Falcon RefinedWeb Dataset

Falcon RefinedWeb
Dataset

A high-quality English web dataset meticulously constructed by the Technology Innovation Institute (TII) in the UAE, containing 968 million records and 600 billion tokens, is the core pre-training data for the Falcon series of large language models, with extreme quality filtering applied to Common Crawl through the MaDR pipeline.

Get the dataset now
968 million records 600 billion tokens ODC-By 1.0 license Falcon training data
📊
968M
Total number of records
📝
600B
Total tokens
🦅
Falcon
Training data for the series of models
📜
ODC-By 1.0
Open license agreement

Dataset Highlights

Industrial-grade web data cleaning pipeline, tailored for pre-training large language models

🌐

Web-scale

Built on Common Crawl, containing 968 million records and 600 billion tokens, large enough to support the pre-training of large language models from scratch.

🔬

MaDR Cleaning Pipeline

Utilizes Macroscopic Data Refinement pipeline, ensuring data quality through content extraction, quality filtering, and deduplication.

🧹

Extreme Deduplication

Employs MinHash + LSH for large-scale fuzzy deduplication, combined with precise substring deduplication, effectively eliminating redundant content and near-duplicate documents.

🔗

URL Filtering

Filters malicious, pornographic, and low-quality domains through blacklists and heuristic rules, controlling the safety and reliability of the dataset from the source.

🌍

Language Recognition

Uses fastText language classifier to accurately filter non-English content, ensuring high language purity of the dataset, suitable for English model training.

💼

Open for Commercial Use

Released under ODC-By 1.0 license, allowing free use and redistribution, including commercial purposes, with attribution required.

Applicable Scenarios

From basic model training to data quality research, covering various NLP applications

🤖

LLM Pre-training

As the core corpus for pre-training large language models from scratch, validated on models such as Falcon-7B/40B/180B

🔍

Data Quality Research

Researching the impact of web data cleaning and filtering strategies on model performance, comparing the effects of different quality pipelines

📏

Filtering Benchmark Testing

As a benchmark dataset for data filtering and deduplication algorithms, evaluating and optimizing the performance of data cleaning pipelines

🏗️

Basic Model Training

Providing high-quality English corpus for custom basic models, supporting various architectures and training paradigms

NLP pre-training Falcon web-crawl high-quality

API Call Example

Quickly obtain Falcon RefinedWeb data through the Ace Data Cloud API

BASH
curl -X GET "https://api.acedata.cloud/datasets/falcon-refinedweb" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json"
# Response Example
{
  "content": "The history of artificial intelligence began in antiquity, with myths, stories and rumors of artificial beings endowed with intelligence...",
  "url": "https://en.wikipedia.org/wiki/History_of_artificial_intelligence",
  "timestamp": "2023-06-15T08:22:31Z",
  "token_count": 1247
}

3 Steps to Get Started Quickly

From browsing to obtaining data, you can start your model training project in just a few minutes

01

Browse the Dataset

View the details of the Falcon RefinedWeb dataset on the Ace Data Cloud platform to understand metadata such as scale, fields, and licensing agreements.

02

Obtain API Token

Register and create an API Token to access data on demand via the RESTful API, without needing to download the entire dataset.

03

Load and Train

Use the data obtained via the API directly for model pre-training, data analysis, or quality research, ready to use out of the box.

Get the Dataset

Start Exploring Falcon RefinedWeb Data

Industry-grade web pre-training corpus with open commercial licensing, available via API immediately. Whether you are training a base model or researching data quality, this dataset is an ideal choice.

Get the Dataset Now