FineWeb is a large-scale high-quality English web text dataset released by Hugging Face, containing 15 trillion tokens, selected from 96 Common Crawl snapshots, specifically designed for pre-training large language models. It employs a multi-stage data quality filtering pipeline (URL filtering, text extraction, deduplication, model quality scoring), surpassing similar datasets like C4, Dolma, and RedPajama.
Industrial-grade high-quality corpus specifically designed for pre-training large language models
Contains 150 trillion tokens, making it the largest open English web text dataset currently available, providing ample corpus for training GPT/LLaMA level large language models.
Employs a multi-stage data cleaning process: URL filtering, trafilatura text extraction, MinHash deduplication, model quality scoring, ensuring high quality for each data point.
In LLM pre-training benchmark tests, FineWeb's performance significantly surpasses mainstream open-source datasets such as C4, Dolma, and RedPajama.
Extracted from 96 complete Common Crawl snapshots from 2013 to 2024, covering a wide range of web content and time spans.
Utilizes the Open Data Commons Attribution license, allowing commercial use, modification, and redistribution with proper attribution.
Seamlessly integrates with the Hugging Face datasets library, supporting streaming data loading, allowing training to start without downloading all data.
From model training to academic research, fully supports the LLM development process
Train large language models at GPT and LLaMA levels from scratch, providing massive high-quality English corpus support
Continuously pre-train existing foundational models for domain adaptation, enhancing language understanding capabilities in specific fields
Research the design and effects of data filtering pipelines, exploring the impact of data quality scores on model performance
Compare the impact of different dataset qualities on model performance, providing experimental basis for dataset selection
The following is a JSONL format example of the FineWeb dataset, with each line containing text content, source URL, token count, and quality score
{"text": "Artificial intelligence continues to reshape industries across the globe...", "url": "https://example.com/ai-article", "token_count": 2847, "quality_score": 0.94}
{"text": "The latest advances in renewable energy technology promise...", "url": "https://example.com/energy-tech", "token_count": 1523, "quality_score": 0.91}
{"text": "Understanding neural network architectures requires...", "url": "https://example.com/nn-guide", "token_count": 4102, "quality_score": 0.97}
Quickly integrate FineWeb into your LLM development process from browsing to training
Browse FineWeb datasets on the Ace Data Cloud platform and view details such as metadata, licensing agreements, and data scale.
Stream load data partitions using the Hugging Face datasets library, or download specific CC snapshot subsets on demand.
Feed the data into your tokenizer and training pipeline, supporting mainstream deep learning frameworks such as PyTorch, JAX, and TensorFlow.
150 trillion tokens of high-quality English corpus, open license, available immediately. Whether you are training the next generation of large language models or researching data quality pipelines, FineWeb is your ideal choice.