Dolma Open Corpus

Dolma Open
Corpus

Dolma is a large-scale open corpus created by Allen AI, containing 3 trillion tokens, integrating six major data sources: Common Crawl, The Stack, C4, Reddit, Wikipedia, and Semantic Scholar, for training the OLMo series language models, and is currently one of the most transparent large-scale pre-training datasets.

Get the dataset now
3T Tokens 6 major data sources ODC-By 1.0 license OLMo training data
📊
3T
Total tokens
🔗
6
Data sources
📜
ODC-By 1.0
Open license agreement
🤖
OLMo
Target model for training

Dataset Highlights

A large-scale, multi-source, fully transparent open pre-training corpus

📏

Trillions of Tokens

Contains approximately 30 trillion tokens of text data, making it one of the largest publicly available pre-training corpora, providing ample data support for training large language models.

🌐

Six Major Data Sources

Integrates six major sources: Common Crawl web pages, The Stack code, C4 filtered text, Reddit conversations, Wikipedia encyclopedia, and Semantic Scholar academic papers.

🔍

Fully Transparent

Allen AI has made the complete data collection, cleaning, deduplication, and filtering processes public, with each processing step being traceable and auditable, setting a new benchmark for dataset transparency.

🔧

Quality Filtering Pipeline

Employs a multi-level quality filtering pipeline, including language detection, content filtering, deduplication, and toxicity detection steps, ensuring the overall quality of the training data.

🔄

Reproducible Processing

All data processing code is open-sourced on GitHub, allowing researchers to fully reproduce the entire processing flow from raw data to the final corpus.

📖

Open License

Uses the ODC-By 1.0 open data license, allowing free use for academic research and commercial applications, with proper attribution required.

Applicable Scenarios

Empowering the AI community from model training to data science research

🧠

LLM Pre-training

Providing diverse and large-scale text data as core pre-training corpus for training foundational models from scratch

📐

Data Ratio Research

Exploring the optimal mixing ratios of different data sources, studying the impact of web pages, code, encyclopedias, academic papers, etc. on model capabilities

🧪

Ablation Experiments

Systematically studying the independent contributions of each data component to model performance by removing or replacing specific data sources

🔬

Reproducible AI Research

Ensuring that research results are verifiable and reproducible based on completely open data and processing workflows, promoting scientific rigor in the AI field

NLP pre-training OLMo open-source multi-source

API Call Example

Quickly obtain Dolma dataset information through the Ace Data Cloud API

PYTHON
import requests
url = "https://api.acedata.cloud/datasets/dolma"
headers = {
    "Authorization": "Bearer YOUR_API_TOKEN",
    "Accept": "application/json"
}
response = requests.get(url, headers=headers)
data = response.json()
# View basic information of the dataset
print(f"Name: {data['name']}")
print(f"Token Count: {data['tokens']}")
print(f"Data Sources: {data['sources']}")
print(f"License: {data['license']}")

3 Steps to Get Started Quickly

From understanding to usage, quickly start your journey with large model training data

01

Browse the Dataset

View the details of the Dolma dataset on the Ace Data Cloud platform, understand the composition of data sources, token scale, and licensing agreements.

02

Get API Token

Register and obtain your API Token to access the dataset through the api.acedata.cloud interface.

03

Download and Train

Download the required data shards via API and start your pre-training or research experiments with the Dolma corpus for your large language model.

Get the Dataset

Start Exploring the Dolma Open Corpus

30 trillion tokens, 6 major data sources, completely transparent processing flow. Whether you are training the next generation of language models or conducting data science research, Dolma is the ideal choice.

Get the Dataset Now