Dataset
RedPajama-Data-1T Dataset
Open 1.2T token pre-training dataset by Together AI reproducing the LLaMA training data recipe, Apache 2.0 licensed.
—Records
—File size
—Delivery formats
—Data updated
RedPajama-Data-1T
RedPajama
Data-1T
RedPajama-Data-1T is an open reproduction version of the LLaMA training dataset created by Together AI, containing 1.2 trillion tokens from seven data sources: CommonCrawl, C4, GitHub, Wikipedia, Books, ArXiv, and StackExchange, licensed under Apache 2.0, supporting transparent and reproducible large language model training.
1.2 trillion tokens
7 major data sources
Apache 2.0 license
Together AI
1.2T
Total tokens
7
Data sources
Apache 2.0
Open license agreement
LLaMA
Training recipe reproduction
Dataset Highlights
Open and transparent trillion-scale pre-training dataset to support research and development of large language models
Trillion Token Scale
Contains 1.2 trillion tokens, fully matching the original training data scale of LLaMA, providing ample data support for pre-training large models.
Seven Data Sources
Covers CommonCrawl, C4, GitHub, Wikipedia, Books, ArXiv, and StackExchange, encompassing diverse fields such as web pages, code, encyclopedias, and academia.
Transparent Processing Workflow
Complete documentation of data processing and filtering pipeline is publicly available, with every operation traceable and auditable, ensuring full transparency of data sources and quality.
Apache 2.0 License
Uses a permissive Apache 2.0 open-source license, supporting academic research and commercial applications without worrying about licensing restrictions.
Quality Filtering
Each data source is processed using domain-specific cleaning rules, including deduplication, language detection, quality scoring, and other multi-dimensional filtering.
Fully Reproducible
Complete methodology and processing workflow have been open-sourced, allowing researchers to reproduce, customize, and extend the dataset to meet their needs.
Applicable Scenarios
Covers various large language model development scenarios from model pre-training to data research
LLM Pre-training
Train large language models from scratch using validated data recipes to replicate LLaMA-level training results
Data Ablation Experiments
Study the impact of different data sources on model performance, quantifying the contribution and importance of data from various fields
Curriculum Learning
Design multi-stage training curricula across data domains, optimizing data mixing ratios and training scheduling strategies
Model Comparison
Conduct fair comparisons of model architectures using standardized training data, eliminating interference from data differences
NLP
pre-training
LLaMA
open-source
trillion-tokens
Quick Start
Quickly access the RedPajama dataset via API
import requests
url = "https://api.acedata.cloud/datasets/redpajama"
headers = {
"Authorization": "Bearer YOUR_API_TOKEN",
"Content-Type": "application/json"
}
params = {
"source": "wikipedia",
"limit": 10
}
response = requests.get(url, headers=headers, params=params)
data = response.json()
# Print the returned data entries
for item in data.get("data", []):
print(item.get("text", "")[:200])
print("---")
3 Steps to Get Started Quickly
From registration to usage, you can start accessing trillion-scale pre-trained data in just a few minutes
01
Register an Account
Register your Ace Data Cloud account at platform.acedata.cloud to quickly complete developer onboarding.
02
Get API Key
Create your API key in the console for authentication and data access authorization.
03
Start Using the Dataset API
Access the RedPajama-Data-1T dataset via the API, querying and downloading pre-trained data from seven major data sources as needed.
Start Exploring the RedPajama Dataset
Open license, trillion-scale, completely transparent. Whether you are training large language models or conducting data research, RedPajama-Data-1T is the ideal choice.
Introduction
Datasets
Pricing