Common Crawl Dataset

Common Crawl
Open Source Web Crawling Dataset

Common Crawl is a large-scale open-source web crawling dataset that provides raw data, metadata, and text extracts from billions of web pages every month, widely used for NLP, search engines, and large language model pre-training research. It includes 3 subsets: Full, Simplified Chinese, Traditional Chinese.

Get the Dataset Now
Billions of Pages/Month 3 Subsets Since 2008 Open License
Common Crawl Dataset
🌐
Billions
Pages Crawled Monthly
📂
3
Data Subsets
📅
2008
Project Start Year
🔓
CC0
Open License Agreement

Dataset Highlights

Open data on an internet scale, driving the next generation of artificial intelligence research

🏗️

Ultra-large Scale

Continuously crawled the internet since 2008, collecting billions of web pages each month, with a cumulative data volume reaching PB level, making it one of the largest publicly available web datasets in the world.

📦

Multi-format Support

Provides three standard formats: WARC (original web pages), WAT (structured metadata), and WET (plain text extraction), to meet different research and engineering needs.

🇨🇳

Chinese Subset

Specifically divided into simplified Chinese and traditional Chinese data partitions, providing high-quality native language corpus for Chinese NLP research and large language model training.

🆓

Free and Open

No registration or API key application required, data hosted on AWS S3, anyone can directly download and use it, truly public domain data.

🧠

NLP Cornerstone

The pre-training corpus for many large language models such as GPT, BERT, and T5 comes from Common Crawl, making it an indispensable data foundation for modern NLP research.

🔄

Regular Updates

New crawled data is released monthly, with consistent formatting, making it convenient for researchers to continuously access the latest internet content for temporal analysis and model iteration.

Applicable Scenarios

From model training to academic research, covering various data-driven applications

🤖

Pre-training of Large Language Models

As a core training data source for large language models like GPT, LLaMA, and BLOOM, providing a vast amount of multilingual text corpus

📝

Web-level NLP

Supports large-scale natural language processing tasks such as language detection, named entity recognition, and sentiment analysis

🔍

Search Engine Research

Used to build and test web search algorithms, webpage ranking models, and information retrieval systems

🀄

Chinese Language Processing

Utilizing subsets of Simplified and Traditional Chinese for research in Chinese word segmentation, text classification, and semantic understanding

Web Crawling NLP Pre-training Large Language Models Chinese Corpus Open Source Data

Data Preview

The following is a header example of WARC format data, showing the structure of a single web page crawl record

WARC
WARC/1.0
WARC-Type: response
WARC-Date: 2024-03-15T08:23:45Z
WARC-Target-URI: https://example.com/article
Content-Type: application/http; msgtype=response
Content-Length: 28456
WARC-Record-ID: <urn:uuid:a1b2c3d4-e5f6-7890-abcd-ef1234567890>
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
<!DOCTYPE html>
<html lang="zh-CN">
<head><title>Example Web Page</title></head>
...

3 Steps to Get Started Quickly

From browsing to usage, you can start your data research project in just a few minutes

01

Browse Datasets

View dataset details on the Ace Data Cloud platform and select the desired subset: Full, Simplified Chinese, or Traditional Chinese.

02

Download or Stream

Download or stream WARC, WAT, WET format files, with data hosted on high-speed storage, supporting resume downloads.

03

Parse and Use

Use tools like warcio to parse the data and extract text content for NLP pipelines or model training.

Get Datasets

Start Exploring Common Crawl Data

An open dataset at internet scale, free to use, and regularly updated. Whether you are training large language models or conducting web data analysis, Common Crawl is an ideal starting point for your data.

Get Datasets Now