meryyllebr543/stack-edu-huggingface

💻 Processed Stack-Edu (Direct Access)

This repository contains a pre-downloaded and processed version of the Stack-Edu dataset. The original dataset consists of SWHIDs pointing to files on Software Heritage. This version contains the full text content, saved in Parquet format and organized by programming language, making it immediately usable for training and analysis without the slow download process.

Stack-Edu is a 125B token dataset of high-quality, educational code filtered from The Stack v2. It was curated by the Hugging Face team for training the StarCoder2 models.

Dataset Structure

The data is organized into one Parquet file per programming language. Each language is available as a separate configuration (subset).

Available Languages (Configurations)

You can load data for any of the following languages by specifying its name in the load_dataset function: c, cpp, csharp, go, java, javascript, markdown, php, python, ruby, rust, shell, sql, swift, typescript.

How to Use

You can easily load any language-specific subset using the datasets library. The dataset card metadata in this README.md allows for direct loading of each configuration.

from datasets import load_dataset

# Load a specific language, for example, Python
# Note: It is recommended to use streaming=True for large datasets to avoid downloading everything at once.
python_data = load_dataset("meryyllebr543/stack-edu-huggingface", name="python", split="train", streaming=True)

# Load the C++ data
cpp_data = load_dataset("meryyllebr543/stack-edu-huggingface", name="cpp", split="train")

# You can then iterate through the dataset
for example in python_data.take(5):
    print(example['text'][:200]) # Print the first 200 characters of the text

Original Dataset Information

For more details on the curation process, please refer to the original Stack-Edu dataset card.

Citation

If you use this dataset, please cite the original work:

@misc{allal2025smollm2smolgoesbig,
      title={SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model},
      author={...},
      year={2025},
      eprint={2502.02737},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}