AlidarAsvarov/dagestan-constitution

Constitution of the Republic of Dagestan — 13 languages

Scans of the Constitution of the Republic of Dagestan in thirteen languages: eleven indigenous languages of Dagestan and the North Caucasus, plus Azerbaijani and Russian. 291 PDF pages across 13 files.

Most PDF pages are two-page book spreads, not single pages. 245 of the 291 are landscape scans of an open book, so the corpus is really 536 book pages. Anyone building an OCR pipeline needs to split them — see Per-file caveats.

The scans now ship with OCR text, manually aligned article by article across all thirteen languages — one wide table, 105 articles, one column per language. See The aligned corpus. The text is machine-generated and unverified for the eleven scanned minority languages; treat it as noisy until a native speaker says otherwise.

Why this dataset

The same legal document appears here in thirteen languages, which makes it a document-aligned parallel corpus. Several of these languages — Agul, Rutul, Tsakhur, Tabasaran, Lak — have very little machine-readable text available anywhere, and almost no OCR ground truth. All of them use Cyrillic orthographies with extended digraphs and letters (гъ, къ, хь, Ӏ, …) that off-the-shelf OCR handles poorly.

The intended uses are OCR training and evaluation for Cyrillic-script minority languages, and — now that the text is aligned at the article level — machine translation and cross-lingual retrieval. For most of these languages this is one of the very few parallel corpora that exists.

Contents

PDF pages is what a reader sees when opening the file. Book pages is what you get after splitting the spreads, and is the number that matters for OCR: it equals 2 × spreads + singles. The singles are covers and title pages.

ISO 639-3	Language	Family	PDF pages	Spreads	Book pages	Size	Source
agx	Agul	Northeast Caucasian (Lezgic)	21	19	40	12.0 MB	scan
ava	Avar	Northeast Caucasian (Avar–Andic)	25	24	49	13.5 MB	scan
azj	North Azerbaijani	Turkic (Oghuz)	17	16	33	10.2 MB	scan
che	Chechen	Northeast Caucasian (Nakh)	21	20	41	11.8 MB	scan
dar	Dargwa	Northeast Caucasian (Dargic)	25	22	47	15.3 MB	scan
kum	Kumyk	Turkic (Kipchak)	21	19	40	11.5 MB	scan
lbe	Lak	Northeast Caucasian (Lak)	22	21	43	12.7 MB	scan
lez	Lezgian	Northeast Caucasian (Lezgic)	21	21	42	34.6 MB	scan
nog	Nogai	Turkic (Kipchak)	22	20	42	12.6 MB	scan
rus	Russian	Indo-European (Slavic)	27	0	27	0.3 MB	born-digital
rut	Rutul	Northeast Caucasian (Lezgic)	22	20	42	12.1 MB	scan
tab	Tabasaran	Northeast Caucasian (Lezgic)	22	20	42	12.7 MB	scan
tkr	Tsakhur	Northeast Caucasian (Lezgic)	25	23	48	14.3 MB	scan

Total: 13 files, 291 PDF pages, 245 spreads, 536 book pages, ~177 MB.

A handful of book pages are blank — the inside covers, and the left half of each title spread, which in several copies holds a library envelope rather than print.

Structure

There are two subsets:

subset	splits	rows	one row is
dag_constitution	train	121	one aligned unit across all 13 languages, one column per language
pdfs	train	13	a PDF: the metadata, and a URL to the scan

dag_constitution is the default, so it is what opens in the Hub preview. It is one wide table: thirteen columns named by ISO 639-3 code (rus, agx, ava, … tkr), and one row per aligned unit — the title, the preamble, each of the eleven chapter headings, the 105 articles, and the closing line, in document order. Alignment is therefore trivial to consume: the languages in a given row are already parallel, so any language pair is just two columns of the same row. A cell is empty where an edition has no counterpart for that unit.

pdfs carries lang, language, family, pdf_pages, spreads, book_pages, text_source, size_bytes, pdf_url. It does not embed the PDFs — it points at them. Fetch the scans themselves with hf download (below).

Note that pdf_pages was called pages in earlier revisions. It was renamed because the number was ambiguous: most PDF pages hold two book pages, so pages meant neither one thing nor the other. book_pages (2 × spreads + singles) is the count you want for OCR.

data/
├── dag_constitution.csv   the alignment, hand-edited (wide: one column per language) — served as-is
└── pdfs.csv               one row per PDF: metadata + a URL to the scan
pdf/
├── metadata.csv    one row per PDF: lang, language, family,
│                   pdf_pages, spreads, book_pages, text_source
├── agx.pdf  ava.pdf  azj.pdf  che.pdf  dar.pdf  kum.pdf  lbe.pdf
├── lez.pdf  nog.pdf  rus.pdf  rut.pdf  tab.pdf  tkr.pdf
└── (one PDF per language, named by ISO 639-3 code)
ocr/
├── pages.jsonl     all 536 book pages, pooled
├── txt/{lang}/     one file per book page, easy to hand-correct
└── docs/{lang}.txt the full text per language
ocr_openrouter.ipynb   scans -> text (vision model, via OpenRouter)
build_hub_data.py      pdf/metadata.csv -> data/pdfs.csv
README.md

data/dag_constitution.csv is served as-is — it is both the editable master and the Hub file. Only data/pdfs.csv is generated (by build_hub_data.py, which adds size_bytes and pdf_url to pdf/metadata.csv). The per-page OCR text under ocr/ is the raw material the alignment was built from; it is kept as a working copy but is not served as a subset.

Loading

from datasets import load_dataset

REPO = "AlidarAsvarov/dagestan-constitution"

# the aligned corpus — default subset, all 13 languages, already parallel
align = load_dataset(REPO, "dag_constitution", split="train")
align[0]["rus"]          # 'КОНСТИТУЦИЯ РЕСПУБЛИКИ ДАГЕСТАН'

# any language pair is two columns of the same row
lez_tab = [(r["lez"], r["tab"]) for r in align if r["lez"] and r["tab"]]

# the scans: metadata and URLs, not the bytes
pdfs = load_dataset(REPO, "pdfs", split="train")
pdfs[0]["pdf_url"]

Both subsets are CSV. datasets infers one file format for the whole repo from the first config, so the two subsets have to share it; the alignment CSV has no multi-line cells, so serving it as CSV is safe. dag_constitution.csv is served exactly as edited — no conversion step.

To fetch the raw PDFs:

hf download AlidarAsvarov/dagestan-constitution --repo-type dataset \
    --include "pdf/*" --local-dir .

or, for everything including the scans:

from huggingface_hub import snapshot_download

path = snapshot_download("AlidarAsvarov/dagestan-constitution", repo_type="dataset")

Both subsets are plain CSV and need nothing but datasets. The PDFs are never loaded by the library, so there is no pdfplumber dependency.

The aligned corpus

The Constitution has 105 articles, and all thirteen editions contain all of them (with one exception, below). Every language marks an article heading the same way — a number next to the local word for article — which is what makes the corpus alignable at all:

Статья 5 (rus, ava, dar, kum) · Статья 5-мур (lbe) · 5-статья (lez) · 5-пу статья (agx) · 5 статья (che, rut) · 5-нши статья (nog) · Маддә 5 (azj) · 5-пи мадда (tab) · МаддаӀ 5 (tkr)

Alignment is manual, done by hand against the scans — and for these languages there was no alternative. LaBSE, LASER, SONAR and vecalign cover none of Agul, Rutul, Tsakhur, Tabasaran, Lak, Dargwa, Avar, Lezgian or Nogai. There is no sentence encoder for them, so the usual bitext-mining stack cannot run here at all. The article headings above are what made a hand-alignment tractable: each unit was matched across editions by its heading, then its text placed in the row.

The columns are the thirteen ISO 639-3 codes — rus, agx, ava, azj, che, dar, kum, lbe, lez, nog, rut, tab, tkr. Rows run in document order — title, preamble, the eleven chapter headings, the 105 articles, closing line — so the canonical Russian article numbering is implicit in that order. A cell is empty where an edition has no counterpart for the unit.

Because every row is already parallel, any of the 78 language pairs is just two columns:

align = load_dataset("AlidarAsvarov/dagestan-constitution", "dag_constitution", split="train")
pairs = [(r["lez"], r["tab"]) for r in align if r["lez"] and r["tab"]]

Alignment issues — the full list

Everything found while aligning by hand, in the order it matters. Nothing here is hypothetical: each item was checked against the Russian text or, where noted, against the scan itself.

1. che is missing an article

The Chechen edition has no counterpart for one Russian article — judicial independence (Russian article 95) is absent from the printed book, not lost in OCR. So che carries 104 of the 105 articles, and its column has exactly one empty cell. In the current alignment the Chechen prosecutor's-office text (which corresponds to Russian article 96) occupies the article-95 row, and the article-96 row is the empty one.

2. azj is a condensed translation, not a close one

The Azerbaijani edition compresses the text — one paragraph where Russian runs three — and is the shortest in the corpus (about 53k characters against Russian's 64k). The articles line up, but within an article the correspondence is loose. Trust it at the article level; read it as a paraphrase, not a sentence-for-sentence rendering.

3. Length outliers that are not misalignments

Several editions omit the Russian amendment annotations — Часть третья утратила силу. — Закон Республики Дагестан от 29.04.2013 N 22. Russian article 76 carries one, so kum, dar and tab look 2–3× shorter there while saying exactly the same thing. Checked by hand; the provisions match. Do not "fix" these.

4. OCR and typesetting quirks in the raw text

These are properties of the per-page OCR text under ocr/, worth knowing before re-deriving anything from it or hand-correcting it:

• Page breaks cut paragraphs in half, often mid-word, with the page number wedged into the gap (…варитта- / 3 / ри кьабул…). The aligned table stitches these back together; the raw ocr/ text does not.
• tkr writes its palochka as І (U+0406), not Ӏ (U+04C0), in some headings — so МаддаӀ and МаддаІ both occur. Normalise it before matching Tsakhur headings.
• The OCR model wrapped some headings in markdown (**Статья 14.**). Strip it before treating a line as a heading.
• Chapter headings belong to no article. Глава 2, 2-ПУ КӀИЛ, ФАСИЛ 2 sit between articles; in the aligned table they occupy their own rows and are kept out of the article text.
• Chapter 11 has no numbered articles. "ЗАКЛЮЧИТЕЛЬНЫЕ И ПЕРЕХОДНЫЕ ПОЛОЖЕНИЯ" follows article 105 and holds only unnumbered provisions.
• Russian puts Статья N in its own paragraph block; every other language inlines it into the first line of the article.

5. A bug in the OCR stage, since fixed

extract_text_layer used plain page.get_text(), which re-wrapped the born-digital Russian raggedly — 41% of lines were fragments, degrading to one word per line in justified passages, and paragraph boundaries were unrecoverable. It now uses page.get_text("blocks"). This was a real defect in the shipped Russian text, and it is fixed in ocr/pages.jsonl, ocr/txt/rus/ and ocr/docs/rus.txt.

How good is it

The alignment is at the article level and was done by hand, so a row pairs the same article across editions; it does not claim the translation is faithful, nor does it align anything below the article. Eight of the twelve translated editions map cleanly onto the Russian numbering with no deviation at all: dar, kum, lbe, lez, nog, rut, tab, tkr. The issues above are confined to azj and che.

azj is a condensed translation, not a close one. It runs one paragraph where Russian runs three and is the shortest text in the corpus. Trust it at article level; read it as a paraphrase. Several editions also omit the Russian amendment annotations ("Часть третья утратила силу…"), which is why their articles can be much shorter than the Russian without anything being misaligned.

Per-file caveats

Several things will bite anyone building a pipeline over this, so they are stated up front rather than left to be discovered.

Most PDF pages are two-page spreads. 245 of the 291 PDF pages are landscape scans of an open book: printed page 8 on the left, page 9 on the right, with a gutter down the middle. The remaining 46 are portrait singles — covers and title pages — plus the 27 pages of rus.pdf. Do not feed a spread to an OCR engine whole. A vision model will often interleave the two pages into one scrambled reading order, and each page gets only half the pixel budget. Split at the gutter first.

The gutter is reliably found as the vertical strip carrying the least ink, searched within a band around the centre so that a wide outer margin cannot win. Across all 245 spreads it sits within a few percent of the midpoint, with one exception worth knowing: on the title spread of ava, che, dar and lbe the left page is entirely blank (it holds a library envelope), so there is no ink minimum to find and the detector drifts. Falling back to the midpoint whenever the detection lands more than ~10% off centre handles those four correctly.

The palochka is printed as a bare vertical stroke. In these editions the palochka is typeset as a plain vertical bar, visually identical to Latin I — Iамал, гIайде. An OCR engine returning I is therefore transcribing the glyph faithfully, not failing.

Normalise it to Ӏ (U+04C0, CYRILLIC LETTER PALOCHKA). This is the capital form, and it is the correct character even inside lowercase words: Dagestani orthographies use the one glyph in both cases. Do not use U+04CF (ӏ, the lowercase palochka) — it is rarely used in practice and will not match text from other sources.

The rewrite is safe when scoped: replacing I, l or 1 with Ӏ only where the character sits inside a Cyrillic word leaves article numbers, years and Глава I untouched.

rus.pdf is not a scan. It is a born-digital export from Microsoft Word (2013), 27 pages, with a real embedded text layer — about 66,000 characters, extractable today with pdftotext rus.pdf -. Every other file is a photograph of a printed page with no text layer whatsoever. So Russian is ground truth, not OCR output, and must not be pooled with OCR'd text without a flag distinguishing the two. It is useful as a structural reference (article numbering, section breaks) for validating an OCR pipeline before trusting it on Agul — and, because the same document exists here as both image and text, as a way to measure a candidate OCR model's character error rate before committing to it.

lez.pdf has /Rotate 270 on its pages. Renderers that honour the rotation entry produce upright spreads; those that ignore it produce sideways ones. It is also physically distinct from the rest: 34.6 MB (roughly triple the others), scanned in October 2024, with no Creator metadata — it predates the rest of the collection and was made with different equipment. It is the only file with no portrait pages at all.

The other eleven were captured with Adobe Scan for iOS in June 2025 and carry its automatic cropping, deskewing, and contrast enhancement. That processing is not reversible from these files; if you need the unprocessed camera images, they are not part of this release.

Provenance

All scans were made by Alidar Asvarov from printed copies held by the Russian State Library (Российская государственная библиотека), Moscow. No third-party scan, image, or digitisation was used.

The specific print edition (publisher, year, shelf mark) is not recorded.

The Russian PDF is a born-digital copy of the official text; it was not produced by the dataset author.

The Constitution of the Republic of Dagestan was adopted by the Constitutional Assembly of the Republic on 10 July 2003.

Licensing

Released under CC0 1.0 — public domain dedication. Two separate things are being licensed, and both are clear:

• The underlying text. Official documents of state bodies, including their official translations, are excluded from copyright protection under Article 1259(6) of the Civil Code of the Russian Federation. The constitutional text and its official translations into the languages of Dagestan are therefore not subject to copyright.
• The scans. Made by the dataset author, who dedicates them to the public domain under CC0. Note that under Russian law a faithful reproduction of a public-domain work does not attract a new copyright, so this dedication mostly serves to remove doubt rather than to waive a right that clearly existed.

Limitations

The OCR text is machine-generated and unverified for eleven of the twelve scanned languages; treat it as noisy until a native-speaker pass says otherwise. Its character error rate has not been measured — rus.pdf exists as both scan and ground-truth text, so it can be, and should be before anyone trusts the text at the character level.

The alignment says two texts are the same article. It does not say the translation is faithful, and it stops at the article level — there is no alignment of paragraphs or sentences below it. It was done by hand against the headings and the Russian text, not by anyone who reads Agul, so it vouches for which article a cell holds, not for the quality of the translation inside it.

Page counts differ across languages (33 to 49 book pages) because typesetting differs, not because content is missing. That is now verified article by article: every edition carries all 105 articles, except che, which is missing Russian article 95.

This is a single legal document in a formal register. It is not a general-purpose language sample, and a model trained only on it will reflect the vocabulary and syntax of constitutional prose.

Citation

@misc{asvarov2026dagestan,
  title        = {Constitution of the Republic of Dagestan: scans in 13 languages},
  author       = {Asvarov, Alidar},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/AlidarAsvarov/dagestan-constitution}}
}