oscar-corpus/mOSCAR

More info can be found here: https://oscar-project.github.io/documentation/versions/mOSCAR/

Paper link: https://arxiv.org/abs/2406.08707

New features:

• Additional filtering steps were applied to remove toxic content (more details in the next version of the paper, coming soon).
• Spanish split is now complete.
• Face detection in images to blur them once downloaded (coordinates are reported on images of size 256 respecting aspect ratio).
• Additional language identification of the documents to improve document-language matching.
• Replace most of Personal Identifiable Information by generic strings.

Previous version remains available, to continue using it:

dataset = load_dataset("oscar-corpus/mOSCAR", revision="v1")

Layout

{
    'images': [{'img_idx': '#000002',
                'sha512': '65c1e5605d48f8753256f758bd442cbdd43e6987691227b1ea6b81430ff36609f46d448c8171546232fe0c258d9e44ce4378f32e8ada5c43c314df5a5e230de2',
                'url': 'https://actuconsommation.fr/wp-content/uploads/2020/05/Disneylands-Japon-1068x712.jpg',
                'faces_loc': [x0, y0, x1, y1]}],
    'metadata': [{'node_order': 'img_#000002|txt_#000000|txt_#000001|txt_#000002|txt_#000003|txt_#000004|txt_#000005|txt_#000006|txt_#000009',
                  'url': 'https://actuconsommation.fr/2020/05/11/disneyland-une-reouverture-sous-haute-securite-a-shanghai-ce-lundi/'}],
    'text': [{'text': 'Disneyland : une réouverture sous haute sécurité à Shanghai ce lundi', 'text_idx': '#000000'},
             {'text': 'Des milliers de visiteurs ont pu pénétrer lundi dans le Disneyland de Shanghai, le premier des six parcs de [...]', text_idx': '#000001'},
             [...] ]
}

Citation

@article{futeral2024moscar,
  title={mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus},
  author={Futeral, Matthieu and Zebaze, Armel and Suarez, Pedro Ortiz and Abadji, Julien and Lacroix, R{\'e}mi and Schmid, Cordelia and Bawden, Rachel and Sagot, Beno{\^\i}t},
  journal={arXiv preprint arXiv:2406.08707},
  year={2024}
}