Corpus of Slovenian periodicals (1771-1914) sPeriodika 1.0

Name: Corpus of Slovenian periodicals (1771-1914) sPeriodika 1.0
License: https://creativecommons.org/licenses/by-sa/4.0/

Dobranić, Filip; Evkoski, Bojan; Ljubešić, Nikola

Corpus of Slovenian periodicals (1771-1914) sPeriodika 1.0

CLARIN.SI data & tools

Avtorji: Dobranić, Filip ; Evkoski, Bojan and Ljubešić, Nikola

Identifikator vnosa: http://hdl.handle.net/11356/1881

URL projekta: https://www.inz.si/en/dihur/

Dokumentirano v: https://doi.org/10.5281/zenodo.13936418
https://aclanthology.org/2024.lrec-main.61/

Datum objave: 2023-12-20

Vrsta: corpus, text

Velikost: 910064957 tokens, 708306576 words, 50904086 sentences, 149391 texts

Jezik(i): Slovenian

Opis: The corpus of Slovenian periodicals sPeriodika contains linguistically annotated periodicals published during the 18th, 19th, and beginning of 20th century (1771-1914). The periodical issues were retreived from Slovenia's national library's digital library service (https://dlib.si) in the form of OCR-ed PDF and TXT files. Before linguistically annotating the documents (lemmatisation, part-of-speech tagging, and named entity recognition) with CLASSLA-Stanza (https://github.com/clarinsi/classla), the OCR-ed texts were corrected with a lightweight and robust approach using cSMTiser (https://github.com/clarinsi/csmtiser), a text normalisation tool based on character-level machine translation. This OCR post-correction model was trained on a set of manually corrected samples (300 random paragraphs at least 100 characters in length) from the original texts, cf. http://hdl.handle.net/11356/1907. The documents in the collection are enriched with the following metadata obtained from dLib: - Document ID (URN) - Periodical name - Document (periodical issue) title - Volume number (if available) - Issue number (if available) - Year of publication - Date of publication (of varying granularity, based on original metadata available) - Source (URL of the original digitised document available at dlib.si) - Image (see below) - Quality (see below) The majority of documents are pagewise aligned with the scanned images of original prints. Using a concordancer the metadata allows for a single-click route to the image of the page in question for further investigation and checking of the OCR results. In other cases, only links to full document PDFs are provided. For custom use without a concordancer, the images are available at https://nl.ijs.si/inz/speriodika/ with individual files having the form <document_id>-<page_number>.jpg. In addition to metadata published by dLib, the corpus contains estimates of the quality of OCR-ed text for individual pages. Pages are classified as "low" quality when mistakes are common and the text itself is mostly suitable for close reading. Otherwise, with pages appropriate for distant reading tasks, the quality metadatum is set to "good". The corpus is available in vertical format with linguistic annotations, as well as JSON files, which contain the corpus texts in all stages of processing - see the sample JSON for README explaining the format and a sample file.