Machine Translation datasets from the KAS corpus KAS-MT 1.0

  Avtorji

Žagar, Aleš ; et al.prikaži vse

Žagar, Aleš ; Kavaš, Matic ; Robnik-Šikonja, Marko ; Erjavec, Tomaž ; Fišer, Darja ; Ljubešić, Nikola ; Ferme, Marko ; Borovič, Mladen ; Boškovič, Borko ; Ojsteršek, Milan ; Hrovat, Goran

  Identifikator vnosa

http://hdl.handle.net/11356/1447

 URL projekta

https://nl.ijs.si/kas/

 Dokumentirano v

https://doi.org/10.5281/zenodo.5562228

 Datum objave

2022-02-04

 Vrsta

corpus, text

 Velikost

21902039 words

 Jezik(i)

English , Slovenian

 Opis

The Machine Translation datasets KAS-MT 1.0 contain automatically sentence-aligned Slovene and English plain-text abstracts from KAS-Abs 2.0 (http://hdl.handle.net/11356/1449) and is meant for studies in machine translation. The setence alignment approach used requires an alignment reliability threshold that omits candidate pairs below a certain value. This value represents a trade-off between the quantity and quality of aligned pairs. We estimate that the default threshold value produces a good-quality dataset for most users. We release three such datasets (files) that reflect a trade-off between quality and quantity of the data. The Normal dataset uses the default reliability threshold and contains 496,102 sentence pairs, the Strict dataset 474,852 sentence pairs, and the Very Strict dataset 425,534 sentence pairs. A file with thesis metadata is also included. The first column in each of the three TSV files gives the confidence that the alignment is correct (higher is better), the second and third are the source and target Slovene and English sentences, while the fourth gives the “merged” state, i.e. whether sentences in the source or target language were merged (sentences do not always exhibit one-to-one mapping). The last column gives the thesis ID. Reference: Žagar, A., Kavaš, M., & Robnik Šikonja, M. (2021). Corpus KAS 2.0: cleaner and with new datasets. In Information Society - IS 2021: Proceedings of the 24th International Multiconference. https://doi.org/10.5281/zenodo.5562228

 Izdajatelj

Faculty of Electrical Engineering and Computer Science, University of Maribor

 

Faculty of Computer and Information Science, University of Ljubljana

 Zahvala

Jožef Stefan Institute CLARIN "CLARIN.SI"

Ministry of Culture C3340-20-278001 "Development of Slovene in a Digital Environment"

European Union EC/H2020/825153 "EMBEDDIA - Cross-Lingual Embeddings for Less-Represented Languages in European News Media"

ARRS (Slovenian Research Agency) J6-2581 "Računalniško podprta večjezična analiza novičarskega diskurza s kontekstualnimi besednimi vložitvami"

 Ključne besede

academic writing PhD theses MSc/MA theses BSc/BA theses machine translation

 Zbirke

CLARIN.SI data & tools

Prikaži polni zapis vnosa