Machine Translation datasets from the KAS corpus KAS-MT 1.0

  Authors

Žagar, Aleš ; et al.show everyone

Žagar, Aleš ; Kavaš, Matic ; Robnik-Šikonja, Marko ; Erjavec, Tomaž ; Fišer, Darja ; Ljubešić, Nikola ; Ferme, Marko ; Borovič, Mladen ; Boškovič, Borko ; Ojsteršek, Milan ; Hrovat, Goran

  Item identifier

http://hdl.handle.net/11356/1447

 Project URL

https://nl.ijs.si/kas/

 Referenced by

https://doi.org/10.5281/zenodo.5562228

 Date issued

2022-02-04

 Type

corpus, text

 Size

21902039 words

 Language(s)

English , Slovenian

 Description

The Machine Translation datasets KAS-MT 1.0 contain automatically sentence-aligned Slovene and English plain-text abstracts from KAS-Abs 2.0 (http://hdl.handle.net/11356/1449) and is meant for studies in machine translation. The setence alignment approach used requires an alignment reliability threshold that omits candidate pairs below a certain value. This value represents a trade-off between the quantity and quality of aligned pairs. We estimate that the default threshold value produces a good-quality dataset for most users. We release three such datasets (files) that reflect a trade-off between quality and quantity of the data. The Normal dataset uses the default reliability threshold and contains 496,102 sentence pairs, the Strict dataset 474,852 sentence pairs, and the Very Strict dataset 425,534 sentence pairs. A file with thesis metadata is also included. The first column in each of the three TSV files gives the confidence that the alignment is correct (higher is better), the second and third are the source and target Slovene and English sentences, while the fourth gives the “merged” state, i.e. whether sentences in the source or target language were merged (sentences do not always exhibit one-to-one mapping). The last column gives the thesis ID. Reference: Žagar, A., Kavaš, M., & Robnik Šikonja, M. (2021). Corpus KAS 2.0: cleaner and with new datasets. In Information Society - IS 2021: Proceedings of the 24th International Multiconference. https://doi.org/10.5281/zenodo.5562228

 Publisher

Faculty of Electrical Engineering and Computer Science, University of Maribor

 

Faculty of Computer and Information Science, University of Ljubljana

 Acknowledgement

Jožef Stefan Institute CLARIN "CLARIN.SI"

Ministry of Culture C3340-20-278001 "Development of Slovene in a Digital Environment"

European Union EC/H2020/825153 "EMBEDDIA - Cross-Lingual Embeddings for Less-Represented Languages in European News Media"

ARRS (Slovenian Research Agency) J6-2581 "Računalniško podprta večjezična analiza novičarskega diskurza s kontekstualnimi besednimi vložitvami"

 Subject(s)

academic writing PhD theses MSc/MA theses BSc/BA theses machine translation

 Collection(s)

CLARIN.SI data & tools

Show full item record