SloBENCH evaluation framework

Name: SloBENCH evaluation framework
License: https://opensource.org/licenses/Apache-2.0

Žitnik, Slavko; Dragar, Frenk

SloBENCH evaluation framework

CLARIN.SI data & tools

Authors: Žitnik, Slavko and Dragar, Frenk

Item identifier: http://hdl.handle.net/11356/1469

Project URL: https://github.com/clarinsi/slobench-eval-docker

Demo URL: https://slobench.cjvt.si

Date issued: 2021-11-30

Type: toolService

Description: The evaluation framework contains public evaluation scripts. All the scripts contain additional Dockerfiles that allow for platform-independent evaluation and exact comparison of results. Pre-built Docker images are available in slobench/eval DockerHub repository. The evaluation framework is used and maintained by the SloBENCH leaderboard Web site team. SloBENCH submitters are able to check their compliance of submissions and evaluate theri model on training/validation data prior to submission. The initial version of SloBENCH contains evaluation scripts with examples of training and testing datasets for nine different tasks: named entity recognition, part-of-speech tagging, lemmatization, dependency parsing, semantic role labeling, translation (ENG-SLO, SLO-ENG), summarization and question answering.

Publisher: CLARIN.SI

Subject(s): evaluation SloBENCH Docker

Collection(s): CLARIN.SI data & tools

Show full item record

Files in this item

This item is

Publicly Available

and licensed under:
Apache License 2.0

Name: slobench-eval-docker-main-2021-12-04.zip
Size: 3.7 MB
Format: application/zip
Description: Commit ID: 4e88c0cc2111dc89602
MD5: d3d8f1c98e980aebfc1bed323dbfad62

Download file Preview

File Preview

slobench-eval-docker-main
- .gitignore12 B
- run.py2 kB
- cjvt-logo.png22 kB
- README.md3 kB
- evaluation_scripts
  - eval_prototype
    - submission.zip274 B
    - README.md658 B
    - evaluate.py1 kB
    - ground_truth.zip273 B
    - Dockerfile147 B
  - eval_translation
    - sample_submission.zip283 B
    - .gitignore38 B
    - translation_eval.py2 kB
    - sample_reference.zip241 B
    - README.md1 kB
    - evaluate.py422 B
    - requirements.txt28 B
    - Dockerfile429 B
  - eval_summarization
    - README.md1 kB
    - .gitignore38 B
    - sample_reference.zip1 MB
    - evaluate.py740 B
    - sample_submission.zip1 MB
    - requirements.txt14 B
    - Dockerfile734 B
    - dependencies
      - ROUGE-1.5.5
        XML
        RegExp.pm5 kB
        Handler
        BuildDOM.pm7 kB
        DOM
        Attr.pod2 kB
        Parser.pod2 kB
        Notation.pod874 B
        Comment.pod438 B
        NodeList.pm695 B
        Entity.pod1 kB
        AttDef.pod662 B
        Text.pod1 kB
        ProcessingInstruction.pod802 B
        DOMImplementation.pod665 B
        DOMException.pm1 kB
        ElementDecl.pod529 B
        Node.pod12 kB
        AttlistDecl.pod1 kB
        Element.pod4 kB
        PerlSAX.pm940 B
        Document.pod5 kB
        CDATASection.pod1 kB
        DocumentFragment.pod2 kB
        CharacterData.pod2 kB
        EntityReference.pod1 kB
        NamedNodeMap.pm5 kB
        NodeList.pod1 kB
        DocumentType.pod4 kB
        XMLDecl.pod686 B
        NamedNodeMap.pod3 kB
        DOM.pm108 kB
        ROUGE-1.5.5.pl101 kB
        runROUGE-test.pl2 kB
        RELEASE-NOTE.txt14 kB
        README.txt15 kB
        data
        smart_common_words.txt3 kB
        WordNet-2.0-Exceptions
        buildExeptionDB.pl679 B
        adj.exc22 kB
        verb.exc37 kB
        noun.exc37 kB
        adv.exc85 B
        WordNet-1.6-Exceptions
        buildExeptionDB.pl679 B
        adj.exc19 kB
        verb.exc81 kB
        noun.exc108 kB
        adv.exc85 B
    - summarization_eval.py2 kB
  - eval_question_answering
    - qa_validation.zip35 kB
    - README.md2 kB
    - .gitignore38 B
    - sample_reference.zip1 kB
    - sample_submission.zip1 kB
    - evaluate.py3 kB
    - qa_eval.py7 kB
    - requirements.txt49 B
    - Dockerfile348 B
    - qa_test.zip123 kB
    - qa_train.zip160 kB
  - eval_sequence_tagging_conllu
    - sample_submission.zip384 kB
    - sample_reference.zip509 kB
    - README.md1 kB
    - evaluate.py1 kB
    - conll18_ud_eval.py27 kB
    - Dockerfile258 B
  - eval_sequence_tagging_tab
    - sample_submission.zip50 kB
    - .gitignore38 B
    - sample_reference.zip50 kB
    - README.md1 kB
    - evaluate.py422 B
    - tagging_tab_eval.py1 kB
    - requirements.txt19 B
    - Dockerfile377 B
- clarin-logo.png28 kB
- cjvt-header.png15 kB

SloBENCH evaluation framework

Files in this item

Partners

Partners

Repository