Japanese web corpus with difficulty levels jpWaC-L 1.0

  Authors

Erjavec, Tomaž ; Hmeljak Sangawa, Kristina and Kawamura, Yoshiko

  Item identifier

http://hdl.handle.net/11356/1047

 Project URL

https://nl.ijs.si/jaslo/index-en.html#jpwac

 Date issued

2008-11-14

 Type

corpus, text

 Size

409030315 tokens

 Language(s)

Japanese

 Description

The corpus contains over 300 million words, with annotations of words and sentences describing their difficulty levels. Words are assigned levels of difficulty according to the Japanese Language Proficiency Test Content Specifications (2004). The difficulty level of the sentences is computed using various heuristics, based on the (difficulty level of) words, sentence length, etc. We distinguish 5 difficulty levels, from L0 (very difficult) to L4 (very easy). The corpus was collected from the Web using WaCkY tools, part-of-speech tagged and lemmatised with Chasen. The Japanese Chasen tags have also been converted to English language based tags. The corpora are made available in vertical format. Structural attributes are <text> and <s> (sentence). Each text gives its @url and @domain. Sentences have the @level attribute, which describes their difficulty level. The positional attributes are: 1. token, as it appears in the text 2. lemma of the word 3. Chasen tag, translated to English 4. original Chasen tag in Japanese 5. difficulty level of the word. The complete corpus is also split into sub-corpora of sentences with the same difficulty level.

 Publisher

Jožef Stefan Institute

 Acknowledgement

Japan Foundation Japanese-Language Education Fellowship "(Kristina Hmeljak Sangawa - March to July 2005)"

ARRS/JSPS Slovenia-Japan bilateral project "(Tomaž Erjavec - November 2008)"

 Subject(s)

difficulty level teaching corpus TEI

 Collection(s)

CLARIN.SI data & tools

Show full item record