HeLI-OTS 1.1
Description
HeLI off-the-shelf language identifier with language models for 200 languages.
Usage:
java -jar HeLI.jar <infile> <outfile>
The program will read the <infile> and classify the language of each line as one of the 200 languages it knows
and writes the results, one ISO 639-3 code per line, into file <outfile>.
You can use the -c option to make the program print a confidence score for the identification after each language code.
Usage:
java -jar HeLI.jar -c <infile> <outfile>
If you omit both of the filenames, the program will read the standard input one line at a time and write the result to standard output.
It can identify c. 3000 sentences per second using one core on a 2021 laptop and around 3 gigabytes of memory.
If you use this program in producing scientific publications, please refer to:
@inproceedings{jauhiainen-etal-2017-evaluation,
title = "Evaluation of language identification methods using 285 languages",
author = "Jauhiainen, Tommi and
Lind{\'e}n, Krister and
Jauhiainen, Heidi",
booktitle = "Proceedings of the 21st Nordic Conference on Computational Linguistics",
month = may,
year = "2017",
address = "Gothenburg, Sweden",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/W17-0221",
pages = "183--191",
}
Producing and publishing this software has been partly supported by The Finnish Research Impact Foundation Tandem Industry Academia -funding in cooperation with Lingsoft.
Files
LanguageModels.zip
Files
(84.4 MB)
Name | Size | Download all |
---|---|---|
md5:906450b3f30d05b30933c8b90b618dc6
|
12.0 kB | Download |
md5:793043d696314ee602e6b4af6befb2b2
|
42.1 MB | Download |
md5:bd6df0b5ca41662ab8431fd5195b23a3
|
17.4 kB | Download |
md5:b6b3ddbb88de79542926c9f3a404eb49
|
39 Bytes | Download |
md5:11faca6aa7115d6f7866f1b30df5c174
|
834 Bytes | Download |
md5:8688132bae548bb78618a9890e8c9ea0
|
42.3 MB | Preview Download |
md5:832f4714255dd27f33aa314be43184a7
|
11.4 kB | Download |
md5:ef7547a7a9b4b1421fb4aaf320b9cbde
|
2.4 kB | Preview Download |
Additional details
References
- Jauhiainen, Tommi et al. (2017). Evaluation of language identification methods using 285 languages. https://www.aclweb.org/anthology/W17-0221