Albanian-English parallel corpus MaCoCu-sq-en 1.0
In: https://macocu.eu/, 2023
Online
unknown
Zugriff:
The Albanian-English parallel corpus MaCoCu-sq-en 1.0 was built by crawling the “.al” internet top-level domain in 2022, extending the crawl dynamically to other domains as well. All the crawling process was carried out by the MaCoCu crawler (https://github.com/macocu/MaCoCu-crawler). Websites containing documents in both target languages were identified and processed using the tool Bitextor (https://github.com/bitextor/bitextor). Considerable effort was devoted into cleaning the extracted text to provide a high-quality parallel corpus. This was achieved by removing boilerplate and near-duplicated paragraphs and documents that are not in one of the targeted languages. Document and segment alignment as implemented in Bitextor were carried out, and Bifixer (https://github.com/bitextor/bifixer) and BicleanerAI (https://github.com/bitextor/bicleaner-ai) were used for fixing, cleaning, and deduplicating the final version of the corpus. The corpus is available in three formats: two sentence-level formats, TXT and TMX, and a document-level TXT format. TMX is an XML-based format and TXT is a tab-separated format. They both consist of pairs of source and target segments (one or several sentences) and additional metadata. The following metadata is included in both sentence-level formats: - source and target document URL; - paragraph ID which includes information on the position of the sentence in the paragraph and in the document (e.g., “p35:77s1/3” which means “paragraph 35 out of 77, sentence 1 out of 3”); - quality score as provided by the tool Bicleaner AI (a likelihood of a pair of sentences being mutual translations, provided with a score between 0 and 1); - similarity score as provided by the sentence alignment tool Bleualign (value between 0 and 1); - personal information identification (“biroamer-entities-detected”): segments containing personal information are flagged, so final users of the corpus can decide whether to use these segments; - translation direction and machine translation identification ...
Titel: |
Albanian-English parallel corpus MaCoCu-sq-en 1.0
|
---|---|
Autor/in / Beteiligte Person: | Bañón, Marta ; Chichirau, Malina ; Esplà-Gomis, Miquel ; Forcada, Mikel L. ; Galiano-Jiménez, Aarón ; García-Romero, Cristian ; Kuzman, Taja ; Ljubešić, Nikola ; van Noord, Rik ; Pla Sempere, Leopoldo ; Ramírez-Sánchez, Gema ; Rupnik, Peter ; Suchomel, Vít ; Toral, Antonio ; Zaragoza-Bernabeu, Jaume |
Link: | |
Zeitschrift: | https://macocu.eu/, 2023 |
Veröffentlichung: | Jožef Stefan Institute ; Prompsit ; Rijksuniversiteit Groningen ; Universitat d'Alacant, 2023 |
Medientyp: | unknown |
Schlagwort: |
|
Sonstiges: |
|