Qustion about pre-processed data file(s)

[rob-ross]

I downloaded the JSONL file at the bottom of this page: https://kaikki.org/dictionary/English/words/index.html

It's named kaikki.org-dictionary-English-words.jsonl and it's 2.69 GB on my machine.

I've been familiarizing myself with the structure. My immediate focus is accessing plural forms of nouns, and I have figured out how to extract them.

1. I'm a bit confused though on what "files" are part of the dump of all wiktionary data. Do I have a complete dump of the English words and relations, etc, or is this file just a subset? The file I have has a total of 1,351,351 entries (individual JSON objects in the JSONL file). But on the Wiktionary home page it shows there are over English 8,286,000 entries. I'm confused about the discrepancy.

2. Also, what are the "extraction errors and warnings" on this page? : https://kaikki.org/dictionary/English/errors/errors.htmlhttps://kaikki.org/dictionary/English/errors/errors.html

3. At first I thought this page of JSON mappings would be useful: https://kaikki.org/dictionary/errors/mapping/

But the first entry I wanted to look at, the args members in the header_templates, takes me to a page that confuses me.

https://kaikki.org/dictionary/errors/mapping/index/head_templates/list/args.html

At the top, for the entries that show "seen in hern [English];", I cannot find arg key names 10-19 either in the entry for "hern" on wiktionary nor in the JSONL file. The only arg key I find is "1", and that is for Hern with a capital H. So, where are the key names 10-19 for hern coming from?

I'm grateful you have provided this post-processed dump of the wiktionary data. The requirements for creating it myself are daunting:
"Extracting all data for all languages from English Wiktionary takes about 1.25 hours on a 128-core dual AMD EPYC 7702 system. The performance is expected to be approximately linear with the number of processor cores, provided you have enough memory (about 10GB/core or 5GB/hyperthread recommended)."

I'm just using a MacBook pro with a single 6-Core Intel Core i9, with total system memory of 16 GB. So processing the wiki data myself is not possible at this time.

Thanks!

[xxyzz]

Please download raw JSONL files at here: https://kaikki.org/dictionary/rawdata.html , it contains data extracted from all pages.

You can ignore the error page. And also ignore all "*_templates" fields, inflection form data are in the "forms" field.

[rob-ross]

What's the difference between this file file and the one at the bottom of this page: https://kaikki.org/dictionary/English/index.html ?

[xxyzz]

The file in your posted link only have English words, it's 2.5GB. The file contains all pages(words in all languages) is 18.5GB.

[rob-ross]

That makes sense. The English-only file is about 3 GB and the other one is about 20 GB.

Thank you!

• Rob