Stats about Common Voice: Kabyle Corpus

Common Voice
#1

I’m sharing with the kab contributors on our FB page some stats about the kab corpus. The corpus was analyzed after tokenization and pos tagging using NLTK Perceptron Tagger. I used a model I have already generated from another corpus.

For graphs and networks I used: matplotlib, numpy, networkx and pylab

I analyzed:
Word length
Sentence lenght
Grammatical classes (tags)
Punctuation VS Alphabet
Verbs/Aspect
Verb occurence
Word Occurence

We use these stats to avoid repetitive words and syntatic forms.

2019-02-07%2012_12_12-Figure%201

2019-02-07%2012_07_55-Figure%201
2019-02-07 12_07_55-Figure 1.png1335×656 15.7 KB
2019-02-07%2012_07_31-Figure%201
2019-02-07 12_07_31-Figure 1.jpg1198×621 28.1 KB
2019-02-07%2012_07_02-Figure%201
2019-02-07 12_07_02-Figure 1.jpg1264×616 45.1 KB
2019-02-07%2012_06_19-Figure%201
2019-02-07 12_06_19-Figure 1.jpg1285×630 35.7 KB
2019-02-07%2012_05_52-Figure%201
2019-02-07%2012_05_29-Figure%201
2019-02-07 12_05_29-Figure 1.jpg1097×605 55.3 KB

#2

Cool! Is this something that can be re-used for other languages? Are the tools to generate this openly available?

Cheers.

#3

Yes but the scripts deal only with Kabyle language. I mean tokenization, POS tag… they are free on Github/Gitlab since months or years :smile :slight_smile: I’ll check if I uploaded the last updates (mozillakab on github and I use mostly French to explain/describe things)