Here is how to process large marathi corpus file to find new words those can be included in the dictionary.
1) Download the data file from:
https://drive.google.com/file/d/1UjZ-X2S77AQyCkHqw2mFXRWYf9WOZS0m/
You will get the reference from MarathiNLP ( Read more... )

apt install ibus-typing-booster
wget https://kagapa.s3.ap-south-1.amazonaws.com/with_acor_N.oxt
unzip with_acor_N.oxt
cp dicts/mr_IN.* /home/ubuntu/.local/share/ibus-typing-booster/data/

These are the 3 commands to use in order to collect unique words used by a website.
time wget -r -p -U Mozilla --wait=1 --limit-rate=350K https://www.weeklysadhana.in/
find /home/ubuntu/myweb/newweb/www.weeklysadhana.in/ -type f -exec cat {} \; > collect.txt
/root/.cargo/bin/rg   -o '\w+' ( Read more... )

def isvalid(myword):
  for y in myword:
      for i in y:
          x = i.encode("raw_unicode_escape")
          if x[:5][-3:] not in [b"090", b"091", b"092", b"093", b"094"]:
              return
          if x[:6][-4:].upper() in [ b"0900", b"0904", b"090E", b"0912", b"0929", b"0934", b"093A", b"093B", b"0946", b"094A", b"094E", b"094F"]:
( Read more... )

import sys
sys.stdout = open("exclude.txt", "w")
mylist = ["्", "ा", "ि", "ी", "ु", "ू", "े", "ै", "ॅ"]
for i in mylist:
for y in mylist:
    print (i + y)
# grep -vf exclude.txt my_corpus2.txt > clean_corpus.txt
# grep -vE "^्|^ा |^ि|^ी|^ु|^ू|^े|^ै|^ॅ"  valid_word11a.txt > valid_word11b.txt

The following tags needs to be added to verbs:
११
SFX R णे ायचं णे
SFX R णे ायचा णे
SFX R णे ायची णे
SFX R णे ायचे णे
SFX R णे ायचो णे
SFX R णे ायच्या णे
SFX R णे ायला णे
SFX R णे ावं णे
SFX R णे ावयास णे
SFX R णे ावेसे णे
SFX R णे ावासा णे
११
SFX R णे तंय णे
SFX R णे तं णे
SFX R णे ताय णे
SFX R णे तोय णे
SFX R णे तेय णे
SFX R णे त णे
SFX R णे ता णे
SFX R णे तो णे
SFX R ( Read more... )

Here is how to download and analyse the wordlist provided by IIT Mumbai ( Read more... )