Error: Trie file version mismatch (5 instead of expected 4)

Deep Speech
#1

I am trying to train a model on the Common Voice dataset. I have followed the documentation correctly and all the versions are correct.
After the 4th epoch it gives me this error:
Error: Trie file version mismatch (5 instead of expected 4). Update your trie file.
terminate called after throwing an instance of ‘int’
Fatal Python error: Aborted

Thread 0x00007f1eee197700 (most recent call first):
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/threading.py”, line 295 in wait
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/queue.py”, line 164 in get
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/summary/writer/event_file_writer.py”, line 159 in run
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/threading.py”, line 916 in _bootstrap_inner
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/threading.py”, line 884 in _bootstrap

Thread 0x00007f1eee998700 (most recent call first):
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/threading.py”, line 295 in wait
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/queue.py”, line 164 in get
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/summary/writer/event_file_writer.py”, line 159 in run
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/threading.py”, line 916 in _bootstrap_inner
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/threading.py”, line 884 in _bootstrap

Current thread 0x00007f1fde149700 (most recent call first):
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ds_ctcdecoder/swigwrapper.py”, line 279 in init
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ds_ctcdecoder/init.py”, line 30 in init
File “/home/ubuntu/DeepSpeech/DeepSpeech/evaluate.py”, line 48 in evaluate
File “./DeepSpeech.py”, line 672 in test
File “./DeepSpeech.py”, line 939 in main
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/absl/app.py”, line 251 in _run_main
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/absl/app.py”, line 300 in run
File “./DeepSpeech.py”, line 962 in

#2

You have downloaded native client for wrong branch version. Try specifying the branch version like this:

python $basedir/bin/taskcluster.py --target . --branch v0.5.1
and add --arch gpu if needed

#3

@Jendker was mostly right, I think it’s the ctcdecoder that is failing here, though. Please ensure you update to ds_ctcdecoder==0.6.0a15 @Moughees_Ahmed

#4

I ran the command:

python util/taskcluster.py --target . --branch v0.5.1 --arch gpu

The error I got:
Downloading https://community-tc.services.mozilla.com/api/index/v1/task/project. deepspeech.deepspeech.native_client.v0.5.1.gpu/artifacts/public/native_client.ta r.xz …
Traceback (most recent call last):
File “util/taskcluster.py”, line 153, in
main()
File “util/taskcluster.py”, line 147, in main
maybe_download_tc(target_dir=args.target, tc_url=get_tc_url(args.arch, args. artifact, args.branch))
File “util/taskcluster.py”, line 57, in maybe_download_tc
urllib.request.urlretrieve(tc_url, target_file, reporthook=(report_progress if progress else None))
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/urllib/request. py”, line 248, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/urllib/request. py”, line 223, in urlopen
return opener.open(url, data, timeout)
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/urllib/request. py”, line 532, in open
response = meth(req, response)
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/urllib/request. py”, line 642, in http_response
‘http’, request, response, code, msg, hdrs)
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/urllib/request. py”, line 570, in error
return self._call_chain(*args)
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/urllib/request. py”, line 504, in _call_chain
result = func(*args)
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/urllib/request. py”, line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

#5

@lissyx my current decoded is ds-ctcdecoder== 0.6.0a14
How do I update?

#6

Also ran this command:

pip3 install $(python3 util/taskcluster.py --decoder)

Got this error

ERROR: ds_ctcdecoder-0.6.0a15-cp36-cp36m-manylinux1_x86_64.whl is not a supported wheel on this platform.
WARNING: You are using pip version 19.1.1, however version 19.3.1 is available.

#7

Update managed to install the ds_ctcdecoder v0.6.0a15.

Do I just resume training?

#8

Yes, you should be able to re-start from previous checkpoint if any was written.