Installation for Training/Transfer Learning Issues

personal.dummy · September 26, 2019, 3:02pm

I have been having a very hard time setting up DeepSpeech for training/transfer learning purposes. I’ve spent days trying to resolve this particular error, reading and attempting all sorts of solutions from various threads of Github and Discourse with no luck.

The current error message I get when trying to run ./bin/run-ldc93s1.sh is:

(deepspeech-venv) lee@lee-VirtualBox:~/tmp/DeepSpeech$ ./bin/run-ldc93s1.sh

[ ! -f DeepSpeech.py ]
[ ! -f data/ldc93s1/ldc93s1.csv ]
[ -d ]
python -c from xdg import BaseDirectory as xdg; print(xdg.save_data_path(“deepspeech/ldc93s1”))
checkpoint_dir=/home/lee/.local/share/deepspeech/ldc93s1
export CUDA_VISIBLE_DEVICES=0
python -u DeepSpeech.py --noshow_progressbar --train_files data/ldc93s1/ldc93s1.csv --test_files data/ldc93s1/ldc93s1.csv --train_batch_size 1 --test_batch_size 1 --n_hidden 100 --epochs 2 --checkpoint_dir /home/lee/.local/share/deepspeech/ldc93s1

WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:429: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, use
tf.py_function, which takes a python function which manipulates tf eager
tensors instead of numpy arrays. It’s easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means tf.py_functions can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:358: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/contrib/rnn/python/ops/lstm_ops.py:696: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
I Restored variables from most recent checkpoint at /home/lee/.local/share/deepspeech/ldc93s1/train-402, step 402
I STARTING Optimization
I Training epoch 0…
I Finished training epoch 0 - loss: 0.223775
I Training epoch 1…
I Finished training epoch 1 - loss: 0.174103
I FINISHED optimization in 0:00:01.358771
Loading the LM will be faster if you build a binary file.
Reading data/lm/lm.binary
----5—10—15—20—25—30—35—40—45—50—55—60—65—70—75—80—85—90—95–100
terminate called after throwing an instance of ‘lm::FormatLoadException’
what(): …/kenlm/lm/read_arpa.cc:65 in void lm::ReadARPACounts(util::FilePiece&, std::vector&) threw FormatLoadException.
first non-empty line was “version https://git-lfs.github.com/spec/v1” not \data. Byte: 43
Aborted (core dumped)

I’ve tried re-cloning the repository as suggested in: https://github.com/mozilla/DeepSpeech/issues/860, but I’m still running into the error.

Please help, what am I doing wrong? I’ve retried so many times to set this up, I have so many virtualbox instances and they eventually all come to this same or related error. (AttributeError: module ‘tensorflow._api.v1.data’ has no attribute ‘get_output_types’)

lissyx · September 26, 2019, 4:38pm

It looks like your git-lfs setup is wrong. We can’t help with “i’ve done what is documented”.

reuben · September 26, 2019, 4:58pm

As for this error, I’m pretty sure it should only happen if you’re running TensorFlow 2.0. Make sure you’re running the documented version of TensorFlow. Current master uses 1.14.

dabinat · September 26, 2019, 6:25pm

Also remember to use the Mozilla fork of Tensorflow.

personal.dummy · September 26, 2019, 6:27pm

First, I just want to thank you two for offering so much of your time to help the community, really do appreciate it.

@lissyx, you’re right. Regarding git-lfs, I’ve been able to get through that issue by following your suggestion on https://github.com/mozilla/DeepSpeech/issues/871. Regarding your second comment, I agree. I’ve taken detailed notes on every command executed during the setup, I’ll gladly share – needs to be cleaned up a bit.

@reuben, you’re right. I tried your suggestion to switch to tensorflow 1.14, and moved past that issue. It was a little confusing because in https://github.com/mozilla/DeepSpeech/issues/2297 it was discussed to downgrade to 1.13.1.

With those two issues resolved, I’ve now run into this:

#####################################
(deepspeech-venv) lee@lee-VirtualBox:~/tmp/DeepSpeech$ ./bin/run-ldc93s1.sh

[ ! -f DeepSpeech.py ]
[ ! -f data/ldc93s1/ldc93s1.csv ]
[ -d ]
python -c from xdg import BaseDirectory as xdg; print(xdg.save_data_path(“deepspeech/ldc93s1”))
checkpoint_dir=/home/lee/.local/share/deepspeech/ldc93s1
export CUDA_VISIBLE_DEVICES=0
python -u DeepSpeech.py --noshow_progressbar --train_files data/ldc93s1/ldc93s1.csv --test_files data/ldc93s1/ldc93s1.csv --train_batch_size 1 --test_batch_size 1 --n_hidden 100 --epochs 2 --checkpoint_dir /home/lee/.local/share/deepspeech/ldc93s1

WARNING:tensorflow:From DeepSpeech.py:836: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

WARNING:tensorflow:From /home/lee/tmp/DeepSpeech/util/config.py:60: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0925 13:53:28.754550 140202457278272 deprecation_wrapper.py:119] From /home/lee/tmp/DeepSpeech/util/config.py:60: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From DeepSpeech.py:818: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

W0925 13:53:28.759094 140202457278272 deprecation_wrapper.py:119] From DeepSpeech.py:818: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

WARNING:tensorflow:From DeepSpeech.py:819: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

W0925 13:53:28.760723 140202457278272 deprecation_wrapper.py:119] From DeepSpeech.py:819: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.

Instructions for updating:

tf.py_func is deprecated in TF V2. Instead, there are two

options available in V2.

tf.py_function takes a python function which manipulates tf eager

tensors instead of numpy arrays. It’s easy to convert a tf eager tensor to

an ndarray (just call tensor.numpy()) but having access to eager tensors

means tf.py_functions can use accelerators such as GPUs as well as

being differentiable using a gradient tape.

tf.numpy_function maintains the semantics of the deprecated tf.py_func

(it is not differentiable, and manipulates numpy arrays). It drops the

stateful argument making all functions stateful.

W0925 13:53:28.790332 140202457278272 deprecation.py:323] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.

Instructions for updating:

tf.py_func is deprecated in TF V2. Instead, there are two

options available in V2.

tf.py_function takes a python function which manipulates tf eager

tensors instead of numpy arrays. It’s easy to convert a tf eager tensor to

an ndarray (just call tensor.numpy()) but having access to eager tensors

means tf.py_functions can use accelerators such as GPUs as well as

being differentiable using a gradient tape.

tf.numpy_function maintains the semantics of the deprecated tf.py_func

(it is not differentiable, and manipulates numpy arrays). It drops the

stateful argument making all functions stateful.

WARNING:tensorflow:From /home/lee/tmp/DeepSpeech/util/feeding.py:45: The name tf.read_file is deprecated. Please use tf.io.read_file instead.

W0925 13:53:28.843681 140202457278272 deprecation_wrapper.py:119] From /home/lee/tmp/DeepSpeech/util/feeding.py:45: The name tf.read_file is deprecated. Please use tf.io.read_file instead.

WARNING:tensorflow:From DeepSpeech.py:372: The name tf.data.Iterator is deprecated. Please use tf.compat.v1.data.Iterator instead.

W0925 13:53:28.932677 140202457278272 deprecation_wrapper.py:119] From DeepSpeech.py:372: The name tf.data.Iterator is deprecated. Please use tf.compat.v1.data.Iterator instead.

WARNING:tensorflow:From DeepSpeech.py:372: DatasetV1.output_types (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.

Instructions for updating:

Use tf.compat.v1.data.get_output_types(dataset).

W0925 13:53:28.935590 140202457278272 deprecation.py:323] From DeepSpeech.py:372: DatasetV1.output_types (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.

Instructions for updating:

Use tf.compat.v1.data.get_output_types(dataset).

WARNING:tensorflow:From DeepSpeech.py:373: DatasetV1.output_shapes (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.

Instructions for updating:

Use tf.compat.v1.data.get_output_shapes(dataset).

W0925 13:53:28.937879 140202457278272 deprecation.py:323] From DeepSpeech.py:373: DatasetV1.output_shapes (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.

Instructions for updating:

Use tf.compat.v1.data.get_output_shapes(dataset).

WARNING:tensorflow:From DeepSpeech.py:374: DatasetV1.output_classes (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.

Instructions for updating:

Use tf.compat.v1.data.get_output_classes(dataset).

W0925 13:53:28.938873 140202457278272 deprecation.py:323] From DeepSpeech.py:374: DatasetV1.output_classes (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.

Instructions for updating:

Use tf.compat.v1.data.get_output_classes(dataset).

WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.

Instructions for updating:

Use tf.compat.v1.data.get_output_types(iterator).

W0925 13:53:28.943069 140202457278272 deprecation.py:323] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.

Instructions for updating:

Use tf.compat.v1.data.get_output_types(iterator).

WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.

Instructions for updating:

Use tf.compat.v1.data.get_output_shapes(iterator).

W0925 13:53:28.946621 140202457278272 deprecation.py:323] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.

Instructions for updating:

Use tf.compat.v1.data.get_output_shapes(iterator).

WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.

Instructions for updating:

Use tf.compat.v1.data.get_output_classes(iterator).

W0925 13:53:28.948177 140202457278272 deprecation.py:323] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.

Instructions for updating:

Use tf.compat.v1.data.get_output_classes(iterator).

WARNING:tensorflow:From DeepSpeech.py:385: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

W0925 13:53:28.950698 140202457278272 deprecation_wrapper.py:119] From DeepSpeech.py:385: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From DeepSpeech.py:208: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

W0925 13:53:28.959286 140202457278272 deprecation_wrapper.py:119] From DeepSpeech.py:208: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.

Instructions for updating:

Call initializer instance with the dtype argument instead of passing it to the constructor

W0925 13:53:29.114466 140202457278272 deprecation.py:506] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.

Instructions for updating:

Call initializer instance with the dtype argument instead of passing it to the constructor

WARNING:tensorflow:Entity <bound method LSTMBlockWrapper.call of <tensorflow.contrib.rnn.python.ops.lstm_ops.LSTMBlockFusedCell object at 0x7f83380b0ef0>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: converting <bound method LSTMBlockWrapper.call of <tensorflow.contrib.rnn.python.ops.lstm_ops.LSTMBlockFusedCell object at 0x7f83380b0ef0>>: AttributeError: module ‘gast’ has no attribute ‘Num’

W0925 13:53:29.205372 140202457278272 ag_logging.py:145] Entity <bound method LSTMBlockWrapper.call of <tensorflow.contrib.rnn.python.ops.lstm_ops.LSTMBlockFusedCell object at 0x7f83380b0ef0>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: converting <bound method LSTMBlockWrapper.call of <tensorflow.contrib.rnn.python.ops.lstm_ops.LSTMBlockFusedCell object at 0x7f83380b0ef0>>: AttributeError: module ‘gast’ has no attribute ‘Num’

WARNING:tensorflow:From DeepSpeech.py:189: The name tf.nn.ctc_loss is deprecated. Please use tf.compat.v1.nn.ctc_loss instead.

W0925 13:53:29.349265 140202457278272 deprecation_wrapper.py:119] From DeepSpeech.py:189: The name tf.nn.ctc_loss is deprecated. Please use tf.compat.v1.nn.ctc_loss instead.

WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.

Instructions for updating:

Use tf.where in 2.0, which has the same broadcast rule as np.where

W0925 13:53:29.430233 140202457278272 deprecation.py:323] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.

Instructions for updating:

Use tf.where in 2.0, which has the same broadcast rule as np.where

WARNING:tensorflow:From DeepSpeech.py:270: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

W0925 13:53:29.646056 140202457278272 deprecation_wrapper.py:119] From DeepSpeech.py:270: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

WARNING:tensorflow:From DeepSpeech.py:329: The name tf.summary.histogram is deprecated. Please use tf.compat.v1.summary.histogram instead.

W0925 13:53:29.732506 140202457278272 deprecation_wrapper.py:119] From DeepSpeech.py:329: The name tf.summary.histogram is deprecated. Please use tf.compat.v1.summary.histogram instead.

WARNING:tensorflow:From DeepSpeech.py:407: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0925 13:53:30.054653 140202457278272 deprecation_wrapper.py:119] From DeepSpeech.py:407: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

WARNING:tensorflow:From DeepSpeech.py:411: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

W0925 13:53:30.303706 140202457278272 deprecation_wrapper.py:119] From DeepSpeech.py:411: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

WARNING:tensorflow:From DeepSpeech.py:413: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

W0925 13:53:30.306632 140202457278272 deprecation_wrapper.py:119] From DeepSpeech.py:413: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

WARNING:tensorflow:From DeepSpeech.py:418: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0925 13:53:30.309255 140202457278272 deprecation_wrapper.py:119] From DeepSpeech.py:418: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.

Instructions for updating:

Use standard file APIs to check for files with this prefix.

W0925 13:53:30.446243 140202457278272 deprecation.py:323] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.

Instructions for updating:

Use standard file APIs to check for files with this prefix.

INFO:tensorflow:Restoring parameters from /home/lee/.local/share/deepspeech/ldc93s1/train-630

I0925 13:53:30.451196 140202457278272 saver.py:1280] Restoring parameters from /home/lee/.local/share/deepspeech/ldc93s1/train-630

WARNING:tensorflow:From DeepSpeech.py:354: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0925 13:53:30.642319 140202457278272 deprecation_wrapper.py:119] From DeepSpeech.py:354: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

I Restored variables from most recent checkpoint at /home/lee/.local/share/deepspeech/ldc93s1/train-630, step 630

I STARTING Optimization

I Training epoch 0…

I Finished training epoch 0 - loss: 0.043179

I Training epoch 1…

I Finished training epoch 1 - loss: 0.044449

I FINISHED optimization in 0:00:01.506567

Error: Trie file version mismatch (3 instead of expected 4). Update your trie file.

terminate called after throwing an instance of ‘int’

Fatal Python error: Aborted

Thread 0x00007f833227c700 (most recent call first):

File “/usr/lib/python3.6/threading.py”, line 295 in wait

File “/usr/lib/python3.6/queue.py”, line 164 in get

File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/summary/writer/event_file_writer.py”, line 159 in run

File “/usr/lib/python3.6/threading.py”, line 916 in _bootstrap_inner

File “/usr/lib/python3.6/threading.py”, line 884 in _bootstrap

Thread 0x00007f8332a7d700 (most recent call first):

File “/usr/lib/python3.6/threading.py”, line 295 in wait

File “/usr/lib/python3.6/queue.py”, line 164 in get

File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/summary/writer/event_file_writer.py”, line 159 in run

File “/usr/lib/python3.6/threading.py”, line 916 in _bootstrap_inner

File “/usr/lib/python3.6/threading.py”, line 884 in _bootstrap

Current thread 0x00007f836da9a740 (most recent call first):

File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/ds_ctcdecoder/swigwrapper.py”, line 240 in init

File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/ds_ctcdecoder/init.py”, line 23 in init

File “/home/lee/tmp/DeepSpeech/evaluate.py”, line 45 in evaluate

File “DeepSpeech.py”, line 554 in test

File “DeepSpeech.py”, line 824 in main

File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/absl/app.py”, line 250 in _run_main

File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/absl/app.py”, line 299 in run

File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/platform/app.py”, line 40 in run

File “DeepSpeech.py”, line 836 in

Aborted (core dumped)
#############################

I saw this being discussed on https://github.com/mozilla/DeepSpeech/issues/2206 so I am attempted to fix with suggested solution:

#############################
(deepspeech-venv) lee@lee-VirtualBox:~/tmp/DeepSpeech$ pip install --upgrade https:://index.taskcluster.net/v1/task/project.deepspeech.deepspeech.native_client.master.cpu-ctc/artifacts/public/ds_ctcdecoder-0.6.0a0-cp36-cp36m-manylinux1_x86_64.whl
Collecting ds-ctcdecoder==0.6.0a0 from https:://index.taskcluster.net/v1/task/project.deepspeech.deepspeech.native_client.master.cpu-ctc/artifacts/public/ds_ctcdecoder-0.6.0a0-cp36-cp36m-manylinux1_x86_64.whl
ERROR: HTTP error 404 while getting https:://index.taskcluster.net/v1/task/project.deepspeech.deepspeech.native_client.master.cpu-ctc/artifacts/public/ds_ctcdecoder-0.6.0a0-cp36-cp36m-manylinux1_x86_64.whl
ERROR: Could not install requirement ds-ctcdecoder==0.6.0a0 from https:://index.taskcluster.net/v1/task/project.deepspeech.deepspeech.native_client.master.cpu-ctc/artifacts/public/ds_ctcdecoder-0.6.0a0-cp36-cp36m-manylinux1_x86_64.whl because of error 404 Client Error: Not Found for url: https:://queue.taskcluster.net/v1/task/YnbB4UWaRfq7sdPhot3N0g/artifacts/public%2Fds_ctcdecoder-0.6.0a0-cp36-cp36m-manylinux1_x86_64.whl
ERROR: Could not install requirement ds-ctcdecoder==0.6.0a0 from https:://index.taskcluster.net/v1/task/project.deepspeech.deepspeech.native_client.master.cpu-ctc/artifacts/public/ds_ctcdecoder-0.6.0a0-cp36-cp36m-manylinux1_x86_64.whl because of HTTP error 404 Client Error: Not Found for url: https:://queue.taskcluster.net/v1/task/YnbB4UWaRfq7sdPhot3N0g/artifacts/public%2Fds_ctcdecoder-0.6.0a0-cp36-cp36m-manylinux1_x86_64.whl for URL
https:://index.taskcluster.net/v1/task/project.deepspeech.deepspeech.native_client.master.cpu-ctc/artifacts/public/ds_ctcdecoder-0.6.0a0-cp36-cp36m-manylinux1_x86_64.whl
#############################

(I put :: for every : in https:// above because I can’t post anything witn more than 5 links)

personal.dummy · September 26, 2019, 6:31pm

@dabinat Thank you too for your time, I see your name come up a lot too. Thank you for the suggestion, I’ll start looking into how to exactly do that. I’ve been installing tensorflow by pip3 install ‘tensorflow==1.14.0’ to install tensorflow.

lissyx · September 26, 2019, 7:04pm

Looks like you still have some stale file somewhere

personal.dummy · September 30, 2019, 3:55pm

@lissyx I was unsure what you meant by stale file so I decided to reinstall/setup the whole thing again. I think its working now because I’m no longer getting any error when attempting to run ./bin/run-ldc93s1.sh.

I’m also testing transfer learning (no GPU) from the checkpoint v0.5.1 and I think its working. My screen shows:

I STARTING Optimization
Epoch 0 | Training | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000

I’ve let it run for about 15 minutes now and haven’t run into an error yet so I assume its working fine. I’m geussing that because I dont have GPU, this will take a really long time. Does this sound right to you?

lissyx · September 30, 2019, 3:57pm

Well no error with LDC test, and this output, it seems okay. You’re right, without GPU it will be slow.

personal.dummy · September 30, 2019, 3:58pm

Sorry, spoke too early, ran into this error seconds after that last comment:

(deepspeech-venv) lee@lee-VirtualBox:~/tmp/DeepSpeech$ ./bin/run-ASR.sh

[! -f DeepSpeech.py ]
./bin/run-ASR.sh: 3: ./bin/run-ASR.sh: [!: not found
python -u DeepSpeech.py --train_files /home/lee/tmp/DS_TRAINING/TRAIN/train.csv --dev_files /home/lee/tmp/DS_TRAINING/DEV/dev.csv --test_files /home/lee/tmp/DS_TRAINING/TEST/test.csv --train_batch_size 7 --dev_batch_size 2 --test_batch_size 1 --n_hidden 2048 --epochs 3
WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
options available in V2.
- tf.py_function takes a python function which manipulates tf eager
  tensors instead of numpy arrays. It’s easy to convert a tf eager tensor to
  an ndarray (just call tensor.numpy()) but having access to eager tensors
  means tf.py_functions can use accelerators such as GPUs as well as
  being differentiable using a gradient tape.
- tf.numpy_function maintains the semantics of the deprecated tf.py_func
  (it is not differentiable, and manipulates numpy arrays). It drops the
  stateful argument making all functions stateful.

W0930 11:44:04.435562 140372350224192 deprecation.py:323] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
options available in V2.
- tf.py_function takes a python function which manipulates tf eager
tensors instead of numpy arrays. It’s easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means tf.py_functions can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
- tf.numpy_function maintains the semantics of the deprecated tf.py_func
(it is not differentiable, and manipulates numpy arrays). It drops the
stateful argument making all functions stateful.

WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.data.get_output_types(iterator).
W0930 11:44:04.580041 140372350224192 deprecation.py:323] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.data.get_output_types(iterator).
WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.data.get_output_shapes(iterator).
W0930 11:44:04.583203 140372350224192 deprecation.py:323] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.data.get_output_shapes(iterator).
WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.data.get_output_classes(iterator).
W0930 11:44:04.585494 140372350224192 deprecation.py:323] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.data.get_output_classes(iterator).
WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:

https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
https://github.com/tensorflow/addons
https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

W0930 11:44:06.635333 140372350224192 lazy_loader.py:50]
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:

https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
https://github.com/tensorflow/addons
https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0930 11:44:06.646873 140372350224192 deprecation.py:506] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:Entity <bound method LSTMBlockWrapper.call of <tensorflow.contrib.rnn.python.ops.lstm_ops.LSTMBlockFusedCell object at 0x7faad049dac8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: converting <bound method LSTMBlockWrapper.call of <tensorflow.contrib.rnn.python.ops.lstm_ops.LSTMBlockFusedCell object at 0x7faad049dac8>>: AttributeError: module ‘gast’ has no attribute ‘Num’
W0930 11:44:06.743133 140372350224192 ag_logging.py:145] Entity <bound method LSTMBlockWrapper.call of <tensorflow.contrib.rnn.python.ops.lstm_ops.LSTMBlockFusedCell object at 0x7faad049dac8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: converting <bound method LSTMBlockWrapper.call of <tensorflow.contrib.rnn.python.ops.lstm_ops.LSTMBlockFusedCell object at 0x7faad049dac8>>: AttributeError: module ‘gast’ has no attribute ‘Num’
WARNING:tensorflow:From DeepSpeech.py:232: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0930 11:44:06.881601 140372350224192 deprecation.py:323] From DeepSpeech.py:232: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
I Initializing variables…
I STARTING Optimization
Epoch 0 | Training | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000 Traceback (most recent call last):
File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 1356, in _do_call
return fn(*args)
File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[4098,7,2048] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
[[{{node tower_0/gradients/zeros_like_2}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “DeepSpeech.py”, line 906, in
absl.app.run(main)
File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/absl/app.py”, line 299, in run
_run_main(main, args)
File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/absl/app.py”, line 250, in _run_main
sys.exit(main(argv))
File “DeepSpeech.py”, line 890, in main
train()
File “DeepSpeech.py”, line 608, in train
train_loss, _ = run_set(‘train’, epoch, train_init_op)
File “DeepSpeech.py”, line 576, in run_set
feed_dict=feed_dict)
File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 950, in run
run_metadata_ptr)
File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 1173, in _run
feed_dict_tensor, options, run_metadata)
File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 1350, in _do_run
run_metadata)
File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[4098,7,2048] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
[[node tower_0/gradients/zeros_like_2 (defined at DeepSpeech.py:308) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Errors may have originated from an input operation.
Input Source operations connected to node tower_0/gradients/zeros_like_2:
tower_0/cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/BlockLSTM (defined at DeepSpeech.py:91)

Original stack trace for ‘tower_0/gradients/zeros_like_2’:
File “DeepSpeech.py”, line 906, in
absl.app.run(main)
File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/absl/app.py”, line 299, in run
_run_main(main, args)
File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/absl/app.py”, line 250, in _run_main
sys.exit(main(argv))
File “DeepSpeech.py”, line 890, in main
train()
File “DeepSpeech.py”, line 449, in train
gradients, loss, non_finite_files = get_tower_results(iterator, optimizer, dropout_rates)
File “DeepSpeech.py”, line 308, in get_tower_results
gradients = optimizer.compute_gradients(avg_loss)
File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/training/optimizer.py”, line 512, in compute_gradients
colocate_gradients_with_ops=colocate_gradients_with_ops)
File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py”, line 158, in gradients
unconnected_gradients)
File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/ops/gradients_util.py”, line 722, in _GradientsHelper
out_grads[i] = control_flow_ops.ZerosLikeOutsideLoop(op, i)
File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py”, line 1338, in ZerosLikeOutsideLoop
return array_ops.zeros_like(val, optimize=False)
File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py”, line 180, in wrapper
return target(*args, **kwargs)
File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py”, line 1916, in zeros_like
return zeros_like_impl(tensor, dtype, name, optimize)
File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py”, line 1976, in zeros_like_impl
return gen_array_ops.zeros_like(tensor, name=name)
File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py”, line 11961, in zeros_like
“ZerosLike”, x=x, name=name)
File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py”, line 788, in _apply_op_helper
op_def=op_def)
File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py”, line 507, in new_func
return func(*args, **kwargs)
File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py”, line 3616, in create_op
op_def=op_def)
File “/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py”, line 2005, in init
self._traceback = tf_stack.extract_stack()

Never ran into this before, I’m looking into solutions.

lissyx · September 30, 2019, 4:05pm

@personal.dummy First, please use code formatting for pasting console output. Second, I saw some OOM, which is unusual on CPU. What are your specs ?

personal.dummy · September 30, 2019, 4:24pm

Apologies for the output, I’ve been looking for how to format it, it hurts my eyes too.

My specs on machine:
Intel® Core™ i7-4790 CPU @ 3.6GHz
Windows 10 Pro
16GB RAM
No GPU

personal.dummy · September 30, 2019, 6:12pm

Okay so good news, I was able to get through transfer learning by changing the batches to 1s. I saw in your readme that if I run into any OOM errors, try reducing it (I chose 1s because I wanted to see if I can get through it without running into errors). By the way, you’re right, I don’t have GPU.

The bad news is, the results are poor. My training setup is listed below:

--checkpoint_dir /home/lee/tmp/CHECKPOINT/deepspeech-0.5.1-checkpoint \
--train_files /home/lee/tmp/DS_TRAINING/TRAIN/train.csv \
--dev_files /home/lee/tmp/DS_TRAINING/DEV/dev.csv \
--test_files /home/lee/tmp/DS_TRAINING/TEST/test.csv \
--train_batch_size 1 \
--dev_batch_size 1 \
--test_batch_size 1 \
--n_hidden 2048 \
--epochs 1 \
#--validation_step 1 \
--early_stop True \
--earlystop_nsteps 6 \
--estop_mean_thresh 0.1 \
--estop_std_thresh 0.1 \
--dropout_rate 0.22 \
--learing_rate 0.000001 \
--report_count 100 \
--use_seq_length False \
--export_dir /home/lee/tmp/EXPORT \
--alphabet_config_path /home/lee/tmp/MODELS/models_TL/alphabet.txt \
--lm_binary_path /home/lee/tmp/MODELS/models_TL/lm.binary \
--lm_trie_path /home/lee/tmp/MODELS/models_TL/trie \

The test clip is about 17 seconds taken from the news. When tested during training, the result came back as:

res: "o h h h o o o o o o h h h h h h h h h h o o o o o o o o o o o o o o o o o o o o o o o h h h o o o o o o h h h h h h h h o o o o o o o o o o o o o o h h h h h h h o o o o o o o o o o o h o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o h h h h h h h h h h h h h h h h h h h h h h h h h h h h h o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o h h h o o o o o o o o o o o o o"

Do you have any suggestions on how to improve these results or where I can learn about making the results better?

personal.dummy · September 30, 2019, 8:07pm

Please ignore last comment, I discovered that I did not run it from the v0.5.1 checkpoint. I have adjusted my training script so that it will run from the appropriate checkpoint.

personal.dummy · October 1, 2019, 1:33pm

After letting it run overnight, finally got the output:

(deepspeech-venv) lee@lee-VirtualBox:~/tmp/DeepSpeech$ ./bin/run-ASR.sh
+ [! -f DeepSpeech.py ]
./bin/run-ASR.sh: 3: ./bin/run-ASR.sh: [!: not found
+ python -u DeepSpeech.py --checkpoint_dir /home/lee/tmp/CHECKPOINT/deepspeech-0.5.1-checkpoint --train_files /home/l                                                                                                                        ee/tmp/DS_TRAINING/TRAIN/train.csv --dev_files /home/lee/tmp/DS_TRAINING/DEV/dev.csv --test_files /home/lee/tmp/DS_TR                                                                                                                        AINING/TEST/test.csv --train_batch_size 1 --dev_batch_size 1 --test_batch_size 1
WARNING:tensorflow:From DeepSpeech.py:836: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead                                                                                                                        .

WARNING:tensorflow:From /home/lee/tmp/DeepSpeech/util/config.py:60: The name tf.ConfigProto is deprecated. Please use                                                                                                                         tf.compat.v1.ConfigProto instead.

W0930 15:42:20.451467 139910847608640 deprecation_wrapper.py:119] From /home/lee/tmp/DeepSpeech/util/config.py:60: Th                                                                                                                        e name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From DeepSpeech.py:818: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.res                                                                                                                        et_default_graph instead.

W0930 15:42:20.456071 139910847608640 deprecation_wrapper.py:119] From DeepSpeech.py:818: The name tf.reset_default_g                                                                                                                        raph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

WARNING:tensorflow:From DeepSpeech.py:819: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_ran                                                                                                                        dom_seed instead.

W0930 15:42:20.457949 139910847608640 deprecation_wrapper.py:119] From DeepSpeech.py:819: The name tf.set_random_seed                                                                                                                         is deprecated. Please use tf.compat.v1.set_random_seed instead.

WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_                                                                                                                        ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
    options available in V2.
    - tf.py_function takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
    - tf.numpy_function maintains the semantics of the deprecated tf.py_func
    (it is not differentiable, and manipulates numpy arrays). It drops the
    stateful argument making all functions stateful.

W0930 15:42:20.517106 139910847608640 deprecation.py:323] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packa                                                                                                                        ges/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and                                                                                                                         will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
    options available in V2.
    - tf.py_function takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
    - tf.numpy_function maintains the semantics of the deprecated tf.py_func
    (it is not differentiable, and manipulates numpy arrays). It drops the
    stateful argument making all functions stateful.

WARNING:tensorflow:From /home/lee/tmp/DeepSpeech/util/feeding.py:45: The name tf.read_file is deprecated. Please use                                                                                                                         tf.io.read_file instead.

W0930 15:42:20.568743 139910847608640 deprecation_wrapper.py:119] From /home/lee/tmp/DeepSpeech/util/feeding.py:45: T                                                                                                                        he name tf.read_file is deprecated. Please use tf.io.read_file instead.

WARNING:tensorflow:From DeepSpeech.py:372: The name tf.data.Iterator is deprecated. Please use tf.compat.v1.data.Iter                                                                                                                        ator instead.

W0930 15:42:20.659285 139910847608640 deprecation_wrapper.py:119] From DeepSpeech.py:372: The name tf.data.Iterator i                                                                                                                        s deprecated. Please use tf.compat.v1.data.Iterator instead.

WARNING:tensorflow:From DeepSpeech.py:372: DatasetV1.output_types (from tensorflow.python.data.ops.dataset_ops) is de                                                                                                                        precated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(dataset)`.
W0930 15:42:20.660744 139910847608640 deprecation.py:323] From DeepSpeech.py:372: DatasetV1.output_types (from tensor                                                                                                                        flow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(dataset)`.
WARNING:tensorflow:From DeepSpeech.py:373: DatasetV1.output_shapes (from tensorflow.python.data.ops.dataset_ops) is d                                                                                                                        eprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(dataset)`.
W0930 15:42:20.663090 139910847608640 deprecation.py:323] From DeepSpeech.py:373: DatasetV1.output_shapes (from tenso                                                                                                                        rflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(dataset)`.
WARNING:tensorflow:From DeepSpeech.py:374: DatasetV1.output_classes (from tensorflow.python.data.ops.dataset_ops) is                                                                                                                         deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(dataset)`.
W0930 15:42:20.665561 139910847608640 deprecation.py:323] From DeepSpeech.py:374: DatasetV1.output_classes (from tens                                                                                                                        orflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(dataset)`.
WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator                                                                                                                        _ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed i                                                                                                                        n a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(iterator)`.
W0930 15:42:20.669818 139910847608640 deprecation.py:323] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packa                                                                                                                        ges/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_o                                                                                                                        ps) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(iterator)`.
WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator                                                                                                                        _ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed                                                                                                                         in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(iterator)`.
W0930 15:42:20.672511 139910847608640 deprecation.py:323] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packa                                                                                                                        ges/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_                                                                                                                        ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(iterator)`.
WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator                                                                                                                        _ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed                                                                                                                         in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(iterator)`.
W0930 15:42:20.674112 139910847608640 deprecation.py:323] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packa                                                                                                                        ges/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator                                                                                                                        _ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(iterator)`.
WARNING:tensorflow:From DeepSpeech.py:385: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder                                                                                                                         instead.

W0930 15:42:20.807493 139910847608640 deprecation_wrapper.py:119] From DeepSpeech.py:385: The name tf.placeholder is                                                                                                                         deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From DeepSpeech.py:208: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.tra                                                                                                                        in.AdamOptimizer instead.

W0930 15:42:20.813153 139910847608640 deprecation_wrapper.py:119] From DeepSpeech.py:208: The name tf.train.AdamOptim                                                                                                                        izer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1                                                                                                                        251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be remo                                                                                                                        ved in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0930 15:42:20.970581 139910847608640 deprecation.py:506] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packa                                                                                                                        ges/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) wi                                                                                                                        th dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:Entity <bound method LSTMBlockWrapper.call of <tensorflow.contrib.rnn.python.ops.lstm_ops.LSTMBloc                                                                                                                        kFusedCell object at 0x7f3f51b90d68>> could not be transformed and will be executed as-is. Please report this to the                                                                                                                         AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach                                                                                                                         the full output. Cause: converting <bound method LSTMBlockWrapper.call of <tensorflow.contrib.rnn.python.ops.lstm_ops                                                                                                                        .LSTMBlockFusedCell object at 0x7f3f51b90d68>>: AttributeError: module 'gast' has no attribute 'Num'
W0930 15:42:21.056357 139910847608640 ag_logging.py:145] Entity <bound method LSTMBlockWrapper.call of <tensorflow.co                                                                                                                        ntrib.rnn.python.ops.lstm_ops.LSTMBlockFusedCell object at 0x7f3f51b90d68>> could not be transformed and will be exec                                                                                                                        uted as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `expor                                                                                                                        t AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method LSTMBlockWrapper.call of <tens                                                                                                                        orflow.contrib.rnn.python.ops.lstm_ops.LSTMBlockFusedCell object at 0x7f3f51b90d68>>: AttributeError: module 'gast' h                                                                                                                        as no attribute 'Num'
WARNING:tensorflow:From DeepSpeech.py:189: The name tf.nn.ctc_loss is deprecated. Please use tf.compat.v1.nn.ctc_loss                                                                                                                         instead.

W0930 15:42:21.177125 139910847608640 deprecation_wrapper.py:119] From DeepSpeech.py:189: The name tf.nn.ctc_loss is                                                                                                                         deprecated. Please use tf.compat.v1.nn.ctc_loss instead.

WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:                                                                                                                        1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed                                                                                                                         in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0930 15:42:21.248890 139910847608640 deprecation.py:323] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packa                                                                                                                        ges/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_                                                                                                                        ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From DeepSpeech.py:270: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.                                                                                                                        scalar instead.

W0930 15:42:21.450597 139910847608640 deprecation_wrapper.py:119] From DeepSpeech.py:270: The name tf.summary.scalar                                                                                                                         is deprecated. Please use tf.compat.v1.summary.scalar instead.

WARNING:tensorflow:From DeepSpeech.py:329: The name tf.summary.histogram is deprecated. Please use tf.compat.v1.summa                                                                                                                        ry.histogram instead.

W0930 15:42:21.541460 139910847608640 deprecation_wrapper.py:119] From DeepSpeech.py:329: The name tf.summary.histogr                                                                                                                        am is deprecated. Please use tf.compat.v1.summary.histogram instead.

WARNING:tensorflow:From DeepSpeech.py:407: The name tf.train.get_or_create_global_step is deprecated. Please use tf.c                                                                                                                        ompat.v1.train.get_or_create_global_step instead.

W0930 15:42:21.888840 139910847608640 deprecation_wrapper.py:119] From DeepSpeech.py:407: The name tf.train.get_or_cr                                                                                                                        eate_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

WARNING:tensorflow:From DeepSpeech.py:411: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summa                                                                                                                        ry.merge_all instead.

W0930 15:42:22.160207 139910847608640 deprecation_wrapper.py:119] From DeepSpeech.py:411: The name tf.summary.merge_a                                                                                                                        ll is deprecated. Please use tf.compat.v1.summary.merge_all instead.

WARNING:tensorflow:From DeepSpeech.py:413: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summ                                                                                                                        ary.FileWriter instead.

W0930 15:42:22.162229 139910847608640 deprecation_wrapper.py:119] From DeepSpeech.py:413: The name tf.summary.FileWri                                                                                                                        ter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

WARNING:tensorflow:From DeepSpeech.py:418: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver                                                                                                                         instead.

W0930 15:42:22.163563 139910847608640 deprecation_wrapper.py:119] From DeepSpeech.py:418: The name tf.train.Saver is                                                                                                                         deprecated. Please use tf.compat.v1.train.Saver instead.

WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py                                                                                                                        :1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in                                                                                                                         a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
W0930 15:42:22.306703 139910847608640 deprecation.py:323] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packa                                                                                                                        ges/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_managemen                                                                                                                        t) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from /home/lee/tmp/CHECKPOINT/deepspeech-0.5.1-checkpoint/model.v0.5.1
I0930 15:42:22.311545 139910847608640 saver.py:1280] Restoring parameters from /home/lee/tmp/CHECKPOINT/deepspeech-0.                                                                                                                        5.1-checkpoint/model.v0.5.1
WARNING:tensorflow:From DeepSpeech.py:354: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.t                                                                                                                        rain.get_global_step instead.

W0930 15:42:26.877884 139910847608640 deprecation_wrapper.py:119] From DeepSpeech.py:354: The name tf.train.get_globa                                                                                                                        l_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

(I had to split this post because I have a 320000 character limit)

personal.dummy · October 1, 2019, 1:34pm

(Post was split due to 320000 character limit)

I Restored variables from most recent checkpoint at /home/lee/tmp/CHECKPOINT/deepspeech-0.5.1-checkpoint/model.v0.5.1                                                                                                                        , step 467356
I STARTING Optimization
Epoch 0 |   Training | Elapsed Time: 1:08:42 | Steps: 7 | Loss: 1569.078017
Epoch 0 | Validation | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dev                                                                                                                        Epoch 0 | Validation | Elapsed Time: 0:00:07 | Steps: 1 | Loss: 959.722717 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/d                                                                                                                        Epoch 0 | Validation | Elapsed Time: 0:00:53 | Steps: 2 | Loss: 5151.055695 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/                                                                                                                        Epoch 0 | Validation | Elapsed Time: 0:00:53 | Steps: 2 | Loss: 5151.055695 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/                                                                                                                        dev.csv
I Saved new best validating model with loss 5151.055695 to: /home/lee/tmp/CHECKPOINT/deepspeech-0.5.1-checkpoint/best                                                                                                                        _dev-467363
Epoch 1 |   Training | Elapsed Time: 0:29:34 | Steps: 5 | Loss: 2428.944629                                                                                                                                                                 WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py:960: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to delete files with this prefix.
W0930 17:21:44.111388 139910847608640 deprecation.py:323] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py:960: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to delete files with this prefix.
Epoch 1 |   Training | Elapsed Time: 1:03:23 | Steps: 7 | Loss: 2752.175293
Epoch 1 | Validation | Elapsed Time: 0:00:50 | Steps: 2 | Loss: 10917.121460 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dev.csv
Epoch 2 |   Training | Elapsed Time: 1:03:20 | Steps: 7 | Loss: 2283.280631
Epoch 2 | Validation | Elapsed Time: 0:00:49 | Steps: 2 | Loss: 2775.877319 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dev.csv
I Saved new best validating model with loss 2775.877319 to: /home/lee/tmp/CHECKPOINT/deepspeech-0.5.1-checkpoint/best_dev-467377
Epoch 3 |   Training | Elapsed Time: 1:02:12 | Steps: 7 | Loss: 1758.814462
Epoch 3 | Validation | Elapsed Time: 0:00:49 | Steps: 2 | Loss: 3010.117950 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dev.csv
Epoch 4 |   Training | Elapsed Time: 1:02:05 | Steps: 7 | Loss: 1960.070443
Epoch 4 | Validation | Elapsed Time: 0:00:49 | Steps: 2 | Loss: 2644.494934 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dev.csv
I Saved new best validating model with loss 2644.494934 to: /home/lee/tmp/CHECKPOINT/deepspeech-0.5.1-checkpoint/best_dev-467391
Epoch 5 |   Training | Elapsed Time: 1:02:04 | Steps: 7 | Loss: 1940.238735
Epoch 5 | Validation | Elapsed Time: 0:00:49 | Steps: 2 | Loss: 2637.237946 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dev.csv
I Saved new best validating model with loss 2637.237946 to: /home/lee/tmp/CHECKPOINT/deepspeech-0.5.1-checkpoint/best_dev-467398
Epoch 6 |   Training | Elapsed Time: 1:02:01 | Steps: 7 | Loss: 1738.707336
Epoch 6 | Validation | Elapsed Time: 0:00:49 | Steps: 2 | Loss: 2566.750397 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dev.csv
I Saved new best validating model with loss 2566.750397 to: /home/lee/tmp/CHECKPOINT/deepspeech-0.5.1-checkpoint/best_dev-467405
Epoch 7 |   Training | Elapsed Time: 1:02:16 | Steps: 7 | Loss: 1724.119023
Epoch 7 | Validation | Elapsed Time: 0:00:49 | Steps: 2 | Loss: 2587.557495 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dev.csv
Epoch 8 |   Training | Elapsed Time: 1:02:11 | Steps: 7 | Loss: 1722.151180
Epoch 8 | Validation | Elapsed Time: 0:00:49 | Steps: 2 | Loss: 2546.124329 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dev.csv
I Saved new best validating model with loss 2546.124329 to: /home/lee/tmp/CHECKPOINT/deepspeech-0.5.1-checkpoint/best_dev-467419
Epoch 9 |   Training | Elapsed Time: 1:02:05 | Steps: 7 | Loss: 1693.717368
Epoch 9 | Validation | Elapsed Time: 0:00:49 | Steps: 2 | Loss: 2553.632019 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dev.csv
Epoch 10 |   Training | Elapsed Time: 1:01:53 | Steps: 7 | Loss: 1692.560926
Epoch 10 | Validation | Elapsed Time: 0:00:49 | Steps: 2 | Loss: 2553.722595 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dev.csv
Epoch 11 |   Training | Elapsed Time: 1:02:04 | Steps: 7 | Loss: 1682.576534
Epoch 11 | Validation | Elapsed Time: 0:00:49 | Steps: 2 | Loss: 2546.759094 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dev.csv
Epoch 12 |   Training | Elapsed Time: 1:02:13 | Steps: 7 | Loss: 1686.977308
Epoch 12 | Validation | Elapsed Time: 0:00:48 | Steps: 2 | Loss: 2551.509094 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dev.csv
Epoch 13 |   Training | Elapsed Time: 1:11:24 | Steps: 7 | Loss: 1680.656084
Epoch 13 | Validation | Elapsed Time: 0:00:49 | Steps: 2 | Loss: 2542.807434 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dev.csv
I Saved new best validating model with loss 2542.807434 to: /home/lee/tmp/CHECKPOINT/deepspeech-0.5.1-checkpoint/best_dev-467454
Epoch 14 |   Training | Elapsed Time: 1:02:23 | Steps: 7 | Loss: 1683.956805
Epoch 14 | Validation | Elapsed Time: 0:00:49 | Steps: 2 | Loss: 2552.203186 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dev.csv
I Early stop triggered as (for last 4 steps) validation loss: 2552.203186 with standard deviation: 3.557418 and mean: 2547.025208
I FINISHED optimization in 16:04:10.745781
Error: Trie file version mismatch (3 instead of expected 4). Update your trie file.
terminate called after throwing an instance of 'int'
Fatal Python error: Aborted

Thread 0x00007f3f50d36700 (most recent call first):
    File "/usr/lib/python3.6/threading.py", line 295 in wait
    File "/usr/lib/python3.6/queue.py", line 164 in get
    File "/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 159 in run
    File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
    File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007f3f51537700 (most recent call first):
    File "/usr/lib/python3.6/threading.py", line 295 in wait
    File "/usr/lib/python3.6/queue.py", line 164 in get
    File "/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 159 in run
    File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
    File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Current thread 0x00007f3f885f3740 (most recent call first):
    File "/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/ds_ctcdecoder/swigwrapper.py", line 240 in init
    File "/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/ds_ctcdecoder/__init__.py", line 23 in __init__
    File "/home/lee/tmp/DeepSpeech/evaluate.py", line 45 in evaluate
    File "DeepSpeech.py", line 554 in test
    File "DeepSpeech.py", line 824 in main
    File "/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/absl/app.py", line 250 in _run_main
    File "/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/absl/app.py", line 299 in run
    File "/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 40 in run
    File "DeepSpeech.py", line 836 in <module>
Aborted (core dumped)

I think the error I’m running into is:

Error: Trie file version mismatch (3 instead of expected 4). Update your trie file.
terminate called after throwing an instance of 'int'
Fatal Python error: Aborted

I remember you mentioned that this means I have a stale file somewhere. Does this mean I have a trie file that is mismatching since the error says “Update your trie file.”?

I’m currently looking here for solutions: https://github.com/mozilla/DeepSpeech/issues/2206.

Do you have any suggestions?

lissyx · October 1, 2019, 1:53pm

Yes, that’s right. Make sure you are training from v0.5.1 git checkout. If you re-generate LM and trie, make sure you do so with v0.5.1 binaries.

personal.dummy · October 2, 2019, 1:08pm

@lissyx I ran the training script overnight again and ran into the same error, please see my output:

(deepspeech-venv) lee@lee-VirtualBox:~/tmp/DeepSpeech$ ./bin/run-ASR.sh
+ [! -f DeepSpeech.py ]
./bin/run-ASR.sh: 3: ./bin/run-ASR.sh: [!: not found
+ python -u DeepSpeech.py --checkpoint_dir /home/lee/tmp/CHECKPOINT/deepspeech-0.5.1-checkpoint --train_files /home/lee/tmp/DS_TRAINING/TRAIN/train.csv --dev_files /home/lee/tmp/DS_TRAINING/DEV/dev.csv --test_files /home/lee/tmp/DS_TRAINING/TEST/test.csv --train_batch_size 1 --dev_batch_size 1 --test_batch_size 1
WARNING:tensorflow:From DeepSpeech.py:836: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

WARNING:tensorflow:From /home/lee/tmp/DeepSpeech/util/config.py:60: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W1001 10:23:47.721463 139667258488640 deprecation_wrapper.py:119] From /home/lee/tmp/DeepSpeech/util/config.py:60: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From DeepSpeech.py:818: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

W1001 10:23:47.726446 139667258488640 deprecation_wrapper.py:119] From DeepSpeech.py:818: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

WARNING:tensorflow:From DeepSpeech.py:819: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

W1001 10:23:47.728003 139667258488640 deprecation_wrapper.py:119] From DeepSpeech.py:819: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
    options available in V2.
    - tf.py_function takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
    - tf.numpy_function maintains the semantics of the deprecated tf.py_func
    (it is not differentiable, and manipulates numpy arrays). It drops the
    stateful argument making all functions stateful.

W1001 10:23:47.813281 139667258488640 deprecation.py:323] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
    options available in V2.
    - tf.py_function takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
    - tf.numpy_function maintains the semantics of the deprecated tf.py_func
    (it is not differentiable, and manipulates numpy arrays). It drops the
    stateful argument making all functions stateful.

WARNING:tensorflow:From /home/lee/tmp/DeepSpeech/util/feeding.py:45: The name tf.read_file is deprecated. Please use tf.io.read_file instead.

W1001 10:23:47.867862 139667258488640 deprecation_wrapper.py:119] From /home/lee/tmp/DeepSpeech/util/feeding.py:45: The name tf.read_file is deprecated. Please use tf.io.read_file instead.

WARNING:tensorflow:From DeepSpeech.py:372: The name tf.data.Iterator is deprecated. Please use tf.compat.v1.data.Iterator instead.

W1001 10:23:47.956069 139667258488640 deprecation_wrapper.py:119] From DeepSpeech.py:372: The name tf.data.Iterator is deprecated. Please use tf.compat.v1.data.Iterator instead.

WARNING:tensorflow:From DeepSpeech.py:372: DatasetV1.output_types (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(dataset)`.
W1001 10:23:47.957960 139667258488640 deprecation.py:323] From DeepSpeech.py:372: DatasetV1.output_types (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(dataset)`.
WARNING:tensorflow:From DeepSpeech.py:373: DatasetV1.output_shapes (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(dataset)`.
W1001 10:23:47.960487 139667258488640 deprecation.py:323] From DeepSpeech.py:373: DatasetV1.output_shapes (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(dataset)`.
WARNING:tensorflow:From DeepSpeech.py:374: DatasetV1.output_classes (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(dataset)`.
W1001 10:23:47.962918 139667258488640 deprecation.py:323] From DeepSpeech.py:374: DatasetV1.output_classes (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(dataset)`.
WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(iterator)`.
W1001 10:23:47.967753 139667258488640 deprecation.py:323] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(iterator)`.
WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(iterator)`.
W1001 10:23:47.970656 139667258488640 deprecation.py:323] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(iterator)`.
WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(iterator)`.
W1001 10:23:47.972325 139667258488640 deprecation.py:323] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(iterator)`.
WARNING:tensorflow:From DeepSpeech.py:385: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

W1001 10:23:48.148529 139667258488640 deprecation_wrapper.py:119] From DeepSpeech.py:385: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From DeepSpeech.py:208: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

W1001 10:23:48.155015 139667258488640 deprecation_wrapper.py:119] From DeepSpeech.py:208: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W1001 10:23:48.302421 139667258488640 deprecation.py:506] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:Entity <bound method LSTMBlockWrapper.call of <tensorflow.contrib.rnn.python.ops.lstm_ops.LSTMBlockFusedCell object at 0x7f069c2ebe48>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method LSTMBlockWrapper.call of <tensorflow.contrib.rnn.python.ops.lstm_ops.LSTMBlockFusedCell object at 0x7f069c2ebe48>>: AttributeError: module 'gast' has no attribute 'Num'
W1001 10:23:48.388300 139667258488640 ag_logging.py:145] Entity <bound method LSTMBlockWrapper.call of <tensorflow.contrib.rnn.python.ops.lstm_ops.LSTMBlockFusedCell object at 0x7f069c2ebe48>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method LSTMBlockWrapper.call of <tensorflow.contrib.rnn.python.ops.lstm_ops.LSTMBlockFusedCell object at 0x7f069c2ebe48>>: AttributeError: module 'gast' has no attribute 'Num'
WARNING:tensorflow:From DeepSpeech.py:189: The name tf.nn.ctc_loss is deprecated. Please use tf.compat.v1.nn.ctc_loss instead.

W1001 10:23:48.522485 139667258488640 deprecation_wrapper.py:119] From DeepSpeech.py:189: The name tf.nn.ctc_loss is deprecated. Please use tf.compat.v1.nn.ctc_loss instead.

WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W1001 10:23:48.618887 139667258488640 deprecation.py:323] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From DeepSpeech.py:270: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

W1001 10:23:48.833139 139667258488640 deprecation_wrapper.py:119] From DeepSpeech.py:270: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

WARNING:tensorflow:From DeepSpeech.py:329: The name tf.summary.histogram is deprecated. Please use tf.compat.v1.summary.histogram instead.

W1001 10:23:48.927037 139667258488640 deprecation_wrapper.py:119] From DeepSpeech.py:329: The name tf.summary.histogram is deprecated. Please use tf.compat.v1.summary.histogram instead.

WARNING:tensorflow:From DeepSpeech.py:407: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W1001 10:23:49.242793 139667258488640 deprecation_wrapper.py:119] From DeepSpeech.py:407: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

WARNING:tensorflow:From DeepSpeech.py:411: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

W1001 10:23:49.475303 139667258488640 deprecation_wrapper.py:119] From DeepSpeech.py:411: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

WARNING:tensorflow:From DeepSpeech.py:413: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

W1001 10:23:49.477800 139667258488640 deprecation_wrapper.py:119] From DeepSpeech.py:413: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

WARNING:tensorflow:From DeepSpeech.py:418: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W1001 10:23:49.523673 139667258488640 deprecation_wrapper.py:119] From DeepSpeech.py:418: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
W1001 10:23:49.693629 139667258488640 deprecation.py:323] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from /home/lee/tmp/CHECKPOINT/deepspeech-0.5.1-checkpoint/train-467461
I1001 10:23:49.699033 139667258488640 saver.py:1280] Restoring parameters from /home/lee/tmp/CHECKPOINT/deepspeech-0.5.1-checkpoint/train-467461
WARNING:tensorflow:From DeepSpeech.py:354: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W1001 10:23:56.195741 139667258488640 deprecation_wrapper.py:119] From DeepSpeech.py:354: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

I Restored variables from most recent checkpoint at /home/lee/tmp/CHECKPOINT/deepspeech-0.5.1-checkpoint/train-467461, step 467461
I STARTING Optimization
Epoch 0 |   Training | Elapsed Time: 1:21:20 | Steps: 7 | Loss: 1682.555433
Epoch 0 | Validation | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/devEpoch 0 | Validation | Elapsed Time: 0:00:13 | Steps: 1 | Loss: 578.138733 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dEpoch 0 | Validation | Elapsed Time: 0:01:30 | Steps: 2 | Loss: 2535.590118 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/Epoch 0 | Validation | Elapsed Time: 0:01:31 | Steps: 2 | Loss: 2535.590118 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dev.csv
I Saved new best validating model with loss 2535.590118 to: /home/lee/tmp/CHECKPOINT/deepspeech-0.5.1-checkpoint/best_dev-467468
Epoch 1 |   Training | Elapsed Time: 0:15:03 | Steps: 3 | Loss: 760.936452                                          WARNING:tensorflow:From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py:960: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to delete files with this prefix.
W1001 12:02:01.948611 139667258488640 deprecation.py:323] From /home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py:960: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to delete files with this prefix.
Epoch 1 |   Training | Elapsed Time: 1:19:22 | Steps: 7 | Loss: 1684.548850
Epoch 1 | Validation | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/devEpoch 1 | Validation | Elapsed Time: 0:00:10 | Steps: 1 | Loss: 579.284729 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dEpoch 1 | Validation | Elapsed Time: 0:01:10 | Steps: 2 | Loss: 2555.643829 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/Epoch 1 | Validation | Elapsed Time: 0:01:10 | Steps: 2 | Loss: 2555.643829 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dev.csv
Epoch 2 |   Training | Elapsed Time: 1:20:34 | Steps: 7 | Loss: 1681.662415
Epoch 2 | Validation | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/devEpoch 2 | Validation | Elapsed Time: 0:00:09 | Steps: 1 | Loss: 577.282166 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dEpoch 2 | Validation | Elapsed Time: 0:01:12 | Steps: 2 | Loss: 2526.384003 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/Epoch 2 | Validation | Elapsed Time: 0:01:12 | Steps: 2 | Loss: 2526.384003 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dev.csv
I Saved new best validating model with loss 2526.384003 to: /home/lee/tmp/CHECKPOINT/deepspeech-0.5.1-checkpoint/best_dev-467482
Epoch 3 |   Training | Elapsed Time: 1:22:25 | Steps: 7 | Loss: 1681.011902
Epoch 3 | Validation | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/devEpoch 3 | Validation | Elapsed Time: 0:00:10 | Steps: 1 | Loss: 579.399658 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dEpoch 3 | Validation | Elapsed Time: 0:01:19 | Steps: 2 | Loss: 2553.752808 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/Epoch 3 | Validation | Elapsed Time: 0:01:19 | Steps: 2 | Loss: 2553.752808 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dev.csv
Epoch 4 |   Training | Elapsed Time: 1:11:01 | Steps: 7 | Loss: 1681.635319
Epoch 4 | Validation | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/devEpoch 4 | Validation | Elapsed Time: 0:00:07 | Steps: 1 | Loss: 577.048096 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dEpoch 4 | Validation | Elapsed Time: 0:00:56 | Steps: 2 | Loss: 2535.951538 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/Epoch 4 | Validation | Elapsed Time: 0:00:56 | Steps: 2 | Loss: 2535.951538 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dev.csv
Epoch 5 |   Training | Elapsed Time: 1:04:23 | Steps: 7 | Loss: 1680.117279
Epoch 5 | Validation | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/devEpoch 5 | Validation | Elapsed Time: 0:00:06 | Steps: 1 | Loss: 577.872864 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dEpoch 5 | Validation | Elapsed Time: 0:00:49 | Steps: 2 | Loss: 2547.607330 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/Epoch 5 | Validation | Elapsed Time: 0:00:49 | Steps: 2 | Loss: 2547.607330 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dev.csv
Epoch 6 |   Training | Elapsed Time: 1:04:02 | Steps: 7 | Loss: 1678.914756
Epoch 6 | Validation | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/devEpoch 6 | Validation | Elapsed Time: 0:00:06 | Steps: 1 | Loss: 576.744080 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dEpoch 6 | Validation | Elapsed Time: 0:00:50 | Steps: 2 | Loss: 2534.232880 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/Epoch 6 | Validation | Elapsed Time: 0:00:50 | Steps: 2 | Loss: 2534.232880 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dev.csv
Epoch 7 |   Training | Elapsed Time: 1:04:24 | Steps: 7 | Loss: 1681.608241
Epoch 7 | Validation | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/devEpoch 7 | Validation | Elapsed Time: 0:00:07 | Steps: 1 | Loss: 578.338135 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dEpoch 7 | Validation | Elapsed Time: 0:00:49 | Steps: 2 | Loss: 2558.250610 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/Epoch 7 | Validation | Elapsed Time: 0:00:49 | Steps: 2 | Loss: 2558.250610 | Dataset: /home/lee/tmp/DS_TRAINING/DEV/dev.csv
I Early stop triggered as (for last 4 steps) validation loss: 2558.250610 with standard deviation: 5.941261 and mean: 2539.263916
I FINISHED optimization in 9:58:00.475419
Error: Trie file version mismatch (3 instead of expected 4). Update your trie file.
terminate called after throwing an instance of 'int'
Fatal Python error: Aborted

Thread 0x00007f0695c7b700 (most recent call first):
    File "/usr/lib/python3.6/threading.py", line 295 in wait
    File "/usr/lib/python3.6/queue.py", line 164 in get
    File "/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 159 in run
    File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
    File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007f069647c700 (most recent call first):
    File "/usr/lib/python3.6/threading.py", line 295 in wait
    File "/usr/lib/python3.6/queue.py", line 164 in get
    File "/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 159 in run
    File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
    File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Current thread 0x00007f06d1543740 (most recent call first):
    File "/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/ds_ctcdecoder/swigwrapper.py", line 240 in init
    File "/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/ds_ctcdecoder/__init__.py", line 23 in __init__
    File "/home/lee/tmp/DeepSpeech/evaluate.py", line 45 in evaluate
    File "DeepSpeech.py", line 554 in test
    File "DeepSpeech.py", line 824 in main
    File "/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/absl/app.py", line 250 in _run_main
    File "/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/absl/app.py", line 299 in run
    File "/home/lee/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 40 in run
    File "DeepSpeech.py", line 836 in <module>
Aborted (core dumped)

For generating the trie and checkpoints, I used what was in the docs:

wget https://github.com/mozilla/DeepSpeech/releases/download/v0.5.1/deepspeech-0.5.1-models.tar.gz
tar xvfz deepspeech-0.5.1-models.tar.gz
wget https://github.com/mozilla/DeepSpeech/releases/download/v0.5.1/deepspeech-0.5.1-checkpoint.tar.gz
tar xvfz deepspeech-0.5.1-checkpoint.tar.gz

I haven’t regenerated from v0.5.1 binaries. Do you think I should?

personal.dummy · October 2, 2019, 1:09pm

My training script below:

--checkpoint_dir '/home/lee/tmp/CHECKPOINT/deepspeech-0.5.1-checkpoint' \
--train_files '/home/lee/tmp/DS_TRAINING/TRAIN/train.csv' \
--dev_files '/home/lee/tmp/DS_TRAINING/DEV/dev.csv' \
--test_files '/home/lee/tmp/DS_TRAINING/TEST/test.csv' \
--train_batch_size 1 \
--dev_batch_size 1 \
--test_batch_size 1 \	
--n_hidden 2048 \	
--epochs 1 \
#--validation_step 1 \
--early_stop True \
--earlystop_nsteps 6 \
--estop_mean_thresh 0.1 \
--estop_std_thresh 0.1 \
--dropout_rate 0.22 \
--learing_rate 0.0001 \
--report_count 100 \
--use_seq_length False \
--export_dir '/home/lee/tmp/EXPORT' \	
--alphabet_config_path '/home/lee/tmp/MODELS/deepspeech-0.5.1-models/alphabet.txt' \
--lm_binary_path '/home/lee/tmp/MODELS/deepspeech-0.5.1-models/lm.binary' \
--lm_trie_path '/home/lee/tmp/MODELS/deepspeech-0.5.1-models/trie' \

lissyx · October 2, 2019, 1:12pm

I already gave you answers. I can’t help more.

I don’t know what you are doing precisely, so I cannot tell.