Finetuning with Common Voice - v0.6.0

Deep Speech
#1

I am trying to finetune the model on CV - the error I get is the following:
How to resolve. Using tf 1.14.0

INFO:tensorflow:Restoring parameters from /home/ubuntu/fcp/deepspeech-0.6.0-checkpoint/best_dev-233784
I0108 13:36:18.091672 139951958247168 saver.py:1280] Restoring parameters from /home/ubuntu/fcp/deepspeech-0.6.0-checkpoint/best_dev-233784
Traceback (most recent call last):
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 1356, in _do_call
return fn(*args)
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.NotFoundError: Key cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias/Adam not found in checkpoint
[[{{node save_1/RestoreV2}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/training/saver.py”, line 1286, in restore
{self.saver_def.filename_tensor_name: save_path})
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 950, in run
run_metadata_ptr)
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 1173, in _run
feed_dict_tensor, options, run_metadata)
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 1350, in _do_run
run_metadata)
File “/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias/Adam not found in checkpoint
[[node save_1/RestoreV2 (defined at DeepSpeech.py:498) ]]

#2

It’s because you’re not using CuDNN RNN. Either specify --use_cudnn_rnn (if you have a compatible GPU) or use --cudnn_checkpoint to specify the location of the pre-trained checkpoint, instead of --checkpoint_dir. In this case, --checkpoint_dir should point to a new location, where the result of the fine-tuning will be stored, separate from the pre-trained checkpoints.

1 Like