Hey guys:
I am training model by using thchs30 datasets. but it is pending in the test decoding predictions step, almost one day.
This is my parameters setting below:
CUDA_VISIBLE_DEVICES=1 python3 -u DeepSpeech.py
–alphabet_config_path $ALPHABET
–lm_binary_path $LM_BINARY
–lm_trie_path $LM_TRIE
–train_files $TRAIN_CSV
–dev_files $DEV_CSV
–test_files $TEST_CSV
–checkpoint_dir “$DATA_DIR/checkpoint”
–export_dir “$DATA_DIR/model”
–train_batch_size 40
–dev_batch_size 40
–test_batch_size 20
–n_hidden 512
–learning_rate 0.0001
–epoch 20
–log_level 0
–summary_secs 10
“$@”
And This is log :
Preprocessing done
2019-01-22 09:04:11.321570: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2019-01-22 09:04:11.321669: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-01-22 09:04:11.321694: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0
2019-01-22 09:04:11.321701: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N
2019-01-22 09:04:11.321973: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 21551 MB memory) -> physical GPU (device: 0, name: Tesla P40, pci bus id: 0000:04:00.0, compute capability: 6.1)
Computing acoustic model predictions…
100% (124 of 124) |##################################################################################################################################################| Elapsed Time: 0:00:43 Time: 0:00:43
Decoding predictions…
N/A% (0 of 124) | | Elapsed Time: 0:00:00 ETA: --:–:--
Does anybody meet same problem? or know how to deal with it?