Epoch 0 | Training | Elapsed Time: 1 day, 9:20:19 | Steps: 87858 | Loss: 6.972850 Killed
WARNING: Logging before flag parsing goes to stderr.
W0831 04:18:35.381413 140353130383104 deprecation_wrapper.py:119] From DeepSpeech_testing.py:853: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.
W0831 04:18:37.063208 140353130383104 deprecation_wrapper.py:119] From /home/administrator/deepspeech/DeepSpeech/DeepSpeech-0.5.1/util/config.py:60: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.
W0831 04:18:37.087497 140353130383104 deprecation_wrapper.py:119] From DeepSpeech_testing.py:835: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.
W0831 04:18:37.087680 140353130383104 deprecation_wrapper.py:119] From DeepSpeech_testing.py:836: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.
W0831 04:36:03.456215 140353130383104 deprecation.py:323] From /usr/local/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
options available in V2.
- tf.py_function takes a python function which manipulates tf eager
tensors instead of numpy arrays. It’s easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means tf.py_functions can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
- tf.numpy_function maintains the semantics of the deprecated tf.py_func
(it is not differentiable, and manipulates numpy arrays). It drops the
stateful argument making all functions stateful.
W0831 04:36:03.543363 140353130383104 deprecation_wrapper.py:119] From DeepSpeech_testing.py:383: The name tf.data.Iterator is deprecated. Please use tf.compat.v1.data.Iterator instead.
W0831 04:36:03.543499 140353130383104 deprecation.py:323] From DeepSpeech_testing.py:383: DatasetV1.output_types (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.data.get_output_types(dataset).
W0831 04:36:03.543672 140353130383104 deprecation.py:323] From DeepSpeech_testing.py:384: DatasetV1.output_shapes (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.data.get_output_shapes(dataset).
W0831 04:36:03.543839 140353130383104 deprecation.py:323] From DeepSpeech_testing.py:385: DatasetV1.output_classes (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.data.get_output_classes(dataset).
W0831 04:36:03.545172 140353130383104 deprecation.py:323] From /usr/local/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.data.get_output_types(iterator).
W0831 04:36:03.545303 140353130383104 deprecation.py:323] From /usr/local/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.data.get_output_shapes(iterator).
W0831 04:36:03.545415 140353130383104 deprecation.py:323] From /usr/local/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.data.get_output_classes(iterator).
W0831 04:36:03.546030 140353130383104 deprecation_wrapper.py:119] From DeepSpeech_testing.py:396: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
W0831 04:36:03.548312 140353130383104 deprecation_wrapper.py:119] From DeepSpeech_testing.py:215: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.
W0831 04:36:03.548471 140353130383104 deprecation_wrapper.py:119] From DeepSpeech_testing.py:251: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.
W0831 04:36:03.631734 140353130383104 deprecation.py:506] From /usr/local/lib/python3.7/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0831 04:36:04.187909 140353130383104 deprecation_wrapper.py:119] From DeepSpeech_testing.py:190: The name tf.nn.ctc_loss is deprecated. Please use tf.compat.v1.nn.ctc_loss instead.
W0831 04:36:04.189303 140353130383104 deprecation.py:323] From DeepSpeech_testing.py:194: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0831 04:36:04.316739 140353130383104 deprecation_wrapper.py:119] From DeepSpeech_testing.py:279: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.
W0831 04:36:04.388463 140353130383104 deprecation_wrapper.py:119] From DeepSpeech_testing.py:340: The name tf.summary.histogram is deprecated. Please use tf.compat.v1.summary.histogram instead.
W0831 04:36:04.525462 140353130383104 deprecation_wrapper.py:119] From DeepSpeech_testing.py:418: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.
W0831 04:36:04.632269 140353130383104 deprecation_wrapper.py:119] From DeepSpeech_testing.py:422: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.
W0831 04:36:04.632952 140353130383104 deprecation_wrapper.py:119] From DeepSpeech_testing.py:424: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.
W0831 04:36:04.649804 140353130383104 deprecation_wrapper.py:119] From DeepSpeech_testing.py:429: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.
W0831 04:36:04.757282 140353130383104 deprecation.py:323] From /usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
I0831 04:36:04.759114 140353130383104 saver.py:1280] Restoring parameters from ./deepspeech-0.5.1-checkpoint/train-555213
W0831 04:36:17.228476 140353130383104 deprecation_wrapper.py:119] From DeepSpeech_testing.py:365: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.
I Restored variables from most recent checkpoint at ./deepspeech-0.5.1-checkpoint/train-555213, step 555214
I STARTING Optimization
Epoch 0 | Training | Elapsed Time: 0:23:25 | Steps: 2044 | Loss: 9.845757
hi , the session get restart after the steps :87858 and again starting the process from the begin but the total steps to be is 137000.
Epoch 0 | Training | Elapsed Time: 1 day, 9:20:19 | Steps: 87858 | Loss: 6.972850 Killed
so it automatically killing the process
I have gtx 1050 4gb graphics card.
command : CUDA_VISIBLE_DEVICES=0 python3.7 DeepSpeech_testing.py --lm_binary_path ./data/deepspeech_dataset/lm.binary --lm_trie_path ./data/deepspeech_dataset/trie --learning_rate 0.0001 --alphabet_config_path ./data/alphabet.txt --beam_width 1024 --use_seq_length false --remove_export true --notest --train_batch_size 30 --train_files ./data/deepspeech_dataset/august_22_prune_dataset.csv --checkpoint_dir ./deepspeech-0.5.1-checkpoint --nodev --export_dir ./export_final_dataset --epochs 100 --dropout_rate4 0.05 --dropout_rate5 0.05
any suggestion
Thanks