Segmentation fault

Deep Speech
#1

Dear Support.

I am facing an error.

I Initializing variables…
I STARTING Optimization
Epoch 0 | Training | Elapsed Time: 0:37:58 | Steps: 3796 | Loss: 57.462339 Fatal Python error: Segmentation fault

Thread 0x00007f6ec7944700 (most recent call first):
File “/home/user/anaconda3/envs/zml/lib/python3.6/threading.py”, line 295 in wait
File “/home/user/anaconda3/envs/zml/lib/python3.6/queue.py”, line 164 in get
File “/home/user/.local/lib/python3.6/site-packages/tensorflow/python/summary/writer/event_file_writer.py”, line 159 in run
File “/home/user/anaconda3/envs/zml/lib/python3.6/threading.py”, line 916 in _bootstrap_inner
File “/home/user/anaconda3/envs/zml/lib/python3.6/threading.py”, line 884 in _bootstrap

Thread 0x00007f7012e17700 (most recent call first):
File “/home/user/anaconda3/envs/zml/lib/python3.6/threading.py”, line 295 in wait
File “/home/user/anaconda3/envs/zml/lib/python3.6/queue.py”, line 164 in get
File “/home/user/.local/lib/python3.6/site-packages/tensorflow/python/summary/writer/event_file_writer.py”, line 159 in run
File “/home/user/anaconda3/envs/zml/lib/python3.6/threading.py”, line 916 in _bootstrap_inner
File “/home/user/anaconda3/envs/zml/lib/python3.6/threading.py”, line 884 in _bootstrap

Current thread 0x00007f70e9fad740 (most recent call first):
File “/home/user/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 1429 in _call_tf_sessionrun
File “/home/user/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 1341 in _run_fn
File “/home/user/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 1356 in _do_call
File “/home/user/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 1350 in _do_run
File “/home/user/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 1173 in _run
File “/home/user/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 950 in run
File “DeepSpeech.py”, line 599 in run_set
File “DeepSpeech.py”, line 631 in train
File “DeepSpeech.py”, line 938 in main
File “/home/user/.local/lib/python3.6/site-packages/absl/app.py”, line 250 in _run_main
File “/home/user/.local/lib/python3.6/site-packages/absl/app.py”, line 299 in run
File “DeepSpeech.py”, line 965 in
./run_training.sh: line 44: 13250 Segmentation fault (core dumped) python3 -u DeepSpeech.py

This is at checkpoint or change of epoch (I guess)

I want to ask that what might be the reason?
I have

TensorFlow: v1.14.0-21-ge77504a
DeepSpeech: v0.6.0-alpha.15-0-gb3787ee

If you can guide.

Greetings.

#2

It isn’t, the stack shows it comes from a training step. I don’t know what’s causing this but I would start by not using Anaconda and seeing if that fixes it.

#3

Getting similar error on running this test

python evaluate.py --test_files data/CV/ur/clips/test.csv

I Restored variables from best validation checkpoint at /home/hashim/.local/share/deepspeech/checkpoints/best_dev-245252, step 245252
Testing model on data/CV/ur/clips/test.csv
Test epoch | Steps: 0 | Elapsed Time: 0:00:00 2019-12-19 16:46:39.923201: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=–tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=–xla_hlo_profile.
2019-12-19 16:46:39.961734: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-12-19 16:46:40.079803: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Fatal Python error: Segmentation fault

Thread 0x00007f1cc68e4740 (most recent call first):
File “/home/hashim/Desktop/Hashim/UrduCorpus/deepspeech-venv-0.6.0/lib/python3.7/site-packages/ds_ctcdecoder/init.py”, line 116 in ctc_beam_search_decoder_batch
File “evaluate.py”, line 122 in run_test
File “evaluate.py”, line 155 in evaluate
File “evaluate.py”, line 168 in main
File “/home/hashim/Desktop/Hashim/UrduCorpus/deepspeech-venv-0.6.0/lib/python3.7/site-packages/absl/app.py”, line 250 in _run_main
File “/home/hashim/Desktop/Hashim/UrduCorpus/deepspeech-venv-0.6.0/lib/python3.7/site-packages/absl/app.py”, line 299 in run
File “evaluate.py”, line 177 in
Segmentation fault (core dumped)

@reuben I am continuing training from the release model, using my own data [train,dev,test].csv and building new lm as described in v0.6.0

Training went well and model was exported, but when i test it, i get the above error

#4

That is not the same error. In your case, it happens in the ds_ctcdecoder package. Please share pip list|grep ctcdecoder ?

#5

pip list|grep ctcdecoder
ds-ctcdecoder 0.6.0

#6

I also tried testing with

deepspeech --model export/output_graph.pb --lm data/lm/lm.binary --trie data/lm/trie --audio data/CV/ur/clips/026-26-muha.wav

but getting following similar error

deepspeech --model export/output_graph.pb --lm data/lm/lm.binary --trie data/lm/trie --audio data/CV/ur/clips/026-26-muha.wav
Loading model from file export/output_graph.pb
TensorFlow: v1.14.0-21-ge77504a
DeepSpeech: v0.6.0-0-g6d43e21
Warning: reading entire model file into memory. Transform model file into an mmapped graph to reduce heap usage.
2019-12-19 17:05:39.753428: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-12-19 17:05:39.754553: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-12-19 17:05:39.781038: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-19 17:05:39.781330: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce GTX 1060 6GB major: 6 minor: 1 memoryClockRate(GHz): 1.7085
pciBusID: 0000:01:00.0
2019-12-19 17:05:39.781338: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2019-12-19 17:05:39.781378: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-19 17:05:39.781625: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-19 17:05:39.781857: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-12-19 17:06:30.447586: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-19 17:06:30.447611: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2019-12-19 17:06:30.447616: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2019-12-19 17:06:30.468836: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-19 17:06:30.469127: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-19 17:06:30.469367: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-19 17:06:30.469602: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5585 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0, compute capability: 6.1)
Loaded model in 51.0s.
Loading language model from files data/lm/lm.binary data/lm/trie
Loaded language model in 0.0106s.
Running inference.
2019-12-19 17:06:30.797118: W tensorflow/core/framework/allocator.cc:107] Allocation of 134217728 exceeds 10% of system memory.
2019-12-19 17:06:30.876340: W tensorflow/core/framework/allocator.cc:107] Allocation of 134217728 exceeds 10% of system memory.
2019-12-19 17:06:31.079532: W tensorflow/core/framework/allocator.cc:107] Allocation of 134217728 exceeds 10% of system memory.
2019-12-19 17:06:31.139260: W tensorflow/core/framework/allocator.cc:107] Allocation of 134217728 exceeds 10% of system memory.
2019-12-19 17:06:31.198683: W tensorflow/core/framework/allocator.cc:107] Allocation of 134217728 exceeds 10% of system memory.
Segmentation fault (core dumped)

#7

That’s a good and a bad news. Good news, it’s not the python package itself, bad news, it’s the ctc decoder.

Are those our files ? Or yours ?

Does the official english released model works ?

#8

It is created from my data

#9

Can you verify with our model ?

#10

Yeah it work if i use the lm of DS. i will try to re-check my lm and create again and verify if its working or not.
as of now, the issue due to incorrect lm and trie.

Thanks for the prompt help @lissyx. As always you rock :+1: :+1:

#11

Please have a look at data/lm it should guide you.