CPU inference never ends

I have ran CPU inference for 1 min audio and it took around 20 sec.
By that logic 30 min audio should take 10 min…
I am running inference for 30 min file for over an hour now - the CPU load is 50% and there are no errors and it still running.

Any suggestions?

deepspeech --model models/output_graph.pbmm --alphabet models/alphabet.txt --lm models/lm.binary --trie models/trie --audio 30m.wav 
Loading model from file models/output_graph.pbmm
TensorFlow: v1.13.1-10-g3e0cc53
DeepSpeech: v0.5.1-0-g4b29b78
2019-10-14 19:08:31.699341: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2019-10-14 19:08:31.738316: E tensorflow/core/framework/op_kernel.cc:1325] OpKernel ('op: "UnwrapDatasetVariant" device_type: "CPU"') for unknown op: UnwrapDatasetVariant
2019-10-14 19:08:31.738390: E tensorflow/core/framework/op_kernel.cc:1325] OpKernel ('op: "WrapDatasetVariant" device_type: "GPU" host_memory_arg: "input_handle" host_memory_arg: "output_handle"') for unknown op: WrapDatasetVariant
2019-10-14 19:08:31.738406: E tensorflow/core/framework/op_kernel.cc:1325] OpKernel ('op: "WrapDatasetVariant" device_type: "CPU"') for unknown op: WrapDatasetVariant
2019-10-14 19:08:31.738544: E tensorflow/core/framework/op_kernel.cc:1325] OpKernel ('op: "UnwrapDatasetVariant" device_type: "GPU" host_memory_arg: "input_handle" host_memory_arg: "output_handle"') for unknown op: UnwrapDatasetVariant
Loaded model in 0.13s.
Loading language model from files models/lm.binary models/trie
Loaded language model in 23.2s.
Warning: original sample rate (48000) is different than 16kHz. Resampling might produce erratic speech recognition.
Running inference.

Dealing with 30 mins at once is likely to consume much more memory. Also, it seems resampling is needed. Who knows where it is blocked right now. Have you checked what process consumes the CPU ? How is the memory and swap usage?

I think you were right. It was memory issue (I have no facts) as I killed that VM. And started afresh with double RAM… It took about 20min on CPU to finish. Thank you.

1 Like

I’m curious how much memory was used :slight_smile: