How can I estimate training time of deepspeech

Deep Speech
#1

I am running deepspeech training on Tedlium-2 and Mozzila open voice.
in approximate 12 hours it reached

  • Training of Epoch 0 - loss - 141.758459
    I used batch size 8 for training, validation and test data with -use_warpctc option. Except these I am using default options.
    Approximately how much time my training should take.
    I am using 1 GeForce GTX 1080 GPU.
#2

How big is your overall dataset? TED-LIUM2 + CommonVoice ? That would be ~1000 hours of audio ?

#3

It should be between 300 - 500 hours. I have used valid sets only

#4

I think that ballpark for ~500 hours on our cluster with 16 TITAN X GPUs is around 2h for one epoch, so 12 hours for one epoch on your 1080 seems consistent.

#5

What total training time I can expect ? Any idea ?

#6

Hello,

Can you provide some hints about the total training time in my case ? Thanks a lot.

#7

I think I just did …

1 Like
#8

Does it mean that on my hardware, to complete 75 epochs it will take ~38 days ?

#9

Yes. Running one full epoch will give you a better estimate.

#10

How much time an epoch will last on TED-LIUM2 if using newly NVIDIA Titan V ? Or what would be the speed up over Titan X Pascal? Will the tensor cores help ?