Checkpoint resuming-no. of epochs do not match on google colab

Deep Speech
#1

i am trying to resuming training the model deepspeech 0.4.1 from checkpoint available at releases page checkpoint-0.4.1
I am trying to run my code on google colab, but the epochs are different.
on my personal device, the epochs resume from around 10240, but on google colab the epochs resume from 1832.

code executed on drive=
!python3 DeepSpeech.py --n_hidden 2048 --checkpoint_dir /content/drive/‘My Drive’/DeepSpeech/checkpoints_cloud --epoch -3 --dev_files /content/drive/‘My Drive’/DeepSpeech/models/dev/dev.csv --test_files /content/drive/‘My Drive’/DeepSpeech/models/test/test.csv --train_files /content/drive/‘My Drive’/DeepSpeech/models/train/train.csv --learning_rate 0.0001 --export_dir /content/drive/‘My Drive’/DeepSpeech/models_new
output=
I Training epoch 1832

code executed on personal device=
python3 DeepSpeech.py --n_hidden 2048 --checkpoint_dir /home/raunak/DeepSpeech/deepspeech-0.4.1-checkpoint --epoch -3 --dev_files /home/raunak/DeepSpeech/models/dev/dev.csv --test_files /home/raunak/DeepSpeech/models/test/test.csv --train_files /home/raunak/DeepSpeech/models/train/train.csv --learning_rate 0.0001 --export_dir /home/raunak/DeepSpeech/models_new
output=
I Training epoch 10246

here deepspeech-0.4.1-checkpoint folder and checkpoints_cloud folder are same. just named differently

the datasets are different too,10246- smaller dataset (30wav,7-8secs each), 1832-larger (180 wavs 10 secs each)

here are few images for reference

Screenshot%20from%202019-03-24%2022-13-02
Screenshot from 2019-03-24 22-13-02.png1457×488 98.7 KB

Screenshot%20from%202019-03-25%2000-49-57
Screenshot from 2019-03-25 00-49-57.png740×496 89.2 KB

Screenshot%20from%202019-03-25%2000-49-41
Screenshot from 2019-03-25 00-49-41.png744×495 83.8 KB

#2

Hi @kondaraunak,

Have you found the fix for this? If not can anyone suggest what went wrong in this scenario i.e when we initialising training from check point which comes with 0.4.1 each time when we initialising it is staring from different epochs with same dataset . what might be the issue?

#3

You can continue training with your dataset. The epoch number is relative to the size of dataset and batch size.

#4

Can you avoid using images ? It’s unusable for proper debugging and indexing.

#5

Yes, I realized this.

#6

Also, please use proper code formatting for console output, for the same readability reasons.