Visualize Training/Validation steps

Deep Speech
#1

I would like to see the training steps visualized on some criterias (learning curve/accuracy/wer). I want to see how the model is learning so I can tune the parameters more efficiently.
I looked into the DeepSpeech.py code and saw no ready-made way to do this. Is there any easy process so, DeepSpeech.py would generate a plot displaying the learning curve/accuracy/wer on training and validation data set.

Moreover, is it possible to find these information from the saved checkpoint files or the exported model?

#2

@lissyx: Is there any option from which we can visualize the training progress graphically in Deep Speech?

#3

If you give models checkpoint dir to tensorboard, you can get some visualisation.

#4

I’ve trained on checkpoints from v0.5.0 on German Common Voice Dataset with the following parameters (changed in train/dev/test ä->ae etc. so that I can use the english pretrained model) As Deep Speech is using Curriculum Learning the loss (orange) looks alright but the validation loss looks weird, doesn’t it? Early stopped after 16 epochs.

DeepSpeech.py
–checkpoint_dir deepspeech-0.5.0/
–train_files german_training_data/clips/train_working1.csv
–dev_files german_training_data/clips/dev_working1.csv
–test_files german_training_data/clips/test_working1.csv
–n_hidden 2048
–epochs 30
–train_batch_size 12
–dev_batch_size 12
–test_batch_size 12
–learning_rate 0.00001
–validation_step 1
–report_count 10

WER: 1.833333, CER: 0.566038, loss: 91.882980

  • src: “langfristig brauchen wir sowieso einen weltraumaufzug”
  • res: “ling sis dik gob mil su mido ein belt vom mauftzug”

weirdvalidation
weirdvalidation.png1899×617 105 KB

#5

As @pete said, you can use tensorboard, but give it the summary_dir path not the checkpoint_dir, like:

tensorboard --logdir=~/.local/share/deepspeech/summaries
this is the default path for Ubuntu users, put your path which set on running training.
You’ll get the loss information by time, for both train and dev.