Model with best validation results from training

Deep Speech
#1

When I’m training and using the early stopping, is the output pb model generated from the epoch with best validation result or from the last one?

I noticed that in the release notes, you state “The weights with the best validation loss were selected at the end of the 30 epochs.”.
Is this done automatically by the training script or do you store each checkpoint during the training and pick it manually for the released model somehow?

#2

Hi, can someone from the team please react to this?

If there’s a relevant documentation or publicly accessible code that does the best model selection it would be very helpful as I assume almost everyone who tries to train a model is interested in the best trained model from the run.

#3

I have a patch here that keeps the best validation loss checkpoint during training: https://gist.github.com/reuben/dcc2deaf85568591e34ce363bc3bac2a (previously discussed and shared in this issue: https://github.com/mozilla/DeepSpeech/issues/1475)

It hasn’t been merged because it’s super hacky, TensorFlow’s MonitoredTrainingSession makes it hard to do it cleanly.

#4

Thanks for the link, is this the solution that you use internally to generate “The weights with the best validation loss were selected at the end of the 30 epochs.” ?

#5

Yes, this is the solution we use internally.

#6

ok, thanks for the clarification