Checkpoint selection for model export and continued training

Deep Speech
#1

I was trying to find the information in the docs about how the checkpoint is being loaded during model export and continued training, but I haven’t see any information.
I have only seen, that your final model was trained for 75 epochs and that the model has been exported from the step when the validation loss was smallest.

But how is it in default case, is the model exported from checkpoint after the epoch where validation loss was smallest or last checkpoint if we specify --export_dir for training? And how is the checkpoint selected if we continue training, is then the last checkpoint used?

#2

TensorFlow will export if the validation loss improves. Training will continue until we reach either the point of early stopping, if enabled, or if we reach the number of epochs that was requested on command line.

#3

Do you mean, that the TensorFlow exports after each epoch if the validation loss improves? That’s not what I have observed even though --export_dir argument is defined.

#4

Ok, my wording was inaccurate. It updates its checkpoints.

Exporting as .pb or .tflite happens after training, only if you specify --export_dir.

#5

Thank you very much!

And if I see correctly export() function will export from the last checkpoint. It can be overridden by feeding best_dev_checkpoint to get_checkpoint_state if we want to get checkpoint after the epoch where validation loss was lowest. Please correct me if I’m wrong.