Output_graph .pb

Deep Speech
#1

Hey

I wan to know if I can generate the output_graph.pb file every epoch

Thx :slight_smile:

#2

Export only happens at the end of the process. Why would you want to do that ?

#3

J’ai lancé l’entraînement du modèle avec 3000 epochs en utilisant le GPU mais malheureusement mon PC s’est éteint au bout du deuxième jour de l’entrainement (J’ai obtenu qlq checkpoint mais pas le modèle (.pb)). Je sais pas vraiment la raison mais Je crois parce qu’il s’est échauffé.
Je voudrai relancer l’entrainement mais j’ai peur de finir par rien obtenir, du coup j’avais cette idée de sauvegarder le modèle et de générer le fichier .pb après chaque (n epochs)

#4

@yasine.nifa Can we keep english here ? French can be used in this other sub-forum, if you need assistance: https://discourse.mozilla.org/c/voice/fr

#5

So, you don’t have to perform an export at every step if you fear about loosing data: checkpoints are here to keep track and restart from a good point.

#6

Did it abruptly shutdown ? It could be a mis-behaving PSU or just too small to drive your system. Can you share details on your hardware setup ?

#7

okey I did not know that
thx :slight_smile:

#8

So I just have to relance the execution.

#9

I am not sure if it abruptly shutdown, because I was sleeping.
ubuntu 18.04.2 LTS
Memory : 15.6 Gio
Processor : Intel® Core™ i7-7700 CPU @ 3.60GHz × 8
Graphic card : GeForce GTX 1070/PCIe/SSE2
operating system : 64-bits
Disk : 783.5 Go

#10

What’s your PSU ? How is your GPU plugged with PCIe power cables ?