I used the following command to train with 0.3.0:
python3 DeepSpeech.py --n_hidden 2048 --checkpoint_dir ~/checkpoint_retrain --epoch -10 --train_files /home/rsandhu/iitm_data/assamese-male-train.csv,/home/rsandhu/iitm_data/assamese-female-train.csv,/home/rsandhu/iitm_data/bengali-male-train.csv,/home/rsandhu/iitm_data/gujarati-male-train.csv,/home/rsandhu/iitm_data/gujarati-female-train.csv,/home/rsandhu/iitm_data/hindi-male-train.csv,/home/rsandhu/iitm_data/hindi-female-train.csv,/home/rsandhu/iitm_data/kannada-male-train.csv,/home/rsandhu/iitm_data/kannada-female-train.csv,/home/rsandhu/iitm_data/malayalam-male-train.csv,/home/rsandhu/iitm_data/malayalam-female-train.csv,/home/rsandhu/iitm_data/manipuri-male-train.csv,/home/rsandhu/iitm_data/manipuri-female-train.csv,/home/rsandhu/iitm_data/rajasthani-male-train.csv,/home/rsandhu/iitm_data/rajasthani-female-train.csv,/home/rsandhu/iitm_data/tamil-male-train.csv,/home/rsandhu/iitm_data/tamil-female-train.csv --dev_files /home/rsandhu/iitm_data/assamese-male-dev.csv,/home/rsandhu/iitm_data/assamese-female-dev.csv,/home/rsandhu/iitm_data/bengali-male-dev.csv,/home/rsandhu/iitm_data/gujarati-male-dev.csv,/home/rsandhu/iitm_data/gujarati-female-dev.csv,/home/rsandhu/iitm_data/hindi-male-dev.csv,/home/rsandhu/iitm_data/hindi-female-dev.csv,/home/rsandhu/iitm_data/kannada-male-dev.csv,/home/rsandhu/iitm_data/kannada-female-dev.csv,/home/rsandhu/iitm_data/malayalam-male-dev.csv,/home/rsandhu/iitm_data/malayalam-female-dev.csv,/home/rsandhu/iitm_data/manipuri-male-dev.csv,/home/rsandhu/iitm_data/manipuri-female-dev.csv,/home/rsandhu/iitm_data/rajasthani-male-dev.csv,/home/rsandhu/iitm_data/rajasthani-female-dev.csv,/home/rsandhu/iitm_data/tamil-male-dev.csv,/home/rsandhu/iitm_data/tamil-female-dev.csv --test_files /home/rsandhu/iitm_data/assamese-male-test.csv,/home/rsandhu/iitm_data/assamese-female-test.csv,/home/rsandhu/iitm_data/bengali-male-test.csv,/home/rsandhu/iitm_data/gujarati-male-test.csv,/home/rsandhu/iitm_data/gujarati-female-test.csv,/home/rsandhu/iitm_data/hindi-male-test.csv,/home/rsandhu/iitm_data/hindi-female-test.csv,/home/rsandhu/iitm_data/kannada-male-test.csv,/home/rsandhu/iitm_data/kannada-female-test.csv,/home/rsandhu/iitm_data/malayalam-male-test.csv,/home/rsandhu/iitm_data/malayalam-female-test.csv,/home/rsandhu/iitm_data/manipuri-male-test.csv,/home/rsandhu/iitm_data/manipuri-female-test.csv,/home/rsandhu/iitm_data/rajasthani-male-test.csv,/home/rsandhu/iitm_data/rajasthani-female-test.csv,/home/rsandhu/iitm_data/tamil-female-test.csv,/home/rsandhu/iitm_data/tamil-male-test.csv --learning_rate 0.0001 --train_batch_size 24 --dev_batch_size 48 --test_batch_size 48 --display_step 0 --validation_step 1 --dropout_rate 0.2 --checkpoint_step 1 --decoder_library_path binaries/libctc_decoder_with_kenlm.so --export_dir ~/new_model
Here are the results of test epoch:
I used the following command for training with 0.4.1 with the same data:
python3 DeepSpeech.py --n_hidden 2048 --checkpoint_dir ~/deepspeech-0.4.1-checkpoint --epoch -10 --train_files /home/rsandhu/iitm_data/assamese-male-train.csv,/home/rsandhu/iitm_data/assamese-female-train.csv,/home/rsandhu/iitm_data/bengali-male-train.csv,/home/rsandhu/iitm_data/gujarati-male-train.csv,/home/rsandhu/iitm_data/gujarati-female-train.csv,/home/rsandhu/iitm_data/hindi-male-train.csv,/home/rsandhu/iitm_data/hindi-female-train.csv,/home/rsandhu/iitm_data/kannada-male-train.csv,/home/rsandhu/iitm_data/kannada-female-train.csv,/home/rsandhu/iitm_data/malayalam-male-train.csv,/home/rsandhu/iitm_data/malayalam-female-train.csv,/home/rsandhu/iitm_data/manipuri-male-train.csv,/home/rsandhu/iitm_data/manipuri-female-train.csv,/home/rsandhu/iitm_data/rajasthani-male-train.csv,/home/rsandhu/iitm_data/rajasthani-female-train.csv,/home/rsandhu/iitm_data/tamil-male-train.csv,/home/rsandhu/iitm_data/tamil-female-train.csv --dev_files /home/rsandhu/iitm_data/assamese-male-dev.csv,/home/rsandhu/iitm_data/assamese-female-dev.csv,/home/rsandhu/iitm_data/bengali-male-dev.csv,/home/rsandhu/iitm_data/gujarati-male-dev.csv,/home/rsandhu/iitm_data/gujarati-female-dev.csv,/home/rsandhu/iitm_data/hindi-male-dev.csv,/home/rsandhu/iitm_data/hindi-female-dev.csv,/home/rsandhu/iitm_data/kannada-male-dev.csv,/home/rsandhu/iitm_data/kannada-female-dev.csv,/home/rsandhu/iitm_data/malayalam-male-dev.csv,/home/rsandhu/iitm_data/malayalam-female-dev.csv,/home/rsandhu/iitm_data/manipuri-male-dev.csv,/home/rsandhu/iitm_data/manipuri-female-dev.csv,/home/rsandhu/iitm_data/rajasthani-male-dev.csv,/home/rsandhu/iitm_data/rajasthani-female-dev.csv,/home/rsandhu/iitm_data/tamil-male-dev.csv,/home/rsandhu/iitm_data/tamil-female-dev.csv --test_files /home/rsandhu/iitm_data/assamese-male-test.csv,/home/rsandhu/iitm_data/assamese-female-test.csv,/home/rsandhu/iitm_data/bengali-male-test.csv,/home/rsandhu/iitm_data/gujarati-male-test.csv,/home/rsandhu/iitm_data/gujarati-female-test.csv,/home/rsandhu/iitm_data/hindi-male-test.csv,/home/rsandhu/iitm_data/hindi-female-test.csv,/home/rsandhu/iitm_data/kannada-male-test.csv,/home/rsandhu/iitm_data/kannada-female-test.csv,/home/rsandhu/iitm_data/malayalam-male-test.csv,/home/rsandhu/iitm_data/malayalam-female-test.csv,/home/rsandhu/iitm_data/manipuri-male-test.csv,/home/rsandhu/iitm_data/manipuri-female-test.csv,/home/rsandhu/iitm_data/rajasthani-male-test.csv,/home/rsandhu/iitm_data/rajasthani-female-test.csv,/home/rsandhu/iitm_data/tamil-female-test.csv,/home/rsandhu/iitm_data/tamil-male-test.csv --learning_rate 0.0001 --train_batch_size 24 --dev_batch_size 48 --test_batch_size 48 --display_step 0 --validation_step 1 --dropout_rate 0.2 --checkpoint_step 1 --lm_alpha 0.75 --lm_beta 1.85 --export_dir ~/new_model
And the results are as below:
I havenât been able to observe the loss for validation for each of the epochs because of WARNING:root:frame length (1536) is greater than FFT size (512), frame will be truncated. Increase NFFT to avoid occurring several hundred times whenever a csv is being preprocessed