I want my users to be able to add custom dictionaries. So the majority of the LM will be the same for all users, but users can add a handful of words to their specific dictionary. I suppose I can add all custom words to the main LM, but in certain cases (like my last name), there are different spellings.
Supporting multiple LM is still an open issue, sadly: https://github.com/mozilla/DeepSpeech/issues/1678