Hi,
I would like to know more about what the IoT enabler package exactly is.
I am interested in Machine learning, AI, data analytics I have developed interest in IoT, however, I do not have much experience with it. Is there something in the project that I can work on?
I also had a doubt about the code repository, is it https://github.com/mozilla/vaani or https://github.com/mozilla/vaani.iot
2 Likes
Hi @oakshweta11,
I believe that actual activity is in the repos linked from https://github.com/mozilla/vaani.iot.
If you don’t get responses from real Project Vaani members (I’m not), don’t hesitate to come and say hello to Project Link, we have a few ideas related to machine learning 
What would you like to know?
The IoT enabler package “Vaani” is an SDK that allows device makers to voice enable their devices (think thermostats, lights, music players…) in addition it allows developers to create “apps” that also control these IoT devices.
In terms of machine learning what experience you have there? We are in the process of opening up to the community a kaggle-like competition based around what is essentially a machine translation task. As is standard for such things, the competition will be judged on BLEU scores. So, if TensorFlow, RNN’s, BRNN’s, LSTM’s, and Sequence-to-Sequence models are not Greek to you and you’re interested, I can give you more details.
Also, we are in the process of creating our own speech recognition engine based off of Deep Speech: Scaling up end-to-end speech recognition, an engine using BRNN and CTC, to convert text to speech. Does that sound of interest?
hi @kdavis!
Experience in ML: I have worked on a project on skin disease detection from images. I have used NNs. I have done courses online that covered the topics listed above. I would like working on the competition task.
I went through the paper, developing a speech recognition engine sounds interesting! Do tell me more about it.
Currently we are collecting data for our training and test sets. So, we’re not ready for you yet 
.
However, I can describe the task in a bit more detail; so, when we’re ready with the data set, you can hit the ground running…
Our current pre-alpha version, see for example the demo[1], allows you to address IoT devices using a limited set of phrases. For example, to turn off a light named “floor lamp” you can only use this phrase
Turn the floor lamp off.
This state of affairs is less than ideal. One would like to address the various IoT devices in a more unconstrained manner and be able to say things such as…
Turn the floor lamp off please.
Please turn the floor lamp off.
It is bright in here; please turn the floor lamp off.
Could you please turn the floor lamp off.
Could you turn the floor lamp off please.
…
The list of possible variations is endless.
So, we are approaching this as a machine learning task. For each canonical sentence, e.g.
Turn the floor lamp off.
we are collecting a set of non-canonical sentences, e.g.
Turn the floor lamp off please.
Please turn the floor lamp off.
It is dark in here; please turn the floor lamp off.
Could you please turn the floor lamp off.
Could you turn the floor lamp off please.
…
then creating a machine learning algorithm that is able to translate from a non-canonical phrase. e.g.
Could you turn the floor lamp off please.
to its canonical form
Turn the floor lamp off.
As this translation process is analogous to the translation of one language to another language, our baseline algorithm uses neural machine translation, a deep learning technique used to translate between languages. In particular, our baseline algorithm is motivated by the work of Bahdanau, Cho, and Bengio[2]. Seems interesting?
With respect to creating our own speech recognition engine based off Deep Speech: Scaling up end-to-end speech recognition, we are advising a set of students that are re-implementing the algorithms there. As a jumping off point for the students, I’ve implemented large parts of the algorithms using TensorFlow in a Python notebook[3]. (The LaTeX equations don’t render correctly in github. So, to see them you have to start your own notebook server.) Note that in this notebook I’ve used BLSTM’S instead of the BRNN from the original paper and I’ve not yet integrated the CTC algorithm[4] released in the newest version of TensorFlow[5]. If you’d be interesting in integrating the CTC algorithm into the notebook that’d be awesome!
As a side-note, it would be great if you could make your library independent from OpenHab, so that it can be reused in other projects.
1 Like
@kdavis Yes sounds interesting! I’ll focus on the machine translation part after I am done with the CTC algorithm integration. If I have any questions regarding the CTC algorithm integration, should I post them here, or is there an IRC channel I can join?
I will keep you updated on the progress I make.
Awesome!
You can post any questions you have here. We’re trying to make it easier to involve the community so switching over to using discourse instead of IRC.
Thanks again!
cool!
I am working on the forward backward algorithm, however, before integrating the code, I tried to run the original notebook, I get this error, I tried to fix it but I keep getting the same message. How should I fix this?:
for the cell:
layer_6 = BiRNN(x, istate_fw, istate_bw, weights, biases)
layer_6 = BiRNN(x, istate_fw, istate_bw, weights, biases)
ValueError Traceback (most recent call last)
in ()
----> 1 layer_6 = BiRNN(x, istate_fw, istate_bw, weights, biases)
in BiRNN(_X, _istate_fw, _istate_bw, _weights, _biases)
29 layer_3,
30 initial_state_fw=_istate_fw,
—> 31 initial_state_bw=_istate_bw)
32
33 # Reshape outputs from a list of n_steps tensors each of shape [batch_size, 2*n_cell_dim]
/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/rnn.pyc in bidirectional_rnn(cell_fw, cell_bw, inputs, initial_state_fw, initial_state_bw, dtype, sequence_length, scope)
360 with vs.variable_scope(name + “_FW”) as fw_scope:
361 output_fw, output_state_fw = rnn(cell_fw, inputs, initial_state_fw, dtype,
–> 362 sequence_length, scope=fw_scope)
363
364 # Backward direction
/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/rnn.pyc in rnn(cell, inputs, initial_state, dtype, sequence_length, scope)
141 zero_output, state, call_cell)
142 else:
–> 143 (output, state) = call_cell()
144
145 outputs.append(output)
/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/rnn.pyc in ()
134 if time > 0: vs.get_variable_scope().reuse_variables()
135 # pylint: disable=cell-var-from-loop
–> 136 call_cell = lambda: cell(input_, state)
137 # pylint: enable=cell-var-from-loop
138 if sequence_length is not None:
/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/rnn_cell.pyc in call(self, inputs, state, scope)
201 # Parameters of gates are concatenated into one multiply for efficiency.
202 c, h = array_ops.split(1, 2, state)
–> 203 concat = linear([inputs, h], 4 * self._num_units, True)
204
205 # i = input_gate, j = new_input, f = forget_gate, o = output_gate
/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/rnn_cell.pyc in linear(args, output_size, bias, bias_start, scope)
697 # Now the computation.
698 with vs.variable_scope(scope or “Linear”):
–> 699 matrix = vs.get_variable(“Matrix”, [total_arg_size, output_size])
700 if len(args) == 1:
701 res = math_ops.matmul(args[0], matrix)
/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.pyc in get_variable(name, shape, dtype, initializer, regularizer, trainable, collections)
332 _get_default_variable_store(), name, shape=shape, dtype=dtype,
333 initializer=initializer, regularizer=regularizer, trainable=trainable,
–> 334 collections=collections)
335
336
/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.pyc in get_variable(self, var_store, name, shape, dtype, initializer, regularizer, trainable, collections, caching_device)
255 full_name, shape=shape, dtype=dtype, initializer=initializer,
256 regularizer=regularizer, reuse=self.reuse, trainable=trainable,
–> 257 collections=collections, caching_device=caching_device)
258
259
/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.pyc in get_variable(self, name, shape, dtype, initializer, regularizer, reuse, trainable, collections, caching_device)
116 " Did you mean to set reuse=True in VarScope? "
117 “Originally defined at:\n\n%s” % (
–> 118 name, “”.join(traceback.format_list(tb))))
119 found_var = self._vars[name]
120 if not shape.is_compatible_with(found_var.get_shape()):
ValueError: Variable BiRNN_FW/BasicLSTMCell/Linear/Matrix already exists, disallowed. Did you mean to set reuse=True in VarScope? Originally defined at:
File “”, line 31, in BiRNN
initial_state_bw=_istate_bw)
File “”, line 1, in
layer_6 = BiRNN(x, istate_fw, istate_bw, weights, biases)
File “/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py”, line 2885, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
What version of tensorflow are you running?
I’ve a hand-compiled version the last commit of which is commit
cb91829
I have tensorflow 0.8.0
I’ll check out the hand compiled version
When I starting writing this the bi-directional stuff wasn’t officially
released and now it looks like they’ve changed the API a bit 
When I starting writing this the bi-directional stuff wasn’t officially
released and now it looks like they’ve changed the API a bit
yup, looks like that
I checked the tensorflow code right now, looks like version 0.8.0 has improvements: https://github.com/tensorflow/tensorflow/blob/r0.8/RELEASE.md
This includes CTC loss and decoders. We might want to check out : https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/ctc
So, all the prep work I did for CTC looks to be unneeded.
However, integrating the TensorFlow CTC is till really helpful!
Yeah! I can work on integrating the TensorFlow CTC
Should I start on a new NoteBook? or will we probably need the prep work done in the future?
You can start on a new notebook if you want and copy over the BLSTM code.
cool, I’ll do that