Where to start debugging when things don't update

Mozilla IoT
#1

Hi all,
I’m pretty new to the Mozilla IoT/WebThings and started to create a bunch of own sensors (ESP32/ESP8266 based, so I’m using the Arduino WebThing lib and version 0.9.2 of the gateway).

I noticed very strange behaviour of the Temperature und MultiLevel Sensors (e.g. only the latest LevelProperty that was added gets updated - I worked around this problem by simply having multiple devices on a single adapter). However, I noticed that at some point all sensors that I created stop updating (reproducible). I didn’t find a pattern yet when this happens, but the experimental logs feature clearly show that at some point they simply stop updating and so do the downloadable logs- although the devices are still reachable (http://IP/things/DeviceName/properties).

I’m a bit lost on how to debug this without any logs showing a device failure or the devices being unreachable, they are still accessible and provide valid data. Any hints on how I can figure out why the gateway stops updating the things? In some cases the devices start to update again after hours, in other cases they don’t or just some of them (even though they are served by the same adapter on the same device and still providing valid data). Is there any way to find out why a thing wasn’t updated anymore? Frankly speaking, I didn’t figure out yet if a call to adapter->update() pushes the values to the gateway or if the gateway pulls the data regularly and the call just updates the internal data structure that is being published.

I’ll file issues for it once I’m able to confirm it’s really a problem with the gateway and not with my code.

I’m very grateful for any hints. Thanks for all the effort you put into the project so far, great work, I love it!

Thanks,
Chris

MultiLevelSensor Only Updates Second Thing
#2

Is there anything relevant in your log files? It would probably be prefixed by “thing-url”.

You can find the log files in ~/.mozilla-iot/log or in the UI via Settings -> Developer -> View Internal Logs.

#3

I think I found the problem by accident… it seems to be related to mDNS and caused by my router.
The logs didn’t give anything useful other than no more logs for the devices - at some point it simply stopped. Is there anyway to see when the gateway tried to reach the devices the last time? What happens exactly when you fire adapter->update()?

#4

No, I don’t believe that adapter logs anything when it’s polling, but you could add a log if you want.

  • mkdir ~/.mozilla-iot/addons/thing-url-adapter/.git
  • Add a console.log() in poll() in ~/.mozilla-iot/addons/thing-url-adapter/thing-url-adapter.js
  • Disable the “Web Thing” add-on in your UI, wait a few seconds, then re-enable it.
MultiLevelSensor Only Updates Second Thing
#5

Excellent, this is where the actual adapter code is… I’ll try to debug from here and will update the thread once I found the problem.

Cheers! Dank Dir!

#6

As promised, a - hopefully - final reply on the topic. The issues I faced seem to have multiple root causes:

  1. Bad USB cable to power the RaspPi
    No kidding, a shitty, cheap USB cable (in my case it was very likely just too thin in diameter) caused the RaspPi to struggle. A very well known problem… dmesg complained about under voltage: Under-voltage detected! (0x00050005). This caused the RaspPi to reboot in most of the times when you put load on it, e.g. viewing the page of the experimental log feature

  2. ESP8266s behave slightly different to ESP32s:
    In my case, whenever the device rebooted (e.g. wtd kicked in) or reconnects to WIFI the mDNS didn’t work reliable afterwards. With ESP32 that’s no problem, but for the ESP8266 an additional MDNS.update() in the main loop worked miracles… Don’t forget to check for connectivity in the main loop of your thing and re-establish if required - this check is missing in every sample code I’ve seen so far.

  3. Deep-sleep doesn’t work with the gateway, the devices lose connectivity and will only occasionally recover. Bear in mind that the ThingAdapter polls rather than getting the data pushed from the thing.

I’ll polish my code a bit and commit it to my repo for reference. So far I have samples for BME280, soil sensors and I’m working on IR and RF examples to control e.g. sockets or LED strips.

Thanks,
Chris