Time skew handling -- how long to get back to correct time?

Mozilla IoT
#1

We just had a long power outage (~19 hr), and a few devices with time based rules are acting “wacky”. The DateTime adapter is reporting the correct time (and is configured for the correct timezone), but the log shows the lights coming on at very odd times. It’s likely “by design”, and I just don’t understand how the various pieces are interacting. This was a “ugly” return of service, which may have surfaced some corner conditions.

The power returned about 17:45 PT, but internet wasn’t available until 19:45. Since I don’t run a local NTP server, I think the gateway (rpi) would have started at the epoch.

One light is controlled by a rule that turns on at “19:00 or ‘dark’”. It triggered at 01:07, was accurate for its morning schedule of 06:00 to 07:30 (time only - no ‘dark’), but came on at 14:17 under the “19:00 or ‘dark’” rule. I’m leaving the rule as is, to see what happens tonight.

My expectation was that everything would be back in sync before today’s 14:17 turn on. (There are issues with initial states that may be in play here.)

What’s the target expectation for a case like this?

–Hal

Also, in looking into this, I noticed that the gateway itself is configured for ‘Europe/London’ timezone. That shouldn’t make a difference, but did surprise me :slight_smile: (At least it’s a real time zone, and not “Cupertino”!)

#2

The Raspberry Pi does not have a real-time clock, so I believe your assumption is correct, that it boots up each time with the time set to the epoch. Before starting the gateway, we kick off a process to set the time via NTP, but do not wait for it to be set.

It’s interesting that the DateTime adapter has the correct time… I’m not sure how that’s happening.

Your expectation should be correct, though, that schedules return to normal once you have NTP back.

As for the time zone… the Raspberry Pi was developed in the UK. Our image is based on Raspbian, which defaults to Europe/London.

How did things go after the first day?

#3

Everything went fine, it’s all back rock solid.

I didn’t track exactly when I noticed that the DateTime adapter had the correct time. It certainly was after the odd 14:17 turn on, so the entire unit could have had the correct time at that point.

I think I’ll open an issue on this – at least that will document the issue. It may be such a corner case that it’s not worth addressing.

#4

It actually sets the date and time to a date/time around the time that the base image was created (as opposed to the epoch), which is still the same concept. Just thought I’d add that clarification.

#5

Thanks for clarifying. I’d been thinking it might make sense to not act on time-of-day events until ntp has responded. (Event triggers & time delays would work fine.)

That would require a strong and unambiguous signal for that “mode”. I’d been thinking “time in last century” might be such a signal to detect the I-haven’t-heard-from-ntp yet condition. But it sounds like something else would be needed.

#6

Update - I believe the system boots with epoch time, but then systemd decides that is incorrect. From dmesg:

[    3.045375] systemd[1]: System time before build time, advancing clock.

Also, man systemd-timesyncd.service points to some files that could be queried to understand the state of the last attempt to sync.

[/me is still thinking about the corner case I ran into]

1 Like