Concerns about Raspi SD card writes

Mozilla IoT
#1

My Gateway currently runs on a Raspi 3. Which works well but I’m concerned about disk writes killing the micro SD card. Right now it looks like the Gateway is written assuming that disk writes aren’t a problem (voluminous syslogs, SQLite activity, logging for graphs, etc.)

But disk writes are a big problem when running on Raspi SD cards. Mounting an external drive on a Raspi is an expensive kludge. Raspis aren’t designed for 24/7/multiple-years use cases.

One way to sidestep disk writes being a problem is to use a conventional PC. A refurb PC doesn’t cost much more than a Raspi, assuming you buy everything new and maybe need an powered external USB hub to support power hungry dongles. Real good business class refurb PCs are in the $90 range.

It isn’t that difficult to install a, say, Debian distro onto a refurb. But what is daunting is your instructions for installing the Gateway software .

Using a cheap refurb would be more attractive if there was a way of installing the gateway on a standard Linux distro with a single apt install. Or (better yet) custom roll a Debian distro with the gateway bundled, like you are now doing with the Raspis.

TIA.

1 Like
#2

It’s hard to get around the disk writes, especially when you have (optional) features like data logging. However, SD cards have certainly gotten better in recent years than they used to be.

We have been considering an installable Debian package, but have not yet done it. There is currently a snap package in development, which we’re hoping to have for the 0.10 release. There is also a Docker image available.

1 Like
#3

You could reduce the number of disk writes by making sys logging be granular. For example, GoDaddy’s cheap web hosting defaults to doing no sys logging (for the user’s hosting account). If you need sys logging to debug something, you can enable it. But it times out after (I think) five days and then reverts to no sys logging. This seems reasonable if there are no deep problems to debug.

I wasn’t familiar with Snap. I looked at it. Am I correct in assuming that it provides roughly the packaging functionality of apt or rpm, only better? If so, then this would be neat and is one more thing to look forward to in 0.10.

For now, I’m crossing my fingers and trusting in my SanDisk High Endurance card. And in backing up.

#4

Snap packages are not really the same as rpm and debs. Each package resides in their own folder with their own needed system libs, etc. They do not share your existing system resources. Not a bad thing, and I use it for Blender, only because the Fedora Blender packages lack in some functionality. Same for GIMP.

The best way IMO to avoid SDCard writes is not to use an SDCard. As I mentioned in another thread here, it is now possible to network boot the RPi3. My goal is to make my RPi3 network bootable in the near future. Perhaps look into it and see what you think.

#5

It’s hard to get around the disk writes, especially when you have (optional) features like data logging.

Just so I understand this, if I enable a log of a lamp that is turned on once in the morning and is turned off in the evening, this is only a few disk writes per day, correct? Once when it is turned on and a second when it is turned off.

Ditto for a motion sensor. There will be a log write when motion is sensed and a 2nd log write when motion stops being sensed.

This is how I’m using thing logs. There are a minuscule number of writes, compared to normal syslogging. If I understand how thing logs work.

#6

Yes, your assessment is correct about data logging.

As for syslogging, you could do a couple things:

  • For the normal system, make /var/log and other log locations tmpfs.
  • For the gateway, you can change the log level here. Alternatively, you could mount a tmpfs to ~/.mozilla-iot/log. However, that would also put your data logs in the tmpfs.
#7

I will also say that it’s helpful to have a lot of logging right now by default, given that we’re not exactly production-level yet. The extra logging helps us figure out a lot of issues.

#8

Is it currently possible to mount and log to a samba share?

#9

I don’t see why not. You’ll just have to do it manually.

#10

My only concern was the fact that since samba uses Windows permissions, it would prevent the Gateway from writing to it, because of it wanting to use Linux perms.

#11

Continuing the discussion from Concerns about Raspi SD card writes:

  1. Using Docker

I built my system using an old laptop with ubuntu -> docker -> IOT… Pulling the latest mozilla IOT docker image is pretty simple. There is a learning curve using docker to install/start/stop images but it’s not difficult if you have linux experience. It eliminates having to support packages and provides a very easy way to pull (install) them.

  1. Log File Growth & Rasp PI SD Cards

There are devices that must be “polled” that cause increased write volumes during normal operation. My ZWave GE Jasco light switches are polled every 20-30 seconds which introduce a log entry per devices per poll. With 3 switches, my logfiles are 2-3MB/day. I don’t care since I personally don’t use a PI, but understand the issue… I would prefer to keep polled events in the logs as it documents what is happening.

In another thread, I suggested supporting “Log Levels” to increase or decrease information being logged which would support both our issues.

  1. Support For SYSLOG ?

Thinking yesterday, I wondered how hard it would be to send/redirect log messages to a syslog host in addition to, or instead of, logging locally.

  1. Support Ram FS to store log files

In one of my jobs we needed to process GBs of real-time data every 15m. I created a Ram FS partition to process these files and summarize the data they contained. Keeping log files in memory or Ram FS is another option that would eliminate disk IO at the cost of loosing them when rebooting…

I also thought that being able to send SNMP alerts would be cool too (but I digress).

Eric Edberg

#12

I recently had an issue where the RPi crashed and on reboots would keep crashing, unknown reason, but reflashing the SD card fixed it so I presume the filesystem got corrupted somehow.

Options to run on some hardware without the need for an SD card will be helpful. I will certainly look at the Docker option when I get chance.

#13

@lacojim That’s more of a general Linux/Samba/cifs problem, not really specific to the gateway. As long as the pi user can write to your mountpoint, it would be fine.

1 Like
#14

In one of my jobs we needed to process GBs of real-time data every 15m. I created a Ram FS partition to process these files and summarize the data they contained. Keeping log files in memory or Ram FS is another option that would eliminate disk IO at the cost of loosing them when rebooting…

My Gateway Pi has never rebooted by itself. So logging to the equivalent of a RAM FS that could be flushed to disk on demand (i.e., before rebooting, or if the RAM FS is close to getting filled) would be a good solution.

If the gateway could do this transparently this would go a long way towards making a Pi work as a gateway 24/7/years.

Running a Pi 24/7/years is possible. I have a Pi Zero that monitors an APC UPS with apcupsd. I just checked it

20:07:29 up 363 days,  5:29,  2 users,  load average: 0.09, 0.09, 0.02

Almost a year. apcupsd only writes to disk when there is a power event (power failure, UPS bi-weekly self test, etc.) So apcupsd disk writes are usually counted per month, not per minute. (This one won’t make it to “years” because the UPS battery is due to be replaced soon.)