Two years later – a small update on running a network of Raspberry Pi based temperature sensors and dashboards via an influxdb/Grafana Pi server.
Hardware changes: nothing much to report – everything is working reliably. A few of the Raspberries Pi Model 2 & 3 have been changed out for Pi Zeros – smaller and still more than powerful enough to run the temperature data gathering. The Pi 4B server hosting the Grafana dashboards and influxdb has been setup to boot/run from a USB3 SSD, the original 4GB memory model was swapped out for an 8GB model during debugging/testing out-of-memory crashes around the “compaction crash loop” issue detailed below.
Software issues: In my initial design I hadn’t set any data retention policy on the influxdb and after ~14 months the database had grown too big to be auto-compressed on the (32-bit kernel) Raspberry Pi 4B server. I quickly found I’d ran into this known issue – I set a sensible date range on my telegraf DB and things looked a lot better. Compaction failure crash loop came back again about 4 months later and I found that the influx dev team were not actively pursuing adding support for “large” DBs in the 32 bit code so I was left debating a few options:
- Export the DB then import onto another (64 bit) machine e.g. laptop running Fedora and let the compaction complete then export/import back to the Pi (too much faff)
- Install 64 bit kernel on the Pi 4B (“experimental”? too scary!)
- Move the influxdb server permanently to another 64 bit machine e.g. Fedora/Debian laptop/server (I wanted to keep the whole thing Pi based)
So for now I chose none of the above and further reduced the data retention time on my temp_logger DB….
I have a simple backup script running every 6 hours that backs up the influxdb to an attached USB stick and to another laptop (which is in turn backed up to a cloud storage service).