As my home monitoring experiment has become such a celebrity and as it has so much going on, I’m trying to make sure I won’t ever lose its configuration and data, no matter what would happen. Here are some tips and reminders for you, too.
In the IT world, so much can go wrong. Hardware can die, files can become corrupt, malware can hit you, hackers can breach your systems, buggy software updates can cause havoc, you can fat-finger some commands or click on the wrong place and remove data… the list is endless. Here are some ways I’m attempting to protect my home monitoring environment.
Table of Contents
Keep software updated
First things first. In today’s malicious world, it’s mandatory that you keep your software updated. With Linux and Zabbix, you don’t have an excuse to skip the updates. Updating your systems is fast and trouble-free. In small environments like home, even major Zabbix upgrades are fast as the database is not very big, so the database migrations that in a corporate environment can take time, will go through in minutes if not faster.
Remember to backup
Keep your backups in good shape. In my case, I do take backups with BackupPC and monitor my backups with Zabbix.
But I don’t trust one environment. What if my BackupPC says Kaboom? A nightly cron job also copies backup archives from my Raspberry Pi to my Mac, which in turn mirrors the backups to my iCloud. And, there’s one more cloud service I’m using for all my backups but not mentioning it here by name.
Test your backups
As long as you have not verified that your backups do actually work, you do not have a working backup. Have a virtual machine into which you can try to restore your backups. See if they work. Test them periodically, either manually or figure out an automated way to do that.
In the case of Zabbix, you can make your primary Zabbix monitor your test environment, and make Zabbix alert if your restore environment Zabbix suddenly starts responding back something else than the regular login page, or if the restore environment database doesn’t come back with some query response you would expect.
Setup a HA cluster
As any hardware can die, it’s not a bad idea to set up a HA cluster. Last winter I was preparing for potential electricity blackouts here in Finland and did setup my laptop to be a secondary node for my Zabbix. This setup has been working very well.
Use strong passwords
Even if it would only be your sandbox environment where you do test new stuff, please remember to use strong passwords. An evil actor can attempt to breach more targets in your network through a single point of failure.
Use ssh keys
Instead of username + password combination, use ssh keys for ssh authentication. Keys are immune to brute-force attempts and with tuning, ssh keys can also be allowed to only connect from specific IPs and run only specific commands. You know how in your ~/.ssh/authorized_keys the lines do start with something like
ssh-rsa aZfgT12b(....
but if you modify it to be
command="/usr/bin/rsync" ssh-rsa aZfgT12b(....
well, then only rsync would be allowed.
Or, for IP address limitation
from="123.123.123.123" ssh-rsa aZfgT12b(....
Of course, these can be combined:
from="123.123.123.123" command="/usr/bin/rsync" ssh-rsa aZfgT12b(....
Obviously, this grants you much more security than traditional logins.
Use HashiCorp Vault
OK, I admit I’m not doing this at home as it would be overkill for my few logins. But, in a larger environment with absolutely critical safety requirements, use HashiCorp Vault for protecting your credentials. Zabbix has native support for it.
Monitor your logs
Setup a centralized log server — it can be your Zabbix server environment, too — and make sure you monitor the logs. My Zabbix gets all my logs, but wouldn’t be a bad idea to use more advanced log monitoring tools, too. Since I already do have ElastiFlow running at home, at some point I might start utilizing Elasticsearch for the logs. Not doing it much yet.
Use chkrootkit, AIDE, others
Tools like AIDE or chkrootkit can help you detect an intrusion. Set them to run in your cron and get alerted in case of any anomalies. Maybe I’ll one day integrate Zabbix with these tools.
Firewall your environment, use VPN
Don’t allow direct access from the Internet to your Zabbix, or your database, or anything really. In my case, my Asus router allows setting up OpenVPN connections, so that’s what I use. Whilst on the go, I just connect to OpenVPN on my phone and do whatever I need remotely through that.
Anything else?
Did I miss something? Let me know in the comments.
This post was originally published on the author’s page.
