Zabbix daemons use various approaches to improve performance. One such measure, introduced in Zabbix 1.8, is configuration cache. It resulted in massive performance improvements (up to 10 times), mostly because of a significantly reduced database access (that we looked at in previous blog posts). This cache slightly changes how Zabbix operates, though, and can introduce some delays in configuration information processing. Zabbix 1.8.6 provides a new feature to help with configuration cache management.
What is Zabbix configuration cache
Zabbix server generates configuration cache, mostly containing information on hosts and items to be monitored. It re-creates this cache by default every 60 seconds. This period can be customised by configuration parameter CacheUpdateFrequency.
The cache size itself may be set with configuration parameter CacheSize and by default is 8MB.
This means that at any given time Zabbix server sees a snapshot of the configuration. If you make changes in the frontend, they are written to the database – but not immediately seen by the server.
Be in control of it
Now, if you are not making constant changes to the Zabbix configuration, reloading this cache every minute is not really necessary, and can be a bit resource intensive on large installations. In those cases it would be useful to increase this interval. But then any changes to the configuration would be noticed much later, which is not that nice either. How could this be solved?
Potential solution – some way to initiate reloading of configuration cache without restarting server daemon. Which is what got implemented for Zabbix 1.8.6.
Previously, no solution existed for modifying any Zabbix daemon characteristics while they are running, so there was a new generic decision to make – how to do that, because there might be further, similar functionality in the future. Eventually, this was implemented as a generic “runtime control” flag that gets an option passed to it. To quote “man zabbix_server”:
-R, --runtime-control <option> Perform administrative functions according to option.
So it would accept additional option. Currently only one is supported – config_cache_reload. When using this option, server process looks at the configuration file and tries to find the PID file, as defined there. If found, signal is sent to the process, specified in the PID file.
# zabbix_server -R config_cache_reload command sent successfully
Upon receiving this signal, server reads configuration from the database and rebuilds the cache. It logs receiving of the signal in the logfile as well:
forced reloading of the configuration cache
If configuration is being read already at that time (either previously initiated by real time control or just a scheduled run), on the commandline success is still reported and entry is added to the logfile, but signal itself is ignored.
Configuration cache and proxies
Now, that was mostly about Zabbix server. The wonderful remote data collector, Zabbix proxy also keeps the configuration cache – what about it? Of course, it was not forgotten.
There are two types of Zabbix proxies – passive one is contacted by the server, active one connects to the server.
Active proxy loads all its configuration data from the server upon startup and then repeats this process every hour by default. This period can be customised by configuration parameter ConfigFrequency. Configuration data is then stored in the local database on the proxy.
Thus, if one would make any configuration changes on the server, it might take up to one hour for these changes to propagate to the proxy (and then potentially a bit more time to reach active Zabbix agents).
It is desirable to have some way to signal proxy that it should re-read this configuration data, especially when doing the initial configuration or testing – but re-reading it from the local database, like server does it, would be no good – it’s synchronisation from the server that matters. The great news is that active Zabbix proxy, upon receiving configuration cache reload request, is smart engouh to actually ask for this configuration from the server first.
Sending the configuration cache reload signal to an active proxy would result in log entries like these:
forced reloading of the configuration cache Received configuration data from server. Datalen 6629
Passive Zabbix proxy is different – there the server connects to the proxy and sends the configuration data. When configuration cache reload signal is sent to a passive proxy, success is reported, logged – but ignored. It is not possible yet to signal server to re-send configuration information to passive proxies.
Happy reloading.