Performance improvements in Zabbix 1.8, part 2

So in part 1 we found out that Zabbix 1.8.1 provided significant performance improvements over 1.6, mostly because of reduced database access. But those who have been following “What’s new” section in Zabbix manual might be aware that both 1.8.2 and 1.8.3 promised even better performance. After 1.8, that sounds quite optimistic, doesn’t it? Let’s try to get some data on that then. To have comparable data we hunted down Zabbix user verwilst again and coerced him to provide some fresh graphs – thanks again.

Database load reduction

While no data was available on the upgrade to 1.8.2, he was able to dig out graphs that show the change to 1.8.3 (from what most likely is 1.8.2). Let’s start with a look at a large scale MySQL query graph, showing one month worth of data.

MySQL queries – upgrade to 1.8.3. Click to see larger size

While the change for going to 1.8.3 is not as significant as move from 1.6 was, it is still an improvement.

  • The deletes don’t seem to change, just as with upgrade from 1.6.
  • Amount of inserts is slightly reduced. The change is not large, but clearly visible on larger scale.
  • Updates seem to drop slightly… but the change is barely noticeable.
  • Something that is not at all easy to see, but inserts have dropped by a significant percentage. The absolute change barely registers on this y axis scale, but there are twice less inserts per second after the upgrade to 1.8.3.

So it’s almost great – database usage has decreased, even if just a little. But… wait. An observant reader might have noticed that values before upgrade to 1.8.3 don’t really match values after upgrade to 1.8.1 – so we are not seeing the difference that 1.8.2 might have introduced.

Let’s try to compare query situation right after upgrade to 1.8.1 and after upgrade to 1.8.3 – that should be the total impact of 1.8.2 and 1.8.3. It will be easier to do so while looking at data for the same period, so we’ll look at approximately 8 hour graphs. Unfortunately, y axis scale is not the same for them, so there will be some manual scaling. Graph wouldn’t look as nice anymore, but at least we should be able to compare it better.

MySQL queries – 1.8.1 vs 1.8.3. Click to see larger size

So on the left hand side is data right after upgrading to 1.8.1. Right hand side shows data from 1.8.3. One thing has not changed much – update rate is roughly the same. Although probably most of you noticed something else before – that normal operation select rate has dropped a lot, again. It went from some 1200-1300 selects per second to 400. That’s reduction more than 3 times. This efficiency seems to have improved the speed housekeeper is capable of running at – while there are more deletions per second during housekeeper runs, they are notably shorter (even though left hand graph shows bigger period on x axis than the right hand one). What’s also very interesting, insert rate during trends has plummeted. If we look carefully, we can see slight bumps at approximately 10 minutes past full hour, but at this y axis scale that’s just a single pixel change. So what looked like a steady insert reduction at one month graph can be seen as significant insert reduction during trend calculation period on 8 hour graph.

One more thing we didn’t look into much yet – select increase during the housekeeper runs. It is still significant, and given that housekeeper finishes faster now it’s also a much bigger increase over normal select rate. While it has dropped from approximately 2000 queries per second to 1400, there seems to be some room for future performance improvements. And the update rate is not impacted by housekeeper runs as much as before.

Of course, the difference is even bigger if we compare 1.6 directly with 1.8.3… so let’s do just that.

MySQL queries – 1.6 vs 1.8.3. Click to see larger size

This shows approximately the same drops for updates and inserts that we observed from 1.6 to 1.8.1 in part 1, but the reduction of selects is simply astounding.

CPU load reduction on Zabbix server

So we saw some significant further reductions in the amount of queries that are being executed on the database server. There’s still one interesting graph… and that is CPU load on the Zabbix server itself.

CPU load on Zabbix server after upgrade to 1.8.3. Click to see larger size

Yep. It just drops down there somewhere and does not climb up again. OK, CPU load on Zabbix server wasn’t huge to begin with… but now it can be barely seen at the 0 mark on the y axis.

Again, for posterity reasons, let’s look at CPU load reduction starting with 1.6

CPU load on Zabbix server from 1.6 till 1.8.3. Click to see larger size

This image seems to be speaking on its own. Hard work on performance improvements has resulted in significant load drops for pretty much every release in 1.8 series so far. Although on this particular system, unlike with database queries, further improvements wouldn’t be visible anymore…

So that’s it on practical effect of the performance improvements in Zabbix 1.8 series for now – but further articles regarding Zabbix performance may appear, so stay tuned (maybe by subscribing to the RSS feed).

Leave a Reply