Zabbix 2.4 features, part 8 – Debugging web and VMware monitoring

In the previous two blog articles we looked at really great improvements in Zabbix 2.4 to help with debugging/troubleshooting – ability to change loglevel for a running daemon (any sub-process, even) and various smaller validation and error reporting improvements. There is yet another improvement in this area, though – improved debugging capabilities for the built-in web and VMware monitoring for the cases when you really have to dig deep.

Articles in 2.4 feature series:

Debugging web and VMware monitoring previously

Web monitoring has been built in Zabbix for quite a long time and it mostly works. When it doesn’t, though… Things get less pleasant, as there’s no logging of the sent or received data – no way to see what exactly failed. Those with some Zabbix experience would turn to log level 4 with happy expectations – only to be disappointed. Even the 4th log level did not log anything from what the webserver sent us.

Built-in VMware monitoring has been added to Zabbix in version 2.2, thus there has been less time for users to want to debug it. Nevertheless, there have been instances where inability to see what exactly VMware SDK returns has hampered debugging efforts.

Debugging them in Zabbix 2.4

Zabbix 2.4 improves debugging capabilities for both web and VMware monitoring. This is done by adding another log level – 5 (also called trace).

Along with the ability to change log level only for specific sub-process of a running daemon, such a feature allows to enable higher log level only for the specific feature to be debugged, instead of having all of the processes spit out very detailed information.

Still, be careful – both web and VMware debugging can log a huge amount of data. Be ready with increased LogFileSize parameter and enough diskspace.

Note that currently it is not possible to set LogLevel to 5 in the configuration file. Bug report ZBX-8725 deals with that.

Web monitoring debug output

For web monitoring, it seems that only responses are logged – requests are not. “Seems” because for this feature specification was never written and there’s only a partial attempt at documenting what exactly was implemented. Having requests logged would be useful in case of a specific header being required for proper monitoring. Assuming that the functionality would be properly documented eventually, there might be a need for a new feature request to log requests, too. Nevertheless, at log level 4 a web monitoring process (http poller) would log information like this:

 10784:20140907:195201.280 process_httptest() go to URL
 10784:20140907:195201.345 In process_step_data() rspcode:200
  time:0.063834 speed:128129.000000

Increasing log level for all web monitoring processes would be done like this:

$ zabbix_server --runtime-control log_level_increase="http poller"
$ zabbix_server --runtime-control log_level_increase="http poller"

Here we are running the command twice assuming the default/previous log level was 3 – we want to reach the new log level number 5, trace. Additionally, we quote process name as it contains a space.

New log level 5 would provide extra information:

 10784:20140907:191653.185 process_httptest() go to URL
 10784:20140907:191653.240 process_httptest() from'

<!DOCTYPE html>
<html lang="en" dir="ltr">

 10784:20140907:191653.242 In process_step_data() rspcode:200
  time:0.054893 speed:148998.000000

Note that the output is significantly shortened – “…” included whole HTML response from the server.

VMware debugging output

Zabbix built-in VMware support uses VMware SDK/API access. It’s XML based and returns a lot of information. When some VMware monitoring fails, it can be very useful – we would know whether the information was present in the XML structure at all, was it identified as we expected it and so on. In previous versions it was fairly complicated to obtain this output – since 2.4 is is fairly easy. Increasing log level to 5 for process vmware collector (remember, names for various internal processes can be seen at the internal item documentation page) will log all received data.

zabbix_server -R log_level_increase="vmware collector"
zabbix_server -R log_level_increase="vmware collector"

This would set log level to 5 (assuming the default of 3) for all VMware collector processes.

The output would look like this:

 5889:20140923:180604.235 log level has been increased to 5 (trace)
 5889:20140923:180629.786 In vmware_service_update()
 5889:20140923:180629.789 In vmware_service_authenticate()
 5889:20140923:180629.847 vmware_service_authenticate() SOAP
  response: <?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenc=
   The session is not authenticated.</faultstring><detail>
   <NotAuthenticatedFault xmlns="urn:vim25"
 5889:20140923:180629.901 vmware_service_authenticate() SOAP
   response: <?xml version="1.0" encoding="UTF-8"?>

That is only a tiny part of the debug output VMware collector generates – it could easily be many megabytes, so make sure to increase log file size limit significantly before starting a debugging session.

This entry was posted in Technical and tagged , , . Bookmark the permalink.

Leave a Reply