In the previous two blog articles we looked at really great improvements in Zabbix 2.4 to help with debugging/troubleshooting – ability to change loglevel for a running daemon (any sub-process, even) and various smaller validation and error reporting improvements. There is yet another improvement in this area, though – improved debugging capabilities for the built-in web and VMware monitoring for the cases when you really have to dig deep.
Articles in 2.4 feature series:
- Part 1 – Multiple LLD filters
- Part 2 – Controlling redirects and header retrieval for web monitoring
- Part 3 – SSL verification and authentication controls
- Part 4 – Web monitoring URL limit increased to 2048
- Part 5 – Custom action condition formula
- Part 6 – Runtime loglevel changing
- Part 7 – Improved troubleshooting
- Part 8 – Debugging web and VMware monitoring
Debugging web and VMware monitoring previously
Web monitoring has been built in Zabbix for quite a long time and it mostly works. When it doesn’t, though… Things get less pleasant, as there’s no logging of the sent or received data – no way to see what exactly failed. Those with some Zabbix experience would turn to log level 4 with happy expectations – only to be disappointed. Even the 4th log level did not log anything from what the webserver sent us.
Built-in VMware monitoring has been added to Zabbix in version 2.2, thus there has been less time for users to want to debug it. Nevertheless, there have been instances where inability to see what exactly VMware SDK returns has hampered debugging efforts.
Debugging them in Zabbix 2.4
Zabbix 2.4 improves debugging capabilities for both web and VMware monitoring. This is done by adding another log level – 5 (also called trace).
Along with the ability to change log level only for specific sub-process of a running daemon, such a feature allows to enable higher log level only for the specific feature to be debugged, instead of having all of the processes spit out very detailed information.
Still, be careful – both web and VMware debugging can log a huge amount of data. Be ready with increased LogFileSize parameter and enough diskspace.
Note that currently it is not possible to set LogLevel to 5 in the configuration file. Bug report ZBX-8725 deals with that.
Web monitoring debug output
For web monitoring, it seems that only responses are logged – requests are not. “Seems” because for this feature specification was never written and there’s only a partial attempt at documenting what exactly was implemented. Having requests logged would be useful in case of a specific header being required for proper monitoring. Assuming that the functionality would be properly documented eventually, there might be a need for a new feature request to log requests, too. Nevertheless, at log level 4 a web monitoring process (http poller) would log information like this:
10784:20140907:195201.280 process_httptest() go to URL "http://www.openstreetmap.org" 10784:20140907:195201.345 In process_step_data() rspcode:200 time:0.063834 speed:128129.000000
Increasing log level for all web monitoring processes would be done like this:
$ zabbix_server --runtime-control log_level_increase="http poller" $ zabbix_server --runtime-control log_level_increase="http poller"
Here we are running the command twice assuming the default/previous log level was 3 – we want to reach the new log level number 5, trace. Additionally, we quote process name as it contains a space.
New log level 5 would provide extra information:
10784:20140907:191653.185 process_httptest() go to URL "http://www.openstreetmap.org" 10784:20140907:191653.240 process_httptest() page.data from http://www.openstreetmap.org:' <!DOCTYPE html> <html lang="en" dir="ltr"> <head> ... </body> </html> ' 10784:20140907:191653.242 In process_step_data() rspcode:200 time:0.054893 speed:148998.000000
Note that the output is significantly shortened – “…” included whole HTML response from the server.
VMware debugging output
Zabbix built-in VMware support uses VMware SDK/API access. It’s XML based and returns a lot of information. When some VMware monitoring fails, it can be very useful – we would know whether the information was present in the XML structure at all, was it identified as we expected it and so on. In previous versions it was fairly complicated to obtain this output – since 2.4 is is fairly easy. Increasing log level to 5 for process vmware collector (remember, names for various internal processes can be seen at the internal item documentation page) will log all received data.
zabbix_server -R log_level_increase="vmware collector" zabbix_server -R log_level_increase="vmware collector"
This would set log level to 5 (assuming the default of 3) for all VMware collector processes.
The output would look like this:
5889:20140923:180604.235 log level has been increased to 5 (trace) 5889:20140923:180629.786 In vmware_service_update() user@https://127.0.0.1/sdk 5889:20140923:180629.789 In vmware_service_authenticate() 'user'@'https://127.0.0.1/sdk' 5889:20140923:180629.847 vmware_service_authenticate() SOAP response: <?xml version="1.0" encoding="UTF-8"?> <soapenv:Envelope xmlns:soapenc= "http://schemas.xmlsoap.org/soap/encoding/" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <soapenv:Body> <soapenv:Fault><faultcode>ServerFaultCode</faultcode><faultstring> The session is not authenticated.</faultstring><detail> <NotAuthenticatedFault xmlns="urn:vim25" xsi:type="NotAuthenticated"><object type="SessionManager">SessionManager</object> <privilegeId>System.View</privilegeId></NotAuthenticatedFault> </detail></soapenv:Fault> </soapenv:Body> </soapenv:Envelope> 5889:20140923:180629.901 vmware_service_authenticate() SOAP response: <?xml version="1.0" encoding="UTF-8"?> <soapenv:Envelope xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" ...
That is only a tiny part of the debug output VMware collector generates – it could easily be many megabytes, so make sure to increase log file size limit significantly before starting a debugging session.