One of the built-in security features in Zabbix is TLS (Transport Layer Security) support for external connections. This means that when your distributed Zabbix proxies or Zabbix agents connect to the Zabbix server (or vice versa), TLS can be used to encrypt all the connections. When the connections are encrypted, third parties cannot read the Zabbix components’ communication, even though they would be able to catch the network traffic in some way.
In specific cases you may still want to inspect the encrypted traffic, for example to troubleshoot some problems with Zabbix agents or proxies. I already wrote a post about troubleshooting Zabbix agent with Wireshark, but the TLS encryption prevents anyone seeing the actual contents of the packets.
Since the traffic is encrypted in the Zabbix components (the server, agents and proxies), there still is a way for you, the Zabbix administrator, to intervene with the encryption so that you can get hold of the unencrypted traffic as well. In this post I will explain the process.
First, let’s demonstrate the TLS encryption between a Zabbix agent and a Zabbix server. I have configured the agent (Zabbix Agent 2 actually) with these lines:
Hostname=Zabbix70-agent ServerActive=zabbixtest.lein.io TLSConnect=psk TLSPSKIdentity=agent-ident TLSPSKFile=/etc/zabbix/psk
In this example I’m using TLS with pre-shared key (PSK), and the key itself is saved in /etc/zabbix/psk. My favorite way of generating a PSK is using OpenSSL:
markku@agent:~$ openssl rand -hex 32 afa34bf1104a1457e11e7d3a9b1ff7f5fb4f494c92ca1a8a9c5e1437f8897416 markku@agent:~$
The same key must also be configured on the Zabbix server frontend, see the Zabbix documentation for the PSK configuration details:
After the configurations I captured the Zabbix traffic for some time on the Zabbix server (using sudo tcpdump port 10051 -v -w zabbix70-tls-agent.pcap), stopped the capture, copied the capture file to my workstation, and opened it with Wireshark.
The capture file can be downloaded here:
- zabbix70-tls-agent.pcap (github.com)
Note: I recommend using Wireshark version 4.1.0 or later when analyzing captures containing Zabbix traffic because the built-in Zabbix protocol support was only added to Wireshark in version 4.1.0.
The packet list looks like this:
As we can see in the Protocol column, there are no Zabbix packets recognized in this capture, there are only TCP and TLS packets (the TCP-marked packets being the “empty” packets for negotiating the actual connectivity).
A side detail: Even though the traffic is encrypted, you can still see the configured Zabbix TLS PSK identity (“agent-ident” in my configuration above) in plain text inside the TLS Client Hello packets, if you ever need to check that in the traffic.
Now that we confirmed that TLS encryption is used and we cannot see the Zabbix traffic contents in the capture, let’s prepare the Zabbix server for the TLS decryption.
As I hinted in the beginning, since we have the TLS connection endpoints under our management, we can do tricks on the hosts to get the encryption keys. TLS negotiates the encryption keys dynamically for each connection, but there is a way to save the keys to a file so that we can later decrypt the captured traffic. (Note: I’m not a protocol-level TLS expert, so please forgive me any possible technical inaccuracies in the detailed explanations. I’ll just call “TLS keys” whatever is needed to get the encryption/decryption done.)
Peter Wu (who, in contrast to me, is a protocol-level TLS expert, and also one of the Wireshark core developers) has kindly published code for a helper library that makes it possible for us to save the TLS session keys on the TLS endpoint. In this demo I will save the keys on the Zabbix server, but the same could be done on the agents/proxies instead if needed.
First I’ll see the TLS library that my Debian-based Zabbix server is using:
markku@zabbixserver:~$ ldd /usr/sbin/zabbix_server | grep ssl
libssl.so.3 => /lib/x86_64-linux-gnu/libssl.so.3 (0x00007f62ee47a000)
markku@zabbixserver:~$ dpkg -l libssl* | grep ^ii
ii libssl3:amd64 3.0.9-1 amd64 Secure Sockets Layer toolkit - shared libraries
markku@zabbixserver:~$
To get and compile the helper library I’ll need to install some utilities:
markku@zabbixserver:~$ sudo apt install git gcc make libssl-dev ... markku@zabbixserver:~$ dpkg -l libssl* | grep ^ii ii libssl-dev:amd64 3.0.9-1 amd64 Secure Sockets Layer toolkit - development files ii libssl3:amd64 3.0.9-1 amd64 Secure Sockets Layer toolkit - shared libraries markku@zabbixserver:~$
I’ll the clone the Peter’s wireshark-notes repo to the server:
markku@zabbixserver:~$ git clone --depth=1 https://git.lekensteyn.nl/peter/wireshark-notes Cloning into 'wireshark-notes'... ... markku@zabbixserver:~$ cd wireshark-notes/src markku@zabbixserver:~/wireshark-notes/src$ ls -l total 28 -rw-r--r-- 1 markku markku 534 Oct 7 15:39 Makefile -rw-r--r-- 1 markku markku 11392 Oct 7 15:39 sslkeylog.c -rw-r--r-- 1 markku markku 7278 Oct 7 15:39 sslkeylog.py -rwxr-xr-x 1 markku markku 2325 Oct 7 15:39 sslkeylog.sh markku@zabbixserver:~/wireshark-notes/src$
Now I can compile the library and make it available on the server:
markku@zabbixserver:~/wireshark-notes/src$ make cc sslkeylog.c -shared -o libsslkeylog.so -fPIC -ldl markku@zabbixserver:~/wireshark-notes/src$ sudo install libsslkeylog.so /usr/local/lib markku@zabbixserver:~/wireshark-notes/src$ ls -l /usr/local/lib/libsslkeylog.so -rwxr-xr-x 1 root root 17336 Oct 7 15:40 /usr/local/lib/libsslkeylog.so markku@zabbixserver:~/wireshark-notes/src$ cd markku@zabbixserver:~$
To use the helper library, a couple of environment variables need to be set. For Zabbix server the easy way is to edit the systemd configuration for zabbix-server service:
markku@zabbixserver:~$ sudo systemctl edit zabbix-server
In the editor that opens I’ll add these in the configuration:
[Service] Environment=LD_PRELOAD=/usr/local/lib/libsslkeylog.so Environment=SSLKEYLOGFILE=/tmp/tls.keys
The variables are kind of self-explanatory: Whenever Zabbix server service is started, the libsslkeylog.so library is loaded first, and the SSLKEYLOGFILE variable sets the location of the file where the keys will be saved.
Now the word of warning: The libsslkeylog.so library, when loaded by a process that uses TLS communication, will save the encryption/decryption keys of all the TLS sessions of the process to the configured file. This means that whoever gets that file and the saved TLS communication will be able to see the decrypted contents of the packets, defeating the whole idea of the TLS encryption. You really don’t want to do this TLS key saving for any longer periods of time. Be sure to remove the configurations (and restart the service) after you have inspected whatever you were inspecting in your system. Or, don’t do any of this at all.
After saving the configuration the Zabbix server needs to be restarted:
markku@zabbixserver:~$ sudo systemctl restart zabbix-server markku@zabbixserver:~$
The TLS keys have now started being saved in the configured file:
markku@zabbixserver:~$ ls -l /tmp/tls.keys -rw-rw-r-- 1 zabbix zabbix 10157 Oct 7 15:45 tls.keys markku@zabbixserver:~$
At this point the Zabbix agent is still communicating actively with the Zabbix server, so I’ll take a new capture with tcpdump (sudo tcpdump port 10051 -v -w zabbix70-tls-agent-2.pcap).
After a short while I’ll stop the capture, and copy the capture file and the TLS key file on my workstation.
Now it’s a good time to disable the TLS key saving as well (besides containing sensitive data, the key file will also grow with each new TLS session so it can quickly get very large), so I’ll edit the Zabbix service configuration, remove the configured lines and restart the service:
markku@zabbixserver:~$ sudo systemctl edit zabbix-server markku@zabbixserver:~$ sudo systemctl restart zabbix-server markku@zabbixserver:~$
When opening the new capture file in Wireshark there is no immediate change in the packet list: the TLS packets are still shown encrypted. Wireshark needs to be specifically configured to read the TLS keys from the separate file.
In Wireshark, I’ll go to Edit – Preferences – Protocols – TLS:
There is the “(Pre)-Master-Secret log filename” field, I’ll use Browse button to select the copied tls.keys file, and save the configuration with OK.
At this point Wireshark reloads the capture file and the Zabbix agent TLS sessions will be decrypted:
Using the “zabbix” display filter will show just the Zabbix protocol packets:
When selecting a Zabbix protocol packet and looking at the packet details, in the lower right pane there are now three tabs: Frame (the encrypted TLS data), Decrypted TLS, and Uncompressed data.
This is because in this example the Zabbix agent 2 also compresses the traffic, and the compressed traffic is then encrypted when sending out to the network. Wireshark can interpret all this because of its built-in knowledge about TLS encryption and the Zabbix protocol structure, as well as the user-supplied TLS decryption keys.
We are now able to analyze the Zabbix agent communication with Wireshark even though the traffic was TLS-encrypted when we captured it.
One more trick about the TLS keys in Wireshark: It is also possible to save the keys inside the capture file when analyzing the traffic, instead of having the keys in a separate file (tls.keys in this example). I’ll go in Edit menu and select Inject TLS Secrets, and then save the capture file in pcapng format. Now the previously loaded keys are embedded in the capture file, and I can clear the “(Pre)-Master-Secret log filename” field in the TLS settings (as the filename setting is not useful in any later Wireshark analysis). The same can also be done in the command line by using editcap --inject-secrets (editcap is part of Wireshark install, see the manual page of editcap for more details).
Here is the second capture file of this demo, with the embedded TLS keys:
- zabbix70-tls-agent-2-with-keys.pcapng (github.com)
Finally some closing comments:
- As demonstrated, when you have administrator/root-level access to the TLS session endpoint (Zabbix server in this example), there can be a possibility to save and decrypt the TLS sessions using external tooling. After all, TLS encryption is based on the negotiation between the TLS-connected endpoints, so if you are the TLS connection endpoint, you have ways to access the plaintext data. If you don’t have sufficient access to the TLS session endpoint, there is no way you can get the decryption keys mid-path.
- Act responsibly when saving the TLS session keys for any traffic, on Zabbix server or otherwise. The encryption is there for a purpose, and saving the TLS keys always carries the risk that someone else gets access to data they wouldn’t have access to otherwise.
- Do not save the TLS session keys with the capture file, unless you are dealing with a test/demo environment, like I had here.
- When troubleshooting Zabbix connections, TLS decryption with Wireshark is not the only way. You should also consider if just increasing the logging level in the Zabbix components brings you enough information to solve your case, or maybe in some specific case you can just disable TLS encryption for an agent for a moment to not have to deal with the decryption at all. But again, usually the encryption is there for a purpose, so you need to evaluate your own situation.
Thanks for the article! Insightful.