Security is an essential dimension of any tool in your IT infrastructure, and Zabbix is no exception. With Zabbix 6.0 LTS, our users will be able to secure their Zabbix instance on multiple layers – from encrypting your network communication to flexible user access control, API token provisioning, and custom user password policies. Let’s take a look at the full set of security features that Zabbix 6.0 LTS provides to its users.
The full recording of the speech is available on the official Zabbix Youtube channel.
Table of Contents
Why do we need security?
Over time the security standards for IT infrastructures and software have greatly developed. We now have a robust set of standards that we have to follow to ensure the uninterrupted and secure delivery of internal and external services. We also need to ensure that any sensitive information stays confidential and minimize the risk for any possible data breaches. In the case of Zabbix, we have the ability to define a flexible set of security measures to ensure our data stays secure:
- We can encrypt our connections between every Zabbix component to protect our data
- Multiple encryption methods are supported
- Zabbix administrators can configure supported cipher suites based on their company policy
- Role-based access enables Zabbix administrators to define a flexible set of roles to restrict access to confidential information
- This is especially vital for multi-tenant environments, where a single Zabbix instance is shared between multiple customers
- Audit logging adds a layer of visibility that can help us detect potential security or configuration problems before they have a real impact
Security in Zabbix
Zabbix consists of multiple components – some of which are mandatory while others are optional. In the diagram below, you can see how the connection between every component can be encrypted. Users have the ability to choose which parts of their Zabbix infrastructure should be encrypted.
In addition to encrypting the communication between all of the Zabbix components, Zabbix administrators can also create flexible user roles for granular access control. The user passwords are stored in the database, encrypted by using bcrypt. With Zabbix 6.0 LTS, we can also define Zabbix password complexity policies – I’m going to cover that specific feature a bit further in this blog post.
As for your secrets – authentication credentials for your devices, SSH usernames and passwords, and other sensitive information – these can be stored in the HashiCorp Vault, which is an optional component specifically for storing your secrets in a very secure fashion.
Zabbix user types
You can have 3 types of users in Zabbix. We need to understand the access restrictions that these user types enforce, before we can talk more about user roles:
- Zabbix Super Admin
- Unlimited access to everything
- Zabbix Admin
- Can create hosts and templates
- Permission-based access to Zabbix entities
- Zabbix User
- Permission-based access to Zabbix entities
- Has access only to the monitoring information
- Has no access to configuration sections in Zabbix GUI
User roles can be used to create a user role of a particular type and further restrict the access for all users that belong to this role. For example, we can have a User Role that is based on the Super Admin type but, instead of having access to every Zabbix section, we can restrict the members of this role to have access only to parts of Zabbix Administration or Configuration sections.
Obtaining API tokens
If you plan on using the Zabbix API, the first step to executing your API workflows is issuing a user.login API call and obtaining an API token, which will be used in all future API calls until the user is logged out.
Now we have a better way of generating the API tokens. You can now generate the API tokens by accessing Administration – General – API tokens. Here you can generate new API tokens, select the user who can use the particular token, and also define an expiration date for the token.
Once the token has been created, we will see a confirmation screen with the generated token information. You should copy and save the token here since you will not be able to obtain the token later! If you forget to save it, you can always generate a new token.
Below we can see a list of our API tokens, their expiration dates, names, users, and other information:
Using the API tokens is exactly the same as you would use them with the old approach. Simply place your token in the auth field of your API call and, if the token is valid and the user has the necessary permissions, the API call will succeed:
Secret macros
Macros can be used in many different parts of Zabbix – from using them in your triggers to create dynamic trigger thresholds to using them to define user names and passwords for different types of checks, such as SNMP, SSH, Zabbix agent checks, and many others. Now we can add an extra layer of obfuscation for our macros by creating secret macros:
- The value of a secret macro is not displayed in the frontend
- The value of a secret macro does not get cloned or exported with host/template export
- Secret macros are stored in the Zabbix database
- Database connections and access must be secured to prevent the potential attackers from obtaining these values
Zabbix + HashiCorp vault
The HashiCorp Vault is an optional component that we can use for storing Zabbix secrets, thus providing an additional layer of security. The secrets are stored in the vault, and every Zabbix reloads the Zabbix configuration cache, the secrets are picked up from the vault (every minute by default):
- HashiCorp Vault can be used for storing secrets such as authentication information and Zabbix database credentials
- the HashiCorp Vault provides a unified interface to access any secret while ensuring tight access control and keeping a detailed audit log
- Initially, the vault is sealed and must be unsealed using unseal keys
- Secrets are stored in the Zabbix configuration cache
- The values of secrets are retrieved on every Zabbix configuration cache update
Initially, we have to provide the vault token and vault URL in the Zabbix server configuration file. If we choose to store the Zabbix database credentials in the secret vault, we also have to provide the path to the location of the secret:
### Option: VaultToken # Vault authentication token that should have been generated # exclusively for Zabbix server with read only permission VaultToken=verysecretrandomlygeneretedvaultstring ### Option: VaultURL # Vault server HTTP[S] URL. System wide CA certificates directory # will be used if SSLCALocation is not specified. VaultURL=https://my.organization.vault:8200 ### Option: VaultDBPath # Vault path from where credentials for database will be retrieved # by keys 'password' and 'username'. VaultDBPath=my/secret/location
Security improvements in Zabbix 6.0 LTS
The security improvements that we discussed previously have been available in the previous non-LTS versions. If you’re migrating from Zabbix 5.0 LTS to Zabbix 6.0 LTS all of these will be new to you. Next, let’s cover the improvements that have been added specifically in the Zabbix 6.0 LTS. If you’re using the latest major Zabbix version – Zabbix 5.4, then all of the following features await you in Zabbix 6.0 LTS.
Improved Audit log
The audit log has seen some major improvements. Not only is the audit log capable of logging every operation performed by the Zabbix server and Zabbix frontend, but it has also received an internal re-design to ensure minimal possible performance impact and improved UX:
- Improved API operation logging
- Scalability improvements when logging a large number of audit log entries
- Quality of life improvements when it comes to filtering the audit log
- More types of operations are now logged in the audit log
- Script execution
- Global macro changes
- Changes resulting from the execution of low-level discovery rules
- And much more
The above image shows that not only new types of operations are being logged in the audit log, but we can also see the result of those operations, such as – results of script executions, macro changes, new users being created in Zabbix, and more.
User password complexity settings
One of the major security improvements that are introduced in Zabbix 6.0 LTS is the ability to define custom password complexity requirements. Zabbix administrators can select between multiple password complexity requirements and apply them for their Zabbix instance:
- Prevent using simple passwords like password
- Set the minimum password length
- Prevent the usage of name/last name/username in the password
- Prevent usage of easy to guess passwords, such as abcd1234, asdf1234
- The passwords analysis is based on a dictionary of ~1 million most common passwords
- The default Admin password on a fresh Zabbix instance is not changed – you have to update it after the new Zabbix instance is deployed!
Monitoring of certificate attributes
Another dimension of security that has improved with the release of Zabbix 6.0 LTS – security monitoring. Now we can use Zabbix Agent 2 to monitor SSL certificate attributes:
- Zabbix Agent 2 built-in plugin for certificate monitoring available starting from Zabbix Agent 2 5.0.15
- Collects information about the website certificate
- Official templates are available out of the box and can be downloaded on our git page
-
web.certificate.get[hostname,<port>,<address
Below you can see an example of the certificate information obtained from the www.zabbix.com website. Here’s what the item key looks like for our example:
zabbix_agent2 -t web.certificate.get[www.zabbix.com]
And the output that is received by Zabbix:
As we can see, the resulting information tells us that the certificate is valid and has been verified successfully. Next, let’s take a look at a different kind of result:
Here we can see that the certificate is still valid, but we also receive a warning that it’s a self-signed certificate.
You may have noticed that the information obtained by this item comes in a JSON containing multiple values. That’s because most of the .get items obtain information in bulk. We can use this item as a master item and create multiple dependent items, each of which will collect an individual value. This can be configured by using JSONPath preprocessing.
Let’s take a look at some trigger examples that we can use together with the certificate monitoring item:
- Check if the certificate is invalid
-
{HOST:cert.validation.str("invalid")} = 1
-
- Check if the certificate expires in less than 7 days
-
({HOST:cert.not_after.last()} - {HOST:cert.not_after.now ()}) / 86400 < 7
-
- Check if the Certificate fingerprint has changed
-
{TEMPLATE_NAME:cert.sha1_fingerprint.diff()}=1
-
User permissions in service trees
With the reworked business service monitoring, we have also added the ability to define access permissions for a particular service or multiple services that have a matching tag. Permissions can be defined for Read-only access or Read-write access.
In the example below, we can see three child services. Our user has read-only access for the first two services while having read-write permissions for the third service. 
If you wish to read more about the re-designed Services section and Zabbix business service monitoring, you can find a blog post based on the Zabbix Summit 2021 speech on the topic here.
Zabbix security best practices
Finally, let’s go through a checklist of best practices that will help you to secure your environment on all levels.
- Define proper user roles for different types of Zabbix users in your organization
- If you have a multi-tenant Zabbix environment, make sure that each tenant can access only the information related to their organization!
- Enable HTTPS on your Zabbix web frontend
- Unencrypted connections to the web frontend mean that an attacker could potentially obtain sensitive information
- Encrypt your connections to the Zabbix database
- Use the secret vault to harden your security and store your secrets in a vault
- Encrypt the traffic between the Zabbix server, Zabbix proxies, and Zabbix agents
- Here you can choose between PSK and Certificate-based encryptions
- Connections between command-line tools like zabbix-sender can and should also be encrypted!
- Agentless checks should also be configured in a secure manner
- HTTP authentication and SSL for HTTP checks
- Public key authentication for SSH checks
- Define custom password complexity settings to prevent weak passwords for your Zabbix users
- Define allow and deny lists for the item keys in your Zabbix agent configuration
- This way you can restrict the collection of sensitive metrics by potential attackers
- Use secret macros instead of displaying your macro values in plain-text
- Keep Zabbix updated and update to the latest minor Zabbix release
- Minor releases contain critical bug and security fixes
- The upgrade process is extremely simple since no changes are made to the database schema in the minor version updates
Advanced Zabbix Security Administration course
If you wish to learn more about securing your Zabbix instance and get some practical experience in a controlled environment while being guided by Zabbix certified trainers – then the Advanced Zabbix Security Administration course is for you! In this course you will have the chance to learn and practice the following topics:
You can find more information about the Advanced Zabbix Security Administration course as well as the course pricing on the Zabbix website.
Questions
Q: Can we encrypt individual Zabbix components, or is it an all-or-nothing approach?
A: Connections between each component can be encrypted individually. You can, for example, encrypt only connections between Zabbix server and Zabbix proxies if you wish to do so. You can even leave connections between some proxies unencrypted, while others can be fully secured.
Q: Can I mix different Zabbix authentication approaches – for example, can I use LDAP together with internal authentication?
A: It is not only possible, but it’s also recommended. If you’re using LDAP authentication – always have a user with internal authentication configured for it. This will save you in situations where something goes wrong with the LDAP servers or connectivity.
