Magic of New Zabbix Agent

Zabbix 4.4 introduces a new type of agent, zabbix_agent2, which offers a wide range of new capabilities and advanced monitoring functions.

Watch the video now.

Contents

1. About Zabbix agent (4:55:11)
2. Current challenges (5:00:52)
3. Meet the brand new agent (5:02:13)
4. It’s all about the plugins (5:05:36)
5. Just Go with it (5:08:07)

About Zabbix agent

Today I will talk about the new Zabbix agent. But before discussing the new, posh features let’s brush up on the basics.

The Zabbix agent is a daemon that gathers metrics locally from the operating system (OS). I would say that Zabbix is the best monitoring tool in the industry. It is very powerful, available on UNIX-like and Windows OS, and it runs on devices with limited resources thanks to its small footprint, which means low resource consumption.

The Zabbix agent allows for easy and limitless resource gathering. In addition, this tool can customize time intervals with flexible/scheduled intervals options to adjust data gathering to one’s needs. For example, data polling can be scheduled every five minutes during the week and once in an hour during weekends.

The Zabbix agent gathers the following out-of-the-box metrics:

• disk and NIC statistics;
• memory consumption and CPU usage (per system, per process);
• hostname, OS release, and other generic metrics;
• listening port state.

It can also parse logs and simple files, read Windows event logs, and gather statistics from a friendly Zabbix server/proxy, etc. Many more Zabbix agent functions are described in the manual.

The Zabbix agent can be extended mainly by using three options: custom modules, user parameters, and system.run[*] item key.

Custom modules are for coding jedies. They are as fast as Zabbix native agent metrics but you will have to learn C.

1. main {
       printf("How to C learn, young padawan!")
   }

User parameters are for scripting ninjas. User parameters come in when Zabbix does not have a specific predefined agent check. User parameters can be used to run any command respecting Zabbix agent privileges and can also be forked.

UserParameter=systemd.units, systemctl list-unit-files

The system.run[*] item key is a simple yet powerful tool. There is no need to worry about user parameters or the config file. Restarting the Zabbix agent to change the command is not required — the agent will pick it up. Basically, with system.run you can do whatever you like. You should still be careful and avoid doing anything like this:

zabbix_agentd.conf:
Server=0.0.0.0/0
EnableRemoteCommands=1
LogRemoteCommands=0
User=root
AllowRoot=1

Operating system:
shell> iptables -F
shell> systemctl stop firewalls
shell> setenforce 0

Current challenges

The old Zabbix agent has some limitations:

• The native Zabbix agent can’t receive third-party traps.
• No scheduled or custom intervals are available for active checks.
• Only one active check process is available for each Zabbix server/proxy record, which starts on the boot and needs to restart every time the config file is changed.
• Not everybody wants to learn or knows C.

Meet the brand-new agent

Zabbix Agent 2 is the new generation of the Zabbix agent written in Go, one of the most popular languages currently. Go allows for compiling the developed program in a single binary and deploying it with some dependency limitations.

The new Zabbix agent is limitless and has:

• scheduled/flexible intervals for both active and passive checks;
• older configuration file support;
• multiple parallel log file readings;
• out-of-the-box systemd monitoring;
• timeouts implemented on the plugin level.

The Zabbix agent has three main components: the connector, the listener, and the scheduler.

The connector covers the communication with the server, item configuration, and metrics buffer. There is 1 connector per 1 ServerActive. The connector is for communication only and does not perform checks.

The listener accepts passive requests and forwards them to the scheduler. Right now, the functionality is limited to that, but it will be extended in the future.

The scheduler covers task queue management with respect to schedule and task concurrency. There is only one scheduler per agent.

With Zabbix Agent 2 you can still use both passive and active checks. Check out the schema below to learn how the checks are performed.

Passive checks

Active checks

The new agent has a two-level task queue:

• Each plugin has a queue of the requested checks.
• The scheduler has a queue of active plugins.

That makes concurrency work better. When a task cannot be executed because of concurrency limits the plugin is taken out from the scheduler queue and returned only when the next task can be executed.

It’s all about the plugins

The brand new Zabbix agent is a whole new platform that is easily extendable with plugins. Writing your own custom Go-based plugins has never been that easy, and the agent features multiple plugins interfaces for different types of tasks. Zabbix Agent 2 offers a framework for every purpose: exporter, watcher, collector, runner, and configurator.

The exporter (plugin.Exporter) is the simplest interface used most of the time to perform polls and return:

• nothing;
• an error;
• single/multiple values.

The watcher allows implementing your own gathering method, for example, by listening to a specific port, and supporting different trap-based communication methods.

The collector is used when you need to gather data on a regular basis and pass it to the exporter, for example, to export CPU related metrics.

The runner interface can be used by a plugin to start/stop some background goroutine.

Just Go with it

Writing your own plugin is very simple. Let’s create a plugin that implements one additional metric called myip that returns the external IP of the host where the agent is running. It will do so by performing an HTTP request to https://api.ipify.org, which returns the IP of client connected. We will need 2 additional Go packages: “ioutil” and “http”. Don’t worry, you don’t need to install them, they come along with Go.

1. First, we need to get Zabbix sources. We need at least 4.4.2 but since it’s not released yet we’ll use the one from repository:

git clone https://git.zabbix.com/scm/zbx/zabbix.git --branch release/4.4 --depth 1 --single-branch zabbix-4.4

2. Let’s make sure we are able to compile Zabbix go agent, you need to have package golang installed:

./bootstrap.sh
./configure --enable-agent2 --prefix=$(pwd)
make -s install

If no compilation errors you are good to go, otherwise you’ll need to fix them before going further.

3. Now, create directory for your module and edit plugin source file:

mkdir src/go/plugins/myip
vi src/go/plugins/myip/myip.go

Add the following contents to the file:

// This is the name of your Go package. All files in a package must use the same name.                                                                                                                                                         
// It implements 1 metric, called myip, which returns the expternal IP address of the                                                                                                                                                          
// host where Agent is running.                                                                                                                                                                                                                
package myip

// Packages we will use. The last one is a must.                                                                                                                                                                                               
// Note, the name "zabbix.com" changed from "zabbix" in 4.4.2 .                                                                                                                                                                                
import (
        "io/ioutil"
        "net/http"
        "zabbix.com/pkg/plugin"
)

// Plugin must define structure and embed plugin.Base structure.                                                                                                                                                                               
type Plugin struct {
        plugin.Base
}

var impl Plugin

// Plugin must implement one or several plugin interfaces.                                                                                                                                                                                     
func (p *Plugin) Export(key string, params []string, ctx plugin.ContextProvider) (result interface{}, err error) {
        // Fetch response from the specified URL, it should be just the IP address.                                                                                                                                                            
        resp, err := http.Get("https://api.ipify.org")

        if err != nil {
                panic(err)
        }

        defer resp.Body.Close()

        body, err := ioutil.ReadAll(resp.Body)

        if err != nil {
                panic(err)
        }

        return string(body), nil
}

func init() {
        // Register our metric, specifying the plugin and metric details.
        // 1 - a pointer to plugin implementation                                                                                                                                                                                              
        // 2 - plugin name                                                                                                                                                                                                                     
        // 3 - metric name (item key)                                                                                                                                                                                                          
        // 4 - metric description                                                                                                                                                                                                              
        //                                                                                                                                                                                                                                     
        // NB! The metric description must end with period, otherwise the agent won't start!                                                                                                                                                   
        plugin.RegisterMetrics(&impl, "Myip", "myip", "Return the external IP of the host where agent is running.")
}

4. Now we need to tell the agent that there’s our plugin to import. Edit file src/go/plugins/plugins.go and append your plugin to the existing list of imports:

import (
        _ "zabbix.com/plugins/log"
        _ "zabbix.com/plugins/systemrun"
        _ "zabbix.com/plugins/zabbix/async"
        _ "zabbix.com/plugins/zabbix/stats"
        _ "zabbix.com/plugins/zabbix/sync"
        _ "zabbix.com/plugins/myip"
)

5. Let’s recompile the agent again:

make -s install

6. Now, it’s time to start the agent with our plugin:

sbin/zabbix_agent2

You should see the following output:

Starting Zabbix Agent 2 [Zabbix server]. (4.4.2rc1)
Press Ctrl+C to exit.

7. From another terminal, try to request new metric:

bin/zabbix_get -s 127.0.0.1 -k myip
aaa.bbb.ccc.ddd

That’s it, your new metric is working, so you can continue adding more!

Note, zabbix_agentd2 works with exactly the same configuration file as zabbix_agentd . See the complete list of command-line options it supports by running it with -h:

sbin/zabbix_agent2 -h
Usage of sbin/zabbix_agent2:
  -R string
        Perform administrative functions (send 'help' for available commands)
  -V    Print program version and exit (shorthand)
  -c string
        Path to the configuration file (shorthand) (default "/tmp/zabbix-4.4/etc/zabbix_agent2.conf")
  -config string
        Path to the configuration file (default "/tmp/zabbix-4.4/etc/zabbix_agent2.conf")
  -f    Run Zabbix agent in foreground (shorthand) (default true)
  -foreground
        Run Zabbix agent in foreground (default true)
  -p    Print known items and exit (shorthand)
  -print
        Print known items and exit
  -t string
        Test specified item and exit (shorthand)
  -test string
        Test specified item and exit
  -version
        Print program version and exit

This new functionality is available in Zabbix Agent 2. In Zabbix 4.4 it is experimental, but in Zabbix 5.0 it is production-ready. For now, it is available only for UNIX-like systems, but Zabbix Agent 2 for Windows is already in development.

The documentation for developers and system administrators will be available for download from our Git repository.

See also: Presentation slides, Agent 2 documentation

About Alexey Petrov

Zabbix Certified Expert & Trainer

View all posts by Alexey Petrov