Today we are talking about a use case when it’s impossible to find a proper way to write a recovery expression for the Zabbix trigger. In other words, we know how to identify problems. But there is no good way to detect when the problem is gone.

This mostly relates to a huge environment, for example:

  • Got one log file. There are hundreds of patterns inside. We respect all of them. We need them
  • SNMP trap item (snmptrap.fallback) with different patterns being written inside

In these situations, the trigger is most likely configured to “Event generation mode: Multiple.” This practically means: when a “problematic metric” hits the instance, it will open +1 additional problem.

Goal:
I just need to receive an email about the record, then close the event.

As a workaround (let’s call it a solution here), we can define an action which will:

  1. contact an API endpoint
  2. manually acknowledge the event and close it

The biggest reason why this functionality is possible is that: when an event hits the action, the operation actually knows the event ID of the problem. The macro {EVENT.ID} saves the day.

To solve the problem, we need to install API characteristics at the global level:

     {$Z_API_PHP}=http://127.0.0.1/api_jsonrpc.php
    {$Z_API_USER}=api
{$Z_API_PASSWORD}=zabbix

NOTE
‘http://127.0.0.1/api_jsonrpc.php’ means the frontend server runs on the same server as systemd:zabbix-server. If it is not the case, we need to plot a front-end address of Zabbix GUI + add ‘api_jsonrpc.php’.

We will have 2 actions. The first one will deliver a notification to email:

After 1 minute, a second action will close the event:

This is a full bash snippet we must put inside. No need to change anything. It works with copy and paste:

URL={$Z_API_PHP}
USER={$Z_API_USER}
PASSWORD={$Z_API_PASSWORD}

# authorization
AUTH=$(curl -sk -X POST -H "Content-Type: application/json" -d '
{
	"jsonrpc": "2.0",
	"method": "user.login",
	"params": {
		"user": "'"$USER"'",
		"password": "'"$PASSWORD"'"
	},
	"id": 1,
	"auth": null
}
' $URL | \
grep -E -o "([0-9a-f]{32,32})")

# acknowledge and close event
curl -sk -X POST -H "Content-Type: application/json" -d '
{
	"jsonrpc": "2.0",
	"method": "event.acknowledge",
	"params": {
		"eventids": "{EVENT.ID}",
		"action": 1,
		"message": "Problem resolved."
	},
	"auth": "'"$AUTH"'",
	"id": 1
}' $URL

# close api key
curl -sk -X POST -H "Content-Type: application/json" -d '
{
    "jsonrpc": "2.0",
    "method": "user.logout",
    "params": [],
    "id": 1,
    "auth": "'"$AUTH"'"
}
' $URL
Subscribe
Notify of
4 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Tanisha Roy
Tanisha Roy
3 years ago

For more information about best Immigration consultant, https://www.visaexperts.com/

gdrd84
gdrd84
3 years ago

This is just what I was looking for! It worked perfect for me (I just had to change the url to: http: //127.0.0.1/zabbix/api_jsonrpc.php in my case)
Thank you very much!

Constantin Oshmyan
Constantin Oshmyan
2 years ago

Hi!

This blog post is very useful; at least – it provides some workaround while ZBXNEXT-2452 is not implemented.

However, since this blog has been published, some new versions of Zabbix (5.4 and 6.0) was published also; so below are my comments regarding to the current LTS version (v6.0).

First of all, v5.4 introduced the API tokens (current link). So, this script could be simplified to avoid login and logout on every operation.

Second, API documentation contains 2 important notes:

  • Read/Write rights for trigger are required to close the event or to change event’s severity.
  • To close an event, manual close should be allowed in the trigger.

So, user (used for authentication in this script) must have read/write permissions onto an appropriate hosts; and trigger itself must permit a manual closing.

Finally, since v5.4 scripts were moved onto global settings: initially you should define a script here (Administration -> Scripts), and only then you are able to choose a defined script during the Action configuration. Good news is that configuring scripts, all macro’s – both system macro (like {EVENT.ID}) and user macro (like {$Z_API_PHP}) could be inserted in the text of script (they will be substituted during script execution), so the script could be simplified once more.

So, finally, you can use the following steps:

  • define a needed Zabbix user with enough permissions to close problems;
  • generate a token for this user;
  • define 2 user macro at the global level: {$Z_API_PHP} (described in this blog article) and {$Z_API_TOKEN} (containing the authentication token, it could be a secret macro);
  • define a script provided below;
  • define an action running this script for closing a needed problems (for example, using some specific tag to mark such problems).

Text of the final script:

# acknowledge and close event
curl -sk -X POST -H "Content-Type: application/json" -d '
{
    "jsonrpc": "2.0",
    "method": "event.acknowledge",
    "params": {
        "eventids": "{EVENT.ID}",
        "action": 1,
        "message": "Problem auto-closed."
    },
    "auth": "{$Z_API_TOKEN}",
    "id": 1
}' {$Z_API_PHP}
Constantin Oshmyan
Constantin Oshmyan
2 years ago

Once more idea: as this script consists of a single command performing an HTTP/HTTPS request, it could be re-written as a webhook.

Pros:

  • avoid an additional dependencies (curl command in this case);
  • avoid an external processes (fork()+exec() for a shell and then a curl).
  • possibility to set some specific timeout value.

Cons:

  • a bit longer script text.

Details:

  • in a script properties (Administration → Scripts) set the “Type”=”Webhook” (instead of “Script”);
  • add a 3 parameters:
  1. eventid = {EVENT.ID}
  2. token = {$Z_API_TOKEN}
  3. url = {$Z_API_PHP}
  • in the “Script” field copy-paste the following JavaScript code:
try {
    var req = new HttpRequest(),
        params = JSON.parse(value),
        eventid = params.eventid,
        token = params.token,
        url = params.url,
        body,
        resp;

    Zabbix.log(4, '[Closing problem] starts: eventid=' + eventid + ', token="' + token +'", url="' + url + '".');

    //Validation
    if (typeof eventid !== 'string' || eventid.trim() === '') {
        throw 'The eventid must be defined in macro "{EVENT.ID}".';
    }

    if (typeof token !== 'string' || token.trim() === '') {
        throw 'The token must be defined in macro "{$Z_API_TOKEN}".';
    }

    if (typeof url !== 'string' || !(url.startsWith('http://') || url.startsWith('https://'))) {
        throw 'The URL (starting with "http://" or "https://") must be defined in macro "{$Z_API_PHP}".';
    }

    if (typeof params.HTTPProxy === 'string' && params.HTTPProxy.trim() !== '') {
        req.SetProxy(params.HTTPProxy);
    }

    //Processing logic
    req.addHeader('Content-Type: application/json');
    body = {
        'jsonrpc': '2.0',
        'method': 'event.acknowledge',
        'params': {
            'eventids': eventid.trim(),
            'action': 1,
            'message': 'Problem auto-closed.'
        },
        'auth': token.trim(),
        'id': 1
    };
    resp = req.post(url.trim(),
        JSON.stringify(body)
    );

    if (req.getStatus() != 200) {
        throw 'Response code: ' + req.getStatus();
    }

    Zabbix.log(4, '[Closing problem] returned: ' + resp);
    resp = JSON.parse(resp);
    if ('error' in resp) {
        throw ('API error: ' + resp.error.message + '; ' + resp.error.data);
    }

} catch (error) {
    Zabbix.log(3, '[Closing problem] cannot close event "' + eventid + '": ' + error);
    throw 'Cannot close event ' + eventid + ': ' + error;
}

return 'OK';
4
0
Would love your thoughts, please comment.x
()
x