The long-awaited release of Zabbix 3.4 is available now, which brought many useful improvements, among which were customizable JMX endpoints and flexible detection of MBeans.

It’s so cool, huh?

If you use Zabbix and you want to monitor Java applications, then yes — it can greatly improve your life, because you had to resort to various tricks before, and now everything works, as they say, out of the box.

And what’s wrong with these JMX endpoints?

Native monitoring of Java applications through JMX has appeared in Zabbix for a long time — since version 2.0 — and since, this functionality has been improving from version to version. But so it happened that in previous releases of JMX, the endpoint was hardcoded into the Zabbix Java Gateway code in such a naive way:

As it turned out, the Java world is full with lots of software that uses different endpoints, and older versions of Zabbix did not allow changing this parameter. This has become a real headache for many users. For example, the popular application server JBoss EAP 6 uses its JBoss Remoting protocol for remote access to JMX instead of RMI, and JMXServiceURL for it should look like this:

service:jmx:remoting-jmx://{HOST.CONN}:{HOST.PORT}

And, for example, WebSphere 8.5 uses RMI over IIOP (instead of JRMP) and JMXServiceURL for it should be like this:

service:jmx:iiop://{HOST.CONN}:{HOST.PORT}/jndi/JMXConnector

Zabbix 3.4 is just set to solve this problem.

I didn’t get anything. JMX, RMI, JNDI? WTF?

Okay, well, let’s take a look at how this works.

JMX (Java Management Extensions) is a Java technology designed to monitor and manage (including remotely) various objects (resources): applications, devices, networks — if only this object was written in Java.

These resources are called MBeans (ManagedBeans). Each such object implements a specific interface, through which you can access the attribute values of this object, as well as call its methods and receive notifications (if the application registers the corresponding “listening” MBeans).

MBeans are registered on the MBean Server — object registry. Any registered object becomes available to applications (more precisely, its interface).

The resources are accessed using JMX connectors, which make the MBean Server available to JMX clients. The JMX connector consists of a client and a server. The connector server connects to the MBean server and takes connection requests from clients. The connector client is usually located on another JVM, and more often than not, on another machine in relation to the connector server.

The JMX API has a standard connection protocol based on Remote Method Invocation (RMI). This protocol allows the JMX client to remotely access MBeans on the MBean server. In addition to regular RMI, there are other protocols: JMXMP, JBoss Remoting, Hessian, Burlap and even HTTP and SNMP.

Using the MBean interface, a client can receive various metrics for this object, and also call public methods.

Schematically, the interaction of components can be represented as follows:

Any application on the Java SE platform “out of the box” has the ability to monitor it: the RMI connector automatically makes your Java application accessible for remote management and monitoring. Just run the application with the required parameters, and JMX clients (and Zabbix Java Gateway is a JMX client) will be able to connect to it remotely and get the necessary metrics.

To specify the JMX client for the specific application to which you want to connect, a special address is used, which is called the JMX endpoint (also JMXServiceURL). Strictly speaking, this is the address of the JMX API connector server. The format of this address is determined by RFC 2609 and RFC 3111. In general, it looks like this:

service:jmx:protocol:sap

Where “service:jmx:” is a constant.

protocol is a transport protocol (one of many: RMI, JMXMP, etc), used to connect to the connector server.

sap is the address at which the connector server can be found. Specified in this format (this is a subset of the syntax defined in RFC 2609): 

//[host[:port]][url-path]

host[:port] is ipv4 host address (or ipv6, enclosed in square brackets) and optional (depending on the protocol) port number.

url-path is optional URL (depends on the protocol). 

The best way to deal with this is by example. It is often possible to find such a JMX endpoint and the way it looks raises lots of questions: service:jmx:rmi://host:port1/jndi/rmi://host:port2/jmxrmi. But in fact, it’s not that bad.

host is the target host where our application is running.

port1 is the RMI server port which we want to connect to.

and port2 is the RMI registry port (the directory where RMI servers are registered). Default: 1099.

If you know that the RMI registry returns the address and port of the RMI server at the request of the client, it becomes clear that the first part here is superfluous. Thus, the address can be shortened to this:

url-path part means literally the following: take the part of the URL that follows immediately behind /jndi/ and at this address execute the JNDI request in the RMI registry to get information about the RMI server. The registry in response will return the host and port of RMI server.

It should be noted that the port in this case will be randomly generated and there may be problems with configuring the firewall. Such cases is where we should use the previous version of the JMX endpoint record, because it allows you to explicitly specify the port.

If you would like to get deeper into JMX, we recommend that you consult the official documentation: https://docs.oracle.com/javase/9/jmx/toc.htm.

Can you show it in practice?

As simple as that. Let’s try to configure JBoss EAP 6.4 monitoring as an example.

First, let’s make some assumptions:

1. You have already installed Zabbix 3.4 and Zabbix Java Gateway. If not, then you can do it in accordance with the documentation.
2. Zabbix Server and Java Gateway are installed with the prefix /usr/local/.
3. JBoss is already installed in /opt/jboss-eap-6.4/ and is running in standalone mode.
4. For simplicity of the experiment, we shall assume that all these components work on the same host.
5. Firewall and SELinux are disabled (or configured accordingly, but this is beyond the scope of the article).

Let’s make some simple settings in zabbix_server.conf:

JavaGateway=127.0.0.1
StartJavaPollers=5

And in the configuration file zabbix_java/settings.sh (or zabbix_java_gateway.conf):

START_POLLERS=5

Check that JBoss listens to its standard management port:

$ netstat -natp | grep 9999
tcp        0      0 127.0.0.1:9999          0.0.0.0:*               LISTEN      10148/java

Now let’s create a host with JMX interface 127.0.0.1:9999 in Zabbix.

If now we just take the standard “Template App Generic Java JMX” and link it to the host, then we probably will get an error:

$ tail -f /tmp/zabbix_java.log

Java Gateway tells us that at the specified endpoint, it is not RMI at all. Well, we already know that this version of JBoss uses the JBoss Remoting protocol instead of RMI, and we just need to start knocking at the right endpoint.

Let’s make the Full Clone “Template App Generic Java JMX” and call it “Template App Generic Java JMX-remoting.” Let’s select all the data elements inside this template and execute the Mass update operation for the JMX endpoint parameter. Let’s enter the following URL:

service:jmx:remoting-jmx://{HOST.CONN}:{HOST.PORT}

Let’s update the configuration cache:

$ /usr/local/sbin/zabbix_server -R config_cache_reload

Error again.

What is it this time?

“Unsupported protocol: remoting-jmx” means that the Java Gateway does not know how to work with the specified protocol. Well, let’s teach him. The advice from the article “JBoss EAP 6 monitoring using remoting-jmx and Zabbix” will help us in this.

Create a ~/needed_modules.txt file with the following content:

jboss-as-remoting
jboss-logging
jboss-logmanager
jboss-marshalling
jboss-remoting
jboss-sasl
jcl-over-slf4j
jul-to-slf4j-stub
log4j-jboss-logmanager
remoting-jmx
slf4j-api
xnio-api
xnio-nio

Execute the command:

$ for i in $(cat ~/needed_modules.txt); do find /opt/jboss-eap-6.4 -iname ${i}*.jar -exec cp {} /usr/local/sbin/zabbix_java/lib/ \; ; done

Thus, Java Gateway will have all the necessary modules for working with jmx-remoting. What’s left is just to restart the Java Gateway, wait a bit and if you did everything right, see that the precious data began to arrive in Zabbix:

And what about JMX discovery?

JMX discovery appeared in Zabbix simultaneously with the advent of native support for monitoring Java applications via JMX. This function is the responsibility of the undocumented (at that moment) jmx.discovery key. The documentation mentioned nothing about it, because it was still a very crude function:

1. Such a discovery doesn’t make much of a point, since there is no possibility for filtering. And hardly anyone is required to discover all existing JMX objects.
2. This is a very slow solution, since there is one request for each MBean, and there can be quite a lot of them. It is very likely that such a test will simply fall off by the timeout.
3. In this form, you can create only one discovery rule within the host, which is very sad, because in practice I would like to create a lot of rules (it’s common for data types to be different for different attributes).

Zabbix 3.4 provides the ability to filter, which immediately solves many problems.

The new check looks like this: jmx.discovery[<discovery mode>, <object name>]

And allows you to specify whether discovery of MBeans or their attributes is required, and what pattern to use for search thereof.

Let’s give it a try! Let’s monitor, for example, garbage collectors. It is known that their names can differ depending on what parameters the JVM is running. So we cannot specify static names and keys for data items — this is just the job for jmx.discovery

Documentation describes four examples of using jmx.discovery:

We opt not to use first two options, because they do not offer nice solutions in terms of performance. Let’s see what the latter two return to us.

For this purpose, we can simply create an item with the right key and text data type. But it’s not very convenient. First, the output will be unformatted and unobvious. Secondly, to change the request, we will have to change the item key and wait for the data to be updated for some time.

This is not our way. Let’s do something like zabbix_get, but instead of the agent we will go to the Java Gateway. To do this, we will slightly modify the script for this new API proposed in this note: add jmx_endpoint to the query and correct the removal of the header from the answer:

#!/usr/bin/env bash

if [ $# != 6 ]
then
   echo "Usage: $0 <JAVA_GATEWAY_HOST> <JAVA_GATEWAY_PORT> <JMX_SERVER> <JMX_PORT> <JMX_ENDPOINT> <KEY>"
   exit;
fi

# create connection
exec 3<>/dev/tcp/$1/$2

# compose message
MSG="{\"request\": \"java gateway jmx\", \"conn\": \"$3\", \"port\": $4, \"jmx_endpoint\": \"$5\", \"keys\": [\"$6\"]}"

# write message length as zero-padded 16-digit hexadecimal number
printf -v LEN '%016x' "${#MSG}"

# prepare message length in little endian representation
BYTES=""
for i in {0..14..2}
do
    BYTES="\\x${LEN:$i:2}$BYTES"
done

# prepend protocol header and message length
printf "ZBXD\\1$BYTES%s" "$MSG" >&3

# output the result skipping 6 bytes of "ZBXD\\1" header and 8 bytes of message length
tail -c+14 <&3

Now we can easily see what the gateway gives us in response to our requests:

$ ./zabbix_get_java.sh 127.0.0.1 10052 127.0.0.1 9999 'service:jmx:remoting-jmx://127.0.0.1:9999' 'jmx.discovery[beans,\"*:type=GarbageCollector,name=*\"]' | jq '.data[0].value | fromjson | .data'

Exactly what we need!

If we needed, say, just a couple of metrics, then we could discover all gc’s and create each metric by the prototype.

1. Create a discovery rule. We are looking for MBeans (by the way, please note that the custom JMX endpoint is used everywhere).

2. Create prototypes for every metric that interests us. The name of the data element and its key can use any macros that we saw in JSON.

By the way, talking about macros. When MBeans are discovered, macros are dynamically generated based on the properties of MBeans (such as type and name).

!!! Pay attention to this problem when using dynamic macros:  https://support.zabbix.com/browse/ZBX-12705

But, say, we want to create all available numerical metrics for garbage collectors.

1. Then we will create an attribute detection rule with a filter by the data type.

2. And just one prototype:

Here is a simple way you can monitor any Java application. Go for it! 🙂

I currently do not have the possibility to upgrade to Zabbix 3.4, what can I do?

The community has come up with many ways to get around this problem. If you do not yet have the option to upgrade to this version, grab the links: one and two.

Result

Thanks to the new implemented capabilities, monitoring of Java applications in Zabbix has ceased to be a pain. On the contrary, with each new version it becomes more and more simple and enjoyable. We will try to please you more!

Stay tuned!

P.S. The article is also available in Russian.

P.S.S. Get acquainted with other Zabbix 3.4 features in our blog post on mass data collection and preprocessing.

Primary sources used in this blog:

• http://java-course.ru/articles/jmx/
• http://www.javaspecialist.ru/2011/04/jmx-firewall.html
• https://docs.oracle.com/javase/tutorial/jmx/remote/index.html
• https://dzone.com/articles/remote-jmx-access-wildfly-or
• https://habrahabr.ru/post/225527/
• https://jcp.org/en/jsr/detail?id=160
• https://www.denniskanbier.nl/blog/monitoring/jboss-eap-6-monitoring-using-remoting-jmx-and-zabbix/
• https://www.ibm.com/developerworks/ru/library/j-jtp09196/index.html
• https://www.ixxus.com/blog/blog201102monitor-and-manage-alfresco-jmx/

About Vadim Ipatov

Developer at Zabbix

View all posts by Vadim Ipatov