Maps for the lazy

Creating complex maps is a time-consuming job. Actually, even designing a rather small map of 25 elements can take you an hour. That’s time you rather want to spend on something useful or fun, unless you’ve got a fetish for repetitive work. All we need to automate this task, is a network/graph library like Networkx and the Zabbix API.

Notice: I’ll use the broader terms node and edge instead of the Zabbix terms map element and map link interchangeably throughout this writing. Node does not refer to distributed monitoring here.

Manual mapping

Why does it take so long to create a map manually? The work consists of two steps, once you know what you’re going to draw:

• Add and set up all the nodes and edges
• Move things around until you are happy with what you see

Adding new map elements isn’t a terribly efficient process: Click, drag, configure, repeat. Since you can’t move multiple elements at once, you have to drag them away from where they spawn one by one. Not being able to move multiple elements is particularly painful when you’re re-arranging major parts of your map. You also can’t bulk-add hosts. Even if you could, what if you wanted to bulk-add images?

If you are adding hosts to a map, each element must be configured to actually represent an existing host. That’s individual configuration, so you can’t just do it by selecting multiple elements. You also can’t clone an existing element, letting it serve as a template. You have to go through the whole creation process again instead. If you’re using icon mapping, you’ll notice that the mappings are not there while you’re editing your map. That forces you to layout elements without knowing their actual size. Needless to say, you won’t get it right on the first try!

By all means, a computer is probably better in lay-outing a complex graph anyway, trying to avoid unnecessary intersections and all that! That job is slightly aggravated by Zabbix’ exclusively using straight edges to connect nodes.

So what?

In the face of the above, we should dismiss the idea of manual editing for any non-trivial map. What we want instead, is something that can process source information that we already have or can easily provide, in order to produce ready-to-use Zabbix maps. This involves transforming information into the domain of map elements and map links, as well as laying out the nodes. The transformation is mostly up to us. We’ll leave the lay-outing to Networkx though, which will serve as an intermediate layer to a layout engine like Graphviz, Matplotlib or one of their own implementation. Let’s put the idea into action! The below provides examples for two different sources of information. The provided scripts use gescheit’s API module.

Map created from Graphviz DOT file

• Script
• Data

The Graphviz’ DOT language is a simple, plain-text syntax to define graphs. Thus it is very well suited to be diff-ed and stored in versioning systems, much like Tex. The three lines below are enough to define a graph containing two connected nodes, A and B:

graph {
  A -- B
}

More often than not, you’ll want to add attributes to your nodes that contain valuable information for your map, such as actual host names, labels, line colours or which icons to use.

graph {
  //Look, ma, comments!
  //Let's define node details

  //We don't want to type the long host names when defining edges
  bob [hostname="bob.example.com"]
  joe [hostname="joe.tx123.ops.example.com"]

  router [label="Our router" zbximage="router"]

  mystery_guest

  //---------------------------------------------
  //Let's define connections between the nodes

  //This notation allows for muliple edges from "router" in one go
  router -- { bob joe }

  //An otherwise unconnected daisy chain of nodes without further details
  host_a -- host_b -- host_c -- host_d
 }

The syntax above defines an undirected graph. That is, the connections don’t imply a direction of flow, like on a flow chart. You could create a directed graph as well, if it helps your layout or understanding. Zabbix still won’t show arrows. Notice, if you’re using Networkx, you’ll have to patch it to allow for the curly brace notation!

You can write your DOT files in any text editor. If you desire visual feedback or syntax validation, you’re better off with vimdot or another specific editor. Sadly, vimdot doesn’t work very well for me.

The example Python script reads the DOT file into a graph. The graph is then lay-outed and the resulting coordinates are supplied to the map.create API call. Nodes that feature a hostname attribute are considered to be Zabbix hosts, while other nodes are created as images. label and color are standard Graphviz attributes. hostname and zbximage are used to hold additional information.

Simple topology map from neighbourhood protocol data

• Script
• Data

This example is sort of a follow-up to my talk at Zabbix Conference 2012. I felt pretty ashamed when I was asked for an actual map rendering and couldn’t provide one. I had produced a few draft renderings, but they were far from what I wanted them to be. I was still using Zabbix 1.8 at that point. Generating 1.8 map XML was a plague, due to the mixed use of XML elements and attributes. Worse yet, the XML stanzas had to be in a particular order, as they were parsed as read. This time around, we’re not generating XML, but we are using the API.

Our source data resembles the neighbourhood export format of Cisco Prime data, Cisco’s management suite, previously known as Cisco Works. You could potentially retrieve similar information via SNMP as well. The CSV file is read and a graph is constructed from the supplied data. In practice, such a graph can easily end up being too large to display on a single map, which calls for segmentation. Furthermore, you may like to add firewalls or remove irrelevant edges. The script has some hints, but I’m not going to cover these topics here. The script is not making any lookups. It just creates image nodes and adds static labels. Let your fantasy run wild about how it could be extended!

Bottom line

Generating your maps with a script can save you a lot of tedious work. Drastically speaking, it doesn’t even matter if the produced layout is total garbage. Even if you have to re-arrange everything, you still saved a lot of clicking and scrolling and waiting and dragging.

As with all generated things, manual modifications are in constant danger of being lost. It’s particularly annoying to lose the fine-adjustment you made to the layout. As long as you’re using hosts as map elements, certain changes in Zabbix configuration will be reflected. This is not the case for images, of course. Newly added hosts won’t magically appear on maps either, so manual labour or clever scripting is still necessary.

You can’t modify Graphviz layout results in a very direct manner. Different algorithms and various settings strongly influence the result, but that needs exploration. The example DOT file is showing some of them — most likely in a poor manner.

But what if …

What if we just defined maps inside Zabbix, using a Graphviz-like syntax in the first place? What if a Javascript library would layout it dynamically? It should be possible to grab existing map links and elements via the API and render a map using a library like D3 or Sigma, instead of using the Zabbix map rendering facility. That would allow for zooming, dynamic navigation and so forth. D3 could also be the weapon of choice, when it comes to chart rendering, covering wishes such as new chart types, overlays, mouse-over effects.

What if maps could be created automatically in the first place? The most prominent use case might be topology. Suites like NAV or Netdisco can gather topology information and display maps based on it. NAV is using D3 for this purpose, by the way. I don’t know if it makes any sense to duplicate the effort of gathering that data. If Zabbix could store and process topology information, a bridge might be an option. For the time being, you can still create static maps with a script periodically. Maps based on dependency are another interesting consideration and so are maps based on IP networks.

Let’s see what the future brings! Hopefully the above example scripts can help save at least one intern from carpal tunnel syndrome or a nervous breakdown.