Importing legacy Cacti/MRTG data into Zabbix

Here at the University of Hawaii at Manoa ITS department, we recently began using Zabbix. Before adopting Zabbix, we monitored our resources with a loosely integrated mix of several software components, including Cacti, MRTG, and uPortal. Having used this old system for quite a while, we have a considerable amount of data which is valuable for trending. So how do we switch to this great Zabbix system but retain all of our trending statistics? The instructions here describe how to convert data from either Cacti or MRTG and import it directly into Zabbix. For those unfamiliar with Cacti or MRTG, they collect network or system statistics using SNMP or custom scripts and save the data in RRD (Round-Robin Database) files. These files are then used to present graphs when requested.

With such a significantly different back-end from Zabbix, importing data from Cacti is a difficult process that requires several scripts and steps. However, using the steps below and the script included here, this task can be accomplished rather quickly.

Converting RRD data and importing to Zabbix

The instructions detailed below require that your system have the following resources:

• An RRD file that contains the legacy data to be imported,
• RRDTool,
• Zabbix version 1.8 installed along with zabbix_sender,
• and the custom conversion script (code at the end of this post).

1. Convert .rrd to .xml using rrdtool

Since RRD files are difficult to interact with, we must first convert our data file to a parse-able format. RRDTool, the tool used by Cacti and MRTG to interact with RRD data files, provides the dump option to convert to RRD files to large XML representations of the data. It can be run with the following command:

$ rrdtool dump filename.rrd > filename.xml

where filename is the name of the particular file we are converting. This gives us a large XML file called filename.xml that contains all of the data that was in the rrd file.

2. Create an item in Zabbix to contain the data

To contain the data being converted from RRD, we need to prepare an item in Zabbix. We can either use an existing host and/or item or create a new one, so long as there is an item in Zabbix that will contain the RRD data. We can even import legacy data to an existing Zabbix item that already has data! Only two things matter: we must be sure to note the host name and item key used, as we will use them in subsequent steps, and the item must be temporary changed to a Zabbix trapper item. For details on creating hosts, items, or Zabbix Trappers, please see the Zabbix documentation.

Note: If data to be imported goes back for more than one year, make sure to increase Keep trends value, otherwise housekeeper process will start deleting those values.

3. Convert the .xml file to Zabbix input format

At the end of this post is the code for a custom script that will convert the XML data to a format usable by zabbix_sender. To use this, copy the code and paste it into a file, say convertRRD.sh. You can then run the script with the following command:

$ convertRRD.sh <inputFile> <hostname> <itemKey> <DataSource> <outputFile>

where inputFile is the .xml file from step 1, hostName is the name of the Zabbix host and and itemKey is the key of the item from step 2, DataSource is the # of the data source in the RRD file, and outputFile is the name of the temporary file for the next step.

For most cases, we will be using DataSource=1, but if you have an RRD file with multiple data sources, first determine which you want. This is done with the command:

$ rrdtool info <filename.rrd> | more

The order the ds[] items appear is what determines the DataSource #.

4. Run zabbix_sender to import converted data

Manually copy the temporary <outputFile> from step 3 to your Zabbix server (e.g., using scp) and run the following command to import the data into Zabbix:

$ zabbix_sender -z 127.0.0.1 -T -i <outputFile>

Again, <outputFile> is the temporary file we created in step 3.

Note: zabbix_sender can be run from a remote machine and send the data to your Zabbix server, but we experienced a timestamp error when doing this. Though the data would import successfully, graphs of the data would go back to 1970 (epoch).  This error only occurred when we used zabbix_sender from a remote source.

If the data was correctly imported, you should see a series of lines that look like:

Info from server: “Processed 250 Failed 0 Total 250 Seconds spent 0.102974”

Depending on the amount of data, this may take a while. An average of 500 data points are imported per second.
Note: The Zabbix configuration parameter CacheUpdateFrequency (default 60 seconds) determines the delay between configuration cache updates. If you create or change a Zabbix item and run zabbix_sender too quickly, the update may not have been done, causing the import to fail.

5. Enable Zabbix data collection

To enable data collection to commence with Zabbix (or continue if using an existing item), you must convert the Zabbix item back to the correct type (e.g., Zabbix agent).

You should now have all of the data from your old .RRD files safely contained in Zabbix. This method allowed us to import a great deal of old statistical data. We had some data going back more than 5 years!

RRD Conversion script code

The code of the custom script that converts the XML data to a format for zabbix_sender is below. Copy and paste it into a shell script and run it with the XML file created in step 1:

#!/bin/bash
FILENAME=$1
HOST=$2
ITEM=$3
DS=$4
OUTFILE=$5

DS=`expr $DS \* 2`
DS=`expr $DS + 7`

[ "$#" -eq 5 ] || { echo "usage: conversion.sh <inputFile> <hostname> \
<itemkey> <DataSource #> <outputFile>"; exit 1; }

cat $FILENAME | awk '/<row><v> +[0-9]/' | awk \
-vhostitem="$HOST $ITEM" -vDS="$DS" '{printf("%s %.0f %.0f\n", \
hostitem, $6, ($(0+DS)*1))}'

About Ben Karsin, University of Hawaii

Ben Karsin, University of Hawaii ITS Department.

View all posts by Ben Karsin, University of Hawaii