Importing legacy Cacti/MRTG data into Zabbix

Here at the University of Hawaii at Manoa ITS department, we recently began using Zabbix. Before adopting Zabbix, we monitored our resources with a loosely integrated mix of several software components, including Cacti, MRTG, and uPortal. Having used this old system for quite a while, we have a considerable amount of data which is valuable for trending. So how do we switch to this great Zabbix system but retain all of our trending statistics? The instructions here describe how to convert data from either Cacti or MRTG and import it directly into Zabbix. For those unfamiliar with Cacti or MRTG, they collect network or system statistics using SNMP or custom scripts and save the data in RRD (Round-Robin Database) files. These files are then used to present graphs when requested.

With such a significantly different back-end from Zabbix, importing data from Cacti is a difficult process that requires several scripts and steps. However, using the steps below and the script included here, this task can be accomplished rather quickly.

Converting RRD data and importing to Zabbix

The instructions detailed below require that your system have the following resources:

  • An RRD file that contains the legacy data to be imported,
  • RRDTool,
  • Zabbix version 1.8 installed along with zabbix_sender,
  • and the custom conversion script (code at the end of this post).

1. Convert .rrd to .xml using rrdtool

Since RRD files are difficult to interact with, we must first convert our data file to a parse-able format. RRDTool, the tool used by Cacti and MRTG to interact with RRD data files, provides the dump option to convert to RRD files to large XML representations of the data. It can be run with the following command:

$ rrdtool dump filename.rrd > filename.xml

where filename is the name of the particular file we are converting. This gives us a large XML file called filename.xml that contains all of the data that was in the rrd file.

2. Create an item in Zabbix to contain the data

To contain the data being converted from RRD, we need to prepare an item in Zabbix. We can either use an existing host and/or item or create a new one, so long as there is an item in Zabbix that will contain the RRD data. We can even import legacy data to an existing Zabbix item that already has data! Only two things matter: we must be sure to note the host name and item key used, as we will use them in subsequent steps, and the item must be temporary changed to a Zabbix trapper item. For details on creating hosts, items, or Zabbix Trappers, please see the Zabbix documentation.

Note: If data to be imported goes back for more than one year, make sure to increase Keep trends value, otherwise housekeeper process will start deleting those values.

3. Convert the .xml file to Zabbix input format

At the end of this post is the code for a custom script that will convert the XML data to a format usable by zabbix_sender. To use this, copy the code and paste it into a file, say convertRRD.sh. You can then run the script with the following command:

$ convertRRD.sh <inputFile> <hostname> <itemKey> <DataSource> <outputFile>

where inputFile is the .xml file from step 1, hostName is the name of the Zabbix host and and itemKey is the key of the item from step 2, DataSource is the # of the data source in the RRD file, and outputFile is the name of the temporary file for the next step.

For most cases, we will be using DataSource=1, but if you have an RRD file with multiple data sources, first determine which you want. This is done with the command:

$ rrdtool info <filename.rrd> | more

The order the ds[] items appear is what determines the DataSource #.

4. Run zabbix_sender to import converted data

Manually copy the temporary <outputFile> from step 3 to your Zabbix server (e.g., using scp) and run the following command to import the data into Zabbix:

$ zabbix_sender -z 127.0.0.1 -T -i <outputFile>

Again, <outputFile> is the temporary file we created in step 3.

Note: zabbix_sender can be run from a remote machine and send the data to your Zabbix server, but we experienced a timestamp error when doing this. Though the data would import successfully, graphs of the data would go back to 1970 (epoch).  This error only occurred when we used zabbix_sender from a remote source.

If the data was correctly imported, you should see a series of lines that look like:

Info from server: “Processed 250 Failed 0 Total 250 Seconds spent 0.102974″

Depending on the amount of data, this may take a while. An average of 500 data points are imported per second.
Note: The Zabbix configuration parameter CacheUpdateFrequency (default 60 seconds) determines the delay between configuration cache updates. If you create or change a Zabbix item and run zabbix_sender too quickly, the update may not have been done, causing the import to fail.

5. Enable Zabbix data collection

To enable data collection to commence with Zabbix (or continue if using an existing item), you must convert the Zabbix item back to the correct type (e.g., Zabbix agent).

You should now have all of the data from your old .RRD files safely contained in Zabbix. This method allowed us to import a great deal of old statistical data. We had some data going back more than 5 years!

RRD Conversion script code

The code of the custom script that converts the XML data to a format for zabbix_sender is below. Copy and paste it into a shell script and run it with the XML file created in step 1:

#!/bin/bash
FILENAME=$1
HOST=$2
ITEM=$3
DS=$4
OUTFILE=$5

DS=`expr $DS \* 2`
DS=`expr $DS + 7`

[ "$#" -eq 5 ] || { echo "usage: conversion.sh <inputFile> <hostname> \
<itemkey> <DataSource #> <outputFile>"
; exit 1; }

cat $FILENAME | awk '/<row><v> +[0-9]/' | awk \
-vhostitem="$HOST $ITEM" -vDS="$DS" '{printf("%s %.0f %.0f\n", \
hostitem, $6, ($(0+DS)*1))}'

About Ben Karsin, University of Hawaii

Ben Karsin, University of Hawaii ITS Department.
This entry was posted in How To, Technical. Bookmark the permalink.

11 Responses to Importing legacy Cacti/MRTG data into Zabbix

  1. Oleksiy Zagorskyi says:

    Awesome!
    It was my old idea to try such approach to show users that it’s possible.
    Thanks that it done already!

    It worth to mention that configuration cache can be updated forcibly by the command:
    shell> zabbix_server -c /etc/zabbix/zabbix_server.conf -R config_cache_reload

    See documentation:
    http://www.zabbix.com/documentation/1.8/manual/processes/zabbix_server

    Again, thanks!

    • Richlv says:

      …and “-c /etc/zabbix/zabbix_server.conf” is redundant in 1.8, as that’s the default path ;)

      • Oleksiy Zagorskyi says:

        nice to know that, thanks ;)

      • Mamadou says:

        I did everything etxcaly as you said 3 times on 3 different fresh installs, and every time I get the following message, and I have no idea how to fix it.Missing file: /usr/share/cacti/lib/adodb/drivers/adodb-“mysql”.inc.phpADONewConnection: Unable to load database driver

        • Ben karsin says:

          At what step in the process are you getting this error message? Is this error showing up when you run rrdtool or zabbix_sender? I am not sure why it would be giving you a mysql error, but the path of the missing file is under Cacti. It could be a problem with your Cacti or RRDtool installation.

  2. André Déo says:

    Congratulations Ben.

    I’ll translate it to Portuguese and post at our Wiki http://zabbixbrasil.org/wiki ;-)

    Hugs,

    Déo

  3. Nocturne says:

    I would like to use the script but it seems like its not gvniig me the full results. when i run the script all of data changes but the graphs still shows the same information, that is with the spikes. maybe am missing out something, what needs to happen after running the script? should i run the indexmaker or wait for the cron job to update the graphs?

    • Ben karsin says:

      Running the script only creates the file containing all of the data in a format usable by zabbix_sender. Once you have it, you need to convert the item to a “Zabbix trapper” (step 2) and run zabbix_sender to import all of the data (step 4). The data wont be visible in Zabbix until zabbix_sender successfully imports the data.

  4. wagner says:

    Hi
    When i run the script:

    ./convertRRD.sh test.xml Router trap 1 router.txt
    it should create the file “router.txt” but nothing happens. the file isn’t created.
    Does anyone has had any success?

    thank you!

    • Ben karsin says:

      It could be a problem with the xml file somehow not being compatible with the script. Have you taken a look at the xml file to see if there are any major issues? What version of RRDtool are you using to do the xml dump?

      Would it be possible to post or email me a snippet of the xml file?

      • cogwheel says:

        RRDtool 1.4.7. The xml looks fine.

        xml example:

        0003
        300
        1425322504

        hdd_used
        GAUGE
        600
        0.0000000000e+00
        NaN


        10462773248
        4.1851092992e+10
        0

        hdd_total
        GAUGE
        600
        0.0000000000e+00
        NaN


        31607156736
        1.2642862694e+11
        0

        AVERAGE
        1

        5.0000000000e-01

        1.0462674002e+10
        1.2378300416e+10 NaN
        0
        “web1_hdd_total.xml” 254552L, 27182358C 1,1 Top
        NaN
        0

        3.1607156736e+10
        3.1607156736e+10
        NaN
        0

        8.9457512038e+093.1607156736e+10

Leave a Reply