In many cases it is quite trivial to collect some data and send them to Xymon in format like this:
Name: Value
e.g. if you have a device that collects weather information, it might look like
Temperature: 19.2 Humidity: 53 Wind: 9.6
In Xymon, this is known as NCV - Name, Colon, Value - formatted data, and Xymon has built-in support to pick up data formatted this way, and collect the data for graphs.
First create your test data. Typically, this is an extension script that sends in some data to Xymon, using a status or data command. If you use status, it will show up as a separate column on the display, with a green/yellow/red color that can trigger alerts. If you use data, Xymon just collects the data into a graph - you must go to the trends column to see the graph. For this example, we'll use status.
So we create an extension script. Here is an example script; it picks two numbers out of the Linux kernel's memory statistics, and reports these to Xymon.
#!/bin/sh
cat /proc/slabinfo | \
egrep "^dentry_cache|^inode_cache" | \
awk '{print $1 " : " $3*$4}' >/tmp/slab.txt
$XYMON $XYMSRV "status $MACHINE.slab green `date`
`cat /tmp/slab.txt`
"
exit 0
Save this script in ~xymon/client/ext/slab, and add a section to the ~xymon/client/etc/clientlaunch.cfg to run it every 5 minutes:
[slabinfo]
ENVFILE /usr/lib/xymon/client/etc/xymonclient.cfg
CMD /usr/lib/xymon/client/ext/slab
INTERVAL 5m
(On the Xymon server itself, you must add this to the file
~xymon/server/etc/tasks.cfg)
After a few minutes, a slab column should appear on your Xymon view of this host, with the data it reports. The output looks like this:
Sun Nov 20 09:03:44 CET 2005 inode_cache : 330624 dentry_cache : 40891068
This is obviously a name-colon-value formatted report, so we'll use the NCV module in Xymon to handle it. Xymon will find two datasets here: The first will be called inodecache, and the second dentrycache (note that Xymon strips off any part of the name that is not a letter or a number; Xymon also limits the length of the dataset name to 19 letters max. since RRD will not handle longer names). To enable this, on the Xymon server edit the ~xymon/server/etc/xymonserver.cfg file. The TEST2RRD setting defines how Xymon tests (status columns) map to RRD datafiles. So you add the new test to this setting, by adding slab=ncv at the end:
TEST2RRD="cpu=la,disk,<...lots more stuff...>,xymond,mysql=ncv,slab=ncv"
slab is the status column name, and =ncv is a token that tells Xymon to send these data through the built-in NCV module.
By default, the Xymon NCV module expects data to be some sort of counter, e.g. number of bytes sent over a network - it uses the RRD DERIVE datatype by default, which is for data that is continuously increasing in value. Some data are not like that - the data in our test script is not - and for those data you'll have to make an extra setting to tell Xymon what RRD data type to use. The RRDtool rrdcreate(1) man-page has a detailed description of the various RRD datatypes. It is available online at http://www.mrtg.org/rrdtool/doc/rrdcreate.en.html
Our test script provides data that goes up and down in value (it is the number of bytes of memory used for a Linux kernel bufffer), and for that kind of data we'll use the RRD GAUGE datatype. So we add an extra setting to xymonserver.cfg:
NCV_slab="inodecache:GAUGE,dentrycache:GAUGE"
This tells the xymond_rrd module that it should create an RRD file with two datasets of type GAUGE instead of the default (DERIVE). The setting must be named NCV_<columnname>.
In some cases it can be useful to use multiple RRD files - one for each dataset - instead of putting all of the datasets into one RRD file. E.g. if you want to add or remove datasets over time - if they are all stored in one RRD file then you have to dump, modify and reload the data from the RRD file. If you store each dataset in a separate file, then you can just delete the file. Xymon supports this if you call this setting SPLITNCV instead of NCV. Then Xymon will store each dataset in an RRD file column,dataset.rrd, e.g. slab,inodecache.rrd, and the name of the dataset will be "lambda". Apart from this, the setup is identical for the NCV- and SPLITNCV-methods.
The xymonserver.cfg file is not reloaded automatically, so you must restart Xymon after making these changes. Or at least, kill the xymond_rrd processes (there are usually two) - xymonlaunch will automatically restart them, and they will then pick up the new settings.
The next time the slab status is updated, Xymon will begin to collect the data. You can check this by looking for the slab.rrd file in the ~xymon/data/rrd/HOSTNAME/ directory. If you want to check the data it collects, the rrdtool dump ~xymon/data/rrd/HOSTNAME/slab.rrd will tell you what it got:
<!-- Round Robin Database Dump --> <rrd> <version> 0001 </version> <step> 300 </step> <!-- Seconds --> <lastupdate> 1132474725 </lastupdate> <!-- 2005-11-20 09:18:45 CET --> <ds> <name> inodecache </name> RRD datatype------> <type> GAUGE </type> <minimal_heartbeat> 600 </minimal_heartbeat> <min> 0.0000000000e+00 </min> <max> NaN </max> <!-- PDP Status --> current value-----> <last_ds> 330624 </last_ds> <value> 0.0000000000e+00 </value> <unknown_sec> 0 </unknown_sec> </ds>
If you go and look at the status page for the slab column, you should not see any graph yet, but a link to xymon graph ncv:slab. One final step is missing.
The final step is to tell Xymon how to create a graph from the data in the RRD file. This is done in the ~xymon/server/etc/graphs.cfg file.
[slab] TITLE Slab info YAXIS Bytes DEF:inode=slab.rrd:inodecache:AVERAGE DEF:dentry=slab.rrd:dentrycache:AVERAGE LINE2:inode#00CCCC:Inode cache LINE2:dentry#FF0000:Dentry cache COMMENT:\n GPRINT:inode:LAST:Inode cache \: %5.1lf%s (cur) GPRINT:inode:MAX: \: %5.1lf%s (max) GPRINT:inode:MIN: \: %5.1lf%s (min) GPRINT:inode:AVERAGE: \: %5.1lf%s (avg)\n GPRINT:dentry:LAST:Dentry cache\: %5.1lf%s (cur) GPRINT:dentry:MAX: \: %5.1lf%s (max) GPRINT:dentry:MIN: \: %5.1lf%s (min) GPRINT:dentry:AVERAGE: \: %5.1lf%s (avg)\n
[slab] is the name of this graph, and it must match the name of your status column if you want the graph to appear together with the status. The TITLE and YAXIS settings define the graph title and the legend on the Y-axis. The rest are definitions for the rrdgraph(1) tool - you should read the RRDtool docs if you want to know in detail how it works. For now, all you need to know is that you must pick out the data you want from the RRD file with a DEF line, like
DEF:inode=slab.rrd:inodecache:AVERAGEwhich gives you an "inode" definition that has the value from the inodecache dataset in the slab.rrd file. This is then used to draw a line on the graph:
LINE2:inode#00CCCC:Inode cacheThe line gets the color #00CCCC (red-green-blue), which is a light greenish-blue color. Note that you can have several lines in one graph, if it makes sense to compare them. You can also use other types of visual effects, e.g. stack values on top of each other (like the vmstat graphs do) - this is described in the rrdgraph man-page. An online version is at http://www.mrtg.org/rrdtool/doc/rrdgraph.en.html.
The GPRINT lines at the end of the graph definition also uses the inode value to print a summary line showing the current, maximum, minimum and average values from the data that has been collected.
Once you have added this section to graphs.cfg, refresh the status page in your browser, and the graph should show up.
If you want the graph included with the other graphs on the trends column, you must add it to the GRAPHS setting in the ~xymon/server/etc/xymonserver.cfg file.
GRAPHS="la,disk,<... lots more ...>,xymonproxy,xymond,slab"Save the file, and when you click on the trends column you should see the slab graph at the bottom of the page.
You probably used the wrong RRD datatype for your data - see step 4. By default, the RRD file expects data that is increasing constantly; if you are tracking some data that just varies up and down, you must use the RRD GAUGE datatype. Note that when you change the RRD datatype, you must delete any existing RRD files - the RRD datatype is defined when the RRD file is created, and cannot be changed on the fly.
Make sure you have ncv listed in the GRAPHS setting in xymonserver.cfg. (Don't ask why - just take my word that it must be there).
The NCV method works fine in many cases, but you may run into a situation where it isn't really suitable. One possible problem with this method is that you can only store data in a single RRD file using the NCV method, and there may be situations where you want to use multiple RRD files for flexibility - e.g. if you are reporting performance for a number of applications, then it is useful to have one RRD file for each application. And since there are different performance metrics for each application, you cannot use the SPLITNCV method.
Xymon supports a different method for collecting data in these cases - you can send a "trends" message to Xymon with the data you want to put into a graph. All hosts appearing on the Xymon server already have a "trends" status column, but if you send a "trends" status message to Xymon, it will not show up in the "trends" column - instead, it will be used to create/update an RRD file with data. And inside the "trends" message, you can define exactly what RRD files you want to use, how they are formatted, what data goes into the RRD file and so on. Here is an example of a "trends" message, using the same data that we used in the example above for collecting some data about the current weather:
data berlin.trends [weather.rrd] DS:temperature:GAUGE:600:U:U 19.2 DS:humidity:GAUGE:600:0:U 72 DS:wind:GAUGE:600:0:U 9.6
This creates an RRD file for the host "berlin", called "weather.rrd". The RRD file has three datasets: temperature, humidity and wind, all of which are of the GAUGE datatype. For more information about the DS definitions, see the .I rrdcreate(1) manpage. The current values for each of the datasets is specified after the DS definition.
If the RRD file already exists, then it just updates the data.
You still need to create graph definitions on the Xymon server, and if you want the trend graphs displayed on the "trends" status page then you must also add the graph name to the "TEST2RRD" and "GRAPHS" settings in .I xymonserver.cfg(5) but it is not necessary to restart the xymond_rrd program.
If you want to create more than one RRD file, you just add more data in the "trends" status message. E.g. if you have three applications with performance metrics:
data appserver.trends [salaryapp.rrd] DS:usercount:GAUGE:600:0:U 210 DS:payrollgeneration:GAUGE:600:0:U 29 [bulkmailapp.rrd] DS:customercount:GAUGE:600:0:U 1922 DS:mailsize:GAUGE:600:0:U 61293 DS:transmissiontime:GAUGE:600:0:U 88
This will create 2 RRD files, salaryapp.rrd with performance metrics from the Salary application, and bulkmailapp.rrd with performance metrics from the bulkmail application. Note that the metrics collected are quite different - you have total control over which RRD files get created, what datasets they contain etc.
To view the graphs, you still need to setup a definition for generating the graphs in the graphs.cfg configuration file, and the graph definition must be listed in both the TEST2RRD- and GRAPHS-settings in xymonserver.cfg - then the graph will appear on the "trends" page.