RRDtool / Xymon / Hobbit script(s)

[th]

I'm attempting to watch and understand performance on my systems through graphing.

Obviously load goes up when I'm capturing more frames (or other things). But my goal is to create a history and graph it. I can then match this history against CPU load, memory consumption, disk growth, etc.

Does anyone have any pre-existing scripts to pull the FPS on each camera from out of the /var/log/messages file and log it RRD style? I've read about perl's File::Tail for taking just the new lines of a file but I think that's over my head for programming.

So, here's to hoping one of you has already created something like this.

And if not, I will eventually post my script/method when I find or create it.

[whatboy]

Like this???

cat /var/log/zm/zm.log | grep " Processing at" | sed 's/ //g' | cut -d 'f' -f1 | cut -d 't' -f4

cat /var/log/zm/zm.log | grep " Capturing at" | sed 's/ //g' | cut -d 'f' -f1 | cut -d 't' -f5

[whatboy]

Or maybe like this...

cat /var/log/zm/zm.log | grep " Capturing at" | cut -d 'C' -f2 | cut -d 't' -f3 | cut -d 'f' -f1 | sed 's/ //g'

cat /var/log/zm/zm.log | grep " Processing at" | cut -d 'C' -f2 | cut -d 't' -f3 | cut -d 'f' -f1 | sed 's/ //g'

[th]

That's how I would have traditionally done this. But I'm afraid that if I do it that way I risk getting repeating values. Or missing a value. A constant tail should do the trick. Here's what I've come up with since I wrote the question:

Install Perl-POE (from rpm or from cpan).

Grab this perl code from the POE site:

http://poe.perl.org/?POE_Cookbook/Watching_Logs

Code: Select all

#!/usr/bin/perl -w
# http://poe.perl.org/?POE_Cookbook/Watching_Logs
use POE qw/Wheel::FollowTail/;
use strict;
$| = 1;
my $filename = $ARGV[0];
die "Usage: $0 <filename>\n"                     unless $filename;
die "$0: $filename: No such file or directory\n" unless -e $filename;
die "$0: $filename: Permission denied\n"         unless -r $filename;
POE::Session->create(
  inline_states => {
    _start => sub {
      $_[HEAP]->{wheel} = POE::Wheel::FollowTail->new(
        Filename   => $_[ARG0],
        InputEvent => 'got_line',
        ErrorEvent => 'got_error',
        SeekBack   => 1024,
      );
      $_[HEAP]->{first} = 0;
    },
    got_line => sub { print "$_[ARG0]\n" if $_[HEAP]->{first}++ },
    got_error => sub { warn "$_[ARG0]\n" },
  },
  args => [$filename],
);
$poe_kernel->run();

Call it something. I called it 'ptail' per a recommendation I read on their site.

Then call it using your own shell script like so:

Code: Select all

#!/bin/bash

PERLSCR=/usr/local/scripts/ptail
CAMNAME=$1

$PERLSCR /var/log/messages | awk "/Capturing/ && /$CAMNAME/ {print \$12}"

Now when I run it with the name of my camera as the argument I get the output I want.

Code: Select all

[root@zmhost tmp]# ./fps.recording.sh Trendnet
19.23
19.23
18.77
19.23

At this point it appeared to be just shell coding. Which is something I'm good at.

But for some reason that I'm not yet able to determine, I can't redirect that output into a file that I specify. I've tried redirecting output after the last line in the script and I've tried redirecting output after the argument in the command line that I'm running. Neither worked. It would have no output as STDOUT or to the file I redirected to. I suspect it has something to do with how it is a "tail" without and end of the output.

...so I'm close already...

[whatboy]

I have installed MRTG and no redundant or repeated values, even when restarting mrtg or the server... ofcourse I don't have that FPS graph, but I have MYSQL and Temp graph among many others...

[th]

I scrapped my previous method in perl and decided to make a few assumptions about the logs always being in the same order. So here's what I did to get this running on Xymon.

Placed a shell script in ~xymon/client/ext/ called fps.sh

Code: Select all

#!/bin/bash

$BB $BBDISP "status $MACHINE.fps green `date`

        cam1 : `tail -250 /var/log/messages | awk \"/Capturing/ && /Foscam/ {print \\$12}\" | tail -1`
        cam2 : `tail -250 /var/log/messages | awk \"/Capturing/ && /CiscoPTZ/ {print \\$12}\" | tail -1`
        cam3 : `tail -250 /var/log/messages | awk \"/Capturing/ && /Trendnet/ {print \\$12}\" | tail -1`
        "

        exit 0

Obviously your camera names will vary. Mine are named after the models since this is a mobile demo system.

Then add to ~xymon/client/etc/clientlaunch.cfg :

Code: Select all

[fps]
        ENVFILE /usr/lib64/xymon/client/etc/hobbitclient.cfg
        CMD /usr/lib64/xymon/client/ext/fps.sh
        LOGFILE $HOBBITCLIENTHOME/logs/fps.log
        INTERVAL 3m

You might want to make the interval longer or shorter. And of course your paths will vary for your xymon client installation.

At this point you want to restart the client and make sure the xymon web server is showing the new column and data.

Now go into the server CLI and modify the /etc/xymon/hobbitserver.cfg and add this to the end of the TEST2RRD line: fps=ncv

Mine, with everything else as default looks like this:

Code: Select all

TEST2RRD="cpu=la,disk,inode,qtree,memory,$PINGCOLUMN=tcp,http=tcp,
dns=tcp,dig=tcp,time=ntpstat,vmstat,iostat,netstat,temperature,apache,bind
,sendmail,mailq,nmailq=mailq,socks,bea,iishealth,citrix,bbgen,bbtest,bbproxy,
hobbitd,files,procs=processes,ports,clock,lines,ops,stats,cifs,JVM,JMS,HitCache,
Session,JDBCConn,ExecQueue,JTA,TblSpace,RollBack,MemReq,InvObj,snapmirr,snaplist,
snapshot,if_load=devmon,temp=devmon,fps=ncv

Next, add another line somewhere after the TEST2RRD line in the same file:

Code: Select all

NCV_fps="cam1:GAUGE,cam2:GAUGE,cam3:GAUGE"

Go to your installation's rrd folder (/var/lib/xymon/rrd/demo.demo.com in my case) and make sure you see fps.rrd showing up. If not, look for errors in the rrd log files within the /var/log/xymon dir.

Now, setup the graphs in /etc/xymon/hobbitgraph.cfg. Add this stanza:

Code: Select all

[fps]
                TITLE FPS info             
                YAXIS FPS
                DEF:cam1=fps.rrd:cam1:AVERAGE
                DEF:cam2=fps.rrd:cam2:AVERAGE
                DEF:cam3=fps.rrd:cam3:AVERAGE
                LINE2:cam1#00CCCC:cam1 FPS
                LINE2:cam2#FF0000:cam2 FPS
                LINE2:cam3#FF00FF:cam3 FPS
                COMMENT:\n
                GPRINT:cam1:LAST:cam1 FPS \: %5.1lf%s (cur)
                GPRINT:cam1:MAX: \: %5.1lf%s (max)
                GPRINT:cam1:MIN: \: %5.1lf%s (min)
                GPRINT:cam1:AVERAGE: \: %5.1lf%s (avg)\n
                GPRINT:cam2:LAST:cam2 FPS\: %5.1lf%s (cur)
                GPRINT:cam2:MAX: \: %5.1lf%s (max)
                GPRINT:cam2:MIN: \: %5.1lf%s (min)
                GPRINT:cam2:AVERAGE: \: %5.1lf%s (avg)\n
                GPRINT:cam3:LAST:cam3 FPS \: %5.1lf%s (cur)
                GPRINT:cam3:MAX: \: %5.1lf%s (max)
                GPRINT:cam3:MIN: \: %5.1lf%s (min)
                GPRINT:cam3:AVERAGE: \: %5.1lf%s (avg)\n

And the last thing you might want to do is have this fps graph show up on the trends page. So go back into the /etc/xymon/hobbitserver.cfg file and add "fps" at any point in the line that looks roughly like this:

Code: Select all

GRAPHS="la,disk,inode,qtree,files,processes,memory,fps,users,vmstat,
iostat,tcp.http,tcp,ncv,netstat,ifstat,mrtg::1,ports,temperature,ntpstat,apache,
bind,sendmail,mailq,socks,bea,iishealth,citrix,bbgen,bbtest,bbproxy,hobbitd,clock
,lines,ops,stats,cifs,JVM,JMS,HitCache,Session,JDBCConn,ExecQueue,JTA,TblSpace
,RollBack,MemReq,InvObj,snapmirr,snaplist,snapshot,devmon::1,if_load::1,temp"

The order of the placement is reflected in the order of the graphs on the trends page. Notice how I put mine in as the 8th item in the list since I wanted it near the top.

And, obviously, everywhere that I have the "cam1" through "cam3" setup, you can keep adding cams in there. Don't forget to change the color in the graphing or it will get ugly down the road.

And I think that's all.