sobrique: (bubble tree)
[personal profile] sobrique
Following on from my previous post about why RRDtool is awesome.
A worked case study.

First off, we take the Raspberry Pi.
Install 'rrdtool' using:
sudo apt-get update rrdtool
And the perl library:
sudo apt-get librrds-perl

Sky routers have a router stats page, on http://192.168.0.1/sky_system.html
(You will need your router username and password)
You can check it works with
wget --http-user username --http-password password http://192.168.0.1/sky_system.html

There's a table in there that looks a bit like:



(Because I don't want to convert the HTML to plain text, I'd suggest looking at this file in nano or 'vi' - you'll quickly find the table you want by searching for the text string 'Router Statistics'.)

We need to get that information, and feed it into an RRD. The easiest way by far is using 'HTML::TableExtract'
You can download and install this by running:
sudo perl -MCPAN -e shell

And then:
install HTML::TableExtract

Whilst you're doing it, add in:
install Bundle::CPAN
install LWP
(LWP you need for this, Bundle::CPAN just has a bunch of useful utility stuff).

Now, LWP is a perl module that is short for 'libwww-perl' and it's a simple way of fetching web pages with perl.
http://search.cpan.org/~gaas/libwww-perl-6.05/lib/LWP.pm

The page above gives a sample, but for our purposes - we want to fetch this page from a sky router.
I'm going to use default values - as you will, of course, have changed your password from the default, you'll need to update accordingly.

#!/usr/bin/perl

use strict;
use warnings;

use LWP;

my $stats_url = "http://192.168.0.1/sky_system.html";
my $router_user = "admin";
my $router_password = "sky";

  my $UserAgent = LWP::UserAgent -> new;
  my $request = HTTP::Request -> new ( GET => $stats_url );
     $request -> authorization_basic ( $router_user, $router_password );
  my $response = $UserAgent -> request ( $request );
  print $response -> content();


Run that, and you'll get a splurge of text, and - within it - you should see your router stats.

Now, you could do things the hard way, and process the page 'by hand'. However, HTML::TableExtract makes it easy.

use HTML::TableExtract;

  my $table_extractor = HTML::TableExtract -> new ( headers => [ ( "Port ", "TxPkts", "RxPkts ",
                                                         "Collision Pkts", "Tx b/s ", "Rx b/s ", )] );
  $table_extractor -> parse ( $response -> content );

  foreach my $table ( $table_extractor -> tables )
  {
    print "Table: ", join ( ",", $table -> coords ), "\n";
    foreach my $row ( $table -> rows )
    {
      print join ( ",", @$row, "\n" );
    }
  }



TableExtractor is useful, because it rummages through your HTML, looking for a table that has the headers you specify, and spits it out in a fairly easy to handle form. It'll find any matches - and does have an option to just assume the first table matches. We can simplify the above to:

  foreach my $row ( $table_extractor -> rows )
  {
    print join ( ",", @$row ),"\n";
  }


And rather handily, the data we want is delivered to us in tabulated rows - in the same sequence as we specified them above, and without having to parse the HTML document.

So, what to do with these values? We record them in an 'RRD'. For details on using rrdtool, refer to the website for it:
RRD Documentation

For my purposes though:
I want to log the stats above sent and received packets and bytes/sec. (Rx denotes 'receive' and Tx denotes 'transmit'.

Do do this, we need to define the 'Data Sources' - packets is a cumulative value, and bytes is an instant one. So we need two different types of data source. RRDtool has 'COUNTER' for the former, and GAUGE for the later.

We define them with:
DS:TxPkts:COUNTER:600:0:U
DS:RxPkts:COUNTER:600:0:U
DS:CollPkt:COUNTER:600:0:U
DS:Txbps:GAUGE:600:0:U
DS:Rxbps:GAUGE:600:0:U

The last two values are ranges - 0 is the minimum, U is 'unknown' (any) for the maximum.
(600 denotes the 'heartbeat' of the sample, and is the maximum amount of time between samples before it's considered 'unknown')

To keep the value we need to define an archive period - RRA:
I'm going to set up 3:

RRA:AVERAGE:0.5:1:1200
RRA:AVERAGE:0.5:12:2400
RRA:AVERAGE:0.5:144:1200

The first says to keep 1 sample per interval (an 'interval' is set at create time, but default is 300s) for 1200 intervals.
The second says consolidate 12 samples (an hour) and keep 2400 of them (100d).
The third says consolidate 144 samples (12 hours) and keep 1200 of them (600d).
The '0.5' is how many datapoints can be missing before a sample is considered 'unknown' - if we use a step of 300s, then that means if we don't have a sample for 600s, it's 'unknown'.

So it goes a bit like this in perl:
sub create_routerstats_rrd
{

  my ( $filename ) = @_;
  return if ( -f $filename );

  RRDs::create ( $filename,
                 "-s 300",
                 "DS:TxPkts:COUNTER:600:0:U",
                 "DS:RxPkts:COUNTER:600:0:U",
                 "DS:CollPkts:COUNTER:600:0:U",
                 "DS:Txbps:GAUGE:600:0:U",
                 "DS:Rxbps:GAUGE:600:0:U",
                 "RRA:AVERAGE:0.5:1:1200",
                 "RRA:AVERAGE:0.5:12:2400",
                 "RRA:AVERAGE:0.5:144:1200",
  );
  if ( RRDs::error ) { print RRDs::error,"\n"; }
}


And because it makes sense to do so, I've put it in a subroutine.

Updating an RRD is _very_ simple - you feed in the values in the order specified, prefixed by a timestamp.
Almost invariably, your 'time' is 'now' so you can use 'N'.

  foreach my $row ( $table_extractor -> rows )
  {
    my ( $port, @data ) = @$row;
    my $rrd_name = ( "$data_dir/$port.rrd" );
    &create_routerstats_rrd($rrd_name);   
    #update the RRD. 
    RRDs::update ( $rrd_name, join ( ":","N", @data ) );
    if ( RRDs::error ) { print RRDs::error,"\n"; }
  }



... and that's it. You've now taken a single sample of your router stats.
Drawing the graphs and displaying them I'll leave for another day (if you're feeling enthusiastic, the answer is 'set up apache, and read the manpage for rrdgraph').

My code now looks like (in full):
#!/usr/bin/perl

use strict;
use warnings;
use LWP;

use RRDs;
use HTML::TableExtract;

my $router_user = "admin";
my $router_password = "sky";
my $stats_url = "http://192.168.0.1/sky_system.html";

my $data_dir = "/home/ed/sky_ro/data";

sub create_routerstats_rrd
{

  my ( $filename ) = @_;
  return if ( -f $filename );

  RRDs::create ( $filename,
                 "-s 300",
                 "DS:TxPkts:COUNTER:600:0:U",
                 "DS:RxPkts:COUNTER:600:0:U",
                 "DS:CollPkts:COUNTER:600:0:U",
                 "DS:Txbps:GAUGE:600:0:U",
                 "DS:Rxbps:GAUGE:600:0:U",
                 "RRA:AVERAGE:0.5:1:1200",
                 "RRA:AVERAGE:0.5:12:2400",
                 "RRA:AVERAGE:0.5:144:1200",
  );
  if ( RRDs::error ) { print RRDs::error,"\n"; }
}


sub collect_router_stats
{
  my $UserAgent = LWP::UserAgent -> new;
  my $request = HTTP::Request -> new ( GET => $stats_url );
     $request -> authorization_basic ( $router_user, $router_password );
  my $response = $UserAgent -> request ( $request );
  $response = $UserAgent -> request ( $request );
  print $response -> content();


  my $table_extractor = HTML::TableExtract -> new ( headers => [ ( "Port ", "TxPkts", "RxPkts ",
                                                         "Collision Pkts", "Tx b/s ", "Rx b/s ", )] );
  $table_extractor -> parse ( $response -> content );

  foreach my $table ( $table_extractor -> tables )
  {
    print "Table: ", join ( ",", $table -> coords ), "\n";
    foreach my $row ( $table -> rows )
    {
      print join ( ",", @$row, "\n" );
    }
  }

  foreach my $row ( $table_extractor -> rows )
  {
    print join ( ",", @$row ),"\n";
    my ( $port, @data ) = @$row;
    my $rrd_name = ( "$data_dir/$port.rrd" );
    &create_routerstats_rrd($rrd_name);
    print join(":", "N",@data ),"\n";
    RRDs::update ( $rrd_name, join ( ":","N", @data ) );
    if ( RRDs::error ) { print RRDs::error,"\n"; }
  }
}

&collect_router_stats();


And it's to be called via 'cron' at 5 minute intervals.


From:
Anonymous( )Anonymous This account has disabled anonymous posting.
OpenID( )OpenID You can comment on this post while signed in with an account from many other sites, once you have confirmed your email address. Sign in using OpenID.
User
Account name:
Password:
If you don't have an account you can create one now.
Subject:
HTML doesn't work in the subject.

Message:

 
Notice: This account is set to log the IP addresses of everyone who comments.
Links will be displayed as unclickable URLs to help prevent spam.

Profile

sobrique: (Default)
sobrique

December 2015

S M T W T F S
  12345
6789101112
13141516171819
20212223242526
2728 293031  

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Sep. 26th, 2017 07:18 am
Powered by Dreamwidth Studios