Aware: Wire Examples

This is a set of examples that should allow you to write custom Aware wire scripts that do what you need extremely quickly. Pick the example that most closely matches your requirements and modify it to your needs. Each example includes an html page to see alerts in a browser (see Reporting).

Discovery

The discover utility comes with the Aware package. It uses the core engine to ping entire subnets to collect all active IP addresses and optionally do tcp port scans as well as DNS name lookup. This is useful to really know what is on your network and can be used to build an initial list of hosts to configure for monitoring.

Since this program is built on the Aware core, it can scan very large numbers of hosts very quickly, allowing you to rapidly discover your network. For example:

Finally, here is another fun example that will discover information about all the SNMP agents (assumes you have an SNMP package installed):

For more sophisticated discovery, try nmap. This is a port scanner with many features.

The discover example includes a sophisticed "wire" file that discovers your network and stores the results in a MySQL database. It uses the discover program as well as snmpget to fill tables that describe the found interfaces, services and SNMP agents on the specified subnet.

Master/Slave

The awared daemon can be run in a master/slave mode. In this mode a master process is started on one machine which acts as the source for configuration files for the slaves. Slaves synchronize themselves with the master automatically.

For example, you might have a set of mail servers, web servers, dns servers, etc. Each of these groups of servers has its own unique aware configuration to monitor local processes, resources, etc. In a master/slave configuration each slave machine only has the award executables installed. When the awared process is started it is configured to get its "wire" file from the master. In this way, the sysadmin need only manage the configuration files in one place. Should a slave be unable to contact the master, it will continue to use its locally cached configuration file. You may keep many different configurations on the master, each slave will sync with its own.

Assume that you have a machine that will act as the master called aware.mydomain.net. Further assume that you have 4 wire configuration files, one for the master which does network service monitoring, while the other three do local monitoring for the mail,web and dns servers. These are kept in /aware/config and are called: master.wire, mail.wire, http.wire, and dns.wire.

If an awared is started in master mode, it starts an http server on the specified port. This server will serve any files in the specified "docroot" directory. You should put your "wire" files for the slaves in this directory and they will get automatically updated. In addition, this server can be as used as a reporting interface.

The Aware web interface uses wire scripts like the above to support slave clients reporting their events back to the central database.

Reporting Using Awared

Using distinct tags to report information allows you to customized the master's home page. This approach also allows you to integrate the Aware data into existing web pages on other servers (i.e., you already have a systems "status" web page that shows all your MRTG data, just put some Aware links into that page to see alert/event data from your Aware monitored systems).

Log File Reports

Every loghandler registers itself and makes its data available via tags. The following tags are available:

simple

The simple example uses a timer to generate events and a loghandler to receive the events and print them to the console:

//
// To run this with output to console only: awared -wire ./example.wire
//
// To run this with web interface: awared -log example.log -wire ./example.wire -master 4444 .
//
// Point your web browser at locahost:4444 (if you are on this machine) else <hostname>:4444
//
// Type: awared -help for other options
//
// NOTE: you need to have LD_LIBRARY_PATH include
// the directory holding the Aware libraries.
// For example: 
//   export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/aware/lib/linux2.4_i586
//

// create an event signatures
set interrupt create event { name: 'timer (1sec)' }

// create a handler to generate this event every second
set cycletime 1
create handler timer { interrupt: $interrupt cycletime: $cycletime }

// create a logger, this one uses the default output of the console (stderr)
set logconsole create hlogger { }

// create a logger that outputs to ./alarm and rotates the filename every hour
set logfile create hlogger { filename: alarm rotate: hour }

// create a log handler to receive the events
create handler log { logger: $logconsole regevent: $interrupt }

// create a log handler to receive the events
create handler log { logger: $logfile regevent: $interrupt }

/* the output to the console should look something like this:

Sun Feb  3 13:26:10 2002 Aware v0.1.1 Copyright (C) 1998-2002 Russell Leighton (russ@elegant-software.com)
Sun Feb  3 13:26:10 2002 Started monitor
Sun Feb  3 13:26:10 2002 id: 0x1 name: 'timer (1sec)' priority: 0 val: [Sun Feb  3 13:26:10 2002]
Sun Feb  3 13:26:11 2002 id: 0x1 name: 'timer (1sec)' priority: 0 val: [Sun Feb  3 13:26:11 2002]
Sun Feb  3 13:26:12 2002 id: 0x1 name: 'timer (1sec)' priority: 0 val: [Sun Feb  3 13:26:12 2002]
Sun Feb  3 13:26:14 2002 id: 0x1 name: 'timer (1sec)' priority: 0 val: [Sun Feb  3 13:26:14 2002]
Sun Feb  3 13:26:15 2002 id: 0x1 name: 'timer (1sec)' priority: 0 val: [Sun Feb  3 13:26:15 2002]
Sun Feb  3 13:26:15 2002 Received signal, stopping.
Sun Feb  3 13:26:15 2002 Stopping monitor...
Sun Feb  3 13:26:15 2002 Stopped

*/

http

Http is an example of monitoring a web server. It sets up a handler that monitors for connection errors as well as 404 errors. Email alerts are generated using a limiter. It uses a fast cycletime: for illustration purposes so that you can quickly generate alerts and log entries.

//
// To run this: awared -wire ./example.wire
//
// To run this with web interface: awared -log example.log -wire
./example.wire -master 4444 .
//
// Point your web browser at locahost:4444 (if you are on this machine) else
<hostname>:4444
//
// Type: awared -help for other options
//
// NOTE: you need to have LD_LIBRARY_PATH include
// the directory holding the Aware libraries.
// For example: 
//   export
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/aware/lib/linux2.4_i586
//

// change this to be your email address
set myemail russ

// create an event signatures
set connect create event { name: "connect"  priority: 1 }
set noconnect create event { name: "noconnect"  priority: 1 }
set four-oh-four create event { name: "File not found"  priority: 1 }
list events $connect $noconnect $four-oh-four 

create probe http { 

 url: http://www.google.com
 // uncomment to use bogus URL to generate 404s (comment out above url)
 // url: http://www.google.com/foobar

 // comment to NOT see connect events
 connect: $connect

 noconnect: $noconnect

 match: $four-oh-four "" "HTTP/.* 404" "foobar page is missing!"

 timeout: 5

 // probe every 5sec NOTE: this is just as an example, probably 60sec is 
 // more reasonable if you are monitoring your web server
 cycletime: 5

}

// log it to stdout
set consolelogger create hlogger { }
create handler log { logger: $consolelogger regevent: $events }

// log it to file
set filelogger create hlogger { filename: alarm rotate: hour }
create handler log { format: "\$n\t\$v" logger: $filelogger regevent:
$events }

// email alerts for 404 and noconnect errors 
//
// NOTICE:
//          we use \$n which will expand to the event's name
//          we use \$v which will expand to the event's value
//          we use \$t which will expand to the event's timestamp
//          we use $myemail which will expand to the string you set for
myemail
//
create handler execp { 
        cmd: "mail -s \"alert: \$n\" $myemail"
        input: "Web server error:\n\t\$n:\$v at \$t\n\n"
        regevent: $noconnect $four-oh-four 
}




/* the output to the console should look something like this:

[russ http]$ awared -wire example.wire
Thu Jul  3 10:46:32 2003 Aware v0.10.1 Copyright (C) 1998-2003 Russell
Leighton (russ@elegant-software.com)
Thu Jul  3 10:46:32 2003 Started monitor
Thu Jul  3 10:46:35 2003 id: 0x801 name: 'connect' source: '' priority: 1
val: 'http://www.google.com'
Thu Jul  3 10:46:51 2003 Received signal, stopping.
Thu Jul  3 10:46:51 2003 Stopping monitor...
Thu Jul  3 10:46:52 2003 Stopped
[russ http]$

*/

/* ...and if you changed it to generate 404s by using a bogus URL, the
output to the console should look something like this:

[russ http]$ awared -wire example.wire
Thu Jul  3 10:47:32 2003 Aware v0.10.1 Copyright (C) 1998-2003 Russell
Leighton (russ@elegant-software.com)
Thu Jul  3 10:47:32 2003 Started monitor
Thu Jul  3 10:47:35 2003 id: 0x801 name: 'connect' source: '' priority: 1
val: 'http://www.google.com/foobar'
Thu Jul  3 10:47:35 2003 id: 0x803 name: 'File not found' source: ''
priority: 1 val: 'foobar page is missing!'
Thu Jul  3 10:47:40 2003 id: 0x803 name: 'File not found' source: ''
priority: 1 val: 'foobar page is missing!'
Thu Jul  3 10:47:46 2003 id: 0x803 name: 'File not found' source: ''
priority: 1 val: 'foobar page is missing!'
Thu Jul  3 10:47:50 2003 Received signal, stopping.
Thu Jul  3 10:47:50 2003 Stopping monitor...
Thu Jul  3 10:47:50 2003 Stopped
[russ http]$


*/

local

The local example uses all the local handlers to monitor the machine on which the awared process is running. This is a realistic example of what should be monitored on a well maintained system.

All events are logged and those that are metric oriented are stored in Round Robin Databases (RRDs) and graphs are generated in the web interface.

network

The network example uses a script called example.sh to create a wire file using all the network availabilty handlers. This a realistic example of using Aware wire scripts as a network monitoring system (NOTE: the web user interface uses wire scripts to implement network monitoring in a similar manner):

MRTG Stylee

MRTG is a very popular traffic monitoring tool. The mrtg_stylee example shows how you can create similar graphs using Aware with the rrdhandler, an interface to rrdtool. Note: this example requires that you have a working snmpget utility. One can be found at: http://net-snmp.sourceforge.net/.

Inserting Into and Querying a Database

The database_insert example illustrates using the MySQL handler. Although this example uses MySQL, it will work with the other supported databases with minor modifications.

Aware allows you to set up a handler that receives events and insert them into a database. This is very useful to log events into a mangement or application database.

In this example a timer handler is used as an event source, in a more realistic example you would have a database handler register to receive events that contain statistics and alarms for insertion into the database. The database handler inserts a row into the database for every event received, doing a macro expansion on the sql string to subsitute event information into the insert statement.

A mysql handler is also created to illustrate how you can use Aware to poll database tables and generate events based on rows returned. This example has the handler query the same table into which we are inserting events, processing the rows inserted in the last five seconds. A more realistic example might have a handler watch an application error table or execute a periodic query to gather statistics derived from the database. Events generate from these queries can be used to trigger handlers to generate alarms, execute external programs or compute statistics (and perhaps insert results back into the database).

Should the handler not be able to connect, a 'noconnect' event may be generated. This illustrates how you can use a database handler to do service checks.

Monitoring a Database

Proactively monitoring your database is essential for keeping a healthy system. By keeping historical data you can anticipate problems before they happen. This example queries a MySQL database for system level parameters, keeping the data in a round robin database and plotting the results. Other databases have similar metrics available. It is recommended you keep multiple timescales (e.g., hourly, daily, weekly,monthly, yearly) so that you can have good historical baselines of system behavior. In addition, if there are related machine/applications (e.g, web servers) you can combine the monitoring plots on one screen so you can visually correllate behavior. You might have a plot of network traffic, http response time for the web server for some important pages of the web application, the ping response time of the web server, the cpu load, memory usage, followed by the database metrics; thus giving you an end-to-end picture of the state of the system over time. The below image is from the example mysqlmon, which is an example of monitoring a MySQL database:

Inside the mysqlmon example is a wire file called sqlserver.wire which is an example wire file to monitor a SQLServer database. In this example a ping handler and an application query are timed, in addition to the database performance metrics. It is good practice to instrument specific application queries that characterize the application performance.

Monitoring Email Message Flow

The email example will test email connectivity using the 'mail' and 'fetchmail' tools and a MySQL database. It will send and recieve email, then generate alarms should email not be received in a timely fashion.

UPNP

Upnp is a simple example of monitoring your Universal Plug-n-Play devices. The UPnP handler will keep track of UPnP services as the become available and unavailable. The cycletime: refers to how often an M-SEARCH message is sent out requesting a response from ALL devices. Since this can cause a burst of network traffic it is recommended that you do this infrequently (e.g., 2 times a day). The handler listens for 'alive' and 'byebye' messages from devices and will immediately respond if you enable the associated event so there is little penalty for using an infrequent cycletime. The only concern is that since UPnP uses an unreliable protocol it is possible to miss some events. If you are on a relatively clean network this should not be an issue and an infrequent cycletime: is desirable.

Special NOTE: to see a result quickly you need to restart a UPnP device after starting this example running. If you have a Windows XP box on the network that has Internet Connection Sharing, reboot it after starting this example running and you should see events on the console.

Snort

The snort example requires snort to be running. It will monitor /var/log/messages for snort alerts and process them.

In this example, the snort file classifications.config is processed by a shell script and is used to configure a logfile handler to filter for snort events by each classification. An rfilterhandler is used to break the event stream into low,mid and high priority events. Low priority events are logged into a file for low priority events, while high priority are logged into a file for high priority events as well as generating alerts. Mid priority events are logged into a file for mid priority events, in addition these middle level alerts are filtered by a rate limiter such that a burst will trigger an alert.

This example illustrates the use of limiters. The limiter is used avoid large numbers of alerts as well as to detect and alert on bursts of suspicious mid priority events. A burst of the same kind of mid priority events in a short period of time may indicate a security issue, while small numbers of mid priority events spread over time are of less concern.

Multicast File Copy

In the multicastcp example the mctx handler will broadcast a file to the specified multicast address. The mcrx handler will receive the file.

This design is intended to run on LANs and typically used to share files on a cluster/grid.

The design assumes a CLEAN network, where dropped packets are very unusual. Although this program will re-send dropped packets, the performance will be adversley affected. In this design a dropped packet is considered an error. If you do see many dropped packets you can lower the tranmit rate and/or troubleshoot your network.

//
// NOTE: run this on a DIFFERENT machine from the receiver OR in a different directory
// to avoid overwritting file
//
// To run this with output to console only: awared -wire ./mctx.wire
//
// Type: awared -help for other options
//
// NOTE: you need to have LD_LIBRARY_PATH include
// the directory holding the Aware libraries.
// For example: 
//   export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/aware/lib/linux2.4_i586
//

set hlogger create hlogger { }

set start create event { name: "start"  priority: 1 }
set stop create event { name: "stop"  priority: 1 }
set sent create event { name: "sent"  priority: 1 }
set error create event { name: "error"  priority: 1 }

create handler mctx { 
        filename: foo.txt
        mcloopback:
        start: $start
        stop: $stop 
        sent: $sent 
        error: $error 
        cycletime: 5
        elogger: $hlogger 
}

create handler log { 
        logger: $hlogger 
        regevent: 
        $start
        $stop
        $sent
        $error
        elogger: $hlogger 
}

//
// NOTE: run this on a DIFFERENT machine from the sender OR in a different directory
// to avoid overwritting file
//
// To run this with output to console only: awared -wire ./mcrx.wire
//
// Type: awared -help for other options
//
// NOTE: you need to have LD_LIBRARY_PATH include
// the directory holding the Aware libraries.
// For example: 
//   export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/aware/lib/linux2.4_i586
//

set hlogger create hlogger { }

set start create event { name: "start"  priority: 1 }
set stop create event { name: "stop"  priority: 1 }
set received create event { name: "received"  priority: 1 }
set error create event { name: "error"  priority: 1 }

create handler mcrx { 
        start: $start
        stop: $stop 
        received: $received 
        error: $error 
        elogger: $hlogger 
}

create handler log { 
        logger: $hlogger 
        regevent: 
        $start
        $stop
        $received
        $error
        elogger: $hlogger 
}