squidview 0.36 (c) 2001, 2002, 2003 Graeme Sheppard - GPL software
www.rillion.net/squidview

--Overview

Squidview is a program meant to display the squid proxy server log
file in a nice fashion, providing the log file is in squid's
native reporting fashion. It has features such as search, report
generation, monitor mode and supports three log files.

Thus, the program can be used to monitor Internet usage on a
networked site. But please note squid has to be running first and
this program is not a proactive resource controller. What it can
do is tell you who and which sites are consuming the most bandwidth.

--Compiling squidview

  $ make

The program is c/c++ and uses the ncurses library for obtaining
some keystrokes. Old gcc compilers can fail compiling this
program, so if so:

  $ make oldgcc

--Installation

For computers with "install" available:

  # make install

Otherwise...

  # make install-all
  
Or if you aren't the superuser the program will still run if
you have read access to the squid log file.

This program assumes the main squid log file is here:
  /usr/local/squid/var/logs/access.log
If it isn't just symlink it appropriately from the following
default. This directory and links are made in the user's home
space when first run by the user.

  graeme@localhost:~/.squidview$ ls -l
  README -> /usr/share/doc/squidview/README
  log1 -> /usr/local/squid/var/logs/access.log
  log2 -> log1
  log3 -> log1
  users
  words

In this case squidview expects the primary log to be "log1",
and the symlink redirects it to where the file actually is.
The same thing applies to the readme.

This tar.gz archive comes with an example access.log named
example.log. Symlink it to view it with squidview.

"users" and "words" are reporting files - these can be symlinks
to /etc if you would rather have these files there. "aliases" is
another text file, this time for mapping IP numbers to user-
defined names.

This package comes with example "users", "words" and "aliases"
files you can build on.

--Using squidview

Squidview shows each proxy request on one line starting with the
user's name (ie the name of the user on the client machine as
reported by identd or similar), flags and then the target (ie
destination) of the request.

Should an identd process not be running on the client, squidview
can display the client IP address instead of "-", or if you are
using an aliases file it can get a name from that. See Reports
about this.

The target bit is truncated if necessary so as to fit the
information on one line. There are two methods of truncation
(discussed in Reports.)

Change the selected line with the cursor control keys or the number
keypad. Down the bottom, on the status bar, is the time the selected
request was made as well how far through the log it is (as a
percentage: 0% top, 100% end). Press 'h' to get some help or 'r'
for this readme file.

Also on the status bar will be "Mon Pri". "Mon" means the program
will update the screen if new proxy requests are made. Toggle this
off by pressing 'm' if you want to remain on a selected line.
"Pri" is the primary log file, to switch to another log file press
the appropriate key (press 'h' for keys.)

--Flags

Between the user and URL columns is the flags field. For example
it may be "w2Rf":

'w': a word match on the URL (see Searches below)
'2': bytes transferred was between 0.25MB and 1MB
'R': the request was a cache refresh hit
'f': part of the current focus

For a bit of help about these flags select the relevant line and
press 'v'.

--Searching

A search forward is made by the right arrow key, backwards is
handled by the left arrow key. But first you need something to
search for. Press 'f' to find a piece of text. Both user names and
http/ftp addresses can cause a match. Request lines which match
are noted by a 'w' in the flags column.

Multiple search strings are possible, and are necessary when you
use skips. Skips tell squidview not to match some requests, such
as when the word "sex" is searched for but not when the target is
on doubleclick.net (that advertisement server likes to use the word
"sex" in URLs.) The following will accomplish that:

!doubleclick.net
sex

The requests with "doubleclick.net" in them will be skipped
because that piece of text is first and it is preceded with an
"!". In doubleclick.net cases the flags column will have a
'-' where the word match would have been.

Note that text you enter with 'f' is placed at the top of the
search list so it has priority. Using 'F' (capital F) will add
search text to the bottom.

Your search words can be saved from the search options menu, and
you can do some other things there: pick up large requests and
focus on a particular user. These two can be turned off when not
needed.

--Navigating the log file

As well as jumping to the beginning or end of the log file, you
can go to a certain percentage through with 'g', or to the
beginning of a certain day with 'j' or 'J'. Of course 'home' and
'end' work too, if you are using a remote shell and they don't,
press '7' or '1' respectfully (look at your number keypad.)

--User lookup

On a selected line you can press 'v' to get a verbose description
of it - this is actually a dump of the line to the screen.
Squidview will try to match the user to those known in a file
called "users", displaying that line in the file. For example
the "users" file could have in it:-

root system administrator

The first word on each line must be the login name with no spaces
between it and the real name.

--Common options

By default if no login name is available the client's IP number is
displayed instead. Change this with "ip instead of null user" to
get, instead, reports of bandwidth attributed to "-". The aliases
file is another option here. You might specify that 192.168.0.15
be displayed as "server4". You need to enable this one because
it's off by default.

Keeping the filename of target also affects the main window. When
on, the target URL is shifted left - but not over the domain - so that
the type of file can be seen on one line. Otherwise the line is
simply truncated to be displayed.

--Log a report

Make a text or CSV report of search hits. A few options are there.
You will need to specify a report file name to view the details.
Otherwise you will just get a summary. Reports are placed in ~/.squidview.

To start or finish the report at a particular point in the log file
highlight the line in the main window, press 'l' and then either
'a' or 'b'. Press 'a' or 'b' again to toggle it.

User bandwidth totals can be calculated. The options are search hits
or "not veto" (which is mostly everything excluding skips mentioned
above.) These are sorted so you can find the heavy internet users.

Bandwidth totals will find the most popular sites for you. This can
be done for one user specifically or for all users as a whole.

When a word hit is detected it can be written in the report (eg
"word hit action: normal text").
Text reports are good for viewing with "less"; CSV ones are intended
for spreadsheets.

In the case of normal text reports you may or may not want to see
the request size. This information takes up a column. Splitting long
lines will show the details on more than one line if need be. Then
again, you may only want all "hits" to be shown on just one line.

The other options are straight forward. Be a bit picky about the CSV
field separator - they put just about any characters in URLs. Try a
tab (yes, just press 'tab', 'enter') or "*".

To get a summary report about a particular user, say "graeme", do
this:
- in search options focus on "graeme"
- go to log a report
- select "domain bandwidth totals"
- select "only focus user graeme"
- and press enter on the previous screen where it says go

You will need to unset the above options for reports to come
out normally again, and for searches (cursor <-, ->) too.

--Filtered reports

After making a general report it is possible to filter it for
just one user. That way you don't have to rescan the log file
with a focus. The downside of this is the target totals the
user surfed to can't be calculated.

--Bugs

On my system the program can segfault when a report is being
generated under certain conditions. This happens when the c++
STL sort function is called too often. Sorting is used to
speed up access to the tables and reduce the time you have
to wait for a report to complete.

The problem doesn't happen if my sort routine is used instead
of the STL one, so mine is used by default. However I get a
20% performance penalty doing it my way.

There are two variables in squidview.conf which direct
sorting operations. "bRepMySort" can be set to "yes" to use
my sort routines, or "no" otherwise. "iRepFast" is an integer.
After this many lines of processing a sort is made. Therefore
the smaller this number the more often sorts are made. I've
settled on a 10000 value for this; your mileage may vary.
Making "iRepFast" 0 disables sorting completely.

Squidview catches segfaults and will tell you if it happened during
the STL sort. But I've noticed STL sorts can cause faults in other
STL routines outside of the sort. Strange.

By all means play around with this and tell me the results of
your findings. I want the bug squashed.

--Author

Graeme Sheppard <graeme@rillion.net> - happy for your feedback.
http://www.rillion.net/squidview
