Basic Installation and use.
===========================

1) Unpack the archive
   tar xzf suckmt.<VERSION>.tar.gz

2) Build and install
   cd suckmt.<VERSION>
   make install
   A working config file (/etc/suckmt.ini) has now been created or 
   your existing /etc/suckmt.ini has been backedup and updated.

   With a new version of suckmt it is very likely new settings have been added.

   A suckmt.ini.sample is included to get an idea on what a running suckmt.ini
   file can look like. This file can be found in /usr/doc/packages/suckmt

3) Add the news group(s) you want.
   You can just add them with a normal text editor or you can do 
   suckmt -i /etc/suckmt.ini -n nl.comp.os.linux

4) Manually edit the settings in the "Suck MT Configuration" 
   section of the suckmt.ini file. I tried to make the config 
   file as self-documenting as possible so it shouldn't be a 
   problem.

   -----------
        News server Hostname            = news.wirehub.nl
        
   This is the hostname of the server you're downloading from.

   -----------
        Storage Directory               = /etc/ppp/news/Msgs
        Batch File                      = /etc/ppp/news/batch

    The "Storage Directory" must be an existing directory that is used to 
    store the downloaded messages into. The filename "Batch File" is the 
    file that is created with all the names of the downloaded files.

   -----------
        Number of threads               = 5

    This is the number of connections that are opened to the server.
    I recommend you experiment with the "Number of threads" value .
    5 seems to work best in my situation.

   -----------
        Connect with Port Number        = 119
        
    This is the portnumber on which the server listens.
    Recommended : DON'T CHANGE IT.
        
   -----------
        Connect with Username           = 
        Connect with Password           = 

    Use these values if you need to connect to the news server with 
    a username and a password.

   -----------
        Debug SOCKET commands           = False

    Setting this to True let's you see a LOT of what is communicated 
    between suckmt and the server. If you turn this on I recommend you 
    set the number of threads to 1 .

   -----------
        Send MODE READER to the server  = True

    Sending the MODE READER command to the server is required for 
    some servers and is considered to be good practise.

   -----------
         Restart File  = /tmp/suckmtrestart
         
    This is the name of the temporary file that is used to restart 
    when suckmt was stopped before all downloads were completed.
         
   -----------
         Logging amount [0(nothing)..7(noise)]  = 2

    This determines how much information is shown on the screen during
    the execution of suckmt. 
    0 means that (almost) nothing is shown. 
    7 means that a LOT of noise about a lot of steps is shown on the screen.
    The default value 2 show a usefull amount of information.
    Note that some of the logged information is also sent to the syslogd.

7) To actually use suckmt just run it
   suckmt -i /etc/suckmt.ini 
   and everything is handled automagically.

   If suckmt is successfull it returns 0, in case of error it will return 
   something else.
      
   If and error situation occurs the suckmt.ini file will NOT be updated.
   You can however safely post the downloaded files to your news server and 
   remove them from the "Storage Directory". The next time suckmt is run 
   the "Restart File" will be used to determine which messages have already 
   been download and can be skipped for that reason.
   
8) Add some kill/keep rules (see further below)


Content of the suckmt.ini file
==============================

The suckmt.ini has been created as a simple general config format for all 
settings relating to suckmt in a human edditable format.

Generic File format
===================

The file consists of sections. The start of the section foo is indicated 
by a line with [foo]. Each section consists of a group of settings. Each 
setting is a name and a value with an '=' in between. Both the name and the 
value are free text as long as it is on 1 line and the name cannot contain an '='
Each name must be unique in the section. If duplicate names are found the last 
one will overwrite the first one.

IMPORTANT: All names are CASE SENSITIVE !!

Sections can be split in muliple part but they will be rejoined automagically.

For example
[foo]
Harry = Man
Jill = !Man

[bar]
Harry = Man
Jill = !Man

[foo]
William = Man


will be automagically rewritten into

[foo]
Harry    = Man
Jill     = !Man
William  = Man

[bar]
Harry  = Man
Jill   = !Man

This feature can be used to update the ini file by just appending a few 
lines using an extremely simple shell script. If in the above example I 
would want to force a new value for Jill in section foo I could do this 
by just appending the following two lines to the file:
[foo]
Jill = Woman

The next time the application is run this new value will be automagically 
intergrated into the existing parts of the config file.

Used contents of suckmt.ini
===========================
You can put a lot of settings into the suckmt.ini that are not used by the 
software. I will only describe the actually used values. Please examine the
suckmt.ini.sample file to get an idea how to store the information.

[Suck MT Copyright Information] 
    contains some unimportant general information about suckmt.

[Suck MT Installation] 
    contains some unimportant statistics about how often you have run suckmt. 

[Suck MT Configuration] 
    contains the connection and storage information you'll know what they 
    mean when you see them.

[News Groups] 
    is a list of all the newsgroups that should be downloaded. The value that 
    is stored is the article number where to start the next download.

[Kill Headers] 
    are the headers that are cause to kill a message. Each line consists of 
    5 items:
    - The header name
    - The value to look for (this is a regular expression)
    - The date when the last ocurrence was found
    - The number of matches that have been found till now
    - The weight of this rule

[Keep Headers] 
    are the headers that are cause to keep a message even if a kill match 
    was found. The format and use of these line are identical to the Kill 
    Headers. Read the next section on how the weights of rules affect the
    outcome of the kill/keep action.


USING KILL AND KEEP RULES
=========================

  GLOBAL KILL RULES
  =================

[Global Kill Rules]
Minimum Lines in a Message   =    -1 ;   100
Maximum Lines in a Message   =   300 ;   100
Maximum Bytes in a Message   = 15000 ;   100
Maximum Groups in a Message  =     6 ;   100

With these killrules the value -1 means that this rule is not applied.
Any other value means what is clearly stated in the name of the rule.
The second value on each line is the weight of this rule. 
The default weight value is 100 but can be changed manually.
This weight value will be explained later.

Maximum messages per newsgroup threshold     = 1000
Only download N messages when over threshold = 50

When I return from a few weeks of vacation I don't want to spend too much
time downloading all the messages from that period. These two settings 
determine what to do when a group has more messages than I care to read or
download. The above example values will make sure that only 50 messages are 
downloaded in each group when the total number of messages exceeds 1000. 
So if there are 950 messages they will still all be downloaded, if there 
are 1001 then only the last 50 will be downloaded. This should probably be 
a per group setting but that isn't possible yet.
With these killrules the value -1 also means that this rule is not applied.


  KILL AND KEEP BASED ON HEADERS 
  ==============================

You can kill based on ANY header field of a message.
To do this you add a line with

headername:value

in the [Kill Headers] section of the suckmt.ini. 
The section [Keep Headers] is the section of the headers that are used to 
keep messages. The way this is used is identical to the kill headers.

If a message contains a header with the headername and the specified value
occurs somewhere in the actual value of the header the message will be
killed. The value is always interpreted as a regular expression. Note that 
all strings are matched case-sensitive and leading and trailing spaces and 
tabs are ignored !!!
Example:

    Content-Type:text/html
or 
    Subject:over[ck]lo(k|ck)

If you want the line to be matched case-IN-sensitive put a ~ as the 
first character of the line.
Example:

    ~Subject:over[ck]lo(k|ck)

After you run suckmt this line will have been changed into something like
this:

    ~Subject:over[ck]lo(k|ck)  = Thu Jan  6 19:49:06 2000 ;     3 ;   100

The date is the start time of the suckmt run when this header was found. 
The number after the first ';' (in this example 3) is the number of 
occurrances this rule has been applied since it was put in the suckmt.ini 
file. Using this information (the date and the number) you can manually inspect 
the suckmt.ini file and you can see how often a kill rule has been applied 
and when the last occurrance was. 

The second number is the weight of this rule. 
The default weight value is 100 but can be changed manually.
This weight value will be explained later.

  ARTICLE BODY 
  ============
  You can also kill/keep if a specific value occurs in the Article Body.
  To do this you can specify the "header" field called ArticleBody.
  This rule will ONLY be matched if the entire article has been downloaded 
  but can still be usefull to reduce the amount of junk that you see in 
  some newsgroups. 

  Example (this one kills articles that refer to a numerical URL):
    ~ArticleBody:http://[0-9]*/ = Wed Mar 01 22:06:37 2000 ;     0 ;   100

  The weight of this rule will be added to the total weight of all previously 
  matched headers. So if you have a keep rule that sets the cumulative weight 
  of a message to 200 and you match the above rule the message will still be 
  kept because the total weight will be 100.
  
  It is important to note that using this feature will probably cause a 
  performance impact because 
    1) This match can only be done AFTER download, 
    2) all messages have a body so all messages will be checked, 
    3) the size of the "header" is significantly larger than the other headers.

  Note that if you set the "Enable Kill Logfile" to true you will get the 
  entire message in the logfile if a match was found (because that is the 
  value that the expression matched against).

  KILL KEEP WEIGHT VALUE
  ======================

For all messages the following is done:
1)  Download the header based on the line from the XOVER command.
2)  Check the lines, bytes and groups against the ALL kill rules in the 
    "Global Kill Rules" section. 
3)  Check the headers against ALL kill and keep rules.
4)  The matched kill weights are substracted from the matched keep weights.
    If the resulting number is negative the message is killed.
    If the resulting number is 0 or larger the message is NOT killed.
5)  The body is downloaded
6)  Check the body against the ArticleBody kill and keep rules.
    If the total from 3) and 6) is negative the message is killed.
    If the total from 3) and 6) is 0 or larger the message is NOT killed.
7)  Write the message to disk.
