$Id: testing-design.txt,v 1.4 2001/09/12 21:17:15 mulix Exp $

A Comprehensive Test Plan for the "syscalltrack" Project
--------------------------------------------------------

TOC

1. Testing Overview
1.1. Kernel Module (st_rules_module) Tests
1.2. Rules Library (st_rules) Tests
1.3. Configuration Utility (sct_config) Tests

2. Testing Framework
2.1 Anatomy of a Test Unit

3. Test Suggestion
3.1. Kernel Module Test Suggestions
3.2. Rules Engine Test Suggestions
3.3. Configuration Utility Test Suggestions

1. Testing Overview
------------------

This section describes the various kinds of tests we want to perform
and their goals. Tests implementations are discussed in section 2. 

1.1. Kernel Module Tests
------------------------

(note: although there are actually two kernel modules, we treat them
here as one functional unit). 

The main goal of the kernel module tests is to establish the module's
stability. A secondary goal is to establish correctness. 

The tests must cover all of the module's functionality and
preferably, all of its code paths. The major areas are:

system call invocation
       stab function
       filter evaluation
       carry out action
       reentrancy (with the same syscall invocation and with other
           syscalls). 

rule registration/deregistration
       rule registration
       syscall hijacking
       rule deregistration
       syscall releasing

generic kernel functionality
       module loading and unloading 
       sysctl interface

1.2 Rules Engine Tests
------------------

The main goal for testing the rules engine are stability and
robustness. Secondary goals include correctness and memory
consumption. 

The tests should cover all of the code paths and every area of
functionality. These include:

primitive types creation/destruction
       handling of illegal input 
       handling of very long or NULL strings

filter evaluation
       resistance to illegal input (ill defined individual filters,
	   matching of wrong types, invalid operations on types, wrong
	   filter trees ['1 1 +' instead of '1 + 1'])   
       
serialization and deserialization       
       resistance to errors in input (invalid input)	
       resistance to errors during serialization (out of memory)

memory consumption
       need to check that we don't leak memory
       
1.3. Configuration Utility Tests
--------------------------------

The main goal when testing the configuration utility is that it builds
correct 'tracking rules'. A secondary goal is robustness against
invalid configuration files. 

The tests should cover the following areas:
   
invalid configuration syntax
      small errors (missing brackets, invalid keywords)
      big structural errors (nested rules, nested filters, top level
	  filters)

invalid configuration semantics
      missing required keywords
      invalid system calls or parameters
      wrong parameter data types


2. Testing Framework
--------------------

The testing framework is a script used to run the all of the different
tests and report to the user their conclusions. It should have the
following features:

It should be modular, so that new tests can be added easily. 
It should catch all test output to avoid clutter and only present it
to the user if the test failed. 
It should be able to set a time limit on tests (to catch infinite
loops and deadlocks)
It should be able to insert rules to the module prior to a test and
remove them after the test is done. 
It should provide the means for a test that signal that it depends on
another test and must be run while the other test is
running. Likewise, it should be possible for a test to specify that it
must be run only while no other test is running. 

2.1 Anatomy of a Test Unit
--------------------------

A Test Unit is a single test, either an executable file or a script,
excersizing one of the afore mentioned areas of functionality. It
might be linked with the code it is testing, or it might communicate
with it via some other means. 

It will be launched by the testing
framework, receiving its input (if any) from stdin and sending its
output (if any) to stdout and stderr. It should return 0 to signal
success, 1 to signal failure. If it failed, it should return as much
diagnostic information as possible to ascertain where it failed. 

It should indicate to the testing framework what rules it needs in the
module before it starts running, so that the testing framework could
set those rules before the test begins and remove them afterwards. 

3. Test Suggestions
-------------------

Here are some tests we could implement to excersize the various areas
of functionality. This list is by no means complete!

3.1 Kernel Module Test Suggestions
----------------------------------

- two processes sleeping for very short times. when either wakes up,
it performs one of the known system calls, with random parameters
(some of which have filters, some of which dont). this test excersizes
syscall invocation and reentrancy. 

- two processes sleeping for very short times. when either wakes up,
it tries to add/delete rules at random. this test excersizes rule
addition and deletion. 

- a script that launches one (or both) of the previous tests, and
proceeds to try and remove the module while they are running. once
removing is succesfull, it should try to load the module while they
are running. 

3.2 Rules Engine Test Suggestions
---------------------------------

- a test that creates various filter types at random, combines them
and feeds them for evaluation. 

- a test that build some legal (!) filters and check that they are
evaluated correctly. 

- a test the builds filters, evaluates them, serializes and
deserializes them again and then evaluates them again. 

3.3 Configuration Utility Testing
---------------------------------

- a script that creates various (legal) configurations and feeds them
to the configuration utility, comparing the resulting rules with rules
constructed directly using the rules engine. 

- a script that creates random "garbage" made of tokens the
configuration utility knows and feeds them to it. human intervention
will be required to make sure that configurations that are flagged as
"correct" are indeed correct. 





	
       
