Hi all Monet and DS hackers,

here comes the new version of the testing environment. Only a few details
have changed since my proposal.


- The testing environment consists of the following tools:
  + Mtest.py	a python script to run tests
  + Mprofile.py	a python script to run tests and collect profiling information
		(see below)
  + Mfilter.py	a python script to filter the test output before running Mdiff 
		(see below)
  + Mdiff	a C program to show the differences of two files as HTML document
  + Mapprove.py	a python script to approve recent test output (see below)
  + Mtimeout	a C program to limit the resouces of a program
		(thanks to Tim Ruhl, tim@ddi.nl)
  + MkillUsers	a bash script to kill test processes that are orphaned
		(used "fuser" to detect such processes)
  (Mtimeout and MkillUsers are currently available on Unix, only)


- In each directory of the current Monet and/or DS source tree tests may be
  provided in a subdirectory called "Tests". Of course, the tests should
  deal with the part/modules where they are located and they should be
  provided and maintained by the respective developer (see also below).


- Each "Tests" directory must contain a file "All" that contains the names
  of all tests in that directory (one name per line).


- Each test named TST consists of
  + a test script which is ONE of the following:
    * an arbitrary executable (e.g., a shell script)        (TST)           (~,^)
       on Windows, executables must be ".(exe|com)"         (TST.(exe|com)) (~,^)
                   and scripts must be ".(bat|cmd)"         (TST.(bat|cmd)) (~,^)
    * a Python script                                       (TST.py)        (~,^)
    * a MIL script to be executed by Mserver                (TST.milM) (',^)
    * a MIL script to be executed by Mserver -single        (TST.milS) (',^)
       (-single has been removed, so .milM & .milS are
        treated equally, now)
    * a MIL script to be executed by MapiClient -lmil       (TST.milC) (",^)
    * a SQL script to be executed by MapiClient -lsql       (TST.sql)  (",^)
    * a XQUERY script to be executed by MapiClient -lxquery (TST.xq)   (",^)
   (~) In case the name of an arbitrary executable or Python script is
        suffixed with ".MAPI" (i.e., TST.MAPI[.(exe|com|bat|cmd|py)]),
        Mtest.py starts an Mserver with a MAPI-listerner on MAPIPORT
        in the background before executing the test. The test can then
        connect via MAPI (on MAPIPORT) to that Mserver.
       Similar, with suffix ".SQL", an Mserver with an SQL-listener on
        SQLPORT is started;
       and, with suffix ".XQUERY", an Mserver with an XQUERY-listener on
        XQUERYPORT is started.
       In both cases, Mtest.py stops the Mserver again, once the test has
        finished.
   (') For Mserver, if several files are present named
        TST_sXX.mil(M|S) (XX={00,01,...,99})
        these are executed by subsequently calling Mserver
   (") For MapiClient, if several files are present named
        (1) TST_sXX.(milC|sql|xq) (XX={00,01,...,99})
            these are executed subsequently using the same Mserver
        (2) TST_pXX.(milC|sql|xq) (XX={00,01,...,99})
            these are executed cuncurrently using the same Mserver
   (^) For each test file TST[.*] involved,
        if a file called
	TST[.*].src  exists instead of  TST[.*],
	TST[.*].src  is expected to contain a single line
	giving the original location of the test file to be used.
	E.g.,  src/modules/plain/Tests/arith.milM.src  contains
		$TSTBLDBASE/$TSTDIR/arith.mil
	in order to use the MIL script extracted from
	src/modules/plain/arith.mx  as test script.
  + a MIL script to be used as prelude for Mserver
     (optional)                                             (TST.prelude)  (^)
  + for MapiClient tests, a MIL script to be used as
     prologue for Mserver, i.e., a script that is executed by Mserver after
     mapi_start()/sql_server_start()/pfstart() is called, but before (the first)
     MapiClient connects to Mserver
     (optional)                                             (TST.prologue) (^)
  + for MapiClient tests, a MIL script to be used as
     epilogue for Mserver, i.e., a script that is executed by Mserver after
     (all) MapiClient(s) have finished
     (optional)                                             (TST.epilogue) (^)
  + a file that contains a list of modules (one per line)
     required by the test  (optional)                       (TST.modules)  (^)
  + a file that contains a list of BATs (one per line)
     (created by previous tests within the same directory)
     required by the test  (optional)                       (TST.BATs)     (^)
  + a set of files used by the arbitrary executable
     (optinal)                                              (TST.*)        (^)
  + stable (i.e. correct) versions of stdout and stderr
     of the test                                            (TST.stable.{out,err})
    operating system (OS) specific stable output can be 
    provided by adding the suffix ".<OSname>" (".`uname`"),
    ".<OSname><OSversion>" (".`uanme``uname -r`"), or
    ".<OSname><OSversion>.(64|32)bit" 
    to the respective filename
  + a file that contains a single integer number which is used as a factor
    to extend the default TIMEOUT for this test, only.      (TST.timeout)

  Using this naming conventions, the test environment can automatically
  decide what to do when called as "Mtest.py TST" (see below), i.e. execute
  one of the following:
  * TST[.exe|.com|.bat|.cmd] TST [TST.prelude]
  * python TST.py [TST.prelude]
  * Mserver --dbname=TSTDB [TST.prelude] < TST.mil(M|S)
  * Mserver --dbname=TSTDB --set mapi_port=$MAPIPORT --dbinit="module(mapi); mapi_start();" [TST.prologue] &
         MapiClient -lmil < TST[_(s|p)XX].milC
     or  TST[.exe|.com|.bat|.cmd] TST [TST.prelude]
     or  python TST.py [TST.prelude]
  * Mserver --dbname=TSTDB --set sql_port=$SQLPORT --dbinit="module(sql_server); sql_server_start();" [TST.prologue] &
         MapiClient -lsql < TST[_(s|p)XX].sql
     or  TST[.exe|.com|.bat|.cmd] TST [TST.prelude]
     or  python TST.py [TST.prelude]
  * Mserver --dbname=TSTDB --set xquery_port=$XQUERYPORT --dbinit="module(pathfinder); pfstart();" [TST.prologue] &
         MapiClient -lxquery -sxml < TST[_(s|p)XX].xq
     or  TST[.exe|.com|.bat|.cmd] TST [TST.prelude]
     or  python TST.py [TST.prelude]

  (On Unix, all M<tool>'s are started with "Mtimeout -timeout TIMEOUT
   M<tool> ..." to kill (probably) hanging M<tool>'s after a certain
   timeout. The default TIMEOUT value is 60 seconds, i.e. 1 minute.
   The TIMEOUT can be changed globally using Mtest.py's "-t" option.
   The TIMEOUT for a single test can be extended be a factor provided
   in a file TST.timeout (see above).
   Additionally, MkillUsers is scheduled as at-job to kill orphaned
   processes that Mtimeout cannot reach any more.)

  (Mserver is called with "--config=$MONETDB_CONF --debug=$GDK_DEBUG $setMONETDB_MOD_PATH --set monet_prompt=".
   See below and "Mserver --help" for details.)

  (MapiClient is called with "--port=$<l>PORT", where <l> in {"MAPI", "SQL", "XQUERY"}.
   See "MapiClient --help" for details.)


- Mtest.py uses three directory trees based at TSTSRCBASE, TSTBLDBASE, and
  TSTTRGBASE, respectively. These trees have similar purpose as the SOURCE,
  BUILD, and PREFIX trees when configuring and compiling Monet: the original
  tests are found in TSTSRCBASE, tests extracted from .mx or .in files
  during testing are fould in TSTBLDBASE, and Mtest.py writes the test
  output to TSTTRGBASE. The default settings are as follows:
	TSTSRCBASE=$MONETDB_SOURCE  (where you cecked out Monet's source tree)
	TSTBLDBASE=$MONETDB_BUILD   (where you called configure and make)
	TSTTRGBASE=$MONETDB_PREFIX  (where you told configure to put Monet)
  You can over rule any of these defaults by setting the respective
  environment variable, or by giving, e.g., --TSTTRGBASE=/tmp as command
  line option to Mtest.py. 

  IMPORTANT NOTE:

  In any case, all directories must be given as ABSOLUTE PHYSICAL PATHS,
  i.e., starting with "/" on Unix (or "[<drive-letter>:]\" on Windows, and
  not containing any symbolic link(s). 
  The latter is a restiction that might be released in the future, but for
  the time being, using paths that contain symbolic links will cause
  problems with Mtest.py.
  Hence, if you compiled Monet in a path that contains a symbolic link,
  e.g., ~/dev/monet -> /net/myhost/export/scratch1/myname/monet,
  you should either
  a) overrule the initial setting of MONETDB_SOURCE, MONETDB_BUILD,
     and MONETDB_PREFIX which were hardcoded during configure as (e.g.)
	MONETDB_SOURCE=~/dev/monet
	MONETDB_BUILD=~/dev/monet/$OSVER
	MONETDB_PREFIX=~/dev/monet/$OSVER
     by
     either setting the respective environment variables (e.g.) as follows
	MONETDB_SOURCE=/net/myhost/export/scratch1/myname/monet
	MONETDB_BUILD=/net/myhost/export/scratch1/myname/monet/$OSVER
	MONETDB_PREFIX=/net/myhost/export/scratch1/myname/monet/$OSVER
     before running Mtest.py 
     or using the respective command line options to Mtest.py
  or
  b) overrule the default settings of TSTSRCBASE, TSTBLDBASE, and
     TSTTRGBASE (e.g.) as follows
	TSTSRCBASE=/net/myhost/export/scratch1/myname/monet
	TSTBLDBASE=/net/myhost/export/scratch1/myname/monet/$OSVER
	TSTTRGBASE=/net/myhost/export/scratch1/myname/monet/$OSVER
     either by setting the repective environment variables
     or by using the respective command line options to Mtest.py (see
     above).


- Each test is identified by its name and its directory path relative to
  TSTSRCBASE, but without the trailing "/Tests".


- Running one or several tests is done by simply calling
  + "Mtest.py <names...>" to run tests <names...> in
                          a) ".",       if "." is a "Tests"-directory and
                                           there is an "All"-file in "."
                          b) "./Tests", if "./Tests/All" exists
  + "Mtest.py -r"         to run all tests found in any subdirectory of "."

  If Mtest.py is not called in (a subdirectory of) $TSTSRCBASE, Mtest.py
  changes ("cd's") to $TSTSRCBASE before starting its work.
  For more details call "Mtest.py --help".


- Environment variables:
  + Mtest.py doesn't need any special environment variables to be set.
  + However, the following environment variables can be set to overrule the
    default settings:
	name		default					description
	----		-------					-----------
	MONETDB_SOURCE	(where you checked out Monet)		base of Monet's source tree
	MONETDB_BUILD	(where you called configure & make)	base of extracted source tree
	MONETDB_PREFIX	(where you installed Monet to)		base of Monet distribution
	TSTSRCBASE	$MONETDB_SOURCE				base of test source tree
	TSTBLDBASE	$MONETDB_BUILD				base of extracted test tree
	TSTTRGBASE	$MONETDB_PREFIX				base for test output
	MAPICLIENT	MapiClient -lmil			mapi-client program	`)
	SQLCLIENT	MapiClient -lsql			sql-client program	`)
	XQUERYCLIENT	MapiClient -lxquery -sxml		xquery-client program	`)

    `) Alternative Mapi-, SQL-, & XQUERY- client programs need to accept (or ignore)
       "--config=", "--host=", & "--port=" options.

    The following variables are currently still used legacy, but they will disappear soon...
	MONETDB_MOD_PATH	$MONETDB_PREFIX/lib;MONETDB_PREFIX/lib/Monet

    The setting of all these environment variables maybe overruled
    by commandline options with the same names, e.g., "--MAPICLIENT=MapiClient.py".

    NOTE: 
    All paths must be given as ABSOLUTE PHYSICAL PATHS (see "IMPORTANT NOTE"
    above).

  + Further environment variables that Mtest.py exports (to be used in test
    scripts) are:
  	name		description
	----		-----------
	HOST		hostname
	OS		`uname`
	OSVER		`uname``uname -r`
	MTIMEOUT	"Mtimeout -timeout $TIMEOUT"
	MDIFF		"$MTIMEOUT Mdiff"
	MSERVER		"$MTIMEOUT Mserver --config=$MONETDB_CONF --debug=$GDK_DEBUG $setMONETDB_MOD_PATH $setGDK_DBFARM $setSQL_LOGDIR --set mapi_port=$MAPIPORT --set sql_port=$SQLPORT --set xquery_port=$XQUERYPORT --set monet_prompt= --trace"
	MAPI_CLIENT	"$MTIMEOUT $MAPICLIENT --config=$MONETDB_CONF --host=$HOST --port=$MAPIPORT"
	SQL_CLIENT	"$MTIMEOUT $SQLCLIENT -u monetdb -P monetdb --host=$HOST --port=$SQLPORT --trace"
	XQUERY_CLIENT	"$MTIMEOUT $XQUERYCLIENT -u monetdb -P monetdb --host=$HOST --port=$XQUERYPORT"
	MAPIPORT	the MAPI   port (random number between 30000 and 39999)
	SQLPORT		the SQL    port (random number between 40000 and 49999)
	XQUERYPORT	the XQUERY port (random number between 50000 and 59999)
	TST		name of the current test
  	TSTDB		name of default test database, unique for each directory
	TSTDIR		current test directory without $TSTSRCBASE/ & /Tests
	TSTSRCDIR	$TSTSRCBASE/$TSTDIR/Tests
	TSTBLDDIR	$TSTBLDBASE/$TSTDIR/Tests
	TSTTRGDIR	$TSTTRGBASE/mTests/$TSTDIR
	MONETDB_CONF	location of MonetDB.conf (default: $MONETDB_PREFIX/etc/MonetDB.conf)
	SQL_LOGDIR	as read from $MONETDB_CONF or set by --sql_logdir=<path>
	GDK_DBFARM	as read from $MONETDB_CONF or set by --dbfarm=<path>
	GDK_DEBUG	debug bitmask for Mserver (default: 10)
	MONETDB_MOD_PATH	new module search path of Mserver (if set via --monet_mod_path=<pathlist>)
	setMONETDB_MOD_PATH	"--set monet_mod_path=$MONETDB_MOD_PATH"	(if --monet_mod_path=<pathlist> is used)
	setGDK_DBFARM	"--set gdk_dbfarm=$GDK_DBFARM"	(if --dbfarm=<path> is used)
	setSQL_LOGDIR	"--set sql_logdir=$SQL_LOGDIR"	(if --sql_logdir=<path> is used)


- For each directory, Mtest.py creates a database $TSTDB in $GDK_DBFARM
  and the respective transaction-log directory $TSTDB in $SQL_LOGDIR
  before running any test.


- A test is skipped (i.e. not executed), 
  if $TST.modules exists and contains modules that are not available, or
  if $TST.BATs exists and contains BATs that are not available in the
  current database $TSTDB.


- While executing test "$TSTSRCDIR/TST", Mtest.py uses $TSTTRGDIR as working
  directory; for convenience, all files "$TSTSRCDIR/TST*" are copied to
  $TSTTRGDIR.


- Mtest.py stores the output of test "$TSTSRCDIR/TST" in
  $TSTTRGDIR/TST.test.{out,err} and checks it against
  $TSTSRCDIR/TST.stable.{out,err}. If system specific stable output files
  ($TSTSRCDIR/TST.stable.{out,err}[.($OS|$OSVER[.(64|32)bit])][.STATIC])
  exist, the most specific stable output is used. Mtest.py stores the
  respective Mdiff results in $TSTTRGDIR/TST.{out,err}.diff.html.
  For each test directory, a common entry point to the test results is
  created as $TSTTRGDIR/.index.html.  
  For the whole test run, a global entry point to the test results is
  created as $TSTTRGBASE/mTests/index.html.

  
- Mfilter.py is used to filter out variable date form the test output to
  avoid no-error differences when running Mdiff. Mfilter.py prefixes all
  lines enclosed in two line containing only "#~BeginVariableOutput~#" and
  "#~EndVariableOutput~#", respectively, with "#~". These lines are then
  ignored during Mdiff.
  
  MIL Example:
	printf("#~BeginVariableOutput~#\n"); cpu.print(); printf("#~EndVariableOutput~#\n);


- A tool Mapprove.py is provided to copy the recent output from
  $TSTTRGDIR/TST.test.{out,err} to $TSTSRCDIR/TST.stable.{out,err}. Lines
  starting with a '!' (i.e. error messages) are eliminated during copying. 
  Of course, $TSTTRGDIR/TST.test.{out,err} have to be checked and claimed
  correct before being approved!
  See "Mapprove.py -?" for command-line options.


- A development session is then supposed to proceed as follows:
  (assuming TSTSRCBASE=$MONETDB_SOURCE and TSTBLDBASE=$MONETDB_BUILD)

	cd $TSTSRCBASE
	run  cvs update
	loop
		loop
			loop
				# write & compile your code
				cd $TSTSRCBASE/$TSTDIR
				edit source
				cd $TSTBLDBASE/$TSTDIR
				[g]make
				[g]make install
			endloop
			loop
				# write, adapt, run your tests ...
				cd $TSTSRCBASE/$TSTDIR
				edit/create  Tests/All
				edit/create  Tests/<names...>.*
				run  Mtest.py <names...>
				# ... and check their results
				check 	$TSTTRGDIR/<names...>.test.{out,err}
					$TSTTRGDIR/<names...>.{out,err}.diff.html
					(cf. $TSTTRGBASE/mTests/index.html)
			endloop
			# approve new/modified correct output of your tests
			run  Mapprove.py <names...>
		endloop
		# run all Monet tests...
		cd $TSTSRCBASE
		run  Mtest.py -r
		# ... and check their results
		check $TSTTRGBASE/mTests/index.html
	endloop
	# approve "intended" impact of your changes on other tests' output
	run  Mapprove.py <names...>
	# checkin all correct new stuff to cvs
	run  cvs ci


- Mprofile.py:
   Mprofile.py works just like Mtest.py, only it uses "prof.py" to add some
   mprof-commands to all TST*.*mil* files before actually running the test.
   Thus, profiling information is collected.
   prof.py reads a list of mil commands from a .Mprofile-commands.lst file
   (specified by Mprofile.py's "--CMDLST" parameter) and wraps all
   occurances of these commands in a mil script in
	pmB("<command>");
	<command>; 
	pmE("<command>");
   Further, prof.py adds a "pmSummary();" at the end of the mil script.


--
| Stefan Manegold     | mailto:Stefan.Manegold@cwi.nl |
| CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
| 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
| The Netherlands     | Fax : +31 (20) 592-4312       |

