  After long development we would like to introduce new
storage mode which is able to index and search through
several millions of documents. You may visit our test
web site where we have indexed 4.3 million documents
and check result output speed:

http://udm.aspseek.com/cgi-bin/search.cgi

  The main idea of new storage mode is that word index
is stored on the disc rather than SQL database. URL information
(table "url") is kept in SQL thought. Word index is devided into 
1 million files located in tree directories structure under /var/tree 
directory of UdmSearch installation. We call this storage mode
 "cache mode" because words index distribution is very similiar 
to squid cache structure.

  To start usage of "cache" mode edit your indexer.conf and
search.htm as usual. The only thing you have to change is
configure "DBMode" parameter:

DBMode cache

  Indexing in the Cache storage mode is to be done in two
stages. The first one is indexing inself. It is to be done
in usual manner. Indexer being executed in cache mode writes
256 logs into /var/raw directory of UdmSearch installation at 
very fast speed. Indexing speed depends first of all on network 
capacity. The second stage is distribution indexer's logs into high 
performance tree structure. This to be done by new application 
"splitter" included into standard release of our search engine.

  splitter has two command line arguments: -f <first log> -t <second log>
If no parameters are specified splitter distributes all 256 logs.
You can limit log range using -f and -t keys specifying parameters in
HEX notation. For example, "splitter -f 00 -t A0" will distribute logs
in the range from 00 to A0. These keys allow to use several splitters
at the same time. This usually give more quick database distribution.
This shell script starts for splitters in background:

#!/bin/sh
splitter -f 00 -t 3f &
splitter -f 40 -t 7f &
splitter -f 80 -t bf &
splitter -f c0 -t ff &

  After each time you run indexer you have to run splitter which
will commin logs created by indexer into database.


  3.1.5 release notes:

1. Parallel indexing is not implemented yet. Do not use several
indexers at the same time! Do not use threaded version of indexer
(wich is built when --enable-linux-threads is specified to configure
script)!

2. "indexer -C" WITHOUT subsection control just removes all files
from word index tree. You should delete all logs manually.

3. "indexer -C ... " WITH one or several of subsection control option 
(-t, -u, -s) does write to logs and you have to run splitter after it.

4. After running splitter you may delete indexer logs located in /var/raw
directory or move them into enother place for backup pusposes.


  Things that will be implemented in near future:

1. Parallel indexing, both threaded version and several simultanious
indexers even on different machines.

2. Automatize database management: remove all manual operation such
as log moving etc.

3. Make it possible to distribute database between several machines.

4. Make file formats platform independant to take in account bytes ordering.
This will allow to use logs, created on i386, on SGI and Sparc machines.
