Full text search indexing
=========================

Dovecot v1.1 supports two FTS indexers: Squat and Lucene. It's also possible to
use both of them at the same time.

Squat
-----

Squat indexes can be used to optimize standard IMAP SEARCH TEXT and BODY
commands. The main advantage of Squat indexes is that they allow substring
matching, while pretty much all other FTS indexes support only matching from
the beginning of words. IMAP searches require substring matching, so Squat
indexes are the only choice to optimize them in a standard way.

The Squat indexes are currently updated only when SEARCH TEXT or SEARCH BODY
command is used. It's planned that in future they could be updated immediately
when messages are being saved.

Squat works by building a trie of all 1..4 character combinations of all words
in messages. For example given a word "world" it indexes "worl", "orld", "rld",
"ld" and "d". Matching message UIDs are stored to each trie node, so Squat
supports giving definitive answers to any searches with a length of 1..4
characters. For longer search words the word is looked up in 4 letter pieces
and the results are merged. The resulting list is a list of messages where the
word *may* exist. This means that those messages are still opened and the word
is searched from them the slow way.

The Squat name comes from Cyrus IMAP [http://cyrusimap.web.cmu.edu/] which
implements slightly similar Squat indexes ("Search QUery Answer Tool").
Dovecot's implementation and file format however is completely different. The
main visible difference is that Dovecot allows updating the index incrementally
instead of requiring to re-read the entire mailbox to build it. Cyrus Squat
also doesn't support search 1..3  characters long searches.

Configuration
-------------

---%<-------------------------------------------------------------------------
protocol imap {
..
  mail_plugins = fts fts_squat
}
...
plugin {
  fts = squat
}
---%<-------------------------------------------------------------------------

Lucene
------

Lucene indexes can be used for non-standard IMAP SEARCHes, so they're useful
only for modified webmail applications and such. Instead of using "TEXT" and
"BODY" search commands, use "X-TEXT-FAST" and "X-BODY-FAST".

Dovecot builds only a single Lucene index for all mailboxes. In future this
would allow support for searching messages quickly from all mailboxes.

Configuration
-------------

---%<-------------------------------------------------------------------------
protocol imap {
..
  mail_plugins = fts fts_lucene
}
...
plugin {
  fts = lucene
}
---%<-------------------------------------------------------------------------

Both
----

It's also possible to use both Squat and Lucene indexes. Lucene will be used
for X-TEXT-FAST and X-BODY-FAST searches while Squat will be used for TEXT,
BODY and HEADER searches.

---%<-------------------------------------------------------------------------
protocol imap {
..
  mail_plugins = fts fts_squat fts_lucene
}
...
plugin {
  fts = squat lucene
}
---%<-------------------------------------------------------------------------

(This file was created from the wiki on 2008-02-21 16:13)
