A few notes on the conversion of the CIA world factbook 2002
============================================================

The CIA world factbook 2002 is available in HTML format in different versions
at the CIA website http://www.cia.gov/cia/publications/factbook/ .
Unfortunately there are no plaintext / rtf / doc versions available from
the CIA server.

The HTML versions differ with regard to maps and other graphical elements
which are provided in the fullblown version but are missing in the "low
bandwidth version" at
http://www.cia.gov/cia/publications/factbook/countrylisting.html
which was used for the creation of the dict version. In addition to the
country listings the 6 appendices found in the full version are included
in the dictionary.


downloading of files
--------------------

The country listings were downloaded using wget:

	wget -r -l1 \
	http://www.cia.gov/cia/publications/factbook/countrylisting.html

The appendices and the copyright info were downloaded manually and saved to
files.


conversion to txt
-----------------

For html to text conversion html2text was used as lynx -dump does not retain
the table structure of the text:

	for i in *.html
	do
		html2text -nobs -width 73 $i >$i.txt
	done

html2text does a good job with regard to the text layout but the original HTML
format leads after conversion to a two coulumn output where both colums have
approximately the same width. Not ideal for this application but it works
reasonably well.


pre-formatting
--------------

In order not to reinvent the wheel the final formatting to dict format was
done by dictfmt. Prior to the dictfmt run the country listings as well as
the 6 appendices and the copyright info needed to be cleaned up. This was
done manually for the appendices and the copyright info as I considered it
not worth the trouble to write a program for this task. The country listings
were processed by a small custom preprocessor written in python (see
convert.py).

convert.py makes a few assumptions where to find its input. If you intend to
use it for your own purposes please adapt the configuration section to your
needs. Make sure that the appendices and country specific files are located
in different directories and that there are no other .txt files in these
directories.

convert.py reads the text files in the countries/ and appendices/
directories as well as the copyright file and writes the preformatted result
to stdout:

	python convert.py > world02.txt

Before exiting the script creates a table of contents.


creating the dict database
--------------------------

you guessed it:

	dictfmt  -f -u \
	http://www.cia.gov/cia/publications/factbook/countrylisting.html \
 	-s "CIA World Factbook 2002"  -f world02 < world02.txt

finally:

	dictzip world02.dict

That's all. Enjoy.


Eckhard
