Pretrained Punkt Models -- Jan Strunk (New version trained after issues 313 and 514 had been corrected)

These models were prepared using the test corpora from Kiss and Strunk (2006).

For information about how to use these models, please confer the tokenization HOWTO:
http://nltk.googlecode.com/svn/trunk/doc/howto/tokenize.html
and chapter 3.8 of the NLTK book:
http://nltk.googlecode.com/svn/trunk/doc/book/ch03.html#sec-segmentation

There are pretrained tokenizers for the following languages:

File                Language            Source                             Contents                Size of training corpus(in tokens)
=====================================================================================================================================
czech.pickle        Czech               Multilingual Corpus 1 (ECI)        Lidove Noviny                   ~345,000
                                                                           Literarni Noviny
-------------------------------------------------------------------------------------------------------------------------------------
danish.pickle       Danish              Avisdata CD-Rom Ver. 1.1. 1995     Berlingske Tidende              ~550,000
                                        (Berlingske Avisdata, Copenhagen)  Weekend Avisen
-------------------------------------------------------------------------------------------------------------------------------------
dutch.pickle        Dutch               Multilingual Corpus 1 (ECI)        De Limburger                    ~340,000
-------------------------------------------------------------------------------------------------------------------------------------
english.pickle      English             Penn Treebank (LDC)                Wall Street Journal             ~469,000             
                    (American)
-------------------------------------------------------------------------------------------------------------------------------------
estonian.pickle     Estonian            University of Tartu, Estonia       Eesti Ekspress                  ~359,000
-------------------------------------------------------------------------------------------------------------------------------------
finnish.pickle      Finnish             Finnish Parole Corpus, Finnish     Books and major national        ~364,000
                                        Text Bank (Suomen Kielen           newspapers
                                        Tekstipankki)
                                        Finnish Center for IT Science
                                        (CSC)
-------------------------------------------------------------------------------------------------------------------------------------
french.pickle       French              Multilingual Corpus 1 (ECI)        Le Monde                        ~370,000
                    (European)
-------------------------------------------------------------------------------------------------------------------------------------
german.pickle       German              Neue Zürcher Zeitung AG            Neue Zürcher Zeitung            ~847,000
                    (Switzerland)       CD-ROM
                    (Uses "ss"
                     instead of "ß")
-------------------------------------------------------------------------------------------------------------------------------------
greek.pickle        Greek               Efstathios Stamatatos              To Vima (TO BHMA)               ~227,000
-------------------------------------------------------------------------------------------------------------------------------------
italian.pickle      Italian             Multilingual Corpus 1 (ECI)        La Stampa, Il Mattino           ~312,000
-------------------------------------------------------------------------------------------------------------------------------------
norwegian.pickle    Norwegian           Centre for Humanities              Bergens Tidende                 ~479,000
                    (Bokmål and         Information Technologies,
                     Nynorsk)           Bergen
-------------------------------------------------------------------------------------------------------------------------------------
portuguese.pickle   Portuguese          CETENFolha Corpus                  Folha de São Paulo              ~321,000
                    (Brazilian)         (Linguateca)
-------------------------------------------------------------------------------------------------------------------------------------
slovene.pickle      Slovene             TRACTOR                            Delo                            ~354,000
                                        Slovene Academy for Arts
                                        and Sciences
-------------------------------------------------------------------------------------------------------------------------------------
spanish.pickle      Spanish             Multilingual Corpus 1 (ECI)        Sur                             ~353,000
                    (European)
-------------------------------------------------------------------------------------------------------------------------------------
swedish.pickle      Swedish             Multilingual Corpus 1 (ECI)        Dagens Nyheter                  ~339,000
                                                                           (and some other texts)
-------------------------------------------------------------------------------------------------------------------------------------
turkish.pickle      Turkish             METU Turkish Corpus                Milliyet                        ~333,000
                                        (Türkçe Derlem Projesi)
                                        University of Ankara
-------------------------------------------------------------------------------------------------------------------------------------

The corpora contained about 400,000 tokens on average and mostly consisted of newspaper text converted to
Unicode using the codecs module.

Kiss, Tibor and Strunk, Jan (2006): Unsupervised Multilingual Sentence Boundary Detection.
Computational Linguistics 32: 485-525.

---- Training Code ----

# import punkt
import nltk.tokenize.punkt

# Make a new Tokenizer
tokenizer = nltk.tokenize.punkt.PunktSentenceTokenizer()

# Read in trainings corpus (one example: Slovene)
import codecs
text = codecs.open("slovene.plain","Ur","iso-8859-2").read()

# Train tokenizer
tokenizer.train(text)

# Dump pickled tokenizer
import pickle
out = open("slovene.pickle","wb")
pickle.dump(tokenizer, out)
out.close()

---------
