nltk.classify.textcat module

A module for language identification using the TextCat algorithm. An implementation of the text categorization algorithm presented in Cavnar, W. B. and J. M. Trenkle, “N-Gram-Based Text Categorization”.

The algorithm takes advantage of Zipf’s law and uses n-gram frequencies to profile languages and text-yet to be identified-then compares using a distance measure.

Language n-grams are provided by the “An Crubadan” project. A corpus reader was created separately to read those files.

For details regarding the algorithm, see:

For details about An Crubadan, see:

class nltk.classify.textcat.TextCat[source]

Bases: object

calc_dist(lang, trigram, text_profile)[source]

Calculate the “out-of-place” measure between the text and language profile for a single trigram

fingerprints = {}

Find the language with the min distance to the text and return its ISO 639-3 code


Calculate the “out-of-place” measure between the text and all languages

last_distances = {}

Create FreqDist of trigrams within text


Get rid of punctuation except apostrophes