Implementation of ‘TnT - A Statisical Part of Speech Tagger’ by Thorsten Brants
- class nltk.tag.tnt.TnT¶
TnT - Statistical POS tagger
DOES NOT AUTOMATICALLY DEAL WITH UNSEEN WORDS
It is possible to provide an untrained POS tagger to create tags for unknown words, see __init__ function
SHOULD BE USED WITH SENTENCE-DELIMITED INPUT
Due to the nature of this tagger, it works best when trained over sentence delimited input.
However it still produces good results if the training data and testing data are separated on all punctuation eg: [,.?!]
Input for training is expected to be a list of sentences where each sentence is a list of (word, tag) tuples
Input for tag function is a single sentence Input for tagdata function is a list of sentences Output is of a similar form
Function provided to process text that is unsegmented
Please see basic_sent_chop()
TnT uses a second order Markov model to produce tags for a sequence of input, specifically:
argmax [Proj(P(t_i|t_i-1,t_i-2)P(w_i|t_i))] P(t_T+1 | t_T)
IE: the maximum projection of a set of probabilities
The set of possible tags for a given word is derived from the training data. It is the set of all tags that exact word has been assigned.
To speed up and get more precision, we can use log addition to instead multiplication, specifically:
- argmax [Sigma(log(P(t_i|t_i-1,t_i-2))+log(P(w_i|t_i)))] +
The probability of a tag for a given word is the linear interpolation of 3 markov models; a zero-order, first-order, and a second order model.
- P(t_i| t_i-1, t_i-2) = l1*P(t_i) + l2*P(t_i| t_i-1) +
l3*P(t_i| t_i-1, t_i-2)
A beam search is used to limit the memory usage of the algorithm. The degree of the beam can be changed using N in the initialization. N represents the maximum number of possible solutions to maintain while tagging.
It is possible to differentiate the tags which are assigned to capitalized words. However this does not result in a significant gain in the accuracy of the results.
- __init__(unk=None, Trained=False, N=1000, C=False)¶
Construct a TnT statistical tagger. Tagger must be trained before being used to tag input.
unk (TaggerI) – instance of a POS tagger, conforms to TaggerI
Trained (bool) – Indication that the POS tagger is trained or not
N (int) – Beam search degree (see above)
C (bool) – Capitalization flag
Initializer, creates frequency distributions to be used for tagging
_lx values represent the portion of the tri/bi/uni taggers to be used to calculate the probability
N value is the number of possible solutions to maintain while tagging. A good value for this is 1000
C is a boolean value which specifies to use or not use the Capitalization of the word as additional information for tagging. NOTE: using capitalization may not increase the accuracy of the tagger
Uses a set of tagged data to train the tagger. If an unknown word tagger is specified, it is trained on the same data.
data (tuple(str)) – List of lists of (word, tag) tuples
Tags each sentence in a list of sentences
:param data:list of list of words :type data: [[string,],] :return: list of list of (word, tag) tuples
Invokes tag(sent) function for each sentence compiles the results into a list of tagged sentences each tagged sentence is a list of (word, tag) tuples
Tags a single sentence
data ([string,]) – list of words
Calls recursive function ‘_tagword’ to produce a list of tags
Associates the sequence of returned tags with the correct words in the input sequence
returns a list of (word, tag) tuples
- nltk.tag.tnt.basic_sent_chop(data, raw=True)¶
Basic method for tokenizing input into sentences for this tagger:
data (str or tuple(str, str)) – list of tokens (words or (word, tag) tuples)
raw (bool) – boolean flag marking the input data as a list of words or a list of tagged words
list of sentences sentences are a list of tokens tokens are the same as the input
Function takes a list of tokens and separates the tokens into lists where each list represents a sentence fragment This function can separate both tagged and raw sequences into basic sentences.
Sentence markers are the set of [,.!?]
This is a simple method which enhances the performance of the TnT tagger. Better sentence tokenization will further enhance the results.