Sample usage for tag¶
Evaluation of Taggers¶
Evaluating the standard NLTK PerceptronTagger using Accuracy, Precision, Recall and F-measure for each of the tags.
>>> from nltk.tag import PerceptronTagger
>>> from nltk.corpus import treebank
>>> tagger = PerceptronTagger()
>>> gold_data = treebank.tagged_sents()[10:20]
>>> print(tagger.accuracy(gold_data))
0.885931...
>>> print(tagger.evaluate_per_tag(gold_data))
Tag | Prec. | Recall | F-measure
-------+--------+--------+-----------
'' | 1.0000 | 1.0000 | 1.0000
, | 1.0000 | 1.0000 | 1.0000
-NONE- | 0.0000 | 0.0000 | 0.0000
. | 1.0000 | 1.0000 | 1.0000
: | 1.0000 | 1.0000 | 1.0000
CC | 1.0000 | 1.0000 | 1.0000
CD | 0.7647 | 1.0000 | 0.8667
DT | 1.0000 | 1.0000 | 1.0000
IN | 1.0000 | 1.0000 | 1.0000
JJ | 0.5882 | 0.8333 | 0.6897
JJR | 1.0000 | 1.0000 | 1.0000
JJS | 1.0000 | 1.0000 | 1.0000
NN | 0.7647 | 0.9630 | 0.8525
NNP | 0.8929 | 1.0000 | 0.9434
NNS | 1.0000 | 1.0000 | 1.0000
POS | 1.0000 | 1.0000 | 1.0000
PRP | 1.0000 | 1.0000 | 1.0000
RB | 0.8000 | 1.0000 | 0.8889
RBR | 0.0000 | 0.0000 | 0.0000
TO | 1.0000 | 1.0000 | 1.0000
VB | 1.0000 | 1.0000 | 1.0000
VBD | 0.8571 | 0.9231 | 0.8889
VBG | 1.0000 | 1.0000 | 1.0000
VBN | 0.8333 | 0.5556 | 0.6667
VBP | 0.5714 | 0.8000 | 0.6667
VBZ | 1.0000 | 1.0000 | 1.0000
WP | 1.0000 | 1.0000 | 1.0000
`` | 1.0000 | 1.0000 | 1.0000
List only the 10 most common tags:
>>> print(tagger.evaluate_per_tag(gold_data, truncate=10, sort_by_count=True))
Tag | Prec. | Recall | F-measure
-------+--------+--------+-----------
IN | 1.0000 | 1.0000 | 1.0000
DT | 1.0000 | 1.0000 | 1.0000
NN | 0.7647 | 0.9630 | 0.8525
NNP | 0.8929 | 1.0000 | 0.9434
NNS | 1.0000 | 1.0000 | 1.0000
-NONE- | 0.0000 | 0.0000 | 0.0000
CD | 0.7647 | 1.0000 | 0.8667
VBD | 0.8571 | 0.9231 | 0.8889
JJ | 0.5882 | 0.8333 | 0.6897
, | 1.0000 | 1.0000 | 1.0000
Similarly, we can display the confusion matrix for this tagger.
>>> print(tagger.confusion(gold_data))
| - |
| N |
| O |
| N J J N N P P R V V V V V |
| ' E C C D I J J J N N N O R R B T V B B B B B W ` |
| ' , - . : C D T N J R S N P S S P B R O B D G N P Z P ` |
-------+-------------------------------------------------------------------------------------+
'' | <3> . . . . . . . . . . . . . . . . . . . . . . . . . . . |
, | .<11> . . . . . . . . . . . . . . . . . . . . . . . . . . |
-NONE- | . . <.> . . . 4 . . 4 . . 7 2 . . . 1 . . . . . . 3 . . . |
. | . . .<10> . . . . . . . . . . . . . . . . . . . . . . . . |
: | . . . . <1> . . . . . . . . . . . . . . . . . . . . . . . |
CC | . . . . . <5> . . . . . . . . . . . . . . . . . . . . . . |
CD | . . . . . .<13> . . . . . . . . . . . . . . . . . . . . . |
DT | . . . . . . .<28> . . . . . . . . . . . . . . . . . . . . |
IN | . . . . . . . .<34> . . . . . . . . . . . . . . . . . . . |
JJ | . . . . . . . . .<10> . . . 1 . . . . 1 . . . . . . . . . |
JJR | . . . . . . . . . . <1> . . . . . . . . . . . . . . . . . |
JJS | . . . . . . . . . . . <1> . . . . . . . . . . . . . . . . |
NN | . . . . . . . . . 1 . .<26> . . . . . . . . . . . . . . . |
NNP | . . . . . . . . . . . . .<25> . . . . . . . . . . . . . . |
NNS | . . . . . . . . . . . . . .<22> . . . . . . . . . . . . . |
POS | . . . . . . . . . . . . . . . <1> . . . . . . . . . . . . |
PRP | . . . . . . . . . . . . . . . . <3> . . . . . . . . . . . |
RB | . . . . . . . . . . . . . . . . . <4> . . . . . . . . . . |
RBR | . . . . . . . . . . . . . . . . . . <.> . . . . . . . . . |
TO | . . . . . . . . . . . . . . . . . . . <2> . . . . . . . . |
VB | . . . . . . . . . . . . . . . . . . . . <1> . . . . . . . |
VBD | . . . . . . . . . . . . . . . . . . . . .<12> . 1 . . . . |
VBG | . . . . . . . . . . . . . . . . . . . . . . <3> . . . . . |
VBN | . . . . . . . . . 2 . . . . . . . . . . . 2 . <5> . . . . |
VBP | . . . . . . . . . . . . 1 . . . . . . . . . . . <4> . . . |
VBZ | . . . . . . . . . . . . . . . . . . . . . . . . . <2> . . |
WP | . . . . . . . . . . . . . . . . . . . . . . . . . . <3> . |
`` | . . . . . . . . . . . . . . . . . . . . . . . . . . . <3>|
-------+-------------------------------------------------------------------------------------+
(row = reference; col = test)
Brill Trainer with evaluation¶
>>> # Perform the relevant imports.
>>> from nltk.tbl.template import Template
>>> from nltk.tag.brill import Pos, Word
>>> from nltk.tag import untag, RegexpTagger, BrillTaggerTrainer, UnigramTagger
>>> # Load some data
>>> from nltk.corpus import treebank
>>> training_data = treebank.tagged_sents()[:100]
>>> baseline_data = treebank.tagged_sents()[100:200]
>>> gold_data = treebank.tagged_sents()[200:300]
>>> testing_data = [untag(s) for s in gold_data]
>>> backoff = RegexpTagger([
... (r'^-?[0-9]+(.[0-9]+)?$', 'CD'), # cardinal numbers
... (r'(The|the|A|a|An|an)$', 'AT'), # articles
... (r'.*able$', 'JJ'), # adjectives
... (r'.*ness$', 'NN'), # nouns formed from adjectives
... (r'.*ly$', 'RB'), # adverbs
... (r'.*s$', 'NNS'), # plural nouns
... (r'.*ing$', 'VBG'), # gerunds
... (r'.*ed$', 'VBD'), # past tense verbs
... (r'.*', 'NN') # nouns (default)
... ])
We’ve now created a simple RegexpTagger, which tags according to the regular expression
rules it has been supplied. This tagger in and of itself does not have a great accuracy.
>>> backoff.accuracy(gold_data)
0.245014...
Neither does a simple UnigramTagger. This tagger is trained on some data,
and will then first try to match unigrams (i.e. tokens) of the sentence it has
to tag to the learned data.
>>> unigram_tagger = UnigramTagger(baseline_data)
>>> unigram_tagger.accuracy(gold_data)
0.581196...
The lackluster accuracy here can be explained with the following example:
>>> unigram_tagger.tag(["I", "would", "like", "this", "sentence", "to", "be", "tagged"])
[('I', 'NNP'), ('would', 'MD'), ('like', None), ('this', 'DT'), ('sentence', None),
('to', 'TO'), ('be', 'VB'), ('tagged', None)]
As you can see, many tokens are tagged as None, as these tokens are OOV (out of vocabulary).
The UnigramTagger has never seen them, and as a result they are not in its database of known terms.
In practice, a UnigramTagger is exclusively used in conjunction with a backoff. Our real
baseline which will use such a backoff. We’ll create a UnigramTagger like before, but now
the RegexpTagger will be used as a backoff for the situations where the UnigramTagger
encounters an OOV token.
>>> baseline = UnigramTagger(baseline_data, backoff=backoff)
>>> baseline.accuracy(gold_data)
0.7537647...
That is already much better. We can investigate the performance further by running
evaluate_per_tag. This method will output the Precision, Recall and F-measure
of each tag.
>>> print(baseline.evaluate_per_tag(gold_data, sort_by_count=True))
Tag | Prec. | Recall | F-measure
-------+--------+--------+-----------
NNP | 0.9674 | 0.2738 | 0.4269
NN | 0.4111 | 0.9136 | 0.5670
IN | 0.9383 | 0.9580 | 0.9480
DT | 0.9819 | 0.8859 | 0.9314
JJ | 0.8167 | 0.2970 | 0.4356
NNS | 0.7393 | 0.9630 | 0.8365
-NONE- | 1.0000 | 0.8345 | 0.9098
, | 1.0000 | 1.0000 | 1.0000
. | 1.0000 | 1.0000 | 1.0000
VBD | 0.6429 | 0.8804 | 0.7431
CD | 1.0000 | 0.9872 | 0.9935
CC | 1.0000 | 0.9355 | 0.9667
VB | 0.7778 | 0.3684 | 0.5000
VBN | 0.9375 | 0.3000 | 0.4545
RB | 0.7778 | 0.7447 | 0.7609
TO | 1.0000 | 1.0000 | 1.0000
VBZ | 0.9643 | 0.6429 | 0.7714
VBG | 0.6415 | 0.9444 | 0.7640
PRP$ | 1.0000 | 1.0000 | 1.0000
PRP | 1.0000 | 0.5556 | 0.7143
MD | 1.0000 | 1.0000 | 1.0000
VBP | 0.6471 | 0.5789 | 0.6111
POS | 1.0000 | 1.0000 | 1.0000
$ | 1.0000 | 0.8182 | 0.9000
'' | 1.0000 | 1.0000 | 1.0000
: | 1.0000 | 1.0000 | 1.0000
WDT | 0.4000 | 0.2000 | 0.2667
`` | 1.0000 | 1.0000 | 1.0000
JJR | 1.0000 | 0.5000 | 0.6667
NNPS | 0.0000 | 0.0000 | 0.0000
RBR | 1.0000 | 1.0000 | 1.0000
-LRB- | 0.0000 | 0.0000 | 0.0000
-RRB- | 0.0000 | 0.0000 | 0.0000
RP | 0.6667 | 0.6667 | 0.6667
EX | 0.5000 | 0.5000 | 0.5000
JJS | 0.0000 | 0.0000 | 0.0000
WP | 1.0000 | 1.0000 | 1.0000
PDT | 0.0000 | 0.0000 | 0.0000
AT | 0.0000 | 0.0000 | 0.0000
It’s clear that although the precision of tagging “NNP” is high, the recall is very low. With other words, we’re missing a lot of cases where the true label is “NNP”. We can see a similar effect with “JJ”.
We can also see a very expected result: The precision of “NN” is low, while the recall
is high. If a term is OOV (i.e. UnigramTagger defers it to RegexpTagger) and
RegexpTagger doesn’t have a good rule for it, then it will be tagged as “NN”. So,
we catch almost all tokens that are truly labeled as “NN”, but we also tag as “NN”
for many tokens that shouldn’t be “NN”.
This method gives us some insight in what parts of the tagger needs more attention, and why. However, it doesn’t tell us what the terms with true label “NNP” or “JJ” are actually tagged as. To help that, we can create a confusion matrix.
>>> print(baseline.confusion(gold_data))
| - |
| - N - |
| L O R N P |
| R N R J J N N N P P P R R V V V V V W |
| ' B E B A C C D E I J J J M N N P N D O R P R B R T V B B B B B D W ` |
| $ ' , - - - . : T C D T X N J R S D N P S S T S P $ B R P O B D G N P Z T P ` |
-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
$ | <9> . . . . . . . . . . . . . . . . . 2 . . . . . . . . . . . . . . . . . . . . |
'' | . <10> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
, | . .<115> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
-LRB- | . . . <.> . . . . . . . . . . . . . . 3 . . . . . . . . . . . . . . . . . . . . |
-NONE- | . . . .<121> . . . . . . . . . . . . . 24 . . . . . . . . . . . . . . . . . . . . |
-RRB- | . . . . . <.> . . . . . . . . . . . . 3 . . . . . . . . . . . . . . . . . . . . |
. | . . . . . .<100> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
: | . . . . . . . <10> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
AT | . . . . . . . . <.> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
CC | . . . . . . . . . <58> . . . . . . . . 4 . . . . . . . . . . . . . . . . . . . . |
CD | . . . . . . . . . . <77> . . . . . . . 1 . . . . . . . . . . . . . . . . . . . . |
DT | . . . . . . . . 1 . .<163> . 4 . . . . 13 . . . . . . . . . . . . . . . . . 3 . . |
EX | . . . . . . . . . . . . <1> . . . . . 1 . . . . . . . . . . . . . . . . . . . . |
IN | . . . . . . . . . . . . .<228> . . . . 8 . . . . . . . . . . . . . 2 . . . . . . |
JJ | . . . . . . . . . . . . . . <49> . . . 86 2 . 4 . . . . 6 . . . . 12 3 . 3 . . . . |
JJR | . . . . . . . . . . . . . . . <3> . . 3 . . . . . . . . . . . . . . . . . . . . |
JJS | . . . . . . . . . . . . . . . . <.> . 2 . . . . . . . . . . . . . . . . . . . . |
MD | . . . . . . . . . . . . . . . . . <19> . . . . . . . . . . . . . . . . . . . . . |
NN | . . . . . . . . . . . . . . 9 . . .<296> . . 5 . . . . . . . . 5 . 9 . . . . . . |
NNP | . . . . . . . . . . . 2 . . . . . . 199 <89> . 26 . . . . 2 . . . . 2 5 . . . . . . |
NNPS | . . . . . . . . . . . . . . . . . . . 1 <.> 3 . . . . . . . . . . . . . . . . . |
NNS | . . . . . . . . . . . . . . . . . . 5 . .<156> . . . . . . . . . . . . . 1 . . . |
PDT | . . . . . . . . . . . 1 . . . . . . . . . . <.> . . . . . . . . . . . . . . . . |
POS | . . . . . . . . . . . . . . . . . . . . . . . <14> . . . . . . . . . . . . . . . |
PRP | . . . . . . . . . . . . . . . . . . 10 . . 2 . . <15> . . . . . . . . . . . . . . |
PRP$ | . . . . . . . . . . . . . . . . . . . . . . . . . <28> . . . . . . . . . . . . . |
RB | . . . . . . . . . . . . 1 4 . . . . 6 . . . . . . . <35> . 1 . . . . . . . . . . |
RBR | . . . . . . . . . . . . . . . . . . . . . . . . . . . <4> . . . . . . . . . . . |
RP | . . . . . . . . . . . . . . . . . . . . . . . . . . 1 . <2> . . . . . . . . . . |
TO | . . . . . . . . . . . . . . . . . . . . . . . . . . . . . <47> . . . . . . . . . |
VB | . . . . . . . . . . . . . . 2 . . . 30 . . . . . . . 1 . . . <21> . . . 3 . . . . |
VBD | . . . . . . . . . . . . . . . . . . 10 . . . . . . . . . . . . <81> . 1 . . . . . |
VBG | . . . . . . . . . . . . . . . . . . 2 . . . . . . . . . . . . . <34> . . . . . . |
VBN | . . . . . . . . . . . . . . . . . . 4 . . . . . . . . . . . . 31 . <15> . . . . . |
VBP | . . . . . . . . . . . . . . . . . . 7 . . . . . . . . . . . 1 . . . <11> . . . . |
VBZ | . . . . . . . . . . . . . . . . . . . . . 15 . . . . . . . . . . . . . <27> . . . |
WDT | . . . . . . . . . . . . . 7 . . . . 1 . . . . . . . . . . . . . . . . . <2> . . |
WP | . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . <2> . |
`` | . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . <10>|
-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
(row = reference; col = test)
Once again we can see that “NN” is the default if the tagger isn’t sure. Beyond that, we can see why the recall for “NNP” is so low: these tokens are often tagged as “NN”. This effect can also be seen for “JJ”, where the majority of tokens that ought to be tagged as “JJ” are actually tagged as “NN” by our tagger.
This tagger will only serve as a baseline for the BrillTaggerTrainer, which uses
templates to attempt to improve the performance of the tagger.
>>> # Set up templates
>>> Template._cleartemplates() #clear any templates created in earlier tests
>>> templates = [Template(Pos([-1])), Template(Pos([-1]), Word([0]))]
>>> # Construct a BrillTaggerTrainer
>>> tt = BrillTaggerTrainer(baseline, templates, trace=3)
>>> tagger1 = tt.train(training_data, max_rules=10)
TBL train (fast) (seqs: 100; tokens: 2417; tpls: 2; min score: 2; min acc: None)
Finding initial useful rules...
Found 618 useful rules.
B |
S F r O | Score = Fixed - Broken
c i o t | R Fixed = num tags changed incorrect -> correct
o x k h | u Broken = num tags changed correct -> incorrect
r e e e | l Other = num tags changed incorrect -> incorrect
e d n r | e
------------------+-------------------------------------------------------
13 14 1 4 | NN->VB if Pos:TO@[-1]
8 8 0 0 | NN->VB if Pos:MD@[-1]
7 10 3 22 | NN->IN if Pos:NNS@[-1]
5 5 0 0 | NN->VBP if Pos:PRP@[-1]
5 5 0 0 | VBD->VBN if Pos:VBZ@[-1]
5 5 0 0 | NNS->NN if Pos:IN@[-1] & Word:asbestos@[0]
4 4 0 0 | NN->-NONE- if Pos:WP@[-1]
4 4 0 3 | NN->NNP if Pos:-NONE-@[-1]
4 6 2 2 | NN->NNP if Pos:NNP@[-1]
4 4 0 0 | NNS->VBZ if Pos:PRP@[-1]
>>> tagger1.rules()[1:3]
(Rule('000', 'NN', 'VB', [(Pos([-1]),'MD')]), Rule('000', 'NN', 'IN', [(Pos([-1]),'NNS')]))
>>> tagger1.print_template_statistics(printunused=False)
TEMPLATE STATISTICS (TRAIN) 2 templates, 10 rules)
TRAIN ( 2417 tokens) initial 555 0.7704 final: 496 0.7948
#ID | Score (train) | #Rules | Template
--------------------------------------------
000 | 54 0.915 | 9 0.900 | Template(Pos([-1]))
001 | 5 0.085 | 1 0.100 | Template(Pos([-1]),Word([0]))
>>> tagger1.accuracy(gold_data)
0.769230...
>>> print(tagger1.evaluate_per_tag(gold_data, sort_by_count=True))
Tag | Prec. | Recall | F-measure
-------+--------+--------+-----------
NNP | 0.8298 | 0.3600 | 0.5021
NN | 0.4435 | 0.8364 | 0.5797
IN | 0.8476 | 0.9580 | 0.8994
DT | 0.9819 | 0.8859 | 0.9314
JJ | 0.8167 | 0.2970 | 0.4356
NNS | 0.7464 | 0.9630 | 0.8410
-NONE- | 1.0000 | 0.8414 | 0.9139
, | 1.0000 | 1.0000 | 1.0000
. | 1.0000 | 1.0000 | 1.0000
VBD | 0.6723 | 0.8696 | 0.7583
CD | 1.0000 | 0.9872 | 0.9935
CC | 1.0000 | 0.9355 | 0.9667
VB | 0.8103 | 0.8246 | 0.8174
VBN | 0.9130 | 0.4200 | 0.5753
RB | 0.7778 | 0.7447 | 0.7609
TO | 1.0000 | 1.0000 | 1.0000
VBZ | 0.9667 | 0.6905 | 0.8056
VBG | 0.6415 | 0.9444 | 0.7640
PRP$ | 1.0000 | 1.0000 | 1.0000
PRP | 1.0000 | 0.5556 | 0.7143
MD | 1.0000 | 1.0000 | 1.0000
VBP | 0.6316 | 0.6316 | 0.6316
POS | 1.0000 | 1.0000 | 1.0000
$ | 1.0000 | 0.8182 | 0.9000
'' | 1.0000 | 1.0000 | 1.0000
: | 1.0000 | 1.0000 | 1.0000
WDT | 0.4000 | 0.2000 | 0.2667
`` | 1.0000 | 1.0000 | 1.0000
JJR | 1.0000 | 0.5000 | 0.6667
NNPS | 0.0000 | 0.0000 | 0.0000
RBR | 1.0000 | 1.0000 | 1.0000
-LRB- | 0.0000 | 0.0000 | 0.0000
-RRB- | 0.0000 | 0.0000 | 0.0000
RP | 0.6667 | 0.6667 | 0.6667
EX | 0.5000 | 0.5000 | 0.5000
JJS | 0.0000 | 0.0000 | 0.0000
WP | 1.0000 | 1.0000 | 1.0000
PDT | 0.0000 | 0.0000 | 0.0000
AT | 0.0000 | 0.0000 | 0.0000
>>> print(tagger1.confusion(gold_data))
| - |
| - N - |
| L O R N P |
| R N R J J N N N P P P R R V V V V V W |
| ' B E B A C C D E I J J J M N N P N D O R P R B R T V B B B B B D W ` |
| $ ' , - - - . : T C D T X N J R S D N P S S T S P $ B R P O B D G N P Z T P ` |
-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
$ | <9> . . . . . . . . . . . . . . . . . 1 . . . . . . . . . . . 1 . . . . . . . . |
'' | . <10> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
, | . .<115> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
-LRB- | . . . <.> . . . . . . . . . 1 . . . . 2 . . . . . . . . . . . . . . . . . . . . |
-NONE- | . . . .<122> . . . . . . . . 1 . . . . 22 . . . . . . . . . . . . . . . . . . . . |
-RRB- | . . . . . <.> . . . . . . . . . . . . 2 1 . . . . . . . . . . . . . . . . . . . |
. | . . . . . .<100> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
: | . . . . . . . <10> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
AT | . . . . . . . . <.> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
CC | . . . . . . . . . <58> . . . . . . . . 2 1 . . . . . . . . . . . . . . 1 . . . . |
CD | . . . . . . . . . . <77> . . . . . . . 1 . . . . . . . . . . . . . . . . . . . . |
DT | . . . . . . . . 1 . .<163> . 5 . . . . 12 . . . . . . . . . . . . . . . . . 3 . . |
EX | . . . . . . . . . . . . <1> . . . . . 1 . . . . . . . . . . . . . . . . . . . . |
IN | . . . . . . . . . . . . .<228> . . . . 8 . . . . . . . . . . . . . 2 . . . . . . |
JJ | . . . . . . . . . . . . . 4 <49> . . . 79 4 . 4 . . . . 6 . . . 1 12 3 . 3 . . . . |
JJR | . . . . . . . . . . . . . 2 . <3> . . 1 . . . . . . . . . . . . . . . . . . . . |
JJS | . . . . . . . . . . . . . . . . <.> . 2 . . . . . . . . . . . . . . . . . . . . |
MD | . . . . . . . . . . . . . . . . . <19> . . . . . . . . . . . . . . . . . . . . . |
NN | . . . . . . . . . . . . . 7 9 . . .<271> 16 . 5 . . . . . . . . 7 . 9 . . . . . . |
NNP | . . . . . . . . . . . 2 . 7 . . . . 163<117> . 26 . . . . 2 . . . 1 2 5 . . . . . . |
NNPS | . . . . . . . . . . . . . . . . . . . 1 <.> 3 . . . . . . . . . . . . . . . . . |
NNS | . . . . . . . . . . . . . . . . . . 5 . .<156> . . . . . . . . . . . . . 1 . . . |
PDT | . . . . . . . . . . . 1 . . . . . . . . . . <.> . . . . . . . . . . . . . . . . |
POS | . . . . . . . . . . . . . . . . . . . . . . . <14> . . . . . . . . . . . . . . . |
PRP | . . . . . . . . . . . . . . . . . . 10 . . 2 . . <15> . . . . . . . . . . . . . . |
PRP$ | . . . . . . . . . . . . . . . . . . . . . . . . . <28> . . . . . . . . . . . . . |
RB | . . . . . . . . . . . . 1 4 . . . . 6 . . . . . . . <35> . 1 . . . . . . . . . . |
RBR | . . . . . . . . . . . . . . . . . . . . . . . . . . . <4> . . . . . . . . . . . |
RP | . . . . . . . . . . . . . . . . . . . . . . . . . . 1 . <2> . . . . . . . . . . |
TO | . . . . . . . . . . . . . . . . . . . . . . . . . . . . . <47> . . . . . . . . . |
VB | . . . . . . . . . . . . . . 2 . . . 4 . . . . . . . 1 . . . <47> . . . 3 . . . . |
VBD | . . . . . . . . . . . . . 1 . . . . 8 1 . . . . . . . . . . . <80> . 2 . . . . . |
VBG | . . . . . . . . . . . . . . . . . . 2 . . . . . . . . . . . . . <34> . . . . . . |
VBN | . . . . . . . . . . . . . . . . . . 4 . . . . . . . . . . . . 25 . <21> . . . . . |
VBP | . . . . . . . . . . . . . 2 . . . . 4 . . . . . . . . . . . 1 . . . <12> . . . . |
VBZ | . . . . . . . . . . . . . . . . . . . . . 13 . . . . . . . . . . . . . <29> . . . |
WDT | . . . . . . . . . . . . . 7 . . . . 1 . . . . . . . . . . . . . . . . . <2> . . |
WP | . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . <2> . |
`` | . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . <10>|
-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
(row = reference; col = test)
>>> tagged, test_stats = tagger1.batch_tag_incremental(testing_data, gold_data)
>>> tagged[33][12:]
[('foreign', 'NN'), ('debt', 'NN'), ('of', 'IN'), ('$', '$'), ('64', 'CD'),
('billion', 'CD'), ('*U*', '-NONE-'), ('--', ':'), ('the', 'DT'), ('third-highest', 'NN'),
('in', 'IN'), ('the', 'DT'), ('developing', 'VBG'), ('world', 'NN'), ('.', '.')]
Regression Tests¶
Sequential Taggers¶
- Add tests for:
make sure backoff is being done correctly.
make sure ngram taggers don’t use previous sentences for context.
make sure ngram taggers see ‘beginning of the sentence’ as a unique context
make sure regexp tagger’s regexps are tried in order
train on some simple examples, & make sure that the size & the generated models are correct.
make sure cutoff works as intended
make sure that ngram models only exclude contexts covered by the backoff tagger if the backoff tagger gets that context correct at all locations.
Regression Testing for issue #1025¶
We want to ensure that a RegexpTagger can be created with more than 100 patterns and does not fail with: “AssertionError: sorry, but this version only supports 100 named groups”
>>> from nltk.tag import RegexpTagger
>>> patterns = [(str(i), 'NNP',) for i in range(200)]
>>> tagger = RegexpTagger(patterns)
Regression Testing for issue #2483¶
Ensure that tagging with pos_tag (PerceptronTagger) does not throw an IndexError when attempting tagging an empty string. What it must return instead is not strictly defined.
>>> from nltk.tag import pos_tag
>>> pos_tag(['', 'is', 'a', 'beautiful', 'day'])
[...]