nltk.tokenize.api module

Tokenizer Interface

class nltk.tokenize.api.StringTokenizer

Bases: TokenizerI

A tokenizer that divides a string into substrings by splitting on the specified string (defined in subclasses).

span_tokenize(s)

Identify the tokens using integer offsets (start_i, end_i), where s[start_i:end_i] is the corresponding token.

Return type:

Iterator[Tuple[int, int]]

tokenize(s)

Return a tokenized copy of s.

Return type:

List[str]

class nltk.tokenize.api.TokenizerI

Bases: ABC

A processing interface for tokenizing a string. Subclasses must define tokenize() or tokenize_sents() (or both).

span_tokenize(s: str) → Iterator[Tuple[int, int]]

Identify the tokens using integer offsets (start_i, end_i), where s[start_i:end_i] is the corresponding token.

Return type:

Iterator[Tuple[int, int]]

Parameters:

s (str)

span_tokenize_sents(strings: List[str]) → Iterator[List[Tuple[int, int]]]

Apply self.span_tokenize() to each element of strings. I.e.:

return [self.span_tokenize(s) for s in strings]

Yield:

List[Tuple[int, int]]

Parameters:

strings (List[str])

Return type:

Iterator[List[Tuple[int, int]]]

abstractmethod tokenize(s: str) → List[str]

Return a tokenized copy of s.

Return type:

List[str]

Parameters:

s (str)

tokenize_sents(strings: List[str]) → List[List[str]]

Apply self.tokenize() to each element of strings. I.e.:

return [self.tokenize(s) for s in strings]

Return type:

List[List[str]]

Parameters:

strings (List[str])