Reference

Class

RiTa

Name

tokens

Description Return an array containing all unique alphabetical words (tokens) in the text.
Example

sentence = "One dog is like the other dog.";
tokens = RiTa.tokens(sentence);

sentence = "'One dog is like the other dog', she'd thought.";
tokens = RiTa.tokens(sentence, { splitContractions: true });

Parameters
Stringthe input
Object
(or Map<String, Object> in Java)
options (optional), the relevant options for the function:

{boolean} options.splitContractions:
Convert contractions (e.g., "I'd" or "she'll") into multiple individual tokens

{boolean} options.includePunct:
Include punctuation tokens in the output

{boolean} options.caseSensitive:
Treat differently cased Strings as separate tokens

{boolean} options.ignoreStopWords:
Ignore words like 'the', 'and', 'a', 'of', etc, as specified in RiTa.STOP_WORDS

{boolean} options.sort:
Return the result array in sorted order

Returns
String[]the set of unique alphabetical tokens in the text
Syntax
RiTa.tokens(text);
RiTa.tokens(text, options);
Platform Java / JavaScript