TokenizedReader

Synopsis

Reads a tokenized corpus: one token per line, empty line separates sentence.

Description

Reads a tokenized corpus: one token per line, empty line separates sentence.

Snippet

📋

<tokenizedreader class="TokenizedReader">
    <source></source>
</tokenizedreader>

Mandatory parameters

source

Path to the file or directory containing the tokenized text.

Optional parameters

constantAnnotationFeatures

Constant features to add to each annotation created by this module.

constantDocumentFeatures

Constant features to add to each document created by this module.

constantSectionFeatures

Constant features to add to each section created by this module.

section

Name of the section containing the tokenized text.

sentenceLayer

Name of the sentence layer.

tokenLayer

Name of the token layer.

Deprecated parameters

sectionName

Deprecated alias for section .

sentenceLayerName

Deprecated alias for sentenceLayer .

tokenLayerName

Deprecated alias for tokenLayer .