corpus processing engine

How to write a plan

The basics

The plan file is an XML file that specifies the sequence of modules and their parameters.

The top-level tag of a plan file is alvisnlp-plan:

<alvisnlp-plan id="foo">

The id attribute is mandatory, its value is the identifier of the plan. Make sure to fill a value that has some meaning for you.

Then, the alvisnlp-plan contains several module tags:

<alvisnlp-plan id="foo">
<read class="TextFileReader">

<words class="WoSMig">


Each tag specifies an AlvisNLP module that will process the corpus. The class attribute is mandatory, it specifies the class of the module, that is what it does on the corpus. The value must be a supported module class.

Finally, each module tag contains tags that specify the parameter values for the module:

<alvisnlp-plan id="foo">
<read class="TextFileReader">

<words class="WoSMig"/>


In this example, we set two parameters in the module read: sourcePath and sectionName, and no parameter in words. The module class documentation should specify which parameters are supported, which are mandatory and what are the default values.

Parameter value conversion

The type of a parameter and conversion of the contents of the tag into the type are documented in the module class description. In this section we review the most used parameter types:


The contents of the tag must be a character string. Leading and trailing whitespaces are trimmed.


The contents of the tag must be a character string. The conversion is in base 10, with an optional a leading sign symbol (+ or -). Leading and trailing whitespaces are trimmed.


The contents of the tag must be a character string.

true values false values
true false
on off
yes no

Leading and trailing whitespaces are trimmed.

SourceStream and TargetStream

A SourceStream and TargetStreamrepresents a file, a directory or an internet resource. SourceStream are input resources, while TargetStream represent outputs. The contents of the tag is either a path in the local filesystem or an URL to an remote resource. The conversion supports the main internet protocols (http, https, ftp) or the standard streams (stdin, stdout, stderr). It also supports the concatenation of several resources, the specification of all files in a directory, with filename filters. It also supports compression schemes.

File, InputFile, InputDirectory, OutputDirectory, OutputFile, ExecutableFile

These types represent resources in the local filesystem, their value cannot be remote URLs. AlvisNLP will check the values according to the type:

parameter type file exists file type permissions
InputDirectory yes directory rx
InputFile yes regular r
OutputDirectory yes 1 directory 1 rwx 1
OutputFile yes 1 directory or regular 2 rwx 2
ExecutableFile yes regular rx
File no regular 3  

1 some ancestor must exist and be a writable directory
2 if the file exists then it must be regular and writable, the innermost existing ancestor must be a writable directory
3 if the file exists then it must be regular


Expression parameters are evaluated only when the module is processing. Expressions allow to set values that depend on the state of the corpus. The value of the tag must follow the element expression syntax. See [[Element expression examples]] and [[Element expression reference]].


Array parameter types can be recognized by a pair of brackets at the end of the type ([]). An array value is a sequence of values of the same type. You can either specify the elements separated with commas (,), or each element inside an enclosed tag (of arbitrary name).

The parameter tag may specify an alternate separator character with the separator attribute.


Mapping parameter types map strings to values, where all values of the mapping are of the same type. The mapping can be specified by separating each entry with commas (,), and by separating the key from the value with an equal sign (=). Alternatively each entry can be specified with a tag, whose name is the entry key, end the contents is converted into a value.

The parameter tag may specify an alternate separator character with the separator attribute, and an alternate key-value separator with the qualifier attribute.

Complex types

Some modules accept parameters with more complex and composite types. Refer to the documentation in the converter reference.


Sequences are sub-parts of the plan that contain modules (or other sequences):

<alvisnlp-plan id="foo">
<read class="TextFileReader">

<words class="WoSMig"/>
<sentences class="SeSMig"/>


Sequences do not alter the order of processing, their purpose is the organization of modules in logical bundles. Note that sequences affect the logging and may help you to read the AlvisNLP log.

Plan import

Plans can be reused inside other plans:

<alvisnlp-plan id="foo">
<read class="TextFileReader">

<import file="/path/to/another/plan.xml"/>

In this example, all the modules specified in /path/to/another/plan.xml will process the corpus as if the plan file had been included.

Plan-level parameters

You can define parameters for the whole plan, so you can set these parameters when the plan is imported.

A Plan-level parameter looks like this in project_species.xml:

<param name="speciesFile">
<alias module="project.species" param="dictFile" />

<species class="SimpleContentsProjector">
<!-- ... snip ... -->

An import of this file could look like:

<import file="project_species.xml">
<speciesFile value="/bibdev/resources/..."/>

This will import the plan specified in project_species.xml and set the parameter speciesFile. Since speciesFile is defined as an alias to the dictFile parameter in project.species, then it will be set.

A Plan-level parameter can be an alias for several parameters in different modules. When importing and setting these parameters, all aliases will have the same value. You have no excuses left: make modular plans!

More on parameters

The parameter tag may have attributes that change the conversion:

Option Effect
inhibitCheck="true" prevents AlvisNLP from checking this parameter value, for instance it will not check for the existence of InputFile parameters.
separator="C" sets the separator character between array elements or mapping entries (default: ,).
qualifier="C" sets the separator character between the key and the value of a mapping entry (default: =).
trim="false" prevents AlvisNLP from trimming leading and trailing whitespaces off the parameter value.
load="..." loads the specified file. This file must be an XML file, AlvisNLP sets the parameter value as if the parameter tag was the root element of this file. This attribute is useful for complex parameter values.

More attributes may be supported for the conversion to specific types.

Command-line control

The plan, especially parameter values, can be controlled from the command line.


alvisnlp -param MODULE PARAM VALUE

The -param option sets the value of parameter PARAM in MODULE. MODULE is the identifier of a module specified in the plan. If the module is inside a sequence, then its identifier is in the form SEQUENCE.MODULE.
The VALUE is a string and it is converted as if it was the contents of the parameter tag.


alvisnlp -param MODULE PARAM XVALUE

The -xparam option behaves the same way as -param but it expects an XML tag instead of a string value. This option is useful if you want to set parameters with conversion options.


alvisnlp -feat KEY VALUE

The -feat options adds a feature pair to the corpus before the processing starts. Expression parameters can get the value of this feature to alter the behavior of the modules.


alvisnlp -entity NAME REPLACEMENT

This option defines an XML entity that is used when the plan file is parsed.


alvisnlp -environmentEntities

This option defines an XML entity for each environment variable.