[
https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Wiesner updated OPENNLP-1539:
------------------------------------
Description:
[Classic (legacy) POS models|https://opennlp.sourceforge.net/models-1.5/]
output tags in the [PENN Treebank POS
tag|https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html]
format.
The modern UD-based models, however, differ in the [longer output
format|https://universaldependencies.org/u/pos/], e.g. "VB" (Penn) vs. "VERB"
(UD). Extended (UD) word features are covered here:
https://universaldependencies.org/u/feat/index.html
This difference results in mismatches and will cause existing IT / tests to
fail, if executed.
To provide compatibility for existing applications and/or use-cases, we need to
provide a way to retrieve both POS formats.
Aims:
- Introduce a constructor parameter for POSTaggerME to configure tag format /
style: Penn or UD style
- Implement a mapping between both POS tag formats: UD <==> Penn
- Update the OpenNLP Manual to explain differences of POS tag format and
configuration parameter
Conceptual idea:
- {{new POSTaggerME("en")}} => by _default_: UD format "as is"
- {{new POSTaggerME("en", POSTagFormat.PENN)}} => by _intention_, here: Penn
style
Benefit:
1. It should be explicit so devs / user see what they will get via
{{POSTagFormat}}. Enum values: POSTagFormat.UD, POSTagFormat.PENN,
POSTagFormat.DEFAULT
2. IT tests can now be formulated to work on both modern and legacy models.
was:
[Classic (legacy) POS models|https://opennlp.sourceforge.net/models-1.5/]
output tags in the [PENN Treebank POS
tag|https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html]
format.
The modern UD-based models, however, differ in the [longer output
format|https://universaldependencies.org/u/pos/], e.g. "VB" (Penn) vs. "VERB"
(UD). Extended features (UD) are covered here:
https://universaldependencies.org/u/feat/index.html
This difference results in mismatches and will cause existing IT / tests to
fail, if executed.
To provide compatibility for existing applications and/or use-cases, we need to
provide a way to retrieve both POS formats.
Aims:
- Introduce a constructor parameter for POSTaggerME to configure tag format /
style: Penn or UD style
- Implement a mapping between both POS tag formats: UD <==> Penn
- Update the OpenNLP Manual to explain differences of POS tag format and
configuration parameter
Conceptual idea:
- {{new POSTaggerME("en")}} => by _default_: UD format "as is"
- {{new POSTaggerME("en", POSTagFormat.PENN)}} => by _intention_, here: Penn
style
Benefit:
1. It should be explicit so devs / user see what they will get via
{{POSTagFormat}}. Enum values: POSTagFormat.UD, POSTagFormat.PENN,
POSTagFormat.DEFAULT
2. IT tests can now be formulated to work on both modern and legacy models.
> Introduce parameter for POSTaggerME to configure output POS tag format
> ----------------------------------------------------------------------
>
> Key: OPENNLP-1539
> URL: https://issues.apache.org/jira/browse/OPENNLP-1539
> Project: OpenNLP
> Issue Type: Improvement
> Components: POS Tagger
> Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
> Reporter: Martin Wiesner
> Priority: Major
>
> [Classic (legacy) POS models|https://opennlp.sourceforge.net/models-1.5/]
> output tags in the [PENN Treebank POS
> tag|https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html]
> format.
> The modern UD-based models, however, differ in the [longer output
> format|https://universaldependencies.org/u/pos/], e.g. "VB" (Penn) vs. "VERB"
> (UD). Extended (UD) word features are covered here:
> https://universaldependencies.org/u/feat/index.html
> This difference results in mismatches and will cause existing IT / tests to
> fail, if executed.
> To provide compatibility for existing applications and/or use-cases, we need
> to provide a way to retrieve both POS formats.
> Aims:
> - Introduce a constructor parameter for POSTaggerME to configure tag format /
> style: Penn or UD style
> - Implement a mapping between both POS tag formats: UD <==> Penn
> - Update the OpenNLP Manual to explain differences of POS tag format and
> configuration parameter
> Conceptual idea:
> - {{new POSTaggerME("en")}} => by _default_: UD format "as is"
> - {{new POSTaggerME("en", POSTagFormat.PENN)}} => by _intention_, here: Penn
> style
> Benefit:
> 1. It should be explicit so devs / user see what they will get via
> {{POSTagFormat}}. Enum values: POSTagFormat.UD, POSTagFormat.PENN,
> POSTagFormat.DEFAULT
> 2. IT tests can now be formulated to work on both modern and legacy models.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
