[ https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Martin Wiesner updated OPENNLP-1539: ------------------------------------ Description: [Classic (legacy) POS models|https://opennlp.sourceforge.net/models-1.5/] output tags in the [PENN Treebank POS tag|https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html] format. The modern UD-based models, however, differ in the [longer output format|https://universaldependencies.org/u/pos/], e.g. "VB" (Penn) vs. "VERB" (UD). This results in mismatches and will cause existing IT / tests to fail, if executed. To provide compatibility for existing applications and/or use-cases, we need to provide a way to retrieve both POS formats. Aims: - Introduce a constructor parameter for POSTaggerME to configure tag format / style: Penn or UD style - Implement a mapping between both POS tag formats: UD <==> Penn - Update the OpenNLP Manual to explain differences of POS tag format and configuration parameter Conceptual idea: - {{new POSTaggerME("en")}} => by _default_: UD format "as is" - {{new POSTaggerME("en", POSTagFormat.PENN)}} => by _intention_, here: Penn style Benefit: 1. It should be explicit so devs / user see what they will get via {{POSTagFormat}}. Enum values: POSTagFormat.UD, POSTagFormat.PENN, POSTagFormat.DEFAULT 2. IT tests can now be formulated to work on both modern and legacy models. was: [Classic (legacy) POS models|https://opennlp.sourceforge.net/models-1.5/] output tags in the [PENN Treebank POS tag|https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html] format. The modern UD-based models, however, differ in the longer output format, e.g. "VB" (Penn) vs. "VERB" (UD). This results in mismatches and will cause existing IT / tests to fail, if executed. To provide compatibility for existing applications and/or use-cases, we need to provide a way to retrieve both POS formats. Aims: - Introduce a constructor parameter for POSTaggerME to configure tag format / style: Penn or UD style - Implement a mapping between both POS tag formats: UD <==> Penn - Update the OpenNLP Manual to explain differences of POS tag format and configuration parameter Conceptual idea: - {{new POSTaggerME("en")}} => by _default_: UD format "as is" - {{new POSTaggerME("en", POSTagFormat.PENN)}} => by _intention_, here: Penn style Benefit: 1. It should be explicit so devs / user see what they will get via {{POSTagFormat}}. Enum values: POSTagFormat.UD, POSTagFormat.PENN, POSTagFormat.DEFAULT 2. IT tests can now be formulated to work on both modern and legacy models. > Introduce parameter for POSTaggerME to configure output POS tag format > ---------------------------------------------------------------------- > > Key: OPENNLP-1539 > URL: https://issues.apache.org/jira/browse/OPENNLP-1539 > Project: OpenNLP > Issue Type: Improvement > Components: POS Tagger > Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0 > Reporter: Martin Wiesner > Priority: Major > > [Classic (legacy) POS models|https://opennlp.sourceforge.net/models-1.5/] > output tags in the [PENN Treebank POS > tag|https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html] > format. > The modern UD-based models, however, differ in the [longer output > format|https://universaldependencies.org/u/pos/], e.g. "VB" (Penn) vs. "VERB" > (UD). This results in mismatches and will cause existing IT / tests to fail, > if executed. > To provide compatibility for existing applications and/or use-cases, we need > to provide a way to retrieve both POS formats. > Aims: > - Introduce a constructor parameter for POSTaggerME to configure tag format / > style: Penn or UD style > - Implement a mapping between both POS tag formats: UD <==> Penn > - Update the OpenNLP Manual to explain differences of POS tag format and > configuration parameter > Conceptual idea: > - {{new POSTaggerME("en")}} => by _default_: UD format "as is" > - {{new POSTaggerME("en", POSTagFormat.PENN)}} => by _intention_, here: Penn > style > Benefit: > 1. It should be explicit so devs / user see what they will get via > {{POSTagFormat}}. Enum values: POSTagFormat.UD, POSTagFormat.PENN, > POSTagFormat.DEFAULT > 2. IT tests can now be formulated to work on both modern and legacy models. -- This message was sent by Atlassian Jira (v8.20.10#820010)