[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format
[ https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17850265#comment-17850265 ] ASF GitHub Bot commented on OPENNLP-1539: - mawiesne merged PR #601: URL: https://github.com/apache/opennlp/pull/601 > Introduce parameter for POSTaggerME to configure output POS tag format > -- > > Key: OPENNLP-1539 > URL: https://issues.apache.org/jira/browse/OPENNLP-1539 > Project: OpenNLP > Issue Type: Improvement > Components: POS Tagger >Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0 >Reporter: Martin Wiesner >Assignee: Richard Zowalla >Priority: Major > Fix For: 2.4.0 > > > [Classic (legacy) POS models|https://opennlp.sourceforge.net/models-1.5/] > output tags in the [PENN Treebank POS > tag|https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html] > format. > The modern UD-based models, however, differ in the [longer output > format|https://universaldependencies.org/u/pos/], e.g. "VB" (Penn) vs. "VERB" > (UD). Extended (UD) word features are covered here: > https://universaldependencies.org/u/feat/index.html > This difference results in mismatches and will cause existing IT / tests to > fail, if executed. Luckily, a mapping table is found here: > https://universaldependencies.org/tagset-conversion/en-penn-uposf.html > To provide compatibility for existing applications and/or use-cases, we need > to provide a way to retrieve both POS formats. > Aims: > - Introduce a constructor parameter for POSTaggerME to configure tag format / > style: Penn or UD style > - Implement a mapping between both POS tag formats: UD <==> Penn > - Update the OpenNLP Manual to explain differences of POS tag format and > configuration parameter > Conceptual idea: > - {{new POSTaggerME("en")}} => by _default_: UD format "as is" > - {{new POSTaggerME("en", POSTagFormat.PENN)}} => by _intention_, here: Penn > style > Benefit: > 1. It should be explicit so devs / user see what they will get via > {{POSTagFormat}}. Enum values: POSTagFormat.UD, POSTagFormat.PENN, > POSTagFormat.DEFAULT > 2. IT tests can now be formulated to work on both modern and legacy models. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format
[ https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849223#comment-17849223 ] ASF GitHub Bot commented on OPENNLP-1539: - rzo1 commented on PR #601: URL: https://github.com/apache/opennlp/pull/601#issuecomment-2128979449 > @rzo1 Did you run into any weirdness with the older Sourceforge models? Added tests. They look good. > Introduce parameter for POSTaggerME to configure output POS tag format > -- > > Key: OPENNLP-1539 > URL: https://issues.apache.org/jira/browse/OPENNLP-1539 > Project: OpenNLP > Issue Type: Improvement > Components: POS Tagger >Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0 >Reporter: Martin Wiesner >Assignee: Richard Zowalla >Priority: Major > Fix For: 2.4.0 > > > [Classic (legacy) POS models|https://opennlp.sourceforge.net/models-1.5/] > output tags in the [PENN Treebank POS > tag|https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html] > format. > The modern UD-based models, however, differ in the [longer output > format|https://universaldependencies.org/u/pos/], e.g. "VB" (Penn) vs. "VERB" > (UD). Extended (UD) word features are covered here: > https://universaldependencies.org/u/feat/index.html > This difference results in mismatches and will cause existing IT / tests to > fail, if executed. Luckily, a mapping table is found here: > https://universaldependencies.org/tagset-conversion/en-penn-uposf.html > To provide compatibility for existing applications and/or use-cases, we need > to provide a way to retrieve both POS formats. > Aims: > - Introduce a constructor parameter for POSTaggerME to configure tag format / > style: Penn or UD style > - Implement a mapping between both POS tag formats: UD <==> Penn > - Update the OpenNLP Manual to explain differences of POS tag format and > configuration parameter > Conceptual idea: > - {{new POSTaggerME("en")}} => by _default_: UD format "as is" > - {{new POSTaggerME("en", POSTagFormat.PENN)}} => by _intention_, here: Penn > style > Benefit: > 1. It should be explicit so devs / user see what they will get via > {{POSTagFormat}}. Enum values: POSTagFormat.UD, POSTagFormat.PENN, > POSTagFormat.DEFAULT > 2. IT tests can now be formulated to work on both modern and legacy models. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format
[ https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849194#comment-17849194 ] ASF GitHub Bot commented on OPENNLP-1539: - mawiesne commented on code in PR #601: URL: https://github.com/apache/opennlp/pull/601#discussion_r1612947718 ## opennlp-tools/src/main/java/opennlp/tools/postag/POSTagFormatMapper.java: ## @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package opennlp.tools.postag; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * A mapping implementation for converting between different POS tag formats. + * This class supports conversion between Penn Treebank (PENN) and Universal Dependencies (UD) formats. + * The conversion is based on the https://universaldependencies.org/tagset-conversion/en-penn-uposf.html;>Universal Dependencies conversion table. + * Please note that when converting from UD to Penn format, there may be ambiguity in some cases. + */ +public class POSTagFormatMapper { + + private static final Logger logger = LoggerFactory.getLogger(POSTagFormatMapper.class); + + private static final Map CONVERSION_TABLE_PENN_TO_UD = new HashMap<>(); + private static final Map CONVERSION_TABLE_UD_TO_PENN = new HashMap<>(); + + static { +/* + * This is a conversion table to convert PENN to UD format as described in + * https://universaldependencies.org/tagset-conversion/en-penn-uposf.html + */ +CONVERSION_TABLE_PENN_TO_UD.put("#", "SYM"); +CONVERSION_TABLE_PENN_TO_UD.put("$", "SYM"); +CONVERSION_TABLE_PENN_TO_UD.put("''", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put(",", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("-LRB-", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("-RRB-", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put(".", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put(":", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("AFX", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("CC", "CCONJ"); +CONVERSION_TABLE_PENN_TO_UD.put("CD", "NUM"); +CONVERSION_TABLE_PENN_TO_UD.put("DT", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("EX", "PRON"); +CONVERSION_TABLE_PENN_TO_UD.put("FW", "X"); +CONVERSION_TABLE_PENN_TO_UD.put("HYPH", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("IN", "ADP"); +CONVERSION_TABLE_PENN_TO_UD.put("JJ", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("JJR", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("JJS", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("LS", "X"); +CONVERSION_TABLE_PENN_TO_UD.put("MD", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("NIL", "X"); +CONVERSION_TABLE_PENN_TO_UD.put("NN", "NOUN"); +CONVERSION_TABLE_PENN_TO_UD.put("NNP", "PROPN"); +CONVERSION_TABLE_PENN_TO_UD.put("NNPS", "PROPN"); +CONVERSION_TABLE_PENN_TO_UD.put("NNS", "NOUN"); +CONVERSION_TABLE_PENN_TO_UD.put("PDT", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("POS", "PART"); +CONVERSION_TABLE_PENN_TO_UD.put("PRP", "PRON"); +CONVERSION_TABLE_PENN_TO_UD.put("PRP$", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("RB", "ADV"); +CONVERSION_TABLE_PENN_TO_UD.put("RBR", "ADV"); +CONVERSION_TABLE_PENN_TO_UD.put("RBS", "ADV"); +CONVERSION_TABLE_PENN_TO_UD.put("RP", "ADP"); +CONVERSION_TABLE_PENN_TO_UD.put("SYM", "SYM"); +CONVERSION_TABLE_PENN_TO_UD.put("TO", "PART"); +CONVERSION_TABLE_PENN_TO_UD.put("UH", "INTJ"); +CONVERSION_TABLE_PENN_TO_UD.put("VB", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBD", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBG", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBN", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBP", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBZ", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("WDT", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("WP", "PRON"); +CONVERSION_TABLE_PENN_TO_UD.put("WP$", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("WRB", "ADV"); + +/* + * Note: The back conversion might lose information. + */ +CONVERSION_TABLE_UD_TO_PENN.put("ADJ", "JJ"); +
[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format
[ https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849174#comment-17849174 ] ASF GitHub Bot commented on OPENNLP-1539: - rzo1 commented on code in PR #601: URL: https://github.com/apache/opennlp/pull/601#discussion_r1612885912 ## opennlp-tools/src/main/java/opennlp/tools/postag/POSTagFormatMapper.java: ## @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package opennlp.tools.postag; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * A mapping implementation for converting between different POS tag formats. + * This class supports conversion between Penn Treebank (PENN) and Universal Dependencies (UD) formats. + * The conversion is based on the https://universaldependencies.org/tagset-conversion/en-penn-uposf.html;>Universal Dependencies conversion table. + * Please note that when converting from UD to Penn format, there may be ambiguity in some cases. + */ +public class POSTagFormatMapper { + + private static final Logger logger = LoggerFactory.getLogger(POSTagFormatMapper.class); + + private static final Map CONVERSION_TABLE_PENN_TO_UD = new HashMap<>(); + private static final Map CONVERSION_TABLE_UD_TO_PENN = new HashMap<>(); + + static { +/* + * This is a conversion table to convert PENN to UD format as described in + * https://universaldependencies.org/tagset-conversion/en-penn-uposf.html + */ +CONVERSION_TABLE_PENN_TO_UD.put("#", "SYM"); +CONVERSION_TABLE_PENN_TO_UD.put("$", "SYM"); +CONVERSION_TABLE_PENN_TO_UD.put("''", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put(",", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("-LRB-", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("-RRB-", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put(".", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put(":", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("AFX", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("CC", "CCONJ"); +CONVERSION_TABLE_PENN_TO_UD.put("CD", "NUM"); +CONVERSION_TABLE_PENN_TO_UD.put("DT", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("EX", "PRON"); +CONVERSION_TABLE_PENN_TO_UD.put("FW", "X"); +CONVERSION_TABLE_PENN_TO_UD.put("HYPH", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("IN", "ADP"); +CONVERSION_TABLE_PENN_TO_UD.put("JJ", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("JJR", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("JJS", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("LS", "X"); +CONVERSION_TABLE_PENN_TO_UD.put("MD", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("NIL", "X"); +CONVERSION_TABLE_PENN_TO_UD.put("NN", "NOUN"); +CONVERSION_TABLE_PENN_TO_UD.put("NNP", "PROPN"); +CONVERSION_TABLE_PENN_TO_UD.put("NNPS", "PROPN"); +CONVERSION_TABLE_PENN_TO_UD.put("NNS", "NOUN"); +CONVERSION_TABLE_PENN_TO_UD.put("PDT", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("POS", "PART"); +CONVERSION_TABLE_PENN_TO_UD.put("PRP", "PRON"); +CONVERSION_TABLE_PENN_TO_UD.put("PRP$", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("RB", "ADV"); +CONVERSION_TABLE_PENN_TO_UD.put("RBR", "ADV"); +CONVERSION_TABLE_PENN_TO_UD.put("RBS", "ADV"); +CONVERSION_TABLE_PENN_TO_UD.put("RP", "ADP"); +CONVERSION_TABLE_PENN_TO_UD.put("SYM", "SYM"); +CONVERSION_TABLE_PENN_TO_UD.put("TO", "PART"); +CONVERSION_TABLE_PENN_TO_UD.put("UH", "INTJ"); +CONVERSION_TABLE_PENN_TO_UD.put("VB", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBD", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBG", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBN", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBP", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBZ", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("WDT", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("WP", "PRON"); +CONVERSION_TABLE_PENN_TO_UD.put("WP$", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("WRB", "ADV"); + +/* + * Note: The back conversion might lose information. + */ +CONVERSION_TABLE_UD_TO_PENN.put("ADJ", "JJ"); +
[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format
[ https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849172#comment-17849172 ] ASF GitHub Bot commented on OPENNLP-1539: - rzo1 commented on code in PR #601: URL: https://github.com/apache/opennlp/pull/601#discussion_r1612883541 ## opennlp-tools/src/main/java/opennlp/tools/postag/POSTagFormatMapper.java: ## @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package opennlp.tools.postag; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * A mapping implementation for converting between different POS tag formats. + * This class supports conversion between Penn Treebank (PENN) and Universal Dependencies (UD) formats. + * The conversion is based on the https://universaldependencies.org/tagset-conversion/en-penn-uposf.html;>Universal Dependencies conversion table. + * Please note that when converting from UD to Penn format, there may be ambiguity in some cases. + */ +public class POSTagFormatMapper { + + private static final Logger logger = LoggerFactory.getLogger(POSTagFormatMapper.class); + + private static final Map CONVERSION_TABLE_PENN_TO_UD = new HashMap<>(); + private static final Map CONVERSION_TABLE_UD_TO_PENN = new HashMap<>(); + + static { +/* + * This is a conversion table to convert PENN to UD format as described in + * https://universaldependencies.org/tagset-conversion/en-penn-uposf.html + */ +CONVERSION_TABLE_PENN_TO_UD.put("#", "SYM"); +CONVERSION_TABLE_PENN_TO_UD.put("$", "SYM"); +CONVERSION_TABLE_PENN_TO_UD.put("''", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put(",", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("-LRB-", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("-RRB-", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put(".", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put(":", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("AFX", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("CC", "CCONJ"); +CONVERSION_TABLE_PENN_TO_UD.put("CD", "NUM"); +CONVERSION_TABLE_PENN_TO_UD.put("DT", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("EX", "PRON"); +CONVERSION_TABLE_PENN_TO_UD.put("FW", "X"); +CONVERSION_TABLE_PENN_TO_UD.put("HYPH", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("IN", "ADP"); +CONVERSION_TABLE_PENN_TO_UD.put("JJ", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("JJR", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("JJS", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("LS", "X"); +CONVERSION_TABLE_PENN_TO_UD.put("MD", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("NIL", "X"); +CONVERSION_TABLE_PENN_TO_UD.put("NN", "NOUN"); +CONVERSION_TABLE_PENN_TO_UD.put("NNP", "PROPN"); +CONVERSION_TABLE_PENN_TO_UD.put("NNPS", "PROPN"); +CONVERSION_TABLE_PENN_TO_UD.put("NNS", "NOUN"); +CONVERSION_TABLE_PENN_TO_UD.put("PDT", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("POS", "PART"); +CONVERSION_TABLE_PENN_TO_UD.put("PRP", "PRON"); +CONVERSION_TABLE_PENN_TO_UD.put("PRP$", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("RB", "ADV"); +CONVERSION_TABLE_PENN_TO_UD.put("RBR", "ADV"); +CONVERSION_TABLE_PENN_TO_UD.put("RBS", "ADV"); +CONVERSION_TABLE_PENN_TO_UD.put("RP", "ADP"); +CONVERSION_TABLE_PENN_TO_UD.put("SYM", "SYM"); +CONVERSION_TABLE_PENN_TO_UD.put("TO", "PART"); +CONVERSION_TABLE_PENN_TO_UD.put("UH", "INTJ"); +CONVERSION_TABLE_PENN_TO_UD.put("VB", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBD", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBG", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBN", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBP", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBZ", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("WDT", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("WP", "PRON"); +CONVERSION_TABLE_PENN_TO_UD.put("WP$", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("WRB", "ADV"); + +/* + * Note: The back conversion might lose information. + */ +CONVERSION_TABLE_UD_TO_PENN.put("ADJ", "JJ"); +
[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format
[ https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849158#comment-17849158 ] ASF GitHub Bot commented on OPENNLP-1539: - mawiesne commented on PR #601: URL: https://github.com/apache/opennlp/pull/601#issuecomment-2128552371 Thx @rzo1 - I left some comments to improve clarity of the doc for the changes in the API. > Introduce parameter for POSTaggerME to configure output POS tag format > -- > > Key: OPENNLP-1539 > URL: https://issues.apache.org/jira/browse/OPENNLP-1539 > Project: OpenNLP > Issue Type: Improvement > Components: POS Tagger >Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0 >Reporter: Martin Wiesner >Assignee: Richard Zowalla >Priority: Major > Fix For: 2.4.0 > > > [Classic (legacy) POS models|https://opennlp.sourceforge.net/models-1.5/] > output tags in the [PENN Treebank POS > tag|https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html] > format. > The modern UD-based models, however, differ in the [longer output > format|https://universaldependencies.org/u/pos/], e.g. "VB" (Penn) vs. "VERB" > (UD). Extended (UD) word features are covered here: > https://universaldependencies.org/u/feat/index.html > This difference results in mismatches and will cause existing IT / tests to > fail, if executed. Luckily, a mapping table is found here: > https://universaldependencies.org/tagset-conversion/en-penn-uposf.html > To provide compatibility for existing applications and/or use-cases, we need > to provide a way to retrieve both POS formats. > Aims: > - Introduce a constructor parameter for POSTaggerME to configure tag format / > style: Penn or UD style > - Implement a mapping between both POS tag formats: UD <==> Penn > - Update the OpenNLP Manual to explain differences of POS tag format and > configuration parameter > Conceptual idea: > - {{new POSTaggerME("en")}} => by _default_: UD format "as is" > - {{new POSTaggerME("en", POSTagFormat.PENN)}} => by _intention_, here: Penn > style > Benefit: > 1. It should be explicit so devs / user see what they will get via > {{POSTagFormat}}. Enum values: POSTagFormat.UD, POSTagFormat.PENN, > POSTagFormat.DEFAULT > 2. IT tests can now be formulated to work on both modern and legacy models. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format
[ https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849157#comment-17849157 ] ASF GitHub Bot commented on OPENNLP-1539: - mawiesne commented on code in PR #601: URL: https://github.com/apache/opennlp/pull/601#discussion_r1612745224 ## opennlp-tools/src/main/java/opennlp/tools/postag/POSTagFormatMapper.java: ## @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package opennlp.tools.postag; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * A mapping implementation for converting between different POS tag formats. + * This class supports conversion between Penn Treebank (PENN) and Universal Dependencies (UD) formats. + * The conversion is based on the https://universaldependencies.org/tagset-conversion/en-penn-uposf.html;>Universal Dependencies conversion table. + * Please note that when converting from UD to Penn format, there may be ambiguity in some cases. + */ +public class POSTagFormatMapper { + + private static final Logger logger = LoggerFactory.getLogger(POSTagFormatMapper.class); + + private static final Map CONVERSION_TABLE_PENN_TO_UD = new HashMap<>(); + private static final Map CONVERSION_TABLE_UD_TO_PENN = new HashMap<>(); + + static { +/* + * This is a conversion table to convert PENN to UD format as described in + * https://universaldependencies.org/tagset-conversion/en-penn-uposf.html + */ +CONVERSION_TABLE_PENN_TO_UD.put("#", "SYM"); +CONVERSION_TABLE_PENN_TO_UD.put("$", "SYM"); +CONVERSION_TABLE_PENN_TO_UD.put("''", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put(",", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("-LRB-", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("-RRB-", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put(".", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put(":", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("AFX", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("CC", "CCONJ"); +CONVERSION_TABLE_PENN_TO_UD.put("CD", "NUM"); +CONVERSION_TABLE_PENN_TO_UD.put("DT", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("EX", "PRON"); +CONVERSION_TABLE_PENN_TO_UD.put("FW", "X"); +CONVERSION_TABLE_PENN_TO_UD.put("HYPH", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("IN", "ADP"); +CONVERSION_TABLE_PENN_TO_UD.put("JJ", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("JJR", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("JJS", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("LS", "X"); +CONVERSION_TABLE_PENN_TO_UD.put("MD", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("NIL", "X"); +CONVERSION_TABLE_PENN_TO_UD.put("NN", "NOUN"); +CONVERSION_TABLE_PENN_TO_UD.put("NNP", "PROPN"); +CONVERSION_TABLE_PENN_TO_UD.put("NNPS", "PROPN"); +CONVERSION_TABLE_PENN_TO_UD.put("NNS", "NOUN"); +CONVERSION_TABLE_PENN_TO_UD.put("PDT", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("POS", "PART"); +CONVERSION_TABLE_PENN_TO_UD.put("PRP", "PRON"); +CONVERSION_TABLE_PENN_TO_UD.put("PRP$", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("RB", "ADV"); +CONVERSION_TABLE_PENN_TO_UD.put("RBR", "ADV"); +CONVERSION_TABLE_PENN_TO_UD.put("RBS", "ADV"); +CONVERSION_TABLE_PENN_TO_UD.put("RP", "ADP"); +CONVERSION_TABLE_PENN_TO_UD.put("SYM", "SYM"); +CONVERSION_TABLE_PENN_TO_UD.put("TO", "PART"); +CONVERSION_TABLE_PENN_TO_UD.put("UH", "INTJ"); +CONVERSION_TABLE_PENN_TO_UD.put("VB", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBD", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBG", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBN", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBP", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBZ", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("WDT", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("WP", "PRON"); +CONVERSION_TABLE_PENN_TO_UD.put("WP$", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("WRB", "ADV"); + +/* + * Note: The back conversion might lose information. + */ +CONVERSION_TABLE_UD_TO_PENN.put("ADJ", "JJ"); +
[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format
[ https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849055#comment-17849055 ] ASF GitHub Bot commented on OPENNLP-1539: - jzonthemtn commented on code in PR #601: URL: https://github.com/apache/opennlp/pull/601#discussion_r1612105341 ## opennlp-tools/src/main/java/opennlp/tools/postag/POSTagFormatMapper.java: ## @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package opennlp.tools.postag; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * A mapping implementation for converting between different POS tag formats. + * This class supports conversion between Penn Treebank (PENN) and Universal Dependencies (UD) formats. + * The conversion is based on the https://universaldependencies.org/tagset-conversion/en-penn-uposf.html;>Universal Dependencies conversion table. + * Please note that when converting from UD to Penn format, there may be ambiguity in some cases. + */ +public class POSTagFormatMapper { + + private static final Logger logger = LoggerFactory.getLogger(POSTagFormatMapper.class); + + private static final Map CONVERSION_TABLE_PENN_TO_UD = new HashMap<>(); + private static final Map CONVERSION_TABLE_UD_TO_PENN = new HashMap<>(); + + static { +/* + * This is a conversion table to convert PENN to UD format as described in + * https://universaldependencies.org/tagset-conversion/en-penn-uposf.html + */ +CONVERSION_TABLE_PENN_TO_UD.put("#", "SYM"); +CONVERSION_TABLE_PENN_TO_UD.put("$", "SYM"); +CONVERSION_TABLE_PENN_TO_UD.put("''", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put(",", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("-LRB-", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("-RRB-", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put(".", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put(":", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("AFX", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("CC", "CCONJ"); +CONVERSION_TABLE_PENN_TO_UD.put("CD", "NUM"); +CONVERSION_TABLE_PENN_TO_UD.put("DT", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("EX", "PRON"); +CONVERSION_TABLE_PENN_TO_UD.put("FW", "X"); +CONVERSION_TABLE_PENN_TO_UD.put("HYPH", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("IN", "ADP"); +CONVERSION_TABLE_PENN_TO_UD.put("JJ", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("JJR", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("JJS", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("LS", "X"); +CONVERSION_TABLE_PENN_TO_UD.put("MD", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("NIL", "X"); +CONVERSION_TABLE_PENN_TO_UD.put("NN", "NOUN"); +CONVERSION_TABLE_PENN_TO_UD.put("NNP", "PROPN"); +CONVERSION_TABLE_PENN_TO_UD.put("NNPS", "PROPN"); +CONVERSION_TABLE_PENN_TO_UD.put("NNS", "NOUN"); +CONVERSION_TABLE_PENN_TO_UD.put("PDT", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("POS", "PART"); +CONVERSION_TABLE_PENN_TO_UD.put("PRP", "PRON"); +CONVERSION_TABLE_PENN_TO_UD.put("PRP$", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("RB", "ADV"); +CONVERSION_TABLE_PENN_TO_UD.put("RBR", "ADV"); +CONVERSION_TABLE_PENN_TO_UD.put("RBS", "ADV"); +CONVERSION_TABLE_PENN_TO_UD.put("RP", "ADP"); +CONVERSION_TABLE_PENN_TO_UD.put("SYM", "SYM"); +CONVERSION_TABLE_PENN_TO_UD.put("TO", "PART"); +CONVERSION_TABLE_PENN_TO_UD.put("UH", "INTJ"); +CONVERSION_TABLE_PENN_TO_UD.put("VB", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBD", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBG", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBN", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBP", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBZ", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("WDT", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("WP", "PRON"); +CONVERSION_TABLE_PENN_TO_UD.put("WP$", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("WRB", "ADV"); + +/* + * Note: The back conversion might lose information. + */ +CONVERSION_TABLE_UD_TO_PENN.put("ADJ", "JJ"); +
[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format
[ https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849016#comment-17849016 ] ASF GitHub Bot commented on OPENNLP-1539: - rzo1 commented on code in PR #601: URL: https://github.com/apache/opennlp/pull/601#discussion_r1611936275 ## opennlp-tools/src/main/java/opennlp/tools/postag/POSTagFormatMapper.java: ## @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package opennlp.tools.postag; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * A mapping implementation for converting between different POS tag formats. + * This class supports conversion between Penn Treebank (PENN) and Universal Dependencies (UD) formats. + * The conversion is based on the https://universaldependencies.org/tagset-conversion/en-penn-uposf.html;>Universal Dependencies conversion table. + * Please note that when converting from UD to Penn format, there may be ambiguity in some cases. + */ +public class POSTagFormatMapper { + + private static final Logger logger = LoggerFactory.getLogger(POSTagFormatMapper.class); + + private static final Map CONVERSION_TABLE_PENN_TO_UD = new HashMap<>(); + private static final Map CONVERSION_TABLE_UD_TO_PENN = new HashMap<>(); + + static { +/* + * This is a conversion table to convert PENN to UD format as described in + * https://universaldependencies.org/tagset-conversion/en-penn-uposf.html + */ +CONVERSION_TABLE_PENN_TO_UD.put("#", "SYM"); +CONVERSION_TABLE_PENN_TO_UD.put("$", "SYM"); +CONVERSION_TABLE_PENN_TO_UD.put("''", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put(",", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("-LRB-", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("-RRB-", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put(".", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put(":", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("AFX", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("CC", "CCONJ"); +CONVERSION_TABLE_PENN_TO_UD.put("CD", "NUM"); +CONVERSION_TABLE_PENN_TO_UD.put("DT", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("EX", "PRON"); +CONVERSION_TABLE_PENN_TO_UD.put("FW", "X"); +CONVERSION_TABLE_PENN_TO_UD.put("HYPH", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("IN", "ADP"); +CONVERSION_TABLE_PENN_TO_UD.put("JJ", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("JJR", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("JJS", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("LS", "X"); +CONVERSION_TABLE_PENN_TO_UD.put("MD", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("NIL", "X"); +CONVERSION_TABLE_PENN_TO_UD.put("NN", "NOUN"); +CONVERSION_TABLE_PENN_TO_UD.put("NNP", "PROPN"); +CONVERSION_TABLE_PENN_TO_UD.put("NNPS", "PROPN"); +CONVERSION_TABLE_PENN_TO_UD.put("NNS", "NOUN"); +CONVERSION_TABLE_PENN_TO_UD.put("PDT", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("POS", "PART"); +CONVERSION_TABLE_PENN_TO_UD.put("PRP", "PRON"); +CONVERSION_TABLE_PENN_TO_UD.put("PRP$", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("RB", "ADV"); +CONVERSION_TABLE_PENN_TO_UD.put("RBR", "ADV"); +CONVERSION_TABLE_PENN_TO_UD.put("RBS", "ADV"); +CONVERSION_TABLE_PENN_TO_UD.put("RP", "ADP"); +CONVERSION_TABLE_PENN_TO_UD.put("SYM", "SYM"); +CONVERSION_TABLE_PENN_TO_UD.put("TO", "PART"); +CONVERSION_TABLE_PENN_TO_UD.put("UH", "INTJ"); +CONVERSION_TABLE_PENN_TO_UD.put("VB", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBD", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBG", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBN", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBP", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBZ", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("WDT", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("WP", "PRON"); +CONVERSION_TABLE_PENN_TO_UD.put("WP$", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("WRB", "ADV"); + +/* + * Note: The back conversion might lose information. + */ +CONVERSION_TABLE_UD_TO_PENN.put("ADJ", "JJ"); +
[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format
[ https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849008#comment-17849008 ] ASF GitHub Bot commented on OPENNLP-1539: - rzo1 commented on PR #601: URL: https://github.com/apache/opennlp/pull/601#issuecomment-2127421991 Need to add an test for it ;-) > Introduce parameter for POSTaggerME to configure output POS tag format > -- > > Key: OPENNLP-1539 > URL: https://issues.apache.org/jira/browse/OPENNLP-1539 > Project: OpenNLP > Issue Type: Improvement > Components: POS Tagger >Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0 >Reporter: Martin Wiesner >Assignee: Richard Zowalla >Priority: Major > Fix For: 2.4.0 > > > [Classic (legacy) POS models|https://opennlp.sourceforge.net/models-1.5/] > output tags in the [PENN Treebank POS > tag|https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html] > format. > The modern UD-based models, however, differ in the [longer output > format|https://universaldependencies.org/u/pos/], e.g. "VB" (Penn) vs. "VERB" > (UD). Extended (UD) word features are covered here: > https://universaldependencies.org/u/feat/index.html > This difference results in mismatches and will cause existing IT / tests to > fail, if executed. Luckily, a mapping table is found here: > https://universaldependencies.org/tagset-conversion/en-penn-uposf.html > To provide compatibility for existing applications and/or use-cases, we need > to provide a way to retrieve both POS formats. > Aims: > - Introduce a constructor parameter for POSTaggerME to configure tag format / > style: Penn or UD style > - Implement a mapping between both POS tag formats: UD <==> Penn > - Update the OpenNLP Manual to explain differences of POS tag format and > configuration parameter > Conceptual idea: > - {{new POSTaggerME("en")}} => by _default_: UD format "as is" > - {{new POSTaggerME("en", POSTagFormat.PENN)}} => by _intention_, here: Penn > style > Benefit: > 1. It should be explicit so devs / user see what they will get via > {{POSTagFormat}}. Enum values: POSTagFormat.UD, POSTagFormat.PENN, > POSTagFormat.DEFAULT > 2. IT tests can now be formulated to work on both modern and legacy models. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format
[ https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849006#comment-17849006 ] ASF GitHub Bot commented on OPENNLP-1539: - jzonthemtn commented on PR #601: URL: https://github.com/apache/opennlp/pull/601#issuecomment-2127416229 @rzo1 Did you run into any weirdness with the older Sourceforge models? > Introduce parameter for POSTaggerME to configure output POS tag format > -- > > Key: OPENNLP-1539 > URL: https://issues.apache.org/jira/browse/OPENNLP-1539 > Project: OpenNLP > Issue Type: Improvement > Components: POS Tagger >Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0 >Reporter: Martin Wiesner >Assignee: Richard Zowalla >Priority: Major > Fix For: 2.4.0 > > > [Classic (legacy) POS models|https://opennlp.sourceforge.net/models-1.5/] > output tags in the [PENN Treebank POS > tag|https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html] > format. > The modern UD-based models, however, differ in the [longer output > format|https://universaldependencies.org/u/pos/], e.g. "VB" (Penn) vs. "VERB" > (UD). Extended (UD) word features are covered here: > https://universaldependencies.org/u/feat/index.html > This difference results in mismatches and will cause existing IT / tests to > fail, if executed. Luckily, a mapping table is found here: > https://universaldependencies.org/tagset-conversion/en-penn-uposf.html > To provide compatibility for existing applications and/or use-cases, we need > to provide a way to retrieve both POS formats. > Aims: > - Introduce a constructor parameter for POSTaggerME to configure tag format / > style: Penn or UD style > - Implement a mapping between both POS tag formats: UD <==> Penn > - Update the OpenNLP Manual to explain differences of POS tag format and > configuration parameter > Conceptual idea: > - {{new POSTaggerME("en")}} => by _default_: UD format "as is" > - {{new POSTaggerME("en", POSTagFormat.PENN)}} => by _intention_, here: Penn > style > Benefit: > 1. It should be explicit so devs / user see what they will get via > {{POSTagFormat}}. Enum values: POSTagFormat.UD, POSTagFormat.PENN, > POSTagFormat.DEFAULT > 2. IT tests can now be formulated to work on both modern and legacy models. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format
[ https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849000#comment-17849000 ] ASF GitHub Bot commented on OPENNLP-1539: - jzonthemtn commented on code in PR #601: URL: https://github.com/apache/opennlp/pull/601#discussion_r1611890953 ## opennlp-tools/src/main/java/opennlp/tools/postag/POSTagFormatMapper.java: ## @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package opennlp.tools.postag; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * A mapping implementation for converting between different POS tag formats. + * This class supports conversion between Penn Treebank (PENN) and Universal Dependencies (UD) formats. + * The conversion is based on the https://universaldependencies.org/tagset-conversion/en-penn-uposf.html;>Universal Dependencies conversion table. + * Please note that when converting from UD to Penn format, there may be ambiguity in some cases. + */ +public class POSTagFormatMapper { + + private static final Logger logger = LoggerFactory.getLogger(POSTagFormatMapper.class); + + private static final Map CONVERSION_TABLE_PENN_TO_UD = new HashMap<>(); + private static final Map CONVERSION_TABLE_UD_TO_PENN = new HashMap<>(); + + static { +/* + * This is a conversion table to convert PENN to UD format as described in + * https://universaldependencies.org/tagset-conversion/en-penn-uposf.html + */ +CONVERSION_TABLE_PENN_TO_UD.put("#", "SYM"); +CONVERSION_TABLE_PENN_TO_UD.put("$", "SYM"); +CONVERSION_TABLE_PENN_TO_UD.put("''", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put(",", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("-LRB-", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("-RRB-", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put(".", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put(":", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("AFX", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("CC", "CCONJ"); +CONVERSION_TABLE_PENN_TO_UD.put("CD", "NUM"); +CONVERSION_TABLE_PENN_TO_UD.put("DT", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("EX", "PRON"); +CONVERSION_TABLE_PENN_TO_UD.put("FW", "X"); +CONVERSION_TABLE_PENN_TO_UD.put("HYPH", "PUNCT"); +CONVERSION_TABLE_PENN_TO_UD.put("IN", "ADP"); +CONVERSION_TABLE_PENN_TO_UD.put("JJ", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("JJR", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("JJS", "ADJ"); +CONVERSION_TABLE_PENN_TO_UD.put("LS", "X"); +CONVERSION_TABLE_PENN_TO_UD.put("MD", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("NIL", "X"); +CONVERSION_TABLE_PENN_TO_UD.put("NN", "NOUN"); +CONVERSION_TABLE_PENN_TO_UD.put("NNP", "PROPN"); +CONVERSION_TABLE_PENN_TO_UD.put("NNPS", "PROPN"); +CONVERSION_TABLE_PENN_TO_UD.put("NNS", "NOUN"); +CONVERSION_TABLE_PENN_TO_UD.put("PDT", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("POS", "PART"); +CONVERSION_TABLE_PENN_TO_UD.put("PRP", "PRON"); +CONVERSION_TABLE_PENN_TO_UD.put("PRP$", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("RB", "ADV"); +CONVERSION_TABLE_PENN_TO_UD.put("RBR", "ADV"); +CONVERSION_TABLE_PENN_TO_UD.put("RBS", "ADV"); +CONVERSION_TABLE_PENN_TO_UD.put("RP", "ADP"); +CONVERSION_TABLE_PENN_TO_UD.put("SYM", "SYM"); +CONVERSION_TABLE_PENN_TO_UD.put("TO", "PART"); +CONVERSION_TABLE_PENN_TO_UD.put("UH", "INTJ"); +CONVERSION_TABLE_PENN_TO_UD.put("VB", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBD", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBG", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBN", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBP", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("VBZ", "VERB"); +CONVERSION_TABLE_PENN_TO_UD.put("WDT", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("WP", "PRON"); +CONVERSION_TABLE_PENN_TO_UD.put("WP$", "DET"); +CONVERSION_TABLE_PENN_TO_UD.put("WRB", "ADV"); + +/* + * Note: The back conversion might lose information. + */ +CONVERSION_TABLE_UD_TO_PENN.put("ADJ", "JJ"); +
[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format
[ https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848997#comment-17848997 ] ASF GitHub Bot commented on OPENNLP-1539: - rzo1 commented on PR #601: URL: https://github.com/apache/opennlp/pull/601#issuecomment-2127381263 Note: The currently failing tests are unrelated to the actual change: ```bash Error:ChunkerModelLoaderTest.initResources:43->lambda$initResources$0:47 Runtime java.io.IOException: Server returned HTTP response code: 503 for URL: https://opennlp.sourceforge.net/models-1.5/en-chunker.bin Error: TokenNameFinderModelLoaderTest.initResources:43->lambda$initResources$0:47 Runtime java.io.IOException: Server returned HTTP response code: 503 for URL: https://opennlp.sourceforge.net/models-1.5/en-ner-location.bin Error: TokenNameFinderModelTest.testNERWithPOSModelV15:122->AbstractModelLoaderTest.downloadVersion15Model:41->AbstractModelLoaderTest.downloadModel:57 ยป IO Server returned HTTP response code: 503 for URL: https://opennlp.sourceforge.net/models-1.5/pt-pos-perceptron.bin ``` > Introduce parameter for POSTaggerME to configure output POS tag format > -- > > Key: OPENNLP-1539 > URL: https://issues.apache.org/jira/browse/OPENNLP-1539 > Project: OpenNLP > Issue Type: Improvement > Components: POS Tagger >Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0 >Reporter: Martin Wiesner >Assignee: Richard Zowalla >Priority: Major > Fix For: 2.4.0 > > > [Classic (legacy) POS models|https://opennlp.sourceforge.net/models-1.5/] > output tags in the [PENN Treebank POS > tag|https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html] > format. > The modern UD-based models, however, differ in the [longer output > format|https://universaldependencies.org/u/pos/], e.g. "VB" (Penn) vs. "VERB" > (UD). Extended (UD) word features are covered here: > https://universaldependencies.org/u/feat/index.html > This difference results in mismatches and will cause existing IT / tests to > fail, if executed. Luckily, a mapping table is found here: > https://universaldependencies.org/tagset-conversion/en-penn-uposf.html > To provide compatibility for existing applications and/or use-cases, we need > to provide a way to retrieve both POS formats. > Aims: > - Introduce a constructor parameter for POSTaggerME to configure tag format / > style: Penn or UD style > - Implement a mapping between both POS tag formats: UD <==> Penn > - Update the OpenNLP Manual to explain differences of POS tag format and > configuration parameter > Conceptual idea: > - {{new POSTaggerME("en")}} => by _default_: UD format "as is" > - {{new POSTaggerME("en", POSTagFormat.PENN)}} => by _intention_, here: Penn > style > Benefit: > 1. It should be explicit so devs / user see what they will get via > {{POSTagFormat}}. Enum values: POSTagFormat.UD, POSTagFormat.PENN, > POSTagFormat.DEFAULT > 2. IT tests can now be formulated to work on both modern and legacy models. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format
[ https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848994#comment-17848994 ] ASF GitHub Bot commented on OPENNLP-1539: - rzo1 opened a new pull request, #601: URL: https://github.com/apache/opennlp/pull/601 ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with OPENNLP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically main)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn clean install at the root opennlp folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file in opennlp folder? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found in opennlp folder? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: This is a **Draft** open for feedback on implementing compatibility with older PENN-based POS models from Sourceforge. > Introduce parameter for POSTaggerME to configure output POS tag format > -- > > Key: OPENNLP-1539 > URL: https://issues.apache.org/jira/browse/OPENNLP-1539 > Project: OpenNLP > Issue Type: Improvement > Components: POS Tagger >Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0 >Reporter: Martin Wiesner >Assignee: Richard Zowalla >Priority: Major > Fix For: 2.4.0 > > > [Classic (legacy) POS models|https://opennlp.sourceforge.net/models-1.5/] > output tags in the [PENN Treebank POS > tag|https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html] > format. > The modern UD-based models, however, differ in the [longer output > format|https://universaldependencies.org/u/pos/], e.g. "VB" (Penn) vs. "VERB" > (UD). Extended (UD) word features are covered here: > https://universaldependencies.org/u/feat/index.html > This difference results in mismatches and will cause existing IT / tests to > fail, if executed. Luckily, a mapping table is found here: > https://universaldependencies.org/tagset-conversion/en-penn-uposf.html > To provide compatibility for existing applications and/or use-cases, we need > to provide a way to retrieve both POS formats. > Aims: > - Introduce a constructor parameter for POSTaggerME to configure tag format / > style: Penn or UD style > - Implement a mapping between both POS tag formats: UD <==> Penn > - Update the OpenNLP Manual to explain differences of POS tag format and > configuration parameter > Conceptual idea: > - {{new POSTaggerME("en")}} => by _default_: UD format "as is" > - {{new POSTaggerME("en", POSTagFormat.PENN)}} => by _intention_, here: Penn > style > Benefit: > 1. It should be explicit so devs / user see what they will get via > {{POSTagFormat}}. Enum values: POSTagFormat.UD, POSTagFormat.PENN, > POSTagFormat.DEFAULT > 2. IT tests can now be formulated to work on both modern and legacy models. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format
[ https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17801225#comment-17801225 ] Martin Wiesner commented on OPENNLP-1539: - Related StOf post is here: https://stackoverflow.com/q/56505268/2849346 > Introduce parameter for POSTaggerME to configure output POS tag format > -- > > Key: OPENNLP-1539 > URL: https://issues.apache.org/jira/browse/OPENNLP-1539 > Project: OpenNLP > Issue Type: Improvement > Components: POS Tagger >Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0 >Reporter: Martin Wiesner >Priority: Major > > [Classic (legacy) POS models|https://opennlp.sourceforge.net/models-1.5/] > output tags in the [PENN Treebank POS > tag|https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html] > format. > The modern UD-based models, however, differ in the longer output format, e.g. > "VB" (Penn) vs. "VERB" (UD). This results in mismatches and will cause > existing IT / tests to fail, if executed. > To provide compatibility for existing applications and/or use-cases, we need > to provide a way to retrieve both POS formats. > Aims: > - Introduce a constructor parameter for POSTaggerME to configure tag format / > style: Penn or UD style > - Implement a mapping between both POS tag formats: UD <==> Penn > - Update the OpenNLP Manual to explain differences of POS tag format and > configuration parameter > Conceptual idea: > - {{new POSTaggerME("en")}} => by _default_: UD format "as is" > - {{new POSTaggerME("en", POSTagFormat.PENN)}} => by _intention_, here: Penn > style > Benefit: > 1. It should be explicit so devs / user see what they will get via > {{POSTagFormat}}. Enum values: POSTagFormat.UD, POSTagFormat.PENN, > POSTagFormat.DEFAULT > 2. IT tests can now be formulated to work on both modern and legacy models. -- This message was sent by Atlassian Jira (v8.20.10#820010)