[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format

2024-05-29 Thread ASF GitHub Bot (Jira)


[ 
https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17850265#comment-17850265
 ] 

ASF GitHub Bot commented on OPENNLP-1539:
-

mawiesne merged PR #601:
URL: https://github.com/apache/opennlp/pull/601




> Introduce parameter for POSTaggerME to configure output POS tag format
> --
>
> Key: OPENNLP-1539
> URL: https://issues.apache.org/jira/browse/OPENNLP-1539
> Project: OpenNLP
>  Issue Type: Improvement
>  Components: POS Tagger
>Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
>Reporter: Martin Wiesner
>Assignee: Richard Zowalla
>Priority: Major
> Fix For: 2.4.0
>
>
> [Classic (legacy) POS models|https://opennlp.sourceforge.net/models-1.5/] 
> output tags in the [PENN Treebank POS 
> tag|https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html]
>  format.
> The modern UD-based models, however, differ in the [longer output 
> format|https://universaldependencies.org/u/pos/], e.g. "VB" (Penn) vs. "VERB" 
> (UD). Extended (UD) word features are covered here: 
> https://universaldependencies.org/u/feat/index.html
> This difference results in mismatches and will cause existing IT / tests to 
> fail, if executed. Luckily, a mapping table is found here: 
> https://universaldependencies.org/tagset-conversion/en-penn-uposf.html
> To provide compatibility for existing applications and/or use-cases, we need 
> to provide a way to retrieve both POS formats. 
> Aims:
> - Introduce a constructor parameter for POSTaggerME to configure tag format / 
> style: Penn or UD style
> - Implement a mapping between both POS tag formats: UD <==> Penn
> - Update the OpenNLP Manual to explain differences of POS tag format and 
> configuration parameter
> Conceptual idea:
> - {{new POSTaggerME("en")}}  => by _default_: UD format "as is"
> - {{new POSTaggerME("en", POSTagFormat.PENN)}} => by _intention_, here: Penn 
> style
> Benefit: 
> 1. It should be explicit so devs / user see what they will get via 
> {{POSTagFormat}}. Enum values: POSTagFormat.UD, POSTagFormat.PENN, 
> POSTagFormat.DEFAULT
> 2. IT tests can now be formulated to work on both modern and legacy models.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format

2024-05-24 Thread ASF GitHub Bot (Jira)


[ 
https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849223#comment-17849223
 ] 

ASF GitHub Bot commented on OPENNLP-1539:
-

rzo1 commented on PR #601:
URL: https://github.com/apache/opennlp/pull/601#issuecomment-2128979449

   > @rzo1 Did you run into any weirdness with the older Sourceforge models?
   
   Added tests. They look good.




> Introduce parameter for POSTaggerME to configure output POS tag format
> --
>
> Key: OPENNLP-1539
> URL: https://issues.apache.org/jira/browse/OPENNLP-1539
> Project: OpenNLP
>  Issue Type: Improvement
>  Components: POS Tagger
>Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
>Reporter: Martin Wiesner
>Assignee: Richard Zowalla
>Priority: Major
> Fix For: 2.4.0
>
>
> [Classic (legacy) POS models|https://opennlp.sourceforge.net/models-1.5/] 
> output tags in the [PENN Treebank POS 
> tag|https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html]
>  format.
> The modern UD-based models, however, differ in the [longer output 
> format|https://universaldependencies.org/u/pos/], e.g. "VB" (Penn) vs. "VERB" 
> (UD). Extended (UD) word features are covered here: 
> https://universaldependencies.org/u/feat/index.html
> This difference results in mismatches and will cause existing IT / tests to 
> fail, if executed. Luckily, a mapping table is found here: 
> https://universaldependencies.org/tagset-conversion/en-penn-uposf.html
> To provide compatibility for existing applications and/or use-cases, we need 
> to provide a way to retrieve both POS formats. 
> Aims:
> - Introduce a constructor parameter for POSTaggerME to configure tag format / 
> style: Penn or UD style
> - Implement a mapping between both POS tag formats: UD <==> Penn
> - Update the OpenNLP Manual to explain differences of POS tag format and 
> configuration parameter
> Conceptual idea:
> - {{new POSTaggerME("en")}}  => by _default_: UD format "as is"
> - {{new POSTaggerME("en", POSTagFormat.PENN)}} => by _intention_, here: Penn 
> style
> Benefit: 
> 1. It should be explicit so devs / user see what they will get via 
> {{POSTagFormat}}. Enum values: POSTagFormat.UD, POSTagFormat.PENN, 
> POSTagFormat.DEFAULT
> 2. IT tests can now be formulated to work on both modern and legacy models.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format

2024-05-24 Thread ASF GitHub Bot (Jira)


[ 
https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849194#comment-17849194
 ] 

ASF GitHub Bot commented on OPENNLP-1539:
-

mawiesne commented on code in PR #601:
URL: https://github.com/apache/opennlp/pull/601#discussion_r1612947718


##
opennlp-tools/src/main/java/opennlp/tools/postag/POSTagFormatMapper.java:
##
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.postag;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A mapping implementation for converting between different POS tag formats.
+ * This class supports conversion between Penn Treebank (PENN) and Universal 
Dependencies (UD) formats.
+ * The conversion is based on the https://universaldependencies.org/tagset-conversion/en-penn-uposf.html;>Universal
 Dependencies conversion table.
+ * Please note that when converting from UD to Penn format, there may be 
ambiguity in some cases.
+ */
+public class POSTagFormatMapper {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(POSTagFormatMapper.class);
+
+  private static final Map CONVERSION_TABLE_PENN_TO_UD = new 
HashMap<>();
+  private static final Map CONVERSION_TABLE_UD_TO_PENN = new 
HashMap<>();
+
+  static {
+/*
+ * This is a conversion table to convert PENN to UD format as described in
+ * https://universaldependencies.org/tagset-conversion/en-penn-uposf.html
+ */
+CONVERSION_TABLE_PENN_TO_UD.put("#", "SYM");
+CONVERSION_TABLE_PENN_TO_UD.put("$", "SYM");
+CONVERSION_TABLE_PENN_TO_UD.put("''", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put(",", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("-LRB-", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("-RRB-", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put(".", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put(":", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("AFX", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("CC", "CCONJ");
+CONVERSION_TABLE_PENN_TO_UD.put("CD", "NUM");
+CONVERSION_TABLE_PENN_TO_UD.put("DT", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("EX", "PRON");
+CONVERSION_TABLE_PENN_TO_UD.put("FW", "X");
+CONVERSION_TABLE_PENN_TO_UD.put("HYPH", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("IN", "ADP");
+CONVERSION_TABLE_PENN_TO_UD.put("JJ", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("JJR", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("JJS", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("LS", "X");
+CONVERSION_TABLE_PENN_TO_UD.put("MD", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("NIL", "X");
+CONVERSION_TABLE_PENN_TO_UD.put("NN", "NOUN");
+CONVERSION_TABLE_PENN_TO_UD.put("NNP", "PROPN");
+CONVERSION_TABLE_PENN_TO_UD.put("NNPS", "PROPN");
+CONVERSION_TABLE_PENN_TO_UD.put("NNS", "NOUN");
+CONVERSION_TABLE_PENN_TO_UD.put("PDT", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("POS", "PART");
+CONVERSION_TABLE_PENN_TO_UD.put("PRP", "PRON");
+CONVERSION_TABLE_PENN_TO_UD.put("PRP$", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("RB", "ADV");
+CONVERSION_TABLE_PENN_TO_UD.put("RBR", "ADV");
+CONVERSION_TABLE_PENN_TO_UD.put("RBS", "ADV");
+CONVERSION_TABLE_PENN_TO_UD.put("RP", "ADP");
+CONVERSION_TABLE_PENN_TO_UD.put("SYM", "SYM");
+CONVERSION_TABLE_PENN_TO_UD.put("TO", "PART");
+CONVERSION_TABLE_PENN_TO_UD.put("UH", "INTJ");
+CONVERSION_TABLE_PENN_TO_UD.put("VB", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBD", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBG", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBN", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBP", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBZ", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("WDT", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("WP", "PRON");
+CONVERSION_TABLE_PENN_TO_UD.put("WP$", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("WRB", "ADV");
+
+/*
+ * Note: The back conversion might lose information.
+ */
+CONVERSION_TABLE_UD_TO_PENN.put("ADJ", "JJ");
+

[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format

2024-05-24 Thread ASF GitHub Bot (Jira)


[ 
https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849174#comment-17849174
 ] 

ASF GitHub Bot commented on OPENNLP-1539:
-

rzo1 commented on code in PR #601:
URL: https://github.com/apache/opennlp/pull/601#discussion_r1612885912


##
opennlp-tools/src/main/java/opennlp/tools/postag/POSTagFormatMapper.java:
##
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.postag;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A mapping implementation for converting between different POS tag formats.
+ * This class supports conversion between Penn Treebank (PENN) and Universal 
Dependencies (UD) formats.
+ * The conversion is based on the https://universaldependencies.org/tagset-conversion/en-penn-uposf.html;>Universal
 Dependencies conversion table.
+ * Please note that when converting from UD to Penn format, there may be 
ambiguity in some cases.
+ */
+public class POSTagFormatMapper {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(POSTagFormatMapper.class);
+
+  private static final Map CONVERSION_TABLE_PENN_TO_UD = new 
HashMap<>();
+  private static final Map CONVERSION_TABLE_UD_TO_PENN = new 
HashMap<>();
+
+  static {
+/*
+ * This is a conversion table to convert PENN to UD format as described in
+ * https://universaldependencies.org/tagset-conversion/en-penn-uposf.html
+ */
+CONVERSION_TABLE_PENN_TO_UD.put("#", "SYM");
+CONVERSION_TABLE_PENN_TO_UD.put("$", "SYM");
+CONVERSION_TABLE_PENN_TO_UD.put("''", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put(",", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("-LRB-", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("-RRB-", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put(".", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put(":", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("AFX", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("CC", "CCONJ");
+CONVERSION_TABLE_PENN_TO_UD.put("CD", "NUM");
+CONVERSION_TABLE_PENN_TO_UD.put("DT", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("EX", "PRON");
+CONVERSION_TABLE_PENN_TO_UD.put("FW", "X");
+CONVERSION_TABLE_PENN_TO_UD.put("HYPH", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("IN", "ADP");
+CONVERSION_TABLE_PENN_TO_UD.put("JJ", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("JJR", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("JJS", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("LS", "X");
+CONVERSION_TABLE_PENN_TO_UD.put("MD", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("NIL", "X");
+CONVERSION_TABLE_PENN_TO_UD.put("NN", "NOUN");
+CONVERSION_TABLE_PENN_TO_UD.put("NNP", "PROPN");
+CONVERSION_TABLE_PENN_TO_UD.put("NNPS", "PROPN");
+CONVERSION_TABLE_PENN_TO_UD.put("NNS", "NOUN");
+CONVERSION_TABLE_PENN_TO_UD.put("PDT", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("POS", "PART");
+CONVERSION_TABLE_PENN_TO_UD.put("PRP", "PRON");
+CONVERSION_TABLE_PENN_TO_UD.put("PRP$", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("RB", "ADV");
+CONVERSION_TABLE_PENN_TO_UD.put("RBR", "ADV");
+CONVERSION_TABLE_PENN_TO_UD.put("RBS", "ADV");
+CONVERSION_TABLE_PENN_TO_UD.put("RP", "ADP");
+CONVERSION_TABLE_PENN_TO_UD.put("SYM", "SYM");
+CONVERSION_TABLE_PENN_TO_UD.put("TO", "PART");
+CONVERSION_TABLE_PENN_TO_UD.put("UH", "INTJ");
+CONVERSION_TABLE_PENN_TO_UD.put("VB", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBD", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBG", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBN", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBP", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBZ", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("WDT", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("WP", "PRON");
+CONVERSION_TABLE_PENN_TO_UD.put("WP$", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("WRB", "ADV");
+
+/*
+ * Note: The back conversion might lose information.
+ */
+CONVERSION_TABLE_UD_TO_PENN.put("ADJ", "JJ");
+

[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format

2024-05-24 Thread ASF GitHub Bot (Jira)


[ 
https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849172#comment-17849172
 ] 

ASF GitHub Bot commented on OPENNLP-1539:
-

rzo1 commented on code in PR #601:
URL: https://github.com/apache/opennlp/pull/601#discussion_r1612883541


##
opennlp-tools/src/main/java/opennlp/tools/postag/POSTagFormatMapper.java:
##
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.postag;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A mapping implementation for converting between different POS tag formats.
+ * This class supports conversion between Penn Treebank (PENN) and Universal 
Dependencies (UD) formats.
+ * The conversion is based on the https://universaldependencies.org/tagset-conversion/en-penn-uposf.html;>Universal
 Dependencies conversion table.
+ * Please note that when converting from UD to Penn format, there may be 
ambiguity in some cases.
+ */
+public class POSTagFormatMapper {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(POSTagFormatMapper.class);
+
+  private static final Map CONVERSION_TABLE_PENN_TO_UD = new 
HashMap<>();
+  private static final Map CONVERSION_TABLE_UD_TO_PENN = new 
HashMap<>();
+
+  static {
+/*
+ * This is a conversion table to convert PENN to UD format as described in
+ * https://universaldependencies.org/tagset-conversion/en-penn-uposf.html
+ */
+CONVERSION_TABLE_PENN_TO_UD.put("#", "SYM");
+CONVERSION_TABLE_PENN_TO_UD.put("$", "SYM");
+CONVERSION_TABLE_PENN_TO_UD.put("''", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put(",", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("-LRB-", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("-RRB-", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put(".", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put(":", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("AFX", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("CC", "CCONJ");
+CONVERSION_TABLE_PENN_TO_UD.put("CD", "NUM");
+CONVERSION_TABLE_PENN_TO_UD.put("DT", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("EX", "PRON");
+CONVERSION_TABLE_PENN_TO_UD.put("FW", "X");
+CONVERSION_TABLE_PENN_TO_UD.put("HYPH", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("IN", "ADP");
+CONVERSION_TABLE_PENN_TO_UD.put("JJ", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("JJR", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("JJS", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("LS", "X");
+CONVERSION_TABLE_PENN_TO_UD.put("MD", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("NIL", "X");
+CONVERSION_TABLE_PENN_TO_UD.put("NN", "NOUN");
+CONVERSION_TABLE_PENN_TO_UD.put("NNP", "PROPN");
+CONVERSION_TABLE_PENN_TO_UD.put("NNPS", "PROPN");
+CONVERSION_TABLE_PENN_TO_UD.put("NNS", "NOUN");
+CONVERSION_TABLE_PENN_TO_UD.put("PDT", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("POS", "PART");
+CONVERSION_TABLE_PENN_TO_UD.put("PRP", "PRON");
+CONVERSION_TABLE_PENN_TO_UD.put("PRP$", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("RB", "ADV");
+CONVERSION_TABLE_PENN_TO_UD.put("RBR", "ADV");
+CONVERSION_TABLE_PENN_TO_UD.put("RBS", "ADV");
+CONVERSION_TABLE_PENN_TO_UD.put("RP", "ADP");
+CONVERSION_TABLE_PENN_TO_UD.put("SYM", "SYM");
+CONVERSION_TABLE_PENN_TO_UD.put("TO", "PART");
+CONVERSION_TABLE_PENN_TO_UD.put("UH", "INTJ");
+CONVERSION_TABLE_PENN_TO_UD.put("VB", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBD", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBG", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBN", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBP", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBZ", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("WDT", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("WP", "PRON");
+CONVERSION_TABLE_PENN_TO_UD.put("WP$", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("WRB", "ADV");
+
+/*
+ * Note: The back conversion might lose information.
+ */
+CONVERSION_TABLE_UD_TO_PENN.put("ADJ", "JJ");
+

[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format

2024-05-23 Thread ASF GitHub Bot (Jira)


[ 
https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849158#comment-17849158
 ] 

ASF GitHub Bot commented on OPENNLP-1539:
-

mawiesne commented on PR #601:
URL: https://github.com/apache/opennlp/pull/601#issuecomment-2128552371

   Thx @rzo1 - I left some comments to improve clarity of the doc for the 
changes in the API.




> Introduce parameter for POSTaggerME to configure output POS tag format
> --
>
> Key: OPENNLP-1539
> URL: https://issues.apache.org/jira/browse/OPENNLP-1539
> Project: OpenNLP
>  Issue Type: Improvement
>  Components: POS Tagger
>Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
>Reporter: Martin Wiesner
>Assignee: Richard Zowalla
>Priority: Major
> Fix For: 2.4.0
>
>
> [Classic (legacy) POS models|https://opennlp.sourceforge.net/models-1.5/] 
> output tags in the [PENN Treebank POS 
> tag|https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html]
>  format.
> The modern UD-based models, however, differ in the [longer output 
> format|https://universaldependencies.org/u/pos/], e.g. "VB" (Penn) vs. "VERB" 
> (UD). Extended (UD) word features are covered here: 
> https://universaldependencies.org/u/feat/index.html
> This difference results in mismatches and will cause existing IT / tests to 
> fail, if executed. Luckily, a mapping table is found here: 
> https://universaldependencies.org/tagset-conversion/en-penn-uposf.html
> To provide compatibility for existing applications and/or use-cases, we need 
> to provide a way to retrieve both POS formats. 
> Aims:
> - Introduce a constructor parameter for POSTaggerME to configure tag format / 
> style: Penn or UD style
> - Implement a mapping between both POS tag formats: UD <==> Penn
> - Update the OpenNLP Manual to explain differences of POS tag format and 
> configuration parameter
> Conceptual idea:
> - {{new POSTaggerME("en")}}  => by _default_: UD format "as is"
> - {{new POSTaggerME("en", POSTagFormat.PENN)}} => by _intention_, here: Penn 
> style
> Benefit: 
> 1. It should be explicit so devs / user see what they will get via 
> {{POSTagFormat}}. Enum values: POSTagFormat.UD, POSTagFormat.PENN, 
> POSTagFormat.DEFAULT
> 2. IT tests can now be formulated to work on both modern and legacy models.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format

2024-05-23 Thread ASF GitHub Bot (Jira)


[ 
https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849157#comment-17849157
 ] 

ASF GitHub Bot commented on OPENNLP-1539:
-

mawiesne commented on code in PR #601:
URL: https://github.com/apache/opennlp/pull/601#discussion_r1612745224


##
opennlp-tools/src/main/java/opennlp/tools/postag/POSTagFormatMapper.java:
##
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.postag;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A mapping implementation for converting between different POS tag formats.
+ * This class supports conversion between Penn Treebank (PENN) and Universal 
Dependencies (UD) formats.
+ * The conversion is based on the https://universaldependencies.org/tagset-conversion/en-penn-uposf.html;>Universal
 Dependencies conversion table.
+ * Please note that when converting from UD to Penn format, there may be 
ambiguity in some cases.
+ */
+public class POSTagFormatMapper {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(POSTagFormatMapper.class);
+
+  private static final Map CONVERSION_TABLE_PENN_TO_UD = new 
HashMap<>();
+  private static final Map CONVERSION_TABLE_UD_TO_PENN = new 
HashMap<>();
+
+  static {
+/*
+ * This is a conversion table to convert PENN to UD format as described in
+ * https://universaldependencies.org/tagset-conversion/en-penn-uposf.html
+ */
+CONVERSION_TABLE_PENN_TO_UD.put("#", "SYM");
+CONVERSION_TABLE_PENN_TO_UD.put("$", "SYM");
+CONVERSION_TABLE_PENN_TO_UD.put("''", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put(",", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("-LRB-", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("-RRB-", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put(".", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put(":", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("AFX", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("CC", "CCONJ");
+CONVERSION_TABLE_PENN_TO_UD.put("CD", "NUM");
+CONVERSION_TABLE_PENN_TO_UD.put("DT", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("EX", "PRON");
+CONVERSION_TABLE_PENN_TO_UD.put("FW", "X");
+CONVERSION_TABLE_PENN_TO_UD.put("HYPH", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("IN", "ADP");
+CONVERSION_TABLE_PENN_TO_UD.put("JJ", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("JJR", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("JJS", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("LS", "X");
+CONVERSION_TABLE_PENN_TO_UD.put("MD", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("NIL", "X");
+CONVERSION_TABLE_PENN_TO_UD.put("NN", "NOUN");
+CONVERSION_TABLE_PENN_TO_UD.put("NNP", "PROPN");
+CONVERSION_TABLE_PENN_TO_UD.put("NNPS", "PROPN");
+CONVERSION_TABLE_PENN_TO_UD.put("NNS", "NOUN");
+CONVERSION_TABLE_PENN_TO_UD.put("PDT", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("POS", "PART");
+CONVERSION_TABLE_PENN_TO_UD.put("PRP", "PRON");
+CONVERSION_TABLE_PENN_TO_UD.put("PRP$", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("RB", "ADV");
+CONVERSION_TABLE_PENN_TO_UD.put("RBR", "ADV");
+CONVERSION_TABLE_PENN_TO_UD.put("RBS", "ADV");
+CONVERSION_TABLE_PENN_TO_UD.put("RP", "ADP");
+CONVERSION_TABLE_PENN_TO_UD.put("SYM", "SYM");
+CONVERSION_TABLE_PENN_TO_UD.put("TO", "PART");
+CONVERSION_TABLE_PENN_TO_UD.put("UH", "INTJ");
+CONVERSION_TABLE_PENN_TO_UD.put("VB", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBD", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBG", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBN", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBP", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBZ", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("WDT", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("WP", "PRON");
+CONVERSION_TABLE_PENN_TO_UD.put("WP$", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("WRB", "ADV");
+
+/*
+ * Note: The back conversion might lose information.
+ */
+CONVERSION_TABLE_UD_TO_PENN.put("ADJ", "JJ");
+

[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format

2024-05-23 Thread ASF GitHub Bot (Jira)


[ 
https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849055#comment-17849055
 ] 

ASF GitHub Bot commented on OPENNLP-1539:
-

jzonthemtn commented on code in PR #601:
URL: https://github.com/apache/opennlp/pull/601#discussion_r1612105341


##
opennlp-tools/src/main/java/opennlp/tools/postag/POSTagFormatMapper.java:
##
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.postag;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A mapping implementation for converting between different POS tag formats.
+ * This class supports conversion between Penn Treebank (PENN) and Universal 
Dependencies (UD) formats.
+ * The conversion is based on the https://universaldependencies.org/tagset-conversion/en-penn-uposf.html;>Universal
 Dependencies conversion table.
+ * Please note that when converting from UD to Penn format, there may be 
ambiguity in some cases.
+ */
+public class POSTagFormatMapper {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(POSTagFormatMapper.class);
+
+  private static final Map CONVERSION_TABLE_PENN_TO_UD = new 
HashMap<>();
+  private static final Map CONVERSION_TABLE_UD_TO_PENN = new 
HashMap<>();
+
+  static {
+/*
+ * This is a conversion table to convert PENN to UD format as described in
+ * https://universaldependencies.org/tagset-conversion/en-penn-uposf.html
+ */
+CONVERSION_TABLE_PENN_TO_UD.put("#", "SYM");
+CONVERSION_TABLE_PENN_TO_UD.put("$", "SYM");
+CONVERSION_TABLE_PENN_TO_UD.put("''", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put(",", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("-LRB-", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("-RRB-", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put(".", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put(":", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("AFX", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("CC", "CCONJ");
+CONVERSION_TABLE_PENN_TO_UD.put("CD", "NUM");
+CONVERSION_TABLE_PENN_TO_UD.put("DT", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("EX", "PRON");
+CONVERSION_TABLE_PENN_TO_UD.put("FW", "X");
+CONVERSION_TABLE_PENN_TO_UD.put("HYPH", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("IN", "ADP");
+CONVERSION_TABLE_PENN_TO_UD.put("JJ", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("JJR", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("JJS", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("LS", "X");
+CONVERSION_TABLE_PENN_TO_UD.put("MD", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("NIL", "X");
+CONVERSION_TABLE_PENN_TO_UD.put("NN", "NOUN");
+CONVERSION_TABLE_PENN_TO_UD.put("NNP", "PROPN");
+CONVERSION_TABLE_PENN_TO_UD.put("NNPS", "PROPN");
+CONVERSION_TABLE_PENN_TO_UD.put("NNS", "NOUN");
+CONVERSION_TABLE_PENN_TO_UD.put("PDT", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("POS", "PART");
+CONVERSION_TABLE_PENN_TO_UD.put("PRP", "PRON");
+CONVERSION_TABLE_PENN_TO_UD.put("PRP$", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("RB", "ADV");
+CONVERSION_TABLE_PENN_TO_UD.put("RBR", "ADV");
+CONVERSION_TABLE_PENN_TO_UD.put("RBS", "ADV");
+CONVERSION_TABLE_PENN_TO_UD.put("RP", "ADP");
+CONVERSION_TABLE_PENN_TO_UD.put("SYM", "SYM");
+CONVERSION_TABLE_PENN_TO_UD.put("TO", "PART");
+CONVERSION_TABLE_PENN_TO_UD.put("UH", "INTJ");
+CONVERSION_TABLE_PENN_TO_UD.put("VB", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBD", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBG", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBN", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBP", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBZ", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("WDT", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("WP", "PRON");
+CONVERSION_TABLE_PENN_TO_UD.put("WP$", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("WRB", "ADV");
+
+/*
+ * Note: The back conversion might lose information.
+ */
+CONVERSION_TABLE_UD_TO_PENN.put("ADJ", "JJ");
+

[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format

2024-05-23 Thread ASF GitHub Bot (Jira)


[ 
https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849016#comment-17849016
 ] 

ASF GitHub Bot commented on OPENNLP-1539:
-

rzo1 commented on code in PR #601:
URL: https://github.com/apache/opennlp/pull/601#discussion_r1611936275


##
opennlp-tools/src/main/java/opennlp/tools/postag/POSTagFormatMapper.java:
##
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.postag;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A mapping implementation for converting between different POS tag formats.
+ * This class supports conversion between Penn Treebank (PENN) and Universal 
Dependencies (UD) formats.
+ * The conversion is based on the https://universaldependencies.org/tagset-conversion/en-penn-uposf.html;>Universal
 Dependencies conversion table.
+ * Please note that when converting from UD to Penn format, there may be 
ambiguity in some cases.
+ */
+public class POSTagFormatMapper {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(POSTagFormatMapper.class);
+
+  private static final Map CONVERSION_TABLE_PENN_TO_UD = new 
HashMap<>();
+  private static final Map CONVERSION_TABLE_UD_TO_PENN = new 
HashMap<>();
+
+  static {
+/*
+ * This is a conversion table to convert PENN to UD format as described in
+ * https://universaldependencies.org/tagset-conversion/en-penn-uposf.html
+ */
+CONVERSION_TABLE_PENN_TO_UD.put("#", "SYM");
+CONVERSION_TABLE_PENN_TO_UD.put("$", "SYM");
+CONVERSION_TABLE_PENN_TO_UD.put("''", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put(",", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("-LRB-", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("-RRB-", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put(".", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put(":", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("AFX", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("CC", "CCONJ");
+CONVERSION_TABLE_PENN_TO_UD.put("CD", "NUM");
+CONVERSION_TABLE_PENN_TO_UD.put("DT", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("EX", "PRON");
+CONVERSION_TABLE_PENN_TO_UD.put("FW", "X");
+CONVERSION_TABLE_PENN_TO_UD.put("HYPH", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("IN", "ADP");
+CONVERSION_TABLE_PENN_TO_UD.put("JJ", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("JJR", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("JJS", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("LS", "X");
+CONVERSION_TABLE_PENN_TO_UD.put("MD", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("NIL", "X");
+CONVERSION_TABLE_PENN_TO_UD.put("NN", "NOUN");
+CONVERSION_TABLE_PENN_TO_UD.put("NNP", "PROPN");
+CONVERSION_TABLE_PENN_TO_UD.put("NNPS", "PROPN");
+CONVERSION_TABLE_PENN_TO_UD.put("NNS", "NOUN");
+CONVERSION_TABLE_PENN_TO_UD.put("PDT", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("POS", "PART");
+CONVERSION_TABLE_PENN_TO_UD.put("PRP", "PRON");
+CONVERSION_TABLE_PENN_TO_UD.put("PRP$", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("RB", "ADV");
+CONVERSION_TABLE_PENN_TO_UD.put("RBR", "ADV");
+CONVERSION_TABLE_PENN_TO_UD.put("RBS", "ADV");
+CONVERSION_TABLE_PENN_TO_UD.put("RP", "ADP");
+CONVERSION_TABLE_PENN_TO_UD.put("SYM", "SYM");
+CONVERSION_TABLE_PENN_TO_UD.put("TO", "PART");
+CONVERSION_TABLE_PENN_TO_UD.put("UH", "INTJ");
+CONVERSION_TABLE_PENN_TO_UD.put("VB", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBD", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBG", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBN", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBP", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBZ", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("WDT", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("WP", "PRON");
+CONVERSION_TABLE_PENN_TO_UD.put("WP$", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("WRB", "ADV");
+
+/*
+ * Note: The back conversion might lose information.
+ */
+CONVERSION_TABLE_UD_TO_PENN.put("ADJ", "JJ");
+

[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format

2024-05-23 Thread ASF GitHub Bot (Jira)


[ 
https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849008#comment-17849008
 ] 

ASF GitHub Bot commented on OPENNLP-1539:
-

rzo1 commented on PR #601:
URL: https://github.com/apache/opennlp/pull/601#issuecomment-2127421991

   Need to add an test for it ;-) 




> Introduce parameter for POSTaggerME to configure output POS tag format
> --
>
> Key: OPENNLP-1539
> URL: https://issues.apache.org/jira/browse/OPENNLP-1539
> Project: OpenNLP
>  Issue Type: Improvement
>  Components: POS Tagger
>Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
>Reporter: Martin Wiesner
>Assignee: Richard Zowalla
>Priority: Major
> Fix For: 2.4.0
>
>
> [Classic (legacy) POS models|https://opennlp.sourceforge.net/models-1.5/] 
> output tags in the [PENN Treebank POS 
> tag|https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html]
>  format.
> The modern UD-based models, however, differ in the [longer output 
> format|https://universaldependencies.org/u/pos/], e.g. "VB" (Penn) vs. "VERB" 
> (UD). Extended (UD) word features are covered here: 
> https://universaldependencies.org/u/feat/index.html
> This difference results in mismatches and will cause existing IT / tests to 
> fail, if executed. Luckily, a mapping table is found here: 
> https://universaldependencies.org/tagset-conversion/en-penn-uposf.html
> To provide compatibility for existing applications and/or use-cases, we need 
> to provide a way to retrieve both POS formats. 
> Aims:
> - Introduce a constructor parameter for POSTaggerME to configure tag format / 
> style: Penn or UD style
> - Implement a mapping between both POS tag formats: UD <==> Penn
> - Update the OpenNLP Manual to explain differences of POS tag format and 
> configuration parameter
> Conceptual idea:
> - {{new POSTaggerME("en")}}  => by _default_: UD format "as is"
> - {{new POSTaggerME("en", POSTagFormat.PENN)}} => by _intention_, here: Penn 
> style
> Benefit: 
> 1. It should be explicit so devs / user see what they will get via 
> {{POSTagFormat}}. Enum values: POSTagFormat.UD, POSTagFormat.PENN, 
> POSTagFormat.DEFAULT
> 2. IT tests can now be formulated to work on both modern and legacy models.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format

2024-05-23 Thread ASF GitHub Bot (Jira)


[ 
https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849006#comment-17849006
 ] 

ASF GitHub Bot commented on OPENNLP-1539:
-

jzonthemtn commented on PR #601:
URL: https://github.com/apache/opennlp/pull/601#issuecomment-2127416229

   @rzo1 Did you run into any weirdness with the older Sourceforge models?




> Introduce parameter for POSTaggerME to configure output POS tag format
> --
>
> Key: OPENNLP-1539
> URL: https://issues.apache.org/jira/browse/OPENNLP-1539
> Project: OpenNLP
>  Issue Type: Improvement
>  Components: POS Tagger
>Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
>Reporter: Martin Wiesner
>Assignee: Richard Zowalla
>Priority: Major
> Fix For: 2.4.0
>
>
> [Classic (legacy) POS models|https://opennlp.sourceforge.net/models-1.5/] 
> output tags in the [PENN Treebank POS 
> tag|https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html]
>  format.
> The modern UD-based models, however, differ in the [longer output 
> format|https://universaldependencies.org/u/pos/], e.g. "VB" (Penn) vs. "VERB" 
> (UD). Extended (UD) word features are covered here: 
> https://universaldependencies.org/u/feat/index.html
> This difference results in mismatches and will cause existing IT / tests to 
> fail, if executed. Luckily, a mapping table is found here: 
> https://universaldependencies.org/tagset-conversion/en-penn-uposf.html
> To provide compatibility for existing applications and/or use-cases, we need 
> to provide a way to retrieve both POS formats. 
> Aims:
> - Introduce a constructor parameter for POSTaggerME to configure tag format / 
> style: Penn or UD style
> - Implement a mapping between both POS tag formats: UD <==> Penn
> - Update the OpenNLP Manual to explain differences of POS tag format and 
> configuration parameter
> Conceptual idea:
> - {{new POSTaggerME("en")}}  => by _default_: UD format "as is"
> - {{new POSTaggerME("en", POSTagFormat.PENN)}} => by _intention_, here: Penn 
> style
> Benefit: 
> 1. It should be explicit so devs / user see what they will get via 
> {{POSTagFormat}}. Enum values: POSTagFormat.UD, POSTagFormat.PENN, 
> POSTagFormat.DEFAULT
> 2. IT tests can now be formulated to work on both modern and legacy models.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format

2024-05-23 Thread ASF GitHub Bot (Jira)


[ 
https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849000#comment-17849000
 ] 

ASF GitHub Bot commented on OPENNLP-1539:
-

jzonthemtn commented on code in PR #601:
URL: https://github.com/apache/opennlp/pull/601#discussion_r1611890953


##
opennlp-tools/src/main/java/opennlp/tools/postag/POSTagFormatMapper.java:
##
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.postag;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A mapping implementation for converting between different POS tag formats.
+ * This class supports conversion between Penn Treebank (PENN) and Universal 
Dependencies (UD) formats.
+ * The conversion is based on the https://universaldependencies.org/tagset-conversion/en-penn-uposf.html;>Universal
 Dependencies conversion table.
+ * Please note that when converting from UD to Penn format, there may be 
ambiguity in some cases.
+ */
+public class POSTagFormatMapper {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(POSTagFormatMapper.class);
+
+  private static final Map CONVERSION_TABLE_PENN_TO_UD = new 
HashMap<>();
+  private static final Map CONVERSION_TABLE_UD_TO_PENN = new 
HashMap<>();
+
+  static {
+/*
+ * This is a conversion table to convert PENN to UD format as described in
+ * https://universaldependencies.org/tagset-conversion/en-penn-uposf.html
+ */
+CONVERSION_TABLE_PENN_TO_UD.put("#", "SYM");
+CONVERSION_TABLE_PENN_TO_UD.put("$", "SYM");
+CONVERSION_TABLE_PENN_TO_UD.put("''", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put(",", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("-LRB-", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("-RRB-", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put(".", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put(":", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("AFX", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("CC", "CCONJ");
+CONVERSION_TABLE_PENN_TO_UD.put("CD", "NUM");
+CONVERSION_TABLE_PENN_TO_UD.put("DT", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("EX", "PRON");
+CONVERSION_TABLE_PENN_TO_UD.put("FW", "X");
+CONVERSION_TABLE_PENN_TO_UD.put("HYPH", "PUNCT");
+CONVERSION_TABLE_PENN_TO_UD.put("IN", "ADP");
+CONVERSION_TABLE_PENN_TO_UD.put("JJ", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("JJR", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("JJS", "ADJ");
+CONVERSION_TABLE_PENN_TO_UD.put("LS", "X");
+CONVERSION_TABLE_PENN_TO_UD.put("MD", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("NIL", "X");
+CONVERSION_TABLE_PENN_TO_UD.put("NN", "NOUN");
+CONVERSION_TABLE_PENN_TO_UD.put("NNP", "PROPN");
+CONVERSION_TABLE_PENN_TO_UD.put("NNPS", "PROPN");
+CONVERSION_TABLE_PENN_TO_UD.put("NNS", "NOUN");
+CONVERSION_TABLE_PENN_TO_UD.put("PDT", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("POS", "PART");
+CONVERSION_TABLE_PENN_TO_UD.put("PRP", "PRON");
+CONVERSION_TABLE_PENN_TO_UD.put("PRP$", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("RB", "ADV");
+CONVERSION_TABLE_PENN_TO_UD.put("RBR", "ADV");
+CONVERSION_TABLE_PENN_TO_UD.put("RBS", "ADV");
+CONVERSION_TABLE_PENN_TO_UD.put("RP", "ADP");
+CONVERSION_TABLE_PENN_TO_UD.put("SYM", "SYM");
+CONVERSION_TABLE_PENN_TO_UD.put("TO", "PART");
+CONVERSION_TABLE_PENN_TO_UD.put("UH", "INTJ");
+CONVERSION_TABLE_PENN_TO_UD.put("VB", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBD", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBG", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBN", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBP", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("VBZ", "VERB");
+CONVERSION_TABLE_PENN_TO_UD.put("WDT", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("WP", "PRON");
+CONVERSION_TABLE_PENN_TO_UD.put("WP$", "DET");
+CONVERSION_TABLE_PENN_TO_UD.put("WRB", "ADV");
+
+/*
+ * Note: The back conversion might lose information.
+ */
+CONVERSION_TABLE_UD_TO_PENN.put("ADJ", "JJ");
+

[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format

2024-05-23 Thread ASF GitHub Bot (Jira)


[ 
https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848997#comment-17848997
 ] 

ASF GitHub Bot commented on OPENNLP-1539:
-

rzo1 commented on PR #601:
URL: https://github.com/apache/opennlp/pull/601#issuecomment-2127381263

   Note: The currently failing tests are unrelated to the actual change:
   
   ```bash
   Error:ChunkerModelLoaderTest.initResources:43->lambda$initResources$0:47 
Runtime java.io.IOException: Server returned HTTP response code: 503 for URL: 
https://opennlp.sourceforge.net/models-1.5/en-chunker.bin
   Error:
TokenNameFinderModelLoaderTest.initResources:43->lambda$initResources$0:47 
Runtime java.io.IOException: Server returned HTTP response code: 503 for URL: 
https://opennlp.sourceforge.net/models-1.5/en-ner-location.bin
   Error:
TokenNameFinderModelTest.testNERWithPOSModelV15:122->AbstractModelLoaderTest.downloadVersion15Model:41->AbstractModelLoaderTest.downloadModel:57
 ยป IO Server returned HTTP response code: 503 for URL: 
https://opennlp.sourceforge.net/models-1.5/pt-pos-perceptron.bin
   ```




> Introduce parameter for POSTaggerME to configure output POS tag format
> --
>
> Key: OPENNLP-1539
> URL: https://issues.apache.org/jira/browse/OPENNLP-1539
> Project: OpenNLP
>  Issue Type: Improvement
>  Components: POS Tagger
>Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
>Reporter: Martin Wiesner
>Assignee: Richard Zowalla
>Priority: Major
> Fix For: 2.4.0
>
>
> [Classic (legacy) POS models|https://opennlp.sourceforge.net/models-1.5/] 
> output tags in the [PENN Treebank POS 
> tag|https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html]
>  format.
> The modern UD-based models, however, differ in the [longer output 
> format|https://universaldependencies.org/u/pos/], e.g. "VB" (Penn) vs. "VERB" 
> (UD). Extended (UD) word features are covered here: 
> https://universaldependencies.org/u/feat/index.html
> This difference results in mismatches and will cause existing IT / tests to 
> fail, if executed. Luckily, a mapping table is found here: 
> https://universaldependencies.org/tagset-conversion/en-penn-uposf.html
> To provide compatibility for existing applications and/or use-cases, we need 
> to provide a way to retrieve both POS formats. 
> Aims:
> - Introduce a constructor parameter for POSTaggerME to configure tag format / 
> style: Penn or UD style
> - Implement a mapping between both POS tag formats: UD <==> Penn
> - Update the OpenNLP Manual to explain differences of POS tag format and 
> configuration parameter
> Conceptual idea:
> - {{new POSTaggerME("en")}}  => by _default_: UD format "as is"
> - {{new POSTaggerME("en", POSTagFormat.PENN)}} => by _intention_, here: Penn 
> style
> Benefit: 
> 1. It should be explicit so devs / user see what they will get via 
> {{POSTagFormat}}. Enum values: POSTagFormat.UD, POSTagFormat.PENN, 
> POSTagFormat.DEFAULT
> 2. IT tests can now be formulated to work on both modern and legacy models.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format

2024-05-23 Thread ASF GitHub Bot (Jira)


[ 
https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848994#comment-17848994
 ] 

ASF GitHub Bot commented on OPENNLP-1539:
-

rzo1 opened a new pull request, #601:
URL: https://github.com/apache/opennlp/pull/601

   ### For all changes:
   - [x] Is there a JIRA ticket associated with this PR? Is it referenced 
in the commit message?
   
   - [x] Does your PR title start with OPENNLP- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [x] Has your PR been rebased against the latest commit within the target 
branch (typically main)?
   
   - [x] Is your initial contribution a single, squashed commit?
   
   ### For code changes:
   - [x] Have you ensured that the full suite of tests is executed via mvn 
clean install at the root opennlp folder?
   - [x] Have you written or updated unit tests to verify your changes?
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
   - [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file in opennlp folder?
   - [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found in opennlp folder?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   
   This is a **Draft** open for feedback on implementing compatibility with 
older PENN-based POS models from Sourceforge. 
   
   




> Introduce parameter for POSTaggerME to configure output POS tag format
> --
>
> Key: OPENNLP-1539
> URL: https://issues.apache.org/jira/browse/OPENNLP-1539
> Project: OpenNLP
>  Issue Type: Improvement
>  Components: POS Tagger
>Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
>Reporter: Martin Wiesner
>Assignee: Richard Zowalla
>Priority: Major
> Fix For: 2.4.0
>
>
> [Classic (legacy) POS models|https://opennlp.sourceforge.net/models-1.5/] 
> output tags in the [PENN Treebank POS 
> tag|https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html]
>  format.
> The modern UD-based models, however, differ in the [longer output 
> format|https://universaldependencies.org/u/pos/], e.g. "VB" (Penn) vs. "VERB" 
> (UD). Extended (UD) word features are covered here: 
> https://universaldependencies.org/u/feat/index.html
> This difference results in mismatches and will cause existing IT / tests to 
> fail, if executed. Luckily, a mapping table is found here: 
> https://universaldependencies.org/tagset-conversion/en-penn-uposf.html
> To provide compatibility for existing applications and/or use-cases, we need 
> to provide a way to retrieve both POS formats. 
> Aims:
> - Introduce a constructor parameter for POSTaggerME to configure tag format / 
> style: Penn or UD style
> - Implement a mapping between both POS tag formats: UD <==> Penn
> - Update the OpenNLP Manual to explain differences of POS tag format and 
> configuration parameter
> Conceptual idea:
> - {{new POSTaggerME("en")}}  => by _default_: UD format "as is"
> - {{new POSTaggerME("en", POSTagFormat.PENN)}} => by _intention_, here: Penn 
> style
> Benefit: 
> 1. It should be explicit so devs / user see what they will get via 
> {{POSTagFormat}}. Enum values: POSTagFormat.UD, POSTagFormat.PENN, 
> POSTagFormat.DEFAULT
> 2. IT tests can now be formulated to work on both modern and legacy models.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (OPENNLP-1539) Introduce parameter for POSTaggerME to configure output POS tag format

2023-12-29 Thread Martin Wiesner (Jira)


[ 
https://issues.apache.org/jira/browse/OPENNLP-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17801225#comment-17801225
 ] 

Martin Wiesner commented on OPENNLP-1539:
-

Related StOf post is here: https://stackoverflow.com/q/56505268/2849346

> Introduce parameter for POSTaggerME to configure output POS tag format
> --
>
> Key: OPENNLP-1539
> URL: https://issues.apache.org/jira/browse/OPENNLP-1539
> Project: OpenNLP
>  Issue Type: Improvement
>  Components: POS Tagger
>Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
>Reporter: Martin Wiesner
>Priority: Major
>
> [Classic (legacy) POS models|https://opennlp.sourceforge.net/models-1.5/] 
> output tags in the [PENN Treebank POS 
> tag|https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html]
>  format.
> The modern UD-based models, however, differ in the longer output format, e.g. 
> "VB" (Penn) vs. "VERB" (UD). This results in mismatches and will cause 
> existing IT / tests to fail, if executed.
> To provide compatibility for existing applications and/or use-cases, we need 
> to provide a way to retrieve both POS formats. 
> Aims:
> - Introduce a constructor parameter for POSTaggerME to configure tag format / 
> style: Penn or UD style
> - Implement a mapping between both POS tag formats: UD <==> Penn
> - Update the OpenNLP Manual to explain differences of POS tag format and 
> configuration parameter
> Conceptual idea:
> - {{new POSTaggerME("en")}}  => by _default_: UD format "as is"
> - {{new POSTaggerME("en", POSTagFormat.PENN)}} => by _intention_, here: Penn 
> style
> Benefit: 
> 1. It should be explicit so devs / user see what they will get via 
> {{POSTagFormat}}. Enum values: POSTagFormat.UD, POSTagFormat.PENN, 
> POSTagFormat.DEFAULT
> 2. IT tests can now be formulated to work on both modern and legacy models.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)