[jira] [Commented] (OPENNLP-1563) SimpleTokenizer.tokenizePos incorrectly splits words with non-spacing letters

2024-05-28 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/OPENNLP-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17850101#comment-17850101
 ] 

ASF GitHub Bot commented on OPENNLP-1563:
-

jzonthemtn commented on PR #602:
URL: https://github.com/apache/opennlp/pull/602#issuecomment-2135699051

   Thanks @demq! 




> SimpleTokenizer.tokenizePos incorrectly splits words with non-spacing letters
> -
>
> Key: OPENNLP-1563
> URL: https://issues.apache.org/jira/browse/OPENNLP-1563
> Project: OpenNLP
>  Issue Type: Bug
>  Components: Tokenizer
>Affects Versions: 2.3.3
>Reporter: Hrayr Matevosyan
>Priority: Major
> Fix For: 2.3.4
>
>
> The tokenizerPos implementation of SimpleTokenizer incorrectly tokenizes 
> words containing non-spacing letters. For example, the Arabic word "طُوّر" 
> gets tokenized to individual letters ["ط", "ُ", "و", "ّ", "ر"].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (OPENNLP-1563) SimpleTokenizer.tokenizePos incorrectly splits words with non-spacing letters

2024-05-28 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/OPENNLP-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849994#comment-17849994
 ] 

ASF GitHub Bot commented on OPENNLP-1563:
-

rzo1 merged PR #602:
URL: https://github.com/apache/opennlp/pull/602




> SimpleTokenizer.tokenizePos incorrectly splits words with non-spacing letters
> -
>
> Key: OPENNLP-1563
> URL: https://issues.apache.org/jira/browse/OPENNLP-1563
> Project: OpenNLP
>  Issue Type: Bug
>  Components: Tokenizer
>Affects Versions: 2.3.3
>Reporter: Hrayr Matevosyan
>Priority: Major
>
> The tokenizerPos implementation of SimpleTokenizer incorrectly tokenizes 
> words containing non-spacing letters. For example, the Arabic word "طُوّر" 
> gets tokenized to individual letters ["ط", "ُ", "و", "ّ", "ر"].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (OPENNLP-1563) SimpleTokenizer.tokenizePos incorrectly splits words with non-spacing letters

2024-05-28 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/OPENNLP-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849993#comment-17849993
 ] 

ASF GitHub Bot commented on OPENNLP-1563:
-

mawiesne commented on PR #602:
URL: https://github.com/apache/opennlp/pull/602#issuecomment-2134979949

   Thx @demq  !




> SimpleTokenizer.tokenizePos incorrectly splits words with non-spacing letters
> -
>
> Key: OPENNLP-1563
> URL: https://issues.apache.org/jira/browse/OPENNLP-1563
> Project: OpenNLP
>  Issue Type: Bug
>  Components: Tokenizer
>Affects Versions: 2.3.3
>Reporter: Hrayr Matevosyan
>Priority: Major
>
> The tokenizerPos implementation of SimpleTokenizer incorrectly tokenizes 
> words containing non-spacing letters. For example, the Arabic word "طُوّر" 
> gets tokenized to individual letters ["ط", "ُ", "و", "ّ", "ر"].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (OPENNLP-1563) SimpleTokenizer.tokenizePos incorrectly splits words with non-spacing letters

2024-05-28 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/OPENNLP-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849906#comment-17849906
 ] 

ASF GitHub Bot commented on OPENNLP-1563:
-

demq commented on code in PR #602:
URL: https://github.com/apache/opennlp/pull/602#discussion_r1616698074


##
opennlp-tools/src/test/java/opennlp/tools/tokenize/SimpleTokenizerTest.java:
##
@@ -128,4 +128,18 @@ void testTokenizationOfStringWithWindowsNewLineTokens() {
 Assertions.assertArrayEquals(new String[] {"a", "\r", "\n", "\r", "\n", 
"b", "\r", "\n", "\r", "\n", "c"},
 tokenizer.tokenize("a\r\n\r\n b\r\n\r\n c"));
   }
+
+  /**
+   * Tests if it can tokenize a word containing a non-spacing character
+   * like Arabic Damma Unicode Character “◌ُ” (U+064F)
+   */
+  @Test
+  void testNonSpacingLetters() {
+String text = "طُوّر";

Review Comment:
   I have just pushed an update with a full sentence.





> SimpleTokenizer.tokenizePos incorrectly splits words with non-spacing letters
> -
>
> Key: OPENNLP-1563
> URL: https://issues.apache.org/jira/browse/OPENNLP-1563
> Project: OpenNLP
>  Issue Type: Bug
>  Components: Tokenizer
>Affects Versions: 2.3.3
>Reporter: Hrayr Matevosyan
>Priority: Major
>
> The tokenizerPos implementation of SimpleTokenizer incorrectly tokenizes 
> words containing non-spacing letters. For example, the Arabic word "طُوّر" 
> gets tokenized to individual letters ["ط", "ُ", "و", "ّ", "ر"].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (OPENNLP-1563) SimpleTokenizer.tokenizePos incorrectly splits words with non-spacing letters

2024-05-28 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/OPENNLP-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849899#comment-17849899
 ] 

ASF GitHub Bot commented on OPENNLP-1563:
-

rzo1 commented on code in PR #602:
URL: https://github.com/apache/opennlp/pull/602#discussion_r1616664137


##
opennlp-tools/src/test/java/opennlp/tools/tokenize/SimpleTokenizerTest.java:
##
@@ -128,4 +128,18 @@ void testTokenizationOfStringWithWindowsNewLineTokens() {
 Assertions.assertArrayEquals(new String[] {"a", "\r", "\n", "\r", "\n", 
"b", "\r", "\n", "\r", "\n", "c"},
 tokenizer.tokenize("a\r\n\r\n b\r\n\r\n c"));
   }
+
+  /**
+   * Tests if it can tokenize a word containing a non-spacing character
+   * like Arabic Damma Unicode Character “◌ُ” (U+064F)
+   */
+  @Test
+  void testNonSpacingLetters() {
+String text = "طُوّر";

Review Comment:
   @demq Can we have a full sentence example here?





> SimpleTokenizer.tokenizePos incorrectly splits words with non-spacing letters
> -
>
> Key: OPENNLP-1563
> URL: https://issues.apache.org/jira/browse/OPENNLP-1563
> Project: OpenNLP
>  Issue Type: Bug
>  Components: Tokenizer
>Affects Versions: 2.3.3
>Reporter: Hrayr Matevosyan
>Priority: Major
>
> The tokenizerPos implementation of SimpleTokenizer incorrectly tokenizes 
> words containing non-spacing letters. For example, the Arabic word "طُوّر" 
> gets tokenized to individual letters ["ط", "ُ", "و", "ّ", "ر"].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (OPENNLP-1563) SimpleTokenizer.tokenizePos incorrectly splits words with non-spacing letters

2024-05-28 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/OPENNLP-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849892#comment-17849892
 ] 

ASF GitHub Bot commented on OPENNLP-1563:
-

demq opened a new pull request, #602:
URL: https://github.com/apache/opennlp/pull/602

   Thank you for contributing to Apache OpenNLP.
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [x] Is there a JIRA ticket associated with this PR? Is it referenced 
in the commit message?
   
   - [x] Does your PR title start with OPENNLP- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [x] Has your PR been rebased against the latest commit within the target 
branch (typically main)?
   
   - [x] Is your initial contribution a single, squashed commit?
   
   ### For code changes:
   - [x] Have you ensured that the full suite of tests is executed via mvn 
clean install at the root opennlp folder?
   - [x] Have you written or updated unit tests to verify your changes?
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
   - [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file in opennlp folder?
   - [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found in opennlp folder?
   
   ### For documentation related changes:
   - [x] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check GitHub Actions for 
build issues and submit an update to your PR as soon as possible.
   




> SimpleTokenizer.tokenizePos incorrectly splits words with non-spacing letters
> -
>
> Key: OPENNLP-1563
> URL: https://issues.apache.org/jira/browse/OPENNLP-1563
> Project: OpenNLP
>  Issue Type: Bug
>  Components: Tokenizer
>Affects Versions: 2.3.3
>Reporter: Hrayr Matevosyan
>Priority: Major
>
> The tokenizerPos implementation of SimpleTokenizer incorrectly tokenizes 
> words containing non-spacing letters. For example, the Arabic word "طُوّر" 
> gets tokenized to individual letters ["ط", "ُ", "و", "ّ", "ر"].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)