[jira] [Commented] (OPENNLP-589) Text format of Events inconsistent across different implementations of EventStreamReaders
[ https://issues.apache.org/jira/browse/OPENNLP-589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17836942#comment-17836942 ] ASF GitHub Bot commented on OPENNLP-589: mawiesne opened a new pull request, #596: URL: https://github.com/apache/opennlp/pull/596 Change - - adjusts input format in RealBasicEventStream to match the other EventStream's format expectations (consistency!) - adds AbstractEventStreamTest as common base for all related event stream classes - adds new RealBasicEventStreamTest to cover the consistency checks - adjusts existing ScaleDoesntMatterTest to match the (consistent) format - adds further test cases for yet untested methods and/or branches (coverage++) - improves existing JavaDoc - adds further JavaDoc along the path Tasks - Thank you for contributing to Apache OpenNLP. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with OPENNLP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically main)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn clean install at the root opennlp folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file in opennlp folder? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found in opennlp folder? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions for build issues and submit an update to your PR as soon as possible. > Text format of Events inconsistent across different implementations of > EventStreamReaders > - > > Key: OPENNLP-589 > URL: https://issues.apache.org/jira/browse/OPENNLP-589 > Project: OpenNLP > Issue Type: Bug > Components: Machine Learning >Affects Versions: maxent-3.0.3, 2.0.0, 2.1.0, 2.2.0, 2.3.0 >Reporter: Marcin Junczys-Dowmunt >Assignee: Martin Wiesner >Priority: Minor > Fix For: 2.3.3 > > > BasicEventStream expects events to be written to text files as: > context1 context2 context3 ... outcome > FileEventStream expects events to be written to text files as: > outcome context1 context2 context3 ... > toString() of Event creates: > outcome [context1 context2 context3 ...] (note the square brackets, which are > part of context predicates when breaking on spaces). > This is highly confusing and took me some time to understand. I guess this > should be unified? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (OPENNLP-589) Text format of Events inconsistent across different implementations of EventStreamReaders
[ https://issues.apache.org/jira/browse/OPENNLP-589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17836943#comment-17836943 ] ASF GitHub Bot commented on OPENNLP-589: mawiesne commented on PR #596: URL: https://github.com/apache/opennlp/pull/596#issuecomment-2054012917 Note: Seems a previously un-tested piece of code isn't liked that much on Windo*s machines. 😀 investigating... > Text format of Events inconsistent across different implementations of > EventStreamReaders > - > > Key: OPENNLP-589 > URL: https://issues.apache.org/jira/browse/OPENNLP-589 > Project: OpenNLP > Issue Type: Bug > Components: Machine Learning >Affects Versions: maxent-3.0.3, 2.0.0, 2.1.0, 2.2.0, 2.3.0 >Reporter: Marcin Junczys-Dowmunt >Assignee: Martin Wiesner >Priority: Minor > Fix For: 2.3.3 > > > BasicEventStream expects events to be written to text files as: > context1 context2 context3 ... outcome > FileEventStream expects events to be written to text files as: > outcome context1 context2 context3 ... > toString() of Event creates: > outcome [context1 context2 context3 ...] (note the square brackets, which are > part of context predicates when breaking on spaces). > This is highly confusing and took me some time to understand. I guess this > should be unified? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (OPENNLP-589) Text format of Events inconsistent across different implementations of EventStreamReaders
[ https://issues.apache.org/jira/browse/OPENNLP-589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17836944#comment-17836944 ] ASF GitHub Bot commented on OPENNLP-589: mawiesne commented on PR #596: URL: https://github.com/apache/opennlp/pull/596#issuecomment-2054014916 > Note: Seems a previously un-tested piece of code isn't liked that much on Windo*s machines. 😀 > > investigating... Done - test now fully passing, see checks. > Text format of Events inconsistent across different implementations of > EventStreamReaders > - > > Key: OPENNLP-589 > URL: https://issues.apache.org/jira/browse/OPENNLP-589 > Project: OpenNLP > Issue Type: Bug > Components: Machine Learning >Affects Versions: maxent-3.0.3, 2.0.0, 2.1.0, 2.2.0, 2.3.0 >Reporter: Marcin Junczys-Dowmunt >Assignee: Martin Wiesner >Priority: Minor > Fix For: 2.3.3 > > > BasicEventStream expects events to be written to text files as: > context1 context2 context3 ... outcome > FileEventStream expects events to be written to text files as: > outcome context1 context2 context3 ... > toString() of Event creates: > outcome [context1 context2 context3 ...] (note the square brackets, which are > part of context predicates when breaking on spaces). > This is highly confusing and took me some time to understand. I guess this > should be unified? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (OPENNLP-589) Text format of Events inconsistent across different implementations of EventStreamReaders
[ https://issues.apache.org/jira/browse/OPENNLP-589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17836991#comment-17836991 ] ASF GitHub Bot commented on OPENNLP-589: rzo1 commented on code in PR #596: URL: https://github.com/apache/opennlp/pull/596#discussion_r1564855870 ## opennlp-tools/src/main/java/opennlp/tools/ml/maxent/RealBasicEventStream.java: ## @@ -49,13 +55,14 @@ public Event read() throws IOException { } private Event createEvent(String obs) { -int lastSpace = obs.lastIndexOf(' '); -if (lastSpace == -1) +int si = obs.indexOf(' '); +if (si == 0) Review Comment: Shouldn't that be `-1` ? Otherwise, it might produce an index out of bound exception in the else branch. ## opennlp-tools/src/test/java/opennlp/tools/ml/maxent/RealBasicEventStreamTest.java: ## @@ -0,0 +1,107 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package opennlp.tools.ml.maxent; + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.nio.charset.StandardCharsets; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import opennlp.tools.ml.AbstractEventStreamTest; +import opennlp.tools.ml.model.Event; +import opennlp.tools.ml.model.RealValueFileEventStream; +import opennlp.tools.util.InputStreamFactory; +import opennlp.tools.util.ObjectStream; +import opennlp.tools.util.PlainTextByLineStream; + +import static org.junit.jupiter.api.Assertions.fail; + +/** + * Verifies that the textual event (input) format in {@link RealBasicEventStream} is: + * + * {@code outcome context1 context2 context3 ...} + * + * and is consistent with {@link RealValueFileEventStream}. Moreover, the test checks that processing + * given input works as expected. + * + * Reported in and adjusted via: + * https://issues.apache.org/jira/projects/OPENNLP/issues/OPENNLP-589";>OPENNLP-589. + * + * @see ObjectStream + * @see RealValueFileEventStream + */ +public class RealBasicEventStreamTest extends AbstractEventStreamTest { + + @Override + protected RealBasicEventStream createEventStream(String input) throws IOException { +InputStreamFactory factory = () -> new ByteArrayInputStream(input.getBytes(StandardCharsets.UTF_8)); +ObjectStream stream = new PlainTextByLineStream(factory, StandardCharsets.UTF_8); +return new RealBasicEventStream(stream); + } + + /** + * See: {@link AbstractEventStreamTest#EVENTS} for the input data. + */ + @Test + void testReadWithValidInput() throws IOException { +try (ObjectStream eventStream = createEventStream(EVENTS)) { + Assertions.assertEquals("other [wc=ic=1.0 w&c=he,ic=2.0 n1wc=lc=3.0 n1w&c=belongs,lc=4.0 n2wc=lc=5.0]", + eventStream.read().toString()); + Assertions.assertEquals("other [wc=lc=1.0 w&c=belongs,lc=2.0 p1wc=ic=3.0 p1w&c=he,ic=4.0 n1wc=lc=5.0]", + eventStream.read().toString()); + Assertions.assertEquals("other [wc=lc=1.0 w&c=to,lc=2.0 p1wc=lc=3.0 p1w&c=belongs,lc=4.0 p2wc=ic=5.0]", + eventStream.read().toString()); + Assertions.assertEquals("org-start [wc=ic=1.0 w&c=apache,ic=2.0 p1wc=lc=3.0 p1w&c=to,lc=4.0]", + eventStream.read().toString()); + Assertions.assertEquals("org-cont [wc=ic=1.0 w&c=software,ic=2.0 p1wc=ic=3.0 p1w&c=apache,ic=4.0]", + eventStream.read().toString()); + Assertions.assertEquals("org-cont [wc=ic=1.0 w&c=foundation,ic=2.0 p1wc=ic=3.0 p1w&c=software,ic=4.0]", + eventStream.read().toString()); + Assertions.assertEquals("other [wc=other=1.0 w&c=.,other=2.0 p1wc=ic=3.0]", + eventStream.read().toString()); + Assertions.assertNull(eventStream.read()); +} + } + + @Test + void testReatWithInvalidNegativeValues() throws IOException { Review Comment: `Reat`-> `Read` > Text format of Events inconsistent across different implementations of > EventStreamReaders > - > > Key: OPENNLP-589 > URL: https://issues.apache.org/jira/browse/OPENNLP-589 > Projec
[jira] [Commented] (OPENNLP-589) Text format of Events inconsistent across different implementations of EventStreamReaders
[ https://issues.apache.org/jira/browse/OPENNLP-589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17836992#comment-17836992 ] ASF GitHub Bot commented on OPENNLP-589: rzo1 commented on code in PR #596: URL: https://github.com/apache/opennlp/pull/596#discussion_r1564855870 ## opennlp-tools/src/main/java/opennlp/tools/ml/maxent/RealBasicEventStream.java: ## @@ -49,13 +55,14 @@ public Event read() throws IOException { } private Event createEvent(String obs) { -int lastSpace = obs.lastIndexOf(' '); -if (lastSpace == -1) +int si = obs.indexOf(' '); +if (si == 0) Review Comment: Shouldn't that be `-1` ? Otherwise, it might produce an index out of bound exception in the else branch (because nothing prevents the caller to just insert `bla` as an element in the event stream (alhough it does not comply with the text format). > Text format of Events inconsistent across different implementations of > EventStreamReaders > - > > Key: OPENNLP-589 > URL: https://issues.apache.org/jira/browse/OPENNLP-589 > Project: OpenNLP > Issue Type: Bug > Components: Machine Learning >Affects Versions: maxent-3.0.3, 2.0.0, 2.1.0, 2.2.0, 2.3.0 >Reporter: Marcin Junczys-Dowmunt >Assignee: Martin Wiesner >Priority: Minor > Fix For: 2.3.3 > > > BasicEventStream expects events to be written to text files as: > context1 context2 context3 ... outcome > FileEventStream expects events to be written to text files as: > outcome context1 context2 context3 ... > toString() of Event creates: > outcome [context1 context2 context3 ...] (note the square brackets, which are > part of context predicates when breaking on spaces). > This is highly confusing and took me some time to understand. I guess this > should be unified? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (OPENNLP-589) Text format of Events inconsistent across different implementations of EventStreamReaders
[ https://issues.apache.org/jira/browse/OPENNLP-589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17838651#comment-17838651 ] ASF GitHub Bot commented on OPENNLP-589: mawiesne merged PR #596: URL: https://github.com/apache/opennlp/pull/596 > Text format of Events inconsistent across different implementations of > EventStreamReaders > - > > Key: OPENNLP-589 > URL: https://issues.apache.org/jira/browse/OPENNLP-589 > Project: OpenNLP > Issue Type: Bug > Components: Machine Learning >Affects Versions: maxent-3.0.3, 2.0.0, 2.1.0, 2.2.0, 2.3.0 >Reporter: Marcin Junczys-Dowmunt >Assignee: Martin Wiesner >Priority: Minor > Fix For: 2.3.3 > > > BasicEventStream expects events to be written to text files as: > context1 context2 context3 ... outcome > FileEventStream expects events to be written to text files as: > outcome context1 context2 context3 ... > toString() of Event creates: > outcome [context1 context2 context3 ...] (note the square brackets, which are > part of context predicates when breaking on spaces). > This is highly confusing and took me some time to understand. I guess this > should be unified? -- This message was sent by Atlassian Jira (v8.20.10#820010)