[jira] [Commented] (NIFI-1701) StreamScanner in PutKafka doesn't handle byte values < -1 properly
[ https://issues.apache.org/jira/browse/NIFI-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219390#comment-15219390 ] Joseph Witt commented on NIFI-1701: --- i think this warrants an 0.6.1 release. The impact of it is that certain bytes can cause the demarcation logic to stop too soon resulting in data loss in delivery to kafka. > StreamScanner in PutKafka doesn't handle byte values < -1 properly > -- > > Key: NIFI-1701 > URL: https://issues.apache.org/jira/browse/NIFI-1701 > Project: Apache NiFi > Issue Type: Bug >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 0.7.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1701) StreamScanner in PutKafka doesn't handle byte values < -1 properly
[ https://issues.apache.org/jira/browse/NIFI-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219413#comment-15219413 ] Joseph Witt commented on NIFI-1701: --- full clean build w/contrib check works out. Reviewed code thoroughly. Tested on two different live clusters with some wild demarcation strategies and source data. All checks out. I am a +1. Let's give it some time to see if others want to check it out. > StreamScanner in PutKafka doesn't handle byte values < -1 properly > -- > > Key: NIFI-1701 > URL: https://issues.apache.org/jira/browse/NIFI-1701 > Project: Apache NiFi > Issue Type: Bug >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 0.7.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1701) StreamScanner in PutKafka doesn't handle byte values < -1 properly
[ https://issues.apache.org/jira/browse/NIFI-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219452#comment-15219452 ] Joseph Witt commented on NIFI-1701: --- left some comments on the PR itself. > StreamScanner in PutKafka doesn't handle byte values < -1 properly > -- > > Key: NIFI-1701 > URL: https://issues.apache.org/jira/browse/NIFI-1701 > Project: Apache NiFi > Issue Type: Bug >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 0.7.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1701) StreamScanner in PutKafka doesn't handle byte values < -1 properly
[
https://issues.apache.org/jira/browse/NIFI-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219487#comment-15219487
]
ASF GitHub Bot commented on NIFI-1701:
--
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/314#discussion_r58008694
--- Diff:
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/StreamScanner.java
---
@@ -55,29 +56,53 @@
*/
boolean hasNext() {
this.data = null;
-if (!this.eos) {
+int j = 0;
+boolean moreData = true;
+byte b;
+while (this.data == null) {
+this.expandBufferIfNecessary();
try {
-boolean keepReading = true;
-while (keepReading) {
-byte b = (byte) this.is.read();
-if (b > -1) {
-baos.write(b);
-if (buffer.addAndCompare(b)) {
-this.data =
Arrays.copyOfRange(baos.getUnderlyingBuffer(), 0, baos.size() -
delimiter.length);
-keepReading = false;
-}
-} else {
-this.data = baos.toByteArray();
-keepReading = false;
-this.eos = true;
+b = (byte) this.is.read();
--- End diff --
is.read() returns an int and -1 and represents end of stream. Once that
has been verified can the byte value as represented in the range of 0 to 255.
Will submit a patch containing a test that proves it is broken and will make
the fix.
> StreamScanner in PutKafka doesn't handle byte values < -1 properly
> --
>
> Key: NIFI-1701
> URL: https://issues.apache.org/jira/browse/NIFI-1701
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Oleg Zhurakousky
>Assignee: Oleg Zhurakousky
>Priority: Critical
> Fix For: 0.7.0
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1701) StreamScanner in PutKafka doesn't handle byte values < -1 properly
[ https://issues.apache.org/jira/browse/NIFI-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219502#comment-15219502 ] Joseph Witt commented on NIFI-1701: --- [~ozhurakousky] Have attached a patch which applies on top of your PR. It fixes the fact that when reading a byte from the input stream what is returned is an int and the value when -1 means end of stream whereas all other values mean there is a byte there and its value is represented between 0 and 255. The logic we had basically immediately cast to a byte then tested for -1. As a result if the byte stream itself contained a byte representing -1 we would improperly block/stop the stream. Created a unit test to show this as broken and then fixed. > StreamScanner in PutKafka doesn't handle byte values < -1 properly > -- > > Key: NIFI-1701 > URL: https://issues.apache.org/jira/browse/NIFI-1701 > Project: Apache NiFi > Issue Type: Bug >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 0.7.0 > > Attachments: > 0001-NIFI-1701-patch-against-pr-314-which-corrects-byte-v.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1701) StreamScanner in PutKafka doesn't handle byte values < -1 properly
[ https://issues.apache.org/jira/browse/NIFI-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220007#comment-15220007 ] Tim Reardon commented on NIFI-1701: --- This code bit my team yesterday. We moved to 0.6.0, and found that PutKafka was truncating messages. This morning we determined that messages containing multi-byte characters are being truncated at the first occurrence. We are not using a message delimiter. Needless to say, we have reverted to back to our 0.5.1 instance, which itself is using the 0.4.1 Kafka nar due to incompatibility with Kafka 0.8.x. > StreamScanner in PutKafka doesn't handle byte values < -1 properly > -- > > Key: NIFI-1701 > URL: https://issues.apache.org/jira/browse/NIFI-1701 > Project: Apache NiFi > Issue Type: Bug >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 0.7.0 > > Attachments: > 0001-NIFI-1701-patch-against-pr-314-which-corrects-byte-v.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1701) StreamScanner in PutKafka doesn't handle byte values < -1 properly
[ https://issues.apache.org/jira/browse/NIFI-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220009#comment-15220009 ] Joseph Witt commented on NIFI-1701: --- Sorry about that Tim. We will initiate an 0.6.1 vote process asap. We have some very useful tests/conditions now and are running this live on highly complex flows where we originally found the problem. Far better now. > StreamScanner in PutKafka doesn't handle byte values < -1 properly > -- > > Key: NIFI-1701 > URL: https://issues.apache.org/jira/browse/NIFI-1701 > Project: Apache NiFi > Issue Type: Bug >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 0.7.0 > > Attachments: > 0001-NIFI-1701-patch-against-pr-314-which-corrects-byte-v.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1701) StreamScanner in PutKafka doesn't handle byte values < -1 properly
[ https://issues.apache.org/jira/browse/NIFI-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224008#comment-15224008 ] Edgardo Vega commented on NIFI-1701: I haven't seen a 0.6.1 vote. Is there something wrong with this patch? What is the hold up? > StreamScanner in PutKafka doesn't handle byte values < -1 properly > -- > > Key: NIFI-1701 > URL: https://issues.apache.org/jira/browse/NIFI-1701 > Project: Apache NiFi > Issue Type: Bug >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 0.7.0 > > Attachments: > 0001-NIFI-1701-patch-against-pr-314-which-corrects-byte-v.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1701) StreamScanner in PutKafka doesn't handle byte values < -1 properly
[ https://issues.apache.org/jira/browse/NIFI-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224031#comment-15224031 ] Joseph Witt commented on NIFI-1701: --- Will have up hopefully today or tomorrow. Patch works very well. > StreamScanner in PutKafka doesn't handle byte values < -1 properly > -- > > Key: NIFI-1701 > URL: https://issues.apache.org/jira/browse/NIFI-1701 > Project: Apache NiFi > Issue Type: Bug >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 0.7.0 > > Attachments: > 0001-NIFI-1701-patch-against-pr-314-which-corrects-byte-v.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1701) StreamScanner in PutKafka doesn't handle byte values < -1 properly
[ https://issues.apache.org/jira/browse/NIFI-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224163#comment-15224163 ] ASF subversion and git services commented on NIFI-1701: --- Commit 25290cedc4f31c3bece3af34333b96d816dd9ec4 in nifi's branch refs/heads/master from [~ozhurakousky] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=25290ce ] NIFI-1701 fixed StreamScanner, added more tests NIFI-1701 additional refactoring, clean up and more tests > StreamScanner in PutKafka doesn't handle byte values < -1 properly > -- > > Key: NIFI-1701 > URL: https://issues.apache.org/jira/browse/NIFI-1701 > Project: Apache NiFi > Issue Type: Bug >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 0.7.0, 0.6.1 > > Attachments: > 0001-NIFI-1701-patch-against-pr-314-which-corrects-byte-v.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1701) StreamScanner in PutKafka doesn't handle byte values < -1 properly
[ https://issues.apache.org/jira/browse/NIFI-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224164#comment-15224164 ] ASF subversion and git services commented on NIFI-1701: --- Commit 25290cedc4f31c3bece3af34333b96d816dd9ec4 in nifi's branch refs/heads/master from [~ozhurakousky] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=25290ce ] NIFI-1701 fixed StreamScanner, added more tests NIFI-1701 additional refactoring, clean up and more tests > StreamScanner in PutKafka doesn't handle byte values < -1 properly > -- > > Key: NIFI-1701 > URL: https://issues.apache.org/jira/browse/NIFI-1701 > Project: Apache NiFi > Issue Type: Bug >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 0.7.0, 0.6.1 > > Attachments: > 0001-NIFI-1701-patch-against-pr-314-which-corrects-byte-v.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1701) StreamScanner in PutKafka doesn't handle byte values < -1 properly
[ https://issues.apache.org/jira/browse/NIFI-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224169#comment-15224169 ] Mark Payne commented on NIFI-1701: -- Code review looks good. Have been running for a while on a cluster and have been seeing great results. Additionally, there are a lot of really good unit tests included here. Nicely done! +1. Merged to master. > StreamScanner in PutKafka doesn't handle byte values < -1 properly > -- > > Key: NIFI-1701 > URL: https://issues.apache.org/jira/browse/NIFI-1701 > Project: Apache NiFi > Issue Type: Bug >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 0.7.0, 0.6.1 > > Attachments: > 0001-NIFI-1701-patch-against-pr-314-which-corrects-byte-v.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1701) StreamScanner in PutKafka doesn't handle byte values < -1 properly
[ https://issues.apache.org/jira/browse/NIFI-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224257#comment-15224257 ] ASF subversion and git services commented on NIFI-1701: --- Commit d29d5a2a6d1fa19bdae90d7a3c467d9e0a98277c in nifi's branch refs/heads/support/nifi-0.6.x from [~ozhurakousky] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=d29d5a2 ] NIFI-1701 fixed StreamScanner, added more tests NIFI-1701 additional refactoring, clean up and more tests > StreamScanner in PutKafka doesn't handle byte values < -1 properly > -- > > Key: NIFI-1701 > URL: https://issues.apache.org/jira/browse/NIFI-1701 > Project: Apache NiFi > Issue Type: Bug >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 0.7.0, 0.6.1 > > Attachments: > 0001-NIFI-1701-patch-against-pr-314-which-corrects-byte-v.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1701) StreamScanner in PutKafka doesn't handle byte values < -1 properly
[ https://issues.apache.org/jira/browse/NIFI-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224258#comment-15224258 ] ASF subversion and git services commented on NIFI-1701: --- Commit d29d5a2a6d1fa19bdae90d7a3c467d9e0a98277c in nifi's branch refs/heads/support/nifi-0.6.x from [~ozhurakousky] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=d29d5a2 ] NIFI-1701 fixed StreamScanner, added more tests NIFI-1701 additional refactoring, clean up and more tests > StreamScanner in PutKafka doesn't handle byte values < -1 properly > -- > > Key: NIFI-1701 > URL: https://issues.apache.org/jira/browse/NIFI-1701 > Project: Apache NiFi > Issue Type: Bug >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 0.7.0, 0.6.1 > > Attachments: > 0001-NIFI-1701-patch-against-pr-314-which-corrects-byte-v.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1701) StreamScanner in PutKafka doesn't handle byte values < -1 properly
[ https://issues.apache.org/jira/browse/NIFI-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224298#comment-15224298 ] ASF GitHub Bot commented on NIFI-1701: -- Github user olegz closed the pull request at: https://github.com/apache/nifi/pull/314 > StreamScanner in PutKafka doesn't handle byte values < -1 properly > -- > > Key: NIFI-1701 > URL: https://issues.apache.org/jira/browse/NIFI-1701 > Project: Apache NiFi > Issue Type: Bug >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 0.7.0, 0.6.1 > > Attachments: > 0001-NIFI-1701-patch-against-pr-314-which-corrects-byte-v.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1701) StreamScanner in PutKafka doesn't handle byte values < -1 properly
[ https://issues.apache.org/jira/browse/NIFI-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224551#comment-15224551 ] ASF subversion and git services commented on NIFI-1701: --- Commit 15bab7dda858f6f20c73a7b729962bb7c1f130b9 in nifi's branch refs/heads/support/nifi-0.6.x from [~ozhurakousky] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=15bab7d ] NIFI-1701 fixed merge conflicts > StreamScanner in PutKafka doesn't handle byte values < -1 properly > -- > > Key: NIFI-1701 > URL: https://issues.apache.org/jira/browse/NIFI-1701 > Project: Apache NiFi > Issue Type: Bug >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 0.7.0, 0.6.1 > > Attachments: > 0001-NIFI-1701-patch-against-pr-314-which-corrects-byte-v.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1701) StreamScanner in PutKafka doesn't handle byte values < -1 properly
[ https://issues.apache.org/jira/browse/NIFI-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224644#comment-15224644 ] ASF subversion and git services commented on NIFI-1701: --- Commit b5e007213bfb8769842fce9f770a9890f2661d8c in nifi's branch refs/heads/support/nifi-0.6.x from [~ozhurakousky] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=b5e0072 ] NIFI-1701 fixed StreamScanner, added more tests NIFI-1701 additional refactoring, clean up and more tests > StreamScanner in PutKafka doesn't handle byte values < -1 properly > -- > > Key: NIFI-1701 > URL: https://issues.apache.org/jira/browse/NIFI-1701 > Project: Apache NiFi > Issue Type: Bug >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 0.7.0, 0.6.1 > > Attachments: > 0001-NIFI-1701-patch-against-pr-314-which-corrects-byte-v.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1701) StreamScanner in PutKafka doesn't handle byte values < -1 properly
[ https://issues.apache.org/jira/browse/NIFI-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224645#comment-15224645 ] ASF subversion and git services commented on NIFI-1701: --- Commit b5e007213bfb8769842fce9f770a9890f2661d8c in nifi's branch refs/heads/support/nifi-0.6.x from [~ozhurakousky] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=b5e0072 ] NIFI-1701 fixed StreamScanner, added more tests NIFI-1701 additional refactoring, clean up and more tests > StreamScanner in PutKafka doesn't handle byte values < -1 properly > -- > > Key: NIFI-1701 > URL: https://issues.apache.org/jira/browse/NIFI-1701 > Project: Apache NiFi > Issue Type: Bug >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 0.7.0, 0.6.1 > > Attachments: > 0001-NIFI-1701-patch-against-pr-314-which-corrects-byte-v.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
