[jira] [Commented] (KAFKA-8824) InMemoryTimeOrderedKeyValueBuffer propagates nulls when supress is configured
[ https://issues.apache.org/jira/browse/KAFKA-8824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913682#comment-16913682 ] ASF GitHub Bot commented on KAFKA-8824: --- guozhangwang commented on pull request #7235: KAFKA-8824: bypass value serde on null URL: https://github.com/apache/kafka/pull/7235 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > InMemoryTimeOrderedKeyValueBuffer propagates nulls when supress is configured > -- > > Key: KAFKA-8824 > URL: https://issues.apache.org/jira/browse/KAFKA-8824 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.3.0 >Reporter: Ferran altimiras >Assignee: John Roesler >Priority: Major > Fix For: 2.4.0, 2.3.1 > > Attachments: Test.java > > > Maybe this is not a bug, but it looks like something is wrong. This didn't > happen in kafka streams 2.2. > > Applying an aggregate() with suppress on kafka 2.3 sends nulls into > Serializer if delayed msgs are received. > Not sure if some data is lost or not(yet). But IMHO getting a null to > serialize the "accumulator" object is suspicious that something is wrong. > > Attached java code to demonstrate it. > With kafka 2.3 -> LongSerde prints NULL, not in kafka 2.2 > > > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (KAFKA-8824) InMemoryTimeOrderedKeyValueBuffer propagates nulls when supress is configured
[ https://issues.apache.org/jira/browse/KAFKA-8824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16912800#comment-16912800 ] ASF GitHub Bot commented on KAFKA-8824: --- vvcephei commented on pull request #7235: KAFKA-8824: bypass value serde on null URL: https://github.com/apache/kafka/pull/7235 In a KTable context, we should not pass `null` into a user-supplied serde. Testing: I verified that the change to the test results in test failures without the patch. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > InMemoryTimeOrderedKeyValueBuffer propagates nulls when supress is configured > -- > > Key: KAFKA-8824 > URL: https://issues.apache.org/jira/browse/KAFKA-8824 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.3.0 >Reporter: Ferran altimiras >Assignee: John Roesler >Priority: Major > Fix For: 2.4.0, 2.3.1 > > Attachments: Test.java > > > Maybe this is not a bug, but it looks like something is wrong. This didn't > happen in kafka streams 2.2. > > Applying an aggregate() with suppress on kafka 2.3 sends nulls into > Serializer if delayed msgs are received. > Not sure if some data is lost or not(yet). But IMHO getting a null to > serialize the "accumulator" object is suspicious that something is wrong. > > Attached java code to demonstrate it. > With kafka 2.3 -> LongSerde prints NULL, not in kafka 2.2 > > > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (KAFKA-8824) InMemoryTimeOrderedKeyValueBuffer propagates nulls when supress is configured
[ https://issues.apache.org/jira/browse/KAFKA-8824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16912802#comment-16912802 ] John Roesler commented on KAFKA-8824: - Ok, here's the patch: https://github.com/apache/kafka/pull/7235 > InMemoryTimeOrderedKeyValueBuffer propagates nulls when supress is configured > -- > > Key: KAFKA-8824 > URL: https://issues.apache.org/jira/browse/KAFKA-8824 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.3.0 >Reporter: Ferran altimiras >Assignee: John Roesler >Priority: Major > Fix For: 2.4.0, 2.3.1 > > Attachments: Test.java > > > Maybe this is not a bug, but it looks like something is wrong. This didn't > happen in kafka streams 2.2. > > Applying an aggregate() with suppress on kafka 2.3 sends nulls into > Serializer if delayed msgs are received. > Not sure if some data is lost or not(yet). But IMHO getting a null to > serialize the "accumulator" object is suspicious that something is wrong. > > Attached java code to demonstrate it. > With kafka 2.3 -> LongSerde prints NULL, not in kafka 2.2 > > > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (KAFKA-8824) InMemoryTimeOrderedKeyValueBuffer propagates nulls when supress is configured
[ https://issues.apache.org/jira/browse/KAFKA-8824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16912788#comment-16912788 ] John Roesler commented on KAFKA-8824: - By the way, many thanks for the repro code. It made it so much faster to get to the root cause of this bug! > InMemoryTimeOrderedKeyValueBuffer propagates nulls when supress is configured > -- > > Key: KAFKA-8824 > URL: https://issues.apache.org/jira/browse/KAFKA-8824 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.3.0, 2.4.0 >Reporter: Ferran altimiras >Priority: Major > Fix For: 2.4.0, 2.3.1 > > Attachments: Test.java > > > Maybe this is not a bug, but it looks like something is wrong. This didn't > happen in kafka streams 2.2. > > Applying an aggregate() with suppress on kafka 2.3 sends nulls into > Serializer if delayed msgs are received. > Not sure if some data is lost or not(yet). But IMHO getting a null to > serialize the "accumulator" object is suspicious that something is wrong. > > Attached java code to demonstrate it. > With kafka 2.3 -> LongSerde prints NULL, not in kafka 2.2 > > > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (KAFKA-8824) InMemoryTimeOrderedKeyValueBuffer propagates nulls when supress is configured
[
https://issues.apache.org/jira/browse/KAFKA-8824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16912787#comment-16912787
]
John Roesler commented on KAFKA-8824:
-
Ah, I think this was just a mistake on my part.
The code in 2.2 used the "FullChangeSerde" to serialize the (old value, new
value) pair when buffering records, which bypasses the serializer for a null
value:
{code}
final byte[] oldBytes = data.oldValue == null ? null :
innerSerializer.serialize(topic, data.oldValue);
{code}
But I refactored it in 2.3, and now it doesn't do the bypass
{code}
final V priorValue = value.oldValue;
serializedPriorValue =
valueSerde.innerSerde().serializer().serialize(changelogTopic, priorValue);
{code}
My apologies. I'll submit a quick PR to fix it. In the mean time, it's safe to
just return "null" from your serde.
> InMemoryTimeOrderedKeyValueBuffer propagates nulls when supress is configured
> --
>
> Key: KAFKA-8824
> URL: https://issues.apache.org/jira/browse/KAFKA-8824
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.3.0, 2.4.0
>Reporter: Ferran altimiras
>Priority: Major
> Fix For: 2.4.0, 2.3.1
>
> Attachments: Test.java
>
>
> Maybe this is not a bug, but it looks like something is wrong. This didn't
> happen in kafka streams 2.2.
>
> Applying an aggregate() with suppress on kafka 2.3 sends nulls into
> Serializer if delayed msgs are received.
> Not sure if some data is lost or not(yet). But IMHO getting a null to
> serialize the "accumulator" object is suspicious that something is wrong.
>
> Attached java code to demonstrate it.
> With kafka 2.3 -> LongSerde prints NULL, not in kafka 2.2
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (KAFKA-8824) InMemoryTimeOrderedKeyValueBuffer propagates nulls when supress is configured
[ https://issues.apache.org/jira/browse/KAFKA-8824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16912169#comment-16912169 ] Ferran altimiras commented on KAFKA-8824: - Hi [~guozhang], could be related to your last improvements? Regards > InMemoryTimeOrderedKeyValueBuffer propagates nulls when supress is configured > -- > > Key: KAFKA-8824 > URL: https://issues.apache.org/jira/browse/KAFKA-8824 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.3.0 >Reporter: Ferran altimiras >Priority: Major > Attachments: Test.java > > > Maybe this is not a bug, but it looks like something is wrong. This didn't > happen in kafka streams 2.2. > > Applying an aggregate() with suppress on kafka 2.3 sends nulls into > Serializer if delayed msgs are received. > Not sure if some data is lost or not(yet). But IMHO getting a null to > serialize the "accumulator" object is suspicious that something is wrong. > > Attached java code to demonstrate it. > With kafka 2.3 -> LongSerde prints NULL, not in kafka 2.2 > > > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
