[jira] [Commented] (KAFKA-8671) NullPointerException occurs if topic associated with GlobalKTable changes
[
https://issues.apache.org/jira/browse/KAFKA-8671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951598#comment-16951598
]
ASF GitHub Bot commented on KAFKA-8671:
---
mjsax commented on pull request #7437: KAFKA-8671: NullPointerException occurs
if topic associated with GlobalKTable changes
URL: https://github.com/apache/kafka/pull/7437
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> NullPointerException occurs if topic associated with GlobalKTable changes
> -
>
> Key: KAFKA-8671
> URL: https://issues.apache.org/jira/browse/KAFKA-8671
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
>Reporter: Alex Leung
>Assignee: Alex Leung
>Priority: Critical
>
> The following NullPointerException occurs when the global/.checkpoint file
> contains a line with a topic previously associated with (but no longer
> configured for) a GlobalKTable:
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.update(GlobalStateUpdateTask.java:85)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.pollAndUpdate(GlobalStreamThread.java:241)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:290){code}
>
> After line 84
> ([https://github.com/apache/kafka/blob/2.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStateUpdateTask.java#L84)]
> `sourceNodeAndDeserializer` is null for the old, but still valid, topic.
> This can be reproduced with the following sequence:
> # create a GlobalKTable associated with topic, 'global-topic1'
> # change the topic associated with the GlobalKTable to 'global-topic2'
> ## at this point, the global/.checkpoint file will contain lines for both
> topics
> # produce messages to previous topic ('global-topic1')
> # the consumer will attempt to consume from global-topic1, but no
> deserializer associated with global-topic1 will be found and the NPE will
> occur
> It looks like the following recent commit has included checkpoint validations
> that may prevent this issue:
> https://github.com/apache/kafka/commit/53b4ce5c00d61be87962f603682873665155cec4#diff-cc98a6c20f2a8483e1849aea6921c34dR425
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-8671) NullPointerException occurs if topic associated with GlobalKTable changes
[
https://issues.apache.org/jira/browse/KAFKA-8671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943181#comment-16943181
]
ASF GitHub Bot commented on KAFKA-8671:
---
amleung21 commented on pull request #7437: KAFKA-8671: NullPointerException
occurs if topic associated with GlobalKTable changes
URL: https://github.com/apache/kafka/pull/7437
A NullPointerException occurs when the global/.checkpoint file contains a
line with an obsolete (but valid) topic. Log an error and throw exception when
non-relevant topic-partitions from checkpoint file are encountered.
Added a unit test to verify that non-relevant topics are detected and an
exception is thrown. Also, manually ran a streams application with a modified
global/.checkpoint file containing an obsolete topic partition and verified
that the error is logged and initialization fails.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> NullPointerException occurs if topic associated with GlobalKTable changes
> -
>
> Key: KAFKA-8671
> URL: https://issues.apache.org/jira/browse/KAFKA-8671
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
>Reporter: Alex Leung
>Assignee: Alex Leung
>Priority: Critical
>
> The following NullPointerException occurs when the global/.checkpoint file
> contains a line with a topic previously associated with (but no longer
> configured for) a GlobalKTable:
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.update(GlobalStateUpdateTask.java:85)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.pollAndUpdate(GlobalStreamThread.java:241)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:290){code}
>
> After line 84
> ([https://github.com/apache/kafka/blob/2.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStateUpdateTask.java#L84)]
> `sourceNodeAndDeserializer` is null for the old, but still valid, topic.
> This can be reproduced with the following sequence:
> # create a GlobalKTable associated with topic, 'global-topic1'
> # change the topic associated with the GlobalKTable to 'global-topic2'
> ## at this point, the global/.checkpoint file will contain lines for both
> topics
> # produce messages to previous topic ('global-topic1')
> # the consumer will attempt to consume from global-topic1, but no
> deserializer associated with global-topic1 will be found and the NPE will
> occur
> It looks like the following recent commit has included checkpoint validations
> that may prevent this issue:
> https://github.com/apache/kafka/commit/53b4ce5c00d61be87962f603682873665155cec4#diff-cc98a6c20f2a8483e1849aea6921c34dR425
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-8671) NullPointerException occurs if topic associated with GlobalKTable changes
[
https://issues.apache.org/jira/browse/KAFKA-8671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943007#comment-16943007
]
ASF GitHub Bot commented on KAFKA-8671:
---
amleung21 commented on pull request #7188: KAFKA-8671: NullPointerException
occurs if topic associated with GlobalKTable changes
URL: https://github.com/apache/kafka/pull/7188
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> NullPointerException occurs if topic associated with GlobalKTable changes
> -
>
> Key: KAFKA-8671
> URL: https://issues.apache.org/jira/browse/KAFKA-8671
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
>Reporter: Alex Leung
>Assignee: Alex Leung
>Priority: Critical
>
> The following NullPointerException occurs when the global/.checkpoint file
> contains a line with a topic previously associated with (but no longer
> configured for) a GlobalKTable:
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.update(GlobalStateUpdateTask.java:85)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.pollAndUpdate(GlobalStreamThread.java:241)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:290){code}
>
> After line 84
> ([https://github.com/apache/kafka/blob/2.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStateUpdateTask.java#L84)]
> `sourceNodeAndDeserializer` is null for the old, but still valid, topic.
> This can be reproduced with the following sequence:
> # create a GlobalKTable associated with topic, 'global-topic1'
> # change the topic associated with the GlobalKTable to 'global-topic2'
> ## at this point, the global/.checkpoint file will contain lines for both
> topics
> # produce messages to previous topic ('global-topic1')
> # the consumer will attempt to consume from global-topic1, but no
> deserializer associated with global-topic1 will be found and the NPE will
> occur
> It looks like the following recent commit has included checkpoint validations
> that may prevent this issue:
> https://github.com/apache/kafka/commit/53b4ce5c00d61be87962f603682873665155cec4#diff-cc98a6c20f2a8483e1849aea6921c34dR425
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-8671) NullPointerException occurs if topic associated with GlobalKTable changes
[
https://issues.apache.org/jira/browse/KAFKA-8671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904152#comment-16904152
]
ASF GitHub Bot commented on KAFKA-8671:
---
amleung21 commented on pull request #7188: KAFKA-8671: NullPointerException
occurs if topic associated with GlobalKTable changes
URL: https://github.com/apache/kafka/pull/7188
Fix NullPointerException that occurs when the global/.checkpoint file
contains a line with an obsolete (but valid) topic by pruning non-relevant
topics from the checkpoint cache.
Added a unit test to verify that non-relevant topics are pruned from the
checkpoint cache by GlobalStateManagerImpl. Also, manually ran a streams
application with a modified global/.checkpoint file containing an obsolete
topic partition and verified that 1) the NullPointerException no longer occurs
and 2) the obsolete topic is removed from the global/.checkpoint file.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> NullPointerException occurs if topic associated with GlobalKTable changes
> -
>
> Key: KAFKA-8671
> URL: https://issues.apache.org/jira/browse/KAFKA-8671
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
>Reporter: Alex Leung
>Assignee: Alex Leung
>Priority: Critical
>
> The following NullPointerException occurs when the global/.checkpoint file
> contains a line with a topic previously associated with (but no longer
> configured for) a GlobalKTable:
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.update(GlobalStateUpdateTask.java:85)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.pollAndUpdate(GlobalStreamThread.java:241)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:290){code}
>
> After line 84
> ([https://github.com/apache/kafka/blob/2.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStateUpdateTask.java#L84)]
> `sourceNodeAndDeserializer` is null for the old, but still valid, topic.
> This can be reproduced with the following sequence:
> # create a GlobalKTable associated with topic, 'global-topic1'
> # change the topic associated with the GlobalKTable to 'global-topic2'
> ## at this point, the global/.checkpoint file will contain lines for both
> topics
> # produce messages to previous topic ('global-topic1')
> # the consumer will attempt to consume from global-topic1, but no
> deserializer associated with global-topic1 will be found and the NPE will
> occur
> It looks like the following recent commit has included checkpoint validations
> that may prevent this issue:
> https://github.com/apache/kafka/commit/53b4ce5c00d61be87962f603682873665155cec4#diff-cc98a6c20f2a8483e1849aea6921c34dR425
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (KAFKA-8671) NullPointerException occurs if topic associated with GlobalKTable changes
[
https://issues.apache.org/jira/browse/KAFKA-8671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891984#comment-16891984
]
Alex Leung commented on KAFKA-8671:
---
Thanks for taking a look, [~guozhang]. I also wanted to add a unit test for
this but haven't had time yet. Will add that and submit a PR.
> NullPointerException occurs if topic associated with GlobalKTable changes
> -
>
> Key: KAFKA-8671
> URL: https://issues.apache.org/jira/browse/KAFKA-8671
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
>Reporter: Alex Leung
>Assignee: Alex Leung
>Priority: Critical
>
> The following NullPointerException occurs when the global/.checkpoint file
> contains a line with a topic previously associated with (but no longer
> configured for) a GlobalKTable:
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.update(GlobalStateUpdateTask.java:85)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.pollAndUpdate(GlobalStreamThread.java:241)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:290){code}
>
> After line 84
> ([https://github.com/apache/kafka/blob/2.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStateUpdateTask.java#L84)]
> `sourceNodeAndDeserializer` is null for the old, but still valid, topic.
> This can be reproduced with the following sequence:
> # create a GlobalKTable associated with topic, 'global-topic1'
> # change the topic associated with the GlobalKTable to 'global-topic2'
> ## at this point, the global/.checkpoint file will contain lines for both
> topics
> # produce messages to previous topic ('global-topic1')
> # the consumer will attempt to consume from global-topic1, but no
> deserializer associated with global-topic1 will be found and the NPE will
> occur
> It looks like the following recent commit has included checkpoint validations
> that may prevent this issue:
> https://github.com/apache/kafka/commit/53b4ce5c00d61be87962f603682873665155cec4#diff-cc98a6c20f2a8483e1849aea6921c34dR425
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (KAFKA-8671) NullPointerException occurs if topic associated with GlobalKTable changes
[
https://issues.apache.org/jira/browse/KAFKA-8671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891971#comment-16891971
]
Guozhang Wang commented on KAFKA-8671:
--
I looked at the diff and it looks promising to me.
Could you file a PR following the contribution guidelines as well?
https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Code+Changes#ContributingCodeChanges-PullRequest
Also cc [~vvcephei] for reviewing this change.
> NullPointerException occurs if topic associated with GlobalKTable changes
> -
>
> Key: KAFKA-8671
> URL: https://issues.apache.org/jira/browse/KAFKA-8671
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
>Reporter: Alex Leung
>Assignee: Alex Leung
>Priority: Critical
>
> The following NullPointerException occurs when the global/.checkpoint file
> contains a line with a topic previously associated with (but no longer
> configured for) a GlobalKTable:
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.update(GlobalStateUpdateTask.java:85)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.pollAndUpdate(GlobalStreamThread.java:241)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:290){code}
>
> After line 84
> ([https://github.com/apache/kafka/blob/2.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStateUpdateTask.java#L84)]
> `sourceNodeAndDeserializer` is null for the old, but still valid, topic.
> This can be reproduced with the following sequence:
> # create a GlobalKTable associated with topic, 'global-topic1'
> # change the topic associated with the GlobalKTable to 'global-topic2'
> ## at this point, the global/.checkpoint file will contain lines for both
> topics
> # produce messages to previous topic ('global-topic1')
> # the consumer will attempt to consume from global-topic1, but no
> deserializer associated with global-topic1 will be found and the NPE will
> occur
> It looks like the following recent commit has included checkpoint validations
> that may prevent this issue:
> https://github.com/apache/kafka/commit/53b4ce5c00d61be87962f603682873665155cec4#diff-cc98a6c20f2a8483e1849aea6921c34dR425
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (KAFKA-8671) NullPointerException occurs if topic associated with GlobalKTable changes
[
https://issues.apache.org/jira/browse/KAFKA-8671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891483#comment-16891483
]
Alex Leung commented on KAFKA-8671:
---
What I did was prune the checkpointFileCache of all entries that are not
associated with a global state store at the time of initializing it. See
[https://github.com/apache/kafka/compare/trunk...amleung21:global_checkpoint_npe_fix_2]
for the potential fix.
> NullPointerException occurs if topic associated with GlobalKTable changes
> -
>
> Key: KAFKA-8671
> URL: https://issues.apache.org/jira/browse/KAFKA-8671
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
>Reporter: Alex Leung
>Assignee: Alex Leung
>Priority: Critical
>
> The following NullPointerException occurs when the global/.checkpoint file
> contains a line with a topic previously associated with (but no longer
> configured for) a GlobalKTable:
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.update(GlobalStateUpdateTask.java:85)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.pollAndUpdate(GlobalStreamThread.java:241)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:290){code}
>
> After line 84
> ([https://github.com/apache/kafka/blob/2.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStateUpdateTask.java#L84)]
> `sourceNodeAndDeserializer` is null for the old, but still valid, topic.
> This can be reproduced with the following sequence:
> # create a GlobalKTable associated with topic, 'global-topic1'
> # change the topic associated with the GlobalKTable to 'global-topic2'
> ## at this point, the global/.checkpoint file will contain lines for both
> topics
> # produce messages to previous topic ('global-topic1')
> # the consumer will attempt to consume from global-topic1, but no
> deserializer associated with global-topic1 will be found and the NPE will
> occur
> It looks like the following recent commit has included checkpoint validations
> that may prevent this issue:
> https://github.com/apache/kafka/commit/53b4ce5c00d61be87962f603682873665155cec4#diff-cc98a6c20f2a8483e1849aea6921c34dR425
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (KAFKA-8671) NullPointerException occurs if topic associated with GlobalKTable changes
[
https://issues.apache.org/jira/browse/KAFKA-8671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891467#comment-16891467
]
Guozhang Wang commented on KAFKA-8671:
--
*(typing too fast)* As for the issue itself, I think the idea of KAFKA-5998
could still be applied: when reading from the checkpoint file, if the
corresponding topic is no longer in our interested changelog topic list, we
should filter it out in our assigned partitions of the restore.consumer.
> NullPointerException occurs if topic associated with GlobalKTable changes
> -
>
> Key: KAFKA-8671
> URL: https://issues.apache.org/jira/browse/KAFKA-8671
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
>Reporter: Alex Leung
>Assignee: Alex Leung
>Priority: Critical
>
> The following NullPointerException occurs when the global/.checkpoint file
> contains a line with a topic previously associated with (but no longer
> configured for) a GlobalKTable:
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.update(GlobalStateUpdateTask.java:85)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.pollAndUpdate(GlobalStreamThread.java:241)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:290){code}
>
> After line 84
> ([https://github.com/apache/kafka/blob/2.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStateUpdateTask.java#L84)]
> `sourceNodeAndDeserializer` is null for the old, but still valid, topic.
> This can be reproduced with the following sequence:
> # create a GlobalKTable associated with topic, 'global-topic1'
> # change the topic associated with the GlobalKTable to 'global-topic2'
> ## at this point, the global/.checkpoint file will contain lines for both
> topics
> # produce messages to previous topic ('global-topic1')
> # the consumer will attempt to consume from global-topic1, but no
> deserializer associated with global-topic1 will be found and the NPE will
> occur
> It looks like the following recent commit has included checkpoint validations
> that may prevent this issue:
> https://github.com/apache/kafka/commit/53b4ce5c00d61be87962f603682873665155cec4#diff-cc98a6c20f2a8483e1849aea6921c34dR425
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (KAFKA-8671) NullPointerException occurs if topic associated with GlobalKTable changes
[
https://issues.apache.org/jira/browse/KAFKA-8671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891465#comment-16891465
]
Guozhang Wang commented on KAFKA-8671:
--
[~aleung181] I've assigned the ticket to you, thanks for contributing!
As for the issue itself.
> NullPointerException occurs if topic associated with GlobalKTable changes
> -
>
> Key: KAFKA-8671
> URL: https://issues.apache.org/jira/browse/KAFKA-8671
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
>Reporter: Alex Leung
>Assignee: Alex Leung
>Priority: Critical
>
> The following NullPointerException occurs when the global/.checkpoint file
> contains a line with a topic previously associated with (but no longer
> configured for) a GlobalKTable:
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.update(GlobalStateUpdateTask.java:85)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.pollAndUpdate(GlobalStreamThread.java:241)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:290){code}
>
> After line 84
> ([https://github.com/apache/kafka/blob/2.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStateUpdateTask.java#L84)]
> `sourceNodeAndDeserializer` is null for the old, but still valid, topic.
> This can be reproduced with the following sequence:
> # create a GlobalKTable associated with topic, 'global-topic1'
> # change the topic associated with the GlobalKTable to 'global-topic2'
> ## at this point, the global/.checkpoint file will contain lines for both
> topics
> # produce messages to previous topic ('global-topic1')
> # the consumer will attempt to consume from global-topic1, but no
> deserializer associated with global-topic1 will be found and the NPE will
> occur
> It looks like the following recent commit has included checkpoint validations
> that may prevent this issue:
> https://github.com/apache/kafka/commit/53b4ce5c00d61be87962f603682873665155cec4#diff-cc98a6c20f2a8483e1849aea6921c34dR425
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (KAFKA-8671) NullPointerException occurs if topic associated with GlobalKTable changes
[
https://issues.apache.org/jira/browse/KAFKA-8671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891431#comment-16891431
]
Alex Leung commented on KAFKA-8671:
---
I cannot assign this Jira to myself. Do you have the power to add me to the
contributor list? :)
> NullPointerException occurs if topic associated with GlobalKTable changes
> -
>
> Key: KAFKA-8671
> URL: https://issues.apache.org/jira/browse/KAFKA-8671
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
>Reporter: Alex Leung
>Priority: Critical
>
> The following NullPointerException occurs when the global/.checkpoint file
> contains a line with a topic previously associated with (but no longer
> configured for) a GlobalKTable:
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.update(GlobalStateUpdateTask.java:85)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.pollAndUpdate(GlobalStreamThread.java:241)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:290){code}
>
> After line 84
> ([https://github.com/apache/kafka/blob/2.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStateUpdateTask.java#L84)]
> `sourceNodeAndDeserializer` is null for the old, but still valid, topic.
> This can be reproduced with the following sequence:
> # create a GlobalKTable associated with topic, 'global-topic1'
> # change the topic associated with the GlobalKTable to 'global-topic2'
> ## at this point, the global/.checkpoint file will contain lines for both
> topics
> # produce messages to previous topic ('global-topic1')
> # the consumer will attempt to consume from global-topic1, but no
> deserializer associated with global-topic1 will be found and the NPE will
> occur
> It looks like the following recent commit has included checkpoint validations
> that may prevent this issue:
> https://github.com/apache/kafka/commit/53b4ce5c00d61be87962f603682873665155cec4#diff-cc98a6c20f2a8483e1849aea6921c34dR425
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (KAFKA-8671) NullPointerException occurs if topic associated with GlobalKTable changes
[
https://issues.apache.org/jira/browse/KAFKA-8671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891216#comment-16891216
]
Matthias J. Sax commented on KAFKA-8671:
No reason. Go ahead :)
> NullPointerException occurs if topic associated with GlobalKTable changes
> -
>
> Key: KAFKA-8671
> URL: https://issues.apache.org/jira/browse/KAFKA-8671
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.0.0
>Reporter: Alex Leung
>Priority: Critical
>
> The following NullPointerException occurs when the global/.checkpoint file
> contains a line with a topic previously associated with (but no longer
> configured for) a GlobalKTable:
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.update(GlobalStateUpdateTask.java:85)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.pollAndUpdate(GlobalStreamThread.java:241)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:290){code}
>
> After line 84
> ([https://github.com/apache/kafka/blob/2.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStateUpdateTask.java#L84)]
> `sourceNodeAndDeserializer` is null for the old, but still valid, topic.
> This can be reproduced with the following sequence:
> # create a GlobalKTable associated with topic, 'global-topic1'
> # change the topic associated with the GlobalKTable to 'global-topic2'
> ## at this point, the global/.checkpoint file will contain lines for both
> topics
> # produce messages to previous topic ('global-topic1')
> # the consumer will attempt to consume from global-topic1, but no
> deserializer associated with global-topic1 will be found and the NPE will
> occur
> It looks like the following recent commit has included checkpoint validations
> that may prevent this issue:
> https://github.com/apache/kafka/commit/53b4ce5c00d61be87962f603682873665155cec4#diff-cc98a6c20f2a8483e1849aea6921c34dR425
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (KAFKA-8671) NullPointerException occurs if topic associated with GlobalKTable changes
[
https://issues.apache.org/jira/browse/KAFKA-8671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16889904#comment-16889904
]
Alex Leung commented on KAFKA-8671:
---
[~mjsax] any reason not to assign this jira to myself and submit a PR?
> NullPointerException occurs if topic associated with GlobalKTable changes
> -
>
> Key: KAFKA-8671
> URL: https://issues.apache.org/jira/browse/KAFKA-8671
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.0.0
>Reporter: Alex Leung
>Priority: Critical
>
> The following NullPointerException occurs when the global/.checkpoint file
> contains a line with a topic previously associated with (but no longer
> configured for) a GlobalKTable:
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.update(GlobalStateUpdateTask.java:85)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.pollAndUpdate(GlobalStreamThread.java:241)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:290){code}
>
> After line 84
> ([https://github.com/apache/kafka/blob/2.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStateUpdateTask.java#L84)]
> `sourceNodeAndDeserializer` is null for the old, but still valid, topic.
> This can be reproduced with the following sequence:
> # create a GlobalKTable associated with topic, 'global-topic1'
> # change the topic associated with the GlobalKTable to 'global-topic2'
> ## at this point, the global/.checkpoint file will contain lines for both
> topics
> # produce messages to previous topic ('global-topic1')
> # the consumer will attempt to consume from global-topic1, but no
> deserializer associated with global-topic1 will be found and the NPE will
> occur
> It looks like the following recent commit has included checkpoint validations
> that may prevent this issue:
> https://github.com/apache/kafka/commit/53b4ce5c00d61be87962f603682873665155cec4#diff-cc98a6c20f2a8483e1849aea6921c34dR425
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (KAFKA-8671) NullPointerException occurs if topic associated with GlobalKTable changes
[
https://issues.apache.org/jira/browse/KAFKA-8671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886676#comment-16886676
]
Alex Leung commented on KAFKA-8671:
---
I was able to reproduce this on trunk.
`checkpointFileCache` populated with everything from .checkpoint
[here|https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStateManagerImpl.java#L123].
Then `globalConsumer` assigned
[here|https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStreamThread.java#L230],
and `globalConsumer.subscriptions.subscription` includes the "old" topic.
When we get to `pollAndUpdate`, leads to the same NPE:
{code:java}
org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.update(GlobalStateUpdateTask.java:86)
org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.pollAndUpdate(GlobalStreamThread.java:241)
org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:290
apache/kafka/streams/processor/internals/GlobalStreamThread.java#L241{code}
> NullPointerException occurs if topic associated with GlobalKTable changes
> -
>
> Key: KAFKA-8671
> URL: https://issues.apache.org/jira/browse/KAFKA-8671
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.0.0
>Reporter: Alex Leung
>Priority: Critical
>
> The following NullPointerException occurs when the global/.checkpoint file
> contains a line with a topic previously associated with (but no longer
> configured for) a GlobalKTable:
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.update(GlobalStateUpdateTask.java:85)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.pollAndUpdate(GlobalStreamThread.java:241)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:290){code}
>
> After line 84
> ([https://github.com/apache/kafka/blob/2.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStateUpdateTask.java#L84)]
> `sourceNodeAndDeserializer` is null for the old, but still valid, topic.
> This can be reproduced with the following sequence:
> # create a GlobalKTable associated with topic, 'global-topic1'
> # change the topic associated with the GlobalKTable to 'global-topic2'
> ## at this point, the global/.checkpoint file will contain lines for both
> topics
> # produce messages to previous topic ('global-topic1')
> # the consumer will attempt to consume from global-topic1, but no
> deserializer associated with global-topic1 will be found and the NPE will
> occur
> It looks like the following recent commit has included checkpoint validations
> that may prevent this issue:
> https://github.com/apache/kafka/commit/53b4ce5c00d61be87962f603682873665155cec4#diff-cc98a6c20f2a8483e1849aea6921c34dR425
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (KAFKA-8671) NullPointerException occurs if topic associated with GlobalKTable changes
[
https://issues.apache.org/jira/browse/KAFKA-8671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885709#comment-16885709
]
Matthias J. Sax commented on KAFKA-8671:
This issue might be related to KAFKA-5998 – if we cannot reproduce on `trunk`
any longer, the fix of KAFKA-5998 might fix this one, too.
> NullPointerException occurs if topic associated with GlobalKTable changes
> -
>
> Key: KAFKA-8671
> URL: https://issues.apache.org/jira/browse/KAFKA-8671
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.0.0
>Reporter: Alex Leung
>Priority: Critical
>
> The following NullPointerException occurs when the global/.checkpoint file
> contains a line with a topic previously associated with (but no longer
> configured for) a GlobalKTable:
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.update(GlobalStateUpdateTask.java:85)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.pollAndUpdate(GlobalStreamThread.java:241)
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:290){code}
>
> After line 84
> ([https://github.com/apache/kafka/blob/2.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStateUpdateTask.java#L84)]
> `sourceNodeAndDeserializer` is null for the old, but still valid, topic.
> This can be reproduced with the following sequence:
> # create a GlobalKTable associated with topic, 'global-topic1'
> # change the topic associated with the GlobalKTable to 'global-topic2'
> ## at this point, the global/.checkpoint file will contain lines for both
> topics
> # produce messages to previous topic ('global-topic1')
> # the consumer will attempt to consume from global-topic1, but no
> deserializer associated with global-topic1 will be found and the NPE will
> occur
> It looks like the following recent commit has included checkpoint validations
> that may prevent this issue:
> https://github.com/apache/kafka/commit/53b4ce5c00d61be87962f603682873665155cec4#diff-cc98a6c20f2a8483e1849aea6921c34dR425
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
