[
https://issues.apache.org/jira/browse/KAFKA-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Bejeck updated KAFKA-4269:
-------------------------------
Description:
I discovered this issue while doing testing for for KAFKA-4114.
KAFKA-4131 fixed the issue of a _single_ KStream with a regex source on
partitioned topics across multiple consumers.
//KAFKA-4131 fixed this case assuming an "foo*" topics are partitioned
KStream kstream = builder.source(Pattern.compile("foo.*"));
KafkaStream stream = new KafkaStreams(builder, props);
stream.start();
This is a new issue where there are _multiple_
KStream instances (and one has a regex source) within a single KafkaStreams
object. When running the second or "following"
consumer there are NPE errors generated in the RecordQueue.addRawRecords method
when attempting to consume records.
For example:
KStream kstream = builder.source(Pattern.compile("foo.*"));
KStream kstream2 = builder.source(.....): //can be regex or named topic sources
KafkaStream stream = new KafkaStreams(builder, props);
stream.start();
By adding an additional KStream instance like above (whether Regex or Named
topic) causes a NPE when run as "follower"
>From my initial debugging I can see the TopicPartition assignments being set
>on the "follower" KafkaStreams instance, but need to track down why and where
>all assignments aren't being set.
was:
I discovered this issue while doing testing for for KAFKA-4114.
KAFKA-4131 fixed the issue of a _single_ KStream with a regex source on
partitioned topics across multiple consumers.
//KAFKA-4131 fixed this case assuming an "foo*" topics are partitioned
KStream kstream = builder.source(Pattern.compile("foo.*"));
KafkaStream stream = new KafkaStreams(builder, props);
stream.start();
This is a new issue where there are _multiple_
KStream instances (and one has a regex source) within a single KafkaStreams
object. When running the second or "following"
consumer there are NPE errors generated in the RecordQueue.addRawRecords method
when attempting to consume records.
For example:
KStream kstream = builder.source(Pattern.compile("foo.*"));
KStream kstream2 = builder.source(.....): //can be regex or named topic sources
KafkaStream stream = new KafkaStreams(builder, props);
stream.start();
By adding an additional KStream instance like above (whether Regex or Named
topic) causes a NPE when run as "follower"
>From my initial debugging I can see the TopicPartition assignments being set
>on the "follower" KafkaStreams instance, but need to track down
why and where all assignments aren't being set.
> Multiple KStream instances with at least one Regex source causes NPE when
> using multiple consumers
> --------------------------------------------------------------------------------------------------
>
> Key: KAFKA-4269
> URL: https://issues.apache.org/jira/browse/KAFKA-4269
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 0.10.1.0
> Reporter: Bill Bejeck
> Assignee: Bill Bejeck
>
> I discovered this issue while doing testing for for KAFKA-4114.
> KAFKA-4131 fixed the issue of a _single_ KStream with a regex source on
> partitioned topics across multiple consumers.
> //KAFKA-4131 fixed this case assuming an "foo*" topics are partitioned
> KStream kstream = builder.source(Pattern.compile("foo.*"));
> KafkaStream stream = new KafkaStreams(builder, props);
> stream.start();
> This is a new issue where there are _multiple_
> KStream instances (and one has a regex source) within a single KafkaStreams
> object. When running the second or "following"
> consumer there are NPE errors generated in the RecordQueue.addRawRecords
> method when attempting to consume records.
> For example:
> KStream kstream = builder.source(Pattern.compile("foo.*"));
> KStream kstream2 = builder.source(.....): //can be regex or named topic
> sources
> KafkaStream stream = new KafkaStreams(builder, props);
> stream.start();
> By adding an additional KStream instance like above (whether Regex or Named
> topic) causes a NPE when run as "follower"
> From my initial debugging I can see the TopicPartition assignments being set
> on the "follower" KafkaStreams instance, but need to track down why and where
> all assignments aren't being set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
