[
https://issues.apache.org/jira/browse/SPARK-19564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liwei Lin updated SPARK-19564:
------------------------------
Description:
In `KafkaOffsetReader`, when error occurs, we abort the existing consumer and
create a new consumer. In our current implementation, the first consumer and
the second consumer would be in the same group, which violates our intention of
the two consumers not being in the same group.
The cause is that, in our current implementation, the first consumer is created
before `groupId` and `nextId` are initialized in the constructor. Then even if
`groupId` and `nextId` are increased during the creation of that first
consumer, `groupId` and `nextId` would still be initialized to default values
in the constructor.
We should make sure that `groupId` and `nextId` are initialized before any
consumer is created.
was:
In KafkaOffsetReader, when error occurs, we abort the existing consumer and
create a new consumer. In our current implementation, the first consumer and
the second consumer would be in the same group, which violates our intention of
the two consumers not being in the same group.
The cause is that, in our current implementation, the first consumer is created
before `groupId` and `nextId` are initialized in the constructor. Then even if
`groupId` and `nextId` are increased during the creation of that first
consumer, `groupId` and `nextId` would still be initialized to default values
in the constructor.
We should make sure that `groupId` and `nextId` are initialized before the
creation of the any consumer.
> KafkaOffsetReader's consumers should not be in the same group
> -------------------------------------------------------------
>
> Key: SPARK-19564
> URL: https://issues.apache.org/jira/browse/SPARK-19564
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 2.1.1, 2.2.0
> Reporter: Liwei Lin
> Priority: Minor
>
> In `KafkaOffsetReader`, when error occurs, we abort the existing consumer and
> create a new consumer. In our current implementation, the first consumer and
> the second consumer would be in the same group, which violates our intention
> of the two consumers not being in the same group.
> The cause is that, in our current implementation, the first consumer is
> created before `groupId` and `nextId` are initialized in the constructor.
> Then even if `groupId` and `nextId` are increased during the creation of that
> first consumer, `groupId` and `nextId` would still be initialized to default
> values in the constructor.
> We should make sure that `groupId` and `nextId` are initialized before any
> consumer is created.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
