[
https://issues.apache.org/jira/browse/FLINK-19039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-19039:
-----------------------------------
Labels: auto-deprioritized-major stale-minor (was:
auto-deprioritized-major)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Parallel Flink Kafka Consumers compete with each other
> ------------------------------------------------------
>
> Key: FLINK-19039
> URL: https://issues.apache.org/jira/browse/FLINK-19039
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka
> Affects Versions: 1.11.1
> Reporter: Ayrat Hudaygulov
> Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
>
> If I'll run multiple Flink instances with same consumer group id they will
> not re-balance partitions with each other, but rather each instance take all
> partitions, effectively not working in parallel at all, and multiplying
> amount of messages processed.
>
> This is because FlinkKafkaConsumer has its own re-balancing mechanism for
> current parallelism level and then just calls:
> `consumerTmp.assign(newPartitionAssignments){color:#cc7832};{color}`
>
> [https://github.com/apache/flink/blob/59714b9d6addb1dbf2171cab937a0e3fec52f2b1/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.java#L422]
>
> I suppose there has to be a way to fallback to default kafka mechanism of
> re-balancing to respect consumer group id, but it's not presented in Flink at
> all.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
