[
https://issues.apache.org/jira/browse/FLINK-39940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-39940:
-----------------------------------
Release Note: On restore, DynamicKafkaSource now refreshes the cluster
properties from the KafkaMetadataService instead of using the checkpointed
ones, while keeping the checkpointed bootstrap.servers. A failing metadata
lookup during restore now fails the job immediately.
> DynamicKafkaSource should refresh restored cluster properties from
> KafkaMetadataService
> ---------------------------------------------------------------------------------------
>
> Key: FLINK-39940
> URL: https://issues.apache.org/jira/browse/FLINK-39940
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka
> Affects Versions: 2.2.1
> Reporter: Bowen Li
> Assignee: Bowen Li
> Priority: Major
> Labels: pull-request-available
> Fix For: kafka-5.1.0
>
>
> {{DynamicKafkaSource}} restores {{KafkaStream}} metadata from the
> checkpointed enumerator state when a job is resumed from a checkpoint or
> savepoint. Before this change, restored sub-enumerators were rebuilt from the
> checkpointed cluster properties without first refreshing them from
> {{{}KafkaMetadataService{}}}. That means non-bootstrap Kafka client
> properties such as authentication or security settings can be stale at
> restore time, and the job can fail before the next periodic metadata refresh
> has a chance to reconcile the latest metadata.
> The restore path should use the latest metadata-service properties
> immediately, while still preserving the checkpointed {{bootstrap.servers}}
> for restored clusters so the enumerator reconnects to the same restored
> cluster identity. Restore-time metadata lookup failures should also fail fast
> instead of being deferred until a later discovery cycle.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
