Purushottam Sinha created FLINK-41116:
-----------------------------------------
Summary: Connectors/Kafka: Upgrade kafka-clients to 4.4.0
Key: FLINK-41116
URL: https://issues.apache.org/jira/browse/FLINK-41116
Project: Flink
Issue Type: Improvement
Components: Connectors / Kafka
Reporter: Purushottam Sinha
Bump {{kafka.version}} from 4.2.0 to 4.4.0 (latest release; 4.3.x is also
affected by the API change below).
Changes needed:
* {{FlinkKafkaInternalProducer#flushNewPartitions}} calls
{{TransactionalRequestResult#await()}}. Since 4.3.0 the only variant is
{{await(long, TimeUnit, String)}}. Use {{await(Long.MAX_VALUE,
TimeUnit.MILLISECONDS, ...)}} to keep the previous wait-forever behavior.
* {{org.jetbrains:annotations}} no longer converges in the e2e test module
({{kafka-clients}} -> {{zstd-jni}} wants 24.1.0, {{testcontainers}} ->
{{duct-tape}} wants 17.0.0). Add a {{dependencyManagement}} pin (24.1.0).
The other internals the connector reaches into by reflection
({{transactionalId}}, {{currentState}}, {{newPartitionsInTransaction}},
{{txnPartitionMap}}, {{transactionStarted}}, {{producerIdAndEpoch}},
{{transitionTo}}, {{addPartitionsToTransactionHandler}}) still exist in 4.4.0.
{{ProducerIdAndEpoch}} moved to {{org.apache.kafka.common.utils.internals}},
which is handled because the connector resolves it from the field type.
Compatibility note: {{await(long, TimeUnit, String)}} does not exist in 4.2.x,
so the connector can no longer be used with {{kafka-clients}} older than 4.3.0.
Verified on 4.4.0: all {{flink-connector-kafka}} unit tests and ITCases pass
(sink 147, rest 398), and {{-Pcheck-convergence}} passes.
{{KafkaWriterFaultToleranceITCase#testFlushExceptionWhenKafkaUnavailable}} is
flaky on both 4.2.0 and 4.4.0 (1 of 3 runs each). The e2e tests were not run.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
