ramu11 opened a new pull request, #28730: URL: https://github.com/apache/camel/pull/28730
# Description This PR adds opt-in read-process-write exactly-once processing support to the Apache Camel Kafka component. When enabled, Kafka output records and the corresponding source consumer offsets are committed atomically in a single Kafka transaction. This prevents output records from becoming visible when processing fails before the source offsets are committed, and allows failed input records to be replayed safely. ### Implementation - Adds the `exactlyOnce` option to the Kafka component configuration and exposes it through generated component and endpoint DSLs. - Captures source consumer group metadata and offsets while polling, preserving partition offset information, including leader epochs. - Uses the transactional producer's `sendOffsetsToTransaction()` API to include source offsets in the output transaction. - Commits the transaction only after successful route processing; processing failures abort the transaction. - Supports batch processing by aggregating offsets per partition, including when the batch body is transformed. - Adds configuration validation to ensure that exactly-once mode is used with compatible consumer and producer settings. - Avoids independent consumer offset commits while exactly-once processing is active. - Handles consumer group rebalance and stale consumer generation scenarios. ### Configuration requirements and limitations For the Kafka source endpoint, configure: - `exactlyOnce=true` - `allowManualCommit=true` - `autoCommitEnable=false` - `breakOnFirstError=true` - An explicit `groupId` For the destination Kafka producer, configure `exactlyOnce=true` and enable transactions using `transacted=true` or a `transactionalId`. Consumers reading the transactional output should use `isolation.level=read_committed`. Kafka brokers version 2.5 or later are required. The current implementation supports one Kafka source and one exactly-once transactional producer per exchange/unit of work. Using an `offsetRepository` or a custom `KafkaManualCommitFactory` is not supported. Split/multicast child exchanges are not supported in exactly-once mode. ### Tests Adds unit and integration coverage for: - Exactly-once configuration validation. - Source offset and consumer group metadata handling. - Transaction commit and abort behavior. - End-to-end read-process-write exactly-once processing. - Batch processing across partitions. - Consumer rebalance and stale generation handling. Existing Kafka routes remain unchanged unless `exactlyOnce` is explicitly enabled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
