[
https://issues.apache.org/jira/browse/BEAM-3526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338206#comment-16338206
]
ASF GitHub Bot commented on BEAM-3526:
--------------------------------------
rangadi opened a new pull request #4481: [BEAM-3526] KakfaIO support for
finalizeCheckpoint()
URL: https://github.com/apache/beam/pull/4481
Adds an option in KafkaIO source to commit offsets to Kafka when
`CheckpointMark#finalizecheckPoint()` is invoked.
Kafka consumer is single threaded and as a result actuall commit to Kafka
might take upto 1 second (KAFKA_POLL_TIMEOUT). This is not necessarily bad
since it could avoid excessive commits when multiple checkpoint marks are
finalized at once by the runner.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support for checkpointMark finalize in KafkaIO
> ----------------------------------------------
>
> Key: BEAM-3526
> URL: https://issues.apache.org/jira/browse/BEAM-3526
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
> Affects Versions: 2.2.0
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Priority: Major
>
> Users have been asking for option to commit offsets back to Kafka inside
> `CheckpointMark.finalizeCheckpoint()'. This option is some what better than
> setting 'AUTO_COMMIT' in Kafka consumer.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
