[
https://issues.apache.org/jira/browse/BEAM-10123?focusedWorklogId=492721&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-492721
]
ASF GitHub Bot logged work on BEAM-10123:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 29/Sep/20 23:23
Start Date: 29/Sep/20 23:23
Worklog Time Spent: 10m
Work Description: boyuanzz commented on pull request #12572:
URL: https://github.com/apache/beam/pull/12572#issuecomment-701043637
> Thanks, it looks fine in general for me. I left several questions, ptal.
>
> My main concern is the following:
>
> * Can we have a dataloss in case of failures during record processing
while an offset of this partition is already committed in parallel pipeline's
branch?
>
That's the usage of `Reshuffle`. When there is a failure in record
processing, the record will not be re-read from Kafka Read, instead it will be
re-read from `Reshuffle`.
> Also, the tests are very needed for this feature.
I'm thinking about having tests with mock Kafka. Do you have
suggestions/ideas around testing?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 492721)
Time Spent: 1h 50m (was: 1h 40m)
> Create CommitOffsetTransform to commit Kafka record offset
> ----------------------------------------------------------
>
> Key: BEAM-10123
> URL: https://issues.apache.org/jira/browse/BEAM-10123
> Project: Beam
> Issue Type: Sub-task
> Components: io-java-kafka
> Reporter: Boyuan Zhang
> Assignee: Boyuan Zhang
> Priority: P3
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
