[
https://issues.apache.org/jira/browse/FLINK-10205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16639654#comment-16639654
]
ASF GitHub Bot commented on FLINK-10205:
----------------------------------------
StefanRRichter commented on issue #6684: [FLINK-10205] Batch Job:
InputSplit Fault tolerant for DataSource…
URL: https://github.com/apache/flink/pull/6684#issuecomment-427323755
@isunjin Ok, I think I was just connecting a different type of problem with
the word "corruption", but the case makes sense because the assumption so far
was probably that there are only global failover and that the
`InputSplitAssigner` is reset. I think in this context the implementation is
ok, I still not the biggest fan of pulling the `InputSplit` concept into
`Execution` but at the same time I see that currently all alternatives that
came to my mind have their own set of problems. Guess it would be ok for me to
merge the PR, but I would like to ask @tillrohrmann for a second opinion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Batch Job: InputSplit Fault tolerant for DataSourceTask
> -------------------------------------------------------
>
> Key: FLINK-10205
> URL: https://issues.apache.org/jira/browse/FLINK-10205
> Project: Flink
> Issue Type: Sub-task
> Components: JobManager
> Affects Versions: 1.6.1, 1.7.0, 1.6.2
> Reporter: JIN SUN
> Assignee: JIN SUN
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Today DataSource Task pull InputSplits from JobManager to achieve better
> performance, however, when a DataSourceTask failed and rerun, it will not get
> the same splits as its previous version. this will introduce inconsistent
> result or even data corruption.
> Furthermore, if there are two executions run at the same time (in batch
> scenario), this two executions should process same splits.
> we need to fix the issue to make the inputs of a DataSourceTask
> deterministic. The propose is save all splits into ExecutionVertex and
> DataSourceTask will pull split from there.
> document:
> [https://docs.google.com/document/d/1FdZdcA63tPUEewcCimTFy9Iz2jlVlMRANZkO4RngIuk/edit?usp=sharing]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
