[ https://issues.apache.org/jira/browse/FLINK-10205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591489#comment-16591489 ]
JIN SUN commented on FLINK-10205: --------------------------------- In my opinion we need deterministic, especially when a task rerun, the output should as same as the previous version. Today's logic might skip the split that previous task processed and assign a different split, this will lead incorrect result. Fabian, we didn't restrict the InputSplit assignment, instead, by add a small piece of code in Execution.java and ExecutionVertex.java, we can make it deterministic. i'm preparing the code, we can have a further discussion when it ready. > Batch Job: InputSplit Fault tolerant for DataSourceTask > ------------------------------------------------------- > > Key: FLINK-10205 > URL: https://issues.apache.org/jira/browse/FLINK-10205 > Project: Flink > Issue Type: Sub-task > Components: JobManager > Reporter: JIN SUN > Priority: Major > Original Estimate: 168h > Remaining Estimate: 168h > > Today DataSource Task pull InputSplits from JobManager to achieve better > performance, however, when a DataSourceTask failed and rerun, it will not get > the same splits as its previous version. this will introduce inconsistent > result or even data corruption. > Furthermore, if there are two executions run at the same time (in batch > scenario), this two executions should process same splits. > we need to fix the issue to make the inputs of a DataSourceTask > deterministic. The propose is save all splits into ExecutionVertex and > DataSourceTask will pull split from there. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)