> On Oct. 21, 2015, 11:19 a.m., Rohini Palaniswamy wrote: > > src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java, > > line 507 > > <https://reviews.apache.org/r/39226/diff/1/?file=1095351#file1095351line507> > > > > Just storing the current plan? how about what part of it has succeeded? > > Abhishek Agarwal wrote: > Success or failure is being inferred through the commit file in the > output path of an intermediate job. If the output path SUCCESS file is > present, then it is assumed that job has succeeded.
Successful operators are being stored as well now, in the state. - Abhishek ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/39226/#review103384 ----------------------------------------------------------- On Nov. 27, 2015, 3:40 p.m., Abhishek Agarwal wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/39226/ > ----------------------------------------------------------- > > (Updated Nov. 27, 2015, 3:40 p.m.) > > > Review request for pig and Rohini Palaniswamy. > > > Repository: pig-git > > > Description > ------- > > Pig scripts can have multiple ETL jobs in the DAG which may take hours to > finish. In case of transient errors, the job fails. When the job is rerun, > all the nodes in Job graph will rerun. Some of these nodes may have already > run successfully. Redundant runs lead to wastage of cluster capacity and > pipeline delays. > > In case of failure, we can persist the graph state. In next run, only the > failed nodes and their successors will rerun. This is of course subject to > preconditions such as > > Pig script has not changed > > Input locations have not changed > > Output data from previous run is intact > > Configuration has not changed > > > Diffs > ----- > > src/org/apache/pig/PigConfiguration.java 54959fe > src/org/apache/pig/PigServer.java ee52472 > > src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java > 595e68c > > src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/plans/MRIntermediateDataVisitor.java > 4b62112 > > src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/plans/MRJobRecovery.java > PRE-CREATION > > src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/plans/MRJobState.java > PRE-CREATION > src/org/apache/pig/impl/PigImplConstants.java 050a243 > src/org/apache/pig/impl/io/FileLocalizer.java f0f9b43 > src/org/apache/pig/tools/grunt/GruntParser.java 439d087 > > Diff: https://reviews.apache.org/r/39226/diff/ > > > Testing > ------- > > > Thanks, > > Abhishek Agarwal > >
