[
https://issues.apache.org/jira/browse/CRUNCH-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13682585#comment-13682585
]
Josh Wills commented on CRUNCH-218:
-----------------------------------
Hey guys-- so I see two issues here; the first one is also related to something
[~jadler] emailed me about today; i.e., it's hard for the planner to know when
it should run a map-only job w/o some significant hints from the user. The
ideal solution to the use case Gabriel outlined above would be an MSCR fusion
that did a pass-through of the map-side output to an identity reducer, so that
we wrote out both outputs on the reduce side and only ran a single job.
Obviously, doing that isn't trivial, otherwise we would have done it already.
I think adding and checking for the _SUCCESS flag (and even checking the
timestamps on the inputs that feed into the checkpointing stuff, as Gabriel
suggested) is do-able within the context of this JIRA. I think the second issue
(the one that Gabriel outlines above) should go into a separate JIRA.
> Add new Target.WriteMode to skip the write and continue pipeline if an output
> target exists
> -------------------------------------------------------------------------------------------
>
> Key: CRUNCH-218
> URL: https://issues.apache.org/jira/browse/CRUNCH-218
> Project: Crunch
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.6.0
> Reporter: Dave Beech
> Assignee: Josh Wills
> Priority: Minor
> Attachments: CRUNCH-218b.patch, CRUNCH-218.patch
>
>
> Quite often I write pipelines which persist data to the filesystem midway
> through the process, and then carry on doing further work.
> If this intermediate data is already present, I think it would be good if I
> could set a write mode which skips over this first half of processing. This
> way I'd avoid running jobs unnecessarily and wasting cluster resources
> regenerating data I already have.
> Example:
> PCollection<B> inter =
> pipeline.read(source).parallelDo(something).parallelDo(somethingElse);
> inter.write(At.sequenceFile('output'), WriteMode.SKIP_IF_EXISTS);
> PCollection<C> final = inter.parallelDo(moreWork);
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
