Micah Whitacre created CRUNCH-405:
-------------------------------------
Summary: Explore adding support for idempotent MRPipeline.plan()
Key: CRUNCH-405
URL: https://issues.apache.org/jira/browse/CRUNCH-405
Project: Crunch
Issue Type: Improvement
Components: Core
Reporter: Micah Whitacre
Assignee: Josh Wills
Talking through a use case with a consumer, they were interested in having the
ability to run the MRPipeline.plan() method one to many times prior to ever
calling the Pipeline.run/done methods. The reason for this was they were
looking at pulling information off the MRExecutor to tweak settings inside of
their DoFns.
Currently the MRPipeline implementation however does not have an idempotent
plan() method as it alters the state of internal values therefore affecting the
full run once done() is called.
It would be nice if we added an idempotent plan() method that could be gather
this information or perhaps a reset option.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
