[
https://issues.apache.org/jira/browse/CRUNCH-105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13487604#comment-13487604
]
Gabriel Reid commented on CRUNCH-105:
-------------------------------------
Thanks for the feedback! Moving the config key to PlanningParameters sounds
like a good plan. Putting it in MSCRPlanner didn't feel very right to me, but I
didn't have a better idea of where to put it at the time.
The dotfile info is indeed available on a pipeline that's been executed as well
-- the internal call to the planner populates it whether you run the pipeline
or not.
I'd like to refactor the code a bit, but that will also involve refactoring the
some other things in the planner to make the internal information more
accessible (the dotfile writer has a bit more logic in it than what I would
have liked right now). I see this as a first step to documenting and
refactoring the planner (as it makes it easier to understand what's going on),
but there's still work that can be done here.
I'd also like to add dotfiles for the two stages of the pre-planning process
(initial graph and graph with intermediate files), as well as making this more
accessible via the API.
I think that this is a useful update for the 0.4.0 release (and I'm going to be
offline for a few days), so I'll move the config key and commit it, and then
tackle the planner refactoring, pre-planning graph building, and API
accessibility of the dotfiles separately.
> Add generation of Graphviz dot files to describe the pipeline job plan
> ----------------------------------------------------------------------
>
> Key: CRUNCH-105
> URL: https://issues.apache.org/jira/browse/CRUNCH-105
> Project: Crunch
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.4.0
> Reporter: Gabriel Reid
> Assignee: Gabriel Reid
> Attachments: CRUNCH-105.patch, innerjoingraph.png
>
>
> It could be very useful to have the ability to create Graphviz[1] dot
> files[2] based on the outcome of the planner. This could facilitate
> visualizing the full graph of DoFns as well as jobs and mappers and reducers
> in a Crunch pipeline.
> [1] http://graphviz.org
> [2] http://en.wikipedia.org/wiki/DOT_language
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
