[
https://issues.apache.org/jira/browse/HIVE-69?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12648792#action_12648792
]
Ashish Thusoo commented on HIVE-69:
-----------------------------------
General comment - need a lot of javadocs and documentation of how the state
machine works. I think it works correctly but lets document as I have indicated
below.
Also the state should not be maintained in the plan but should be in the
treewalker. The state is never really used in the execution time and we should
only try to put things that are used in the execution time in the plan.
Otherwise, this looks much cleaner than the pervious monolithic blob that we
had. Thanks...
Inline Comments
ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java:32 Is this
just a place holder right now?
ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java:112 Lets add proper
javadocs!!
ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java:132 javadocs. What
is this used for?
ql/src/java/org/apache/hadoop/hive/ql/parse/GenMapRedTaskProcessor.java:57
Move the new into the constructor?
ql/src/java/org/apache/hadoop/hive/ql/parse/GenMapRedTaskProcessor.java:1
javadocs needed.
ql/src/java/org/apache/hadoop/hive/ql/parse/GenMapRedTaskProcessor.java:117
Please put a comment on how this state machine works. What are the transitions,
how the transitions are made and what are the actions taken by each transition.
ql/src/java/org/apache/hadoop/hive/ql/plan/mapredWork.java:28 This is purely
compile time information that is used to generate the plan (cut the plan into
multiple map reduce plans), so we should not be storing this in the plan. This
seems to be specific to the GenMapRedPlanWalker, so I think we should be
maintaining this state there in a hashmap.
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:2986 Looks
much cleaner with the walker stuff... This is cool...
ql/src/java/org/apache/hadoop/hive/ql/parse/GenMapRedWalker.java:44 I
thought we were going to move this logic into the dispatcher? no?
ql/src/java/org/apache/hadoop/hive/ql/parse/GenMapRedTaskProcessor.java:116
Also a description of the call stack that leads to there transitions would be
very helpful.
> genMapRedTasks does not use the tree walker and uses implicit state which
> makes it difficult to enhance
> -------------------------------------------------------------------------------------------------------
>
> Key: HIVE-69
> URL: https://issues.apache.org/jira/browse/HIVE-69
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Namit Jain
>
> In SemanticAnalyzer, genmapredtasks() does not use a tree walker. For
> map-side joins, the taskplan needs to be enhanced to be possibly
> broken at MapSink also. Basically, the code is very difficult to enhance
> since there are implicit assumptions that reduce sink is the only
> operator where the plan breaks.
> This should be enhanced so that the user can implement their own task
> generation logic which is independent of the tree walking.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
