[ https://issues.apache.org/jira/browse/HIVE-4318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13627793#comment-13627793 ]
Gopal V commented on HIVE-4318: ------------------------------- I agree with Gunther's conclusion - moving this out would be the immediate fix for performance reasons. But a built-in profiler is an advantage in many ways, considering how something like yourkit takes ~45 mins to run through a 2 min query & requires a license key to inspect the output. There is definitely middle ground to be found here. The operator base-class hooks need to be removed for sure since it pollutes the common code-path and allocates objects in the inner loop. But the feature is a good idea & should be set aside for an implementation revisit later. > OperatorHooks hit performance even when not used > ------------------------------------------------ > > Key: HIVE-4318 > URL: https://issues.apache.org/jira/browse/HIVE-4318 > Project: Hive > Issue Type: Bug > Components: Query Processor > Environment: Ubuntu LXC (64 bit) > Reporter: Gopal V > Assignee: Gunther Hagleitner > Attachments: HIVE-4318.1.patch > > > Operator Hooks inserted into Operator.java cause a performance hit even when > it is not being used. > For a count(1) query tested with & without the operator hook calls. > {code:title=with} > 2013-04-09 07:33:58,920 Stage-1 map = 100%, reduce = 100%, Cumulative CPU > 84.07 sec > Total MapReduce CPU Time Spent: 1 minutes 24 seconds 70 msec > OK > 28800991 > Time taken: 40.407 seconds, Fetched: 1 row(s) > {code} > {code:title=without} > 2013-04-09 07:33:02,355 Stage-1 map = 100%, reduce = 100%, Cumulative CPU > 68.48 sec > ... > Total MapReduce CPU Time Spent: 1 minutes 8 seconds 480 msec > OK > 28800991 > Time taken: 35.907 seconds, Fetched: 1 row(s) > {code} > The effect is multiplied by the number of operators in the pipeline that has > to forward the row - the more operators there are the, the slower the query. > The modification made to test this was > {code:title=Operator.java} > --- ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java > +++ ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java > @@ -526,16 +526,16 @@ public void process(Object row, int tag) throws > HiveException { > return; > } > OperatorHookContext opHookContext = new OperatorHookContext(this, row, > tag); > - preProcessCounter(); > - enterOperatorHooks(opHookContext); > + //preProcessCounter(); > + //enterOperatorHooks(opHookContext); > processOp(row, tag); > - exitOperatorHooks(opHookContext); > - postProcessCounter(); > + //exitOperatorHooks(opHookContext); > + //postProcessCounter(); > } > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira