-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11032/#review20501
-----------------------------------------------------------
Tested disabling this rule in three ways:
PIG_CONF_DIR=/etc/pig/conf ./bin/pig -c query.pig
set pig.optimizer.rules.disabled 'ColumnMapKeyPrune';
PIG_CONF_DIR=/etc/pig/conf ./bin/pig
-Dpig.optimizer.rules.disabled=ColumnMapKeyPrune -c query.pig
--set pig.optimizer.rules.disabled 'ColumnMapKeyPrune';
PIG_CONF_DIR=/etc/pig/conf ./bin/pig -t ColumnMapKeyPrune -c query.pig
--set pig.optimizer.rules.disabled 'ColumnMapKeyPrune';
The rules report looked like this in all cases:
2013-05-13 20:16:15,594 [main] INFO
org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer -
{RULES_ENABLED=[AddForEach, DuplicateForEachColumnRewrite,
GroupByConstParallelSetter, ImplicitSplitInserter, LimitOptimizer,
LoadTypeCastInserter, MergeFilter, MergeForEach, PartitionFilterOptimizer,
PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter],
RULES_DISABLED=[ColumnMapKeyPrune, FilterLogicExpressionSimplifier]}
Note that FilterLogicExpressionSimplifier is disabled by default.
src/org/apache/pig/Main.java
<https://reviews.apache.org/r/11032/#comment42245>
This conditional has been removed to fix what I believe is a bug. Here we
only publish this property if rules have been disabled. However, in
HExecutionEngine.java line 251 the property is required to exist. I believe
this bug snuck in because FilterLogicExpressionSimplifier is disabled by
default so there's typically something disabled.
- Travis Crawford
On May 13, 2013, 8:35 p.m., Travis Crawford wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11032/
> -----------------------------------------------------------
>
> (Updated May 13, 2013, 8:35 p.m.)
>
>
> Review request for pig, Julien Le Dem, Bill Graham, and Feng Peng.
>
>
> Description
> -------
>
> Update pig to allow disabling optimizations via pig properties. Currently
> optimizations must be disabled via command-line options. Pig properties can
> be set in pig.properties, "set" commands in scripts themselves, and
> command-line -D options.
>
> The use-case is, for scripts that require certain optimizations to be
> disabled, allowing the script itself to disable the optimization. Currently
> whatever runs the script needs to specially handle disabling the optimization
> for that specific query.
>
>
> This addresses bug PIG-3317.
> https://issues.apache.org/jira/browse/PIG-3317
>
>
> Diffs
> -----
>
> src/docs/src/documentation/content/xdocs/perf.xml 108ae7e
> src/org/apache/pig/Main.java f97ed9f
> src/org/apache/pig/PigConstants.java ea77e97
> src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java
> 4dab4e8
> src/org/apache/pig/newplan/logical/optimizer/LogicalPlanOptimizer.java
> d26f381
> test/org/apache/pig/test/TestEvalPipeline2.java 39cf807
>
> Diff: https://reviews.apache.org/r/11032/diff/
>
>
> Testing
> -------
>
> Manually tested on a fully-distributed cluster.
>
> THIS FAILS:
> PIG_CONF_DIR=/etc/pig/conf ./bin/pig -c query.pig
>
> THIS WORKS:
> PIG_CONF_DIR=/etc/pig/conf ./bin/pig
> -Dpig.optimizer.rules.disabled=ColumnMapKeyPrune -c query.pig
>
> Notice how "-Dpig.optimizer.rules.disabled=ColumnMapKeyPrune" specifies a pig
> property, which could be in pig.properties, or the script itself.
>
>
> Failure message:
>
> Pig Stack Trace
> ---------------
> ERROR 2229: Couldn't find matching uid -1 for project (Name: Project Type:
> bytearray Uid: 97550 Input: 0 Column: 1)
>
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1067: Unable to
> explain alias null
> at org.apache.pig.PigServer.explain(PigServer.java:1057)
> at
> org.apache.pig.tools.grunt.GruntParser.explainCurrentBatch(GruntParser.java:419)
> at
> org.apache.pig.tools.grunt.GruntParser.processExplain(GruntParser.java:351)
> at org.apache.pig.tools.grunt.Grunt.checkScript(Grunt.java:98)
> at org.apache.pig.Main.run(Main.java:607)
> at org.apache.pig.Main.main(Main.java:152)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2000:
> Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
> at
> org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:122)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:281)
> at org.apache.pig.PigServer.compilePp(PigServer.java:1380)
> at org.apache.pig.PigServer.explain(PigServer.java:1042)
> ... 10 more
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2229:
> Couldn't find matching uid -1 for project (Name: Project Type: bytearray Uid:
> 97550 Input: 0 Column: 1)
> at
> org.apache.pig.newplan.logical.optimizer.ProjectionPatcher$ProjectionRewriter.visit(ProjectionPatcher.java:91)
> at
> org.apache.pig.newplan.logical.expression.ProjectExpression.accept(ProjectExpression.java:207)
> at
> org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
> at
> org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
> at
> org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
> at
> org.apache.pig.newplan.logical.relational.LOInnerLoad.accept(LOInnerLoad.java:128)
> at
> org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at
> org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:124)
> at
> org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:76)
> at
> org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
> at
> org.apache.pig.newplan.logical.optimizer.ProjectionPatcher.transformed(ProjectionPatcher.java:48)
> at
> org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:113)
> ... 13 more
>
>
> Thanks,
>
> Travis Crawford
>
>
