[
https://issues.apache.org/jira/browse/PIG-4928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15332126#comment-15332126
]
Rohini Palaniswamy edited comment on PIG-4928 at 6/15/16 5:21 PM:
------------------------------------------------------------------
{code}
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.<init>(MapReduceLauncher.java:102)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRExecutionEngine.<init>(MRExecutionEngine.java:34)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRExecType.getExecutionEngine(MRExecType.java:50)
at org.apache.pig.impl.PigContext.<init>(PigContext.java:264)
at org.apache.pig.impl.PigContext.<init>(PigContext.java:243)
at org.apache.pig.impl.util.Utils.parseSchema(Utils.java:259)
at org.apache.pig.impl.util.Utils.getSchemaFromString(Utils.java:235)
at
org.apache.pig.scripting.jython.JythonScriptEngine.registerFunctions(JythonScriptEngine.java:362)
at org.apache.pig.PigServer.registerCode(PigServer.java:619)
at
org.apache.pig.tools.grunt.GruntParser.processRegister(GruntParser.java:471)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:550)
{code}
{code}
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.<init>(MapReduceLauncher.java:102)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRExecutionEngine.<init>(MRExecutionEngine.java:34)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRExecType.getExecutionEngine(MRExecType.java:50)
at org.apache.pig.impl.PigContext.<init>(PigContext.java:264)
at org.apache.pig.impl.PigContext.<init>(PigContext.java:243)
at org.apache.pig.impl.util.Utils.parseSchema(Utils.java:259)
at org.apache.pig.impl.util.Utils.getSchemaFromString(Utils.java:235)
at
org.apache.hive.hcatalog.pig.PigHCatUtil.<clinit>(PigHCatUtil.java:99)
at org.apache.hive.hcatalog.pig.HCatLoader.<init>(HCatLoader.java:75)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at java.lang.Class.newInstance(Class.java:442)
at
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:740)
at
org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:885)
{code}
{code}
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.<init>(MapReduceLauncher.java:102)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRExecutionEngine.<init>(MRExecutionEngine.java:34)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRExecType.getExecutionEngine(MRExecType.java:50)
at org.apache.pig.impl.PigContext.<init>(PigContext.java:264)
at org.apache.pig.impl.PigContext.<init>(PigContext.java:243)
at org.apache.pig.impl.util.Utils.parseSchema(Utils.java:259)
at org.apache.pig.impl.util.Utils.getSchemaFromString(Utils.java:235)
at
org.apache.pig.scripting.jython.JythonFunction.<init>(JythonFunction.java:62)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:738)
at
org.apache.pig.newplan.logical.expression.UserFuncExpression.getFieldSchema(UserFuncExpression.java:221)
at
org.apache.pig.newplan.logical.optimizer.FieldSchemaResetter.execute(SchemaResetter.java:264)
at
org.apache.pig.newplan.logical.expression.AllSameExpressionVisitor.visit(AllSameExpressionVisitor.java:143)
at
org.apache.pig.newplan.logical.expression.UserFuncExpression.accept(UserFuncExpression.java:112)
at
org.apache.pig.newplan.ReverseDependencyOrderWalker.walk(ReverseDependencyOrderWalker.java:70)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
at
org.apache.pig.newplan.logical.optimizer.SchemaResetter.visitAll(SchemaResetter.java:67)
at
org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:122)
at
org.apache.pig.newplan.logical.relational.LOGenerate.accept(LOGenerate.java:245)
at
org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
at
org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:114)
at
org.apache.pig.parser.LogicalPlanBuilder.buildForeachOp(LogicalPlanBuilder.java:1055)
at
org.apache.pig.parser.LogicalPlanGenerator.foreach_clause(LogicalPlanGenerator.java:15896)
{code}
and two more instances of JythonFunction initiation from TypeCheckingRelVisitor
and LogToPhyTranslationVisitor totalling 5 for a script that used one jython
udf and one hcat load statement.
Use of new PigContext() in Utils.parseConstant() and parseSchema() causes this.
was (Author: rohini):
{code}
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.<init>(MapReduceLauncher.java:102)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRExecutionEngine.<init>(MRExecutionEngine.java:34)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRExecType.getExecutionEngine(MRExecType.java:50)
at org.apache.pig.impl.PigContext.<init>(PigContext.java:264)
at org.apache.pig.impl.PigContext.<init>(PigContext.java:243)
at org.apache.pig.impl.util.Utils.parseSchema(Utils.java:259)
at org.apache.pig.impl.util.Utils.getSchemaFromString(Utils.java:235)
at
org.apache.pig.scripting.jython.JythonScriptEngine.registerFunctions(JythonScriptEngine.java:362)
at org.apache.pig.PigServer.registerCode(PigServer.java:619)
at
org.apache.pig.tools.grunt.GruntParser.processRegister(GruntParser.java:471)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:550)
{code}
{code}
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.<init>(MapReduceLauncher.java:102)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRExecutionEngine.<init>(MRExecutionEngine.java:34)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRExecType.getExecutionEngine(MRExecType.java:50)
at org.apache.pig.impl.PigContext.<init>(PigContext.java:264)
at org.apache.pig.impl.PigContext.<init>(PigContext.java:243)
at org.apache.pig.impl.util.Utils.parseSchema(Utils.java:259)
at org.apache.pig.impl.util.Utils.getSchemaFromString(Utils.java:235)
at
org.apache.hive.hcatalog.pig.PigHCatUtil.<clinit>(PigHCatUtil.java:99)
at org.apache.hive.hcatalog.pig.HCatLoader.<init>(HCatLoader.java:75)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at java.lang.Class.newInstance(Class.java:442)
at
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:740)
at
org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:885)
{code}
{code}
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.<init>(MapReduceLauncher.java:102)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRExecutionEngine.<init>(MRExecutionEngine.java:34)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRExecType.getExecutionEngine(MRExecType.java:50)
at org.apache.pig.impl.PigContext.<init>(PigContext.java:264)
at org.apache.pig.impl.PigContext.<init>(PigContext.java:243)
at org.apache.pig.impl.util.Utils.parseSchema(Utils.java:259)
at org.apache.pig.impl.util.Utils.getSchemaFromString(Utils.java:235)
at
org.apache.pig.scripting.jython.JythonFunction.<init>(JythonFunction.java:62)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:738)
at
org.apache.pig.newplan.logical.expression.UserFuncExpression.getFieldSchema(UserFuncExpression.java:221)
at
org.apache.pig.newplan.logical.optimizer.FieldSchemaResetter.execute(SchemaResetter.java:264)
at
org.apache.pig.newplan.logical.expression.AllSameExpressionVisitor.visit(AllSameExpressionVisitor.java:143)
at
org.apache.pig.newplan.logical.expression.UserFuncExpression.accept(UserFuncExpression.java:112)
at
org.apache.pig.newplan.ReverseDependencyOrderWalker.walk(ReverseDependencyOrderWalker.java:70)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
at
org.apache.pig.newplan.logical.optimizer.SchemaResetter.visitAll(SchemaResetter.java:67)
at
org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:122)
at
org.apache.pig.newplan.logical.relational.LOGenerate.accept(LOGenerate.java:245)
at
org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
at
org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:114)
at
org.apache.pig.parser.LogicalPlanBuilder.buildForeachOp(LogicalPlanBuilder.java:1055)
at
org.apache.pig.parser.LogicalPlanGenerator.foreach_clause(LogicalPlanGenerator.java:15896)
{code}
and two more instances of JythonFunction initiation from TypeCheckingRelVisitor
and LogToPhyTranslationVisitor totalling 5.
Use of new PigContext() in Utils.parseConstant() and parseSchema() causes this.
> MRExecutionEngine gets initialized even with Tez
> ------------------------------------------------
>
> Key: PIG-4928
> URL: https://issues.apache.org/jira/browse/PIG-4928
> Project: Pig
> Issue Type: Improvement
> Reporter: Rohini Palaniswamy
>
> In PIG-4921, made the log message in MapreduceLauncher shutdown hook to be
> info. Saw that it was logged multiple times when using jython udfs or
> HCatLoader.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
