Hi Daniel, You've been very helpful in your replies. I believe the missing context is that Chen is working on adding Tez Local Mode (TEZ-684) and is trying to integrate with Pig. The question as I interpret it is whether Pig needs or creates a MiniCluster in local mode. Chen can you confirm this?
Jon On Mon, May 26, 2014 at 2:55 AM, Daniel Dai <[email protected]> wrote: > Hi, Chen, > > Please note Pig local mode does not run tez. It uses MR engine. > Tez-local mode is not implemented (depend on TEZ-235). You will need a > real cluster or use TezMiniCluster to test Pig on Tez. > > Thanks, > Daniel > > On Sun, May 25, 2014 at 11:42 AM, Chen He <[email protected]> wrote: > > Hi Daniel > > > > I was testing the pig local mode. Does pig need mini cluster when it runs > > in local mode? > > > > Regards! > > > > Chen > > > > > > On Sun, May 25, 2014 at 1:40 PM, Daniel Dai <[email protected]> > wrote: > > > >> Which Hadoop are you running against? Also 2.2? > >> > >> On Sun, May 25, 2014 at 6:52 AM, Chen He <[email protected]> wrote: > >> > Hi Daniel > >> > > >> > Thank you for the reply. I compiled Tez and Pig against hadoop 2.2. > >> > > >> > Regards! > >> > > >> > Chen > >> > > >> > > >> > On Thu, May 22, 2014 at 5:22 PM, Daniel Dai <[email protected]> > >> wrote: > >> > > >> >> Which version of Hadoop are you using? Seem LocalResource changed > from > >> >> interface to class in 2.1 (YARN-711). If you are using pre-2.1, this > >> >> might be the case. > >> >> > >> >> Thanks, > >> >> Daniel > >> >> > >> >> On Thu, May 22, 2014 at 3:06 PM, Chen He <[email protected]> wrote: > >> >> > Hi Daniel > >> >> > > >> >> > Thank you for the reply. I did as you suggested but get following > >> errors: > >> >> > > >> >> > ERROR 2998: Unhandled internal error. Found interface > >> >> > org.apache.hadoop.yarn.api.records.LocalResource, but class was > >> expected > >> >> > > >> >> > java.lang.IncompatibleClassChangeError: Found interface > >> >> > org.apache.hadoop.yarn.api.records.LocalResource, but class was > >> expected > >> >> > at > >> >> > > >> >> > >> > org.apache.pig.backend.hadoop.executionengine.tez.TezResourceManager.getTezResources(TezResourceManager.java:124) > >> >> > at > >> >> > > >> >> > >> > org.apache.pig.backend.hadoop.executionengine.tez.TezResourceManager.addTezResources(TezResourceManager.java:96) > >> >> > at > >> >> > > >> >> > >> > org.apache.pig.backend.hadoop.executionengine.tez.TezPlanContainer.getLocalResources(TezPlanContainer.java:104) > >> >> > at > >> >> > > >> >> > >> > org.apache.pig.backend.hadoop.executionengine.tez.TezJobControlCompiler.getJob(TezJobControlCompiler.java:106) > >> >> > at > >> >> > > >> >> > >> > org.apache.pig.backend.hadoop.executionengine.tez.TezJobControlCompiler.compile(TezJobControlCompiler.java:89) > >> >> > at > >> >> > > >> >> > >> > org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher.launchPig(TezLauncher.java:92) > >> >> > at > >> >> > > >> >> > >> > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:277) > >> >> > at org.apache.pig.PigServer.launchPlan(PigServer.java:1378) > >> >> > at > >> >> > > >> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1363) > >> >> > at org.apache.pig.PigServer.execute(PigServer.java:1352) > >> >> > at org.apache.pig.PigServer.executeBatch(PigServer.java:403) > >> >> > at org.apache.pig.PigServer.executeBatch(PigServer.java:386) > >> >> > at > >> >> > > >> > org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:170) > >> >> > at > >> >> > > >> >> > >> > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:232) > >> >> > at > >> >> > > >> >> > >> > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:203) > >> >> > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) > >> >> > at org.apache.pig.Main.run(Main.java:608) > >> >> > at org.apache.pig.Main.main(Main.java:156) > >> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> >> > at > >> >> > > >> >> > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > >> >> > at > >> >> > > >> >> > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> >> > at java.lang.reflect.Method.invoke(Method.java:606) > >> >> > at org.apache.hadoop.util.RunJar.main(RunJar.java:208) > >> >> > > >> >> > Is this a problem that caused by hadoop incompatiblilty? Which > >> version of > >> >> > Hadoop is the best? > >> >> > > >> >> > Regards! > >> >> > > >> >> > Chen > >> >> > > >> >> > > >> >> > On Thu, May 22, 2014 at 4:19 PM, Daniel Dai <[email protected] > > > >> >> wrote: > >> >> > > >> >> >> Need to set PIG_CLASSPATH: > >> >> >> export PIG_CLASSPATH=$TEZ_HOME/*:$TEZ_HOME/lib/*:$TEZ_HOME/conf > >> >> >> > >> >> >> Thanks, > >> >> >> Daniel > >> >> >> > >> >> >> On Thu, May 22, 2014 at 2:11 PM, Chen He <[email protected]> > wrote: > >> >> >> > Any one met this error before? > >> >> >> > > >> >> >> > [main] ERROR org.apache.pig.Main - ERROR 2998: Unhandled > internal > >> >> error. > >> >> >> > org/apache/tez/common/counters/DAGCounter > >> >> >> > > >> >> >> > Any reply will be appreciated! > >> >> >> > > >> >> >> > Regards! > >> >> >> > > >> >> >> > Chen > >> >> >> > >> >> >> -- > >> >> >> CONFIDENTIALITY NOTICE > >> >> >> NOTICE: This message is intended for the use of the individual or > >> >> entity to > >> >> >> which it is addressed and may contain information that is > >> confidential, > >> >> >> privileged and exempt from disclosure under applicable law. If the > >> >> reader > >> >> >> of this message is not the intended recipient, you are hereby > >> notified > >> >> that > >> >> >> any printing, copying, dissemination, distribution, disclosure or > >> >> >> forwarding of this communication is strictly prohibited. If you > have > >> >> >> received this communication in error, please contact the sender > >> >> immediately > >> >> >> and delete it from your system. Thank You. > >> >> >> > >> >> > >> >> -- > >> >> CONFIDENTIALITY NOTICE > >> >> NOTICE: This message is intended for the use of the individual or > >> entity to > >> >> which it is addressed and may contain information that is > confidential, > >> >> privileged and exempt from disclosure under applicable law. If the > >> reader > >> >> of this message is not the intended recipient, you are hereby > notified > >> that > >> >> any printing, copying, dissemination, distribution, disclosure or > >> >> forwarding of this communication is strictly prohibited. If you have > >> >> received this communication in error, please contact the sender > >> immediately > >> >> and delete it from your system. Thank You. > >> >> > >> > >> -- > >> CONFIDENTIALITY NOTICE > >> NOTICE: This message is intended for the use of the individual or > entity to > >> which it is addressed and may contain information that is confidential, > >> privileged and exempt from disclosure under applicable law. If the > reader > >> of this message is not the intended recipient, you are hereby notified > that > >> any printing, copying, dissemination, distribution, disclosure or > >> forwarding of this communication is strictly prohibited. If you have > >> received this communication in error, please contact the sender > immediately > >> and delete it from your system. Thank You. > >> > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
