Hi Jon, Daniel Daniel was right. I tried pig local mode using "pig -x local -x tez …..". It tries to connect mini cluster and finally timeout. The error that I mentioned before : "ERROR 2998: Unhandled internal error. Found interface org.apache.hadoop.yarn.api.records.LocalResource, but class was expected" is because I have a hadoop 0.23 mini cluster running in background.
Chen On Mon, May 26, 2014 at 9:39 AM, Jonathan Eagles <[email protected]> wrote: > Hi Daniel, > > You've been very helpful in your replies. I believe the missing context is > that Chen is working on adding Tez Local Mode (TEZ-684) and is trying to > integrate with Pig. The question as I interpret it is whether Pig needs or > creates a MiniCluster in local mode. Chen can you confirm this? > > Jon > > > On Mon, May 26, 2014 at 2:55 AM, Daniel Dai <[email protected]> wrote: > > > Hi, Chen, > > > > Please note Pig local mode does not run tez. It uses MR engine. > > Tez-local mode is not implemented (depend on TEZ-235). You will need a > > real cluster or use TezMiniCluster to test Pig on Tez. > > > > Thanks, > > Daniel > > > > On Sun, May 25, 2014 at 11:42 AM, Chen He <[email protected]> wrote: > > > Hi Daniel > > > > > > I was testing the pig local mode. Does pig need mini cluster when it > runs > > > in local mode? > > > > > > Regards! > > > > > > Chen > > > > > > > > > On Sun, May 25, 2014 at 1:40 PM, Daniel Dai <[email protected]> > > wrote: > > > > > >> Which Hadoop are you running against? Also 2.2? > > >> > > >> On Sun, May 25, 2014 at 6:52 AM, Chen He <[email protected]> wrote: > > >> > Hi Daniel > > >> > > > >> > Thank you for the reply. I compiled Tez and Pig against hadoop 2.2. > > >> > > > >> > Regards! > > >> > > > >> > Chen > > >> > > > >> > > > >> > On Thu, May 22, 2014 at 5:22 PM, Daniel Dai <[email protected]> > > >> wrote: > > >> > > > >> >> Which version of Hadoop are you using? Seem LocalResource changed > > from > > >> >> interface to class in 2.1 (YARN-711). If you are using pre-2.1, > this > > >> >> might be the case. > > >> >> > > >> >> Thanks, > > >> >> Daniel > > >> >> > > >> >> On Thu, May 22, 2014 at 3:06 PM, Chen He <[email protected]> > wrote: > > >> >> > Hi Daniel > > >> >> > > > >> >> > Thank you for the reply. I did as you suggested but get following > > >> errors: > > >> >> > > > >> >> > ERROR 2998: Unhandled internal error. Found interface > > >> >> > org.apache.hadoop.yarn.api.records.LocalResource, but class was > > >> expected > > >> >> > > > >> >> > java.lang.IncompatibleClassChangeError: Found interface > > >> >> > org.apache.hadoop.yarn.api.records.LocalResource, but class was > > >> expected > > >> >> > at > > >> >> > > > >> >> > > >> > > > org.apache.pig.backend.hadoop.executionengine.tez.TezResourceManager.getTezResources(TezResourceManager.java:124) > > >> >> > at > > >> >> > > > >> >> > > >> > > > org.apache.pig.backend.hadoop.executionengine.tez.TezResourceManager.addTezResources(TezResourceManager.java:96) > > >> >> > at > > >> >> > > > >> >> > > >> > > > org.apache.pig.backend.hadoop.executionengine.tez.TezPlanContainer.getLocalResources(TezPlanContainer.java:104) > > >> >> > at > > >> >> > > > >> >> > > >> > > > org.apache.pig.backend.hadoop.executionengine.tez.TezJobControlCompiler.getJob(TezJobControlCompiler.java:106) > > >> >> > at > > >> >> > > > >> >> > > >> > > > org.apache.pig.backend.hadoop.executionengine.tez.TezJobControlCompiler.compile(TezJobControlCompiler.java:89) > > >> >> > at > > >> >> > > > >> >> > > >> > > > org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher.launchPig(TezLauncher.java:92) > > >> >> > at > > >> >> > > > >> >> > > >> > > > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:277) > > >> >> > at org.apache.pig.PigServer.launchPlan(PigServer.java:1378) > > >> >> > at > > >> >> > > > >> > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1363) > > >> >> > at org.apache.pig.PigServer.execute(PigServer.java:1352) > > >> >> > at org.apache.pig.PigServer.executeBatch(PigServer.java:403) > > >> >> > at org.apache.pig.PigServer.executeBatch(PigServer.java:386) > > >> >> > at > > >> >> > > > >> > > org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:170) > > >> >> > at > > >> >> > > > >> >> > > >> > > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:232) > > >> >> > at > > >> >> > > > >> >> > > >> > > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:203) > > >> >> > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) > > >> >> > at org.apache.pig.Main.run(Main.java:608) > > >> >> > at org.apache.pig.Main.main(Main.java:156) > > >> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > > >> >> > at > > >> >> > > > >> >> > > >> > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > >> >> > at > > >> >> > > > >> >> > > >> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > >> >> > at java.lang.reflect.Method.invoke(Method.java:606) > > >> >> > at org.apache.hadoop.util.RunJar.main(RunJar.java:208) > > >> >> > > > >> >> > Is this a problem that caused by hadoop incompatiblilty? Which > > >> version of > > >> >> > Hadoop is the best? > > >> >> > > > >> >> > Regards! > > >> >> > > > >> >> > Chen > > >> >> > > > >> >> > > > >> >> > On Thu, May 22, 2014 at 4:19 PM, Daniel Dai < > [email protected] > > > > > >> >> wrote: > > >> >> > > > >> >> >> Need to set PIG_CLASSPATH: > > >> >> >> export PIG_CLASSPATH=$TEZ_HOME/*:$TEZ_HOME/lib/*:$TEZ_HOME/conf > > >> >> >> > > >> >> >> Thanks, > > >> >> >> Daniel > > >> >> >> > > >> >> >> On Thu, May 22, 2014 at 2:11 PM, Chen He <[email protected]> > > wrote: > > >> >> >> > Any one met this error before? > > >> >> >> > > > >> >> >> > [main] ERROR org.apache.pig.Main - ERROR 2998: Unhandled > > internal > > >> >> error. > > >> >> >> > org/apache/tez/common/counters/DAGCounter > > >> >> >> > > > >> >> >> > Any reply will be appreciated! > > >> >> >> > > > >> >> >> > Regards! > > >> >> >> > > > >> >> >> > Chen > > >> >> >> > > >> >> >> -- > > >> >> >> CONFIDENTIALITY NOTICE > > >> >> >> NOTICE: This message is intended for the use of the individual > or > > >> >> entity to > > >> >> >> which it is addressed and may contain information that is > > >> confidential, > > >> >> >> privileged and exempt from disclosure under applicable law. If > the > > >> >> reader > > >> >> >> of this message is not the intended recipient, you are hereby > > >> notified > > >> >> that > > >> >> >> any printing, copying, dissemination, distribution, disclosure > or > > >> >> >> forwarding of this communication is strictly prohibited. If you > > have > > >> >> >> received this communication in error, please contact the sender > > >> >> immediately > > >> >> >> and delete it from your system. Thank You. > > >> >> >> > > >> >> > > >> >> -- > > >> >> CONFIDENTIALITY NOTICE > > >> >> NOTICE: This message is intended for the use of the individual or > > >> entity to > > >> >> which it is addressed and may contain information that is > > confidential, > > >> >> privileged and exempt from disclosure under applicable law. If the > > >> reader > > >> >> of this message is not the intended recipient, you are hereby > > notified > > >> that > > >> >> any printing, copying, dissemination, distribution, disclosure or > > >> >> forwarding of this communication is strictly prohibited. If you > have > > >> >> received this communication in error, please contact the sender > > >> immediately > > >> >> and delete it from your system. Thank You. > > >> >> > > >> > > >> -- > > >> CONFIDENTIALITY NOTICE > > >> NOTICE: This message is intended for the use of the individual or > > entity to > > >> which it is addressed and may contain information that is > confidential, > > >> privileged and exempt from disclosure under applicable law. If the > > reader > > >> of this message is not the intended recipient, you are hereby notified > > that > > >> any printing, copying, dissemination, distribution, disclosure or > > >> forwarding of this communication is strictly prohibited. If you have > > >> received this communication in error, please contact the sender > > immediately > > >> and delete it from your system. Thank You. > > >> > > > > -- > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity > to > > which it is addressed and may contain information that is confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified > that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender > immediately > > and delete it from your system. Thank You. > > >
