Hi Daniel Thank you for your reply. Any suggestion about how can I run pig local mode using Tez without minicluster? I am sure the code on Tez side is ready. What changes need to be made on PIG? I am not very familar with PIG.
Regards! Chen On Mon, May 26, 2014 at 8:22 PM, Chen He <[email protected]> wrote: > Hi Jon, Daniel > > Daniel was right. I tried pig local mode using "pig -x local -x tez …..". > It tries to connect mini cluster and finally timeout. The error that I > mentioned before : "ERROR 2998: Unhandled internal error. Found interface > org.apache.hadoop.yarn.api.records.LocalResource, but class was expected" > is because I have a hadoop 0.23 mini cluster running in background. > > Chen > > > On Mon, May 26, 2014 at 9:39 AM, Jonathan Eagles <[email protected]>wrote: > >> Hi Daniel, >> >> You've been very helpful in your replies. I believe the missing context is >> that Chen is working on adding Tez Local Mode (TEZ-684) and is trying to >> integrate with Pig. The question as I interpret it is whether Pig needs or >> creates a MiniCluster in local mode. Chen can you confirm this? >> >> Jon >> >> >> On Mon, May 26, 2014 at 2:55 AM, Daniel Dai <[email protected]> >> wrote: >> >> > Hi, Chen, >> > >> > Please note Pig local mode does not run tez. It uses MR engine. >> > Tez-local mode is not implemented (depend on TEZ-235). You will need a >> > real cluster or use TezMiniCluster to test Pig on Tez. >> > >> > Thanks, >> > Daniel >> > >> > On Sun, May 25, 2014 at 11:42 AM, Chen He <[email protected]> wrote: >> > > Hi Daniel >> > > >> > > I was testing the pig local mode. Does pig need mini cluster when it >> runs >> > > in local mode? >> > > >> > > Regards! >> > > >> > > Chen >> > > >> > > >> > > On Sun, May 25, 2014 at 1:40 PM, Daniel Dai <[email protected]> >> > wrote: >> > > >> > >> Which Hadoop are you running against? Also 2.2? >> > >> >> > >> On Sun, May 25, 2014 at 6:52 AM, Chen He <[email protected]> wrote: >> > >> > Hi Daniel >> > >> > >> > >> > Thank you for the reply. I compiled Tez and Pig against hadoop 2.2. >> > >> > >> > >> > Regards! >> > >> > >> > >> > Chen >> > >> > >> > >> > >> > >> > On Thu, May 22, 2014 at 5:22 PM, Daniel Dai <[email protected] >> > >> > >> wrote: >> > >> > >> > >> >> Which version of Hadoop are you using? Seem LocalResource changed >> > from >> > >> >> interface to class in 2.1 (YARN-711). If you are using pre-2.1, >> this >> > >> >> might be the case. >> > >> >> >> > >> >> Thanks, >> > >> >> Daniel >> > >> >> >> > >> >> On Thu, May 22, 2014 at 3:06 PM, Chen He <[email protected]> >> wrote: >> > >> >> > Hi Daniel >> > >> >> > >> > >> >> > Thank you for the reply. I did as you suggested but get >> following >> > >> errors: >> > >> >> > >> > >> >> > ERROR 2998: Unhandled internal error. Found interface >> > >> >> > org.apache.hadoop.yarn.api.records.LocalResource, but class was >> > >> expected >> > >> >> > >> > >> >> > java.lang.IncompatibleClassChangeError: Found interface >> > >> >> > org.apache.hadoop.yarn.api.records.LocalResource, but class was >> > >> expected >> > >> >> > at >> > >> >> > >> > >> >> >> > >> >> > >> org.apache.pig.backend.hadoop.executionengine.tez.TezResourceManager.getTezResources(TezResourceManager.java:124) >> > >> >> > at >> > >> >> > >> > >> >> >> > >> >> > >> org.apache.pig.backend.hadoop.executionengine.tez.TezResourceManager.addTezResources(TezResourceManager.java:96) >> > >> >> > at >> > >> >> > >> > >> >> >> > >> >> > >> org.apache.pig.backend.hadoop.executionengine.tez.TezPlanContainer.getLocalResources(TezPlanContainer.java:104) >> > >> >> > at >> > >> >> > >> > >> >> >> > >> >> > >> org.apache.pig.backend.hadoop.executionengine.tez.TezJobControlCompiler.getJob(TezJobControlCompiler.java:106) >> > >> >> > at >> > >> >> > >> > >> >> >> > >> >> > >> org.apache.pig.backend.hadoop.executionengine.tez.TezJobControlCompiler.compile(TezJobControlCompiler.java:89) >> > >> >> > at >> > >> >> > >> > >> >> >> > >> >> > >> org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher.launchPig(TezLauncher.java:92) >> > >> >> > at >> > >> >> > >> > >> >> >> > >> >> > >> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:277) >> > >> >> > at org.apache.pig.PigServer.launchPlan(PigServer.java:1378) >> > >> >> > at >> > >> >> > >> > >> >> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1363) >> > >> >> > at org.apache.pig.PigServer.execute(PigServer.java:1352) >> > >> >> > at org.apache.pig.PigServer.executeBatch(PigServer.java:403) >> > >> >> > at org.apache.pig.PigServer.executeBatch(PigServer.java:386) >> > >> >> > at >> > >> >> > >> > >> >> > >> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:170) >> > >> >> > at >> > >> >> > >> > >> >> >> > >> >> > >> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:232) >> > >> >> > at >> > >> >> > >> > >> >> >> > >> >> > >> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:203) >> > >> >> > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) >> > >> >> > at org.apache.pig.Main.run(Main.java:608) >> > >> >> > at org.apache.pig.Main.main(Main.java:156) >> > >> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >> > >> >> > at >> > >> >> > >> > >> >> >> > >> >> > >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> > >> >> > at >> > >> >> > >> > >> >> >> > >> >> > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> > >> >> > at java.lang.reflect.Method.invoke(Method.java:606) >> > >> >> > at org.apache.hadoop.util.RunJar.main(RunJar.java:208) >> > >> >> > >> > >> >> > Is this a problem that caused by hadoop incompatiblilty? Which >> > >> version of >> > >> >> > Hadoop is the best? >> > >> >> > >> > >> >> > Regards! >> > >> >> > >> > >> >> > Chen >> > >> >> > >> > >> >> > >> > >> >> > On Thu, May 22, 2014 at 4:19 PM, Daniel Dai < >> [email protected] >> > > >> > >> >> wrote: >> > >> >> > >> > >> >> >> Need to set PIG_CLASSPATH: >> > >> >> >> export PIG_CLASSPATH=$TEZ_HOME/*:$TEZ_HOME/lib/*:$TEZ_HOME/conf >> > >> >> >> >> > >> >> >> Thanks, >> > >> >> >> Daniel >> > >> >> >> >> > >> >> >> On Thu, May 22, 2014 at 2:11 PM, Chen He <[email protected]> >> > wrote: >> > >> >> >> > Any one met this error before? >> > >> >> >> > >> > >> >> >> > [main] ERROR org.apache.pig.Main - ERROR 2998: Unhandled >> > internal >> > >> >> error. >> > >> >> >> > org/apache/tez/common/counters/DAGCounter >> > >> >> >> > >> > >> >> >> > Any reply will be appreciated! >> > >> >> >> > >> > >> >> >> > Regards! >> > >> >> >> > >> > >> >> >> > Chen >> > >> >> >> >> > >> >> >> -- >> > >> >> >> CONFIDENTIALITY NOTICE >> > >> >> >> NOTICE: This message is intended for the use of the individual >> or >> > >> >> entity to >> > >> >> >> which it is addressed and may contain information that is >> > >> confidential, >> > >> >> >> privileged and exempt from disclosure under applicable law. If >> the >> > >> >> reader >> > >> >> >> of this message is not the intended recipient, you are hereby >> > >> notified >> > >> >> that >> > >> >> >> any printing, copying, dissemination, distribution, disclosure >> or >> > >> >> >> forwarding of this communication is strictly prohibited. If you >> > have >> > >> >> >> received this communication in error, please contact the sender >> > >> >> immediately >> > >> >> >> and delete it from your system. Thank You. >> > >> >> >> >> > >> >> >> > >> >> -- >> > >> >> CONFIDENTIALITY NOTICE >> > >> >> NOTICE: This message is intended for the use of the individual or >> > >> entity to >> > >> >> which it is addressed and may contain information that is >> > confidential, >> > >> >> privileged and exempt from disclosure under applicable law. If the >> > >> reader >> > >> >> of this message is not the intended recipient, you are hereby >> > notified >> > >> that >> > >> >> any printing, copying, dissemination, distribution, disclosure or >> > >> >> forwarding of this communication is strictly prohibited. If you >> have >> > >> >> received this communication in error, please contact the sender >> > >> immediately >> > >> >> and delete it from your system. Thank You. >> > >> >> >> > >> >> > >> -- >> > >> CONFIDENTIALITY NOTICE >> > >> NOTICE: This message is intended for the use of the individual or >> > entity to >> > >> which it is addressed and may contain information that is >> confidential, >> > >> privileged and exempt from disclosure under applicable law. If the >> > reader >> > >> of this message is not the intended recipient, you are hereby >> notified >> > that >> > >> any printing, copying, dissemination, distribution, disclosure or >> > >> forwarding of this communication is strictly prohibited. If you have >> > >> received this communication in error, please contact the sender >> > immediately >> > >> and delete it from your system. Thank You. >> > >> >> > >> > -- >> > CONFIDENTIALITY NOTICE >> > NOTICE: This message is intended for the use of the individual or >> entity to >> > which it is addressed and may contain information that is confidential, >> > privileged and exempt from disclosure under applicable law. If the >> reader >> > of this message is not the intended recipient, you are hereby notified >> that >> > any printing, copying, dissemination, distribution, disclosure or >> > forwarding of this communication is strictly prohibited. If you have >> > received this communication in error, please contact the sender >> immediately >> > and delete it from your system. Thank You. >> > >> > >
