Hi, Chen, The hook for for tez local mode is:
bin/pig -x tez_local You might need to set PIG_OPTS="-Dopt.fetch=false" to prevent it bypassing the tez engine. How do you run tutorial example? Using MR local mode? Thanks, Daniel On Tue, May 27, 2014 at 9:32 AM, Chen He <[email protected]> wrote: > We are running pig tutorial example and get following errors. I think we > are pick up some hadoop config that specify HDFS instead of localFS. > > Here are the errors from pig.log > > Pig Stack Trace > --------------- > ERROR 6007: Unable to check name hdfs://localhost:9000/user/chenhe > > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error > during parsing. Unable to check name hdfs://localhost:9000/user/chenhe > at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1736) > at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1431) > at org.apache.pig.PigServer.parseAndBuild(PigServer.java:375) > at org.apache.pig.PigServer.executeBatch(PigServer.java:400) > at org.apache.pig.PigServer.executeBatch(PigServer.java:386) > at > org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:170) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:232) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:203) > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) > at org.apache.pig.Main.run(Main.java:608) > at org.apache.pig.Main.main(Main.java:156) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.util.RunJar.main(RunJar.java:208) > Caused by: Failed to parse: Pig script failed to parse: > <file ../../../../tutorial/scripts/script1-local.pig, line 28, column 6> > pig script failed to validate: > org.apache.pig.backend.datastorage.DataStorageException: ERROR 6007: Unable > to check name hdfs://localhost:9000/user/chenhe > at > org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:196) > at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1723) > ... 15 more > Caused by: > <file ../../../../tutorial/scripts/script1-local.pig, line 28, column 6> > pig script failed to validate: > org.apache.pig.backend.datastorage.DataStorageException: ERROR 6007: Unable > to check name hdfs://localhost:9000/user/chenhe > at > org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:881) > at > org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568) > at > org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625) > at > org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102) > at > org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560) > at > org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421) > at > org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188) > ... 16 more > Caused by: org.apache.pig.backend.datastorage.DataStorageException: ERROR > 6007: Unable to check name hdfs://localhost:9000/user/chenhe > at > org.apache.pig.backend.hadoop.datastorage.HDataStorage.isContainer(HDataStorage.java:207) > at > org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:128) > at > org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:138) > at > org.apache.pig.parser.QueryParserUtils.getCurrentDir(QueryParserUtils.java:90) > at > org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:873) > ... 22 more > Caused by: java.net.ConnectException: Call From > guardwritten-dl.corp.yahoo.com/10.74.90.55 to localhost:9000 failed on > connection exception: java.net.ConnectException: Connection refused; For > more details see: http://wiki.apache.org/hadoop/ConnectionRefused > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:700) > at org.apache.hadoop.ipc.Client.call(Client.java:1098) > at > org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:195) > at com.sun.proxy.$Proxy6.getFileInfo(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:102) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:67) > at com.sun.proxy.$Proxy6.getFileInfo(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1305) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:734) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1272) > at > org.apache.pig.backend.hadoop.datastorage.HDataStorage.isContainer(HDataStorage.java:200) > ... 26 more > Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735) > at > org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) > at > org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:462) > at > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:559) > at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:207) > at org.apache.hadoop.ipc.Client.getConnection(Client.java:1204) > at org.apache.hadoop.ipc.Client.call(Client.java:1074) > ... 39 more > ================================================================================ > > > > On Tue, May 27, 2014 at 11:04 AM, Chen He <[email protected]> wrote: > >> Hi Daniel >> >> Thank you for your reply. Any suggestion about how can I run pig local >> mode using Tez without minicluster? I am sure the code on Tez side is >> ready. What changes need to be made on PIG? I am not very familar with PIG. >> >> Regards! >> >> Chen >> >> >> On Mon, May 26, 2014 at 8:22 PM, Chen He <[email protected]> wrote: >> >>> Hi Jon, Daniel >>> >>> Daniel was right. I tried pig local mode using "pig -x local -x tez …..". >>> It tries to connect mini cluster and finally timeout. The error that I >>> mentioned before : "ERROR 2998: Unhandled internal error. Found >>> interface org.apache.hadoop.yarn.api.records.LocalResource, but class >>> was expected" >>> is because I have a hadoop 0.23 mini cluster running in background. >>> >>> Chen >>> >>> >>> On Mon, May 26, 2014 at 9:39 AM, Jonathan Eagles <[email protected]>wrote: >>> >>>> Hi Daniel, >>>> >>>> You've been very helpful in your replies. I believe the missing context >>>> is >>>> that Chen is working on adding Tez Local Mode (TEZ-684) and is trying to >>>> integrate with Pig. The question as I interpret it is whether Pig needs >>>> or >>>> creates a MiniCluster in local mode. Chen can you confirm this? >>>> >>>> Jon >>>> >>>> >>>> On Mon, May 26, 2014 at 2:55 AM, Daniel Dai <[email protected]> >>>> wrote: >>>> >>>> > Hi, Chen, >>>> > >>>> > Please note Pig local mode does not run tez. It uses MR engine. >>>> > Tez-local mode is not implemented (depend on TEZ-235). You will need a >>>> > real cluster or use TezMiniCluster to test Pig on Tez. >>>> > >>>> > Thanks, >>>> > Daniel >>>> > >>>> > On Sun, May 25, 2014 at 11:42 AM, Chen He <[email protected]> wrote: >>>> > > Hi Daniel >>>> > > >>>> > > I was testing the pig local mode. Does pig need mini cluster when it >>>> runs >>>> > > in local mode? >>>> > > >>>> > > Regards! >>>> > > >>>> > > Chen >>>> > > >>>> > > >>>> > > On Sun, May 25, 2014 at 1:40 PM, Daniel Dai <[email protected]> >>>> > wrote: >>>> > > >>>> > >> Which Hadoop are you running against? Also 2.2? >>>> > >> >>>> > >> On Sun, May 25, 2014 at 6:52 AM, Chen He <[email protected]> wrote: >>>> > >> > Hi Daniel >>>> > >> > >>>> > >> > Thank you for the reply. I compiled Tez and Pig against hadoop >>>> 2.2. >>>> > >> > >>>> > >> > Regards! >>>> > >> > >>>> > >> > Chen >>>> > >> > >>>> > >> > >>>> > >> > On Thu, May 22, 2014 at 5:22 PM, Daniel Dai < >>>> [email protected]> >>>> > >> wrote: >>>> > >> > >>>> > >> >> Which version of Hadoop are you using? Seem LocalResource changed >>>> > from >>>> > >> >> interface to class in 2.1 (YARN-711). If you are using pre-2.1, >>>> this >>>> > >> >> might be the case. >>>> > >> >> >>>> > >> >> Thanks, >>>> > >> >> Daniel >>>> > >> >> >>>> > >> >> On Thu, May 22, 2014 at 3:06 PM, Chen He <[email protected]> >>>> wrote: >>>> > >> >> > Hi Daniel >>>> > >> >> > >>>> > >> >> > Thank you for the reply. I did as you suggested but get >>>> following >>>> > >> errors: >>>> > >> >> > >>>> > >> >> > ERROR 2998: Unhandled internal error. Found interface >>>> > >> >> > org.apache.hadoop.yarn.api.records.LocalResource, but class was >>>> > >> expected >>>> > >> >> > >>>> > >> >> > java.lang.IncompatibleClassChangeError: Found interface >>>> > >> >> > org.apache.hadoop.yarn.api.records.LocalResource, but class was >>>> > >> expected >>>> > >> >> > at >>>> > >> >> > >>>> > >> >> >>>> > >> >>>> > >>>> org.apache.pig.backend.hadoop.executionengine.tez.TezResourceManager.getTezResources(TezResourceManager.java:124) >>>> > >> >> > at >>>> > >> >> > >>>> > >> >> >>>> > >> >>>> > >>>> org.apache.pig.backend.hadoop.executionengine.tez.TezResourceManager.addTezResources(TezResourceManager.java:96) >>>> > >> >> > at >>>> > >> >> > >>>> > >> >> >>>> > >> >>>> > >>>> org.apache.pig.backend.hadoop.executionengine.tez.TezPlanContainer.getLocalResources(TezPlanContainer.java:104) >>>> > >> >> > at >>>> > >> >> > >>>> > >> >> >>>> > >> >>>> > >>>> org.apache.pig.backend.hadoop.executionengine.tez.TezJobControlCompiler.getJob(TezJobControlCompiler.java:106) >>>> > >> >> > at >>>> > >> >> > >>>> > >> >> >>>> > >> >>>> > >>>> org.apache.pig.backend.hadoop.executionengine.tez.TezJobControlCompiler.compile(TezJobControlCompiler.java:89) >>>> > >> >> > at >>>> > >> >> > >>>> > >> >> >>>> > >> >>>> > >>>> org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher.launchPig(TezLauncher.java:92) >>>> > >> >> > at >>>> > >> >> > >>>> > >> >> >>>> > >> >>>> > >>>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:277) >>>> > >> >> > at org.apache.pig.PigServer.launchPlan(PigServer.java:1378) >>>> > >> >> > at >>>> > >> >> > >>>> > >> >>>> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1363) >>>> > >> >> > at org.apache.pig.PigServer.execute(PigServer.java:1352) >>>> > >> >> > at >>>> org.apache.pig.PigServer.executeBatch(PigServer.java:403) >>>> > >> >> > at >>>> org.apache.pig.PigServer.executeBatch(PigServer.java:386) >>>> > >> >> > at >>>> > >> >> > >>>> > >> >>>> > >>>> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:170) >>>> > >> >> > at >>>> > >> >> > >>>> > >> >> >>>> > >> >>>> > >>>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:232) >>>> > >> >> > at >>>> > >> >> > >>>> > >> >> >>>> > >> >>>> > >>>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:203) >>>> > >> >> > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) >>>> > >> >> > at org.apache.pig.Main.run(Main.java:608) >>>> > >> >> > at org.apache.pig.Main.main(Main.java:156) >>>> > >> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >>>> Method) >>>> > >> >> > at >>>> > >> >> > >>>> > >> >> >>>> > >> >>>> > >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>> > >> >> > at >>>> > >> >> > >>>> > >> >> >>>> > >> >>>> > >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> > >> >> > at java.lang.reflect.Method.invoke(Method.java:606) >>>> > >> >> > at org.apache.hadoop.util.RunJar.main(RunJar.java:208) >>>> > >> >> > >>>> > >> >> > Is this a problem that caused by hadoop incompatiblilty? Which >>>> > >> version of >>>> > >> >> > Hadoop is the best? >>>> > >> >> > >>>> > >> >> > Regards! >>>> > >> >> > >>>> > >> >> > Chen >>>> > >> >> > >>>> > >> >> > >>>> > >> >> > On Thu, May 22, 2014 at 4:19 PM, Daniel Dai < >>>> [email protected] >>>> > > >>>> > >> >> wrote: >>>> > >> >> > >>>> > >> >> >> Need to set PIG_CLASSPATH: >>>> > >> >> >> export >>>> PIG_CLASSPATH=$TEZ_HOME/*:$TEZ_HOME/lib/*:$TEZ_HOME/conf >>>> > >> >> >> >>>> > >> >> >> Thanks, >>>> > >> >> >> Daniel >>>> > >> >> >> >>>> > >> >> >> On Thu, May 22, 2014 at 2:11 PM, Chen He <[email protected]> >>>> > wrote: >>>> > >> >> >> > Any one met this error before? >>>> > >> >> >> > >>>> > >> >> >> > [main] ERROR org.apache.pig.Main - ERROR 2998: Unhandled >>>> > internal >>>> > >> >> error. >>>> > >> >> >> > org/apache/tez/common/counters/DAGCounter >>>> > >> >> >> > >>>> > >> >> >> > Any reply will be appreciated! >>>> > >> >> >> > >>>> > >> >> >> > Regards! >>>> > >> >> >> > >>>> > >> >> >> > Chen >>>> > >> >> >> >>>> > >> >> >> -- >>>> > >> >> >> CONFIDENTIALITY NOTICE >>>> > >> >> >> NOTICE: This message is intended for the use of the >>>> individual or >>>> > >> >> entity to >>>> > >> >> >> which it is addressed and may contain information that is >>>> > >> confidential, >>>> > >> >> >> privileged and exempt from disclosure under applicable law. >>>> If the >>>> > >> >> reader >>>> > >> >> >> of this message is not the intended recipient, you are hereby >>>> > >> notified >>>> > >> >> that >>>> > >> >> >> any printing, copying, dissemination, distribution, >>>> disclosure or >>>> > >> >> >> forwarding of this communication is strictly prohibited. If >>>> you >>>> > have >>>> > >> >> >> received this communication in error, please contact the >>>> sender >>>> > >> >> immediately >>>> > >> >> >> and delete it from your system. Thank You. >>>> > >> >> >> >>>> > >> >> >>>> > >> >> -- >>>> > >> >> CONFIDENTIALITY NOTICE >>>> > >> >> NOTICE: This message is intended for the use of the individual or >>>> > >> entity to >>>> > >> >> which it is addressed and may contain information that is >>>> > confidential, >>>> > >> >> privileged and exempt from disclosure under applicable law. If >>>> the >>>> > >> reader >>>> > >> >> of this message is not the intended recipient, you are hereby >>>> > notified >>>> > >> that >>>> > >> >> any printing, copying, dissemination, distribution, disclosure or >>>> > >> >> forwarding of this communication is strictly prohibited. If you >>>> have >>>> > >> >> received this communication in error, please contact the sender >>>> > >> immediately >>>> > >> >> and delete it from your system. Thank You. >>>> > >> >> >>>> > >> >>>> > >> -- >>>> > >> CONFIDENTIALITY NOTICE >>>> > >> NOTICE: This message is intended for the use of the individual or >>>> > entity to >>>> > >> which it is addressed and may contain information that is >>>> confidential, >>>> > >> privileged and exempt from disclosure under applicable law. If the >>>> > reader >>>> > >> of this message is not the intended recipient, you are hereby >>>> notified >>>> > that >>>> > >> any printing, copying, dissemination, distribution, disclosure or >>>> > >> forwarding of this communication is strictly prohibited. If you have >>>> > >> received this communication in error, please contact the sender >>>> > immediately >>>> > >> and delete it from your system. Thank You. >>>> > >> >>>> > >>>> > -- >>>> > CONFIDENTIALITY NOTICE >>>> > NOTICE: This message is intended for the use of the individual or >>>> entity to >>>> > which it is addressed and may contain information that is confidential, >>>> > privileged and exempt from disclosure under applicable law. If the >>>> reader >>>> > of this message is not the intended recipient, you are hereby notified >>>> that >>>> > any printing, copying, dissemination, distribution, disclosure or >>>> > forwarding of this communication is strictly prohibited. If you have >>>> > received this communication in error, please contact the sender >>>> immediately >>>> > and delete it from your system. Thank You. >>>> > >>>> >>> >>> >> -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
