We are running pig tutorial example and get following errors. I think we
are pick up some hadoop config that specify HDFS instead of localFS.
Here are the errors from pig.log
Pig Stack Trace
---------------
ERROR 6007: Unable to check name hdfs://localhost:9000/user/chenhe
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error
during parsing. Unable to check name hdfs://localhost:9000/user/chenhe
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1736)
at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1431)
at org.apache.pig.PigServer.parseAndBuild(PigServer.java:375)
at org.apache.pig.PigServer.executeBatch(PigServer.java:400)
at org.apache.pig.PigServer.executeBatch(PigServer.java:386)
at
org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:170)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:232)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:203)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
at org.apache.pig.Main.run(Main.java:608)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: Failed to parse: Pig script failed to parse:
<file ../../../../tutorial/scripts/script1-local.pig, line 28, column 6>
pig script failed to validate:
org.apache.pig.backend.datastorage.DataStorageException: ERROR 6007: Unable
to check name hdfs://localhost:9000/user/chenhe
at
org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:196)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1723)
... 15 more
Caused by:
<file ../../../../tutorial/scripts/script1-local.pig, line 28, column 6>
pig script failed to validate:
org.apache.pig.backend.datastorage.DataStorageException: ERROR 6007: Unable
to check name hdfs://localhost:9000/user/chenhe
at
org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:881)
at
org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568)
at
org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625)
at
org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
at
org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
at
org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
at
org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
... 16 more
Caused by: org.apache.pig.backend.datastorage.DataStorageException: ERROR
6007: Unable to check name hdfs://localhost:9000/user/chenhe
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.isContainer(HDataStorage.java:207)
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:128)
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:138)
at
org.apache.pig.parser.QueryParserUtils.getCurrentDir(QueryParserUtils.java:90)
at
org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:873)
... 22 more
Caused by: java.net.ConnectException: Call From
guardwritten-dl.corp.yahoo.com/10.74.90.55 to localhost:9000 failed on
connection exception: java.net.ConnectException: Connection refused; For
more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:700)
at org.apache.hadoop.ipc.Client.call(Client.java:1098)
at
org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:195)
at com.sun.proxy.$Proxy6.getFileInfo(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:102)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:67)
at com.sun.proxy.$Proxy6.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1305)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:734)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1272)
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.isContainer(HDataStorage.java:200)
... 26 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:462)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:559)
at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:207)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1204)
at org.apache.hadoop.ipc.Client.call(Client.java:1074)
... 39 more
================================================================================
On Tue, May 27, 2014 at 11:04 AM, Chen He <[email protected]> wrote:
> Hi Daniel
>
> Thank you for your reply. Any suggestion about how can I run pig local
> mode using Tez without minicluster? I am sure the code on Tez side is
> ready. What changes need to be made on PIG? I am not very familar with PIG.
>
> Regards!
>
> Chen
>
>
> On Mon, May 26, 2014 at 8:22 PM, Chen He <[email protected]> wrote:
>
>> Hi Jon, Daniel
>>
>> Daniel was right. I tried pig local mode using "pig -x local -x tez …..".
>> It tries to connect mini cluster and finally timeout. The error that I
>> mentioned before : "ERROR 2998: Unhandled internal error. Found
>> interface org.apache.hadoop.yarn.api.records.LocalResource, but class
>> was expected"
>> is because I have a hadoop 0.23 mini cluster running in background.
>>
>> Chen
>>
>>
>> On Mon, May 26, 2014 at 9:39 AM, Jonathan Eagles <[email protected]>wrote:
>>
>>> Hi Daniel,
>>>
>>> You've been very helpful in your replies. I believe the missing context
>>> is
>>> that Chen is working on adding Tez Local Mode (TEZ-684) and is trying to
>>> integrate with Pig. The question as I interpret it is whether Pig needs
>>> or
>>> creates a MiniCluster in local mode. Chen can you confirm this?
>>>
>>> Jon
>>>
>>>
>>> On Mon, May 26, 2014 at 2:55 AM, Daniel Dai <[email protected]>
>>> wrote:
>>>
>>> > Hi, Chen,
>>> >
>>> > Please note Pig local mode does not run tez. It uses MR engine.
>>> > Tez-local mode is not implemented (depend on TEZ-235). You will need a
>>> > real cluster or use TezMiniCluster to test Pig on Tez.
>>> >
>>> > Thanks,
>>> > Daniel
>>> >
>>> > On Sun, May 25, 2014 at 11:42 AM, Chen He <[email protected]> wrote:
>>> > > Hi Daniel
>>> > >
>>> > > I was testing the pig local mode. Does pig need mini cluster when it
>>> runs
>>> > > in local mode?
>>> > >
>>> > > Regards!
>>> > >
>>> > > Chen
>>> > >
>>> > >
>>> > > On Sun, May 25, 2014 at 1:40 PM, Daniel Dai <[email protected]>
>>> > wrote:
>>> > >
>>> > >> Which Hadoop are you running against? Also 2.2?
>>> > >>
>>> > >> On Sun, May 25, 2014 at 6:52 AM, Chen He <[email protected]> wrote:
>>> > >> > Hi Daniel
>>> > >> >
>>> > >> > Thank you for the reply. I compiled Tez and Pig against hadoop
>>> 2.2.
>>> > >> >
>>> > >> > Regards!
>>> > >> >
>>> > >> > Chen
>>> > >> >
>>> > >> >
>>> > >> > On Thu, May 22, 2014 at 5:22 PM, Daniel Dai <
>>> [email protected]>
>>> > >> wrote:
>>> > >> >
>>> > >> >> Which version of Hadoop are you using? Seem LocalResource changed
>>> > from
>>> > >> >> interface to class in 2.1 (YARN-711). If you are using pre-2.1,
>>> this
>>> > >> >> might be the case.
>>> > >> >>
>>> > >> >> Thanks,
>>> > >> >> Daniel
>>> > >> >>
>>> > >> >> On Thu, May 22, 2014 at 3:06 PM, Chen He <[email protected]>
>>> wrote:
>>> > >> >> > Hi Daniel
>>> > >> >> >
>>> > >> >> > Thank you for the reply. I did as you suggested but get
>>> following
>>> > >> errors:
>>> > >> >> >
>>> > >> >> > ERROR 2998: Unhandled internal error. Found interface
>>> > >> >> > org.apache.hadoop.yarn.api.records.LocalResource, but class was
>>> > >> expected
>>> > >> >> >
>>> > >> >> > java.lang.IncompatibleClassChangeError: Found interface
>>> > >> >> > org.apache.hadoop.yarn.api.records.LocalResource, but class was
>>> > >> expected
>>> > >> >> > at
>>> > >> >> >
>>> > >> >>
>>> > >>
>>> >
>>> org.apache.pig.backend.hadoop.executionengine.tez.TezResourceManager.getTezResources(TezResourceManager.java:124)
>>> > >> >> > at
>>> > >> >> >
>>> > >> >>
>>> > >>
>>> >
>>> org.apache.pig.backend.hadoop.executionengine.tez.TezResourceManager.addTezResources(TezResourceManager.java:96)
>>> > >> >> > at
>>> > >> >> >
>>> > >> >>
>>> > >>
>>> >
>>> org.apache.pig.backend.hadoop.executionengine.tez.TezPlanContainer.getLocalResources(TezPlanContainer.java:104)
>>> > >> >> > at
>>> > >> >> >
>>> > >> >>
>>> > >>
>>> >
>>> org.apache.pig.backend.hadoop.executionengine.tez.TezJobControlCompiler.getJob(TezJobControlCompiler.java:106)
>>> > >> >> > at
>>> > >> >> >
>>> > >> >>
>>> > >>
>>> >
>>> org.apache.pig.backend.hadoop.executionengine.tez.TezJobControlCompiler.compile(TezJobControlCompiler.java:89)
>>> > >> >> > at
>>> > >> >> >
>>> > >> >>
>>> > >>
>>> >
>>> org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher.launchPig(TezLauncher.java:92)
>>> > >> >> > at
>>> > >> >> >
>>> > >> >>
>>> > >>
>>> >
>>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:277)
>>> > >> >> > at org.apache.pig.PigServer.launchPlan(PigServer.java:1378)
>>> > >> >> > at
>>> > >> >> >
>>> > >>
>>> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1363)
>>> > >> >> > at org.apache.pig.PigServer.execute(PigServer.java:1352)
>>> > >> >> > at
>>> org.apache.pig.PigServer.executeBatch(PigServer.java:403)
>>> > >> >> > at
>>> org.apache.pig.PigServer.executeBatch(PigServer.java:386)
>>> > >> >> > at
>>> > >> >> >
>>> > >>
>>> >
>>> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:170)
>>> > >> >> > at
>>> > >> >> >
>>> > >> >>
>>> > >>
>>> >
>>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:232)
>>> > >> >> > at
>>> > >> >> >
>>> > >> >>
>>> > >>
>>> >
>>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:203)
>>> > >> >> > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
>>> > >> >> > at org.apache.pig.Main.run(Main.java:608)
>>> > >> >> > at org.apache.pig.Main.main(Main.java:156)
>>> > >> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>> Method)
>>> > >> >> > at
>>> > >> >> >
>>> > >> >>
>>> > >>
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>> > >> >> > at
>>> > >> >> >
>>> > >> >>
>>> > >>
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> > >> >> > at java.lang.reflect.Method.invoke(Method.java:606)
>>> > >> >> > at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>>> > >> >> >
>>> > >> >> > Is this a problem that caused by hadoop incompatiblilty? Which
>>> > >> version of
>>> > >> >> > Hadoop is the best?
>>> > >> >> >
>>> > >> >> > Regards!
>>> > >> >> >
>>> > >> >> > Chen
>>> > >> >> >
>>> > >> >> >
>>> > >> >> > On Thu, May 22, 2014 at 4:19 PM, Daniel Dai <
>>> [email protected]
>>> > >
>>> > >> >> wrote:
>>> > >> >> >
>>> > >> >> >> Need to set PIG_CLASSPATH:
>>> > >> >> >> export
>>> PIG_CLASSPATH=$TEZ_HOME/*:$TEZ_HOME/lib/*:$TEZ_HOME/conf
>>> > >> >> >>
>>> > >> >> >> Thanks,
>>> > >> >> >> Daniel
>>> > >> >> >>
>>> > >> >> >> On Thu, May 22, 2014 at 2:11 PM, Chen He <[email protected]>
>>> > wrote:
>>> > >> >> >> > Any one met this error before?
>>> > >> >> >> >
>>> > >> >> >> > [main] ERROR org.apache.pig.Main - ERROR 2998: Unhandled
>>> > internal
>>> > >> >> error.
>>> > >> >> >> > org/apache/tez/common/counters/DAGCounter
>>> > >> >> >> >
>>> > >> >> >> > Any reply will be appreciated!
>>> > >> >> >> >
>>> > >> >> >> > Regards!
>>> > >> >> >> >
>>> > >> >> >> > Chen
>>> > >> >> >>
>>> > >> >> >> --
>>> > >> >> >> CONFIDENTIALITY NOTICE
>>> > >> >> >> NOTICE: This message is intended for the use of the
>>> individual or
>>> > >> >> entity to
>>> > >> >> >> which it is addressed and may contain information that is
>>> > >> confidential,
>>> > >> >> >> privileged and exempt from disclosure under applicable law.
>>> If the
>>> > >> >> reader
>>> > >> >> >> of this message is not the intended recipient, you are hereby
>>> > >> notified
>>> > >> >> that
>>> > >> >> >> any printing, copying, dissemination, distribution,
>>> disclosure or
>>> > >> >> >> forwarding of this communication is strictly prohibited. If
>>> you
>>> > have
>>> > >> >> >> received this communication in error, please contact the
>>> sender
>>> > >> >> immediately
>>> > >> >> >> and delete it from your system. Thank You.
>>> > >> >> >>
>>> > >> >>
>>> > >> >> --
>>> > >> >> CONFIDENTIALITY NOTICE
>>> > >> >> NOTICE: This message is intended for the use of the individual or
>>> > >> entity to
>>> > >> >> which it is addressed and may contain information that is
>>> > confidential,
>>> > >> >> privileged and exempt from disclosure under applicable law. If
>>> the
>>> > >> reader
>>> > >> >> of this message is not the intended recipient, you are hereby
>>> > notified
>>> > >> that
>>> > >> >> any printing, copying, dissemination, distribution, disclosure or
>>> > >> >> forwarding of this communication is strictly prohibited. If you
>>> have
>>> > >> >> received this communication in error, please contact the sender
>>> > >> immediately
>>> > >> >> and delete it from your system. Thank You.
>>> > >> >>
>>> > >>
>>> > >> --
>>> > >> CONFIDENTIALITY NOTICE
>>> > >> NOTICE: This message is intended for the use of the individual or
>>> > entity to
>>> > >> which it is addressed and may contain information that is
>>> confidential,
>>> > >> privileged and exempt from disclosure under applicable law. If the
>>> > reader
>>> > >> of this message is not the intended recipient, you are hereby
>>> notified
>>> > that
>>> > >> any printing, copying, dissemination, distribution, disclosure or
>>> > >> forwarding of this communication is strictly prohibited. If you have
>>> > >> received this communication in error, please contact the sender
>>> > immediately
>>> > >> and delete it from your system. Thank You.
>>> > >>
>>> >
>>> > --
>>> > CONFIDENTIALITY NOTICE
>>> > NOTICE: This message is intended for the use of the individual or
>>> entity to
>>> > which it is addressed and may contain information that is confidential,
>>> > privileged and exempt from disclosure under applicable law. If the
>>> reader
>>> > of this message is not the intended recipient, you are hereby notified
>>> that
>>> > any printing, copying, dissemination, distribution, disclosure or
>>> > forwarding of this communication is strictly prohibited. If you have
>>> > received this communication in error, please contact the sender
>>> immediately
>>> > and delete it from your system. Thank You.
>>> >
>>>
>>
>>
>
