Help with pigsetup
I sent this to pig apache user mailing list but have got no response. Not sure if that list is still active. thought I will post here if someone is able to help me. I am in process of installing and learning pig. I have a hadoop cluster and when I try to run pig in mapreduce mode it errors out: Hadoop version is hadoop-0.20.203.0 and pig version is pig-0.8.1 Error before Pig is launched ERROR 2999: Unexpected internal error. Failed to create DataStorage java.lang.RuntimeException: Failed to create DataStorage at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75) at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:58) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:214) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134) at org.apache.pig.impl.PigContext.connect(PigContext.java:183) at org.apache.pig.PigServer.init(PigServer.java:226) at org.apache.pig.PigServer.init(PigServer.java:215) at org.apache.pig.tools.grunt.Grunt.init(Grunt.java:55) at org.apache.pig.Main.run(Main.java:452) at org.apache.pig.Main.main(Main.java:107) Caused by: java.io.IOException: Call to dsdb1/172.18.60.96:54310 failed on local exception: java.io.EOFException at org.apache.hadoop.ipc.Client.wrapException(Client.java:775) at org.apache.hadoop.ipc.Client.call(Client.java:743) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) at $Proxy0.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359) at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:207) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:170) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95) at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72) ... 9 more Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
Re: Help with pigsetup
I think Jonathan Coveney's reply on user@pig answered your question. Its basically an issue of hadoop version differences between the one Pig 0.8.1 release got bundled with vs. Hadoop 0.20.203 release which is newer. On Thu, May 26, 2011 at 10:26 PM, Mohit Anchlia mohitanch...@gmail.com wrote: I sent this to pig apache user mailing list but have got no response. Not sure if that list is still active. thought I will post here if someone is able to help me. I am in process of installing and learning pig. I have a hadoop cluster and when I try to run pig in mapreduce mode it errors out: Hadoop version is hadoop-0.20.203.0 and pig version is pig-0.8.1 Error before Pig is launched ERROR 2999: Unexpected internal error. Failed to create DataStorage java.lang.RuntimeException: Failed to create DataStorage at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75) at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:58) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:214) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134) at org.apache.pig.impl.PigContext.connect(PigContext.java:183) at org.apache.pig.PigServer.init(PigServer.java:226) at org.apache.pig.PigServer.init(PigServer.java:215) at org.apache.pig.tools.grunt.Grunt.init(Grunt.java:55) at org.apache.pig.Main.run(Main.java:452) at org.apache.pig.Main.main(Main.java:107) Caused by: java.io.IOException: Call to dsdb1/172.18.60.96:54310 failed on local exception: java.io.EOFException at org.apache.hadoop.ipc.Client.wrapException(Client.java:775) at org.apache.hadoop.ipc.Client.call(Client.java:743) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) at $Proxy0.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359) at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:207) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:170) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95) at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72) ... 9 more Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446) -- Harsh J
Re: Help with pigsetup
For some reason I don't see that reply from Jonathan in my Inbox. I'll try to google it. What should be my next step in that case? I can't use pig then? On Thu, May 26, 2011 at 10:00 AM, Harsh J ha...@cloudera.com wrote: I think Jonathan Coveney's reply on user@pig answered your question. Its basically an issue of hadoop version differences between the one Pig 0.8.1 release got bundled with vs. Hadoop 0.20.203 release which is newer. On Thu, May 26, 2011 at 10:26 PM, Mohit Anchlia mohitanch...@gmail.com wrote: I sent this to pig apache user mailing list but have got no response. Not sure if that list is still active. thought I will post here if someone is able to help me. I am in process of installing and learning pig. I have a hadoop cluster and when I try to run pig in mapreduce mode it errors out: Hadoop version is hadoop-0.20.203.0 and pig version is pig-0.8.1 Error before Pig is launched ERROR 2999: Unexpected internal error. Failed to create DataStorage java.lang.RuntimeException: Failed to create DataStorage at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75) at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:58) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:214) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134) at org.apache.pig.impl.PigContext.connect(PigContext.java:183) at org.apache.pig.PigServer.init(PigServer.java:226) at org.apache.pig.PigServer.init(PigServer.java:215) at org.apache.pig.tools.grunt.Grunt.init(Grunt.java:55) at org.apache.pig.Main.run(Main.java:452) at org.apache.pig.Main.main(Main.java:107) Caused by: java.io.IOException: Call to dsdb1/172.18.60.96:54310 failed on local exception: java.io.EOFException at org.apache.hadoop.ipc.Client.wrapException(Client.java:775) at org.apache.hadoop.ipc.Client.call(Client.java:743) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) at $Proxy0.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359) at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:207) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:170) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95) at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72) ... 9 more Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446) -- Harsh J
Re: Help with pigsetup
I'll repost it here then :) Here is what I had to do to get pig running with a different version of Hadoop (in my case, the cloudera build but I'd try this as well): build pig-withouthadoop.jar by running ant jar-withouthadoop. Then, when you run pig, put the pig-withouthadoop.jar on your classpath as well as your hadoop jar. In my case, I found that scripts only worked if I additionally manually registered the antlr jar: register /path/to/pig/build/ivy/lib/Pig/antlr-runtime-3.2.jar; 2011/5/26 Mohit Anchlia mohitanch...@gmail.com For some reason I don't see that reply from Jonathan in my Inbox. I'll try to google it. What should be my next step in that case? I can't use pig then? On Thu, May 26, 2011 at 10:00 AM, Harsh J ha...@cloudera.com wrote: I think Jonathan Coveney's reply on user@pig answered your question. Its basically an issue of hadoop version differences between the one Pig 0.8.1 release got bundled with vs. Hadoop 0.20.203 release which is newer. On Thu, May 26, 2011 at 10:26 PM, Mohit Anchlia mohitanch...@gmail.com wrote: I sent this to pig apache user mailing list but have got no response. Not sure if that list is still active. thought I will post here if someone is able to help me. I am in process of installing and learning pig. I have a hadoop cluster and when I try to run pig in mapreduce mode it errors out: Hadoop version is hadoop-0.20.203.0 and pig version is pig-0.8.1 Error before Pig is launched ERROR 2999: Unexpected internal error. Failed to create DataStorage java.lang.RuntimeException: Failed to create DataStorage at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75) at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:58) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:214) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134) at org.apache.pig.impl.PigContext.connect(PigContext.java:183) at org.apache.pig.PigServer.init(PigServer.java:226) at org.apache.pig.PigServer.init(PigServer.java:215) at org.apache.pig.tools.grunt.Grunt.init(Grunt.java:55) at org.apache.pig.Main.run(Main.java:452) at org.apache.pig.Main.main(Main.java:107) Caused by: java.io.IOException: Call to dsdb1/172.18.60.96:54310 failed on local exception: java.io.EOFException at org.apache.hadoop.ipc.Client.wrapException(Client.java:775) at org.apache.hadoop.ipc.Client.call(Client.java:743) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) at $Proxy0.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359) at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:207) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:170) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95) at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72) ... 9 more Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446) -- Harsh J
Re: Help with pigsetup
On Thu, May 26, 2011 at 10:06 AM, Jonathan Coveney jcove...@gmail.com wrote: I'll repost it here then :) Here is what I had to do to get pig running with a different version of Hadoop (in my case, the cloudera build but I'd try this as well): build pig-withouthadoop.jar by running ant jar-withouthadoop. Then, when you run pig, put the pig-withouthadoop.jar on your classpath as well as your hadoop jar. In my case, I found that scripts only worked if I additionally manually registered the antlr jar: Thanks Jonathan! I will give it a shot. register /path/to/pig/build/ivy/lib/Pig/antlr-runtime-3.2.jar; Is this a windows command? Sorry, have not used this before. 2011/5/26 Mohit Anchlia mohitanch...@gmail.com For some reason I don't see that reply from Jonathan in my Inbox. I'll try to google it. What should be my next step in that case? I can't use pig then? On Thu, May 26, 2011 at 10:00 AM, Harsh J ha...@cloudera.com wrote: I think Jonathan Coveney's reply on user@pig answered your question. Its basically an issue of hadoop version differences between the one Pig 0.8.1 release got bundled with vs. Hadoop 0.20.203 release which is newer. On Thu, May 26, 2011 at 10:26 PM, Mohit Anchlia mohitanch...@gmail.com wrote: I sent this to pig apache user mailing list but have got no response. Not sure if that list is still active. thought I will post here if someone is able to help me. I am in process of installing and learning pig. I have a hadoop cluster and when I try to run pig in mapreduce mode it errors out: Hadoop version is hadoop-0.20.203.0 and pig version is pig-0.8.1 Error before Pig is launched ERROR 2999: Unexpected internal error. Failed to create DataStorage java.lang.RuntimeException: Failed to create DataStorage at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75) at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:58) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:214) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134) at org.apache.pig.impl.PigContext.connect(PigContext.java:183) at org.apache.pig.PigServer.init(PigServer.java:226) at org.apache.pig.PigServer.init(PigServer.java:215) at org.apache.pig.tools.grunt.Grunt.init(Grunt.java:55) at org.apache.pig.Main.run(Main.java:452) at org.apache.pig.Main.main(Main.java:107) Caused by: java.io.IOException: Call to dsdb1/172.18.60.96:54310 failed on local exception: java.io.EOFException at org.apache.hadoop.ipc.Client.wrapException(Client.java:775) at org.apache.hadoop.ipc.Client.call(Client.java:743) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) at $Proxy0.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359) at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:207) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:170) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95) at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72) ... 9 more Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446) -- Harsh J
Re: Help with pigsetup
I've built pig-withouthadoop.jar and have copied it to my linux box. Now how do I put hadoop-core-0.20.203.0.jar and pig-withouthadoop.jar in the classpath. Is it by using CLASSPATH variable? On Thu, May 26, 2011 at 10:18 AM, Mohit Anchlia mohitanch...@gmail.com wrote: On Thu, May 26, 2011 at 10:06 AM, Jonathan Coveney jcove...@gmail.com wrote: I'll repost it here then :) Here is what I had to do to get pig running with a different version of Hadoop (in my case, the cloudera build but I'd try this as well): build pig-withouthadoop.jar by running ant jar-withouthadoop. Then, when you run pig, put the pig-withouthadoop.jar on your classpath as well as your hadoop jar. In my case, I found that scripts only worked if I additionally manually registered the antlr jar: Thanks Jonathan! I will give it a shot. register /path/to/pig/build/ivy/lib/Pig/antlr-runtime-3.2.jar; Is this a windows command? Sorry, have not used this before. 2011/5/26 Mohit Anchlia mohitanch...@gmail.com For some reason I don't see that reply from Jonathan in my Inbox. I'll try to google it. What should be my next step in that case? I can't use pig then? On Thu, May 26, 2011 at 10:00 AM, Harsh J ha...@cloudera.com wrote: I think Jonathan Coveney's reply on user@pig answered your question. Its basically an issue of hadoop version differences between the one Pig 0.8.1 release got bundled with vs. Hadoop 0.20.203 release which is newer. On Thu, May 26, 2011 at 10:26 PM, Mohit Anchlia mohitanch...@gmail.com wrote: I sent this to pig apache user mailing list but have got no response. Not sure if that list is still active. thought I will post here if someone is able to help me. I am in process of installing and learning pig. I have a hadoop cluster and when I try to run pig in mapreduce mode it errors out: Hadoop version is hadoop-0.20.203.0 and pig version is pig-0.8.1 Error before Pig is launched ERROR 2999: Unexpected internal error. Failed to create DataStorage java.lang.RuntimeException: Failed to create DataStorage at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75) at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:58) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:214) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134) at org.apache.pig.impl.PigContext.connect(PigContext.java:183) at org.apache.pig.PigServer.init(PigServer.java:226) at org.apache.pig.PigServer.init(PigServer.java:215) at org.apache.pig.tools.grunt.Grunt.init(Grunt.java:55) at org.apache.pig.Main.run(Main.java:452) at org.apache.pig.Main.main(Main.java:107) Caused by: java.io.IOException: Call to dsdb1/172.18.60.96:54310 failed on local exception: java.io.EOFException at org.apache.hadoop.ipc.Client.wrapException(Client.java:775) at org.apache.hadoop.ipc.Client.call(Client.java:743) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) at $Proxy0.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359) at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:207) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:170) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95) at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72) ... 9 more Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446) -- Harsh J
Re: Help with pigsetup
I added to PIG_CLASSPATH and went past the error but now I get a different error. Looks like I need to add some other jars but not sure which one. export PIG_CLASSPATH=$HADOOP_CONF_DIR:$HADOOP_HOME/hadoop-core-0.20.203.0.jar:$PIG_HOME/../pig-withouthadoop.jar ERROR 2998: Unhandled internal error. org/apache/commons/configuration/Configuration java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:37) at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.clinit(DefaultMetricsSystem.java:34) at org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:196) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159) at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:409) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:395) at org.apache.hadoop.fs.FileSystem$Cache$Key.init(FileSystem.java:1418) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1319) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:109) at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72) at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:58) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:196) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:116) at org.apache.pig.impl.PigContext.connect(PigContext.java:187) at org.apache.pig.PigServer.init(PigServer.java:243) at org.apache.pig.PigServer.init(PigServer.java:228) at org.apache.pig.tools.grunt.Grunt.init(Grunt.java:46) at org.apache.pig.Main.run(Main.java:484) at org.apache.pig.Main.main(Main.java:108) Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.Configuration at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) On Thu, May 26, 2011 at 10:55 AM, Mohit Anchlia mohitanch...@gmail.com wrote: I've built pig-withouthadoop.jar and have copied it to my linux box. Now how do I put hadoop-core-0.20.203.0.jar and pig-withouthadoop.jar in the classpath. Is it by using CLASSPATH variable? On Thu, May 26, 2011 at 10:18 AM, Mohit Anchlia mohitanch...@gmail.com wrote: On Thu, May 26, 2011 at 10:06 AM, Jonathan Coveney jcove...@gmail.com wrote: I'll repost it here then :) Here is what I had to do to get pig running with a different version of Hadoop (in my case, the cloudera build but I'd try this as well): build pig-withouthadoop.jar by running ant jar-withouthadoop. Then, when you run pig, put the pig-withouthadoop.jar on your classpath as well as your hadoop jar. In my case, I found that scripts only worked if I additionally manually registered the antlr jar: Thanks Jonathan! I will give it a shot. register /path/to/pig/build/ivy/lib/Pig/antlr-runtime-3.2.jar; Is this a windows command? Sorry, have not used this before. 2011/5/26 Mohit Anchlia mohitanch...@gmail.com For some reason I don't see that reply from Jonathan in my Inbox. I'll try to google it. What should be my next step in that case? I can't use pig then? On Thu, May 26, 2011 at 10:00 AM, Harsh J ha...@cloudera.com wrote: I think Jonathan Coveney's reply on user@pig answered your question. Its basically an issue of hadoop version differences between the one Pig 0.8.1 release got bundled with vs. Hadoop 0.20.203 release which is newer. On Thu, May 26, 2011 at 10:26 PM, Mohit Anchlia mohitanch...@gmail.com wrote: I sent this to pig apache user mailing list but have got no response. Not sure if that list is still active. thought I will post here if someone is able to help me. I am in process of installing and learning pig. I have a hadoop cluster and when I try to run pig in mapreduce mode it errors out: Hadoop version is hadoop-0.20.203.0 and pig version is pig-0.8.1 Error before Pig is launched ERROR 2999: Unexpected internal error. Failed to create DataStorage java.lang.RuntimeException: Failed to create
Re: Help with pigsetup
I added all the jars in the classpath in HADOOP_HOME/lib and now I get to the grunt prompt. Will try the tutorials and see how it behaves :) Thanks for your help! On Thu, May 26, 2011 at 9:56 AM, Mohit Anchlia mohitanch...@gmail.com wrote: I sent this to pig apache user mailing list but have got no response. Not sure if that list is still active. thought I will post here if someone is able to help me. I am in process of installing and learning pig. I have a hadoop cluster and when I try to run pig in mapreduce mode it errors out: Hadoop version is hadoop-0.20.203.0 and pig version is pig-0.8.1 Error before Pig is launched ERROR 2999: Unexpected internal error. Failed to create DataStorage java.lang.RuntimeException: Failed to create DataStorage at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75) at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:58) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:214) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134) at org.apache.pig.impl.PigContext.connect(PigContext.java:183) at org.apache.pig.PigServer.init(PigServer.java:226) at org.apache.pig.PigServer.init(PigServer.java:215) at org.apache.pig.tools.grunt.Grunt.init(Grunt.java:55) at org.apache.pig.Main.run(Main.java:452) at org.apache.pig.Main.main(Main.java:107) Caused by: java.io.IOException: Call to dsdb1/172.18.60.96:54310 failed on local exception: java.io.EOFException at org.apache.hadoop.ipc.Client.wrapException(Client.java:775) at org.apache.hadoop.ipc.Client.call(Client.java:743) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) at $Proxy0.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359) at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:207) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:170) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95) at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72) ... 9 more Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
