"Failed to create DataStorage" is due to incompatible hadoop jars. Check https://cwiki.apache.org/confluence/display/PIG/FAQ#FAQ-Q%3AWhatshallIdoifIsaw%22FailedtocreateDataStorage%22%3F
Daniel On Thu, Sep 8, 2011 at 11:20 PM, kiranprasad <kiranprasa...@imimobile.com>wrote: > Hi > > For Cluster I have 3 VMs(10.0.0.61-master, 10.0.0.62,10.0.0.63 - Slaves) > and another VM 10.0.0.64 in which I ve installed PIG from which I run the > PIG commands. > Hadoop version : hadoop-0.20.2-cdh3u0 and PIG : pig-0.8.1 > I ve updated the xmls , please find the below > > mapred site.xml > -------------- > <configuration> > <property> > <name>mapred.job.tracker</**name> > <value>10.0.0.61:8021</value> > </property> > </configuration> > > > core-site.xml > ---------- > <configuration> > <property> > <name>fs.default.name</name> > <value>hdfs://10.0.0.61:8020</**value> > > </property> > </configuration> > > Hdfs-site.xml > ---------------- > <configuration> > <property> > <name>fs.default.name</name> > <value>hdfs://10.0.0.61:8020</**value> > </property> > <property> > <name>mapred.job.tracker</**name> > <value>10.0.0.61:8021</value> > > </property> > <property> > <name>dfs.replication</name> > <value>2</value> > </property> > > masters > --------- > 10.0.0.61 > > slaves > -------- > > 10.0.0.62 > 10.0.0.63 > > Have started start-all.sh. and checked with jps command in > master(10.0.0.61) I got response as > > 4313 Jps > 3357 NameNode > 3580 JobTracker > 3502 SecondaryNameNode > > and in slave I got > > 2662 DataNode > 2902 Jps > 2759 TaskTracker > > So the Cluster is working... > > But When I tried to connect to PIG i.e. bin/pig (I ve installed PIG in > 10.0.0.64 this is out of Hadoop Cluster). > I am getting below exception again. Please help close this. > > > [kiranprasad.g@pig1 pig-0.8.1]$ bin/pig > 2011-09-09 17:13:29,507 [main] INFO org.apache.pig.Main - Logging error > messages to: /home/kiranprasad.g/pig-0.8.1/**pig_1315568609504.log > 2011-09-09 17:13:29,676 [main] INFO org.apache.pig.backend.hadoop.** > executionengine.**HExecutionEngine - Connecting to hadoop file system at: > hdfs://10.0.0.61:8020 > 2011-09-09 17:13:29,804 [main] ERROR org.apache.pig.Main - ERROR 2999: > Unexpected internal error. Failed to create DataStorage > > > Regards > Kiran.G > -----Original Message----- From: Daniel Dai > Sent: Friday, September 09, 2011 2:44 AM > > To: user@pig.apache.org > Subject: Re: ERROR 2999: Unexpected internal error. Failed to create > DataStorage > > It try to connect to 8020 port and hadoop is not there. Why you have > different port in different config files? > > Daniel > > On Thu, Sep 8, 2011 at 2:07 AM, kiranprasad <kiranprasa...@imimobile.com>* > *wrote: > > Hi >> Iam getting the below exception when I do hadoop fs -ls / >> >> Most commands print help when invoked w/o parameters. >> [kiranprasad.g@pig1 hadoop-0.20.2-cdh3u0]$ bin/hadoop fs -ls/ >> 11/09/12 00:06:54 INFO ipc.Client: Retrying connect to server: / >> 10.0.0.61:8020. Already tried 0 time(s). >> 11/09/12 00:06:56 INFO ipc.Client: Retrying connect to server: / >> 10.0.0.61:8020. Already tried 1 time(s). >> 11/09/12 00:06:57 INFO ipc.Client: Retrying connect to server: / >> 10.0.0.61:8020. Already tried 2 time(s). >> 11/09/12 00:06:58 INFO ipc.Client: Retrying connect to server: / >> 10.0.0.61:8020. Already tried 3 time(s). >> 11/09/12 00:07:00 INFO ipc.Client: Retrying connect to server: / >> 10.0.0.61:8020. Already tried 4 time(s). >> 11/09/12 00:07:01 INFO ipc.Client: Retrying connect to server: / >> 10.0.0.61:8020. Already tried 5 time(s). >> 11/09/12 00:07:02 INFO ipc.Client: Retrying connect to server: / >> 10.0.0.61:8020. Already tried 6 time(s). >> 11/09/12 00:07:04 INFO ipc.Client: Retrying connect to server: / >> 10.0.0.61:8020. Already tried 7 time(s). >> 11/09/12 00:07:05 INFO ipc.Client: Retrying connect to server: / >> 10.0.0.61:8020. Already tried 8 time(s). >> 11/09/12 00:07:06 INFO ipc.Client: Retrying connect to server: / >> 10.0.0.61:8020. Already tried 9 time(s). >> Bad connection to FS. command aborted. exception: Call to >> /10.0.0.61:8020failed >> on connection exception: java.net.ConnectException: Connection >> >> refused >> >> >> Regards >> Kiran.G >> >> IMImobile Plot 770, Rd. 44 Jubilee Hills, Hyderabad - 500033 >> M +91 9000170909 T +91 40 2355 5945 - Ext: 229 www.imimobile.com >> -----Original Message----- From: Ashutosh Chauhan >> Sent: Thursday, September 08, 2011 11:41 AM >> >> To: user@pig.apache.org >> Subject: Re: ERROR 2999: Unexpected internal error. Failed to create >> DataStorage >> >> Are you able to run >> hadoop fs -ls / >> if so, is your HADOOP_HOME and HADOOP_CONF_DIR environment variables set >> correctly? >> >> Ashutosh >> >> On Wed, Sep 7, 2011 at 21:58, kiranprasad <kiranprasa...@imimobile.com>** >> ** >> wrote: >> >> Even after updating the core-site.xml file I am getting the same >> >>> exception, >>> Please help. >>> >>> >>> Regards >>> Kiran.G >>> >>> IMImobile Plot 770, Rd. 44 Jubilee Hills, Hyderabad - 500033 >>> M +91 9000170909 T +91 40 2355 5945 - Ext: 229 www.imimobile.com >>> -----Original Message----- From: Ashutosh Chauhan >>> Sent: Wednesday, September 07, 2011 8:35 PM >>> >>> To: user@pig.apache.org >>> Subject: Re: ERROR 2999: Unexpected internal error. Failed to create >>> DataStorage >>> >>> Kiran, >>> >>> I guess your problem is config file. You have different value of >>> fs.default.name in hdfs-site.xml (10.0.0.61:8020) then in >>> mapred-site.xml >>> and core-site.xml (10.0.0.61:9000) Make them consistent and then try. >>> >>> Hope it helps, >>> Ashutosh >>> >>> On Wed, Sep 7, 2011 at 07:16, kiranprasad <kiranprasa...@imimobile.com>* >>> *** >>> ** >>> wrote: >>> >>> Hi >>> >>> >>>> Ive checked all the files are configured. >>>> >>>> For this Iam using 4 VMs (10.0.0.61,10.0.0.62,10.0.0.**** >>>> ****63,10.0.0.64) >>>> 1 VM(10.0.0.61) is for namenode , 2nd VM is for mapreduce(10.0.0.62), >>>> 3rd >>>> is datanode(10.0.0.63) ie is slave. >>>> I ve configured the same in masters and slaves file. >>>> >>>> core-site.xml >>>> ------------------ >>>> <configuration> >>>> <property> >>>> <name>fs.default.name</name> >>>> <value>hdfs://10.0.0.61:9000</********value> >>>> </property> >>>> </configuration> >>>> >>>> mapred-site.xml >>>> ------------------ >>>> <configuration> >>>> <property> >>>> <name>mapred.job.tracker</********name> >>>> <value>10.0.0.62:9000</value> >>>> </property> >>>> </configuration> >>>> >>>> hdfs-site.xml >>>> -------------------- >>>> <configuration> >>>> <property> >>>> <name>fs.default.name</name> >>>> <value>hdfs://10.0.0.61:8020</********value> >>>> </property> >>>> <property> >>>> <name>mapred.job.tracker</********name> >>>> <value>10.0.0.62:8021</value> >>>> </property> >>>> <property> >>>> <name>dfs.replication</name> >>>> <value>2</value> >>>> </property> >>>> </configuration> >>>> >>>> masters >>>> --------------- >>>> 10.0.0.61 >>>> 10.0.0.62 >>>> >>>> slaves >>>> --------- >>>> 10.0.0.63 >>>> >>>> But still getting the same exception. >>>> >>>> >>>> Regards >>>> Kiran.G >>>> >>>> IMImobile Plot 770, Rd. 44 Jubilee Hills, Hyderabad - 500033 >>>> M +91 9000170909 T +91 40 2355 5945 - Ext: 229 www.imimobile.com >>>> -----Original Message----- From: Marek Miglinski >>>> Sent: Wednesday, September 07, 2011 5:09 PM >>>> >>>> To: user@pig.apache.org >>>> Subject: RE: ERROR 2999: Unexpected internal error. Failed to create >>>> DataStorage >>>> >>>> Check if you have configured /etc/hadoop/conf/* files properly. >>>> >>>> >>>> Marek M. >>>> >>>> -----Original Message----- >>>> From: kiranprasad [mailto:kiranprasad.g@******imimob**ile.com<http://** >>>> imimobile.com <http://imimobile.com>> >>>> >>>> <kiranprasad.g@**imimobile.com <kiranprasa...@imimobile.com>> >>>> ] >>>> Sent: Wednesday, September 07, 2011 2:23 PM >>>> To: user@pig.apache.org >>>> Subject: Re: ERROR 2999: Unexpected internal error. Failed to create >>>> DataStorage >>>> >>>> Hi >>>> >>>> I ve started the namenode and datanode but still iam getting the same >>>> exception. >>>> >>>> Kiran.G >>>> >>>> IMImobile Plot 770, Rd. 44 Jubilee Hills, Hyderabad - 500033 M +91 >>>> 9000170909 T +91 40 2355 5945 - Ext: 229 www.imimobile.com-----Original >>>> Message----- >>>> From: Marek Miglinski >>>> Sent: Wednesday, September 07, 2011 2:29 PM >>>> To: user@pig.apache.org >>>> Subject: RE: ERROR 2999: Unexpected internal error. Failed to create >>>> DataStorage >>>> >>>> Check if Hadoop services are running: >>>> >>>> ${pathToHadoop service folder}/ hadoop-${version}-namenode status >>>> ${pathToHadoop service folder}/ hadoop-${version}-******** >>>> secondarynameode >>>> status ${pathToHadoop service folder}/ hadoop-${version}-datanode status >>>> ${pathToHadoop service folder}/ hadoop-${version}-jobtracker status (if >>>> you >>>> use it) ${pathToHadoop service folder}/ hadoop-${version}-tasktracker >>>> status >>>> (if you use it) >>>> >>>> Example: >>>> /etc/init.d/hadoop-0.20-********namenode status >>>> >>>> >>>> Marek M. >>>> >>>> -----Original Message----- >>>> From: kiranprasad [mailto:kiranprasad.g@******imimob**ile.com<http://** >>>> imimobile.com <http://imimobile.com>> >>>> >>>> <kiranprasad.g@**imimobile.com <kiranprasa...@imimobile.com>> >>>> ] >>>> Sent: Wednesday, September 07, 2011 11:55 AM >>>> To: user@pig.apache.org >>>> Subject: ERROR 2999: Unexpected internal error. Failed to create >>>> DataStorage >>>> >>>> Hi >>>> >>>> Iam new to PIG, trying to set up HADOOP cluster. >>>> >>>> The error Iam getting is >>>> >>>> [kiranprasad.g@pig1 pig-0.8.1]$ bin/pig >>>> 2011-09-07 19:45:50,606 [main] INFO org.apache.pig.Main - Logging error >>>> messages to: /home/kiranprasad.g/pig-0.8.1/** >>>> ******pig_1315404950603.log >>>> 2011-09-07 19:45:50,764 [main] INFO >>>> org.apache.pig.backend.hadoop.********executionengine.******** >>>> HExecutionEngine >>>> - >>>> Connecting to hadoop file system at: hdfs://10.0.0.61:0 >>>> 2011-09-07 19:45:52,171 [main] INFO org.apache.hadoop.ipc.Client - >>>> Retrying connect to server: /10.0.0.61:0. Already tried 0 time(s). >>>> 2011-09-07 19:45:53,523 [main] INFO org.apache.hadoop.ipc.Client - >>>> Retrying connect to server: /10.0.0.61:0. Already tried 1 time(s). >>>> 2011-09-07 19:45:55,096 [main] INFO org.apache.hadoop.ipc.Client - >>>> Retrying connect to server: /10.0.0.61:0. Already tried 2 time(s). >>>> 2011-09-07 19:45:56,447 [main] INFO org.apache.hadoop.ipc.Client - >>>> Retrying connect to server: /10.0.0.61:0. Already tried 3 time(s). >>>> 2011-09-07 19:45:57,755 [main] INFO org.apache.hadoop.ipc.Client - >>>> Retrying connect to server: /10.0.0.61:0. Already tried 4 time(s). >>>> 2011-09-07 19:45:59,081 [main] INFO org.apache.hadoop.ipc.Client - >>>> Retrying connect to server: /10.0.0.61:0. Already tried 5 time(s). >>>> 2011-09-07 19:46:00,394 [main] INFO org.apache.hadoop.ipc.Client - >>>> Retrying connect to server: /10.0.0.61:0. Already tried 6 time(s). >>>> 2011-09-07 19:46:01,734 [main] INFO org.apache.hadoop.ipc.Client - >>>> Retrying connect to server: /10.0.0.61:0. Already tried 7 time(s). >>>> 2011-09-07 19:46:03,050 [main] INFO org.apache.hadoop.ipc.Client - >>>> Retrying connect to server: /10.0.0.61:0. Already tried 8 time(s). >>>> 2011-09-07 19:46:04,373 [main] INFO org.apache.hadoop.ipc.Client - >>>> Retrying connect to server: /10.0.0.61:0. Already tried 9 time(s). >>>> 2011-09-07 19:46:04,384 [main] ERROR org.apache.pig.Main - ERROR 2999: >>>> Unexpected internal error. Failed to create DataStorage Details at >>>> logfile: >>>> /home/kiranprasad.g/pig-0.8.1/********pig_1315404950603.log >>>> >>>> Please Help >>>> >>>> Regards >>>> Kiran.G >>>> >>>> IMImobile Plot 770, Rd. 44 Jubilee Hills, Hyderabad - 500033 M +91 >>>> 9000170909 T +91 40 2355 5945 - Ext: 229 www.imimobile.com >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>> >>> >> >> > >
