Can you go to the Resource Manager UI and look at the diagnostics and task logs of job_1464584017709_0003 to see what the actual stacktrace is? Most likely there are some connection issues with either hdfs or hbase and these can retry for a really long time before erroring out. Only that can explain taking 1 hour before failing when there are two records involved.
On Sun, May 29, 2016 at 10:47 PM, rituparna sinha < rituparna.si...@heritageit.edu> wrote: > Hello, > I have an issue regarding storing data in HBase table using > pig. The details are as follows. > > I have hadoop-2.5.1 cluster with one master and one data node, each having > 2GB RAM. Upon this in the master I have HBase-1.1.4 in pseudo distributed > mode. Till this everything is working fine. Then I have installed pig. > First I untarred pig-15 src-tar .gz and then untarred the pig-15.tar.gz. > Then I executed the command i.e. > > ant clean tar -Dhadoopversion=23 -Dhbase95.version=1.1.2 > -Dforrest.home=<downloaded forrest location> > > It worked succesfully. Next I went to grunt shell in mapreduce mode. I am > able to load an input file from hdfs into a pig variable using load > command. My input file contains only 2 records. > > 1975,32 > 1950,45 > > I am also successfully been able to dump the data. But when I try to store > the dumped data into hbase table , I see the following > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *2016-05-30 10:28:47,461 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 0% complete2016-05-30 10:28:47,462 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - Running jobs are [job_1464584017709_0003]2016-05-30 10:29:22,722 [main] > INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 50% complete2016-05-30 10:29:22,722 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - Running jobs are [job_1464584017709_0003]2016-05-30 11:11:30,888 [main] > WARN > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to > stop immediately on failure.2016-05-30 11:11:30,889 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - job job_1464584017709_0003 has failed! Stop running all dependent > jobs2016-05-30 11:11:30,889 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 100% complete2016-05-30 11:11:30,964 [main] INFO > org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at > hadoop-master/192.168.36.10:8032 <http://192.168.36.10:8032>2016-05-30 > 11:11:31,084 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - > Application state is completed. FinalApplicationStatus=FAILED. Redirecting > to job history server2016-05-30 11:11:32,227 [main] INFO > org.apache.hadoop.mapred.ClientServiceDelegate - Application state is > completed. FinalApplicationStatus=FAILED. Redirecting to job history > server2016-05-30 11:11:32,614 [main] INFO > org.apache.hadoop.mapred.ClientServiceDelegate - Application state is > completed. FinalApplicationStatus=FAILED. Redirecting to job history > server2016-05-30 11:11:32,741 [main] INFO > org.apache.hadoop.mapred.ClientServiceDelegate - Application state is > completed. FinalApplicationStatus=FAILED. Redirecting to job history > server2016-05-30 11:11:33,240 [main] ERROR > org.apache.pig.tools.pigstats.PigStats - ERROR 0: java.io.IOException: > org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): > java.lang.NullPointerException at > > org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getTaskReports(HistoryClientService.java:319) > at > > org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getTaskReports(MRClientProtocolPBServiceImpl.java:185) > at > > org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:285) > at > > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) at > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) at > java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:415) at > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)2016-05-30 > 11:11:33,240 [main] ERROR > org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil - 1 map reduce > job(s) failed!2016-05-30 11:11:33,240 [main] INFO > org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script Statistics: > HadoopVersion PigVersion UserId StartedAt FinishedAt > Features2.5.1 0.15.0-SNAPSHOT hduser 2016-05-30 10:28:44 > 2016-05-30 11:11:33 UNKNOWNFailed!Failed Jobs:JobId Alias > Feature Message Outputsjob_1464584017709_0003 rit MAP_ONLY > Message: Job failed! hbase://T1,Input(s):Failed to read data from > "/input/god.txt"Output(s):Failed to produce result in > "hbase://T1"Counters:Total records written : 0Total bytes written : > 0Spillable Memory Manager spill count : 0Total bags proactively spilled: > 0Total records proactively spilled: 0Job > DAG:job_1464584017709_00032016-05-30 11:11:33,241 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - Failed!grunt>* > > > > For your information the store command is taking aprox 1 hour to get the > error above. Kindly help as I am very new to this environment and is not > very comfortable with java. > > > Thanks and Regards, > Rituparna Sinha, > Asst. Professor, > Dept. of IT > Heritage Institute of Technology > > > > > > > > > Thanks and Regards, > Rituparna Sinha, > Asst. Professor, > Dept. of IT > Heritage Institute of Technology > > -- > > > *Disclaimer:* > This e-mail contains confidential information belonging to Heritage > Institute of Technology & is intended solely for the Addresses. The > unauthorized disclosure, use dissemination or copying (either whole or > partial of this e-mail, or any information it contains), is prohibited. > E-mail is susceptible to alteration and their integrity cannot be > guaranteed. Heritage Institute of Technology shall not be liable of this > e-mail if modified or falsified. If you are not the intended recipient of > this e-mail, please delete it immediately from your system & notify the > sender of the wrong delivery and the mail deletion. >