Environment: hadoop 1.0.3, hbase 0.94.1, pig 0.11.1
I am running a pig script in Java program, I get the following error sometimes
but not all the time. What the program does is it loads a file from hdfs, do
some transformation and store it into hbase. My program is multi-threaded. And
I've already made PigServer thread-safe and I have "/user/root" directory
created in hdfs. Here is the snippet of the program and the exception I've got.
Please advise.
pigServer =PigFactory.getServer();
URL path = getClass().getClassLoader().getResource("cfg/concatall.py");
LOG.info("CDNResolve2Hbase: reading concatall.py file from "+ path.toString());
pigServer.getPigContext().getProperties().setProperty(PigContext.JOB_NAME,"CDNResolve2Hbase");
pigServer.registerQuery("A = load '"+ inputPath +"' using PigStorage('\t') as
(ip:chararray, do:chararray, cn:chararray, cdn:chararray,
firsttime:chararray, updatetime:chararray);");
pigServer.registerCode(path.toString(),"jython","myfunc");
pigServer.registerQuery("B = foreach A generate myfunc.concatall('"+
extractTimestamp (inputPath)+"',ip,do,cn), cdn, SUBSTRING(firsttime,0,8);");
outputTable ="hbase://"+ outputTable;ExecJob job = pigServer.store
("B",outputTable,"org.apache.pig.backend.hadoop.hbase.HBaseStorage('d:cdn
d:dtime')");
and my PigFactory has the following code
privatestaticThreadLocal<PigServer> pigServer
=newThreadLocal<PigServer>();publicstaticsynchronizedPigServer
getServer(){if(pigServer.get()==null){try{ printClassPath();Properties prop
=SystemUtils.getCfg(); pigServer.set(newPigServer(ExecType.MAPREDUCE,
prop));return pigServer.get();}catch(Exception e){ LOG.error("error in starting
PigServer:", e);returnnull;}}return pigServer.get();}
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during
parsing. Unable to check name hdfs://DC-001:9000/user/root at
org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1607) at
org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1546) at
org.apache.pig.PigServer.registerQuery(PigServer.java:516) at
org.apache.pig.PigServer.registerQuery(PigServer.java:529) at
com.hugedata.cdnserver.datanalysis.CDNResolve2Hbase.execute(Unknown Source) at
com.hugedata.cdnserver.DatAnalysis.cdnResolve2Hbase(Unknown Source) at
com.hugedata.cdnserver.task.HandleDomainNameLogTask.execute(Unknown Source) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273) at
org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:264)
at
org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)
at org.quartz.core.JobRunShell.run(JobRunShell.java:203) at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)
Caused by: Failed to parse: Pig script failed to parse: pig script failed to
validate: org.apache.pig.backend.datastorage.DataStorageException: ERROR 6007:
Unable to check name hdfs://DC-001:9000/user/root at
org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191) at
org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1599)
... 15 more Caused by: pig script failed to validate:
org.apache.pig.backend.datastorage.DataStorageException: ERROR 6007: Unable to
check name hdfs://DC-001:9000/user/root at
org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:835)
at
org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3236)
at
org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1315)
at
org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:799)
at
org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:517)
at
org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:392)
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:184)
... 16 more Caused by: org.apache.pig.backend.datastorage.DataStorageException:
ERROR 6007: Unable to check name hdfs://DC-001:9000/user/root at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.isContainer(HDataStorage.java:207)
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:128)
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:138)
at
org.apache.pig.parser.QueryParserUtils.getCurrentDir(QueryParserUtils.java:91)
at
org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:827)
... 22 more Caused by: java.io.IOException: Filesystem closed at
org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:264) at
org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:873) at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:513)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:768) at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.isContainer(HDataStorage.java:200)
... 26 more
java hbase pig
Thanks. Please advise.
Teresa
