HbaseStorage is broken in Pig 0.5.0, see https://issues.apache.org/jira/browse/PIG-970
The fix for that has been checked into trunk. You can either check
out from trunk and build to get that, or can check out from the 0.5.0
branch and then apply the patches in PIG-970 to that code base. If
you choose the latter let me know and I'll help you figure out which
files in the JIRA you actually need.
Alan.
On Nov 20, 2009, at 8:35 AM, joris lops wrote:
Hi all,
We use pig 0.20.1 from a release distribution (no trunk).
The hbase-0.20.1/conf/hbase-site.xml is included in CLASSPATH.
The hbase-site.xml content (is this correct?):
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://localhost:9000/hbase</value>
<description>The directory shared by region servers.
</description>
</property>
</configuration>
Is my URL to the database correct ('hbase://test') and which pig
version
should I use (mapreduce or local?)
Now I get different results and a stacktrace (from the commands
below).
Maybe this is helpful; the bold part in the stacktrace indicates a
wrong URL
to the databasetable, is this an correct interpretation of the
stacktrace?
pig -x mapreduce
grunt> B = load 'hbase://test' using
org.apache.pig.backend.hadoop.hbase.HBaseStorage('data') as (col_a);
grunt> dump B;
still the same result:
Retrying connect to server: localhost/127.0.0.1:60000 <
http://127.0.0.1:60000>. Already tried 0 time(s).
pig-0.5.0/bin/pig -x local
grunt> B = load 'hbase://test' using
org.apache.pig.backend.hadoop.hbase.HBaseStorage('data') as (col_a);
grunt> dump B;
2009-11-20 17:15:01,725 [main] ERROR
org.apache.pig.tools.grunt.Grunt -
ERROR 1002: Unable to store alias B
Details at logfile: /Users/jorislops/Desktop/pig_1258733695965.log
Pig Stack Trace
---------------
ERROR 1002: Unable to store alias B
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066:
Unable to
open iterator for alias B
at org.apache.pig.PigServer.openIterator(PigServer.java:475)
at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:
532)
at
org
.apache
.pig
.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:
190)
at
org
.apache
.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:166)
at
org
.apache
.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:142)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:75)
at org.apache.pig.Main.main(Main.java:363)
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR
1002:
Unable to store alias B
at org.apache.pig.PigServer.store(PigServer.java:530)
at org.apache.pig.PigServer.openIterator(PigServer.java:458)
... 6 more
Caused by: org.apache.pig.backend.executionengine.ExecException:
ERROR 0:
Wrong FS: hbase://test, expected: file:///
at
org
.apache
.pig
.backend
.local
.executionengine
.LocalExecutionEngine.execute(LocalExecutionEngine.java:184)
at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:
773)
at org.apache.pig.PigServer.store(PigServer.java:522)
... 7 more
*Caused by: java.lang.IllegalArgumentException: Wrong FS: hbase://
test,
expected: file:///*
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:305)
at
org
.apache
.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:47)
at
org
.apache
.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:
357)
at
org
.apache
.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:643)
at
org
.apache
.pig
.backend
.hadoop.datastorage.HDataStorage.isContainer(HDataStorage.java:203)
at
org
.apache
.pig
.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:
131)
at
org
.apache
.pig
.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:
147)
at
org.apache.pig.impl.io.FileLocalizer.fullPath(FileLocalizer.java:532)
at org.apache.pig.impl.io.FileLocalizer.open(FileLocalizer.java:
346)
at
org
.apache
.pig
.backend
.hadoop
.executionengine
.physicalLayer.relationalOperators.POLoad.setUp(POLoad.java:103)
at
org
.apache
.pig
.backend
.hadoop
.executionengine
.physicalLayer.relationalOperators.POLoad.getNext(POLoad.java:131)
at
org
.apache
.pig
.backend
.hadoop
.executionengine
.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:
231)
at
org
.apache
.pig
.backend
.local
.executionengine
.physicalLayer.counters.POCounter.getNext(POCounter.java:71)
at
org
.apache
.pig
.backend
.hadoop
.executionengine
.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:
231)
at
org
.apache
.pig
.backend
.hadoop
.executionengine
.physicalLayer.relationalOperators.POStore.getNext(POStore.java:117)
at
org
.apache
.pig
.backend
.local
.executionengine.LocalPigLauncher.runPipeline(LocalPigLauncher.java:
146)
at
org
.apache
.pig
.backend
.local
.executionengine.LocalPigLauncher.launchPig(LocalPigLauncher.java:109)
at
org
.apache
.pig
.backend
.local
.executionengine
.LocalExecutionEngine.execute(LocalExecutionEngine.java:165)
... 9 more
Thanks,
Joris
2009/11/19 Jeff Zhang <zjf...@gmail.com>
Hi Morris,
Do you use the pig in trunk ?
If you want to use hbase, you should put hbase configuration in
hbase-site.xml, and put this file on your classpath.
Jeff Zhang
On Thu, Nov 19, 2009 at 8:20 AM, Morris Swertz <m.a.swe...@rug.nl>
wrote:
Hi all,
I try to load data from HBase into pig with HBaseStorage.
Something is
going wrong because no data from HBase (test table) shows up in
Pig; only
errors.
I configured the Hadoop and HBase in Pseudo-Distributed Operation
mode.
What follows are the commands that I did and the output it produced.
//try with pig in remote mode!
pig -x mapreduce
B = load 'hbase://test' using
org.apache.pig.backend.hadoop.hbase.HBaseStorage('data') as (col_a);
dump B;
output:
009-11-19 13:56:02,810 [main] INFO
org
.apache
.pig
.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
2009-11-19 13:56:02,810 [main] INFO
org
.apache
.pig
.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
2009-11-19 13:56:04,708 [main] INFO
org
.apache
.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2009-11-19 13:56:04,729 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM
Metrics
with processName=JobTracker, sessionId= - already initialized
2009-11-19 13:56:04,739 [Thread-5] WARN
org.apache.hadoop.mapred.JobClient
- Use GenericOptionsParser for parsing the arguments. Applications
should
implement Tool for the same.
2009-11-19 13:56:05,024 [Thread-5] INFO
org.apache.pig.backend.hadoop.hbase.HBaseStorage - tablename:
file:/Users/jorislops/Desktop/pig-0.5.0/test
2009-11-19 13:56:05,231 [main] INFO
org
.apache
.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2009-11-19 13:56:06,222 [Thread-5] INFO
org.apache.hadoop.ipc.Client -
Retrying connect to server: localhost/127.0.0.1:60000 <
http://127.0.0.1:60000>. Already tried 0 time(s).
2009-11-19 13:56:06,222 [Thread-5] INFO
org.apache.hadoop.ipc.Client -
Retrying connect to server: localhost/127.0.0.1:60000 <
http://127.0.0.1:60000>. Already tried 1 time(s).
//port 60000 is used by a java program
pig -x local
B = load 'test' using
org.apache.pig.backend.hadoop.hbase.HBaseStorage('data') as (col_a);
dump B;
output:
2009-11-19 13:53:18,425 [main] INFO
org.apache.pig.backend.local.executionengine.LocalPigLauncher -
Successfully stored result in: "file:/tmp/temp-1663248768/
tmp-1939618752"
2009-11-19 13:53:18,436 [main] INFO
org.apache.pig.backend.local.executionengine.LocalPigLauncher -
Records
written : 0
2009-11-19 13:53:18,436 [main] INFO
org.apache.pig.backend.local.executionengine.LocalPigLauncher -
Bytes
written : 0
2009-11-19 13:53:18,436 [main] INFO
org.apache.pig.backend.local.executionengine.LocalPigLauncher - 100%
complete!
2009-11-19 13:53:18,436 [main] INFO
org.apache.pig.backend.local.executionengine.LocalPigLauncher -
Success!!
//there is nothing in /tmp/temp-1663248768/tmp-1939618752 (it's
empty)
I tried different paths to the HBase table 'hbase://test', 'test',
hbase://localhost:60000/test
How I stated the system (Hadoop + HBase) is started and I verified
that's
working as I expected.
bin/hadoop namenode -format
bin/start-all.sh
//both Namenode and Jobtrackter are running verified by
http://localhost:50070 and http://localhost:500040
bin/start-hbase.sh
//both mater and regionserver are running check by localhost:60010
localhost:20 localhost:30
//also zookeeper Quorum is started at port localhost:2181
//fill a test table in hbase
hbase-0.20.1/bin/hbase shell
create 'test', 'data'
put 'test', 'row1', 'data', 'value1'
scan 'test'
//localhost:60010 show that the test table is in HBase.
Hope that someone knows the solution.
Thanks,
Joris
