Hi,
I tried to give him the jars containing those classes both by java class path
and by pig.additional.jars and by REGISTER, all failed. It is even more strange
that it already has classes for hbase client and zookeeper but not slf.
Please advice!
Thanks,
Class path:
export CLASSPATH="$(~/tools/hbase-0.98.14-hadoop1/bin/hbase classpath)"
Pig script:
SET hbase.zookeeper.quorum localhost;
-- pig.exec.reducers.max defines the upper bound on the number of reducers
(default is 999)
SET pig.exec.reducers.max 2;
REGISTER /Users/dnmaras/tools/hbase-0.98.14-hadoop1/lib/slf4j-api-1.6.4.jar;
REGISTER /Users/dnmaras/tools/hbase-0.98.14-hadoop1/lib/slf4j-log4j12-1.6.4.jar;
REGISTER /Users/dnmaras/tools/hbase-0.98.14-hadoop1/lib/zookeeper-3.4.6.jar;
daily_rows = LOAD 'hbase://dt_campaign'
USING org.apache.pig.backend.hadoop.hbase.HBaseStorage(
'v:clid,v:caid, v:dt, v:dth, v:imo, v:im, v:ex, v:ltr, v:clo, v:cl,
v:imcl, v:rev',
'-loadKey true -gte ${client}2015${month} -lte ${client}2015${month}Z')
AS (rowkey:chararray, clid:chararray, caid:chararray, dt:chararray,
imo:double, im:double, ex:double, ltr:double, clo:double, cl:double,
imcl:double, rev:double);
describe daily_rows;
monthly_rows_grp = GROUP daily_rows BY SUBSTRING(dt, 0, 7);
describe monthly_rows_grp;
monthly_rows = FOREACH monthly_rows_grp GENERATE
group as ym,
COUNT(daily_rows) as inputrows,
CONCAT('M', SUBSTRING(MIN(daily_rows.rowkey), 1, 26)) AS rowkey,
MIN(daily_rows.clid) AS clid,
MIN(daily_rows.caid) AS caid,
SUM(daily_rows.imo) AS imo,
SUM(daily_rows.im) AS im,
SUM(daily_rows.ex) AS ex,
SUM(daily_rows.ltr) AS ltr,
SUM(daily_rows.clo) AS clo,
SUM(daily_rows.cl) AS cl,
SUM(daily_rows.imcl) AS imcl,
SUM(daily_rows.rev) AS rev;
describe monthly_rows;
STORE monthly_rows INTO 'dt_campaign' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage(
'v:ym, v:irows, v:clid, v:caid, v:imo, v:im, v:ex, v:ltr, v:clo, v:cl,
v:imcl, v:rev'
);
So the slf LoggerFactory is in:
REGISTER /Users/dnmaras/tools/hbase-0.98.14-hadoop1/lib/slf4j-api-1.6.4.jar;
which is also in the classpath and running java slf.LoggerFactory gives that it
does not have main (not that it does not exist).
Pig run:
rm -f pig*; pig -Dpig.additional.jars=slf4j-api-1.6.4.jar -x local -p month=01
-p client=29
/Users/dnmaras/ds/bcs-batch/modules/bcs-pig/src/main/resources/scripts/granularities.pig;
cat pig*
15/10/30 12:13:10 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
15/10/30 12:13:10 INFO pig.ExecTypeProvider: Picked LOCAL as the ExecType
2015-10-30 12:13:10,381 [main] INFO org.apache.pig.Main - Apache Pig version
0.14.0 (r1640057) compiled Nov 16 2014, 18:01:24
2015-10-30 12:13:10,381 [main] INFO org.apache.pig.Main - Logging error
messages to: /private/tmp/pig_1446199990380.log
2015-10-30 12:13:10,443 [main] INFO org.apache.pig.impl.util.Utils - Default
bootup file /Users/dnmaras/.pigbootup not found
2015-10-30 12:13:10,495 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to
hadoop file system at: file:///
daily_rows: {rowkey: chararray,clid: chararray,caid: chararray,dt:
chararray,imo: double,im: double,ex: double,ltr: double,clo: double,cl:
double,imcl: double,rev: double}
monthly_rows_grp: {group: chararray,daily_rows: {(rowkey: chararray,clid:
chararray,caid: chararray,dt: chararray,imo: double,im: double,ex: double,ltr:
double,clo: double,cl: double,imcl: double,rev: double)}}
monthly_rows: {ym: chararray,inputrows: long,rowkey: chararray,clid:
chararray,caid: chararray,imo: double,im: double,ex: double,ltr: double,clo:
double,cl: double,imcl: double,rev: double}
2015-10-30 12:13:11,331 [main] ERROR
org.apache.hadoop.hbase.mapreduce.TableOutputFormat - java.io.IOException:
java.lang.reflect.InvocationTargetException
2015-10-30 12:13:11,334 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
2999: Unexpected internal error. java.io.IOException:
java.lang.reflect.InvocationTargetException
Details at logfile: /private/tmp/pig_1446199990380.log
2015-10-30 12:13:11,349 [main] INFO org.apache.pig.Main - Pig script completed
in 3 seconds and 213 milliseconds (3213 ms)
Pig Stack Trace
---------------
ERROR 2999: Unexpected internal error. java.io.IOException:
java.lang.reflect.InvocationTargetException
java.lang.RuntimeException: java.io.IOException:
java.lang.reflect.InvocationTargetException
at
org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:211)
at
org.apache.pig.backend.hadoop.hbase.HBaseStorage.getOutputFormat(HBaseStorage.java:904)
at
org.apache.pig.newplan.logical.visitor.InputOutputFileValidatorVisitor.visit(InputOutputFileValidatorVisitor.java:69)
at
org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66)
at
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
at
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at
org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
at
org.apache.pig.newplan.logical.relational.LogicalPlan.validate(LogicalPlan.java:212)
at org.apache.pig.PigServer$Graph.compile(PigServer.java:1767)
at org.apache.pig.PigServer$Graph.access$300(PigServer.java:1443)
at org.apache.pig.PigServer.execute(PigServer.java:1356)
at org.apache.pig.PigServer.executeBatch(PigServer.java:415)
at org.apache.pig.PigServer.executeBatch(PigServer.java:398)
at
org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
at org.apache.pig.Main.run(Main.java:624)
at org.apache.pig.Main.main(Main.java:170)
Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
at
org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:383)
at
org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:360)
at
org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:244)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:187)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:149)
at
org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:206)
... 21 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
at
org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:381)
... 26 more
Caused by: java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
at org.apache.zookeeper.ZooKeeper.<clinit>(ZooKeeper.java:94)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:112)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:132)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:165)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134)
at
org.apache.hadoop.hbase.client.ZooKeeperKeepAliveConnection.<init>(ZooKeeperKeepAliveConnection.java:43)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(HConnectionManager.java:1722)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:82)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:794)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:627)
... 31 more
Caused by: java.lang.ClassNotFoundException: org.slf4j.LoggerFactory
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 41 more
