Pig 0.11 cannot work with HBase 0.98
Hi Guys We are recently trying to upgrade our HBase from 0.94 to 0.98, our original PIG setup seems not work with the new HBase anymore, it throws the following class not found exception: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/filter/WritableByteArrayComparable at java.lang.Class.forName0(Native Method) I checked the source code of HBase 0.98, there seems no such a class WritableByteArrayComparable for 0.98, any thoughts? Thanks! Cheers Ramon
when will hbase create the zookeeper znode 'root-region-server’ is created? Hbase 0.94
Hello,
I am trying to debug a coprocessor code on hbase 0.94.24, which seems to work
well on 0.94.5, but I cannot make it work on 0.94.24.
Here is the copy of some coprocessor init code:
public class TestEndpoint implements TestIface, HTableWrapper {
…
@Override
public void start(CoprocessorEnvironment env) throws IOException {
this.env = env;
conf = env.getConfiguration();
HBaseAdmin admin = new HBaseAdmin(conf);
if (!admin.tableExists(SOME_TABLE)) {
//do something if this table is not there
…
When hbase starts, in the log file I noticed that when regionserver load this
coprocessor, it will hang there inside the admin.tableExists() function. That
API will try to access the zookeeper znode ‘root-region-server’, so I start the
‘hbase zkcli’ and run ‘ls /hbase’ at that time, and I found the znode
‘root-region-server’ is not created. Since the coprocessor want to access a
table, it must look up the –ROOT- region, which location is saved in that
znode, but that znode is not there. Then it hangs there for ever. If I disable
this coprocessor, hbase can start good and I can see ‘root-region-server’ znode
created there.
This coprocessor code is claimed to work well on 0.94.5, so I am wondering if
there is something changed about the sequence of ‘load coprocessor’ and the
‘create root-region-server znode’ in hbase 0.94 serials after 0.94.5?
So my basic question is : when znode ‘root-region-server’ is created? Who
create it? And is there any fixed timing sequence of this initialization
between the coprocessor loading time?
By the way, I cannot find anywhere to download a 0.94.5 hbase source code, can
anyone tell me if there is somewhere I can find it?
I know old version is obsolete , but this is not for production, but for
research, so please help me if you have any idea. Thanks very much in advance.
Thanks,
Ming
Re: How to add HBase dependencies and conf with spark-submit?
Thanks, Soumitra Kumar, I didn’t know why you put hbase-protocol.jar in SPARK_CLASSPATH, while add hbase-protocol.jar, hbase-common.jar, hbase-client.jar, htrace-core.jar in --jar, but it did work. Actually, I put all these four jars in SPARK_CLASSPATH along with HBase conf directory. 2014-10-15 22:39 GMT+08:00 Soumitra Kumar kumar.soumi...@gmail.com: I am writing to HBase, following are my options: export SPARK_CLASSPATH=/opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.jar spark-submit \ --jars /opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.jar,/opt/cloudera/parcels/CDH/lib/hbase/hbase-common.jar,/opt/cloudera/parcels/CDH/lib/hbase/hbase-client.jar,/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core.jar \ - Original Message - From: Fengyun RAO raofeng...@gmail.com To: u...@spark.apache.org, user@hbase.apache.org Sent: Wednesday, October 15, 2014 6:29:21 AM Subject: Re: How to add HBase dependencies and conf with spark-submit? +user@hbase 2014-10-15 20:48 GMT+08:00 Fengyun RAO raofeng...@gmail.com : We use Spark 1.1, and HBase 0.98.1-cdh5.1.0, and need to read and write an HBase table in Spark program. I notice there are: spark.driver.extraClassPath spark.executor.extraClassPath properties to manage extra ClassPath, over even an deprecated SPARK_CLASSPATH. The problem is what classpath or jars should we append? I can simplely add the whole `hbase classpath`, which is huge, but this leads to dependencies conflict, e.g. HBase uses guava-12 while Spark uses guava-14.
Re: Pig 0.11 cannot work with HBase 0.98
See https://issues.apache.org/jira/browse/HBASE-6658 See this thread as well: http://search-hadoop.com/m/T9Y7N1zRrGX1/Writablebytearraycomparable+hbasesubj=Re+Pig+cannot+load+data+using+hbasestorage Cheers On Oct 16, 2014, at 12:03 AM, Ramon Wang ra...@appannie.com wrote: Hi Guys We are recently trying to upgrade our HBase from 0.94 to 0.98, our original PIG setup seems not work with the new HBase anymore, it throws the following class not found exception: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/filter/WritableByteArrayComparable at java.lang.Class.forName0(Native Method) I checked the source code of HBase 0.98, there seems no such a class WritableByteArrayComparable for 0.98, any thoughts? Thanks! Cheers Ramon
Re: Schema migration in Hbase
How many rows are there in test1 table ? Please consider approach #1 for efficiency. Cheers On Oct 15, 2014, at 10:24 PM, Vimal Jain vkj...@gmail.com wrote: Hi, I have a hbase table( say test1) with 3 cfs ( a,b,c) and i have bunch of cqs in each of these cf. I also have one more table ( say test2) with some cfs on same cluster. My requirement is to move data from test1 to test2 through my custom logic ( as in which cf:cq from test1 will go to which cf:cq in test2). I thought there are ways 2 achieve this. 1) Writing some Java client code to perform get on test1 and perform put on test2. 2) Through writing a Jruby script and executing it on hbase shell. I preferred the 2nd way and started writing the script. i wrote something like this for ( all rowkeys in test1) do somevariable = get 'test1' , '1' ,'a:abc' put 'test2' ,'1','a:abc',somevariable done But its not working(Sometimes syntax error , sometimes empty value is put into test2 ) .I am new to Jruby scripting , so please redirect me to any tutorial/blog which can help me achieve this. I am using hbase 0.94.17. -- Thanks and Regards, Vimal Jain
Re: Schema migration in Hbase
There will be around 2 million rows in test1. Any specific reason for approach #1 being more efficient ? I thought approach #-2 being efficient as the ruby script will run on same machine ( I have only one node cluster ) ,so there wont be any network calls whereas in approach # 1 , i was planning to write java code and deploy it on different machine so this will involve network calls. Even if i choose approach # 1 , is there a possibility of doing this via Jruby script , i mean is it even possible ? On Thu, Oct 16, 2014 at 1:57 PM, Ted Yu yuzhih...@gmail.com wrote: How many rows are there in test1 table ? Please consider approach #1 for efficiency. Cheers On Oct 15, 2014, at 10:24 PM, Vimal Jain vkj...@gmail.com wrote: Hi, I have a hbase table( say test1) with 3 cfs ( a,b,c) and i have bunch of cqs in each of these cf. I also have one more table ( say test2) with some cfs on same cluster. My requirement is to move data from test1 to test2 through my custom logic ( as in which cf:cq from test1 will go to which cf:cq in test2). I thought there are ways 2 achieve this. 1) Writing some Java client code to perform get on test1 and perform put on test2. 2) Through writing a Jruby script and executing it on hbase shell. I preferred the 2nd way and started writing the script. i wrote something like this for ( all rowkeys in test1) do somevariable = get 'test1' , '1' ,'a:abc' put 'test2' ,'1','a:abc',somevariable done But its not working(Sometimes syntax error , sometimes empty value is put into test2 ) .I am new to Jruby scripting , so please redirect me to any tutorial/blog which can help me achieve this. I am using hbase 0.94.17. -- Thanks and Regards, Vimal Jain -- Thanks and Regards, Vimal Jain
Re: when will hbase create the zookeeper znode 'root-region-server’ is created? Hbase 0.94
On Thu, Oct 16, 2014 at 2:18 AM, Liu, Ming (HPIT-GADSC) ming.l...@hp.com wrote: By the way, I cannot find anywhere to download a 0.94.5 hbase source code, can anyone tell me if there is somewhere I can find it? I know old version is obsolete , but this is not for production, but for research, so please help me if you have any idea. Thanks very much in advance. In most cases you can get old versions of the HBase source from the ASF archives: http://archive.apache.org/dist/hbase/hbase-0.94.5/ -- Sean
Re: Schema migration in Hbase
bq. ruby script will run on same machine ( I have only one node cluster ) ,so there wont be any network calls I don't think the above is true: ruby script would still need to contact zookeeper to find the location of -ROOT-. There is no short circuit read for ruby script. Normal RPC is used to communicate with region server. Approach #2 is possible but you need to escape the row keys properly. Cheers On Thu, Oct 16, 2014 at 1:54 AM, Vimal Jain vkj...@gmail.com wrote: There will be around 2 million rows in test1. Any specific reason for approach #1 being more efficient ? I thought approach #-2 being efficient as the ruby script will run on same machine ( I have only one node cluster ) ,so there wont be any network calls whereas in approach # 1 , i was planning to write java code and deploy it on different machine so this will involve network calls. Even if i choose approach # 1 , is there a possibility of doing this via Jruby script , i mean is it even possible ? On Thu, Oct 16, 2014 at 1:57 PM, Ted Yu yuzhih...@gmail.com wrote: How many rows are there in test1 table ? Please consider approach #1 for efficiency. Cheers On Oct 15, 2014, at 10:24 PM, Vimal Jain vkj...@gmail.com wrote: Hi, I have a hbase table( say test1) with 3 cfs ( a,b,c) and i have bunch of cqs in each of these cf. I also have one more table ( say test2) with some cfs on same cluster. My requirement is to move data from test1 to test2 through my custom logic ( as in which cf:cq from test1 will go to which cf:cq in test2). I thought there are ways 2 achieve this. 1) Writing some Java client code to perform get on test1 and perform put on test2. 2) Through writing a Jruby script and executing it on hbase shell. I preferred the 2nd way and started writing the script. i wrote something like this for ( all rowkeys in test1) do somevariable = get 'test1' , '1' ,'a:abc' put 'test2' ,'1','a:abc',somevariable done But its not working(Sometimes syntax error , sometimes empty value is put into test2 ) .I am new to Jruby scripting , so please redirect me to any tutorial/blog which can help me achieve this. I am using hbase 0.94.17. -- Thanks and Regards, Vimal Jain -- Thanks and Regards, Vimal Jain
Re: when will hbase create the zookeeper znode 'root-region-server’ is created? Hbase 0.94
Ming: The tar ball in the archive contains source code. See example below: $ tar tzvf hbase-0.94.5.tar.gz | grep '\.java' | grep Assignment -rw-r--r-- 0 jenkins jenkins 47982 Feb 7 2013 hbase-0.94.5/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java -rw-r--r-- 0 jenkins jenkins 136645 Feb 7 2013 hbase-0.94.5/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java FYI On Thu, Oct 16, 2014 at 7:18 AM, Sean Busbey bus...@cloudera.com wrote: On Thu, Oct 16, 2014 at 2:18 AM, Liu, Ming (HPIT-GADSC) ming.l...@hp.com wrote: By the way, I cannot find anywhere to download a 0.94.5 hbase source code, can anyone tell me if there is somewhere I can find it? I know old version is obsolete , but this is not for production, but for research, so please help me if you have any idea. Thanks very much in advance. In most cases you can get old versions of the HBase source from the ASF archives: http://archive.apache.org/dist/hbase/hbase-0.94.5/ -- Sean
Re: Snapshot failure
Hi, I got a change finally to apply the timeout changes and it lasted longer before it started failing, but its now just throwing the same errors on procedure.Subprocedure: Subprocedure pool is full! Looking through the logs, it looks like the nodes may not correctly updating they completion state after finishing a snapshot, eg. 2014-09-25 21:03:15,713 INFO [regionserver16020-EventThread] procedure.ZKProcedureMemberRpcs: Received procedure start children changed event: /hbase/online-snapshot/acquired 2014-09-25 21:03:15,713 DEBUG [regionserver16020-EventThread] procedure.ZKProcedureMemberRpcs: Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2014-09-25 21:03:15,715 DEBUG [regionserver16020-EventThread] procedure.ZKProcedureMemberRpcs: Found procedure znode: /hbase/online-snapshot/acquired/requestData-20140925-210256 2014-09-25 21:03:15,718 DEBUG [regionserver16020-EventThread] procedure.ZKProcedureMemberRpcs: start proc data length is 57 2014-09-25 21:03:15,718 DEBUG [regionserver16020-EventThread] procedure.ZKProcedureMemberRpcs: Found data for znode:/hbase/online-snapshot/acquired/requestData-20140925-210256 2014-09-25 21:03:15,718 DEBUG [regionserver16020-EventThread] snapshot.RegionServerSnapshotManager: Launching subprocedure for snapshot requestData-20140925-210256 from table requestData 2014-09-25 21:03:15,718 DEBUG [regionserver16020-EventThread] procedure.ProcedureMember: Submitting new Subprocedure:requestData-20140925-210256 2014-09-25 21:03:15,719 DEBUG [member: '###-005,16020,1411640680181' subprocedure-pool1-thread-1] procedure.Subprocedure: Starting subprocedure 'requestData-20140925-210256' with timeout 60ms 2014-09-25 21:03:15,719 DEBUG [member: '###-005,16020,1411640680181' subprocedure-pool1-thread-1] errorhandling.TimeoutExceptionInjector: Scheduling process timer to run in: 60 ms 2014-09-25 21:03:15,720 DEBUG [member: '###-005,16020,1411640680181' subprocedure-pool1-thread-1] procedure.Subprocedure: Subprocedure 'requestData-20140925-210256' starting 'acquire' stage 2014-09-25 21:03:15,720 DEBUG [member: '###-005,16020,1411640680181' subprocedure-pool1-thread-1] procedure.Subprocedure: Subprocedure 'requestData-20140925-210256' locally acquired 2014-09-25 21:03:15,720 DEBUG [member: '###-005,16020,1411640680181' subprocedure-pool1-thread-1] procedure.ZKProcedureMemberRpcs: Member: '###-005,16020,1411640680181' joining acquired barrier fo r procedure (requestData-20140925-210256) in zk 2014-09-25 21:03:15,724 DEBUG [member: '###-005,16020,1411640680181' subprocedure-pool1-thread-1] procedure.ZKProcedureMemberRpcs: Watch for global barrier reached:/hbase/online-snapshot/reached/request Data-20140925-210256 2014-09-25 21:03:15,739 DEBUG [member: '###-005,16020,1411640680181' subprocedure-pool1-thread-1] procedure.Subprocedure: Subprocedure 'requestData-20140925-210256' coordinator notified of 'acquire', wa iting on 'reached' or 'abort' from coordinator 2014-09-25 21:03:15,825 INFO [regionserver16020-EventThread] procedure.ZKProcedureMemberRpcs: Received created event:/hbase/online-snapshot/reached/requestData-20140925-210256 2014-09-25 21:03:15,825 DEBUG [regionserver16020-EventThread] procedure.ZKProcedureMemberRpcs: Recieved reached global barrier:/hbase/online-snapshot/reached/requestData-20140925-210256 2014-09-25 21:03:15,825 DEBUG [member: '###-005,16020,1411640680181' subprocedure-pool1-thread-1] procedure.Subprocedure: Subprocedure 'requestData-20140925-210256' received 'reached' from coordinator. ... 2014-09-25 21:03:38,026 DEBUG [member: '###-005,16020,1411640680181' subprocedure-pool1-thread-1] snapshot.RegionServerSnapshotManager: Completed 5 local region snapshots. 2014-09-25 21:03:38,026 DEBUG [member: '###-005,16020,1411640680181' subprocedure-pool1-thread-1] snapshot.RegionServerSnapshotManager: cancelling 0 tasks for snapshot ###-005,16020,1411640680181 2014-09-25 21:03:38,026 DEBUG [member: '###-005,16020,1411640680181' subprocedure-pool1-thread-1] procedure.Subprocedure: Subprocedure 'requestData-20140925-210256' locally completed 2014-09-25 21:03:38,026 DEBUG [member: '###-005,16020,1411640680181' subprocedure-pool1-thread-1] procedure.ZKProcedureMemberRpcs: Marking procedure 'requestData-20140925-210256' completed for member ' ###-005,16020,1411640680181' in zk 2014-09-25 21:03:38,028 DEBUG [member: '###-005,16020,1411640680181' subprocedure-pool1-thread-1] procedure.Subprocedure: Subprocedure 'requestData-20140925-210256' has notified controller of completion 2014-09-25 21:03:38,028 DEBUG [member: '###-005,16020,1411640680181' subprocedure-pool1-thread-1] errorhandling.TimeoutExceptionInjector: Marking timer as complete - no error notifications will be recei ved for this timer. 2014-09-25 21:03:38,028 DEBUG [member:
Re: How to add HBase dependencies and conf with spark-submit?
Great, it worked. I don't have an answer what is special about SPARK_CLASSPATH vs --jars, just found the working setting through trial an error. - Original Message - From: Fengyun RAO raofeng...@gmail.com To: Soumitra Kumar kumar.soumi...@gmail.com Cc: u...@spark.apache.org, user@hbase.apache.org Sent: Thursday, October 16, 2014 12:50:01 AM Subject: Re: How to add HBase dependencies and conf with spark-submit? Thanks, Soumitra Kumar, I didn’t know why you put hbase-protocol.jar in SPARK_CLASSPATH, while add hbase-protocol.jar , hbase-common.jar , hbase-client.jar , htrace-core.jar in --jar, but it did work. Actually, I put all these four jars in SPARK_CLASSPATH along with HBase conf directory. 2014-10-15 22:39 GMT+08:00 Soumitra Kumar kumar.soumi...@gmail.com : I am writing to HBase, following are my options: export SPARK_CLASSPATH=/opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.jar spark-submit \ --jars /opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.jar,/opt/cloudera/parcels/CDH/lib/hbase/hbase-common.jar,/opt/cloudera/parcels/CDH/lib/hbase/hbase-client.jar,/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core.jar \ - Original Message - From: Fengyun RAO raofeng...@gmail.com To: u...@spark.apache.org , user@hbase.apache.org Sent: Wednesday, October 15, 2014 6:29:21 AM Subject: Re: How to add HBase dependencies and conf with spark-submit? +user@hbase 2014-10-15 20:48 GMT+08:00 Fengyun RAO raofeng...@gmail.com : We use Spark 1.1, and HBase 0.98.1-cdh5.1.0, and need to read and write an HBase table in Spark program. I notice there are: spark.driver.extraClassPath spark.executor.extraClassPath properties to manage extra ClassPath, over even an deprecated SPARK_CLASSPATH. The problem is what classpath or jars should we append? I can simplely add the whole `hbase classpath`, which is huge, but this leads to dependencies conflict, e.g. HBase uses guava-12 while Spark uses guava-14.
hbase-client Put serialization exception
Anyone had a problem using hbase-client on hadoop2? Seems like the Put class is missing a method (either default public constructor or some init method) and throws an exception when my MR jobs starts up. I’m using: HDP 2.1 with hbase-client.0.98.0.2.1.4.0-632-hadoop2.jar Stack trace: hadoop com.att.bdcoe.platform.persistence.mapreduce.jobs.GeoAnalyticFormatBulkLoader /user/hbase/scada /user/hbase/output 14/10/16 13:25:46 INFO impl.TimelineClientImpl: Timeline service address: http://dn02.platform.bigtdata.io:8188/ws/v1/timeline/ 14/10/16 13:25:46 INFO client.RMProxy: Connecting to ResourceManager at dn02.platform.bigtdata.io/172.16.6.27:8050 14/10/16 13:25:47 INFO input.FileInputFormat: Total input paths to process : 1 14/10/16 13:25:47 INFO mapreduce.JobSubmitter: Cleaning up the staging area /user/hbase/.staging/job_1410461150931_0063 14/10/16 13:25:47 ERROR jobs.GeoAnalyticFormatBulkLoader: java.lang.NoSuchMethodException: org.apache.hadoop.hbase.client.Put.init() 14/10/16 13:25:47 WARN hdfs.DFSClient: DataStreamer Exception org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/hbase/.staging/job_1410461150931_0063/job.split: File does not exist. Holder DFSClient_NONMAPREDUCE_-1551666999_1 does not have any open files. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2952) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2772) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2680) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:590) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) at org.apache.hadoop.ipc.Client.call(Client.java:1410) at org.apache.hadoop.ipc.Client.call(Client.java:1363) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy15.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103) at com.sun.proxy.$Proxy15.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525) 14/10/16 13:25:47 ERROR hdfs.DFSClient: Failed to close file /user/hbase/.staging/job_1410461150931_0063/job.split org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/hbase/.staging/job_1410461150931_0063/job.split: File does not exist. Holder DFSClient_NONMAPREDUCE_-1551666999_1 does not have any open files. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2952) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2772) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2680) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:590) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at
Re: hbase-client Put serialization exception
Do you have more information about the NoSuchMethodException w.r.t. Put() ctor ? Put() ctor doesn't exist in 0.98 Can you check GeoAnalyticFormatBulkLoader ? The other exceptions are for hdfs. Cheers On Thu, Oct 16, 2014 at 11:35 AM, THORMAN, ROBERT D rt2...@att.com wrote: Anyone had a problem using hbase-client on hadoop2? Seems like the Put class is missing a method (either default public constructor or some init method) and throws an exception when my MR jobs starts up. I’m using: HDP 2.1 with hbase-client.0.98.0.2.1.4.0-632-hadoop2.jar Stack trace: hadoop com.att.bdcoe.platform.persistence.mapreduce.jobs.GeoAnalyticFormatBulkLoader /user/hbase/scada /user/hbase/output 14/10/16 13:25:46 INFO impl.TimelineClientImpl: Timeline service address: http://dn02.platform.bigtdata.io:8188/ws/v1/timeline/ 14/10/16 13:25:46 INFO client.RMProxy: Connecting to ResourceManager at dn02.platform.bigtdata.io/172.16.6.27:8050 14/10/16 13:25:47 INFO input.FileInputFormat: Total input paths to process : 1 14/10/16 13:25:47 INFO mapreduce.JobSubmitter: Cleaning up the staging area /user/hbase/.staging/job_1410461150931_0063 14/10/16 13:25:47 ERROR jobs.GeoAnalyticFormatBulkLoader: java.lang.NoSuchMethodException: org.apache.hadoop.hbase.client.Put.init() 14/10/16 13:25:47 WARN hdfs.DFSClient: DataStreamer Exception org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/hbase/.staging/job_1410461150931_0063/job.split: File does not exist. Holder DFSClient_NONMAPREDUCE_-1551666999_1 does not have any open files. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2952) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2772) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2680) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:590) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) at org.apache.hadoop.ipc.Client.call(Client.java:1410) at org.apache.hadoop.ipc.Client.call(Client.java:1363) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy15.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103) at com.sun.proxy.$Proxy15.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525) 14/10/16 13:25:47 ERROR hdfs.DFSClient: Failed to close file /user/hbase/.staging/job_1410461150931_0063/job.split org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/hbase/.staging/job_1410461150931_0063/job.split: File does not exist. Holder DFSClient_NONMAPREDUCE_-1551666999_1 does not have any open files. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2952) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2772) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2680) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:590) at
Re: hbase-client Put serialization exception
Can you confirm that you're using the same version of hbase in your project dependencies as with your runtime system? Seems like you might have some 0.94 mixed in somewhere. On Thu, Oct 16, 2014 at 2:57 PM, Ted Yu yuzhih...@gmail.com wrote: Do you have more information about the NoSuchMethodException w.r.t. Put() ctor ? Put() ctor doesn't exist in 0.98 Can you check GeoAnalyticFormatBulkLoader ? The other exceptions are for hdfs. Cheers On Thu, Oct 16, 2014 at 11:35 AM, THORMAN, ROBERT D rt2...@att.com wrote: Anyone had a problem using hbase-client on hadoop2? Seems like the Put class is missing a method (either default public constructor or some init method) and throws an exception when my MR jobs starts up. I’m using: HDP 2.1 with hbase-client.0.98.0.2.1.4.0-632-hadoop2.jar Stack trace: hadoop com.att.bdcoe.platform.persistence.mapreduce.jobs.GeoAnalyticFormatBulkLoader /user/hbase/scada /user/hbase/output 14/10/16 13:25:46 INFO impl.TimelineClientImpl: Timeline service address: http://dn02.platform.bigtdata.io:8188/ws/v1/timeline/ 14/10/16 13:25:46 INFO client.RMProxy: Connecting to ResourceManager at dn02.platform.bigtdata.io/172.16.6.27:8050 14/10/16 13:25:47 INFO input.FileInputFormat: Total input paths to process : 1 14/10/16 13:25:47 INFO mapreduce.JobSubmitter: Cleaning up the staging area /user/hbase/.staging/job_1410461150931_0063 14/10/16 13:25:47 ERROR jobs.GeoAnalyticFormatBulkLoader: java.lang.NoSuchMethodException: org.apache.hadoop.hbase.client.Put.init() 14/10/16 13:25:47 WARN hdfs.DFSClient: DataStreamer Exception org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/hbase/.staging/job_1410461150931_0063/job.split: File does not exist. Holder DFSClient_NONMAPREDUCE_-1551666999_1 does not have any open files. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2952) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2772) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2680) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:590) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) at org.apache.hadoop.ipc.Client.call(Client.java:1410) at org.apache.hadoop.ipc.Client.call(Client.java:1363) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy15.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103) at com.sun.proxy.$Proxy15.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525) 14/10/16 13:25:47 ERROR hdfs.DFSClient: Failed to close file /user/hbase/.staging/job_1410461150931_0063/job.split org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/hbase/.staging/job_1410461150931_0063/job.split: File does not exist. Holder DFSClient_NONMAPREDUCE_-1551666999_1 does not have any open files. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2952) at
Re: custom filter on hbase 0.96
Hi, can anyone help with above? Feels like I'm missing something obvious. On Wednesday, October 15, 2014, Nishanth S nishanth.2...@gmail.com wrote: Thanks Ted .I will take a look. -Nishanth On Wed, Oct 15, 2014 at 3:43 PM, Ted Yu yuzhih...@gmail.com javascript:; wrote: Nishanth: Good question. As a general coding guide, writing unit test is always a good start. Using Matt's case as an example, take a look at TestPrefixFilter. There're various unit tests for Filters in hbase code. Cheers On Wed, Oct 15, 2014 at 2:30 PM, Nishanth S nishanth.2...@gmail.com javascript:; wrote: Hi Ted , Since I am also working on similar thing is there a way we can first test the filter on client side?.You know what I mean without disrupting others who are using the same cluster for other work? Thanks, Nishanth On Wed, Oct 15, 2014 at 3:17 PM, Ted Yu yuzhih...@gmail.com javascript:; wrote: bq. Or create a new file, compile it into ... You should go with the above approach. On Wed, Oct 15, 2014 at 2:08 PM, Matt K matvey1...@gmail.com javascript:; wrote: Hi all, I'm trying to get a custom filter to work on HBase 0.96. After some searching, I found that starting from 0.96, the implementer is required to implement toByteArray and parseFrom methods, using Protocol Buffers. But I'm having trouble with the how. The proto file for the existing filters is located here: https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/protobuf/Filter.proto Am I supposed to modify that file? Or create a new file, compile it into Java, and package it up with the filter? In the meantime, I've taken a shortcut that's not working. Here's my code: http://pastebin.com/iHFKu9Xz I'm using PrefixFilter, which comes with HBase, since I'm also filtering by prefix. However, that errors out with the following: http://pastebin.com/zBg47p6Z Thanks in advance for helping! -Matt -- www.calcmachine.com - easy online calculator.
RE: when will hbase create the zookeeper znode 'root-region-server’ is created? Hbase 0.94
Thanks Ted and Sean, It is great to find the archives, why I did not find them for so long time... :-) The original coprocessor author reply me yesterday, it is my fault. In fact, that coprocessor is loaded after the hbase startup, not during hbase regionserver startup time. In their client application code , they invoke addCoprocessor() to load the coprocessor at run-time. So I should not put them into the hbase-site.xml. When loaded later, the HBaseAdmin.tableExist() can work, since at that time, all initialization is done. So this is not a version issue, it can work in 0.94.5 and 0.94.24 as well. Thank you all for the help. Ming -Original Message- From: Ted Yu [mailto:yuzhih...@gmail.com] Sent: Thursday, October 16, 2014 10:29 PM To: user@hbase.apache.org Subject: Re: when will hbase create the zookeeper znode 'root-region-server’ is created? Hbase 0.94 Ming: The tar ball in the archive contains source code. See example below: $ tar tzvf hbase-0.94.5.tar.gz | grep '\.java' | grep Assignment -rw-r--r-- 0 jenkins jenkins 47982 Feb 7 2013 hbase-0.94.5/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java -rw-r--r-- 0 jenkins jenkins 136645 Feb 7 2013 hbase-0.94.5/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java FYI On Thu, Oct 16, 2014 at 7:18 AM, Sean Busbey bus...@cloudera.com wrote: On Thu, Oct 16, 2014 at 2:18 AM, Liu, Ming (HPIT-GADSC) ming.l...@hp.com wrote: By the way, I cannot find anywhere to download a 0.94.5 hbase source code, can anyone tell me if there is somewhere I can find it? I know old version is obsolete , but this is not for production, but for research, so please help me if you have any idea. Thanks very much in advance. In most cases you can get old versions of the HBase source from the ASF archives: http://archive.apache.org/dist/hbase/hbase-0.94.5/ -- Sean
Re: when will hbase create the zookeeper znode 'root-region-server’ is created? Hbase 0.94
0.94.5 to 0.94.24 are all releases in the 0.94 train. There is no change w.r.t. the timing of 'root-region-server’ znode creation. Cheers On Thu, Oct 16, 2014 at 8:19 PM, Liu, Ming (HPIT-GADSC) ming.l...@hp.com wrote: Thanks Ted and Sean, It is great to find the archives, why I did not find them for so long time... :-) The original coprocessor author reply me yesterday, it is my fault. In fact, that coprocessor is loaded after the hbase startup, not during hbase regionserver startup time. In their client application code , they invoke addCoprocessor() to load the coprocessor at run-time. So I should not put them into the hbase-site.xml. When loaded later, the HBaseAdmin.tableExist() can work, since at that time, all initialization is done. So this is not a version issue, it can work in 0.94.5 and 0.94.24 as well. Thank you all for the help. Ming -Original Message- From: Ted Yu [mailto:yuzhih...@gmail.com] Sent: Thursday, October 16, 2014 10:29 PM To: user@hbase.apache.org Subject: Re: when will hbase create the zookeeper znode 'root-region-server’ is created? Hbase 0.94 Ming: The tar ball in the archive contains source code. See example below: $ tar tzvf hbase-0.94.5.tar.gz | grep '\.java' | grep Assignment -rw-r--r-- 0 jenkins jenkins 47982 Feb 7 2013 hbase-0.94.5/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java -rw-r--r-- 0 jenkins jenkins 136645 Feb 7 2013 hbase-0.94.5/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java FYI On Thu, Oct 16, 2014 at 7:18 AM, Sean Busbey bus...@cloudera.com wrote: On Thu, Oct 16, 2014 at 2:18 AM, Liu, Ming (HPIT-GADSC) ming.l...@hp.com wrote: By the way, I cannot find anywhere to download a 0.94.5 hbase source code, can anyone tell me if there is somewhere I can find it? I know old version is obsolete , but this is not for production, but for research, so please help me if you have any idea. Thanks very much in advance. In most cases you can get old versions of the HBase source from the ASF archives: http://archive.apache.org/dist/hbase/hbase-0.94.5/ -- Sean
issue about migrate hbase table from 0.94 to 0.98
hi，maillist: how can do do data migrate from hbase 0.94 to 0.96 ,copy file from old hadoop cluster to new hadoop cluster is OK ,any one can help me?
