Problem with Scan operation over a Secure HBase 1.0.0-CDH5.4
Hello, We are running a Secure HBase cluster (enabling kerberos authentication and setting up hbase authorization) and we are trying to execute operations using a Java client. We are using the following configuration. import org.apache.hadoop.security.*; szQuorum="node01.example.com,node02.example.com,node01.example.com"; config = HBaseConfiguration.create(); config.set("hbase.zookeeper.quorum", szQuorum); config.set("hbase.zookeeper.property.clientPort", "2181"); config.set("hbase.security.authentication", "kerberos"); config.set("hadoop.security.authentication", "kerberos"); config.set("hbase.master.kerberos.principal","hbase/node03.example@example.com"); config.set("hbase.regionserver.kerberos.principal","hbase/node03.example@example.com"); UserGroupInformation.setConfiguration(config); ugi = UserGroupInformation.loginUserFromKeytabAndReturnUGI("hbase/node03.example@example.com","/var/run/cloudera-scm-agent/process/224-hbase-REGIONSERVER/hbase.keytab"); UserGroupInformation.setLoginUser(ugi); hconn = HConnectionManager.createConnection(config); hti=conn.getTable("exampletbl"); scan = new Scan(); rsScanner=hti.getScanner(scan); While we are able to create a table, puts and gets, when we try to execute a scan after a few seconds we get the following exceptions: 97976 [hconnection-0x4f2c9ba6-shared--pool1-t6] DEBUG org.apache.hadoop.hbase.security.HBaseSaslRpcClient - Have sent token of size 674 from initSASLContext. 97977 [hconnection-0x4f2c9ba6-shared--pool1-t6] WARN org.apache.hadoop.security.UserGroupInformation - PriviledgedActionExceptionas:hbase/node01.example@example.com(auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed 97977 [hconnection-0x4f2c9ba6-shared--pool1-t6] DEBUG org.apache.hadoop.security.UserGroupInformation - PrivilegedAction as:hbase/node01.example@example.com(auth:KERBEROS) from:org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.handleSaslConnectionFailure(RpcClientImpl.java:631) 97977 [hconnection-0x4f2c9ba6-shared--pool1-t6] WARN org.apache.hadoop.hbase.ipc.AbstractRpcClient - Couldn't setup connection for hbase/node01.example@example.com tohbase/node01.example@example.com We have run the kinit and setup the jaas.conf in the JAVA_OPTIONS of our Java application. export JAVA_OPTIONS=" -Djava.security.auth.login.config=/var/run/cloudera-scm-agent/process/224-hbase-REGIONSERVER/jaas.conf -Dsun.security.krb5.debug=true " kinit -k -t /var/run/cloudera-scm-agent/process/224-hbase-REGIONSERVER/hbase.keytabhbase/hbase/node03.example@example.com klist -f Ticket cache: FILE:/tmp/krb5cc_0 Default principal:hbase/hbase/node03.example@example.com Valid starting ExpiresService principal 06/17/15 17:37:31 06/18/15 17:37:31 krbtgt/example@example.com renew until 06/22/15 17:37:31, Flags: FRI less /var/run/cloudera-scm-agent/process/224-hbase-REGIONSERVER/jaas.conf Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true useTicketCache=true keyTab="hbase.keytab" principal="hbase/hbase/node03.example@example.com"; }; Is there any missing configuration? Thanks, Gaby
Question on upper bound of column qualifiers for a Row HBASE-0.98
Hello, We are working on HBase 0.98-CDH 5.2. We are creating a table where a row may contain a great amount of column qualifiers. Does it exists a limit on the number of column qualifiers a row may contain? Does the row have a limited size? Thanks, Gaby
Bucket cache configuration
Hello, We are working on performance tunning on HBase 0.98 CDH 5.1 using bucket cache. We were able to set up the bucket cache configuration on all storage nodes but we wanted to know further details on how the MaxDirectMemorySize is related to the bucket cache size and the Java heap size. How does the MaxDirectMemorySize affects performance in HBase? Our current configuration is the following: hbase.bucketcache.ioengine offheap hbase-site.xml hbase.bucketcache.percentage.in.combinedcache 0.8 hbase-site.xml hbase.bucketcache.size 1 hbase-site.xml -XX:MaxDirectMemorySize=12g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:-CMSConcurrentMTEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled Java Heap Size: 12G Thanks, Gaby
Re: Create custom filter on HBase 0.96.1.1-cdh5.0.1
Hi all, Here is the code snippet from the implementation of the parseFrom and toByteArray code. Is there something wrong I am doing there? Thanks, 1. publicKeyValueFilter(byte[]key,byte[]value) 2. { 3. m_keyBytes=key; 4. m_valueBytes=value; 5. } 6. 7. 8. /** 9. * Filter the row if the K/V pair is not found in the row data. 10. * 11. * @param cell the cell in question 12. * 13. */ 14. publicFilter.ReturnCodefilterKeyValue(Cell cell) 15. { 16. if(Bytes.equals(m_keyBytes, cell.getQualifier())){// CellUtil.cloneQualifier(cell))) { 17. 18. byte[]valueBytes=cell.getValue();// CellUtil.cloneValue(cell); 19. // found the matched k/v, do not filter this row 20. if(Bytes.equals(m_valueBytes, valueBytes)){ 21. debug("filterKeyValue: row matches K/V pair"); 22. 23. m_bFilterRow=false; 24. } 25. } 26. else{ 27. debug("filterKeyValue: row not matches key"); 28. } 29. 30. // include all columns 31. returnFilter.ReturnCode.INCLUDE; 32. } 33. 34. publicstatic Filter parseFrom(byte[]pbBytes)throwsDeserializationException 35. { 36. try{ 37. ByteArrayInputStreambais=newByteArrayInputStream(pbBytes); 38. DataInputStreaminput=newDataInputStream(bais); 39. 40. debug("parseFrom: read key byte array from byte array"); 41. byte[]keyBytes=Bytes.readByteArray(input); 42. 43. debug("parseFrom: read value byte array from byte array"); 44. byte[]valueBytes=Bytes.readByteArray(input); 45. 46. returnnewKeyValueFilter(keyBytes, valueBytes); 47. } 48. catch(IOExceptione){ 49. thrownewDeserializationException(e); 50. } 51. } 52. 53. publicbyte[]toByteArray()throwsIOException 54. { 55. ByteArrayOutputStreambaos=newByteArrayOutputStream(60/*size*/); 56. DataOutputStreamdos=newDataOutputStream(baos); 57. 58. debug("toByteArray: write key into byte array stream"); 59. Bytes.writeByteArray(dos, m_keyBytes); 60. 61. debug("toByteArray: write value into byte array stream"); 62. Bytes.writeByteArray(dos, m_valueBytes); 63. dos.flush(); 64. baos.flush(); 65. 66. returnbaos.toByteArray(); 67. } On 20/08/2014 06:46 p.m., Ted Yu wrote: Can you show implementation for parseFrom(byte[]) - using pastebin ? If possible, seeing the code for whole class would help us understand better. Cheers On Wed, Aug 20, 2014 at 4:18 PM, gabriela montiel < gabriela.mont...@oracle.com> wrote: Hi all, I have been working on migrating a custom filter used in HBase 0.94 to make it work on HBase 0.96.1.1. This custom filter extends the FilterBase API and receives only two byte arrays. According to the documentation both toByteArray() and parseFrom(byte[]) should be implemented. After adding this APIs, copying the jar to all region servers and restarting HBase, I noticed that the filter is not working (all records are retrieved). I don't get any errors, but checking out the trace the constructor and toByteArray() APIs are only called but no other operation on the filter is executed. I saw that you need to use FilterProtos from the protobuf to write/read the byte array to rebuild the filter, however there is no documentation on how to update these APIs to add my custom filter. Do you have any idea on what we are doing wrong? Thanks, Gaby
Create custom filter on HBase 0.96.1.1-cdh5.0.1
Hi all, I have been working on migrating a custom filter used in HBase 0.94 to make it work on HBase 0.96.1.1. This custom filter extends the FilterBase API and receives only two byte arrays. According to the documentation both toByteArray() and parseFrom(byte[]) should be implemented. After adding this APIs, copying the jar to all region servers and restarting HBase, I noticed that the filter is not working (all records are retrieved). I don't get any errors, but checking out the trace the constructor and toByteArray() APIs are only called but no other operation on the filter is executed. I saw that you need to use FilterProtos from the protobuf to write/read the byte array to rebuild the filter, however there is no documentation on how to update these APIs to add my custom filter. Do you have any idea on what we are doing wrong? Thanks, Gaby
Re: How to validate HBase configuration (for region servers) from HBase admin page?
Hi St.Ack, We are working with HBase 0.96.1.1-cdh5.0.1 using 10 region servers. We configured the bucket cache and set the -XX:MaxDirectMemorySize to 32g, heap size of 12g, and hbase.regionserver.count=40. The bucket cache configuration is the following (added in hbase-site.xml): hbase.regionserver.handler.count 40 hbase.bucketcache.ioengine offheap hbase.bucketcache.percentage.in.combinedcache 0.8 hbase.bucketcache.size 4 After restarting HBase, we checked out the HBase process from console we are able to set the -XX:MaxDirectMemorySize=32, as well as the memory heap size we configured. However, when we go to the http://:60010/conf, the*hbase.regionserver.count* is still set up to 30 (presumably the default value) and the bucketcache parameters added (bucketcache.ioengine, bucketcache.percentage.in.combinedcache, bucketcache.size) are not showed up, so we are not sure if the parameters are taken into account or not. Is there a way to validate that the bucketcache parameters as well as the new value for the hbase.regionserver.count are really taken from the hbase-site.xml we changed? Thanks, Gaby On 28/07/2014 02:23 p.m., Stack wrote: And don't forget to set -XX:MaxDirectMemorySize (from 9.6.4.5.1. Enable SlabCache in the refguide -- http://hbase.apache.org/book.html). This you set where you pass your server JVM args. St.Ack On Mon, Jul 28, 2014 at 12:21 PM, Stack wrote: Probably a question for the cloudera hbase list? (Did you enter the block cache configs via CM safety valve mechanism?) Yours, St.Ack On Mon, Jul 28, 2014 at 11:54 AM, Gabriela Montiel < gabriela.mont...@oracle.com> wrote: Hi Shengzhe, We executed a rolling restart on HBase through Cloudera 5, but after completing the site retrieved the same configurations. Thanks, Gaby Sent from my Samsung Mobile Original message From: yao Date:28/07/2014 13:43 (GMT-06:00) To: dev@hbase.apache.org Subject: Re: How to validate HBase configuration (for region servers) from HBase admin page? Hi Gaby, Did you restart your regionservers ? If not, HBase won't pick up the up-to-date configuration. Best Shengzhe On Mon, Jul 28, 2014 at 11:38 AM, Ted Yu wrote: bq. hbase-site.xml Can you check the hbase-site.xml which is in classpath of the region server ? What release of hbase are you using ? Cheers On Mon, Jul 28, 2014 at 11:28 AM, gabriela montiel < gabriela.mont...@oracle.com> wrote: Hi all, We have set up an HBase cluster with 10 region nodes. We modified the default configurations to handle bucket cache in the hbase-site.xml (for all region servers) as follows: hbase.regionserver.handler.count 40 hbase.bucketcache.ioengine offheap hbase.bucketcache.percentage.in.combinedcache 0.8 hbase.bucketcache.size 4 However, when we access the HBase configuration page on the HBase Web admin, we are not able to see the updated configurations, even for the ones not related to bucket cache: hbase.regionserver.handler.count*30*hbase-site.xml Can you give us a hint on why the configurations are not seen in the HBase configuration site, and how we can check out the HBase configuration for all region servers? Thanks, Gaby -- Regards, Gaby <http://www.oracle.com> Gabriela Montiel Moreno | Senior Member of Technical Staff Phone: +52 33 12531343 Oracle Semantic Technoliges ORACLE Mexico <http://www.oracle.com/commitment> Oracle is committed to developing practices and products that help protect the environment
Re: How to validate HBase configuration (for region servers) from HBase admin page?
Hi Shengzhe, We executed a rolling restart on HBase through Cloudera 5, but after completing the site retrieved the same configurations. Thanks, Gaby Sent from my Samsung Mobile Original message From: yao Date:28/07/2014 13:43 (GMT-06:00) To: dev@hbase.apache.org Subject: Re: How to validate HBase configuration (for region servers) from HBase admin page? Hi Gaby, Did you restart your regionservers ? If not, HBase won't pick up the up-to-date configuration. Best Shengzhe On Mon, Jul 28, 2014 at 11:38 AM, Ted Yu wrote: > bq. hbase-site.xml > > Can you check the hbase-site.xml which is in classpath of the region server > ? > > What release of hbase are you using ? > > Cheers > > > On Mon, Jul 28, 2014 at 11:28 AM, gabriela montiel < > gabriela.mont...@oracle.com> wrote: > > > Hi all, > > > > We have set up an HBase cluster with 10 region nodes. We modified the > > default configurations to handle bucket cache in the hbase-site.xml (for > > all region servers) as follows: > > > > > > hbase.regionserver.handler.count > > 40 > > > > > > hbase.bucketcache.ioengine > > offheap > > > > > > hbase.bucketcache.percentage.in.combinedcache > > 0.8 > > > > > > hbase.bucketcache.size > > 4 > > > > > > However, when we access the HBase configuration page on the HBase Web > > admin, we are not able to see the updated configurations, even for the > ones > > not related to bucket cache: > > > > hbase.regionserver.handler.count > name>*30*hbase-site.xml > > > > > > Can you give us a hint on why the configurations are not seen in the > HBase > > configuration site, and how we can check out the HBase configuration for > > all region servers? > > > > Thanks, > > > > Gaby > > > > > > > > > > > > > > > > > > > > >
How to validate HBase configuration (for region servers) from HBase admin page?
Hi all, We have set up an HBase cluster with 10 region nodes. We modified the default configurations to handle bucket cache in the hbase-site.xml (for all region servers) as follows: hbase.regionserver.handler.count 40 hbase.bucketcache.ioengine offheap hbase.bucketcache.percentage.in.combinedcache 0.8 hbase.bucketcache.size 4 However, when we access the HBase configuration page on the HBase Web admin, we are not able to see the updated configurations, even for the ones not related to bucket cache: hbase.regionserver.handler.count*30*hbase-site.xml Can you give us a hint on why the configurations are not seen in the HBase configuration site, and how we can check out the HBase configuration for all region servers? Thanks, Gaby