Create custom filter on HBase 0.96.1.1-cdh5.0.1
Hi all, I have been working on migrating a custom filter used in HBase 0.94 to make it work on HBase 0.96.1.1. This custom filter extends the FilterBase API and receives only two byte arrays. According to the documentation both toByteArray() and parseFrom(byte[]) should be implemented. After adding this APIs, copying the jar to all region servers and restarting HBase, I noticed that the filter is not working (all records are retrieved). I don't get any errors, but checking out the trace the constructor and toByteArray() APIs are only called but no other operation on the filter is executed. I saw that you need to use FilterProtos from the protobuf to write/read the byte array to rebuild the filter, however there is no documentation on how to update these APIs to add my custom filter. Do you have any idea on what we are doing wrong? Thanks, Gaby
Re: Create custom filter on HBase 0.96.1.1-cdh5.0.1
Hi all,
Here is the code snippet from the implementation of the parseFrom and
toByteArray code. Is there something wrong I am doing there?
Thanks,
1.
publicKeyValueFilter(byte[]key,byte[]value)
2.
{
3.
m_keyBytes=key;
4.
m_valueBytes=value;
5.
}
6.
7.
8.
/**
9.
* Filter the row if the K/V pair is not found in the row data.
10.
*
11.
* @param cell the cell in question
12.
*
13.
*/
14.
publicFilter.ReturnCodefilterKeyValue(Cell cell)
15.
{
16.
if(Bytes.equals(m_keyBytes, cell.getQualifier())){//
CellUtil.cloneQualifier(cell))) {
17.
18.
byte[]valueBytes=cell.getValue();// CellUtil.cloneValue(cell);
19.
// found the matched k/v, do not filter this row
20.
if(Bytes.equals(m_valueBytes, valueBytes)){
21.
debug("filterKeyValue: row matches K/V pair");
22.
23.
m_bFilterRow=false;
24.
}
25.
}
26.
else{
27.
debug("filterKeyValue: row not matches key");
28.
}
29.
30.
// include all columns
31.
returnFilter.ReturnCode.INCLUDE;
32.
}
33.
34.
publicstatic Filter
parseFrom(byte[]pbBytes)throwsDeserializationException
35.
{
36.
try{
37.
ByteArrayInputStreambais=newByteArrayInputStream(pbBytes);
38.
DataInputStreaminput=newDataInputStream(bais);
39.
40.
debug("parseFrom: read key byte array from byte array");
41.
byte[]keyBytes=Bytes.readByteArray(input);
42.
43.
debug("parseFrom: read value byte array from byte array");
44.
byte[]valueBytes=Bytes.readByteArray(input);
45.
46.
returnnewKeyValueFilter(keyBytes, valueBytes);
47.
}
48.
catch(IOExceptione){
49.
thrownewDeserializationException(e);
50.
}
51.
}
52.
53.
publicbyte[]toByteArray()throwsIOException
54.
{
55.
ByteArrayOutputStreambaos=newByteArrayOutputStream(60/*size*/);
56.
DataOutputStreamdos=newDataOutputStream(baos);
57.
58.
debug("toByteArray: write key into byte array stream");
59.
Bytes.writeByteArray(dos, m_keyBytes);
60.
61.
debug("toByteArray: write value into byte array stream");
62.
Bytes.writeByteArray(dos, m_valueBytes);
63.
dos.flush();
64.
baos.flush();
65.
66.
returnbaos.toByteArray();
67.
}
On 20/08/2014 06:46 p.m., Ted Yu wrote:
Can you show implementation for parseFrom(byte[]) - using pastebin ?
If possible, seeing the code for whole class would help us understand
better.
Cheers
On Wed, Aug 20, 2014 at 4:18 PM, gabriela montiel <
gabriela.mont...@oracle.com> wrote:
Hi all,
I have been working on migrating a custom filter used in HBase 0.94 to
make it work on HBase 0.96.1.1. This custom filter extends the FilterBase
API and receives only two byte arrays. According to the documentation both
toByteArray() and parseFrom(byte[]) should be implemented. After adding
this APIs, copying the jar to all region servers and restarting HBase, I
noticed that the filter is not working (all records are retrieved). I don't
get any errors, but checking out the trace the constructor and
toByteArray() APIs are only called but no other operation on the filter is
executed.
I saw that you need to use FilterProtos from the protobuf to write/read
the byte array to rebuild the filter, however there is no documentation on
how to update these APIs to add my custom filter. Do you have any idea on
what we are doing wrong?
Thanks,
Gaby
Bucket cache configuration
Hello, We are working on performance tunning on HBase 0.98 CDH 5.1 using bucket cache. We were able to set up the bucket cache configuration on all storage nodes but we wanted to know further details on how the MaxDirectMemorySize is related to the bucket cache size and the Java heap size. How does the MaxDirectMemorySize affects performance in HBase? Our current configuration is the following: hbase.bucketcache.ioengine offheap hbase-site.xml hbase.bucketcache.percentage.in.combinedcache 0.8 hbase-site.xml hbase.bucketcache.size 1 hbase-site.xml -XX:MaxDirectMemorySize=12g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:-CMSConcurrentMTEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled Java Heap Size: 12G Thanks, Gaby
Question on upper bound of column qualifiers for a Row HBASE-0.98
Hello, We are working on HBase 0.98-CDH 5.2. We are creating a table where a row may contain a great amount of column qualifiers. Does it exists a limit on the number of column qualifiers a row may contain? Does the row have a limited size? Thanks, Gaby
How to validate HBase configuration (for region servers) from HBase admin page?
Hi all, We have set up an HBase cluster with 10 region nodes. We modified the default configurations to handle bucket cache in the hbase-site.xml (for all region servers) as follows: hbase.regionserver.handler.count 40 hbase.bucketcache.ioengine offheap hbase.bucketcache.percentage.in.combinedcache 0.8 hbase.bucketcache.size 4 However, when we access the HBase configuration page on the HBase Web admin, we are not able to see the updated configurations, even for the ones not related to bucket cache: hbase.regionserver.handler.count*30*hbase-site.xml Can you give us a hint on why the configurations are not seen in the HBase configuration site, and how we can check out the HBase configuration for all region servers? Thanks, Gaby
Re: How to validate HBase configuration (for region servers) from HBase admin page?
Hi Shengzhe, We executed a rolling restart on HBase through Cloudera 5, but after completing the site retrieved the same configurations. Thanks, Gaby Sent from my Samsung Mobile Original message From: yao Date:28/07/2014 13:43 (GMT-06:00) To: dev@hbase.apache.org Subject: Re: How to validate HBase configuration (for region servers) from HBase admin page? Hi Gaby, Did you restart your regionservers ? If not, HBase won't pick up the up-to-date configuration. Best Shengzhe On Mon, Jul 28, 2014 at 11:38 AM, Ted Yu wrote: > bq. hbase-site.xml > > Can you check the hbase-site.xml which is in classpath of the region server > ? > > What release of hbase are you using ? > > Cheers > > > On Mon, Jul 28, 2014 at 11:28 AM, gabriela montiel < > gabriela.mont...@oracle.com> wrote: > > > Hi all, > > > > We have set up an HBase cluster with 10 region nodes. We modified the > > default configurations to handle bucket cache in the hbase-site.xml (for > > all region servers) as follows: > > > > > > hbase.regionserver.handler.count > > 40 > > > > > > hbase.bucketcache.ioengine > > offheap > > > > > > hbase.bucketcache.percentage.in.combinedcache > > 0.8 > > > > > > hbase.bucketcache.size > > 4 > > > > > > However, when we access the HBase configuration page on the HBase Web > > admin, we are not able to see the updated configurations, even for the > ones > > not related to bucket cache: > > > > hbase.regionserver.handler.count > name>*30*hbase-site.xml > > > > > > Can you give us a hint on why the configurations are not seen in the > HBase > > configuration site, and how we can check out the HBase configuration for > > all region servers? > > > > Thanks, > > > > Gaby > > > > > > > > > > > > > > > > > > > > >
Re: How to validate HBase configuration (for region servers) from HBase admin page?
Hi St.Ack, We are working with HBase 0.96.1.1-cdh5.0.1 using 10 region servers. We configured the bucket cache and set the -XX:MaxDirectMemorySize to 32g, heap size of 12g, and hbase.regionserver.count=40. The bucket cache configuration is the following (added in hbase-site.xml): hbase.regionserver.handler.count 40 hbase.bucketcache.ioengine offheap hbase.bucketcache.percentage.in.combinedcache 0.8 hbase.bucketcache.size 4 After restarting HBase, we checked out the HBase process from console we are able to set the -XX:MaxDirectMemorySize=32, as well as the memory heap size we configured. However, when we go to the http://:60010/conf, the*hbase.regionserver.count* is still set up to 30 (presumably the default value) and the bucketcache parameters added (bucketcache.ioengine, bucketcache.percentage.in.combinedcache, bucketcache.size) are not showed up, so we are not sure if the parameters are taken into account or not. Is there a way to validate that the bucketcache parameters as well as the new value for the hbase.regionserver.count are really taken from the hbase-site.xml we changed? Thanks, Gaby On 28/07/2014 02:23 p.m., Stack wrote: And don't forget to set -XX:MaxDirectMemorySize (from 9.6.4.5.1. Enable SlabCache in the refguide -- http://hbase.apache.org/book.html). This you set where you pass your server JVM args. St.Ack On Mon, Jul 28, 2014 at 12:21 PM, Stack wrote: Probably a question for the cloudera hbase list? (Did you enter the block cache configs via CM safety valve mechanism?) Yours, St.Ack On Mon, Jul 28, 2014 at 11:54 AM, Gabriela Montiel < gabriela.mont...@oracle.com> wrote: Hi Shengzhe, We executed a rolling restart on HBase through Cloudera 5, but after completing the site retrieved the same configurations. Thanks, Gaby Sent from my Samsung Mobile Original message From: yao Date:28/07/2014 13:43 (GMT-06:00) To: dev@hbase.apache.org Subject: Re: How to validate HBase configuration (for region servers) from HBase admin page? Hi Gaby, Did you restart your regionservers ? If not, HBase won't pick up the up-to-date configuration. Best Shengzhe On Mon, Jul 28, 2014 at 11:38 AM, Ted Yu wrote: bq. hbase-site.xml Can you check the hbase-site.xml which is in classpath of the region server ? What release of hbase are you using ? Cheers On Mon, Jul 28, 2014 at 11:28 AM, gabriela montiel < gabriela.mont...@oracle.com> wrote: Hi all, We have set up an HBase cluster with 10 region nodes. We modified the default configurations to handle bucket cache in the hbase-site.xml (for all region servers) as follows: hbase.regionserver.handler.count 40 hbase.bucketcache.ioengine offheap hbase.bucketcache.percentage.in.combinedcache 0.8 hbase.bucketcache.size 4 However, when we access the HBase configuration page on the HBase Web admin, we are not able to see the updated configurations, even for the ones not related to bucket cache: hbase.regionserver.handler.count*30*hbase-site.xml Can you give us a hint on why the configurations are not seen in the HBase configuration site, and how we can check out the HBase configuration for all region servers? Thanks, Gaby -- Regards, Gaby <http://www.oracle.com> Gabriela Montiel Moreno | Senior Member of Technical Staff Phone: +52 33 12531343 Oracle Semantic Technoliges ORACLE Mexico <http://www.oracle.com/commitment> Oracle is committed to developing practices and products that help protect the environment
Problem with Scan operation over a Secure HBase 1.0.0-CDH5.4
Hello,
We are running a Secure HBase cluster (enabling kerberos authentication and
setting up hbase authorization) and we are trying to execute operations using a
Java client. We are using the following configuration.
import org.apache.hadoop.security.*;
szQuorum="node01.example.com,node02.example.com,node01.example.com";
config = HBaseConfiguration.create();
config.set("hbase.zookeeper.quorum", szQuorum);
config.set("hbase.zookeeper.property.clientPort", "2181");
config.set("hbase.security.authentication", "kerberos");
config.set("hadoop.security.authentication", "kerberos");
config.set("hbase.master.kerberos.principal","hbase/node03.example@example.com");
config.set("hbase.regionserver.kerberos.principal","hbase/node03.example@example.com");
UserGroupInformation.setConfiguration(config);
ugi =
UserGroupInformation.loginUserFromKeytabAndReturnUGI("hbase/node03.example@example.com","/var/run/cloudera-scm-agent/process/224-hbase-REGIONSERVER/hbase.keytab");
UserGroupInformation.setLoginUser(ugi);
hconn = HConnectionManager.createConnection(config);
hti=conn.getTable("exampletbl");
scan = new Scan();
rsScanner=hti.getScanner(scan);
While we are able to create a table, puts and gets, when we try to execute a
scan after a few seconds we get the following exceptions:
97976 [hconnection-0x4f2c9ba6-shared--pool1-t6] DEBUG
org.apache.hadoop.hbase.security.HBaseSaslRpcClient - Have sent token of size
674 from initSASLContext.
97977 [hconnection-0x4f2c9ba6-shared--pool1-t6] WARN
org.apache.hadoop.security.UserGroupInformation -
PriviledgedActionExceptionas:hbase/node01.example@example.com(auth:KERBEROS)
cause:org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
GSS initiate failed
97977 [hconnection-0x4f2c9ba6-shared--pool1-t6] DEBUG
org.apache.hadoop.security.UserGroupInformation - PrivilegedAction
as:hbase/node01.example@example.com(auth:KERBEROS)
from:org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.handleSaslConnectionFailure(RpcClientImpl.java:631)
97977 [hconnection-0x4f2c9ba6-shared--pool1-t6] WARN
org.apache.hadoop.hbase.ipc.AbstractRpcClient - Couldn't setup connection for
hbase/node01.example@example.com tohbase/node01.example@example.com
We have run the kinit and setup the jaas.conf in the JAVA_OPTIONS of our Java
application.
export JAVA_OPTIONS="
-Djava.security.auth.login.config=/var/run/cloudera-scm-agent/process/224-hbase-REGIONSERVER/jaas.conf
-Dsun.security.krb5.debug=true "
kinit -k -t
/var/run/cloudera-scm-agent/process/224-hbase-REGIONSERVER/hbase.keytabhbase/hbase/node03.example@example.com
klist -f
Ticket cache: FILE:/tmp/krb5cc_0
Default principal:hbase/hbase/node03.example@example.com
Valid starting ExpiresService principal
06/17/15 17:37:31 06/18/15 17:37:31 krbtgt/example@example.com
renew until 06/22/15 17:37:31, Flags: FRI
less /var/run/cloudera-scm-agent/process/224-hbase-REGIONSERVER/jaas.conf
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
useTicketCache=true
keyTab="hbase.keytab"
principal="hbase/hbase/node03.example@example.com";
};
Is there any missing configuration?
Thanks,
Gaby
