See this blog post: http://www.flurry.com/2012/12/06/exploring-dynamic-loading-of-custom-filters-i#.VEcNtNR4rZg
Cheers On Tue, Oct 21, 2014 at 6:48 PM, Kevin <kevin.macksa...@gmail.com> wrote: > Also, if you do end up using dynamic loading, you'll need a way to version > your filters because the RS will not reload a JAR if it changes. > > On Tue, Oct 21, 2014 at 9:46 PM, Kevin <kevin.macksa...@gmail.com> wrote: > > > I haven't tried dynamic loading of filters on RS, but I know it does > > exist. See https://issues.apache.org/jira/browse/HBASE-9301. > > > > If you still can't get it to work, then I suggest distributing your > > filters to the RS and restart them. Let us know how everything works out. > > > > On Tue, Oct 21, 2014 at 9:02 PM, Matt K <matvey1...@gmail.com> wrote: > > > >> Thanks Kevin! > >> > >> I was under impression, probably mistakingly, that as of 0.96 placing > >> the filter on hdfs under hbase lib directory is sufficient and RS should > >> load the filter dynamically from hdfs. Is that not the case? > >> > >> On Tuesday, October 21, 2014, Kevin <kevin.macksa...@gmail.com> wrote: > >> > >> > BTW, the error looks like you didn't distribute your custom filter to > >> your > >> > region servers. > >> > > >> > On Tue, Oct 21, 2014 at 1:34 PM, Kevin <kevin.macksa...@gmail.com > >> > <javascript:;>> wrote: > >> > > >> > > Matt, > >> > > > >> > > You should create your own proto file and compile that with the > Google > >> > > Protocol Buffer compiler. Take a look at the > SingleColumnValueFilter's > >> > > code: > >> > > > >> > > >> > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java#L327 > >> > > > >> > > You will need to override `public byte[] toByteArray()` and `public > >> > static > >> > > Filter parseFrom(byte[] pbBytes)`. The output of toByteArray() > should > >> be > >> > > the byte[] from serializing with the protocol buffer. This will also > >> be > >> > the > >> > > input to parseFrom(byte[]), which is called using reflection on the > >> > > server-side to instantiate your custom filter and use it. > >> > > > >> > > On Sun, Oct 19, 2014 at 11:31 AM, Matt K <matvey1...@gmail.com > >> > <javascript:;>> wrote: > >> > > > >> > >> Anyone? > >> > >> > >> > >> On Thursday, October 16, 2014, Matt K <matvey1...@gmail.com > >> > <javascript:;>> wrote: > >> > >> > >> > >> > Hi, can anyone help with above? Feels like I'm missing something > >> > >> obvious. > >> > >> > > >> > >> > On Wednesday, October 15, 2014, Nishanth S < > >> nishanth.2...@gmail.com > >> > <javascript:;> > >> > >> > <javascript:_e(%7B%7D,'cvml','nishanth.2...@gmail.com > >> <javascript:;>');>> > >> > wrote: > >> > >> > > >> > >> >> Thanks Ted .I will take a look. > >> > >> >> > >> > >> >> -Nishanth > >> > >> >> > >> > >> >> On Wed, Oct 15, 2014 at 3:43 PM, Ted Yu <yuzhih...@gmail.com > >> > <javascript:;>> wrote: > >> > >> >> > >> > >> >> > Nishanth: > >> > >> >> > Good question. > >> > >> >> > > >> > >> >> > As a general coding guide, writing unit test is always a good > >> > start. > >> > >> >> Using > >> > >> >> > Matt's case as an example, take a look at TestPrefixFilter. > >> > >> >> > > >> > >> >> > There're various unit tests for Filters in hbase code. > >> > >> >> > > >> > >> >> > Cheers > >> > >> >> > > >> > >> >> > On Wed, Oct 15, 2014 at 2:30 PM, Nishanth S < > >> > nishanth.2...@gmail.com <javascript:;> > >> > >> > > >> > >> >> > wrote: > >> > >> >> > > >> > >> >> > > Hi Ted , > >> > >> >> > > Since I am also working on similar thing is there a way we > >> can > >> > >> first > >> > >> >> > test > >> > >> >> > > the filter on client side?.You know what I mean without > >> > disrupting > >> > >> >> > others > >> > >> >> > > who are using the same cluster for other work? > >> > >> >> > > > >> > >> >> > > Thanks, > >> > >> >> > > Nishanth > >> > >> >> > > > >> > >> >> > > On Wed, Oct 15, 2014 at 3:17 PM, Ted Yu < > yuzhih...@gmail.com > >> > <javascript:;>> > >> > >> wrote: > >> > >> >> > > > >> > >> >> > > > bq. Or create a new file, compile it into ... > >> > >> >> > > > > >> > >> >> > > > You should go with the above approach. > >> > >> >> > > > > >> > >> >> > > > On Wed, Oct 15, 2014 at 2:08 PM, Matt K < > >> matvey1...@gmail.com > >> > <javascript:;>> > >> > >> >> wrote: > >> > >> >> > > > > >> > >> >> > > > > Hi all, > >> > >> >> > > > > > >> > >> >> > > > > I'm trying to get a custom filter to work on HBase 0.96. > >> > After > >> > >> >> some > >> > >> >> > > > > searching, I found that starting from 0.96, the > >> implementer > >> > is > >> > >> >> > required > >> > >> >> > > > to > >> > >> >> > > > > implement "toByteArray" and "parseFrom" methods, using > >> > Protocol > >> > >> >> > > Buffers. > >> > >> >> > > > > But I'm having trouble with the "how". > >> > >> >> > > > > > >> > >> >> > > > > The "proto" file for the existing filters is located > here: > >> > >> >> > > > > > >> > >> >> > > > > > >> > >> >> > > > > >> > >> >> > > > >> > >> >> > > >> > >> >> > >> > >> > >> > > >> > https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/protobuf/Filter.proto > >> > >> >> > > > > > >> > >> >> > > > > Am I supposed to modify that file? Or create a new file, > >> > >> compile > >> > >> >> it > >> > >> >> > > into > >> > >> >> > > > > Java, and package it up with the filter? > >> > >> >> > > > > > >> > >> >> > > > > In the meantime, I've taken a shortcut that's not > working. > >> > >> Here's > >> > >> >> my > >> > >> >> > > > code: > >> > >> >> > > > > http://pastebin.com/iHFKu9Xz > >> > >> >> > > > > > >> > >> >> > > > > I'm using "PrefixFilter", which comes with HBase, since > >> I'm > >> > >> also > >> > >> >> > > > filtering > >> > >> >> > > > > by "prefix". However, that errors out with the > following: > >> > >> >> > > > > http://pastebin.com/zBg47p6Z > >> > >> >> > > > > > >> > >> >> > > > > Thanks in advance for helping! > >> > >> >> > > > > > >> > >> >> > > > > -Matt > >> > >> >> > > > > > >> > >> >> > > > > >> > >> >> > > > >> > >> >> > > >> > >> >> > >> > >> > > >> > >> > > >> > >> > -- > >> > >> > www.calcmachine.com - easy online calculator. > >> > >> > > >> > >> > >> > >> > >> > >> -- > >> > >> www.calcmachine.com - easy online calculator. > >> > >> > >> > > > >> > > > >> > > >> > >> > >> -- > >> www.calcmachine.com - easy online calculator. > >> > > > > >
