Re: DISCUSSION: lets do a developer workshop on near-term work

[ramkrishna vasudevan]

Hi Andy

Based on our POCs done, we expect around 20% improvement in latency.  For
scans it will be little lesser than 20%.

Regards
Ram


On Sun, Jul 19, 2015 at 10:20 AM, Andrew Purtell 
wrote:

> Hi Ram,
>
> Do you have any targets for what you are measuring? What are the goals you
> guys are working toward with the off heaping changes?
>
>
> > On Jul 18, 2015, at 9:16 PM, ramkrishna vasudevan <
> ramkrishna.s.vasude...@gmail.com> wrote:
> >
> > Thanks Vladimir.
> > Yeah, the reports that were attached specifically captured the 95/99th
> > percentile.
> > The reason for checking the server side perf was to specifically see the
> > improvement in the server side and also the client was sending large
> > results in multiple threads. So wanted to avoid the n/w interference. I
> > think it was a general practice that we were following.
> > We Wil do some more tests and get some latest readings with bigger data
> > sets.
> > Sent from mobile.
> >> On Jul 19, 2015 1:05 AM, "Andrew Purtell" 
> wrote:
> >>
> >> +1
> >>
> >> Yeah, something like that, with aspirational targets for improvement
> from
> >> current releases. Then what to measure, the tests to run, and criteria
> for
> >> evaluation are clear and organized and we're able to better assess how
> the
> >> work in progress is meeting its goals (or not)
> >>
> >>
> >>
> >> On Jul 18, 2015, at 12:05 PM, Vladimir Rodionov  >
> >> wrote:
> >>
> > Umbrella jira to make sure we can have blocks cached in offheap
> backed
> >>> cache. In the entire read path, we can refer to this offheap buffer and
> >>> avoid onheap copying.
> >>>
> >>> I think, on a read path, the most important improvement we could
> imagine
> >> is
> >>> elimination or reducing of object creations (KVs, iterators etc).
> >>> object reuse, byte buffers reuse or offheap buffers reuse, API change
> >> etc.
> >>> If this is a part of this JIRA, then I would easily define a goal:
> >>> improving 95/99% latency of a read operations. Not performance, but
> >> latency
> >>> matters
> >>>
> >>> -Vlad
> >>>
> >>>
> >>>
> >>> On Sat, Jul 18, 2015 at 11:24 AM, Andrew Purtell <
> >> andrew.purt...@gmail.com>
> >>> wrote:
> >>>
>  That's not a realistic or useful test scenario, unless the goal is to
>  accelerate queries where all cells are filtered at the server.
> 
> 
> 
> > On Jul 18, 2015, at 11:02 AM, Anoop John 
> >> wrote:
> >
> > No Andy. 11425 having doc attached to it. At the end of it, we have
> >> added
> > perf numbers in a cluster testing.  This was done using PE get and
> scan
> > tests with filtering all cells at server (to not consider n/w
> bandwidth
> > constraints)
> >
> > -Anoop-
> >
> > On Sat, Jul 18, 2015 at 9:30 PM, Andrew Purtell <
>  andrew.purt...@gmail.com>
> > wrote:
> >
> >> We have some microbenchmarks, not evidence of differences seen from
> a
> >> client application. I'm not saying that microbenchmarks are not
> >> totally
> >> necessary and a great start - they are - but that they don't measure
> >> an
>  end
> >> goal. Furthermore unless I've missed one somewhere we don't have a
> >> JIRA
>  or
> >> design doc that states a clear end goal metric like the strawman I
> >> threw
> >> together in my previous mail. A measurable system level goal and
> some
>  data
> >> from full cluster testing would go a lot further toward letting all
> of
>  us
> >> evaluate the potential and payoff of the work. In the meantime we
> >> should
> >> probably be assembling these changes on a branch instead of in
> trunk,
>  for
> >> as long as the goal is not clearly defined and the payoff and
> >> potential
>  for
> >> perf regressions is untested and unknown.
> >>
> >>
> >>> On Jul 18, 2015, at 8:05 AM, Anoop John 
> >> wrote:
> >>>
> >>> Thanks Andy and Lars.  The parent jira has doc attached which
> >> contains
> >> some
> >>> perf gain numbers..  We will be doing more tests in next 2 weeks
>  (before
> >>> end of this month) and will publish them.   Yes it will be great if
> >> it
>  is
> >>> more IST friendly time :-)
> >>>
> >>> -Anoop-
> >>>
> >>> On Fri, Jul 17, 2015 at 9:44 PM, Andrew Purtell <
> >> andrew.purt...@gmail.com>
> >>> wrote:
> >>>
> > I can represent your side Ram (and Anoop). I've been known always
>  argue
>  both side of a discussion and to never take sides easily (drives
> >> some
> >> folks
>  crazy).
> 
>  I can vouch for this (smile)
> 
>  I also can offer support for off heaping there. At the same time
> we
> >> do
>  have a gap where we can't point to a timeline of improvements
> (yet,
> >> anyway)
>  with benchmarks showing gains where your goals need them. For
> >> example,
>  stock HBase in one JVM can address max N GB for response time
> >> distribut

Re: DISCUSSION: lets do a developer workshop on near-term work

[Sean Busbey]

Can y'all move discussion of the off heaping work (or perf feature dev
generally) to a new thread?

-- 
Sean
On Jul 20, 2015 6:44 AM, "ramkrishna vasudevan" <
ramkrishna.s.vasude...@gmail.com> wrote:

> Hi Andy
>
> Based on our POCs done, we expect around 20% improvement in latency.  For
> scans it will be little lesser than 20%.
>
> Regards
> Ram
>
>
> On Sun, Jul 19, 2015 at 10:20 AM, Andrew Purtell  >
> wrote:
>
> > Hi Ram,
> >
> > Do you have any targets for what you are measuring? What are the goals
> you
> > guys are working toward with the off heaping changes?
> >
> >
> > > On Jul 18, 2015, at 9:16 PM, ramkrishna vasudevan <
> > ramkrishna.s.vasude...@gmail.com> wrote:
> > >
> > > Thanks Vladimir.
> > > Yeah, the reports that were attached specifically captured the 95/99th
> > > percentile.
> > > The reason for checking the server side perf was to specifically see
> the
> > > improvement in the server side and also the client was sending large
> > > results in multiple threads. So wanted to avoid the n/w interference. I
> > > think it was a general practice that we were following.
> > > We Wil do some more tests and get some latest readings with bigger data
> > > sets.
> > > Sent from mobile.
> > >> On Jul 19, 2015 1:05 AM, "Andrew Purtell" 
> > wrote:
> > >>
> > >> +1
> > >>
> > >> Yeah, something like that, with aspirational targets for improvement
> > from
> > >> current releases. Then what to measure, the tests to run, and criteria
> > for
> > >> evaluation are clear and organized and we're able to better assess how
> > the
> > >> work in progress is meeting its goals (or not)
> > >>
> > >>
> > >>
> > >> On Jul 18, 2015, at 12:05 PM, Vladimir Rodionov <
> vladrodio...@gmail.com
> > >
> > >> wrote:
> > >>
> > > Umbrella jira to make sure we can have blocks cached in offheap
> > backed
> > >>> cache. In the entire read path, we can refer to this offheap buffer
> and
> > >>> avoid onheap copying.
> > >>>
> > >>> I think, on a read path, the most important improvement we could
> > imagine
> > >> is
> > >>> elimination or reducing of object creations (KVs, iterators etc).
> > >>> object reuse, byte buffers reuse or offheap buffers reuse, API change
> > >> etc.
> > >>> If this is a part of this JIRA, then I would easily define a goal:
> > >>> improving 95/99% latency of a read operations. Not performance, but
> > >> latency
> > >>> matters
> > >>>
> > >>> -Vlad
> > >>>
> > >>>
> > >>>
> > >>> On Sat, Jul 18, 2015 at 11:24 AM, Andrew Purtell <
> > >> andrew.purt...@gmail.com>
> > >>> wrote:
> > >>>
> >  That's not a realistic or useful test scenario, unless the goal is
> to
> >  accelerate queries where all cells are filtered at the server.
> > 
> > 
> > 
> > > On Jul 18, 2015, at 11:02 AM, Anoop John 
> > >> wrote:
> > >
> > > No Andy. 11425 having doc attached to it. At the end of it, we have
> > >> added
> > > perf numbers in a cluster testing.  This was done using PE get and
> > scan
> > > tests with filtering all cells at server (to not consider n/w
> > bandwidth
> > > constraints)
> > >
> > > -Anoop-
> > >
> > > On Sat, Jul 18, 2015 at 9:30 PM, Andrew Purtell <
> >  andrew.purt...@gmail.com>
> > > wrote:
> > >
> > >> We have some microbenchmarks, not evidence of differences seen
> from
> > a
> > >> client application. I'm not saying that microbenchmarks are not
> > >> totally
> > >> necessary and a great start - they are - but that they don't
> measure
> > >> an
> >  end
> > >> goal. Furthermore unless I've missed one somewhere we don't have a
> > >> JIRA
> >  or
> > >> design doc that states a clear end goal metric like the strawman I
> > >> threw
> > >> together in my previous mail. A measurable system level goal and
> > some
> >  data
> > >> from full cluster testing would go a lot further toward letting
> all
> > of
> >  us
> > >> evaluate the potential and payoff of the work. In the meantime we
> > >> should
> > >> probably be assembling these changes on a branch instead of in
> > trunk,
> >  for
> > >> as long as the goal is not clearly defined and the payoff and
> > >> potential
> >  for
> > >> perf regressions is untested and unknown.
> > >>
> > >>
> > >>> On Jul 18, 2015, at 8:05 AM, Anoop John 
> > >> wrote:
> > >>>
> > >>> Thanks Andy and Lars.  The parent jira has doc attached which
> > >> contains
> > >> some
> > >>> perf gain numbers..  We will be doing more tests in next 2 weeks
> >  (before
> > >>> end of this month) and will publish them.   Yes it will be great
> if
> > >> it
> >  is
> > >>> more IST friendly time :-)
> > >>>
> > >>> -Anoop-
> > >>>
> > >>> On Fri, Jul 17, 2015 at 9:44 PM, Andrew Purtell <
> > >> andrew.purt...@gmail.com>
> > >>> wrote:
> > >>>
> > > I can represent your side Ram (and Anoop). I've been known
> always
> >  argue
> > >>>

Re: DISCUSSION: lets do a developer workshop on near-term work

[lars hofhansl]

Personally, I think that is a reasonable way to test the internal friction of 
the server. I've been doing a lot of tests like that and found a lot of 
inefficiencies in HBase that way.For cases where we return all Cells back to a 
(remote) client improving the server by 10 or 20% would mostly go unnoticed.

Analytics (aggregates via Phoenix of direct coprocessors) will be more 
important going forward, so improving that part is important.
I completely agree that end-to-end (by which I mean data shipped to the client) 
testing is important, it's just I'd expect us to work on different areas (put 
Protobufs on a diet, have a streaming protocol, etc).
-- Lars

 From: Andrew Purtell 
 To: "dev@hbase.apache.org"  
 Sent: Saturday, July 18, 2015 11:24 AM
 Subject: Re: DISCUSSION: lets do a developer workshop on near-term work
   
That's not a realistic or useful test scenario, unless the goal is to 
accelerate queries where all cells are filtered at the server. 





> On Jul 18, 2015, at 11:02 AM, Anoop John  wrote:
> 
> No Andy. 11425 having doc attached to it. At the end of it, we have added
> perf numbers in a cluster testing.  This was done using PE get and scan
> tests with filtering all cells at server (to not consider n/w bandwidth
> constraints)
> 
> -Anoop-
> 
> On Sat, Jul 18, 2015 at 9:30 PM, Andrew Purtell 
> wrote:
> 
>> We have some microbenchmarks, not evidence of differences seen from a
>> client application. I'm not saying that microbenchmarks are not totally
>> necessary and a great start - they are - but that they don't measure an end
>> goal. Furthermore unless I've missed one somewhere we don't have a JIRA or
>> design doc that states a clear end goal metric like the strawman I threw
>> together in my previous mail. A measurable system level goal and some data
>> from full cluster testing would go a lot further toward letting all of us
>> evaluate the potential and payoff of the work. In the meantime we should
>> probably be assembling these changes on a branch instead of in trunk, for
>> as long as the goal is not clearly defined and the payoff and potential for
>> perf regressions is untested and unknown.
>> 
>> 
>>> On Jul 18, 2015, at 8:05 AM, Anoop John  wrote:
>>> 
>>> Thanks Andy and Lars.  The parent jira has doc attached which contains
>> some
>>> perf gain numbers..  We will be doing more tests in next 2 weeks (before
>>> end of this month) and will publish them.  Yes it will be great if it is
>>> more IST friendly time :-)
>>> 
>>> -Anoop-
>>> 
>>> On Fri, Jul 17, 2015 at 9:44 PM, Andrew Purtell <
>> andrew.purt...@gmail.com>
>>> wrote:
>>> 
> I can represent your side Ram (and Anoop). I've been known always argue
 both side of a discussion and to never take sides easily (drives some
>> folks
 crazy).
 
 I can vouch for this (smile)
 
 I also can offer support for off heaping there. At the same time we do
 have a gap where we can't point to a timeline of improvements (yet,
>> anyway)
 with benchmarks showing gains where your goals need them. For example,
 stock HBase in one JVM can address max N GB for response time
>> distribution
 D; dev version of HBase in off heap branch can address max N' GB for
 distribution D', where N' > N and D > D' (distribution D' statistically
 shows better/lower response times).
 
 
 
> On Jul 17, 2015, at 6:56 AM, lars hofhansl  wrote:
> 
> I'm in favor of anything that improves performance (and preferably
 doesn't set us back into a world that's worse than C due to the lack of
 pointers in Java).Never said "I don't like it", it's just that I'm
>> perhaps
 asking for more numbers and justification in weighing the pros and cons.
> I can represent your side Ram (and Anoop). I've been known always argue
 both side of a discussion and to never take sides easily (drives some
>> folks
 crazy). And Stack's there too, he yell at me where needed :)
> 
> Perhaps we can do it a bit later in the evening so there is a fighting
 chance that folks on IST can participate. I know that some of our folks
>> on
 IST would love to participate in the backup discussion).
> 
> Like Enis, I'm also happy to host. We're in Downtown SF. I'd just need
 an approx. number of folks.
> 
> -- Lars
> 
>    From: ramkrishna vasudevan 
> To: "dev@hbase.apache.org" ; lars hofhansl <
 la...@apache.org>
> Sent: Wednesday, July 15, 2015 10:10 AM
> Subject: Re: DISCUSSION: lets do a developer workshop on near-term work
> 
> Hi
> What time will it be on August 26th?
> @LarsYa. I know that you are not generally in favour of this offheaping
 stuff.  May be if we (from India) can attend this meeting remotely your
 thoughts can be discussed and also the current state of this work.
> RegardsRam
> 
> 
> On Wed, Jul 15, 2015 at 9:28 PM, lars hofhansl 
>> wrote:
> 
> Works for me. I'll be b

Re: DISCUSSION: lets do a developer workshop on near-term work

[Andrew Purtell]

Returning all cells to a client is the other extreme and I don't think that 
would be a great test either. 

Personally I think for testing big change sets well we need a range of 
workloads. The extreme cases (filter all, filter none) are useful data points 
but not great if measured in isolation. I think YCSB is a reasonable option for 
that these days now that it is maintained. It comes with 6 or so canned 
workloads. Not a bad start.


> On Jul 20, 2015, at 6:01 AM, lars hofhansl  wrote:
> 
> Personally, I think that is a reasonable way to test the internal friction of 
> the server. I've been doing a lot of tests like that and found a lot of 
> inefficiencies in HBase that way.For cases where we return all Cells back to 
> a (remote) client improving the server by 10 or 20% would mostly go unnoticed.
> 
> Analytics (aggregates via Phoenix of direct coprocessors) will be more 
> important going forward, so improving that part is important.
> I completely agree that end-to-end (by which I mean data shipped to the 
> client) testing is important, it's just I'd expect us to work on different 
> areas (put Protobufs on a diet, have a streaming protocol, etc).
> -- Lars
> 
> From: Andrew Purtell 
> To: "dev@hbase.apache.org"  
> Sent: Saturday, July 18, 2015 11:24 AM
> Subject: Re: DISCUSSION: lets do a developer workshop on near-term work
> 
> That's not a realistic or useful test scenario, unless the goal is to 
> accelerate queries where all cells are filtered at the server. 
> 
> 
> 
> 
> 
>> On Jul 18, 2015, at 11:02 AM, Anoop John  wrote:
>> 
>> No Andy. 11425 having doc attached to it. At the end of it, we have added
>> perf numbers in a cluster testing.  This was done using PE get and scan
>> tests with filtering all cells at server (to not consider n/w bandwidth
>> constraints)
>> 
>> -Anoop-
>> 
>> On Sat, Jul 18, 2015 at 9:30 PM, Andrew Purtell 
>> wrote:
>> 
>>> We have some microbenchmarks, not evidence of differences seen from a
>>> client application. I'm not saying that microbenchmarks are not totally
>>> necessary and a great start - they are - but that they don't measure an end
>>> goal. Furthermore unless I've missed one somewhere we don't have a JIRA or
>>> design doc that states a clear end goal metric like the strawman I threw
>>> together in my previous mail. A measurable system level goal and some data
>>> from full cluster testing would go a lot further toward letting all of us
>>> evaluate the potential and payoff of the work. In the meantime we should
>>> probably be assembling these changes on a branch instead of in trunk, for
>>> as long as the goal is not clearly defined and the payoff and potential for
>>> perf regressions is untested and unknown.
>>> 
>>> 
 On Jul 18, 2015, at 8:05 AM, Anoop John  wrote:
 
 Thanks Andy and Lars.  The parent jira has doc attached which contains
>>> some
 perf gain numbers..  We will be doing more tests in next 2 weeks (before
 end of this month) and will publish them.  Yes it will be great if it is
 more IST friendly time :-)
 
 -Anoop-
 
 On Fri, Jul 17, 2015 at 9:44 PM, Andrew Purtell <
>>> andrew.purt...@gmail.com>
 wrote:
 
>> I can represent your side Ram (and Anoop). I've been known always argue
> both side of a discussion and to never take sides easily (drives some
>>> folks
> crazy).
> 
> I can vouch for this (smile)
> 
> I also can offer support for off heaping there. At the same time we do
> have a gap where we can't point to a timeline of improvements (yet,
>>> anyway)
> with benchmarks showing gains where your goals need them. For example,
> stock HBase in one JVM can address max N GB for response time
>>> distribution
> D; dev version of HBase in off heap branch can address max N' GB for
> distribution D', where N' > N and D > D' (distribution D' statistically
> shows better/lower response times).
> 
> 
> 
>> On Jul 17, 2015, at 6:56 AM, lars hofhansl  wrote:
>> 
>> I'm in favor of anything that improves performance (and preferably
> doesn't set us back into a world that's worse than C due to the lack of
> pointers in Java).Never said "I don't like it", it's just that I'm
>>> perhaps
> asking for more numbers and justification in weighing the pros and cons.
>> I can represent your side Ram (and Anoop). I've been known always argue
> both side of a discussion and to never take sides easily (drives some
>>> folks
> crazy). And Stack's there too, he yell at me where needed :)
>> 
>> Perhaps we can do it a bit later in the evening so there is a fighting
> chance that folks on IST can participate. I know that some of our folks
>>> on
> IST would love to participate in the backup discussion).
>> 
>> Like Enis, I'm also happy to host. We're in Downtown SF. I'd just need
> an approx. number of folks.
>> 
>> -- Lars
>> 
>> From: ramkrishna vas

Re: DISCUSSION: lets do a developer workshop on near-term work

[Andrew Purtell]

Cool, thanks. 

Is a 20% latency reduction the most we can expect or do you think there is room 
for more improvement? Just curious. 

Is latency reduction the only goal? Anything here about supporting larger 
heaps? Is there something we can measure in that regard?

Hope you see my point and there's enough here to prime a goals and metrics 
discussion at the pow wow or on the relevant JIRAs. 

> On Jul 20, 2015, at 4:43 AM, ramkrishna vasudevan 
>  wrote:
> 
> Hi Andy
> 
> Based on our POCs done, we expect around 20% improvement in latency.  For
> scans it will be little lesser than 20%.
> 
> Regards
> Ram
> 
> 
> On Sun, Jul 19, 2015 at 10:20 AM, Andrew Purtell 
> wrote:
> 
>> Hi Ram,
>> 
>> Do you have any targets for what you are measuring? What are the goals you
>> guys are working toward with the off heaping changes?
>> 
>> 
 On Jul 18, 2015, at 9:16 PM, ramkrishna vasudevan <
>>> ramkrishna.s.vasude...@gmail.com> wrote:
>>> 
>>> Thanks Vladimir.
>>> Yeah, the reports that were attached specifically captured the 95/99th
>>> percentile.
>>> The reason for checking the server side perf was to specifically see the
>>> improvement in the server side and also the client was sending large
>>> results in multiple threads. So wanted to avoid the n/w interference. I
>>> think it was a general practice that we were following.
>>> We Wil do some more tests and get some latest readings with bigger data
>>> sets.
>>> Sent from mobile.
 On Jul 19, 2015 1:05 AM, "Andrew Purtell" 
>> wrote:
 
 +1
 
 Yeah, something like that, with aspirational targets for improvement
>> from
 current releases. Then what to measure, the tests to run, and criteria
>> for
 evaluation are clear and organized and we're able to better assess how
>> the
 work in progress is meeting its goals (or not)
 
 
 
 On Jul 18, 2015, at 12:05 PM, Vladimir Rodionov >> 
 wrote:
 
>>> Umbrella jira to make sure we can have blocks cached in offheap
>> backed
> cache. In the entire read path, we can refer to this offheap buffer and
> avoid onheap copying.
> 
> I think, on a read path, the most important improvement we could
>> imagine
 is
> elimination or reducing of object creations (KVs, iterators etc).
> object reuse, byte buffers reuse or offheap buffers reuse, API change
 etc.
> If this is a part of this JIRA, then I would easily define a goal:
> improving 95/99% latency of a read operations. Not performance, but
 latency
> matters
> 
> -Vlad
> 
> 
> 
> On Sat, Jul 18, 2015 at 11:24 AM, Andrew Purtell <
 andrew.purt...@gmail.com>
> wrote:
> 
>> That's not a realistic or useful test scenario, unless the goal is to
>> accelerate queries where all cells are filtered at the server.
>> 
>> 
>> 
>>> On Jul 18, 2015, at 11:02 AM, Anoop John 
 wrote:
>>> 
>>> No Andy. 11425 having doc attached to it. At the end of it, we have
 added
>>> perf numbers in a cluster testing.  This was done using PE get and
>> scan
>>> tests with filtering all cells at server (to not consider n/w
>> bandwidth
>>> constraints)
>>> 
>>> -Anoop-
>>> 
>>> On Sat, Jul 18, 2015 at 9:30 PM, Andrew Purtell <
>> andrew.purt...@gmail.com>
>>> wrote:
>>> 
 We have some microbenchmarks, not evidence of differences seen from
>> a
 client application. I'm not saying that microbenchmarks are not
 totally
 necessary and a great start - they are - but that they don't measure
 an
>> end
 goal. Furthermore unless I've missed one somewhere we don't have a
 JIRA
>> or
 design doc that states a clear end goal metric like the strawman I
 threw
 together in my previous mail. A measurable system level goal and
>> some
>> data
 from full cluster testing would go a lot further toward letting all
>> of
>> us
 evaluate the potential and payoff of the work. In the meantime we
 should
 probably be assembling these changes on a branch instead of in
>> trunk,
>> for
 as long as the goal is not clearly defined and the payoff and
 potential
>> for
 perf regressions is untested and unknown.
 
 
> On Jul 18, 2015, at 8:05 AM, Anoop John 
 wrote:
> 
> Thanks Andy and Lars.  The parent jira has doc attached which
 contains
 some
> perf gain numbers..  We will be doing more tests in next 2 weeks
>> (before
> end of this month) and will publish them.   Yes it will be great if
 it
>> is
> more IST friendly time :-)
> 
> -Anoop-
> 
> On Fri, Jul 17, 2015 at 9:44 PM, Andrew Purtell <
 andrew.purt...@gmail.com>
> wrote:
> 
>>> I can represent your side Ram (and Anoop). I've been known always
>> argue

Re: DISCUSSION: lets do a developer workshop on near-term work

[Anoop John]

We will be doing some more large data tests in coming week Andy..   Will
report back more.  Also will do a write up , in what all ways the work
might help us.  As Sean said, we will continue in another thread if any
thing further..  Will soon write back on the test result.  Thanks.

-Anoop-

On Mon, Jul 20, 2015 at 9:59 PM, Andrew Purtell 
wrote:

> Cool, thanks.
>
> Is a 20% latency reduction the most we can expect or do you think there is
> room for more improvement? Just curious.
>
> Is latency reduction the only goal? Anything here about supporting larger
> heaps? Is there something we can measure in that regard?
>
> Hope you see my point and there's enough here to prime a goals and metrics
> discussion at the pow wow or on the relevant JIRAs.
>
> > On Jul 20, 2015, at 4:43 AM, ramkrishna vasudevan <
> ramkrishna.s.vasude...@gmail.com> wrote:
> >
> > Hi Andy
> >
> > Based on our POCs done, we expect around 20% improvement in latency.  For
> > scans it will be little lesser than 20%.
> >
> > Regards
> > Ram
> >
> >
> > On Sun, Jul 19, 2015 at 10:20 AM, Andrew Purtell <
> andrew.purt...@gmail.com>
> > wrote:
> >
> >> Hi Ram,
> >>
> >> Do you have any targets for what you are measuring? What are the goals
> you
> >> guys are working toward with the off heaping changes?
> >>
> >>
>  On Jul 18, 2015, at 9:16 PM, ramkrishna vasudevan <
> >>> ramkrishna.s.vasude...@gmail.com> wrote:
> >>>
> >>> Thanks Vladimir.
> >>> Yeah, the reports that were attached specifically captured the 95/99th
> >>> percentile.
> >>> The reason for checking the server side perf was to specifically see
> the
> >>> improvement in the server side and also the client was sending large
> >>> results in multiple threads. So wanted to avoid the n/w interference. I
> >>> think it was a general practice that we were following.
> >>> We Wil do some more tests and get some latest readings with bigger data
> >>> sets.
> >>> Sent from mobile.
>  On Jul 19, 2015 1:05 AM, "Andrew Purtell" 
> >> wrote:
> 
>  +1
> 
>  Yeah, something like that, with aspirational targets for improvement
> >> from
>  current releases. Then what to measure, the tests to run, and criteria
> >> for
>  evaluation are clear and organized and we're able to better assess how
> >> the
>  work in progress is meeting its goals (or not)
> 
> 
> 
>  On Jul 18, 2015, at 12:05 PM, Vladimir Rodionov <
> vladrodio...@gmail.com
> >>>
>  wrote:
> 
> >>> Umbrella jira to make sure we can have blocks cached in offheap
> >> backed
> > cache. In the entire read path, we can refer to this offheap buffer
> and
> > avoid onheap copying.
> >
> > I think, on a read path, the most important improvement we could
> >> imagine
>  is
> > elimination or reducing of object creations (KVs, iterators etc).
> > object reuse, byte buffers reuse or offheap buffers reuse, API change
>  etc.
> > If this is a part of this JIRA, then I would easily define a goal:
> > improving 95/99% latency of a read operations. Not performance, but
>  latency
> > matters
> >
> > -Vlad
> >
> >
> >
> > On Sat, Jul 18, 2015 at 11:24 AM, Andrew Purtell <
>  andrew.purt...@gmail.com>
> > wrote:
> >
> >> That's not a realistic or useful test scenario, unless the goal is
> to
> >> accelerate queries where all cells are filtered at the server.
> >>
> >>
> >>
> >>> On Jul 18, 2015, at 11:02 AM, Anoop John 
>  wrote:
> >>>
> >>> No Andy. 11425 having doc attached to it. At the end of it, we have
>  added
> >>> perf numbers in a cluster testing.  This was done using PE get and
> >> scan
> >>> tests with filtering all cells at server (to not consider n/w
> >> bandwidth
> >>> constraints)
> >>>
> >>> -Anoop-
> >>>
> >>> On Sat, Jul 18, 2015 at 9:30 PM, Andrew Purtell <
> >> andrew.purt...@gmail.com>
> >>> wrote:
> >>>
>  We have some microbenchmarks, not evidence of differences seen
> from
> >> a
>  client application. I'm not saying that microbenchmarks are not
>  totally
>  necessary and a great start - they are - but that they don't
> measure
>  an
> >> end
>  goal. Furthermore unless I've missed one somewhere we don't have a
>  JIRA
> >> or
>  design doc that states a clear end goal metric like the strawman I
>  threw
>  together in my previous mail. A measurable system level goal and
> >> some
> >> data
>  from full cluster testing would go a lot further toward letting
> all
> >> of
> >> us
>  evaluate the potential and payoff of the work. In the meantime we
>  should
>  probably be assembling these changes on a branch instead of in
> >> trunk,
> >> for
>  as long as the goal is not clearly defined and the payoff and
>  potential
> >> for
>  perf regressions is untested and u

Re: DISCUSSION: lets do a developer workshop on near-term work

[Stephen Jiang]

[Let us move back to the main topic - a meeting to talk about the next
direction on HBASE development]

Are we firm on the *August 26th* meeting date?

Given the long list of topics from St.Ack, even a one day meeting might not
cover all of them (in depth).  We need to either trim the topic list or
limit the time to discuss a single topic (30 min for one topic enough?).

Thanks
Stephen


On Mon, Jul 20, 2015 at 9:50 AM, Anoop John  wrote:

> We will be doing some more large data tests in coming week Andy..   Will
> report back more.  Also will do a write up , in what all ways the work
> might help us.  As Sean said, we will continue in another thread if any
> thing further..  Will soon write back on the test result.  Thanks.
>
> -Anoop-
>
> On Mon, Jul 20, 2015 at 9:59 PM, Andrew Purtell 
> wrote:
>
> > Cool, thanks.
> >
> > Is a 20% latency reduction the most we can expect or do you think there
> is
> > room for more improvement? Just curious.
> >
> > Is latency reduction the only goal? Anything here about supporting larger
> > heaps? Is there something we can measure in that regard?
> >
> > Hope you see my point and there's enough here to prime a goals and
> metrics
> > discussion at the pow wow or on the relevant JIRAs.
> >
> > > On Jul 20, 2015, at 4:43 AM, ramkrishna vasudevan <
> > ramkrishna.s.vasude...@gmail.com> wrote:
> > >
> > > Hi Andy
> > >
> > > Based on our POCs done, we expect around 20% improvement in latency.
> For
> > > scans it will be little lesser than 20%.
> > >
> > > Regards
> > > Ram
> > >
> > >
> > > On Sun, Jul 19, 2015 at 10:20 AM, Andrew Purtell <
> > andrew.purt...@gmail.com>
> > > wrote:
> > >
> > >> Hi Ram,
> > >>
> > >> Do you have any targets for what you are measuring? What are the goals
> > you
> > >> guys are working toward with the off heaping changes?
> > >>
> > >>
> >  On Jul 18, 2015, at 9:16 PM, ramkrishna vasudevan <
> > >>> ramkrishna.s.vasude...@gmail.com> wrote:
> > >>>
> > >>> Thanks Vladimir.
> > >>> Yeah, the reports that were attached specifically captured the
> 95/99th
> > >>> percentile.
> > >>> The reason for checking the server side perf was to specifically see
> > the
> > >>> improvement in the server side and also the client was sending large
> > >>> results in multiple threads. So wanted to avoid the n/w
> interference. I
> > >>> think it was a general practice that we were following.
> > >>> We Wil do some more tests and get some latest readings with bigger
> data
> > >>> sets.
> > >>> Sent from mobile.
> >  On Jul 19, 2015 1:05 AM, "Andrew Purtell"  >
> > >> wrote:
> > 
> >  +1
> > 
> >  Yeah, something like that, with aspirational targets for improvement
> > >> from
> >  current releases. Then what to measure, the tests to run, and
> criteria
> > >> for
> >  evaluation are clear and organized and we're able to better assess
> how
> > >> the
> >  work in progress is meeting its goals (or not)
> > 
> > 
> > 
> >  On Jul 18, 2015, at 12:05 PM, Vladimir Rodionov <
> > vladrodio...@gmail.com
> > >>>
> >  wrote:
> > 
> > >>> Umbrella jira to make sure we can have blocks cached in offheap
> > >> backed
> > > cache. In the entire read path, we can refer to this offheap buffer
> > and
> > > avoid onheap copying.
> > >
> > > I think, on a read path, the most important improvement we could
> > >> imagine
> >  is
> > > elimination or reducing of object creations (KVs, iterators etc).
> > > object reuse, byte buffers reuse or offheap buffers reuse, API
> change
> >  etc.
> > > If this is a part of this JIRA, then I would easily define a goal:
> > > improving 95/99% latency of a read operations. Not performance, but
> >  latency
> > > matters
> > >
> > > -Vlad
> > >
> > >
> > >
> > > On Sat, Jul 18, 2015 at 11:24 AM, Andrew Purtell <
> >  andrew.purt...@gmail.com>
> > > wrote:
> > >
> > >> That's not a realistic or useful test scenario, unless the goal is
> > to
> > >> accelerate queries where all cells are filtered at the server.
> > >>
> > >>
> > >>
> > >>> On Jul 18, 2015, at 11:02 AM, Anoop John 
> >  wrote:
> > >>>
> > >>> No Andy. 11425 having doc attached to it. At the end of it, we
> have
> >  added
> > >>> perf numbers in a cluster testing.  This was done using PE get
> and
> > >> scan
> > >>> tests with filtering all cells at server (to not consider n/w
> > >> bandwidth
> > >>> constraints)
> > >>>
> > >>> -Anoop-
> > >>>
> > >>> On Sat, Jul 18, 2015 at 9:30 PM, Andrew Purtell <
> > >> andrew.purt...@gmail.com>
> > >>> wrote:
> > >>>
> >  We have some microbenchmarks, not evidence of differences seen
> > from
> > >> a
> >  client application. I'm not saying that microbenchmarks are not
> >  totally
> >  necessary and a great start - they are - but that they don't
> > measure
> >  an
> > >> en

[jira] [Created] (HBASE-14123) HBase Backup/Restore Phase 2

[Vladimir Rodionov (JIRA)]

Vladimir Rodionov created HBASE-14123:
-

 Summary: HBase Backup/Restore Phase 2
 Key: HBASE-14123
 URL: https://issues.apache.org/jira/browse/HBASE-14123
 Project: HBase
  Issue Type: Umbrella
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov


Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14124) Failed backup is not handled properly in incremental mode

[Vladimir Rodionov (JIRA)]

Vladimir Rodionov created HBASE-14124:
-

 Summary: Failed backup is not handled properly in incremental mode
 Key: HBASE-14124
 URL: https://issues.apache.org/jira/browse/HBASE-14124
 Project: HBase
  Issue Type: Bug
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov


BackupHandler failedBackup method does not clean failed incremental backup 
artefacts on HDFS (and in HBase).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Merge branch hbase-11339 HBase MOB to trunk

[Ted Yu]

Tuesday July 21 PDT is coming up soon.

I noticed that Jingcheng's email has Jon, Annop and Ram's names at the
bottom.
Does this mean that Jon, Anoop and Ram already gave (implicit) +1 ?

Cheers

On Fri, Jul 17, 2015 at 3:33 AM, Ted Yu  wrote:

> QA run for HBASE-11339  is
> green.
>
> IntegrationTestIngestWithMOB has been run which passes.
>
> +1 on merging to master branch.
>
> On Fri, Jul 17, 2015 at 2:01 AM, Jingcheng Du 
> wrote:
>
>> Hi all,
>>
>> The Moderate Object Storage (MOB) feature (HBASE-11339[1]) is modified I/O
>> and compaction path that allows individual moderately sized values
>> (100KB-10MB) to be stored so that write amplification is reduced when
>> compared to the normal I/O path. MOB is defined in the column family
>> and it is almost isolated with other components, the features and
>> performance
>> cannot be effected in normal columns. A detailed design doc and user guide
>> can
>> be found on the hbase-11339 jira. The code reside in the feature branch
>> hbase-11339[2], and now the latest mega patch for trunk is
>> available in RB[3].
>>
>> Jon had brought this in another DISCUSSION thread some days ago. We
>> collected
>> all the feedbacks there and had tried to incorporate them.
>>
>> Some improvements, like encryption of mob files, finding
>> corrupt mob files and dangling reference cells, have been implemented.
>>
>> The other point that was discussed is the use of MR for doing the MOB
>> compaction in phase-1
>> of the implementation. Now the dependency is removed, we have an inbuilt
>> MOB
>> compaction that
>> can be run automatically and it does not need MR anymore. Still the MR
>> compaction is provided
>> as a tool for users.
>>
>> Considering the fact that the concerns from the discussion thread have
>> been
>> addressed, we are putting
>> the merge to trunk up for a vote.
>> [] +1 proceed with merge to trunk.
>> [] 0 no opinion
>> [] -1 do not merge to trunk, because ...
>>
>> Our current merge vote policy requires at least 3 +1s from committers to
>> move forward[4]. The vote runs for 72 hours (the weekends are excluded),
>> and
>> it will be closed at the midnight of
>> Tuesday July 21 PDT.
>>
>> Thanks,
>> Jingcheng, Jon, Ram and Anoop.
>>
>> [1]: https://issues.apache.org/jira/browse/HBASE-11339
>> [2]: https://github.com/apache/hbase/tree/hbase-11339
>> [3]: https://reviews.apache.org/r/36391/
>> [4]: https://hbase.apache.org/book.html#_decisions
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-hbase.679495.n3.nabble.com/VOTE-Merge-branch-hbase-11339-HBase-MOB-to-trunk-tp4073308.html
>> Sent from the HBase Developer mailing list archive at Nabble.com.
>>
>
>

[jira] [Created] (HBASE-14125) HBAse Backup/Restore Phase 2: Cancel backup

[Vladimir Rodionov (JIRA)]

Vladimir Rodionov created HBASE-14125:
-

 Summary: HBAse Backup/Restore Phase 2: Cancel backup
 Key: HBASE-14125
 URL: https://issues.apache.org/jira/browse/HBASE-14125
 Project: HBase
  Issue Type: New Feature
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov


Cancel backup operation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Merge branch hbase-11339 HBase MOB to trunk

[Sean Busbey]

+1

-- 
Sean
On Jul 17, 2015 2:01 AM, "Jingcheng Du"  wrote:

> Hi all,
>
> The Moderate Object Storage (MOB) feature (HBASE-11339[1]) is modified I/O
> and compaction path that allows individual moderately sized values
> (100KB-10MB) to be stored so that write amplification is reduced when
> compared to the normal I/O path. MOB is defined in the column family
> and it is almost isolated with other components, the features and
> performance
> cannot be effected in normal columns. A detailed design doc and user guide
> can
> be found on the hbase-11339 jira. The code reside in the feature branch
> hbase-11339[2], and now the latest mega patch for trunk is
> available in RB[3].
>
> Jon had brought this in another DISCUSSION thread some days ago. We
> collected
> all the feedbacks there and had tried to incorporate them.
>
> Some improvements, like encryption of mob files, finding
> corrupt mob files and dangling reference cells, have been implemented.
>
> The other point that was discussed is the use of MR for doing the MOB
> compaction in phase-1
> of the implementation. Now the dependency is removed, we have an inbuilt
> MOB
> compaction that
> can be run automatically and it does not need MR anymore. Still the MR
> compaction is provided
> as a tool for users.
>
> Considering the fact that the concerns from the discussion thread have been
> addressed, we are putting
> the merge to trunk up for a vote.
> [] +1 proceed with merge to trunk.
> [] 0 no opinion
> [] -1 do not merge to trunk, because ...
>
> Our current merge vote policy requires at least 3 +1s from committers to
> move forward[4]. The vote runs for 72 hours (the weekends are excluded),
> and
> it will be closed at the midnight of
> Tuesday July 21 PDT.
>
> Thanks,
> Jingcheng, Jon, Ram and Anoop.
>
> [1]: https://issues.apache.org/jira/browse/HBASE-11339
> [2]: https://github.com/apache/hbase/tree/hbase-11339
> [3]: https://reviews.apache.org/r/36391/
> [4]: https://hbase.apache.org/book.html#_decisions
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/VOTE-Merge-branch-hbase-11339-HBase-MOB-to-trunk-tp4073308.html
> Sent from the HBase Developer mailing list archive at Nabble.com.
>

Re: [VOTE] Merge branch hbase-11339 HBase MOB to trunk

[Andrew Purtell]

We do have three +1s on this thread which is already sufficient for a
branch merge unless there is a veto.


On Mon, Jul 20, 2015 at 1:53 PM, Ted Yu  wrote:

> Tuesday July 21 PDT is coming up soon.
>
> I noticed that Jingcheng's email has Jon, Annop and Ram's names at the
> bottom.
> Does this mean that Jon, Anoop and Ram already gave (implicit) +1 ?
>
> Cheers
>
> On Fri, Jul 17, 2015 at 3:33 AM, Ted Yu  wrote:
>
> > QA run for HBASE-11339 <
> https://issues.apache.org/jira/browse/HBASE-11339> is
> > green.
> >
> > IntegrationTestIngestWithMOB has been run which passes.
> >
> > +1 on merging to master branch.
> >
> > On Fri, Jul 17, 2015 at 2:01 AM, Jingcheng Du 
> > wrote:
> >
> >> Hi all,
> >>
> >> The Moderate Object Storage (MOB) feature (HBASE-11339[1]) is modified
> I/O
> >> and compaction path that allows individual moderately sized values
> >> (100KB-10MB) to be stored so that write amplification is reduced when
> >> compared to the normal I/O path. MOB is defined in the column family
> >> and it is almost isolated with other components, the features and
> >> performance
> >> cannot be effected in normal columns. A detailed design doc and user
> guide
> >> can
> >> be found on the hbase-11339 jira. The code reside in the feature branch
> >> hbase-11339[2], and now the latest mega patch for trunk is
> >> available in RB[3].
> >>
> >> Jon had brought this in another DISCUSSION thread some days ago. We
> >> collected
> >> all the feedbacks there and had tried to incorporate them.
> >>
> >> Some improvements, like encryption of mob files, finding
> >> corrupt mob files and dangling reference cells, have been implemented.
> >>
> >> The other point that was discussed is the use of MR for doing the MOB
> >> compaction in phase-1
> >> of the implementation. Now the dependency is removed, we have an inbuilt
> >> MOB
> >> compaction that
> >> can be run automatically and it does not need MR anymore. Still the MR
> >> compaction is provided
> >> as a tool for users.
> >>
> >> Considering the fact that the concerns from the discussion thread have
> >> been
> >> addressed, we are putting
> >> the merge to trunk up for a vote.
> >> [] +1 proceed with merge to trunk.
> >> [] 0 no opinion
> >> [] -1 do not merge to trunk, because ...
> >>
> >> Our current merge vote policy requires at least 3 +1s from committers to
> >> move forward[4]. The vote runs for 72 hours (the weekends are excluded),
> >> and
> >> it will be closed at the midnight of
> >> Tuesday July 21 PDT.
> >>
> >> Thanks,
> >> Jingcheng, Jon, Ram and Anoop.
> >>
> >> [1]: https://issues.apache.org/jira/browse/HBASE-11339
> >> [2]: https://github.com/apache/hbase/tree/hbase-11339
> >> [3]: https://reviews.apache.org/r/36391/
> >> [4]: https://hbase.apache.org/book.html#_decisions
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://apache-hbase.679495.n3.nabble.com/VOTE-Merge-branch-hbase-11339-HBase-MOB-to-trunk-tp4073308.html
> >> Sent from the HBase Developer mailing list archive at Nabble.com.
> >>
> >
> >
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: [VOTE] Merge branch hbase-11339 HBase MOB to trunk

[Matteo Bertozzi]

+1

Matteo


On Mon, Jul 20, 2015 at 3:26 PM, Andrew Purtell  wrote:

> We do have three +1s on this thread which is already sufficient for a
> branch merge unless there is a veto.
>
>
> On Mon, Jul 20, 2015 at 1:53 PM, Ted Yu  wrote:
>
> > Tuesday July 21 PDT is coming up soon.
> >
> > I noticed that Jingcheng's email has Jon, Annop and Ram's names at the
> > bottom.
> > Does this mean that Jon, Anoop and Ram already gave (implicit) +1 ?
> >
> > Cheers
> >
> > On Fri, Jul 17, 2015 at 3:33 AM, Ted Yu  wrote:
> >
> > > QA run for HBASE-11339 <
> > https://issues.apache.org/jira/browse/HBASE-11339> is
> > > green.
> > >
> > > IntegrationTestIngestWithMOB has been run which passes.
> > >
> > > +1 on merging to master branch.
> > >
> > > On Fri, Jul 17, 2015 at 2:01 AM, Jingcheng Du 
> > > wrote:
> > >
> > >> Hi all,
> > >>
> > >> The Moderate Object Storage (MOB) feature (HBASE-11339[1]) is modified
> > I/O
> > >> and compaction path that allows individual moderately sized values
> > >> (100KB-10MB) to be stored so that write amplification is reduced when
> > >> compared to the normal I/O path. MOB is defined in the column family
> > >> and it is almost isolated with other components, the features and
> > >> performance
> > >> cannot be effected in normal columns. A detailed design doc and user
> > guide
> > >> can
> > >> be found on the hbase-11339 jira. The code reside in the feature
> branch
> > >> hbase-11339[2], and now the latest mega patch for trunk is
> > >> available in RB[3].
> > >>
> > >> Jon had brought this in another DISCUSSION thread some days ago. We
> > >> collected
> > >> all the feedbacks there and had tried to incorporate them.
> > >>
> > >> Some improvements, like encryption of mob files, finding
> > >> corrupt mob files and dangling reference cells, have been implemented.
> > >>
> > >> The other point that was discussed is the use of MR for doing the MOB
> > >> compaction in phase-1
> > >> of the implementation. Now the dependency is removed, we have an
> inbuilt
> > >> MOB
> > >> compaction that
> > >> can be run automatically and it does not need MR anymore. Still the MR
> > >> compaction is provided
> > >> as a tool for users.
> > >>
> > >> Considering the fact that the concerns from the discussion thread have
> > >> been
> > >> addressed, we are putting
> > >> the merge to trunk up for a vote.
> > >> [] +1 proceed with merge to trunk.
> > >> [] 0 no opinion
> > >> [] -1 do not merge to trunk, because ...
> > >>
> > >> Our current merge vote policy requires at least 3 +1s from committers
> to
> > >> move forward[4]. The vote runs for 72 hours (the weekends are
> excluded),
> > >> and
> > >> it will be closed at the midnight of
> > >> Tuesday July 21 PDT.
> > >>
> > >> Thanks,
> > >> Jingcheng, Jon, Ram and Anoop.
> > >>
> > >> [1]: https://issues.apache.org/jira/browse/HBASE-11339
> > >> [2]: https://github.com/apache/hbase/tree/hbase-11339
> > >> [3]: https://reviews.apache.org/r/36391/
> > >> [4]: https://hbase.apache.org/book.html#_decisions
> > >>
> > >>
> > >>
> > >> --
> > >> View this message in context:
> > >>
> >
> http://apache-hbase.679495.n3.nabble.com/VOTE-Merge-branch-hbase-11339-HBase-MOB-to-trunk-tp4073308.html
> > >> Sent from the HBase Developer mailing list archive at Nabble.com.
> > >>
> > >
> > >
> >
>
>
>
> --
> Best regards,
>
>- Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

[jira] [Created] (HBASE-14126) I'm using play framework, creating a sample Hbase project. And I keep getting this error: tried to access method com.google.common.base.Stopwatch.()V from class or

[Lesley Cheung (JIRA)]

Lesley Cheung created HBASE-14126:
-

 Summary: I'm using play framework, creating a sample Hbase 
project. And I keep getting this error: tried to access method 
com.google.common.base.Stopwatch.()V from class 
org.apache.hadoop.hbase.zookeeper.MetaTableLocator
 Key: HBASE-14126
 URL: https://issues.apache.org/jira/browse/HBASE-14126
 Project: HBase
  Issue Type: Task
 Environment: Ubuntu; Play Framework 2.4.2; Hbase 1.0
Reporter: Lesley Cheung
Assignee: Lesley Cheung


The simple Hbase project works in Maven. But when I use play framework , it 
keeps showing this error. I modified the lib dependency many times, but I just 
can't eliminate this error. Some one please help me! 

 
java.lang.IllegalAccessError: tried to access method 
com.google.common.base.Stopwatch.()V from class 
org.apache.hadoop.hbase.zookeeper.MetaTableLocator
at 
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:434)
at 
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:60)
at 
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1123)
at 
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1110)
at 
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1262)
at 
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1126)
at 
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:369)
at 
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:320)
at 
org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:206)
at 
org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1496)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1119)
at utils.BigQueue.run(BigQueue.java:68)
at 
akka.actor.LightArrayRevolverScheduler$$anon$2$$anon$1.run(Scheduler.scala:242)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
at 
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Merge branch hbase-11339 HBase MOB to trunk

[Elliott Clark]

+1

On Mon, Jul 20, 2015 at 3:30 PM, Matteo Bertozzi 
wrote:

> +1
>
> Matteo
>
>
> On Mon, Jul 20, 2015 at 3:26 PM, Andrew Purtell 
> wrote:
>
> > We do have three +1s on this thread which is already sufficient for a
> > branch merge unless there is a veto.
> >
> >
> > On Mon, Jul 20, 2015 at 1:53 PM, Ted Yu  wrote:
> >
> > > Tuesday July 21 PDT is coming up soon.
> > >
> > > I noticed that Jingcheng's email has Jon, Annop and Ram's names at the
> > > bottom.
> > > Does this mean that Jon, Anoop and Ram already gave (implicit) +1 ?
> > >
> > > Cheers
> > >
> > > On Fri, Jul 17, 2015 at 3:33 AM, Ted Yu  wrote:
> > >
> > > > QA run for HBASE-11339 <
> > > https://issues.apache.org/jira/browse/HBASE-11339> is
> > > > green.
> > > >
> > > > IntegrationTestIngestWithMOB has been run which passes.
> > > >
> > > > +1 on merging to master branch.
> > > >
> > > > On Fri, Jul 17, 2015 at 2:01 AM, Jingcheng Du <
> jingcheng...@intel.com>
> > > > wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> The Moderate Object Storage (MOB) feature (HBASE-11339[1]) is
> modified
> > > I/O
> > > >> and compaction path that allows individual moderately sized values
> > > >> (100KB-10MB) to be stored so that write amplification is reduced
> when
> > > >> compared to the normal I/O path. MOB is defined in the column family
> > > >> and it is almost isolated with other components, the features and
> > > >> performance
> > > >> cannot be effected in normal columns. A detailed design doc and user
> > > guide
> > > >> can
> > > >> be found on the hbase-11339 jira. The code reside in the feature
> > branch
> > > >> hbase-11339[2], and now the latest mega patch for trunk is
> > > >> available in RB[3].
> > > >>
> > > >> Jon had brought this in another DISCUSSION thread some days ago. We
> > > >> collected
> > > >> all the feedbacks there and had tried to incorporate them.
> > > >>
> > > >> Some improvements, like encryption of mob files, finding
> > > >> corrupt mob files and dangling reference cells, have been
> implemented.
> > > >>
> > > >> The other point that was discussed is the use of MR for doing the
> MOB
> > > >> compaction in phase-1
> > > >> of the implementation. Now the dependency is removed, we have an
> > inbuilt
> > > >> MOB
> > > >> compaction that
> > > >> can be run automatically and it does not need MR anymore. Still the
> MR
> > > >> compaction is provided
> > > >> as a tool for users.
> > > >>
> > > >> Considering the fact that the concerns from the discussion thread
> have
> > > >> been
> > > >> addressed, we are putting
> > > >> the merge to trunk up for a vote.
> > > >> [] +1 proceed with merge to trunk.
> > > >> [] 0 no opinion
> > > >> [] -1 do not merge to trunk, because ...
> > > >>
> > > >> Our current merge vote policy requires at least 3 +1s from
> committers
> > to
> > > >> move forward[4]. The vote runs for 72 hours (the weekends are
> > excluded),
> > > >> and
> > > >> it will be closed at the midnight of
> > > >> Tuesday July 21 PDT.
> > > >>
> > > >> Thanks,
> > > >> Jingcheng, Jon, Ram and Anoop.
> > > >>
> > > >> [1]: https://issues.apache.org/jira/browse/HBASE-11339
> > > >> [2]: https://github.com/apache/hbase/tree/hbase-11339
> > > >> [3]: https://reviews.apache.org/r/36391/
> > > >> [4]: https://hbase.apache.org/book.html#_decisions
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> View this message in context:
> > > >>
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/VOTE-Merge-branch-hbase-11339-HBase-MOB-to-trunk-tp4073308.html
> > > >> Sent from the HBase Developer mailing list archive at Nabble.com.
> > > >>
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> >- Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>

Re: [VOTE] Merge branch hbase-11339 HBase MOB to trunk

[ramkrishna vasudevan]

I can cast my explicit +1 too.
Any way we now have enough +1s for the vote to pass.  Thanks to all those
who voted.

Regards
Ram

On Tue, Jul 21, 2015 at 6:23 AM, Elliott Clark  wrote:

> +1
>
> On Mon, Jul 20, 2015 at 3:30 PM, Matteo Bertozzi 
> wrote:
>
> > +1
> >
> > Matteo
> >
> >
> > On Mon, Jul 20, 2015 at 3:26 PM, Andrew Purtell 
> > wrote:
> >
> > > We do have three +1s on this thread which is already sufficient for a
> > > branch merge unless there is a veto.
> > >
> > >
> > > On Mon, Jul 20, 2015 at 1:53 PM, Ted Yu  wrote:
> > >
> > > > Tuesday July 21 PDT is coming up soon.
> > > >
> > > > I noticed that Jingcheng's email has Jon, Annop and Ram's names at
> the
> > > > bottom.
> > > > Does this mean that Jon, Anoop and Ram already gave (implicit) +1 ?
> > > >
> > > > Cheers
> > > >
> > > > On Fri, Jul 17, 2015 at 3:33 AM, Ted Yu  wrote:
> > > >
> > > > > QA run for HBASE-11339 <
> > > > https://issues.apache.org/jira/browse/HBASE-11339> is
> > > > > green.
> > > > >
> > > > > IntegrationTestIngestWithMOB has been run which passes.
> > > > >
> > > > > +1 on merging to master branch.
> > > > >
> > > > > On Fri, Jul 17, 2015 at 2:01 AM, Jingcheng Du <
> > jingcheng...@intel.com>
> > > > > wrote:
> > > > >
> > > > >> Hi all,
> > > > >>
> > > > >> The Moderate Object Storage (MOB) feature (HBASE-11339[1]) is
> > modified
> > > > I/O
> > > > >> and compaction path that allows individual moderately sized values
> > > > >> (100KB-10MB) to be stored so that write amplification is reduced
> > when
> > > > >> compared to the normal I/O path. MOB is defined in the column
> family
> > > > >> and it is almost isolated with other components, the features and
> > > > >> performance
> > > > >> cannot be effected in normal columns. A detailed design doc and
> user
> > > > guide
> > > > >> can
> > > > >> be found on the hbase-11339 jira. The code reside in the feature
> > > branch
> > > > >> hbase-11339[2], and now the latest mega patch for trunk is
> > > > >> available in RB[3].
> > > > >>
> > > > >> Jon had brought this in another DISCUSSION thread some days ago.
> We
> > > > >> collected
> > > > >> all the feedbacks there and had tried to incorporate them.
> > > > >>
> > > > >> Some improvements, like encryption of mob files, finding
> > > > >> corrupt mob files and dangling reference cells, have been
> > implemented.
> > > > >>
> > > > >> The other point that was discussed is the use of MR for doing the
> > MOB
> > > > >> compaction in phase-1
> > > > >> of the implementation. Now the dependency is removed, we have an
> > > inbuilt
> > > > >> MOB
> > > > >> compaction that
> > > > >> can be run automatically and it does not need MR anymore. Still
> the
> > MR
> > > > >> compaction is provided
> > > > >> as a tool for users.
> > > > >>
> > > > >> Considering the fact that the concerns from the discussion thread
> > have
> > > > >> been
> > > > >> addressed, we are putting
> > > > >> the merge to trunk up for a vote.
> > > > >> [] +1 proceed with merge to trunk.
> > > > >> [] 0 no opinion
> > > > >> [] -1 do not merge to trunk, because ...
> > > > >>
> > > > >> Our current merge vote policy requires at least 3 +1s from
> > committers
> > > to
> > > > >> move forward[4]. The vote runs for 72 hours (the weekends are
> > > excluded),
> > > > >> and
> > > > >> it will be closed at the midnight of
> > > > >> Tuesday July 21 PDT.
> > > > >>
> > > > >> Thanks,
> > > > >> Jingcheng, Jon, Ram and Anoop.
> > > > >>
> > > > >> [1]: https://issues.apache.org/jira/browse/HBASE-11339
> > > > >> [2]: https://github.com/apache/hbase/tree/hbase-11339
> > > > >> [3]: https://reviews.apache.org/r/36391/
> > > > >> [4]: https://hbase.apache.org/book.html#_decisions
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> View this message in context:
> > > > >>
> > > >
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/VOTE-Merge-branch-hbase-11339-HBase-MOB-to-trunk-tp4073308.html
> > > > >> Sent from the HBase Developer mailing list archive at Nabble.com.
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > >- Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
>

Re: [VOTE] Merge branch hbase-11339 HBase MOB to trunk

[Anoop John]

As it was implicit  (because of the name in the mail) I have not voted.
Here is my explicit +1

-Anoop-

On Tue, Jul 21, 2015 at 11:17 AM, ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com> wrote:

> I can cast my explicit +1 too.
> Any way we now have enough +1s for the vote to pass.  Thanks to all those
> who voted.
>
> Regards
> Ram
>
> On Tue, Jul 21, 2015 at 6:23 AM, Elliott Clark  wrote:
>
> > +1
> >
> > On Mon, Jul 20, 2015 at 3:30 PM, Matteo Bertozzi <
> theo.berto...@gmail.com>
> > wrote:
> >
> > > +1
> > >
> > > Matteo
> > >
> > >
> > > On Mon, Jul 20, 2015 at 3:26 PM, Andrew Purtell 
> > > wrote:
> > >
> > > > We do have three +1s on this thread which is already sufficient for a
> > > > branch merge unless there is a veto.
> > > >
> > > >
> > > > On Mon, Jul 20, 2015 at 1:53 PM, Ted Yu  wrote:
> > > >
> > > > > Tuesday July 21 PDT is coming up soon.
> > > > >
> > > > > I noticed that Jingcheng's email has Jon, Annop and Ram's names at
> > the
> > > > > bottom.
> > > > > Does this mean that Jon, Anoop and Ram already gave (implicit) +1 ?
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Fri, Jul 17, 2015 at 3:33 AM, Ted Yu 
> wrote:
> > > > >
> > > > > > QA run for HBASE-11339 <
> > > > > https://issues.apache.org/jira/browse/HBASE-11339> is
> > > > > > green.
> > > > > >
> > > > > > IntegrationTestIngestWithMOB has been run which passes.
> > > > > >
> > > > > > +1 on merging to master branch.
> > > > > >
> > > > > > On Fri, Jul 17, 2015 at 2:01 AM, Jingcheng Du <
> > > jingcheng...@intel.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Hi all,
> > > > > >>
> > > > > >> The Moderate Object Storage (MOB) feature (HBASE-11339[1]) is
> > > modified
> > > > > I/O
> > > > > >> and compaction path that allows individual moderately sized
> values
> > > > > >> (100KB-10MB) to be stored so that write amplification is reduced
> > > when
> > > > > >> compared to the normal I/O path. MOB is defined in the column
> > family
> > > > > >> and it is almost isolated with other components, the features
> and
> > > > > >> performance
> > > > > >> cannot be effected in normal columns. A detailed design doc and
> > user
> > > > > guide
> > > > > >> can
> > > > > >> be found on the hbase-11339 jira. The code reside in the feature
> > > > branch
> > > > > >> hbase-11339[2], and now the latest mega patch for trunk is
> > > > > >> available in RB[3].
> > > > > >>
> > > > > >> Jon had brought this in another DISCUSSION thread some days ago.
> > We
> > > > > >> collected
> > > > > >> all the feedbacks there and had tried to incorporate them.
> > > > > >>
> > > > > >> Some improvements, like encryption of mob files, finding
> > > > > >> corrupt mob files and dangling reference cells, have been
> > > implemented.
> > > > > >>
> > > > > >> The other point that was discussed is the use of MR for doing
> the
> > > MOB
> > > > > >> compaction in phase-1
> > > > > >> of the implementation. Now the dependency is removed, we have an
> > > > inbuilt
> > > > > >> MOB
> > > > > >> compaction that
> > > > > >> can be run automatically and it does not need MR anymore. Still
> > the
> > > MR
> > > > > >> compaction is provided
> > > > > >> as a tool for users.
> > > > > >>
> > > > > >> Considering the fact that the concerns from the discussion
> thread
> > > have
> > > > > >> been
> > > > > >> addressed, we are putting
> > > > > >> the merge to trunk up for a vote.
> > > > > >> [] +1 proceed with merge to trunk.
> > > > > >> [] 0 no opinion
> > > > > >> [] -1 do not merge to trunk, because ...
> > > > > >>
> > > > > >> Our current merge vote policy requires at least 3 +1s from
> > > committers
> > > > to
> > > > > >> move forward[4]. The vote runs for 72 hours (the weekends are
> > > > excluded),
> > > > > >> and
> > > > > >> it will be closed at the midnight of
> > > > > >> Tuesday July 21 PDT.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Jingcheng, Jon, Ram and Anoop.
> > > > > >>
> > > > > >> [1]: https://issues.apache.org/jira/browse/HBASE-11339
> > > > > >> [2]: https://github.com/apache/hbase/tree/hbase-11339
> > > > > >> [3]: https://reviews.apache.org/r/36391/
> > > > > >> [4]: https://hbase.apache.org/book.html#_decisions
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> View this message in context:
> > > > > >>
> > > > >
> > > >
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/VOTE-Merge-branch-hbase-11339-HBase-MOB-to-trunk-tp4073308.html
> > > > > >> Sent from the HBase Developer mailing list archive at
> Nabble.com.
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > >
> > > >- Andy
> > > >
> > > > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein
> > > > (via Tom White)
> > > >
> > >
> >
>

[jira] [Resolved] (HBASE-14126) I'm using play framework, creating a sample Hbase project. And I keep getting this error: tried to access method com.google.common.base.Stopwatch.()V from class o

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John resolved HBASE-14126.

Resolution: Invalid
  Assignee: (was: Lesley Cheung)

Pls ask these kind of questions in dev@ or user@

> I'm using play framework, creating a sample Hbase project. And I keep getting 
> this error: tried to access method com.google.common.base.Stopwatch.()V 
> from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
> -
>
> Key: HBASE-14126
> URL: https://issues.apache.org/jira/browse/HBASE-14126
> Project: HBase
>  Issue Type: Task
> Environment: Ubuntu; Play Framework 2.4.2; Hbase 1.0
>Reporter: Lesley Cheung
>
> The simple Hbase project works in Maven. But when I use play framework , it 
> keeps showing this error. I modified the lib dependency many times, but I 
> just can't eliminate this error. Some one please help me! 
>  
> java.lang.IllegalAccessError: tried to access method 
> com.google.common.base.Stopwatch.()V from class 
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator
>   at 
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:434)
>   at 
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:60)
>   at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1123)
>   at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1110)
>   at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1262)
>   at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1126)
>   at 
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:369)
>   at 
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:320)
>   at 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:206)
>   at 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
>   at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1496)
>   at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1119)
>   at utils.BigQueue.run(BigQueue.java:68)
>   at 
> akka.actor.LightArrayRevolverScheduler$$anon$2$$anon$1.run(Scheduler.scala:242)
>   at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
>   at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)