Re: [VOTE] Which Orca representation?

[Nicolas Liochon]

Let me start :-)


[+1] nkeywal's favorite
[ +0.9] Black version of nkeywal's favorite
[+0] Page 1 from Apache_HBase_Orca_Logo_round5.pdf
<
https://issues.apache.org/jira/secure/attachment/12648671/Apache_HBase_Orca_Logo_round5.pdf
>
[-0] Page 2 from Apache_HBase_Orca_Logo_round5.pdf
<
https://issues.apache.org/jira/secure/attachment/12648671/Apache_HBase_Orca_Logo_round5.pdf
>
[-0] Page 3 from Apache_HBase_Orca_Logo_round5.pdf
<
https://issues.apache.org/jira/secure/attachment/12648671/Apache_HBase_Orca_Logo_round5.pdf
>
[ -0] Page 4 from Apache_HBase_Orca_Logo_round5.pdf
<
https://issues.apache.org/jira/secure/attachment/12648671/Apache_HBase_Orca_Logo_round5.pdf
>
[-0] Enis hbasecon talk
[-0 ] JMS orca
[ -0] Plus orca
[ -0] Miscellaneous #1
[ -0] Miscellaneous #2
[ -1 :-)] None of the above!



On Tue, Aug 19, 2014 at 11:38 PM, Stack  wrote:

> On the thread http://search-hadoop.com/m/DHED4uBtYz1, we voted an Orca as
> our mascot.
>
> Below let us pick which Orca representation to use going forward.
>
> Intentionally, we are not voting on an 'hbase logo' + Orca combination.  We
> are justing voting on an Orca representation. Let putting together our
> current logo and the chosen Orca image be a separate project (done by a
> professional!). I think it ok voting on the image only because how they are
> combined will vary with deployment context. We are voting on the image only
> also because getting a set of 'hbase logo' and Orca combinations that are
> of equal treatment to put under your noses is too much work.
>
> So, please vote on which Orca representation we should use going forward as
> the Apache HBase mascot.  First review the options and then add your vote
> in the section on the end.
>
> First up are the classics:
>
> https://issues.apache.org/jira/secure/attachment/12511525/apache%20hbase%20orca%20logo_Proof%203.pdf
>  There are four on this page.  Number #1 is nkeywals's favorite.  Lets vote
> on #1 and #2 only from this page (Where #2 is the black version of
> nkeywal's favorite).
>
> This page has four from which to choose:
> https://issues.apache.org/jira/secure/attachment/12648671/Apache_HBase
> _Orca_Logo_round5.pdf The options are called out below as "Page N from
> Apache_HBase_Orca_Logo_round5.pdf
> <
> https://issues.apache.org/jira/secure/attachment/12648671/Apache_HBase_Orca_Logo_round5.pdf
> >",
> etc.
>
> Enis used this one in his hbase 1.0 hbasecon talk:
>
> http://depositphotos.com/2900573/stock-illustration-killer-whale-tattoo.html
>
> Here is an image that just needs attribution arranged in a few ways by JMS:
>
> https://issues.apache.org/jira/secure/attachment/12662112/proposal_3_logo.png
>
> https://issues.apache.org/jira/secure/attachment/12662110/proposal_2_logo.png
>
> https://issues.apache.org/jira/secure/attachment/12661953/orca_free_vector_on_top_66percent_levelled.png
>
> https://issues.apache.org/jira/secure/attachment/12661954/orca_free_vector_some_selections.png
>
> Lets call this option the 'plus_orca'
> https://issues.apache.org/jira/secure/attachment/12514291/plus_orca.png
>
> And finally, these two the miscellaneous orcas #1 and #2:
> https://issues.apache.org/jira/secure/attachment/12514301/more_orcas2.png
>
> Fill in an 'x' into the box below for the one you want.
>
> [ ] nkeywal's favorite
> [ ] Black version of nkeywal's favorite
> [ ] Page 1 from Apache_HBase_Orca_Logo_round5.pdf
> <
> https://issues.apache.org/jira/secure/attachment/12648671/Apache_HBase_Orca_Logo_round5.pdf
> >
> [ ] Page 2 from Apache_HBase_Orca_Logo_round5.pdf
> <
> https://issues.apache.org/jira/secure/attachment/12648671/Apache_HBase_Orca_Logo_round5.pdf
> >
> [ ] Page 3 from Apache_HBase_Orca_Logo_round5.pdf
> <
> https://issues.apache.org/jira/secure/attachment/12648671/Apache_HBase_Orca_Logo_round5.pdf
> >
> [ ] Page 4 from Apache_HBase_Orca_Logo_round5.pdf
> <
> https://issues.apache.org/jira/secure/attachment/12648671/Apache_HBase_Orca_Logo_round5.pdf
> >
> [ ] Enis hbasecon talk
> [ ] JMS orca
> [ ] Plus orca
> [ ] Miscellaneous #1
> [ ] Miscellaneous #2
> [ ] None of the above!
>

Re: Shout-out for Misty

[Nicolas Liochon]

+1, it's a great to have you...

Nicolas


On Wed, Aug 20, 2014 at 12:17 PM, Anoop John  wrote:

> Great work!  Thanks a lot Misty...
>
>
> -Anoop-
>
> On Wed, Aug 20, 2014 at 11:56 AM, ramkrishna vasudevan <
> ramkrishna.s.vasude...@gmail.com> wrote:
>
> > Great job !! Keep it up.!!!
> >
> > Regards
> > Ram
> >
> >
> > On Wed, Aug 20, 2014 at 11:49 AM, rajeshbabu chintaguntla <
> > rajeshbabu.chintagun...@huawei.com> wrote:
> >
> > > Good work. Thanks Misty!
> > >
> > > Thanks,
> > > Rajeshbabu.
> > > 
> > > This e-mail and its attachments contain confidential information from
> > > HUAWEI, which
> > > is intended only for the person or entity whose address is listed
> above.
> > > Any use of the
> > > information contained herein in any way (including, but not limited to,
> > > total or partial
> > > disclosure, reproduction, or dissemination) by persons other than the
> > > intended
> > > recipient(s) is prohibited. If you receive this e-mail in error, please
> > > notify the sender by
> > > phone or email immediately and delete it!
> > >
> > > 
> > > From: Srikanth Srungarapu [srikanth...@gmail.com]
> > > Sent: Wednesday, August 20, 2014 11:38 AM
> > > To: dev@hbase.apache.org
> > > Cc: hbase-user
> > > Subject: Re: Shout-out for Misty
> > >
> > > Keep rocking, Misty :)
> > >
> > >
> > > On Tue, Aug 19, 2014 at 10:56 PM, Bharath Vissapragada <
> > > bhara...@cloudera.com> wrote:
> > >
> > > > +1. Thanks Misty.
> > > >
> > > >
> > > > On Wed, Aug 20, 2014 at 11:23 AM, Nick Dimiduk 
> > > wrote:
> > > >
> > > > > Our docs are getting a lot of love lately, courtesy of one Misty
> > > > > Stanley-Jones. As someone who joined this community by way of
> > > > > documentation, I'd like to say: Thank you, Misty!
> > > > >
> > > > > -n
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Bharath Vissapragada
> > > > 
> > > >
> > >
> >
>

Re: [ANNOUNCE] New HBase PMC members: Matteo Bertozzi, Nick Dimiduk, and Jeffrey Zhong

[Nicolas Liochon]

Congrats, guys!


On Tue, Aug 26, 2014 at 7:26 AM, Jeremy Carroll  wrote:

> Congratulations. I have seen so many of these names in JIRA. ;)
>
>
> On Mon, Aug 25, 2014 at 5:24 PM, Jonathan Hsieh  wrote:
>
> > On behalf of the Apache HBase PMC, I am happy to belatedly announce and
> > welcome Matteo Bertozzi, Nick Dimiduk, Jeffrey Zhong to the Apache HBase
> > PMC.
> >
> > * Matteo (mbertozzi) is responsible for HBase snapshots, has worked on
> > smoothing security in areas, and driving some new scheduling and
> throttling
> > related features into HBase.
> >
> > * Nick (ndimiduk) is an author of "HBase In Action", the driver on the
> > HBase typing efforts, and recently worked on bucket cache performance and
> > hardening.
> >
> > * Jeffrey (jefferyz) introduced the MTTR-improving distributed log replay
> > feature to HBase and has been an active member of the HBase-related
> Apache
> > Phoenix project.
> >
> > Please join me in congratulating Matteo, Nick and Jeffery on their new
> > roles!
> >
> > Jon
> >
> > --
> > // Jonathan Hsieh (shay)
> > // HBase Tech Lead, Software Engineer, Cloudera
> > // j...@cloudera.com // @jmhsieh
> >
>

Re: Should scan check the limitation of the number of versions?

[Nicolas Liochon]

(moving to user)

In your first scenario (put "table", "row1", "cf:a", "value1", 100 then put
"table", "row1", "cf:a", "value1", 200), there is no deletion, so the
setting KEEP_DELETED_CELLS is not used at all
The behavior you describe is "as expected": there are two versions until
the compaction occurs and removes the version not needed, depending on the
configuration.
There are some optimizations around this: we skip reading early if the
timestamps of what we're reading is not in the scan range. So we don't know
if there is a newer value.

What's the use case you're looking at?

Nicolas












On Tue, Aug 26, 2014 at 3:36 AM, tobe  wrote:

> @andrew Actually I don't want to see row in TIMERANGE => [0, 150] because
> it's the overdue version. Should I set {KEEP_DELETED_CELLS => 'true'}? My
> problem is that even though I don't keep deleted cells, I will get the
> result which is not what I expect.
>
>
> On Tue, Aug 26, 2014 at 9:24 AM, Andrew Purtell 
> wrote:
>
> > On Mon, Aug 25, 2014 at 6:13 PM, tobe  wrote:
> >
> > > @lars I have set {KEEP_DELETED_CELLS => 'false'} in that table. I will
> > get
> > > the same result before manually running `flush`. You can try the
> > commands I
> > > gave and it's 100% repro.
> > >
> >
> > ​You need KEEP_DELETED_CELLS => 'true'. ​
> >
> >
> >
> > On Mon, Aug 25, 2014 at 6:13 PM, tobe  wrote:
> >
> > > @lars I have set {KEEP_DELETED_CELLS => 'false'} in that table. I will
> > get
> > > the same result before manually running `flush`. You can try the
> > commands I
> > > gave and it's 100% repro.
> > >
> > >
> > > On Tue, Aug 26, 2014 at 2:20 AM, lars hofhansl 
> wrote:
> > >
> > > > Queries of past time ranges only work correctly when
> KEEP_DELETED_CELLS
> > > is
> > > > enabled for the column families.
> > > >
> > > >
> > > > 
> > > >  From: tobe 
> > > > To: hbase-dev 
> > > > Cc: "u...@hbase.apache.org" 
> > > > Sent: Monday, August 25, 2014 4:32 AM
> > > > Subject: Re: Should scan check the limitation of the number of
> > versions?
> > > >
> > > >
> > > > I haven't read the code deeply but I have an idea(not sure whether
> it's
> > > > right or not). When we scan the the columns, we will skip the one
> which
> > > > doesn't match(deleted). Can we use a counter to record this? For each
> > > skip,
> > > > we add one until it reaches the restrictive number of versions. But
> we
> > > have
> > > > to consider mvcc and others, which seems more complex.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Aug 25, 2014 at 5:54 PM, tobe  wrote:
> > > >
> > > > > So far, I have found two problems about this.
> > > > >
> > > > > Firstly, HBase-11675 <
> > > https://issues.apache.org/jira/browse/HBASE-11675
> > > > >.
> > > > > It's a little tricky and rarely happens. But it asks users to be
> > > careful
> > > > of
> > > > > compaction which occurs on server side. They may get different
> > results
> > > > > before and after the major compaction.
> > > > >
> > > > > Secondly, if you put a value with timestamp 100, then put another
> > value
> > > > on
> > > > > the same column with timestamp 200. Here we set the number of
> version
> > > as
> > > > 1.
> > > > > So when we get the value of this column, we will get the latest one
> > > with
> > > > > timestamp 200 and that's right. But if I get with a timerange form
> 0
> > to
> > > > > 150, I may get the first value with timestamp 100 before compaction
> > > > > happens. And after compaction happens, you will never get this
> value
> > > even
> > > > > you run the same command.
> > > > >
> > > > > It's easy to repro, follow this steps:
> > > > > hbase(main):001:0> create "table", "cf"
> > > > > hbase(main):003:0> put "table", "row1", "cf:a", "value1", 100
> > > > > hbase(main):003:0> put "table", "row1", "cf:a", "value1", 200
> > > > > hbase(main):026:0> get "table", "row1", {TIMERANGE => [0, 150]}  //
> > > > before
> > > > > flush
> > > > >row1  column=cf:a, timestamp=100, value=value1
> > > > > hbase(main):060:0> flush "table"
> > > > > hbase(main):082:0> get "table", "row1", {TIMERANGE => [0, 150]}  //
> > > after
> > > > > flush
> > > > >0 row(s) in 0.0050 seconds
> > > > >
> > > > > I think the reason of that is we have three restriction to remove
> > data:
> > > > > delete, ttl and versions. Any time we get or scan the data, we will
> > > check
> > > > > the delete mark and ttl to make sure it will not return to users.
> But
> > > for
> > > > > versions, we don't check this limitation. Our output relies on the
> > > > > compaction to cleanup the overdue data. Is it possible to add this
> > > > > condition within scan(get is implemented as scan)?
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> >- Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>

Re: I wonder if it makes sense to restrict the HBase build to one build per jenkins host

[Nicolas Liochon]

It used to be the case for precommit builds: there was a single worker.
It's much simpler this way.
It seems it has changed, likely less than a year ago. I see on
https://builds.apache.org/computer/H0/load-statistics that there are 2
workers.
We're can't do what we want on these machines, as they are used by other
projects however. In the past, Giri was administrating these machines, I
don't know if it's still the case (Enis? Devaraj? Stack? Do you guys know?)

I guess that the root issue is that hdfs & other builds are not
parallelized as HBase are, so they need multiple workers to really use the
multiple cores of the build env...

Nicolas


On Wed, Aug 27, 2014 at 10:45 PM, Andrew Purtell 
wrote:

> +1, if it's possible on ASF Jenkins
>
> We also share with other projects. I've seen build reports implicating
> Oozie tests for supposed HBase test zombies.
>
> When we had cloud based Jenkins running at Trend Micro and Intel we
> launched instances on demand as we needed workers, but never more than one
> worker per instance concurrently. Definitely helps to reduce failures due
> to unexpected state from interactions between tests.
>
>
> On Wed, Aug 27, 2014 at 1:28 PM, Alex Newman  wrote:
>
> > I have noticed that HBase tests can cause a good deal of inherent test
> > interference. We have had much better luck running one and only one
> > set of the tests at a time. I notice that this currently is not
> > happening. I don't have any rights on jenkins, so perhaps someone else
> > could give this a shot. I'd be glad to describe how do it. Another
> > option would be to run the build from within a container (We do this
> > at WanDISCO). Finally I got the build running on travis ci.  Checkout
> > https://github.com/OhmData/hbase-public/tree/travis for an example of
> > how to get this done.
> >
>
>
>
> --
> Best regards,
>
>- Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Re: [DISCUSSION] Dropping support for Hadoop 1.0 in 0.98

[Nicolas Liochon]

+1
Le 31 oct. 2014 23:49, "Andrew Purtell"  a écrit :

> Based on the positive responses thus far, and unless we see an objection
> between now and then, I plan to resolve HBASE-12397 next week by removing
> support in 0.98 branch for Hadoop 1.0 (but not Hadoop 1.1) in time for
> release 0.98.8.
>
> On Fri, Oct 31, 2014 at 3:34 PM, Ted Yu  wrote:
>
> > Adding user@.
> >
> > I would +1 this motion.
> >
> > On Fri, Oct 31, 2014 at 12:18 PM, Sean Busbey 
> wrote:
> >
> > > +1, presuming we wouldn't change our position on hadoop 1.0 for 0.94.
> > >
> > > For the curious, here is the full support matrix Andrew is referencing:
> > > http://hbase.apache.org/book.html#d0e1440
> > >
> > > On Fri, Oct 31, 2014 at 1:13 PM, Andrew Purtell 
> > > wrote:
> > >
> > > > No, we'll still need a -hadoop1 and -hadoop2 munged build of 0.98.
> I'm
> > > only
> > > > suggesting removing support for version 1.0. Other version 1.x would
> > > remain
> > > > active in the compatibility list.
> > > >
> > > > On Fri, Oct 31, 2014 at 11:01 AM, Konstantin Boudnik  >
> > > > wrote:
> > > >
> > > > > Absolutely makes sense! it will make a lot of things easier,
> really.
> > > The
> > > > > infamous need for classifiers will finally go away!
> > > > >
> > > > > On Fri, Oct 31, 2014 at 10:56AM, Andrew Purtell wrote:
> > > > > > Hadoop 1.0 is an ancient, and I believe dead, version that
> > certainly
> > > > > nobody
> > > > > > should use today. We have a chronic problem on the 0.98 branch
> with
> > > > > changes
> > > > > > tested on only Hadoop 2 later are found to break builds against
> > > Hadoop
> > > > 1,
> > > > > > since only 0.94 and 0.98 still support Hadoop 1.x. See
> > > > > > https://issues.apache.org/jira/browse/HBASE-12397 as an example.
> > > This
> > > > > issue
> > > > > > also illustrates that dropping support for 1.0 while retaining
> > > support
> > > > > for
> > > > > > 1.1 and later versions of Hadoop 1.x can reduce cross-version
> > > > > compatibility
> > > > > > complexity for at least the API involved in that issue, and
> > certainly
> > > > > > others. This in my opinion is a good thing.
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best regards,
> > > > > >
> > > > > >- Andy
> > > > > >
> > > > > > Problems worthy of attack prove their worth by hitting back. -
> Piet
> > > > Hein
> > > > > > (via Tom White)
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > >
> > > >- Andy
> > > >
> > > > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein
> > > > (via Tom White)
> > > >
> > >
> > >
> > >
> > > --
> > > Sean
> > >
> >
>
>
>
> --
> Best regards,
>
>- Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Re: Please welcome our latest committer, Sean Busbey

[Nicolas Liochon]

Congratulations and welcome, Sean!

On Fri, Dec 5, 2014 at 6:55 AM, Sean Busbey  wrote:

> On Thu, Dec 4, 2014 at 2:10 PM, Stack  wrote:
>
> > Sean has been doing excellent work around these environs. Your PMC made
> him
> > a committer in recognition.  Welcome Sean!
> >
> > St.Ack
> >
>
> Thanks everyone for your kind words. Hopefully I can repay y'all with more
> reviews. ;)
>
> --
> Sean
>

Re: All builds are recently failing with timeout or fork errors, let's change settings

[Nicolas Liochon]

I see in https://builds.apache.org/computer/ubuntu-2/load-statistics (used
for the 0.98 build mentionned by Andrew above) that we have a configuration
with 2 executors.
It means that jenkins tries to run 2 builds in parallel, each of these
builds will trigger its own set of surefire forks.

iirc, in the past:
 - we were not building on these machines, we were using only the hadoop
pool of machines
 - these machines were configured with 1 executor

>From what I see, there are two sets of machines
 - H*, for hadoop projects. H0 (for example) is configured with a single
executor.
 - ubuntu*, for everybody: ubuntu2 (for example) is configured with 2
executors.

0.98 and PreCommit-HBASE-Build are configured with: (ubuntu||Hadoop) &&
!jenkins-cloud-4GB && !H11

So it depends: lucky = H*. Unlucky = ubuntu*

I don't know who changed this, nor why, but may be we should not go to
ubuntu* machines. Or, if it's possible, we should have a different config
for these machines.



On Mon, Jan 19, 2015 at 7:11 PM, Andrew Purtell 
wrote:

> The 0.98 build is still showing this problem (latest as of now at
> https://builds.apache.org/job/hbase-0.98/803), so I went ahead and made
> the
> proposed change, but only to the 0.98 builds. I'll let you know if it
> provides any improvement.
>
>
> On Sun, Jan 18, 2015 at 10:00 AM, Andrew Purtell  >
> wrote:
>
> > Forked VMs are being killed in the 0.98 builds. That suggests
> > infrastructure issues.
> >
> > Having only one test execute in a forked runner does mean the finding of
> a
> > zombie and thread dumps or other state from the runner will identify and
> > characterize a sick test with no unrelated state mixed in.
> >
> >
> > > On Jan 17, 2015, at 7:43 PM, Stack  wrote:
> > >
> > > Agree, try anything to get our blues back.  We add back the //ism after
> > all
> > > settles.
> > >
> > > Do you think something has changed in INFRA Andy? Is it more contended?
> > Or,
> > > more likely, is it that we've been committing stuff that has
> destabilized
> > > builds? We had a good streak of blue there for a while. It just took
> some
> > > work fixing breakage and watching jenkins to make sure breakage didn't
> > > sneak in, but we've lapsed for sure.
> > >
> > > St.Ack
> > >
> > >> On Sat, Jan 17, 2015 at 9:19 AM, Dima Spivak 
> > wrote:
> > >>
> > >> Not running tests in parallel will definitely cut down on Surefire
> > >> flakiness (and in contention that sometimes leads to false failures in
> > >> resource-hungry tests), but it will probably also balloon test run
> > times to
> > >> about two hours. Probably worth it in the short term, but we
> > >> eventually need to do something about some of these heavy tests.
> > >>
> > >> -Dima
> > >>
> > >> On Friday, January 16, 2015, Andrew Purtell  >
> > >> wrote:
> > >>
> > >>> You might have missed the larger issue Ted.
> > >>>
> > >>>
> >  On Jan 16, 2015, at 4:48 PM, Ted Yu  > >> >
> > >>> wrote:
> > 
> >  With HBASE-12874, we should get a green build for branch-1.0
> > 
> >  FYI
> > 
> >  On Fri, Jan 16, 2015 at 12:20 PM, Andrew Purtell <
> apurt...@apache.org
> > >>> >
> >  wrote:
> > 
> > > See BUILDS-49 tracking issues specifically with 0.98 jobs, but I
> just
> > > noticed trunk, branch-1, and branch-1.0 all failed after I checked
> in
> > >> a
> > > shell doc fix due to a timeout or fork failure.
> > >
> > > I propose we update all Jenkins jobs to not run tests in parallel,
> > >> i.e.
> > >>> add
> > > "-Dsurefire.firstPartForkCount=1 -Dsurefire.secondPartForkCount=1"
> > >
> > > --
> > > Best regards,
> > >
> > >  - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> > >> Hein
> > > (via Tom White)
> > >>
> >
>

Re: 答复: Please welcome our newest committers, Andrey Stepachev and Duo Zhang!

[Nicolas Liochon]

Congrats, guys!
Le 10 mars 2015 17:56, "Anoop John"  a écrit :

> Congrats both of you..  Welcome..
>
> -Anoop-
>
> On Tue, Mar 10, 2015 at 11:41 AM, 冯宏华  wrote:
>
> > Cons and Welcome! Andrey and Duo:-)
> > 
> > 发件人: saint@gmail.com  代表 Stack <
> st...@duboce.net>
> > 发送时间: 2015年3月10日 5:52
> > 收件人: HBase Dev List
> > 主题: Please welcome our newest committers, Andrey Stepachev and Duo Zhang!
> >
> > These fellas are great and have been doing nice, juicy contribs over the
> > last year or two. Please welcome them to the hbase committer crew.
> >
> > Welcome aboard lads!
> > St.Ack
> >
>

Re: [DISCUSS] Dependency compatibility

[Nicolas Liochon]

>There's no reason our HDFS usage should be exposed in the HBase client code
I did look at this in the past, IIRC, our dependency was we use
hadoop-common code to read our XML configuration files
I would +1 a code duplication to remove the dependency.

I also think it is important for the end user.


On Fri, Mar 13, 2015 at 5:43 PM, Sean Busbey  wrote:

> On Fri, Mar 13, 2015 at 11:18 AM, Andrew Purtell 
> wrote:
>
> > > I'm -1 (non-binding) on weakening our compatibility promises. The more
> > we can
> > isolate our users from the impact of changes upstream the better.
> >
> > We can't though in general. Making compatibility promises we can't keep
> > because our upstreams don't (see the dependencies section of Hadoop's
> > compatibility guidelines) is ultimately an untenable position. *If* we
> had
> > some complete dependency isolation for MapReduce and coprocessors
> committed
> > then this could be a different conversation. Am I misstating this?
> >
>
>
> > ​In this specific instance we do have another option, so we could defer
> > this to a later time when a really unavoidable dependency change
> happens...
> > like a Guava update affecting HDFS. (We had one of those before.) We can
> > document the Jackson classpath issue with Hadoop >= 2.6 and provide
> > remediation advice in the troubleshooting section of the manual.
> >
> >
> I think we can solve this generally for Hadoop 2.6.0+. There's no reason
> our HDFS usage should be exposed in the HBase client code, and I think the
> application classpath feature for YARN in that version can isolate us on
> the MR side. I am willing to do this work in time for 1.1. Realistically I
> don't know the timeline for that version yet. If it turns out the work is
> more involved or my time is more constrained then I think, I'm willing to
> accept promise weakening as a practical matter.
>
> I'd be much more comfortable weakening our dependency promises for
> coprocessor than doing it in general. Folks running coprocessors should
> already be more risk tolerant and familiar with our internals.
>
> For upstreams that don't have the leverage on us of Hadoop, we solve this
> problem simply by not updating dependencies that we can't trust to not
> break our downstreams.
>
>
>
> > I would be disappointed to see a VOTE thread. That means we failed to
> reach
> > consensus and needed to fall back to process to resolve differences.
> >
> >
>
> That's fair. What about the wider audience issue on user@? There's no
> reason our DISCUSS threads couldn't go there as well.
>
>
>
> > Why don't we do the doc update and call it a day?
> >
>
> I've been burned by dependency changes in projects I rely on many times in
> the past, usually over changes in code sections that folks didn't think
> were likely to be used. So I'm very willing to do work now to save
> downstream users of HBase that same headache.
>
> --
> Sean
>

Re: [ANNOUNCE] Sean Busbey joins the Apache HBase PMC

[Nicolas Liochon]

Congrats, Sean.
Le 27 mars 2015 06:25, "张铎"  a écrit :

> Congratulations!
>
> 2015-03-27 13:12 GMT+08:00 Rajeshbabu Chintaguntla <
> chrajeshbab...@gmail.com
> >:
>
> > Congratulations Sean!
> >
> > Thanks,
> > Rajeshbabu.
> >
> > On Fri, Mar 27, 2015 at 10:28 AM, liushaohui 
> > wrote:
> >
> > > Congratulations, Sean!
> > >
> > > -Shaohui Liu
> > >
> > >
> > > On 03/27/2015 12:55 PM, Jerry He wrote:
> > >
> > >> Congratulations!!
> > >> On Mar 26, 2015 8:41 PM, "Pankaj kr"  wrote:
> > >>
> > >>  Congrats Sean...!!
> > >>>
> > >>> Regards,
> > >>> Pankaj
> > >>> -Original Message-
> > >>> From: Ted Yu [mailto:yuzhih...@gmail.com]
> > >>> Sent: 27 March 2015 08:29
> > >>> To: dev@hbase.apache.org; lars hofhansl
> > >>> Cc: u...@hbase.apache.org
> > >>> Subject: Re: [ANNOUNCE] Sean Busbey joins the Apache HBase PMC
> > >>>
> > >>> Congratulations, Sean.
> > >>>
> > >>> On Thu, Mar 26, 2015 at 2:49 PM, lars hofhansl
> > >>>  > >>> wrote:
> > >>>
> > >>>  Yeah! :)
> > From: Andrew Purtell 
> >    To: "u...@hbase.apache.org" ; "
> >  dev@hbase.apache.org" 
> >    Sent: Thursday, March 26, 2015 10:26 AM
> >    Subject: [ANNOUNCE] Sean Busbey joins the Apache HBase PMC
> > 
> >  On behalf of the Apache HBase PMC I"m pleased to announce that Sean
> >  Busbey has accepted our invitation to become a PMC member on the
> >  Apache HBase project. Sean has been an active and positive
> contributor
> >  in many areas, including on project meta-concerns such as
> versioning,
> >  build infrastructure, code reviews, etc. He's a natural and we're
> >  looking forward to many more future contributions.
> > 
> >  Welcome to the PMC, Sean!
> > 
> >  --
> >  Best regards,
> > 
> > - Andy
> > 
> >  Problems worthy of attack prove their worth by hitting back. - Piet
> >  Hein (via Tom White)
> > 
> > 
> > 
> > 
> > 
> > >
> >
>

Re: [ANNOUNCE] New HBase committer Mikhail Antonov

[Nicolas Liochon]

Congrats, Mikhail!

On Wed, Jul 1, 2015 at 5:32 PM, Jonathan Hsieh  wrote:

> Good stuff Mikhail!
>
> On Tue, Jun 30, 2015 at 5:14 PM, Andrew Purtell 
> wrote:
>
> > On behalf of the Apache HBase PMC, I am pleased to announce that Mikhail
> > Antonov has accepted the PMC's invitation to become a committer on the
> > project. We appreciate all of Mikhail's generous contributions thus far
> and
> > look forward to his continued involvement.
> >
> > Congratulations and welcome, Mikhail!
> >
> > --
> > Best regards,
> >
> >- Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
>
>
> --
> // Jonathan Hsieh (shay)
> // HBase Tech Lead, Software Engineer, Cloudera
> // j...@cloudera.com // @jmhsieh
>

Re: [ANNOUNCE] New HBase committer Esteban Gutierrez

[Nicolas Liochon]

Welcome aboard, Esteban.

On Wed, Jul 1, 2015 at 3:45 PM, Pankaj kr  wrote:

>
> Congrats Esteban..!! :)
>
> -Original Message-
> From: ramkrishna vasudevan [mailto:ramkrishna.s.vasude...@gmail.com]
> Sent: 01 July 2015 09:42
> To: dev@hbase.apache.org
> Subject: Re: [ANNOUNCE] New HBase committer Esteban Gutierrez
>
> Congrats and best wishes Esteban !!
>
> On Wed, Jul 1, 2015 at 9:30 AM, Anoop John  wrote:
>
> > Congrats Esteban..  Great work from you..  Keep going... All the best!
> >
> > -Anoop-
> >
> > On Wed, Jul 1, 2015 at 8:38 AM, Ted Malaska 
> > wrote:
> >
> > > Wow great job Esteban.  Wow
> > >
> > > On Tue, Jun 30, 2015 at 8:15 PM, Andrew Purtell 
> > > wrote:
> > >
> > > > On behalf of the Apache HBase PMC, I am pleased to announce that
> > Esteban
> > > > Gutierrez has accepted the PMC's invitation to become a committer on
> > the
> > > > project. We appreciate all of Esteban's generous contributions thus
> far
> > > and
> > > > look forward to his continued involvement.
> > > >
> > > > Congratulations and welcome, Esteban!
> > > >
> > > > --
> > > > Best regards,
> > > >
> > > >- Andy
> > > >
> > > > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein
> > > > (via Tom White)
> > > >
> > >
> >
>

Re: [DISCUSS] correcting abusive behavior on mailing lists was (Re: [DISCUSS] Multi-Cluster HBase Client)

[Nicolas Liochon]

> You’re a developer, or just some higher level primate that pounds code?

And it's not the first time. A 3 month ban seems ok to me.

On Wed, Jul 1, 2015 at 5:46 PM, Cody Marcel  wrote:

> Seems odd to create extra work for already crazy busy volunteers. You would
> basically be moderating one individual. Just objectively look at whether
> the value he adds to the community outweighs the time and emotional capitol
> spent in these conversations. I suspect the fact the this thread is
> happening is because some people have reached their limits to the abuse.
>
> On Tue, Jun 30, 2015 at 8:08 PM, lars hofhansl  wrote:
>
> > Right. Hence the moderating or temporary ban.
> >   From: Andrew Purtell 
> >  To: "dev@hbase.apache.org" 
> > Cc: Hbase-User 
> >  Sent: Tuesday, June 30, 2015 7:28 PM
> >  Subject: Re: [DISCUSS] correcting abusive behavior on mailing lists was
> > (Re: [DISCUSS] Multi-Cluster HBase Client)
> >
> > Huh? This is not about being caremad over some technical issue, this is
> > about preserving a productive space for communication free from toxic
> > people casting abuse while being "funny" and "smart". Please go back on
> the
> > original thread and re-read the interaction with Sean. Are you going to
> > defend that as within the bounds of acceptable behavior? Is this the
> first
> > time that has happened?
> >
> >
> >
> >
> > > On Jun 30, 2015, at 5:58 PM, lars hofhansl  wrote:
> > >
> > > Moderating is better than outright banning, I think.While Micheal is
> > sometimes infuriating, he's also funny and smart.
> > > Can we have a group of moderators? I'd volunteer, but not if I'm the
> > only one.
> > >
> > > -- Lars
> > >  From: Stack 
> > > To: Hbase-User 
> > > Cc: "dev@hbase.apache.org" 
> > > Sent: Tuesday, June 30, 2015 12:25 PM
> > > Subject: Re: [DISCUSS] correcting abusive behavior on mailing lists was
> > (Re: [DISCUSS] Multi-Cluster HBase Client)
> > >
> > >> On Tue, Jun 30, 2015 at 12:18 PM, Stack  wrote:
> > >>
> > >> I volunteer to help moderate.
> > >> St.Ack
> > > Sorry. Changed my mind. I'm not going to do more work because of
> Michael.
> > > Let the filters do the work for us. Ban him for 3 months.
> > >
> > >
> > > St.Ack
> > >
> > >
> > >
> > >>
> > >>> On Tue, Jun 30, 2015 at 12:08 PM, Sean Busbey 
> > wrote:
> > >>>
> > >>> (I took the liberty of moving us off of Ted's feature discussion
> > thread)
> > >>>
> > >>> A three month ban sounds better than a permanent ban. But it still
> > seems
> > >>> likely to just result in the ban-ee walking away from teh community
> or
> > >>> escalating.
> > >>>
> > >>> Would we consider enforcing moderation on posts for the three month
> > period
> > >>> instead of an outright ban?
> > >>>
> > >>> It's more work for whomever has moderator rights on the mailing list,
> > but
> > >>> it provides a better feedback cycle for improved behavior. Presuming
> > that
> > >>> I
> > >>> don't already have such access as a PMC member, I'd be willing to
> > >>> volunteer
> > >>> to help take on part of the burden.
> > >>>
> > >>> -Sean
> > >>>
> > >>> On Tue, Jun 30, 2015 at 1:59 PM, Andrew Purtell  >
> > >>> wrote:
> > >>>
> >  It was suggested privately that we try a temporary ban first, to be
> > >>> clear
> >  that current behavior and communication is unacceptable and won't be
> >  tolerated further, yet allow for the possibility of a return to the
> >  community should the message be received. So let me amend my
> proposal
> > -
> > >>> a
> >  three month temporary ban.
> > 
> >  Also, the reason I am asking for public feedback before calling for
> a
> > >>> vote
> >  is this would be the first time in the history of the project we
> would
> > >>> take
> >  this very unfortunate step. It pains me personally but enough is
> > >>> enough, in
> >  my opinion. However, if you are not comfortable with that then
> please
> > >>> speak
> >  up and I won't ask the PMC to vote on this.
> > 
> > 
> >  On Tue, Jun 30, 2015 at 11:41 AM, Andrew Purtell <
> apurt...@apache.org
> > >
> >  wrote:
> > 
> > > I've had enough and would like to ask the user and dev communities
> if
> >  they
> > > would mind if we vote on a permanent ban of Michael Segal - any and
> > >>> all
> > > email accounts he may choose to set up - from all HBase mailing
> > lists.
> >  The
> > > basic lack of courtesy and constant naysaying is corrosive. Nobody
> > >>> trying
> > > to volunteer their time here deserves his continuing abuse.
> > >
> > >
> > > On Tue, Jun 30, 2015 at 7:55 AM, Michael Segel <
> >  michael_se...@hotmail.com>
> > > wrote:
> > >
> > >> Sean,
> > >>
> > >> You’re a developer, or just some higher level primate that pounds
> > >>> code?
> > >>
> > >> I don’t want to embarrass you, but what do they teach in
> engineering
> > >> schools these days?
> > 
> > 
> >  --
> >  Best regards,
> > 
> > >>>

Re: Please welcome our newest committer, Rajeshbabu Chintaguntla

[Nicolas Liochon]

Congrats, Rajesh!


On Wed, Sep 11, 2013 at 10:37 PM, lars hofhansl  wrote:

> Welcome Rajesh.
>
>
>
> 
>  From: ramkrishna vasudevan 
> To: "dev@hbase.apache.org" ; "u...@hbase.apache.org"
> 
> Sent: Wednesday, September 11, 2013 9:17 AM
> Subject: Please welcome our newest committer, Rajeshbabu Chintaguntla
>
>
> Hi All,
>
> Please join me in welcoming Rajeshbabu (Rajesh) as our new HBase committer.
> Rajesh has been there for more than a year and has been solving some very
> good bugs around the Assignment Manger area.  He has been working on other
> stuff like HBase-Mapreduce performance improvement, migration scripts and
> off late in the Secondary Index related things.
>
> Rajesh has made his first commit to the pom.xml already.
> Once again, congratulations and welcome to this new role (smile).
>
> Cheers
> Ram
>

Re: [VOTE] The 1st hbase-0.96.0 release candidate is available for download

[Nicolas Liochon]

There is HBASE-9521 that I would like to see in the RC.
The mvn site failed on hadoop qa. I restarted it this morning / night, but
hadoop qa is down now.
Tested locally, it works.

Stack, do you mind if I commit it?

Nicolas


On Mon, Sep 16, 2013 at 5:38 PM, Stack  wrote:

> On Wed, Sep 11, 2013 at 1:26 PM, Stack  wrote:
>
> > On Wed, Sep 11, 2013 at 11:22 AM, Nick Dimiduk 
> wrote:
> >
> >> Likewise, some subset of Jon's HBASE-9245 should be resolved before we
> cut
> >> the next RC.
> >>
> >>
> > All sounds good by me.  I've upped to blocker items raised here but it
> > looks like I could still cut an RC tomorrow or perhaps the day after
> going
> > by what is outstanding.
> >
>
> I know I keep saying I'm going to cut a new RC and then the days go by, but
> it looks like all the asks are in (I will commit the outstanding this
> morning unless owners get to it), so RC this afternoon.
>
> Thanks for your patience,
> St.Ack
>

Re: [VOTE] The 6th hbase-0.96.0 release candidate is available for download

[Nicolas Liochon]

I'm +1.
I've downloaded the released, deployed it. Ran YCSB, kills the nodes. I've
ran a trunk client vs. the RC5, it works.
As Elliott on the good news, the modules, the recovery, MTTR ( ;-), ...
As him, I don't see any open issue that would break the wire compatibility.
We will need to fix the bad news he mentions. This is doable in a point
release.

I think it would be great to deliver now our 96.0 and do as the 0.94: a
minor release per month.

Thanks all,

Nicolas



On Fri, Oct 18, 2013 at 9:37 PM, Elliott Clark  wrote:

> I'm -0 this being the first 96 release.
>
> For the good news:
> * On a small cluster I've run all of the IT tests with both the calm
> chaos monkey and the slow deterministic chaos monkey.  Everything
> passed and looked good.  4 tests  of 5 hours each pass consistently.
> * Runs of YCSB Reads were better than stock 0.94 (naggles and short
> scans were the big wins).
> * This works well with Hadoop 2.2.0
> * Thanks to the znode watcher our MTTR if a region server crashes is
> pretty great.
> * Protobuf means our wire compat story is great.
> * The client module and the Testing util module mean that for
> downstreamers using HBase is easier than ever.
> * I see really large jumps in read and write throughput when the
> balancer runs so it looks like the new balancer does a good job.
> * Tar's are signed and look good
> * The jars are in place.
>
> For the Bad news:
> * This release won't scale up to large clusters busy cluster.  The
> client uses way too many threads.
> * Not a single IT test that uses the client from lots of hosts passes
> (TestLoadAndVerify or Big Linked List)
> * YCSB inserts fail if run from more than one driver node at a time
> * YCSB perf is pretty far behind 0.94 until compaction dominate.
>
> For the in-between news:
> * Upping the write buffer and lowering the number of async actions
> outstanding per server help and can help work around things.  But they
> are just work arounds and not everyone can up the write buffer enough
> to make this go away.
> * This can be fixed without changing anything on the wire so fixing
> this in a point release should be possible.
>
>
> On a larger live serving cluster I can't see this release being
> deployed ever.  But I also can't see a short term fix that can be
> applied in time for another rc to be cut.
>
>
> On Thu, Oct 17, 2013 at 2:49 PM, Ted Yu  wrote:
> > +1
> >
> > I did the following on a single node cluster which runs HBase against
> > hadoop-2:
> >
> > Checked out layouts for artifacts
> > Ran TestAcidGuarantees
> > Performed several commands using HBase shell
> > Ran rowcounter on a small table and verified the result through shell
> > 'count' command.
> >
> > All was good.
> >
> > On Tue, Oct 15, 2013 at 7:14 PM, Enis Söztutar 
> wrote:
> >
> >> Spent the day testing the RC. Here is my +1 based on:
> >>
> >>  - Downloaded artifacts for -hadoop1-bin -hadoop2-bin and -src
> >>  - Verified checksums for the artifacts (except for RMD160)
> >>Note: It would be useful if we can get these in a machine readable
> >> format to do automated verification with {sha*|md5}sum -c (for the next
> >> time)
> >>  - Verified signatures
> >>  - Checked out documentation
> >>  - Checked out layouts for artifacts
> >>  - Verified hadoop versions
> >>  - Run local cluster
> >>  - Build source artifact with hadoop1 and hadoop2
> >>  - Run small unit tests
> >>  - Deployed on a 7 node cluster with hadoop-2
> >>  - Run small tests including LoadTestTool, ITBigLinkedList,
> ITLoadAndVerify
> >> and some shell smoke tests
> >>
> >> Plus we have been running the tip of 0.96 branch through our test
> harness
> >> including many IT's with and without CM configurations.
> >>
> >> Let's ship it !
> >>
> >> Enis
> >>
> >>
> >> On Tue, Oct 15, 2013 at 6:25 PM, Devaraj Das 
> wrote:
> >>
> >> > +1. Based on some large scale tests with bits very close to RC5. Found
> >> two
> >> > issues - HBASE-9773 and HBASE-9777 and neither of them are release
> >> blockers
> >> > in my mind.
> >> >
> >> >
> >> > On Fri, Oct 11, 2013 at 4:20 PM, Stack  wrote:
> >> >
> >> > > hbase-0.96.0 will be our next major release.  It is intended to
> >> supplant
> >> > > the 0.94.x series.
> >> > >
> >> > > hbase-0.96.0RC5 is our sixth hbase-0.96.0 release candidate.
> >> > >
> >> > > The signed tarballs are available here:
> >> > >
> >> > >  http://people.apache.org/~stack/hbase-0.96.0RC5<
> >> > > http://people.apache.org/~stack/0.96.0RC5/>
> >> > > / 
> >> > >
> >> > > Note that hbase-0.96.0 comes in two flavors; a build that includes
> and
> >> > runs
> >> > > on hadoop-1.x and another for hadoop-2.x.  You must chose the hbase
> >> that
> >> > > suits your hadoop context.
> >> > >
> >> > > Both sets of hbase artifacts, for hadoop1 and for hadoop2, are
> >> available
> >> > in
> >> > > this maven staging repo:
> >> > >
> >> > >
> >> https://repository.apache.org/content/repositories/orgapachehbase-165/
> >> > >
> >> >

Re: DISCUSSION: Move to GIT?

[Nicolas Liochon]

I don't really care. I will follow.

I suppose we will keep our integration with JIRA, so we will have to submit
a patch anyway to get it validated by pre-commit stuff?

Nicolas



On Fri, Oct 25, 2013 at 8:30 PM, Sergey Shelukhin wrote:

> +1, also only use svn to commit
>
>
> On Fri, Oct 25, 2013 at 11:10 AM, Enis Söztutar  wrote:
>
> > +1000.
> >
> > Enis
> >
> >
> > On Fri, Oct 25, 2013 at 10:58 AM, Jonathan Hsieh 
> wrote:
> >
> > > +1.  I've actually only used SVN to commit patches.
> > >
> > > I'd suggest we maintain the same commit style (straight line rebased
> > > commits per release branch, branches off trunk for major features)
> after
> > > the move to git as opposed to the github style (lots of branch/merges).
> > >
> > > Jon.
> > >
> > >
> > > On Fri, Oct 25, 2013 at 10:44 AM, Elliott Clark 
> > wrote:
> > >
> > > > +1 I've actually been using git to do most of my work lately.
> > > >
> > > >
> > > > On Fri, Oct 25, 2013 at 10:16 AM, 黄浩松  wrote:
> > > >
> > > > > git is more powerful and flexible.
> > > > >
> > > > > 2013/10/26 Ted Yu :
> > > > > > +1 too.
> > > > > >
> > > > > >
> > > > > > On Fri, Oct 25, 2013 at 10:07 AM, Jesse Yates <
> > > jesse.k.ya...@gmail.com
> > > > > >wrote:
> > > > > >
> > > > > >> +1
> > > > > >>
> > > > > >> Do all my work in git anyways :)
> > > > > >>
> > > > > >> On Friday, October 25, 2013, Stack wrote:
> > > > > >>
> > > > > >> > At yesterday's dev meetup -- minutes to follow -- it was
> > suggested
> > > > we
> > > > > >> move
> > > > > >> > the project to git as our repo of truth.
> > > > > >> >
> > > > > >> > What do folks think?  How patches are contributed and
> committer
> > > will
> > > > > have
> > > > > >> > to change to match git-flow.
> > > > > >> >
> > > > > >> > Josh Eiser, an accumulo committer, was at the dev meetup
> > yesterday
> > > > and
> > > > > >> was
> > > > > >> > kind enough to write up the steps involved moving their
> project
> > > > over.
> > > > > >>  See
> > > > > >> > the discussion on how they come up w/ a process in particular.
> > > > > >> >
> > > > > >> > St.Ack
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > -- Forwarded message --
> > > > > >> > From: Josh Elser >
> > > > > >> > Date: Fri, Oct 25, 2013 at 12:25 AM
> > > > > >> > Subject: Accumulo's transition to git
> > > > > >> > To: Stack >
> > > > > >> >
> > > > > >> >
> > > > > >> > Stack,
> > > > > >> >
> > > > > >> > As promised, here's the info that I had collected when
> Accumulo
> > > made
> > > > > the
> > > > > >> > transition from SVN to Git.
> > > > > >> >
> > > > > >> > Make sure you have some mailing list consensus, and then open
> up
> > > an
> > > > > INFRA
> > > > > >> > ticket. For us, we requested the following things
> > > > > >> >
> > > > > >> > 1. Specify which SVN branches you want in Git, and if you want
> > > them
> > > > > >> > renamed. Also, which you want the default Git branch to be on
> > > clone.
> > > > > >> >
> > > > > >> > 2. If you have any contribs, specify which other SVN path you
> > want
> > > > as
> > > > > >> their
> > > > > >> > own Git repos.
> > > > > >> >
> > > > > >> > 3. Specify the format of the subject for the git commit
> > > > notifications
> > > > > >> email
> > > > > >> > and to which mailing list.
> > > > > >> >
> > > > > >> > 4. Request update of mirroring on git.a.o and
> > github.com/apache/.
> > > ..
> > > > > >> >
> > > > > >> > 5. Transition from svn2jira to git2jira (honestly, I don't
> > > remember
> > > > > what
> > > > > >> > this was anymore.. maybe commenting on Jira for issue mentions
> > in
> > > > > >> commits)
> > > > > >> >
> > > > > >> > 6. Request delivery of pull-requests to a given mailing list.
> > > > > >> >
> > > > > >> > 7. Request update reviewboard to the new repo.
> > > > > >> >
> > > > > >> > Our INFRA ticket can be found at [1]
> > > > > >> >
> > > > > >> > FWIW, we also had a huge discussion about how we were going to
> > use
> > > > > Git so
> > > > > >> > that we didn't constantly duplicate a bunch of commits
> > > > cherry-picking
> > > > > >> > changes everywhere. There was a bunch of teaching we had to do
> > to
> > > > make
> > > > > >> sure
> > > > > >> > we had a good commit history and could still use features like
> > > > > >> git-bisect.
> > > > > >> > The result of that is at [2].
> > > > > >> >
> > > > > >> > Joe Stein also sent us Kafka's use of Git that was helpful
> when
> > > > > writing
> > > > > >> > down how users should contribute patches. [3]
> > > > > >> >
> > > > > >> > I hope all this helps! Despite the pain we went through in
> > getting
> > > > > this
> > > > > >> all
> > > > > >> > set up (dealing with coordination of INFRA to lock the SVN
> > repos,
> > > > > >> transfer
> > > > > >> > to Git, verify accuracy, and people who then didn't know how
> to
> > > use
> > > > > >> Git), I
> > > > > >> > think everyone is happy we did it.
> > > > > >> >
> > > > > >> > - Josh
> > > > > >> >
> > > > > >> > [1] https://issues.apache.org/**ji

Re: HBase 0.96.0 performace

[Nicolas Liochon]

This test targets the client and the communication layers of HBase. It uses
a coprocessor that keeps the region empty (HBASE-10001), and all the
internal layers (memstore, cache, ...) are skipped. The region server and
the client are run on the same machine, this way we're not limited by the
network. What remains when there is no disk i/o not network i/o is the cpu,
and, indeed, that's the issue. Protobuf comes with a lot of object
creations and array copies. The difference between the 0.96 and the .96
head comes from the removal of many of these copies / creations. The
remaining gap with 0.94 could come from the remaining array copies, that's
what I'm looking at right now.

This test has some specificities:
 - it uses YCSB: YCSB does puts of 1KB size which can be small or large,
depending on your own use case. Smaller would mean that the cost of the
array copies is less important.
- there no network nor disk. Usually the disk i/o are the bottleneck.
- the data is not inserted in HBase: the memstore stays empty, we don't
write on HDFS (WAL or flush or compactions). The gets are done on an empty
table.
- there is a single region server, so by definition it gets all the load.
It maximizes the visibility of any GC on the server.
- the client and the region are on the same machine, so they are struggling
for the CPU. YCSB uses quite a lot of CPU resources as well.

So it's a development test to highlight a specific problem, not a real
benchmark.

Cheers,

Nicolas








On Tue, Nov 26, 2013 at 6:31 AM, lars hofhansl  wrote:

> As a general comment for 0.94 vs 0.96 testing... Please always make sure
> we are testing against the same default options.
>
> In 0.96 we increased the default blockCache, enabled scanner caching,
> disabled Nagle's, increased the handler count, etc.
> In order to compare apples to apples we should always add the following
> config options to 0.94 (in hbase-site.xml)
>
> hbase.regionserver.global.memstore.lowerLimit = 0.38
> hfile.block.cache.size = 0.4
> hbase.ipc.client.tcpnodelay = true
> hbase.regionserver.handler.count = 30
> hbase.client.scanner.caching = 100
>
> There might be more, but these are obvious ones.
>
> Thanks.
>
> -- Lars
>
>
>
>
> 
>  From: Jerry He 
> To: dev 
> Sent: Monday, November 25, 2013 2:09 PM
> Subject: Re: HBase 0.96.0 performace
>
>
> Hi, Ted
>
> Do you have the slides?
> I don't have the full slides.  The image was passed to me by a co-worker in
> China.  We are evaluating it.
> I am copying the numbers below:
>
> Release3 YCSB clients (puts/seconds) 5 YCSB clients
> (puts/seconds)
> 0.94.14 178K  220K
> 0.96.0   20K   15K
> 0.96.1   173K  168K
>
> Release10 YCSB clients (reads/seconds) 25 YCSB clients
> (reads/seconds)
> 0.94.14 50K52K
> 0.96.0   23K   23K
> 0.96.1   52K   54K
>
>
>
> On Mon, Nov 25, 2013 at 1:25 PM, Ted Yu  wrote:
>
> > Jerry:
> > Where did you obtain the picture ?
> >
> > If you got it from slides posted on China Hadoop Summit website, can you
> > share the link ?
> >
> > Cheers
> >
> >
> > On Tue, Nov 26, 2013 at 3:55 AM, Ted Yu  wrote:
> >
> > > The picture didn't go through.
> > >
> > > Correction: the comparison between hbase 0.96.0 and 0.96.1 was done by
> > > Nicolas Liochon. This was not done at EBay.
> > >
> > > In my talk, there was another comparison to show that MTTR improved a
> lot
> > > from 0.94 to 0.96 - this was done at EBay.
> > >
> > > When Stack cuts 0.96.1 RC0, you would get the list of JIRAs that
> > > contributed to the speedup from 0.96.0 to 0.96.1
> > >
> > > Cheers
> > >
> > >
> > > On Tue, Nov 26, 2013 at 3:50 AM, Jerry He  wrote:
> > >
> > >> Hi,  Ted, and hbase experts
> > >>
> > >> Below picture is a performance comparison between hbase 0.96.0 and
> > >> 0.96.1, shared by Ted Yu in China Hadoop Summit today. This perf
> > testing is
> > >> said to be executed on ebay's real cluster.
> > >>
> > >> It is surprising 0.96.0 is such worse compared to 0.96.1 and even
> > >> 0.94.14.  Are these numbers official and the performance degradation
> > true?
> > >>   What patch in 0.96.1 fixed this?
> > >>
> > >>
> > >>
> > >>
> > >>
> > >
> >
>

Re: Integration Test Slow Do wn

[Nicolas Liochon]

I'm going to have a look at this today, Elliott.

Le 6 déc. 2013 00:31, "Elliott Clark"  a écrit :

> On Thu, Dec 5, 2013 at 3:27 PM, Devaraj Das  wrote:
>
> > Elliott, do you know how much it takes without chaosmonkey
>
>
> That's what the Calm monkey does.  So 4.5 hours to 6.3
>

Re: Integration Test Slow Do wn

[Nicolas Liochon]

There was a real issue. HBASE-10097 should fix it.

Let me know how it works for you, Elliott.
If it's possible, I would be interested to see your settings and the
history of the test run time, typically vs. 0.96.0.

Thanks Elliott,

Nicolas




On Fri, Dec 6, 2013 at 9:19 AM, Nicolas Liochon  wrote:

> I'm going to have a look at this today, Elliott.
>
> Le 6 déc. 2013 00:31, "Elliott Clark"  a écrit :
>
>  On Thu, Dec 5, 2013 at 3:27 PM, Devaraj Das  wrote:
>>
>> > Elliott, do you know how much it takes without chaosmonkey
>>
>>
>> That's what the Calm monkey does.  So 4.5 hours to 6.3
>>
>

Re: We haven't had a clean hadoopqa build since 10/2/13.

[Nicolas Liochon]

We're alone on the precommit but not on the postcommit.


On Wed, Dec 11, 2013 at 9:15 PM, Elliott Clark  wrote:

> Fixing the site build so that it doesn't take an install first seems to be
> eluding me. The other option is to make the qa bot do an install.  But for
> that to be safe we need to make sure that the maven repositories aren't
> shared on any concurrently running hadoop qa runs.  Can someone with
> jenkins perms see if that's already set up ?
>
>
> On Wed, Dec 11, 2013 at 7:59 AM, Jonathan Hsieh  wrote:
>
> > great - the fix can be done there. I did a git bisect to find the commits
> > that caused the problem and emailed the committer.
> >
> > On Wednesday, December 11, 2013, Ted Yu wrote:
> >
> > > For the site build error,
> > > HBASE-9729 was
> > > logged.
> > >
> > > I do agree with Jon that no new warnings should be introduced.
> > >
> > > Currently 3 new Findbugs appear in QA reports.
> > >
> > > Cheers
> > >
> > >
> > > On Wed, Dec 11, 2013 at 7:13 AM, Jonathan Hsieh  > >
> > > wrote:
> > >
> > > > Just a reminder -- we do not want to introduce new javadoc warnings,
> > > > findbugs warnings, or break the docs/site compile.  Please clean up
> > after
> > > > you commits!  If you are committing something else and a warning
> shows
> > > up,
> > > > please investigate enough in order ping to the committer responsible
> > > > (email/jira) or even better just fix the problem.
> > > >
> > > > Thanks,
> > > > Jon.
> > > >
> > > >
> > > > --
> > > > // Jonathan Hsieh (shay)
> > > > // Software Engineer, Cloudera
> > > > // j...@cloudera.com 
> > > >
> > >
> >
> >
> > --
> > // Jonathan Hsieh (shay)
> > // Software Engineer, Cloudera
> > // j...@cloudera.com
> >
>

Re: Forked large unit tests seem slow to terminate

[Nicolas Liochon]

> I wonder if there is anything that can be done to wait for the forked
process to actually terminate?
As far as I know, no. We could be hacky, and, in the afterClass, start a
deamon thread that will do a kill -9 after a few seconds. Or something
similar, but that's very hacky imho.

> Is this a Surefire bug?
I don't think so, even if surefire has its share of bugs.
The issue could be in HBase itself: surefire execute the "afterClass"
method, this method does finish, but HBase then gets stuck in the shutdown
hooks.




On Fri, Dec 13, 2013 at 1:26 AM, Andrew Purtell  wrote:

> We eventually have no zombies, at least the zombie detector does not find
> anything, but meanwhile on memory limited EC2 instances I'm seeing tests
> killed by the OOM killer, very likely meaning that previous tests are still
> in the process of shutting down and are hanging around in the process
> table, consuming RAM, VMM, etc. I didn't see this before the switch to
> Hadoop 2 as default. I'm not suggesting we switch back. I wonder if there
> is anything that can be done to wait for the forked process to actually
> terminate? Is this a Surefire bug?
>
> --
> Best regards,
>
>- Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Re: Please welcome our newest committer, Liang Xie!

[Nicolas Liochon]

Welcome!


On Mon, Dec 23, 2013 at 3:35 AM, Aleksandr Shulman wrote:

> Congrats Liang!
>
>
> On Fri, Dec 20, 2013 at 2:11 PM, Sergey Shelukhin  >wrote:
>
> > Congrats!
> >
> >
> > On Thu, Dec 19, 2013 at 2:05 PM, Nick Dimiduk 
> wrote:
> >
> > > Congrats Liang!
> > >
> > >
> > > On Wed, Dec 18, 2013 at 4:47 PM, Stack  wrote:
> > >
> > > > 谢良 has been doing great work for ages now; we're glad to have him on
> > > board
> > > > ("One of us!").  Keep up the great work 谢良.
> > > >
> > > > St.Ack
> > > >
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
>
>
> --
> Best Regards,
>
> Aleks Shulman
> 847.814.5804
> Cloudera
>

Re: HBASE-6104

[Nicolas Liochon]

I've been bitten by this in the past as well. Our precommit env is specific
to hadoop. It launches a specific script, and this script overrides the
jenkins property (source ${WORKSPACE}/nightly/hudsonEnv.sh)
We don't own this script. May be we should just copy its content into the
jenkins shell parameter, but don't remember if there is an impact.
HBASE-8731 was about changing the script itself.


On Thu, Dec 26, 2013 at 6:14 AM, Andrew Purtell  wrote:

> Ok, so I spoke too soon, the ASF Jenkins builds for 0.98 are both set up to
> use "1.6 (latest)", but I had a test here that would fail on one build but
> not another, locally on 1.6u43, and on some ASF Jenkins builds but not
> others, and never on a 1.7 build (locally). Or a precommit build, which was
> 1.6u20. I thought JRE version was the variable but there is something else
> uncontrolled here. That's actually worse. So scratch my above suggestions.
> I am just going to cut my losses and move on.
>
> I will not update the ASF Jenkins job configurations until when/if there is
> further discussion.
>
>
> On Wed, Dec 25, 2013 at 9:02 PM, Andrew Purtell 
> wrote:
>
> > I should also mention the precommit builds all passed, which suggests to
> > me they are using JDK 7 also.
> >
> >
> > On Wed, Dec 25, 2013 at 9:00 PM, Andrew Purtell  >wrote:
> >
> >> HBASE-6104 introduced a feature developed and tested with JDK 7.
> >> Subsequent to review and commit, two builds on ASF Jenkins began
> failing,
> >> the 0.98 on Hadoop 1.1 build, and the trunk build. After chasing a red
> >> herring that looked like a test only failure for a bit it seems the
> problem
> >> was deeper and manifested on JDK 6. I have reverted the change,
> unscheduled
> >> the feature from 0.98, and may return to it some time in the future
> after
> >> the sting wears off. In the meantime, I have two suggestions:
> >>
> >> 1. We should set all of the primary builds like trunk and 0.98 on Hadoop
> >> 1.1 to use JDK 7, like the other builds. The inconsistent use of JDK/JRE
> >> version will be an occasional source of confusion.
> >>
> >> 2. We should explicitly set up jobs that use JDK/JRE 6 and name them as
> >> such.
> >>
> >> --
> >> Best regards,
> >>
> >>- Andy
> >>
> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> >> (via Tom White)
> >>
> >
> >
> >
> > --
> > Best regards,
> >
> >- Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
>
>
> --
> Best regards,
>
>- Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Re: 1.5.0-SNAPSHOT conflicting with hbase-proto 0.96.x

[Nicolas Liochon]

Should we not rename ZeroCopyLiteralByteString to something like
HBasePrivateZeroCopyLiteralByteString to be sure that we won't have name
conflicts in the future?



On Mon, Jan 27, 2014 at 3:36 AM, tsuna  wrote:

> Yes, I just ran into the same issue.  Sigh.  I guess one workaround is
> to try to make AsyncHBase's jar show up first on the class path,
> although that's not always as easy to do as it might sound.
>
> I'm copying hbase-dev so that they are aware of the problem.  My trick
> was ported to HBase in HBASE-9867, but it wasn't ported properly.  It
> doesn't look like the zeroCopyGetBytes helper is used in HBase (I
> checked 0.96, 0.98, and trunk).  So the best course of action seems to
> be to fix the signature of the method in HBase.
>
> I created an issue and proposed a patch for HBase here:
> https://issues.apache.org/jira/browse/HBASE-10422
>
> On Tue, Jan 14, 2014 at 1:49 AM,   wrote:
> > * hbase-protocol-0.96.1.1-hadoop2.jar
> > * asynchbase-1.5.0-SNAPSHOT.jar
> >
> > Both packages contains class
> > "com.google.protobuf.ZeroCopyLiteralByteString", but with different API:
> > * public static byte[] zeroCopyGetBytes(ByteString buf)
> > * public static byte[] zeroCopyGetBytes(final LiteralByteString buf)
> >
> >
> > When running a program with both dependencies, following error will
> occur in
> > most cases:
> >
> > 14/01/14 13:45:41 ERROR async.RegionClient:1093 - Unexpected exception
> from
> > downstream on [id: 0x33adfae4, /127.0.0.1:37648 => /127.0.0.1:34508]
> > java.lang.NoSuchMethodError:
> >
> com.google.protobuf.ZeroCopyLiteralByteString.zeroCopyGetBytes(Lcom/google/protobuf/ByteString;)[B
> > at org.hbase.async.Bytes.get(Bytes.java:297)
> > at org.hbase.async.KeyValue.fromCell(KeyValue.java:311)
> > at org.hbase.async.GetRequest.convertResult(GetRequest.java:514)
> > at org.hbase.async.GetRequest.extractResponse(GetRequest.java:496)
> > at
> >
> org.hbase.async.RegionClient$1GetClosestRowBefore.deserialize(RegionClient.java:651)
> > at org.hbase.async.RegionClient.decode(RegionClient.java:1308)
> > at org.hbase.async.RegionClient.decode(RegionClient.java:89)
> > at
> >
> org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500)
> > at
> >
> org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
> > at
> org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1080)
> > at
> >
> org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:2652)
> > at
> > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> > at
> > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> > at
> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
> > at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:744)
> >
> > It will be very simple to rename ZeroCopyLiteralByteString class in
> > asynchbase into something different and republish new jar.
> > Hope, someone may propose better solution.
>
>
>
> --
> Benoit "tsuna" Sigoure
>

Re: 1.5.0-SNAPSHOT conflicting with hbase-proto 0.96.x

[Nicolas Liochon]

Hum the API is different today, and will remain different after
HBASE-10422.
https://github.com/apache/hbase/blob/trunk/hbase-protocol/src/main/java/com/google/protobuf/ZeroCopyLiteralByteString.java

has:  public static ByteString wrap(final byte[] array, int offset, int
length) {

And this method is not in
https://github.com/tsuna/asynchbase/blob/master/src/protobuf/ZeroCopyLiteralByteString.java

It's too brittle. I created HBASE-10431 for this.



On Mon, Jan 27, 2014 at 5:35 PM, tsuna  wrote:

> On Mon, Jan 27, 2014 at 1:02 AM, Nicolas Liochon 
> wrote:
> > Should we not rename ZeroCopyLiteralByteString to something like
> > HBasePrivateZeroCopyLiteralByteString to be sure that we won't have name
> > conflicts in the future?
>
> I don't mind keeping the same name as long as we agree on the API.
> I don't expect this class to change much if at all anyway.  It's just
> really unfortunate that this method was changed, what's more with a
> signature that renders it unusable.
>
> --
> Benoit "tsuna" Sigoure
>

Re: DISCUSSION: Enis for 1.0.0 RM

[Nicolas Liochon]

+1


On Thu, Feb 13, 2014 at 5:35 AM, ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com> wrote:

> +1 from me.
>
>
> On Thu, Feb 13, 2014 at 1:34 AM, Andrew Purtell  >wrote:
>
> > That's great!
> >
> >
> > > On Feb 11, 2014, at 3:20 PM, Stack  wrote:
> > >
> > > Over on the tail of concurrent thread 'DISCUSSION: 1.0.0', Enis
> > volunteered
> > > (again) as 1.0.0 Release Manager.
> > >
> > > Unless objection, I'd say let this be so.  We could run a vote but my
> > > thinking is that any one interested in RM'ing can just volunteer to do
> > the
> > > next one; it's healthy spreading the RM role IMO.
> > >
> > > St.Ack
> >
>

Re: DISCUSS: We need a mascot, a totem HBASE-4920

[Nicolas Liochon]

+1 for the Orca.
Personally, I like this one:
https://issues.apache.org/jira/secure/attachment/12511412/HBase%20Orca%20Logo.jpg


On Tue, Mar 4, 2014 at 8:55 AM, Andrew Purtell  wrote:

> Go for it, I'd say.
>
>
> On Tue, Mar 4, 2014 at 11:31 AM, Stack  wrote:
>
> > Our Esteban reminded me of this old issue today voting we just take on an
> > Orca as our mascot:
> >
> > Unless objection, I'll just put up
> >
> >
> https://issues.apache.org/jira/secure/attachment/12505693/Orca_479990801.jpgas
> > our mascot from here on out and close out this two year old item.
> >
> > St.Ack
> >
>
>
>
> --
> Best regards,
>
>- Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Re: DISCUSS: We need a mascot, a totem HBASE-4920

[Nicolas Liochon]

Yes, let's decide first on the animal itself (well we're done it seems),
and use another discussion thread for the picture.


On Wed, Mar 5, 2014 at 7:56 AM, Eric Charles  wrote:

> On 03/04/2014 11:19 PM, Jean-Marc Spaggiari wrote:
>
>> Can we not have Stack as the mascot? ;)
>>
>
> Well, Stack's place is on the HBase chair, not as a puppet mascot on my
> desk :)
>
>
>  Else, I'm fine with the Orca too,
>>
>
> +1 Not sure if Orca is a fish, but it is fast, powerful and human can
> trust it (until a certain point, I've seen a reportage on it) - just like
> HBase...
>
>
>  as I will be fine with almost anything else.
>>
>>
> Not me - would you agree with a baby dog?
> https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcSy1Z-
> dTEBG6lZqm5ai2ky5vtcWPxOTMLg-q9RI63ifYLnFTmS8Ig
>
> Apart from this, if this a thread for Orca of to choose a picture. I read
> two different concerns here. Maybe worth to quickly bootstrap a vote for
> Orca, and after a vote with a list of proposals (I've got my favorite, but
> prefer casting a vote to avoid to be lost in the mass)
>
>
>
>
>> 2014-03-04 17:13 GMT-05:00 Ted Yu :
>>
>>  +1
>>>
>>>
>>> On Tue, Mar 4, 2014 at 2:12 PM, Enis Söztutar 
>>> wrote:
>>>
>>>  Orca +1.


 On Tue, Mar 4, 2014 at 12:00 PM, Todd Lipcon  wrote:

  On Tue, Mar 4, 2014 at 11:44 AM, Stack  wrote:
>
> So I suggest that we put up the
>> graphic that is in the head of this thread for now; it is a clean,
>> non-nonsense representation of the non-nonsense fish.
>>
>
>
> That's nonsense! Orcas are not fish!
>
> Nonsensically
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
>

>>>
>>

Re: Re:Re: Please welcome our newest committer, Mr. Honghua Feng

[Nicolas Liochon]

Congrats and welcome!



On Thu, Mar 13, 2014 at 6:34 AM, rajeshbabu chintaguntla <
rajeshbabu.chintagun...@huawei.com> wrote:

> Congratulations Honghua.
>
> Thanks,
> Rajeshbabu.
> 
> From: Chunhui Shen [zju...@163.com]
> Sent: Thursday, March 13, 2014 8:00 AM
> To: dev@hbase.apache.org
> Subject: Re:Re: Please welcome our newest committer, Mr. Honghua Feng
>
> Congrats !!!
>
>
>
>
>
> At 2014-03-13 10:07:46,haosdent  wrote:
> >Congratulations!
> >
> >
> >On Thu, Mar 13, 2014 at 9:59 AM, Rabbit's Foot
> >wrote:
> >
> >> Congratulations and welcome!
> >>
> >>
> >> 2014-03-13 7:59 GMT+08:00 Andrew Purtell :
> >>
> >> > Congratulations!
> >> >
> >> >
> >> > On Wed, Mar 12, 2014 at 2:52 PM, Stack  wrote:
> >> >
> >> > > Please welcome our latest committer, 冯宏华.  He has contributed some
> >> sweet
> >> > > patches since he started hanging out in these parts.  Keep up the
> great
> >> > > work Honghua.
> >> > >
> >> > > St.Ack
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Best regards,
> >> >
> >> >- Andy
> >> >
> >> > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> >> > (via Tom White)
> >> >
> >>
> >
> >
> >
> >--
> >Best Regards,
> >Haosdent Huang
>

Re: HBase region server failure issues

[Nicolas Liochon]

What you described seems to be the favored nodes feature, but there are
still some open (and stale...) jiras there: HBASE-9116 and cie.
You may also want to look at the hbase.master.distributed.log.replay
option, as is allows writes during recovery.
And for the client there is hbase.status.published that can help...





On Wed, Apr 16, 2014 at 6:35 AM, Claudiu Soroiu  wrote:

> Yes, overall the second WAL would contain the same data but differently
> distributed.
> A server will have in the second WAL data from the regions that it will
> take over if they fail.
> It is just an idea as it might not be good to duplicate the data across the
> cluster.
>
>
>
>
> On Wed, Apr 16, 2014 at 12:36 AM, Ted Yu  wrote:
>
> > Would the second WAL contain the same contents as the first ?
> >
> > We already have the code that adds interceptor on the calls to the
> > namenode#getBlockLocations so that blocks on the same DN as the dead RS
> are
> > placed at the end of the priority queue..
> > See addLocationsOrderInterceptor()
> > in hbase-server/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java
> >
> > This is for faster recovery in case regionserver is deployed on the same
> > box as the datanode.
> >
> >
> > On Tue, Apr 15, 2014 at 1:43 PM, Claudiu Soroiu 
> wrote:
> >
> > > First of all, thanks for the clarifications.
> > >
> > > **how about 300 regions with 3x replication?  Or 1000 regions? This
> > > is going to be 3000 files. on HDFS. per one RS.**
> > >
> > > Now i see that the trade-off is how to reduce the recovery time without
> > > affecting the overall performance of the cluster.
> > > Having too many WAL's affects the write performance.
> > > Basically multiple WAL's might improve the process but the number of
> > WAL's
> > > should be relatively small.
> > >
> > > Would it be feasible to know ahead of time where a region might
> activate
> > in
> > > case of a failure and have for each region server a second WAL file
> > > containing backup edits?
> > > E.g. If machine B crashes then a region will be assigned to node A,
>  one
> > to
> > > node C, etc.
> > > Also another view would be: Server A will backup a region from Server B
> > if
> > > crashes, a region from server C, etc. Basically this second WAL will
> > > contain the data that is needed to fast recover a crashed node.
> > > This adds additional redundancy and some degree of complexity to the
> > > solution but ensures data locality in case of a crash and faster
> > recovery.
> > >
> > > **What did you do Claudiu to get the time down?**
> > >
> > >  Decreased the hdfs block size to 64 megs for now.
> > >  Enabled settings to avoid hdfs stale nodes.
> > >  Cluster I tested this was relatively small - 10 computers.
> > >  I did tuning for zookeeper sessions to keep the heartbeat at 5 seconds
> > for
> > > the moment, and plan to decrease this value.
> > >  At this point dfs.heartbeat.interval is set at the default 3 seconds,
> > but
> > > this I also plan to decrease and perform a more intensive test.
> > >  (Decreasing the times is based on the experience with our current
> system
> > > configured at 1.2 seconds and didn't had any issues even under heavy
> > loads,
> > > obviously stop the world GC times should be smaller that the heartbeat
> > > interval)
> > >  And I remember i did some changes for the reconnect intervals of the
> > > client to allow him to reconnect to the region as fast as possible.
> > >  I am in an early stage of experimenting with hbase but there are lot
> of
> > > things to test/check...
> > >
> > >
> > >
> > >
> > > On Tue, Apr 15, 2014 at 11:03 PM, Vladimir Rodionov
> > > wrote:
> > >
> > > > *We also had a global HDFS file limit to contend with*
> > > >
> > > > Yes, we have been seeing this from time to time in our production
> > > clusters.
> > > > Periodic purging of old files helps, but the issue is obvious.
> > > >
> > > > -Vladimir Rodionov
> > > >
> > > >
> > > > On Tue, Apr 15, 2014 at 11:58 AM, Stack  wrote:
> > > >
> > > > > On Mon, Apr 14, 2014 at 1:47 PM, Claudiu Soroiu  >
> > > > wrote:
> > > > >
> > > > > > 
> > > > >
> > > > > After some tunning I managed to
> > > > > > reduce it to 8 seconds in total and for the moment it fits the
> > needs.
> > > > > >
> > > > >
> > > > > What did you do Claudiu to get the time down?
> > > > > Thanks,
> > > > > St.Ack
> > > > >
> > > >
> > >
> >
>

Re: On coprocessor API evolution

[Nicolas Liochon]

Hi,

(With Apache still lagging on mails, it may be difficult to have a
discussion...)

For 1.0+, I think that registering observer as proposed in 11125 works well.
For 0.98, could we do something like this?
 - new coprocessor hooks can be added between minor releases
 - existing coprocessors hooks are not removed between minor releases
 - a coprocessor can extend the default implementation. Binary
compatibility when migrating to a newer minor release is ensured.
 - a coprocessor can implement directly the interface, but in this case the
application needs to be updated and recompiled between minor releases .
 - new hooks are always tagged with @since. This helps the coprocessor
developer if he needs to support multiple minor version.
 - between major release, everything can happen.

fwiw, Java 8 supports default implementations in interfaces:
http://docs.oracle.com/javase/tutorial/java/IandI/defaultmethods.html

Cheers,

Nicolas





On Thu, May 15, 2014 at 3:13 AM, Andrew Purtell  wrote:

> Because coprocessor APIs are so tightly bound with internals, if we apply
> suggested rules like as mentioned on HBASE-11054:
>
>   I'd say policy should be no changes to method apis across minor
> versions
>
> This will lock coprocessor based components to the limitations of the API
> as we encounter them. Core code does not suffer this limitation, we are
> otherwise free to refactor and change internal methods. For example, if we
> apply this policy to the 0.98 branch, then we will have to abandon further
> security feature development there and move to trunk only. This is because
> we already are aware that coprocessor APIs as they stand are insufficient
> still.
>
> Coprocessor APIs are a special class of internal method. We have had a
> tension between allowing freedom of movement for developing them out and
> providing some measure of stability for implementors for a while.
>
> It is my belief that the way forward is something like HBASE-11125. Perhaps
> we can take this discussion to that JIRA and have this long overdue
> conversation.
>
> Regarding security features specifically, I would also like to call your
> attention to HBASE-11127. I think security has been an optional feature
> long enough, it is becoming a core requirement for the project, so should
> be moved into core. Sure, we can therefore sidestep any issues with
> coprocessor API sufficiency for hosting security features. However, in my
> opinion we should pursue both HBASE-11125 and HBASE-11127; the first to
> provide the relative stability long asked for by coprocessor API users, the
> latter to cleanly solve emerging issues with concurrency and versioning.
>
>
> --
> Best regards,
>
>- Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Re: ANNOUNCEMENT: Git Migration In Progress (WAS => Re: Git Migration)

[Nicolas Liochon]

Can we now commit again, or is the migration still in progress?

Thanks,

Nicolas


On Fri, May 23, 2014 at 7:31 AM, Stack  wrote:

> I added to the refguide here:
> http://hbase.apache.org/book.html#git.patch.flow
>
> Also updated our build box references so point to git instead of svn.
>
> St.Ack
>
>
> On Thu, May 22, 2014 at 11:06 AM, Enis Söztutar  wrote:
>
> > Thanks guys for checking.
> >
> > Can we at least agree on always using something like the following flow
> for
> > checking in for now:
> >  - Commit the patch to trunk.
> >  - Try to cherry-pick the patch to 0.98 / 0.96 if possible
> >  - If not, manually commit the patch to the branch.
> >
> > If the patch is applicable to the branch without issues, we should
> > cherry-pick which will help us in merges / comparisons etc.
> >
> > Enis
> >
> >
> > On Thu, May 22, 2014 at 10:16 AM, Stack  wrote:
> >
> > > What Andy said.
> > >
> > > I checked trunk and 0.96 branch content (compensating for above
> commits).
> > >  I confirmed list of branches and tags are the same.
> > >
> > > Thanks for sending the note saying repo is open again Andy.
> > >
> > > St.Ack
> > >
> > >
> > > On Thu, May 22, 2014 at 9:45 AM, Andrew Purtell 
> > > wrote:
> > >
> > > > That is unfortunate, because there was not an all clear sent to dev@
> .
> > I
> > > > suppose we are "lucky" that otherwise the diffs are fine.  So I guess
> > > it's
> > > > open season on the Git repo then. Would have been nice for folks to
> > have
> > > > waited for Stack or someone else to write back verifying file
> contents
> > > were
> > > > good.
> > > >
> > > >
> > > > On Thu, May 22, 2014 at 9:43 AM, Anoop John 
> > > wrote:
> > > >
> > > > > No Andy. Those were commits to Git after the migration.
> > > > >
> > > > > -Anoop-
> > > > >
> > > > > On Thu, May 22, 2014 at 10:11 PM, Andrew Purtell <
> > apurt...@apache.org
> > > > > >wrote:
> > > > >
> > > > > > So someone made a commit to SVN **after** the migration was in
> > > > progress??
> > > > > >
> > > > > >
> > > > > > On Thu, May 22, 2014 at 9:39 AM, Ted Yu 
> > wrote:
> > > > > >
> > > > > > > Andrew:
> > > > > > > The diff shown in http://pastebin.com/Pvk3BH4i corresponds to
> > > > > > HBASE-11219
> > > > > > > which was integrated to master and 0.98 last night.
> > > > > > >
> > > > > > > In my local git workspace for 0.98, I do see this change.
> > > > > > >
> > > > > > > FYI
> > > > > > >
> > > > > > >
> > > > > > > On Thu, May 22, 2014 at 9:28 AM, Andrew Purtell <
> > > apurt...@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Infra has closed the migration ticket.
> > > > > > > >
> > > > > > > > I looked at tags for trunk/master and 0.98, and these look
> > fine.
> > > > > > > >
> > > > > > > > Unfortunately there are differences between SVN checkouts and
> > Git
> > > > > > > > checkouts. SVN has changes on trunk/master and 0.98 that did
> > not
> > > > make
> > > > > > it
> > > > > > > > over to Git looks like.
> > > > > > > >
> > > > > > > > master/trunk: http://pastebin.com/dQ6SU2Dz
> > > > > > > >
> > > > > > > > 0.98: http://pastebin.com/Pvk3BH4i
> > > > > > > >
> > > > > > > > 0.96: Good!
> > > > > > > >
> > > > > > > > 0.94: Good!
> > > > > > > >
> > > > > > > > 0.89-fb​: Good!
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, May 21, 2014 at 10:55 PM, Stack 
> > > wrote:
> > > > > > > >
> > > > > > > > > Thanks T.
> > > > > > > > >
> > > > > > > > > The trunk test is still running fine.  Checkout local looks
> > > good
> > > > > too.
> > > > > > >  I
> > > > > > > > > tried a branch.  It seems right too.  Asking about
> > discrepancy
> > > in
> > > > > the
> > > > > > > tag
> > > > > > > > > listings between the branches up in the INFRA issue.git.
> > >  Working
> > > > > on
> > > > > > > file
> > > > > > > > > compares of svn and git checkouts  Will report back.
> > > > > > > > >
> > > > > > > > > St.Ack
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, May 21, 2014 at 10:02 PM, Ted Yu <
> > yuzhih...@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I pointed trunk Jenkins job to git repo and triggered a
> > > build.
> > > > > > > > > >
> > > > > > > > > > So far the tests are running fine.
> > > > > > > > > >
> > > > > > > > > > FYI
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, May 21, 2014 at 7:42 PM, Ted Yu <
> > yuzhih...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I 'git clone'd master branch.
> > > > > > > > > > >
> > > > > > > > > > > Ran mvn package.
> > > > > > > > > > > Ran some tests.
> > > > > > > > > > > Checked 'git log'
> > > > > > > > > > >
> > > > > > > > > > > Looks Okay.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Wed, May 21, 2014 at 7:23 PM, Stack <
> st...@duboce.net
> > >
> > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Migration looks done:
> > > > > > > > > > https://git-wip-us.apache.org/repos/asf?p=hbase.git
> > > > > >

Re: [DISCUSSION] Avoiding merge commits

[Nicolas Liochon]

+1


On Fri, May 23, 2014 at 7:46 PM, Jesse Yates wrote:

> +1 that's how I do all my git development and what I advocate at salesforce
> as well.
> On May 23, 2014 10:39 AM, "Andrew Purtell"  wrote:
>
> > I recommend we do not push merge commits upstream. I suppose it is easy
> > enough to filter them out when looking at history but there is no need to
> > be merging upstream branches into your local tracking branch when you can
> > rebase instead. In this way we can avoid polluting the history in the
> > master repository with unnecessary merge commit entries. (And maybe some
> > devs will be merging upstream into tracking branches or merging commits
> > from local feature branches several times per day, and these will all
> > accumulate...)
> >
> > When updating your local tracking branch from upstream, use git fetch
> > upstream && git rebase upstream/branch instead of 'git merge'.
> >
> > When developing features on a local branch it's possible to do a squash
> > commit from the feature branch to the tracking branch using 'git rebase'
> > instead of 'git merge', then a push of the single squashed commit from
> the
> > tracking branch to the upstream branch.
> >
> > If these workflow choices are acceptable by consensus we can update the
> > 'how to commit' document with an illustration of the workflow with
> example
> > commands.
> >
> >
> > --
> > Best regards,
> >
> >- Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>

Re: [VOTE] Merge branch HBASE-10070 to trunk

[Nicolas Liochon]

+1 for the integration, whatever the solution in git.


On Mon, Jun 9, 2014 at 7:46 PM, Enis Söztutar  wrote:

> Thanks Jon.
>
> I think we'll try the rebase approach first. I'll start the effort today
> and see how far along I can get with that. Rebasing each patch might be a
> bit more work actually. If this turns out painful, we'll revert to the
> merge approach similar to what you describe.
>
> Enis
>
>
> On Sat, Jun 7, 2014 at 7:41 AM, Jonathan Hsieh  wrote:
>
> > On Fri, Jun 6, 2014 at 5:05 PM, Enis Söztutar 
> wrote:
> >
> > > My preference would be to do the rebase-then-merge style (last style in
> > > your comment). For each patch, I am hoping for all the changes between
> > > committed version and rebased version to be mechanical. I like having a
> > > linear history with explicit commit-to-patch mapping.
> > >
> > > If the changes are of a mechanical nature and that each patch compiles
> > and
> > passes unit tests along the way, then rebase-then-merge is perfectly fine
> > by me.
> >
> > Even though snapshots were fairly orthogonal to the rest of the codebase,
> > when we did merges we had problems maintaining the compiles and passes
> with
> > every commit invariants when we tried rebase-then-merge approach.  After
> > each rebase we would end up doing a bunch of work and still end up
> failing
> > unit tests.   In that case we (jesse, matteo, myself) went to the merge
> > approach.   We actually merged into the snapshot branch a few times
> fixing
> > things due to changes in trunk that broke parts of the snapshots branch.
> > Here's a more accurate picture of what the commit history looked like in
> > git (we dev'ed in git and in end had to recommit this all in svn).
> >
> > m  (essentially empty merge patch).
> > |\
> > t |  (trunk patch with no impact)
> > | s6* (patch to fix newly introduced problem)
> > |/|
> > t |
> > t*|   (trunk patch that broke snapshots again)
> > | s5* (branch bug fixes due to merge)
> > | s4* (mechanical fixups due to the merge)
> > |/|
> > t |
> > t |
> > t |
> > | s3
> > | s2
> > | s1
> > |/
> > t
> > t
> >
> > This approach was captured in github and had the added benefit of
> > preserving exact history, and having known good points preserving the
> > compile/unit test invariants on trunk.  (unfortunately, some of this
> > history was lost when we ported the git history over to svn, but that
> isn't
> > a problem any more).If you want to see the whole thing, look at my
> git
> > repo:
> >
> > git remote add jmhsieh g...@github.com:jmhsieh/hbase.git
> > git log --oneline --graph --color jmhsieh/hbase-7290
> >
> >
> > Can we do history-preserving merge with the first style? I can do the
> > > rebases and upload interdiff if that is big so that we can compare on a
> > > per-patch basis.
> > >
> > >
> > It is possible.
> >
> >
> >
> >
> >
> > > Enis
> > >
> > >
> > > On Fri, Jun 6, 2014 at 4:36 PM, Jonathan Hsieh 
> wrote:
> > >
> > > > When we merged snapshots branch in we did this:
> > > >
> > > > t = trunk commit
> > > > s = snapshot branch commit
> > > > m = merge point.
> > > >
> > > > During work:
> > > > t
> > > > t
> > > > t
> > > > |  s3
> > > > |  s2
> > > > |  s1
> > > > |/
> > > > t
> > > > t
> > > >
> > > > During after merge:
> > > > m  (essentially empty merge patch).
> > > > t \
> > > > t |
> > > > t |
> > > > | s4* (fixups due to the merge)
> > > > | s3
> > > > | s2
> > > > | s1
> > > > |/
> > > > t
> > > > t
> > > >
> > > > Does your proposal mean you are going to do this instead?
> > > >
> > > > s3* (modified due to rebase)
> > > > s2* (modified due to rebase)
> > > > s1
> > > > t
> > > > t
> > > > t
> > > > | s (now dead hbase-10070 branch)
> > > > | s
> > > > | s
> > > > | s
> > > > |/
> > > > t
> > > > t
> > > >
> > > > If it is the then we should probably have a review of the modified
> > > patches.
> > > >  If it the same as the snapshot merge, then we just need a review of
> > the
> > > > merge point delta (as well as a full review).
> > > >
> > > > Personally I prefer the merge for these large feature branches -- it
> > > > guarantees that each commit is compilable, and reflects what you guys
> > > have
> > > > been testing for a while.  If you go with the last approach you might
> > > have
> > > > stuff broken, and in the mainline commit path.
> > > >
> > > > Jon.
> > > >
> > > >
> > > >
> > > > On Tue, Jun 3, 2014 at 11:16 AM, Andrew Purtell  >
> > > > wrote:
> > > >
> > > > >  ​
> > > > > On Mon, Jun 2, 2014 at 2:24 PM, Enis Söztutar 
> > wrote:
> > > > >
> > > > > > This VOTE is for merging back the remaining changes in branch to
> > > trunk.
> > > > > If
> > > > > > passes, we will rebase the branch on top of current trunk, in
> which
> > > we
> > > > > will
> > > > > > keep the commit-per-issue log history. After that we will do a
> git
> > > > merge
> > > > > > for the branch keeping the history clean and not squashing the
> > > > commits. I
> > > > > > expect rebasing to be straightforward, however with some manual
> > > > conf

Re: Please fix your git name/address on your commits.

[Nicolas Liochon]

As Matteo.


On Tue, Jun 10, 2014 at 7:18 PM, Matteo Bertozzi 
wrote:

> I think that @unknown is a result from the svn conversion (at least looking
> at my commits)
>
> Matteo
>
>
>
> On Tue, Jun 10, 2014 at 6:09 PM, Jonathan Hsieh  wrote:
>
> > Now that we've a few weeks with the git repo I've noticed that some of
> use
> > are showing up with funny names like name@unknown or @*MacBook*.
> >  Can
> > you all fix your emails before you next commits (my assumption is either
> > apache.org or company email) ?
> >
> > To set just in that project, see the first answer here:
> >
> >
> http://stackoverflow.com/questions/6116548/how-to-tell-git-to-use-the-correct-identity-name-and-email-for-a-given-project
> >
> > Here's a list of authors names from the past 500 commits.
> >
> >  Andrew Kyle Purtell 
> >  Andrew Purtell 
> >  anoopsamjohn 
> >  anoopsjohn 
> >  Devaraj Das 
> >  eclark 
> >  Enis Soztutar 
> >  fenghh 
> >  Gary Helmling 
> >  Jean-Daniel Cryans 
> >  Jeffrey Zhong 
> >  jeffreyz 
> >  Jesse Yates 
> >  Jimmy Xiang 
> >  Jonathan Hsieh 
> >  Jonathan M Hsieh 
> >  jxiang 
> >  larsh 
> >  Lars Hofhansl 
> >  liangxie 
> >  Matteo Bertozzi 
> >  mbertozzi 
> >  Michael Stack 
> >  Michael Stack 
> >  ndimiduk 
> >  Nick Dimiduk 
> >  Nicolas Liochon 
> >  nkeywal 
> >  rajeshbabu 
> >  Ramkrishna 
> >  ramkrishna 
> >  sershe 
> >  Ted Yu 
> >  tedyu 
> >  Zhihong Yu 
> >  zjusch 
> >
> >
> > --
> > // Jonathan Hsieh (shay)
> > // HBase Tech Lead, Software Engineer, Cloudera
> > // j...@cloudera.com // @jmhsieh
> >
>

jdk 1.7 & trunk

[Nicolas Liochon]

Hi all,

HBASE-11297 just broke the build because it uses a ConcurrentLinkedDeque.

Should we be 1.7 only for trunk / 1.0?
This would mean using the 1.7 features.

What about .98?

We would need to update our precommit env, it still builds with 1.6 today...

Shall we start a vote?

Nicolas

Re: jdk 1.7 & trunk

[Nicolas Liochon]

If hadoop does not support the 1.6 we can't support it either.
But we can drop the 1.6 ealier than them...


On Wed, Jun 25, 2014 at 6:54 PM, Ted Yu  wrote:

> Here is the thread in hadoop w.r.t. JDK version:
>
> http://search-hadoop.com/m/LgpTk2GHceb2/moving+to+jdk+major&subj=Re+Moving+to+JDK7+JDK8+and+new+major+releases
>
> Sounds like support for JDK6 may be dropped in hadoop-2.6 or hadoop-2.7.
>
> Cheers
>
>
> On Wed, Jun 25, 2014 at 9:45 AM, Nick Dimiduk  wrote:
>
> > I think it's time to move master/1.0 to 1.7, though, where is Hadoop on
> > this? We should support 1.6 if we plan to continue to support hadoop
> > versions that support 1.6.
> >
> >
> > On Wed, Jun 25, 2014 at 9:15 AM, Nicolas Liochon 
> > wrote:
> >
> > > Hi all,
> > >
> > > HBASE-11297 just broke the build because it uses a
> ConcurrentLinkedDeque.
> > >
> > > Should we be 1.7 only for trunk / 1.0?
> > > This would mean using the 1.7 features.
> > >
> > > What about .98?
> > >
> > > We would need to update our precommit env, it still builds with 1.6
> > > today...
> > >
> > > Shall we start a vote?
> > >
> > > Nicolas
> > >
> >
>

Re: jdk 1.7 & trunk

[Nicolas Liochon]

Ok Enis I like the definition you put in the thread.

Dropping support explicitly means that we stop building against
JDK6 in jenkins, and won't try to fix issues if they are jdk6 only. Release
testing and unit tests are done on jdk7.

And it implies; we may start to use 1.7 specific features.

With this definition:
1) Do we drop support for JDK6 in .98
2) Do we drop support for JDK6 in 1.0

Does anyone think we need to discuss this or can we start a vote on this?
If there is no objection I will start the vote thread in an hour or two.

Cheers,

N.



On Wed, Jun 25, 2014 at 8:14 PM, Enis Söztutar  wrote:

> See the thread:
>
>
> https://mail-archives.apache.org/mod_mbox/hbase-dev/201404.mbox/%3CCAMUu0w_KgwTLWkTc3W85H4yYuDSaEn=Ncf7ziRhhMZ=hfod...@mail.gmail.com%3E
>
> There was no conclusion in that thread. But it should be ok to start a VOTE
> I guess.
>
> Enis
>
>
> On Wed, Jun 25, 2014 at 10:29 AM, Andrew Purtell 
> wrote:
>
> > Er, I mean no user should be running on a runtime less than 7, they are
> all
> > EOL...
> >
> >
> > On Wed, Jun 25, 2014 at 10:28 AM, Andrew Purtell 
> > wrote:
> >
> > >
> > > On Wed, Jun 25, 2014 at 9:15 AM, Nicolas Liochon 
> > > wrote:
> > >
> > >> Should we be 1.7 only for trunk / 1.0?
> > >> This would mean using the 1.7 features.
> > >>
> > >
> > > I think this is prudent. Hadoop common is having a similar discussion
> and
> > > I think converging on consensus that they would be ok with their trunk
> > > including features only available in 7.
> > >
> > >
> > >> What about .98?
> > >>
> > >
> > > ​I don't think this is an option, because although no user should be
> > > running with a 7 runtime (and in fact performance conscious users
> should
> > be
> > > looking hard at 8), vendors will still have to support customers on 6.
> ​
> > >
> > > --
> > > Best regards,
> > >
> > >- Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> >- Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>

Re: jdk 1.7 & trunk

[Nicolas Liochon]

> it would mean that you'd need to upgrade Java in order to get HBase bug
fixes :)
Yes if you're using hbase 0.98 with the JDK 1.6 in production.

If that's the case yes we should not drop the support.





On Wed, Jun 25, 2014 at 8:54 PM, Jean-Daniel Cryans 
wrote:

> On Wed, Jun 25, 2014 at 11:48 AM, Nicolas Liochon 
> wrote:
>
> > Ok Enis I like the definition you put in the thread.
> >
> > Dropping support explicitly means that we stop building against
> > JDK6 in jenkins, and won't try to fix issues if they are jdk6 only.
> Release
> > testing and unit tests are done on jdk7.
> >
> > And it implies; we may start to use 1.7 specific features.
> >
> > With this definition:
> > 1) Do we drop support for JDK6 in .98
> >
>
> That's an easy -1, it would mean that you'd need to upgrade Java in order
> to get HBase bug fixes :)
>
>
> > 2) Do we drop support for JDK6 in 1.0
> >
> > Does anyone think we need to discuss this or can we start a vote on this?
> > If there is no objection I will start the vote thread in an hour or two.
> >
> > Cheers,
> >
> > N.
> >
> >
> >
> > On Wed, Jun 25, 2014 at 8:14 PM, Enis Söztutar 
> wrote:
> >
> > > See the thread:
> > >
> > >
> > >
> >
> https://mail-archives.apache.org/mod_mbox/hbase-dev/201404.mbox/%3CCAMUu0w_KgwTLWkTc3W85H4yYuDSaEn=Ncf7ziRhhMZ=hfod...@mail.gmail.com%3E
> > >
> > > There was no conclusion in that thread. But it should be ok to start a
> > VOTE
> > > I guess.
> > >
> > > Enis
> > >
> > >
> > > On Wed, Jun 25, 2014 at 10:29 AM, Andrew Purtell 
> > > wrote:
> > >
> > > > Er, I mean no user should be running on a runtime less than 7, they
> are
> > > all
> > > > EOL...
> > > >
> > > >
> > > > On Wed, Jun 25, 2014 at 10:28 AM, Andrew Purtell <
> apurt...@apache.org>
> > > > wrote:
> > > >
> > > > >
> > > > > On Wed, Jun 25, 2014 at 9:15 AM, Nicolas Liochon <
> nkey...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Should we be 1.7 only for trunk / 1.0?
> > > > >> This would mean using the 1.7 features.
> > > > >>
> > > > >
> > > > > I think this is prudent. Hadoop common is having a similar
> discussion
> > > and
> > > > > I think converging on consensus that they would be ok with their
> > trunk
> > > > > including features only available in 7.
> > > > >
> > > > >
> > > > >> What about .98?
> > > > >>
> > > > >
> > > > > ​I don't think this is an option, because although no user should
> be
> > > > > running with a 7 runtime (and in fact performance conscious users
> > > should
> > > > be
> > > > > looking hard at 8), vendors will still have to support customers on
> > 6.
> > > ​
> > > > >
> > > > > --
> > > > > Best regards,
> > > > >
> > > > >- Andy
> > > > >
> > > > > Problems worthy of attack prove their worth by hitting back. - Piet
> > > Hein
> > > > > (via Tom White)
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > >
> > > >- Andy
> > > >
> > > > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein
> > > > (via Tom White)
> > > >
> > >
> >
>

Re: jdk 1.7 & trunk

[Nicolas Liochon]

Yeah. It would also be so much simpler when it comes to backporting from
master to 0.98, and 0.98 could very likely live a long life, a la 0.94. So
the sooner we're clear on this the better.
 And anyway, Hadoop will likely drop the JDK6 in a 2.x release, so we will
be stuck with a 2.5 or so if we don't stop the JDK6 support.

Lastly, HBase 0.94 is still alive and kicking for the 1.6 lovers.

> We can vote.  Could also just decide.

Let's try to decide here. If I understand correctly, for 1.0 we're done: we
don't support JDK6
For 0.98, I chatted offline with Enis & Stack, dropping the JDK6 support is
not a showstopper for them.
Andrew would be ok as well.

So is it at least acceptable for all of us to drop the JDK6 support in 0.98
and 1.0?

Thanks,

Nicolas





On Wed, Jun 25, 2014 at 9:54 PM, Konstantin Boudnik  wrote:

> Back in time of JDK6 GA - when I was still working in Sun's JDK team - we
> had
> companies sitting on 1.4 and paying _a lot_ of money for Sun support of it.
> So...
>
> That said, I think moving to JDK7 is pretty much has happened already for
> HBase, because e.g. 0.98.2 can not be build with JDK6 because we see
>   https://issues.apache.org/jira/browse/HBASE-8479
> in Bigtop CI.
>
> Cos
>
> On Wed, Jun 25, 2014 at 10:29AM, Andrew Purtell wrote:
> > Er, I mean no user should be running on a runtime less than 7, they are
> all
> > EOL...
> >
> >
> > On Wed, Jun 25, 2014 at 10:28 AM, Andrew Purtell 
> > wrote:
> >
> > >
> > > On Wed, Jun 25, 2014 at 9:15 AM, Nicolas Liochon 
> > > wrote:
> > >
> > >> Should we be 1.7 only for trunk / 1.0?
> > >> This would mean using the 1.7 features.
> > >>
> > >
> > > I think this is prudent. Hadoop common is having a similar discussion
> and
> > > I think converging on consensus that they would be ok with their trunk
> > > including features only available in 7.
> > >
> > >
> > >> What about .98?
> > >>
> > >
> > > ​I don't think this is an option, because although no user should be
> > > running with a 7 runtime (and in fact performance conscious users
> should be
> > > looking hard at 8), vendors will still have to support customers on 6.
> ​
> > >
> > > --
> > > Best regards,
> > >
> > >- Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> >- Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
>

Re: jdk 1.7 & trunk

[Nicolas Liochon]

Anyone has anything to add?

If no we're done with the decision I think:
- master / 0.99 / 1.0 becomes JDK7 only
- 0.98 continue to support the JDK6 and JDK7.


Cheers,

Nicolas






On Thu, Jun 26, 2014 at 5:51 PM, Ted Yu  wrote:

> bq. -1 on dropping JDK6 support for anything earlier
>
> I am with J-D and Lars
>
> Cheers
>
>
> On Thu, Jun 26, 2014 at 8:06 AM, lars hofhansl  wrote:
>
> > +1 on moving to JDK7 only for 1.0+
> > -1 on dropping JDK6 support for anything earlier (i.e. 0.98, 0.96, 0.94).
> >
> > If there are issues with JDK6 in 0.98 already that would be a major bug
> > that we need to fix, IMHO.
> >
> >
> > -- Lars
> >
> >
> >
> > 
> >  From: Enis Söztutar 
> > To: "dev@hbase.apache.org" 
> > Sent: Wednesday, June 25, 2014 12:53 PM
> > Subject: Re: jdk 1.7 & trunk
> >
> >
> > My vote will be
> >
> > 1) Not drop support in JDK6 in 0.98.x series as per above.
> >
> > 2) Drop support to JDK6 in 1.x series.
> >
> > Also see recent jira: https://issues.apache.org/jira/browse/HBASE-11102
> >
> > Enis
> >
> >
> >
> >
> >
> > On Wed, Jun 25, 2014 at 12:13 PM, Stack  wrote:
> >
> > > On Wed, Jun 25, 2014 at 11:48 AM, Nicolas Liochon 
> > > wrote:
> > >
> > > > Ok Enis I like the definition you put in the thread.
> > > >
> > > > Dropping support explicitly means that we stop building against
> > > > JDK6 in jenkins, and won't try to fix issues if they are jdk6 only.
> > > Release
> > > > testing and unit tests are done on jdk7.
> > > >
> > > >
> > > This is good by me.
> > >
> > >
> > >
> > > > And it implies; we may start to use 1.7 specific features.
> > > >
> > > >
> > > Are there explicitly 1.7 features we want to make use of?  Looking at
> > this
> > > list, we might be able to do without:
> > >
> > >
> >
> http://marxsoftware.blogspot.com/2011/03/jdk-7-new-interfaces-classes-enums-and.html
> > >  Unless distinct advantage to be had, suggest we avoid breaking our
> being
> > > able to run on 1.6. 1.0 hbase will be out in a world that is hadoop
> > > 2.3.x-2.5.x  These do not preclude running on 1.6.
> > >
> > >
> > >
> > > > With this definition:
> > > > 1) Do we drop support for JDK6 in .98
> > > >
> > >
> > > No.  Not over a point release I'd say.
> > >
> > >
> > >
> > > > 2) Do we drop support for JDK6 in 1.0
> > > >
> > > >
> > > Thinking on it, I'd say yes.  1.6 is EOL.  1.7 makes you faster.
>  Caveat
> > > suggestion above that we try and avoid gratuitously breaking 1.6.
> > >
> > >
> > > > Does anyone think we need to discuss this or can we start a vote on
> > this?
> > > > If there is no objection I will start the vote thread in an hour or
> > two.
> > > >
> > >
> > > We can vote.  Could also just decide.
> > >
> > > St.Ack
> > >
> >
>

Re: Proposal: Matteo for 2.0.0 RM

[Nicolas Liochon]

+1

On Fri, Aug 28, 2015 at 12:59 AM,  wrote:

> +1
>   From: Stack 
>  To: HBase Dev List 
>  Sent: Thursday, August 27, 2015 10:22 AM
>  Subject: Proposal: Matteo for 2.0.0 RM
>
> Last night at the HBase dev workshop, during discussion of 2.0.0 (what will
> be in it, when will it come out), it was noted that there is as yet no RM
> for hbase-2.0.0.
>
> A bunch of us suggested Matteo and he was up for it.
>
> Folks good with that? Anyone else interested in being RM?
>
> +1 on Matteo from me,
> St.Ack
>
>
>
>

Re: Changing resolution in JIRA

[Nicolas Liochon]

fwiw, I gave it a try and I can't change the resolution myself, even if I
have the admin bit on the jira.
But may be a jira guru can do something about this...

On Mon, Nov 9, 2015 at 10:17 AM, Lars Francke 
wrote:

> Hi,
>
> LarsG and I have been looking at a couple of JIRA issues (861 to be
> precise) that are listed as Resolved & Fixed but without a Fix version.
> Some of them should have been closed as Invalid instead of Fixed and
> various other problems.
>
> We weren't able to change the Resolution though. We worked around it by
> reopening and closing again. Is there any other way to do this or a set of
> permissions you could give me and/or Lars George?
>
> Thanks!
>
> Cheers,
> Lars
>

Re: Please welcome our newest committer, Jeffrey Zhong

[Nicolas Liochon]

Congrats. It's well deserved :-).


On Wed, Jun 5, 2013 at 11:03 PM, Sergey Shelukhin wrote:

> Congrats! :)
>
> On Tue, Jun 4, 2013 at 6:00 PM, lars hofhansl  wrote:
>
> > Welcome Jeffrey.
> >
> >
> >
> > 
> >  From: Stack 
> > To: HBase Dev List 
> > Sent: Tuesday, June 4, 2013 3:47 AM
> > Subject: Please welcome our newest committer, Jeffrey Zhong
> >
> >
> > Please welcome our newest committer, the mighty Jeffrey Zhong.  He has
> been
> > doing great stuff of late especially speeding up distributed log split
> and
> > we're glad to have him on board.
> >
> > Welcome and keep up the good work Jeffrey,
> > St.Ack
> >
>

Re: VOTE: hbase-0.95.1 release candidate 1 is available for download

[Nicolas Liochon]

Hi,

I've tried to test the hadoop2 maven release with ycsb.

If I build hbase locally, with
 mvn clean install -DskipTests -Dhadoop.profile=2.0
then I can compile ycsb, providing 0.95.1 as the hbase version.
=> Compilation & execution works.

If I provide 0.95.0-hadoop2-SNAPSHOT (the version found in
https://repository.apache.org/content/groups/snapshots/org/apache/hbase,
but there is a bunch of files with different dates there)

Then it builds, but I have a failure on execution.
Caused by: java.lang.NoSuchMethodError:
org.apache.hadoop.net.NetUtils.getInputStream(Ljava/net/Socket;)Ljava/io/InputStream;
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:814)

When I dig with mvn dependency:tree, I see hadoop2 dependencies only, as
expected.

=> Is that the version to use? If not, which one should I use, from which
repo?


As well, by default, we don't configure log4j, so we don't have anything on
screen: log4j does not output to screen by default for security reasons (if
the data is confidential). We should probably change this.

Cheers,

Nicolas







On Fri, Jun 7, 2013 at 12:37 AM, Stack  wrote:

> Here is our second 0.95.1 release candidate.  Should we put this out as
> 0.95.1?
>
> Since 0.95.1 is "just" a "Development" Series release [1], lets keep the
> vote period short.  Please vote by this Tuesday, June 11th.
>
> The release artifacts may be downloaded from:
>
>   http://people.apache.org/~stack/hbase-0.95.1RC1/<
> http://people.apache.org/~stack/hbase-0.95.1RC0/>
>
> Almost 200 issues have been closed since we put out 0.95.0 [2].
>
> All feedback is welcome.  We'd be interested in getting comments on
> everything from the packaging, layout, through documentation, UI, and of
> course,
> any bugs found.  I deployed SNAPSHOTs to maven and would particularly like
> to hear from downstream projects on whether our maven packaging is
> palatable.
>
> Thanks,
> St.Ack
>
> 1. http://hbase.apache.org/book.html#hbase.versioning
> 2.
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20fixVersion%20%3D%20%220.95.1%22%20AND%20status%20%3D%20Resolved%20ORDER%20BY%20priority%20DESC
>

Re: VOTE: hbase-0.95.1 release candidate 1 is available for download

[Nicolas Liochon]

Thanks Elliott,

I was trying something similar that worked in the past, but couldn't make
it work on this last release. I will retry with yours later (my machine is
running a test right now), except if it already worked for you on this
0.95.1 release?

As well, it's good enough for an alpha release, but do know what we will do
for the official one?
IIRC, duplicating the pom.xml was considered as a possible solution?



On Tue, Jun 11, 2013 at 12:48 AM, Elliott Clark  wrote:

> So the maven pom's still point to 1.0 as default no matter what version of
> the client you point at.  I've gotten around this by being very explicit:
> 
>
>   
>   org.apache.hbase
>   hbase-client
>   ${hbase.version}
>
>   
>   
>   org.apache.hadoop
>   hadoop-core
>   
>   
>   org.mortbay.jetty
>   jetty
>   
>   
>   com.sun.jdmk
>   jmxtools
>   
>   
>   com.sun.jmx
>   jmxri
>   
>   
>   
>   
>   org.apache.hadoop
>   hadoop-common
>   2.0.0-alpha
>   
>   
>
>
> It's not awesome but it got me working.
>
>
>
> On Mon, Jun 10, 2013 at 3:43 PM, Nicolas Liochon 
> wrote:
>
> > Hi,
> >
> > I've tried to test the hadoop2 maven release with ycsb.
> >
> > If I build hbase locally, with
> >  mvn clean install -DskipTests -Dhadoop.profile=2.0
> > then I can compile ycsb, providing 0.95.1 as the hbase version.
> > => Compilation & execution works.
> >
> > If I provide 0.95.0-hadoop2-SNAPSHOT (the version found in
> > https://repository.apache.org/content/groups/snapshots/org/apache/hbase,
> > but there is a bunch of files with different dates there)
> >
> > Then it builds, but I have a failure on execution.
> > Caused by: java.lang.NoSuchMethodError:
> >
> >
> org.apache.hadoop.net.NetUtils.getInputStream(Ljava/net/Socket;)Ljava/io/InputStream;
> > at
> >
> >
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:814)
> >
> > When I dig with mvn dependency:tree, I see hadoop2 dependencies only, as
> > expected.
> >
> > => Is that the version to use? If not, which one should I use, from which
> > repo?
> >
> >
> > As well, by default, we don't configure log4j, so we don't have anything
> on
> > screen: log4j does not output to screen by default for security reasons
> (if
> > the data is confidential). We should probably change this.
> >
> > Cheers,
> >
> > Nicolas
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Jun 7, 2013 at 12:37 AM, Stack  wrote:
> >
> > > Here is our second 0.95.1 release candidate.  Should we put this out as
> > > 0.95.1?
> > >
> > > Since 0.95.1 is "just" a "Development" Series release [1], lets keep
> the
> > > vote period short.  Please vote by this Tuesday, June 11th.
> > >
> > > The release artifacts may be downloaded from:
> > >
> > >   http://people.apache.org/~stack/hbase-0.95.1RC1/<
> > > http://people.apache.org/~stack/hbase-0.95.1RC0/>
> > >
> > > Almost 200 issues have been closed since we put out 0.95.0 [2].
> > >
> > > All feedback is welcome.  We'd be interested in getting comments on
> > > everything from the packaging, layout, through documentation, UI, and
> of
> > > course,
> > > any bugs found.  I deployed SNAPSHOTs to maven and would particularly
> > like
> > > to hear from downstream projects on whether our maven packaging is
> > > palatable.
> > >
> > > Thanks,
> > > St.Ack
> > >
> > > 1. http://hbase.apache.org/book.html#hbase.versioning
> > > 2.
> > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20fixVersion%20%3D%20%220.95.1%22%20AND%20status%20%3D%20Resolved%20ORDER%20BY%20priority%20DESC
> > >
> >
>

Re: VOTE: hbase-0.95.1 release candidate 1 is available for download

[Nicolas Liochon]

I've not tested the public release, only my own build. I also haven't
tested hadoop1.
I've got a test in progress, let me run it for a while and coming back to
you.

My general feeling from my previous tests is that this rc is much more
stable than 0.95.0 or trunk. But let's wait a little to confirm this.

In any case, we will need to have maven working for the RC2 imho. If it has
to be a file copy, well, let's do that.


On Tue, Jun 11, 2013 at 2:00 AM, Stack  wrote:

> Thanks for testing Mr. Liochon.   Sounds like our mvn publishing needs a
> bit of work still around hadoop2 (hadoop1 works?).  You +1 on this as dev
> release?
>
> St.Ack
>
>
> On Mon, Jun 10, 2013 at 3:43 PM, Nicolas Liochon 
> wrote:
>
> > Hi,
> >
> > I've tried to test the hadoop2 maven release with ycsb.
> >
> > If I build hbase locally, with
> >  mvn clean install -DskipTests -Dhadoop.profile=2.0
> > then I can compile ycsb, providing 0.95.1 as the hbase version.
> > => Compilation & execution works.
> >
> > If I provide 0.95.0-hadoop2-SNAPSHOT (the version found in
> > https://repository.apache.org/content/groups/snapshots/org/apache/hbase,
> > but there is a bunch of files with different dates there)
> >
> > Then it builds, but I have a failure on execution.
> > Caused by: java.lang.NoSuchMethodError:
> >
> >
> org.apache.hadoop.net.NetUtils.getInputStream(Ljava/net/Socket;)Ljava/io/InputStream;
> > at
> >
> >
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:814)
> >
> > When I dig with mvn dependency:tree, I see hadoop2 dependencies only, as
> > expected.
> >
> > => Is that the version to use? If not, which one should I use, from which
> > repo?
> >
> >
> > As well, by default, we don't configure log4j, so we don't have anything
> on
> > screen: log4j does not output to screen by default for security reasons
> (if
> > the data is confidential). We should probably change this.
> >
> > Cheers,
> >
> > Nicolas
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Jun 7, 2013 at 12:37 AM, Stack  wrote:
> >
> > > Here is our second 0.95.1 release candidate.  Should we put this out as
> > > 0.95.1?
> > >
> > > Since 0.95.1 is "just" a "Development" Series release [1], lets keep
> the
> > > vote period short.  Please vote by this Tuesday, June 11th.
> > >
> > > The release artifacts may be downloaded from:
> > >
> > >   http://people.apache.org/~stack/hbase-0.95.1RC1/<
> > > http://people.apache.org/~stack/hbase-0.95.1RC0/>
> > >
> > > Almost 200 issues have been closed since we put out 0.95.0 [2].
> > >
> > > All feedback is welcome.  We'd be interested in getting comments on
> > > everything from the packaging, layout, through documentation, UI, and
> of
> > > course,
> > > any bugs found.  I deployed SNAPSHOTs to maven and would particularly
> > like
> > > to hear from downstream projects on whether our maven packaging is
> > > palatable.
> > >
> > > Thanks,
> > > St.Ack
> > >
> > > 1. http://hbase.apache.org/book.html#hbase.versioning
> > > 2.
> > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20fixVersion%20%3D%20%220.95.1%22%20AND%20status%20%3D%20Resolved%20ORDER%20BY%20priority%20DESC
> > >
> >
>

large heaps

[Nicolas Liochon]

Hi there,

During the hackathon I had some discussions around GC on large heaps.

This guy, who seems to know what he is talking about, and had a patch
accepted in hotspot jdk, said in 2011 that he's got a configuration working
reasonably well with large heaps at that time :

"I was able to keep GC pause on 32Gb Oracle Coherence storage node below
150ms on 8 core server."

(in http://java.dzone.com/articles/how-tame-java-gc-pauses)

There is a lot of stuff in his blog, some of it in Russian only, but at
least one of us will understand it.

http://blog.ragozin.info/2011/07/openjdk-patch-cutting-down-gc-pause.html
http://fr.slideshare.net/aragozin/garbage-collection-in-jvm-mailru-fin

Cheers,

Nicolas

Re: large heaps

[Nicolas Liochon]

Thanks a lot Todd, your conclusion does confirm the feeling I had around
large heaps with HBase.


On Thu, Jun 13, 2013 at 7:30 AM, Todd Lipcon  wrote:

> Hey Nicolas,
>
> I've corresponded with that guy a few times in the past -- back when i
> was attempting to hack some patches into G1 for better performance on
> HBase. The end result of that investigation was the MSLAB feature
> which made it into 0.90.x.
>
> The main thing I learned about GC is that big heaps aren't in
> themselves problematic -- they don't tend to make young gen pauses
> take longer. The only problem is if you eventually hit a
> stop-the-world CMS pause, the size of the heap linearly effects the
> length of the pause. So, the trick is avoiding stop-the-world CMS.
>
> In order to avoid that, you need to do a few things:
> - make sure you don't have any short-lived super-large objects: when
> large objects are promoted from the young generation, they need to
> find contiguous space in the old gen. If you allocate, say, a 400MB
> array, even if it's short lived, it's unlikely you'll find 400MB of
> contiguous space in the old gen without defragmenting. This will cause
> a STW pause.
>
> If you have some super-large objects allocated at startup, that's OK,
> they'll just park themselves in the old gen and not cause trouble.
>
> - make sure that most of your objects are "around the same size". This
> prevents fragmentation build-up in the old gen.
>
> - move big memory consumers off-heap if possible
>
> We've done a pretty good job of the above so far, and with a bit more
> careful analysis I think it's possible to fully avoid old-gen STW
> pauses.
>
> -Todd
>
>
> On Wed, Jun 12, 2013 at 8:35 PM, Nicolas Liochon 
> wrote:
> > Hi there,
> >
> > During the hackathon I had some discussions around GC on large heaps.
> >
> > This guy, who seems to know what he is talking about, and had a patch
> > accepted in hotspot jdk, said in 2011 that he's got a configuration
> working
> > reasonably well with large heaps at that time :
> >
> > "I was able to keep GC pause on 32Gb Oracle Coherence storage node below
> > 150ms on 8 core server."
> >
> > (in http://java.dzone.com/articles/how-tame-java-gc-pauses)
> >
> > There is a lot of stuff in his blog, some of it in Russian only, but at
> > least one of us will understand it.
> >
> >
> http://blog.ragozin.info/2011/07/openjdk-patch-cutting-down-gc-pause.html
> > http://fr.slideshare.net/aragozin/garbage-collection-in-jvm-mailru-fin
> >
> > Cheers,
> >
> > Nicolas
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Re: Extend the Hadoop QA bot to handle dependencies

[Nicolas Liochon]

Yes, it's a good idea.
It seems feasible. The code executed by HadoopQA is in the hbase repo in
dev-support/test-patch.sh, so we control it. It's easy to add  tests there.
One thing to take care about however is how long it takes to do that. maven
plugins are surprising sometimes.
As well, if the plugin outputs cleanly the undeclared dependencies, we
could just fix them all at the beginning instead of doing a diff. We did
this for the javadoc warnings, and it worked.

Cheers,

Nicolas


On Thu, Jul 11, 2013 at 11:27 AM, Lars Francke wrote:

> Hi,
>
> I don't know much (basically nothing) about how the Hadoop QA bot is
> implemented but I have a suggestion for how it could be improved and
> would like some opinions and maybe pointers.
>
> Maven and dependency handling is a topic not very many people find
> interesting. Understandably. But that means that often people
> introduce dependencies without declaring them in the pom file (see
> HBASE-8917[1] for one example but we had this problem before, years
> ago). That often leads to weird errors at some point when an upstream
> dependency changes.
>
> There's a plugin[2] that can analyze dependencies and their usage and
> report unused dependencies and used but undeclared ones (it's not
> perfect but good enough). I think it'd be great if the Hadoop QA bot
> could analyze a before and after snapshot of the output of this plugin
> and do a -1 if anyone introduces a new dependency without declaring
> it.
>
> Does that sound like a good idea?
>
> Cheers,
> Lars
>
> [1] 
> [2] <
> http://maven.apache.org/plugins/maven-dependency-plugin/analyze-mojo.html>
>

Re: Connection reference counting

[Nicolas Liochon]

I had a look, I think it's a bug.
If the connection holds resources that should be freed, well they won't be
freed.
It's used in the minicluster shutdown. So we could imagine scenarios which
that create some test flakyness. I doubt it happens in real life, but imho
it's better to fix it (it there are no different opinions, I will do it if
you don't ;-).

Cheers,

Nicolas


On Wed, Jul 17, 2013 at 10:03 AM, Lars George  wrote:

> Hi,
>
> I am working on an issue around Thrift 2 (HBASE-7035), and I am testing
> how this all works a bit. See
> https://github.com/larsgeorge/hbase-scanner-test for the code, which runs
> against a local HBase instance. Now, there is an issue that connections do
> not get closed as I would have expected. I close the connection in a loop
> over a scanner that gets single rows. Although I close it, it keeps on
> going. Scanners may or may not be reference counted - any code calling
> HCM.getConnection(conf) increases the count.
>
> So I tried a HCM.deleteAllConnections, which does this:
>
>   public static void deleteAllConnections() {
> synchronized (HBASE_INSTANCES) {
>   Set connectionKeys = new HashSet();
>   connectionKeys.addAll(HBASE_INSTANCES.keySet());
>   for (HConnectionKey connectionKey : connectionKeys) {
> deleteConnection(connectionKey, false);
>   }
>   HBASE_INSTANCES.clear();
> }
>   }
>
> It iterates over all connections and "deletes" them. Then is clears the
> entire reference list! The issue is that deleteConnection() is using the
> refcounts and is *not* closing the connection when it is still referenced.
> The final clear() call simply drops them from the list of managed
> connections. This means that we now might have dangling open connections,
> and unless you hold a reference there is no way that you can talk to it
> again, not even using deleteStaleConnection() for example.
>
> Is that wanted? Just curious, since obviously this is not a biggie in real
> life.
>
> Lars
>
>
>
>
>
>

Re: Connection reference counting

[Nicolas Liochon]

;-)

For backward compatibility it would be better to keep the existing
behaviour and add a method, but I by default I would be lazy here and just
fix a behaviour that seems broken.


On Wed, Jul 17, 2013 at 7:06 PM, Lars George  wrote:

> Hi Nicholas,
>
> Knock yourself out :)
>
> I would either call deleteConnection() with the force flag set to true
> (like deleteStaleConnection does) or check if the connection is closed -
> using isClosed() - and only if it is remove it from the internal list.
>
> Best would be to expose the force flag to the users with an overloaded
> deleteAllConnections() plus the above part of not removing not closed
> connections.
>
> Lars
>
>
>
> On Jul 17, 2013, at 6:31 PM, Nicolas Liochon  wrote:
>
> > I had a look, I think it's a bug.
> > If the connection holds resources that should be freed, well they won't
> be
> > freed.
> > It's used in the minicluster shutdown. So we could imagine scenarios
> which
> > that create some test flakyness. I doubt it happens in real life, but
> imho
> > it's better to fix it (it there are no different opinions, I will do it
> if
> > you don't ;-).
> >
> > Cheers,
> >
> > Nicolas
> >
> >
> > On Wed, Jul 17, 2013 at 10:03 AM, Lars George 
> wrote:
> >
> >> Hi,
> >>
> >> I am working on an issue around Thrift 2 (HBASE-7035), and I am testing
> >> how this all works a bit. See
> >> https://github.com/larsgeorge/hbase-scanner-test for the code, which
> runs
> >> against a local HBase instance. Now, there is an issue that connections
> do
> >> not get closed as I would have expected. I close the connection in a
> loop
> >> over a scanner that gets single rows. Although I close it, it keeps on
> >> going. Scanners may or may not be reference counted - any code calling
> >> HCM.getConnection(conf) increases the count.
> >>
> >> So I tried a HCM.deleteAllConnections, which does this:
> >>
> >>  public static void deleteAllConnections() {
> >>synchronized (HBASE_INSTANCES) {
> >>  Set connectionKeys = new HashSet();
> >>  connectionKeys.addAll(HBASE_INSTANCES.keySet());
> >>  for (HConnectionKey connectionKey : connectionKeys) {
> >>deleteConnection(connectionKey, false);
> >>  }
> >>  HBASE_INSTANCES.clear();
> >>}
> >>  }
> >>
> >> It iterates over all connections and "deletes" them. Then is clears the
> >> entire reference list! The issue is that deleteConnection() is using the
> >> refcounts and is *not* closing the connection when it is still
> referenced.
> >> The final clear() call simply drops them from the list of managed
> >> connections. This means that we now might have dangling open
> connections,
> >> and unless you hold a reference there is no way that you can talk to it
> >> again, not even using deleteStaleConnection() for example.
> >>
> >> Is that wanted? Just curious, since obviously this is not a biggie in
> real
> >> life.
> >>
> >> Lars
> >>
> >>
> >>
> >>
> >>
> >>
>

Re: Resolved JIRAs

[Nicolas Liochon]

Hi Lars,

The committer changes the value to resolved when the patch is committed.
The release manager changes the value to closed when the version which
includes the fix is released.

Cheers,

Nicolas


On Mon, Jul 29, 2013 at 3:06 PM, Lars George  wrote:

> Hi,
>
> Sorry to ask this lame question, but what is our policy of setting a
> "Resolved" issue to "Closed"?
>
> Thanks,
> Lars
>
>

Re: ycsb and hbase 0.95.1 hadoop2

[Nicolas Liochon]

You can use the branch master here: https://github.com/nkeywal/YCSB
Built it with:
mvn clean package -DskipTests -Dhbase-96 -Dhbase.version=0.97.0-SNAPSHOT
-Dhadoop.profile=2.0 -Dhadoop-two.version=2.0.5
(obviously, change the hbase.version & hadoop-two.version as you need).

I'm awaiting the finalized hadoop2 flags in hbase pom to update this and
push it to the main ycsb repo.

Cheers,

Nicolas


On Mon, Jul 29, 2013 at 6:04 PM, Nick Dimiduk  wrote:

> Hi Paul,
>
> I think Nicolas Liochon maintains a personal set of branches for this.
> Maybe he can point you in the right direction.
>
> Thanks,
> Nick
>
> On Mon, Jul 29, 2013 at 12:48 AM, Paul Baclace  >wrote:
>
> > [This might look like a user list question, or a question for another
> > project, but people on the hbase dev list use ycsb to test hbase, and the
> > side effect of my progress on this benefits hbase more than any other
> > project... please indulge me.]
> >
> > I have been unable to build ycsb for hbase 0.95.1  hadoop2, even though I
> > added the artifacts to my local repo like so:
> >
> > mvn install:install-file -Dfile=../hbase-0.95.1-**
> > hadoop2/lib/hadoop-mapreduce-**client-core-2.0.2-alpha.jar
> > -DgroupId=org.apache.hadoop -DartifactId=hadoop-mapreduce-**client-core
> > -Dversion=2.0.2-alpha -Dpackaging=jar
> >
> > mvn install:install-file -Dfile=../hbase-0.95.1-**
> > hadoop2/lib/hbase-client-0.95.**1-hadoop2.jar -DgroupId=org.apache.hbase
> > -DartifactId=hbase-client -Dversion=0.95.1-hadoop2 -Dpackaging=jar
> >
> > I looked at the pom.xml in the jar files to determine the parameters,
> > modified ycsb pom.xml and hbase/pom.xml; ycsb  "mvn clean package" has
> > errors, not finding basic stuff:
> >
> > [ERROR] /Volumes/pebproject/ThirdEye/**LSI/ws/YCSB/hbase/src/main/**
> > java/com/yahoo/ycsb/db/**HBaseClient.java:[34,29] package
> > org.apache.hadoop.conf does not exist
> > [ERROR] /Volumes/pebproject/ThirdEye/**LSI/ws/YCSB/hbase/src/main/**
> > java/com/yahoo/ycsb/db/**HBaseClient.java:[35,30] cannot find symbol
> > symbol  : class KeyValue
> > location: package org.apache.hadoop.hbase
> > [ERROR] /Volumes/pebproject/ThirdEye/**LSI/ws/YCSB/hbase/src/main/**
> > java/com/yahoo/ycsb/db/**HBaseClient.java:[46,35] cannot find symbol
> > symbol  : class Bytes
> > location: package org.apache.hadoop.hbase.util
> > [ERROR] /Volumes/pebproject/ThirdEye/**LSI/ws/YCSB/hbase/src/main/**
> > java/com/yahoo/ycsb/db/**HBaseClient.java:[47,30] cannot find symbol
> > symbol  : class HBaseConfiguration
> > location: package org.apache.hadoop.hbase
> > [... more not shown for brevity ]
> >
> > I suspect ycsb pom.xml [https://github.com/**brianfrankcooper/YCSB<
> https://github.com/brianfrankcooper/YCSB>]
> > is not ready for packaging changes made in hbase 0.95 OR could it be that
> > adding jars to a local maven repo, because they are not yet on a
> > centralized server, simply does not suffice because not all dependencies
> > are added?   HBaseConfiguration is in hbase-common-0.95.1-hadoop2.**jar ,
> > but how to add all the turtles all the way down?
> >
> > What's the secret to using ycsb for the latest HBase?
> >
> >
>

Re: Resolved JIRAs

[Nicolas Liochon]

It's my understanding as well. I think exceptions are tolerated (the first
RM to release being the only one to close), but I may be wrong.


On Mon, Jul 29, 2013 at 3:57 PM, Ted Yu  wrote:

> Since different branches have their own release cycles, my understanding is
> that different JIRA(s) should be opened for the respective branch(es) on
> the same fix.
> This way, RM can close the issue independently.
>
> Cheers
>
> On Mon, Jul 29, 2013 at 6:46 AM, Lars George 
> wrote:
>
> > Hi Nicholas,
> >
> > Wait, what if the issue is against multiple branches which are still
> > maintained and released? Who is closing the issue then?
> >
> > Lars
> >
> > On Jul 29, 2013, at 3:18 PM, Nicolas Liochon  wrote:
> >
> > > Hi Lars,
> > >
> > > The committer changes the value to resolved when the patch is
> committed.
> > > The release manager changes the value to closed when the version which
> > > includes the fix is released.
> > >
> > > Cheers,
> > >
> > > Nicolas
> > >
> > >
> > > On Mon, Jul 29, 2013 at 3:06 PM, Lars George 
> > wrote:
> > >
> > >> Hi,
> > >>
> > >> Sorry to ask this lame question, but what is our policy of setting a
> > >> "Resolved" issue to "Closed"?
> > >>
> > >> Thanks,
> > >> Lars
> > >>
> > >>
> >
> >
>

Re: ycsb and hbase 0.95.1 hadoop2

[Nicolas Liochon]

No, I've got nothing in my setting.xml
I build HBase locally (mvn install -Dhadoop.profile=2.0 -DskipTests), so
yes, it goes into my .m2


I've just rechecked with the last 0.97 as of now, it does compile. It
worked as well 2 weeks ago with the 0.95 branch. I'm not sure about the
95.1. I remember some nasty maven issues.

If you're using the maven repo for the 0.95.1, IIRC there is no version
available that works well with hadoop 2 with a maven-built client (it's ok
if you build hbase locally with the right hadoop version).





On Mon, Jul 29, 2013 at 8:50 PM, Paul Baclace wrote:

> Thanks for your help, Nicolas. I adjusted the params and I get a
> ycsb/db/HBaseClient.java compile error (below) without any missing package
> errors, so I'm wondering how it can work for you. I am building by adding
> artifacts from 0.95.1 release to my .m2; and you are building a snapshot
> and "installing" that to .m2 (or other repo), I assume.
>
> It appears to be missing hbase-common which I added to .m2 and the
> compiler debug shows a classpath that only contains
> hbase-server-0.95.1-hadoop2.**jar (plus ycsb and general jars). Does your
> mvn -X compile cmd line for ycsb/db/HBaseClient.java contain more hbase or
> hadoop jars? Do you have a settings.xml that makes mvn pull in more
> dependencies?
>
>
> [ERROR] 
> YCSB/hbase/src/main/java/com/**yahoo/ycsb/db/HBaseClient.**java:[34,30]
> package org.apache.hadoop.conf does not exist
> [ERROR] 
> /YCSB/hbase/src/main/java/com/**yahoo/ycsb/db/HBaseClient.**java:[35,31]
> cannot find symbol
> symbol  : class KeyValue
>
>
>
> On 20130729 9:29 , Nicolas Liochon wrote:
>
>> You can use the branch master here: 
>> https://github.com/nkeywal/**YCSB<https://github.com/nkeywal/YCSB>
>> Built it with:
>> mvn clean package -DskipTests -Dhbase-96 -Dhbase.version=0.97.0-**
>> SNAPSHOT
>> -Dhadoop.profile=2.0 -Dhadoop-two.version=2.0.5
>> (obviously, change the hbase.version & hadoop-two.version as you need).
>>
>> I'm awaiting the finalized hadoop2 flags in hbase pom to update this and
>> push it to the main ycsb repo.
>>
>> Cheers,
>>
>> Nicolas
>>
>>
>> On Mon, Jul 29, 2013 at 6:04 PM, Nick Dimiduk  wrote:
>>
>>  Hi Paul,
>>>
>>> I think Nicolas Liochon maintains a personal set of branches for this.
>>> Maybe he can point you in the right direction.
>>>
>>> Thanks,
>>> Nick
>>>
>>> On Mon, Jul 29, 2013 at 12:48 AM, Paul Baclace >>
>>>> wrote:
>>>> [This might look like a user list question, or a question for another
>>>> project, but people on the hbase dev list use ycsb to test hbase, and
>>>> the
>>>> side effect of my progress on this benefits hbase more than any other
>>>> project... please indulge me.]
>>>>
>>>> I have been unable to build ycsb for hbase 0.95.1  hadoop2, even though
>>>> I
>>>> added the artifacts to my local repo like so:
>>>>
>>>> mvn install:install-file -Dfile=../hbase-0.95.1-**
>>>> hadoop2/lib/hadoop-mapreduce-client-core-2.0.2-alpha.jar
>>>> -DgroupId=org.apache.hadoop -DartifactId=hadoop-mapreduce-**
>>>> **client-core
>>>> -Dversion=2.0.2-alpha -Dpackaging=jar
>>>>
>>>> mvn install:install-file -Dfile=../hbase-0.95.1-**
>>>> hadoop2/lib/hbase-client-0.95.1-hadoop2.jar
>>>> -DgroupId=org.apache.hbase
>>>> -DartifactId=hbase-client -Dversion=0.95.1-hadoop2 -Dpackaging=jar
>>>>
>>>> I looked at the pom.xml in the jar files to determine the parameters,
>>>> modified ycsb pom.xml and hbase/pom.xml; ycsb  "mvn clean package" has
>>>> errors, not finding basic stuff:
>>>>
>>>> [ERROR] YCSB/hbase/src/main/**
>>>>
>>>> java/com/yahoo/ycsb/db/HBaseClient.java:[34,29] package
>>>> org.apache.hadoop.conf does not exist
>>>> [ERROR] YCSB/hbase/src/main/**
>>>>
>>>> java/com/yahoo/ycsb/db/HBaseClient.java:[35,30] cannot find symbol
>>>> symbol  : class KeyValue
>>>> location: package org.apache.hadoop.hbase
>>>> [ERROR] YCSB/hbase/src/main/**
>>>>
>>>> java/com/yahoo/ycsb/db/HBaseClient.java:[46,35] cannot find symbol
>>>> symbol  : class Bytes
>>>> location: package org.apache.hadoop.hbase.util
>>>> [ERROR] YCSB/hbase/src/main/**
>>>>
>>>> java/com/yahoo/ycsb/db/HBaseClient.java:[47,30] cannot find symbol
>>>> symbol  : class HBaseConfiguration
>>>> location: package org.apache.hadoop.hbase
>>>> [... more not shown for brevity ]
>>>>
>>>> I suspect ycsb pom.xml 
>>>> [https://github.com/brianfrankcooper/YCSB<https://github.com/**brianfrankcooper/YCSB>
>>>> <
>>>>
>>> https://github.com/**brianfrankcooper/YCSB<https://github.com/brianfrankcooper/YCSB>
>>> >]
>>>
>>>> is not ready for packaging changes made in hbase 0.95 OR could it be
>>>> that
>>>> adding jars to a local maven repo, because they are not yet on a
>>>> centralized server, simply does not suffice because not all dependencies
>>>> are added?   HBaseConfiguration is in hbase-common-0.95.1-hadoop2.jar
>>>> ,
>>>> but how to add all the turtles all the way down?
>>>>
>>>> What's the secret to using ycsb for the latest HBase?
>>>>
>>>>
>>>>
>

Re: ycsb and hbase 0.95.1 hadoop2

[Nicolas Liochon]

0.95.2 is not yet finished. As Ted suggested, you can build HBase locally
from the sources (the 0.95 branch is usually good as the patches are tested
before being committed).

When 0.95.2 will be out I will redo the integration with ycsb. Hopefully it
will be simple to make it work with all hadoop versions on the standards
maven repo...


On Mon, Jul 29, 2013 at 10:14 PM, Ted Yu  wrote:

> You can perform the following command in 0.95 workspace:
>
> mvn clean install -DskipTests
>
> Cheers
>
> On Mon, Jul 29, 2013 at 1:11 PM, Paul Baclace  >wrote:
>
> > On 20130729 12:18 , Nicolas Liochon wrote:
> >
> >> No, I've got nothing in my setting.xml
> >> I build HBase locally (mvn install -Dhadoop.profile=2.0 -DskipTests), so
> >> yes, it goes into my .m2
> >>
> >>
> >> I've just rechecked with the last 0.97 as of now, it does compile. It
> >> worked as well 2 weeks ago with the 0.95 branch. I'm not sure about the
> >> 95.1. I remember some nasty maven issues.
> >>
> > ((I rather be attacked by angry ants than nasty mavens.))
> >
> >
> >> If you're using the maven repo for the 0.95.1, IIRC there is no version
> >> available that works well with hadoop 2 with a maven-built client (it's
> ok
> >> if you build hbase locally with the right hadoop version).
> >>
> > Is there an experimental or nightly 0.95.2 maven repo I can connect to? I
> > can probably use 0.95.2 if all the tests passed; it looks like
> > https://builds.apache.org/job/**HBase-0.95/373<
> https://builds.apache.org/job/HBase-0.95/373>passed.
> >
> >
> >
> >
> >
> >>
> >>
> >>
> >>
> >> On Mon, Jul 29, 2013 at 8:50 PM, Paul Baclace  >> >wrote:
> >>
> >>  Thanks for your help, Nicolas. I adjusted the params and I get a
> >>> ycsb/db/HBaseClient.java compile error (below) without any missing
> >>> package
> >>> errors, so I'm wondering how it can work for you. I am building by
> adding
> >>> artifacts from 0.95.1 release to my .m2; and you are building a
> snapshot
> >>> and "installing" that to .m2 (or other repo), I assume.
> >>>
> >>> It appears to be missing hbase-common which I added to .m2 and the
> >>> compiler debug shows a classpath that only contains
> >>> hbase-server-0.95.1-hadoop2.jar (plus ycsb and general jars). Does
> >>> your
> >>> mvn -X compile cmd line for ycsb/db/HBaseClient.java contain more hbase
> >>> or
> >>> hadoop jars? Do you have a settings.xml that makes mvn pull in more
> >>> dependencies?
> >>>
> >>>
> >>> [ERROR] YCSB/hbase/src/main/java/com/yahoo/ycsb/db/HBaseClient.
> >>> java:[34,30]
> >>> package org.apache.hadoop.conf does not exist
> >>> [ERROR]
> /YCSB/hbase/src/main/java/com/yahoo/ycsb/db/HBaseClient.
> >>> java:[35,31]
> >>> cannot find symbol
> >>> symbol  : class KeyValue
> >>>
> >>>
> >>>
> >>> On 20130729 9:29 , Nicolas Liochon wrote:
> >>>
> >>>  You can use the branch master here:
> https://github.com/nkeywal/YCSB<https://github.com/nkeywal/**YCSB>
> >>>> <https://github.com/**nkeywal/YCSB <https://github.com/nkeywal/YCSB>>
> >>>> Built it with:
> >>>> mvn clean package -DskipTests -Dhbase-96 -Dhbase.version=0.97.0-**
> >>>> SNAPSHOT
> >>>> -Dhadoop.profile=2.0 -Dhadoop-two.version=2.0.5
> >>>> (obviously, change the hbase.version & hadoop-two.version as you
> need).
> >>>>
> >>>> I'm awaiting the finalized hadoop2 flags in hbase pom to update this
> and
> >>>> push it to the main ycsb repo.
> >>>>
> >>>> Cheers,
> >>>>
> >>>> Nicolas
> >>>>
> >>>>
> >>>> On Mon, Jul 29, 2013 at 6:04 PM, Nick Dimiduk 
> >>>> wrote:
> >>>>
> >>>>   Hi Paul,
> >>>>
> >>>>> I think Nicolas Liochon maintains a personal set of branches for
> this.
> >>>>> Maybe he can point you in the right direction.
> >>>>>
> >>>>> Thanks,
> >>>>> Nick
> >>>>>
> >>>>> On Mon, Jul 29, 2013 at 12:48 AM, Paul Baclace <
> paul.bacl...@gmail.com
> >>>>>

Re: Resolved JIRAs

[Nicolas Liochon]

+1 for #3 as well
Le 30 juil. 2013 06:44, "Ted Yu"  a écrit :

> I would lean toward #3.
>
> Cheers
>
> On Mon, Jul 29, 2013 at 9:30 PM, lars hofhansl  wrote:
>
> > As long as we all agree. :)
> >
> > We have three options:
> >
> > 1. separate jiras for each version
> > 2. last release closes jira
> > 3. first release closes jira, separate jiras needed for further changes
> >
> > It should also be easy to determine which jiras need to be close and be
> > able to do that in bulk. That is easy in #1 and #3, but hard for #2.
> > #1 and #2 are easier to understand.
> > #3 can be confusing.
> > #1 is cumbersome.
> >
> > My vote would remain with #3.
> >
> > -- Lars
> >
> >
> >
> > 
> >  From: Enis Söztutar 
> > To: "dev@hbase.apache.org" ; lars hofhansl <
> > la...@apache.org>
> > Sent: Monday, July 29, 2013 7:01 PM
> > Subject: Re: Resolved JIRAs
> >
> >
> >
> > I think it makes sense to group fix versions in the same jira as long as
> > there is no significant time delay between patches getting in. First
> > release closing the issue also makes sense, since closing is basically
> > marking the issue as complete. If addendums are needed, we can do another
> > jira.
> >
> >
> >
> > On Mon, Jul 29, 2013 at 2:45 PM, lars hofhansl  wrote:
> >
> > Yeah, but that would be a lot of extra jiras to file.
> > >I think we can co-fix issues across multiple branches exactly until one
> > of them is released.
> > >
> > >We should not add new patches over long time spans to the same jira
> > anyway.
> > >
> > >
> > >
> > >-- Lars
> > >
> > >
> > >
> > >- Original Message -
> > >
> > >From: Ted Yu 
> > >To: dev@hbase.apache.org
> > >Cc: lars hofhansl 
> > >Sent: Monday, July 29, 2013 2:39 PM
> > >Subject: Re: Resolved JIRAs
> > >
> > >bq. another way to do this is not have issues that target multiple
> > >branches/releases.
> > >
> > >+1
> > >
> > >On Mon, Jul 29, 2013 at 2:34 PM, Andrew Purtell 
> > wrote:
> > >
> > >> > The argument could also be made that *any* release should lead to
> > closing
> > >> the issue
> > >>
> > >> For issues that have multiple commit/target versions, we can close it
> > after
> > >> the first release goes out but then we can't change it's state if
> > there's
> > >> an issue with another branch/release, maybe as simple as making sure
> it
> > got
> > >> committed there or (re)testing. That could be fine, I have no strong
> > >> opinion.
> > >>
> > >> Or another way to do this is not have issues that target multiple
> > >> branches/releases.
> > >>
> > >>
> > >> On Mon, Jul 29, 2013 at 2:22 PM, lars hofhansl 
> > wrote:
> > >>
> > >> > Hmm... That would would be difficult to track in bulk, though.
> > >> > It's true that I have closed all 0.94.x issues when 0.94.x is
> > released.
> > >> > That has been very helpful to identify jiras that folks mislabel
> later
> > >> > (which happens very frequently).
> > >> >
> > >> > The argument could also be made that *any* release should lead to
> > closing
> > >> > the issue (as long as it has a fix for said version, of course).At
> > that
> > >> > point the code is out in the wild and is used, any change now should
> > >> > require a new jira.
> > >> >
> > >> > -- Lars
> > >> >
> > >> >
> > >> >
> > >> > - Original Message -
> > >> > From: Andrew Purtell 
> > >> > To: dev@hbase.apache.org
> > >> > Cc:
> > >> > Sent: Monday, July 29, 2013 12:19 PM
> > >> > Subject: Re: Resolved JIRAs
> > >> >
> > >> > On Mon, Jul 29, 2013 at 11:06 AM, Lars George <
> lars.geo...@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > That is exactly my point, ie the former case. If I commit to all
> > major
> > >> > > branches within a day as is common, but the branches release at
> > various
> > >> > > times, who is going to close the issue? The release manager who
> > >> releases
> > >> > > first?
> > >> >
> > >> >
> > >> > IMHO:
> > >> >
> > >> > The commiter should set the state to 'Resolved' after the change is
> > >> applied
> > >> > to all desired target branches.
> > >> >
> > >> > The RM for the _last_ affected release should set the state to
> > 'Closed',
> > >> > essentially garbage collecting when refcount goes to 0.
> > >> >
> > >> > --
> > >> > Best regards,
> > >> >
> > >> >- Andy
> > >> >
> > >> > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein
> > >> > (via Tom White)
> > >> >
> > >> >
> > >>
> > >>
> > >> --
> > >> Best regards,
> > >>
> > >>- Andy
> > >>
> > >> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > >> (via Tom White)
> > >>
> > >
> > >
> >
>

Re: HBase master failover

[Nicolas Liochon]

Yes, it makes sense. Even a 1 minute timeout is not ideal in this case: we
know that the work to do server side is trivial, and we know it's
idempotent so we can retry. So I would to tend to use a specific setting to
use for such operations.

Could you please create a jira for this?

Thanks,

Nicolas



On Tue, Aug 6, 2013 at 9:46 AM, Julian Zhou  wrote:

> Hi Community,
> Could you help if this case makes sense for 0.94 or trunk?
> Default of "hbase.rpc.timeout" is 6 ms (1 min). User sometimes
> increase them to a bigger value such as 60 ms (10 mins) for many
> concurrent loading application from client. Some user share the same
> hbase-site.xml for both client and server. HRegionServer
> #tryRegionServerReport via rpc channel to report to live master, but
> there was a window for master failover scenario. That region server
> attemping to connect to master, which was just killed, backup master
> took the ative role immediately and put to /hbase/master, but region
> server was still waiting for the rpc timeout from connecting to the dead
> master. If "hbase.rpc.timeout" is too long, this master failover process
> will be long due to long rpc timeout from dead master.
>
> If so, could we seperate with 2 options, "hbase.rpc.timeout" is still
> for hbase client, while "hbase.rpc.internal.timeout" was for this
> regionserver/master rpc channel, which could be set shorted value
> without affect real client rpc timeout value?
>
> --
> Best Regards, Julian
>
>

Re: HBase master failover

[Nicolas Liochon]

Thanks Julian. I've added a comment in the jira, let's continue there, it
will be easier to track later.


On Tue, Aug 6, 2013 at 5:44 PM, Julian Zhou  wrote:

> Thanks Nicolas. HBASE-9139: "Independent timeout configuration for rpc
> channel between cluster nodes" has been opened to track it. So do you think
> "hbase.rpc.internal.timeout" is a suitable name for this configuration?
>
> 于 8/6/2013 4:19 PM, Nicolas Liochon 写道:
>
>  Yes, it makes sense. Even a 1 minute timeout is not ideal in this case: we
>> know that the work to do server side is trivial, and we know it's
>> idempotent so we can retry. So I would to tend to use a specific setting
>> to
>> use for such operations.
>>
>> Could you please create a jira for this?
>>
>> Thanks,
>>
>> Nicolas
>>
>>
>>
>> On Tue, Aug 6, 2013 at 9:46 AM, Julian Zhou  wrote:
>>
>>  Hi Community,
>>> Could you help if this case makes sense for 0.94 or trunk?
>>> Default of "hbase.rpc.timeout" is 6 ms (1 min). User sometimes
>>> increase them to a bigger value such as 60 ms (10 mins) for many
>>> concurrent loading application from client. Some user share the same
>>> hbase-site.xml for both client and server. HRegionServer
>>> #tryRegionServerReport via rpc channel to report to live master, but
>>> there was a window for master failover scenario. That region server
>>> attemping to connect to master, which was just killed, backup master
>>> took the ative role immediately and put to /hbase/master, but region
>>> server was still waiting for the rpc timeout from connecting to the dead
>>> master. If "hbase.rpc.timeout" is too long, this master failover process
>>> will be long due to long rpc timeout from dead master.
>>>
>>> If so, could we seperate with 2 options, "hbase.rpc.timeout" is still
>>> for hbase client, while "hbase.rpc.internal.timeout" was for this
>>> regionserver/master rpc channel, which could be set shorted value
>>> without affect real client rpc timeout value?
>>>
>>> --
>>> Best Regards, Julian
>>>
>>>
>>>
>
> --
> Best Regards, Julian
>
>

Re: [ANNOUNCE] Secondary Index in HBase - from Huawei

[Nicolas Liochon]

Well done, Rajesh!


On Tue, Aug 13, 2013 at 8:44 AM, Anoop John  wrote:

> Good to see this Rajesh. Thanks a lot to Huawei HBase team!
>
> -Anoop-
>
> On Tue, Aug 13, 2013 at 11:49 AM, rajeshbabu chintaguntla <
> rajeshbabu.chintagun...@huawei.com> wrote:
>
> > Hi,
> >
> > We have been working on implementing secondary index in HBase, and had
> > shared an overview of our design in the 2012  Hadoop Technical Conference
> > at Beijing(http://bit.ly/hbtc12-hindex). We are pleased to open source
> it
> > today.
> >
> > The project is available on github.
> >   https://github.com/Huawei-Hadoop/hindex
> >
> > It is 100% Java, compatible with Apache HBase 0.94.8, and is open sourced
> > under Apache Software License v2.
> >
> > Following features are supported currently.
> > -  multiple indexes on table,
> > -  multi column index,
> > -  index based on part of a column value,
> > -  equals and range condition scans using index, and
> > -  bulk loading data to indexed table (Indexing done with bulk
> > load)
> >
> > We now plan to raise HBase JIRA(s) to make it available in Apache
> release,
> > and can hopefully continue our work on this in the community.
> >
> > Regards
> > Rajeshbabu
> >
> >
>

Re: [VOTE] The 1st hbase-0.96.0 release candidate is available for download

[Nicolas Liochon]

It wanted to run ycsb on the release, but HBASE-9334 breaks the client code
compatibility. Was it the intend?


On Sat, Sep 7, 2013 at 10:19 AM, Stack  wrote:

> On Sat, Sep 7, 2013 at 4:42 AM, Elliott Clark  wrote:
>
> > So I posted an update in HBASE-9338[1].  For me this sinks the release
> > and I'm -1 biding until we can figure out what's causing this data
> > loss.
> >
> > Thanks Devaraj for testing. Good to know that others are also seeing a
> > failure.  If other could test too I would really appreciate it.
> >
> > 1. https://issues.apache.org/jira/browse/HBASE-9338#comment-13760594
>
>
> Agree this is a sinker.  Will roll new RC soon as we figure it.  Meantime
> any other comments on this RC?
>
> Thanks,
> St.Ack
>

Re: HBase - stable versions

[Nicolas Liochon]

That's linux terminology. 0.95 is a developper release It should not go in
production. When it's ready for production, it will be released as 0.96
0.96 should be ready soon, tests (and fixes are in progress). There is
already a release candidate available: 0.96.RC0.
There should be a new release candidate (soon as well :-))
For details about the 0.96 RC0 see this thread:
http://comments.gmane.org/gmane.comp.java.hadoop.hbase.devel/39592

Cheers,

Nicolas


On Tue, Sep 10, 2013 at 8:22 AM, Kiru Pakkirisamy  wrote:

> BTW, can somebody explain the function/purpose of 0.95.2. Do the community
> expect 0.95.2 to be used in a prod env or does it have to 0.96.0 for that ?
> Also, I have some development hiccups with it (like cannot find the jar on
> the maven repo etc, if somebody can provide pointers that would be great).
>
>
> Regards,
> - kiru
>
>
> 
>  From: Ameya Kanitkar 
> To: u...@hbase.apache.org; Kiru Pakkirisamy 
> Cc: "dev@hbase.apache.org" 
> Sent: Monday, September 9, 2013 5:02 PM
> Subject: Re: HBase - stable versions
>
>
> We (Groupon), will also stick to 0.94 for near future.
>
> Ameya
>
>
> On Mon, Sep 9, 2013 at 4:03 PM, Kiru Pakkirisamy
> wrote:
>
> > When is 0.96 release being planned ? Right now we are testing against
> > 0.95.2 as this does not seem to have the HBASE-9410 bug.
> >
> >
> > Regards,
> > - kiru
> >
> > 
> >  From: Enis Söztutar 
> > To: hbase-user 
> > Cc: "dev@hbase.apache.org" 
> > Sent: Wednesday, September 4, 2013 6:20 PM
> > Subject: Re: HBase - stable versions
> >
> >
> > As long as there is interest for 0.94, we will care for 0.94. However,
> when
> > 0.96.0 comes out, it will be marked as the next stable release, so I
> expect
> > that we would promote newcomers that branch.
> >
> > Any committer can propose any branch and release candidate any time, so
> if
> > there are road blocks for 0.94.x mainline, you might as well propose a
> new
> > branch.
> >
> > Enis
> >
> >
> > On Wed, Sep 4, 2013 at 4:29 PM, Varun Sharma 
> wrote:
> >
> > > We, at Pinterest, are also going to stay on 0.94 for a while since it
> has
> > > worked well for us and we don't have the resources to test 0.96 in the
> > EC2
> > > environment. That may change in the future but we don't know when...
> > >
> > >
> > > On Wed, Sep 4, 2013 at 1:53 PM, Andrew Purtell 
> > > wrote:
> > >
> > > > If LarsH is willing to stay on as RM for 0.94 then IMHO we should
> > proceed
> > > > as today with the exception that 0.96 is what the stable symlink
> points
> > > to.
> > > >
> > > > As long as 0.94 has someone willing to RM and users such as
> Salesforce
> > > then
> > > > there will be individuals there and in the community motivated to
> keep
> > it
> > > > in good working order with occasional point releases. We should not
> > throw
> > > > up roadblocks or adopt an arbitrary policy, as long as new features
> > > arrive
> > > > in the branch as backports, and the changes maintain our point
> release
> > > > compatibility criteria (rolling restarts possible, no API
> regressions).
> > > >
> > > >
> > > > On Tue, Sep 3, 2013 at 5:30 PM, lars hofhansl 
> > wrote:
> > > >
> > > > > With 0.96 being imminent we should start a discussion about
> > continuing
> > > > > support for 0.94.
> > > > >
> > > > > 0.92 became stale pretty soon after 0.94 was released.
> > > > > The relationship between 0.94 and 0.96 is slightly different,
> though:
> > > > >
> > > > > 1. 0.92.x could be upgraded to 0.94.x without downtime
> > > > > 2. 0.92 clients and servers are mutually compatible with 0.94
> clients
> > > and
> > > > > servers
> > > > > 3. the user facing API stayed backward compatible
> > > > >
> > > > > None of the above is true when moving from 0.94 to 0.96+.
> > > > > Upgrade from 0.94 to 0.96 will require a one-way upgrade process
> > > > including
> > > > > downtime, and client and server need to be upgraded in lockstep.
> > > > >
> > > > > I would like to have an informal poll about who's using 0.94 and is
> > > > > planning to continue to use it; and who is planning to upgrade from
> > > 0.94
> > > > to
> > > > > 0.96.
> > > > > Should we officially continue support for 0.94? How long?
> > > > >
> > > > > Thanks.
> > > > >
> > > > > -- Lars
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > >
> > > >- Andy
> > > >
> > > > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein
> > > > (via Tom White)
> > > >
> > >
> >
>

Re: 0.95 Error in Connecting

[Nicolas Liochon]

(redirected user mailing list, dev mailing list in bcc)

Various comments:
- you should not need to add the hadoop jar in your client application pom,
they will come with hbase. But this should not the cause of your issue.
- what does the server say in its logs?
- I'm suprised by this:  Client
environment:zookeeper.version=3.3.2-1031432, built on 11/05/2010 05:32 GMT
   => HBase is built with the version 3.4.5.

May be you can share the code & the logs on pastebin?

Nicolas


On Tue, Sep 10, 2013 at 3:19 AM, David Williams wrote:

>
> Hi all,
>
> I am working on a api demo that talks to hbase.  Today I upgraded to 0.95
> to get access to the hbase-client 0.95 libraries.  I unpacked the 0.95
> binaries on my system, and started hbase.  I logged into Hbase shell, and
> checked status etc.  Then I added the client libs for hadoop 1.2.1 and
> hbase 0.95 to my pom.xml and ran a unit test which checks if I can read and
> write a simple test value to a table, which I created before hand.  The
> output is a stack trace and some timeouts.  The ip addresses correspond to
> my machine on the local network.  It then repeats this on the command line.
> What should I try next?  My goal is to simply programmatically read and
> write to a local hbase on Mac OS X running in pseudo distributed mode.
>
>
> ---
>  T E S T S
> ---
> Running com.example.hbase.HConnectionTest
> 13/09/09 18:06:32 INFO annotation.ClassPathBeanDefinitionScanner: JSR-330
> 'javax.inject.Named' annotation found and supported for component scanning
> 13/09/09 18:06:32 INFO annotation.AnnotationConfigApplicationContext:
> Refreshing
> org.springframework.context.annotation.AnnotationConfigApplicationContext@4b40de18:
> startup date [Mon Sep 09 18:06:32 PDT 2013]; root of context hierarchy
> 13/09/09 18:06:32 INFO annotation.ClassPathBeanDefinitionScanner: JSR-330
> 'javax.inject.Named' annotation found and supported for component scanning
> 13/09/09 18:06:32 INFO annotation.ClassPathBeanDefinitionScanner: JSR-330
> 'javax.inject.Named' annotation found and supported for component scanning
> 13/09/09 18:06:32 INFO annotation.AnnotationConfigApplicationContext:
> Refreshing
> org.springframework.context.annotation.AnnotationConfigApplicationContext@27c549b9:
> startup date [Mon Sep 09 18:06:32 PDT 2013]; root of context hierarchy
> 13/09/09 18:06:32 INFO annotation.ClassPathBeanDefinitionScanner: JSR-330
> 'javax.inject.Named' annotation found and supported for component scanning
> 13/09/09 18:06:32 INFO annotation.AutowiredAnnotationBeanPostProcessor:
> JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
> 13/09/09 18:06:32 INFO support.DefaultListableBeanFactory:
> Pre-instantiating singletons in
> org.springframework.beans.factory.support.DefaultListableBeanFactory@594f8a87:
> defining beans
> [org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,config,org.springframework.context.annotation.ConfigurationClassPostProcessor.importAwareProcessor,properties,hTablePool,appHealthCheck,healthCheck,validate,jsonProvider,submit,hConnection,jaxRsServer,cxf,hadoopConfiguration,jaxRsApiApplication,decode];
> root of factory hierarchy
> 13/09/09 18:06:32 INFO zookeeper.ZooKeeper: Client
> environment:zookeeper.version=3.3.2-1031432, built on 11/05/2010 05:32 GMT
> 13/09/09 18:06:32 INFO zookeeper.ZooKeeper: Client environment:host.name
> =10.14.49.129
> 13/09/09 18:06:32 INFO zookeeper.ZooKeeper: Client
> environment:java.version=1.7.0_25
> 13/09/09 18:06:32 INFO zookeeper.ZooKeeper: Client
> environment:java.vendor=Oracle Corporation
> 13/09/09 18:06:32 INFO zookeeper.ZooKeeper: Client
> environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre
> 13/09/09 18:06:32 INFO zookeeper.ZooKeeper: Client
> environment:java.class.path=/Users/dwilliams/IdeaProjects/vinapi/target/test-classes:/Users/dwilliams/IdeaProjects/vinapi/target/classes:/Users/dwilliams/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.3/commons-beanutils-core-1.8.3.jar:/Users/dwilliams/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:/Users/dwilliams/.m2/repository/commons-beanutils/commons-beanutils/1.8.3/commons-beanutils-1.8.3.jar:/Users/dwilliams/.m2/repository/org/apache/cxf/cxf-rt-frontend-jaxrs/2.7.2/cxf-rt-frontend-jaxrs-2.7.2.jar:/Users/dwilliams/.m2/repository/org/apache/cxf/cxf-api/2.7.2/cxf-api-2.7.2.jar:/Users/dwilliams/.m2/repository/org/codehaus/woodstox/woodstox-core-asl/4.1.4/woodstox-core-asl-4.1.4.jar:/Users/dwilliams/.m2/repository/org/codehaus/woodstox/stax2-api/3.1.1/stax2-api-3.1.1.jar:/Users/dwilliams/.m2/repository/org/apache/ws/xm

Re: Invitation to use Google Talk

[Nicolas Liochon]

Sorry about that. I was fighting with Google Hangout.


On Fri, May 17, 2013 at 7:44 PM, Jean-Daniel Cryans wrote:

> Nicolas is such a nice guy.
>
> On Fri, May 17, 2013 at 10:42 AM, Jean-Marc Spaggiari
>  wrote:
> > ;)
> >
> > I think Nicolas want to connect with all of us on GTalk ;)
> >
> > 2013/5/17 Jonathan Hsieh 
> >
> >> Hein? (jd says this is french for "eh?")
> >>
> >>
> >> On Fri, May 17, 2013 at 10:34 AM, Google Talk  >> >wrote:
> >>
> >> >
> ---
> >> >
> >> > You've been invited by Nicolas Liochon to use Google Talk.
> >> >
> >> > If you already have a Google account, login to Gmail and accept this
> chat
> >> > invitation:
> >> >
> >> >
> >>
> http://mail.google.com/mail/b-1bf20423a5-b930ddab15-yDRzYsPA77muBtGE9ZE0AZSGGFY
> >> >
> >> > To sign up for a Google account and get started with Google Talk, you
> can
> >> > visit:
> >> >
> >> >
> >>
> http://mail.google.com/mail/a-1bf20423a5-b930ddab15-yDRzYsPA77muBtGE9ZE0AZSGGFY?pc=en-rf---a
> >> >
> >> > Learn more at:
> >> > http://www.google.com/intl/en/landing/accounts/
> >> >
> >> >
> >> > Thanks,
> >> > The Google Team
> >> >
> >>
> >>
> >>
> >> --
> >> // Jonathan Hsieh (shay)
> >> // Software Engineer, Cloudera
> >> // j...@cloudera.com
> >>
>

Re: patch waiting for HBASE-8025

[Nicolas Liochon]

Hi Dave,

I'm having a look.

Nicolas


On Thu, Mar 14, 2013 at 2:59 PM, Dave Latham  wrote:

> Is anyone available to take a look at (and hopefully commit) the small
> patch for HBASE-8025?
>
> Thanks,
> Dave
>

Marking normal priority after getting exception

[Nicolas Liochon]

Hi,

I'm doing some recovery tests (unplugging a server) I've got something new:
hundreds of line like this one in the region server logs. Does anyone know
where it comes from?

Thanks,


2013-03-26 17:59:40,693 DEBUG
org.apache.hadoop.hbase.regionserver.QosFunction: Marking normal priority
after getting
exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException:
Region is not online: 3dfeaf95dd18d06c24fc679a0ab016d2
2013-03-26 17:59:40,693 DEBUG
org.apache.hadoop.hbase.regionserver.QosFunction: Marking normal priority
after getting
exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException:
Region is not online: 3dfeaf95dd18d06c24fc679a0ab016d2
2013-03-26 17:59:40,897 DEBUG
org.apache.hadoop.hbase.regionserver.QosFunction: Marking normal priority
after getting
exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException:
Region is not online: 3dfeaf95dd18d06c24fc679a0ab016d2
2013-03-26 17:59:40,897 DEBUG
org.apache.hadoop.hbase.regionserver.QosFunction: Marking normal priority
after getting
exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException:
Region is not online: 3dfeaf95dd18d06c24fc679a0ab016d2
2013-03-26 17:59:41,100 DEBUG
org.apache.hadoop.hbase.regionserver.QosFunction: Marking normal priority
after getting
exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException:
Region is not online: 3dfeaf95dd18d06c24fc679a0ab016d2

Re: Marking normal priority after getting exception

[Nicolas Liochon]

Yes on trunk. I found this line, which seems to be written for guava.
That's the global picture behind this line that I don't have :-)


On Tue, Mar 26, 2013 at 6:15 PM, Ted Yu  wrote:

> You tested with trunk, right ?
>
> Here it is:
>
>   if (LOG.isDebugEnabled()) LOG.debug("Marking normal priority after
> getting exception=" + ex);
>
> ./hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/QosFunction.java
>
> On Tue, Mar 26, 2013 at 10:07 AM, Nicolas Liochon 
> wrote:
>
> > Hi,
> >
> > I'm doing some recovery tests (unplugging a server) I've got something
> new:
> > hundreds of line like this one in the region server logs. Does anyone
> know
> > where it comes from?
> >
> > Thanks,
> >
> >
> > 2013-03-26 17:59:40,693 DEBUG
> > org.apache.hadoop.hbase.regionserver.QosFunction: Marking normal priority
> > after getting
> > exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException:
> > Region is not online: 3dfeaf95dd18d06c24fc679a0ab016d2
> > 2013-03-26 17:59:40,693 DEBUG
> > org.apache.hadoop.hbase.regionserver.QosFunction: Marking normal priority
> > after getting
> > exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException:
> > Region is not online: 3dfeaf95dd18d06c24fc679a0ab016d2
> > 2013-03-26 17:59:40,897 DEBUG
> > org.apache.hadoop.hbase.regionserver.QosFunction: Marking normal priority
> > after getting
> > exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException:
> > Region is not online: 3dfeaf95dd18d06c24fc679a0ab016d2
> > 2013-03-26 17:59:40,897 DEBUG
> > org.apache.hadoop.hbase.regionserver.QosFunction: Marking normal priority
> > after getting
> > exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException:
> > Region is not online: 3dfeaf95dd18d06c24fc679a0ab016d2
> > 2013-03-26 17:59:41,100 DEBUG
> > org.apache.hadoop.hbase.regionserver.QosFunction: Marking normal priority
> > after getting
> > exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException:
> > Region is not online: 3dfeaf95dd18d06c24fc679a0ab016d2
> >
>

Re: Marking normal priority after getting exception

[Nicolas Liochon]

Thanks, Stack. There is a log line for each failed get and in this test
there is a lot of gets. I will dig to understand if it can occur in
reasonable production conditions as well...


On Tue, Mar 26, 2013 at 7:57 PM, Stack  wrote:

> When a request goes into a server, a function is called to evaluate whether
> it should be high or low priority (The mechanism for making the
> function-to-call pluggable is a guava facility).  This mechanism has been
> in place for ever.
>
> On the recent rpc commit, this QoS stuff had to be refactored.   Previous,
> in essence, it undid the request to figure what region the request was for
> because if it was say .META., then the request should be given a high
> priority.  With the move to pb and the refactor of the way we do parameters
> and requests, there is less to undo.
>
> As part of the refactor, I enabled logging.  At a minimum, this exposed the
> following interesting phenomeon: If we get a request for a region that is
> not on the particular server, NotServingRegionException is thrown, which
> QoS then catches and sets priority to 'normal'.. then it lets the request
> come into the regionserver and it throws NSRE again.
>
> Anyway, if it is logging too much turn it down... I can.  I enabled it so
> could see what was going on.
>
> Sorry about that,
> St.Ack
>
>
> On Tue, Mar 26, 2013 at 10:17 AM, Nicolas Liochon 
> wrote:
>
> > Yes on trunk. I found this line, which seems to be written for guava.
> > That's the global picture behind this line that I don't have :-)
> >
> >
> > On Tue, Mar 26, 2013 at 6:15 PM, Ted Yu  wrote:
> >
> > > You tested with trunk, right ?
> > >
> > > Here it is:
> > >
> > >   if (LOG.isDebugEnabled()) LOG.debug("Marking normal priority
> after
> > > getting exception=" + ex);
> > >
> > >
> >
> ./hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/QosFunction.java
> > >
> > > On Tue, Mar 26, 2013 at 10:07 AM, Nicolas Liochon 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I'm doing some recovery tests (unplugging a server) I've got
> something
> > > new:
> > > > hundreds of line like this one in the region server logs. Does anyone
> > > know
> > > > where it comes from?
> > > >
> > > > Thanks,
> > > >
> > > >
> > > > 2013-03-26 17:59:40,693 DEBUG
> > > > org.apache.hadoop.hbase.regionserver.QosFunction: Marking normal
> > priority
> > > > after getting
> > > >
> exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException:
> > > > Region is not online: 3dfeaf95dd18d06c24fc679a0ab016d2
> > > > 2013-03-26 17:59:40,693 DEBUG
> > > > org.apache.hadoop.hbase.regionserver.QosFunction: Marking normal
> > priority
> > > > after getting
> > > >
> exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException:
> > > > Region is not online: 3dfeaf95dd18d06c24fc679a0ab016d2
> > > > 2013-03-26 17:59:40,897 DEBUG
> > > > org.apache.hadoop.hbase.regionserver.QosFunction: Marking normal
> > priority
> > > > after getting
> > > >
> exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException:
> > > > Region is not online: 3dfeaf95dd18d06c24fc679a0ab016d2
> > > > 2013-03-26 17:59:40,897 DEBUG
> > > > org.apache.hadoop.hbase.regionserver.QosFunction: Marking normal
> > priority
> > > > after getting
> > > >
> exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException:
> > > > Region is not online: 3dfeaf95dd18d06c24fc679a0ab016d2
> > > > 2013-03-26 17:59:41,100 DEBUG
> > > > org.apache.hadoop.hbase.regionserver.QosFunction: Marking normal
> > priority
> > > > after getting
> > > >
> exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException:
> > > > Region is not online: 3dfeaf95dd18d06c24fc679a0ab016d2
> > > >
> > >
> >
>

Re: Disable flaky tests, and let Jenkin stay blue?

[Nicolas Liochon]

I'm between +0 and -0.5
+0 because I like green status. They help to detect regression.

-0.5 because
   - If we can't afford to fix it now I guess it won't be fixed in the
future:  we will continue to keep it in the codebase (i.e. paying the cost
of updating it when we change an interface), but without any added value as
we don't run it.
   - some tests failures are actually issues in the main source code. Ok,
they're often minor, but still they are issues. Last example I have is from
today: the one found by Jeff related HBASE-8204.
- and sometimes it shows lacks in the way we test (for example, the
waitFor stuff, while quite obvious in a way, was added only very recently).
- often a flaky test is better than no test at all: they can still
detect regressions.
- I also don't understand why the precommit seems to be now better than
the main build.

For me, doing it in a case by case way would be simpler (using the
component owners: it a test on a given component is flaky, the decision can
be taken between the people who want to remove the test and the component
owners, with a jira, an analysis and a traced decision)

Cheers,

Nicolas



On Tue, Apr 2, 2013 at 7:09 PM, Jimmy Xiang  wrote:

> We have not seen couple blue Jenkin builds for 0.95/trunk for quite some
> time.  Because of this, sometimes we ignore the precommit build failures,
> which could let some bugs (code or test) sneaked in.
>
> I was wondering if it is time to disable all flaky tests and let Jenkin
> stay blue.  We can maintain a list of tests disabled, and get them back
> once they are fixed. For each disabled test, if someone wants to get it
> back, please file a jira so that we don't duplicate the effort and work on
> the same one.
>
> As to how to define a test flaky, to me, if a test fails twice in the last
> 10/20 runs, then it is flaky, if there is no apparent env issue.
>
> We have different Jenkins job for hadoop 1 and hadoop 2.  If a test is
> flaky for either one, it is flaky.
>
> What do you think?
>
> Thanks,
> Jimmy
>

Re: VOTE: hbase-0.95RC1, the second "Development" Series Release is available [WAS -> VOTE: hbase-0.95.0RC0, the first "Developer Release" release candidate is available for download and vote]

[Nicolas Liochon]

If I'm not wrong, the pom.xml for hbase-hadoop2 gets the hadoop1-compat
because hadoop.profile=1.0 by default. If so, it means that a user must
specify it from it's maven build line as well (i.e. having the dependency
on hbase-hadoop2  is not enough).

Is this voluntary? May be it should be documented?


On Wed, Apr 3, 2013 at 10:02 AM, Elliott Clark  wrote:

> +1
>
> Verified the gpg signature for hadoop2 and source tar balls.
> Spun up a local instance.
> Created tables. Put data, got data
> Ran TestAcidGuarantees for about 2 hours.
> Tried an online schema change while running test acid. (seemed to work but
> caused a huge drop in throughput)
> Created a snapshot while test acid was going on.
> Ran apache-rat on the source
>
> Native libs aren't present (are they supposed to be there for mlock ?)
> I filed some minor jiras but everything seems good for a developer release.
>
>
> On Tue, Apr 2, 2013 at 10:12 PM, Stack  wrote:
>
> > Here is the second 0.95.0 release candidate.  Should we put this out as
> > 0.95.0?  Please
> > vote by friday, April 5th.
> >
> > See the refguide [1] for definition of what a "Development" Series
> Release
> > is (or read below where we call it a 'developer release").  In short, it
> is
> > a preview release, not for production, put out early so us devs start
> > getting feedback the sooner.
> >
> > The release is available here:
> >
> >   http://people.apache.org/~stack/hbase-0.95.0RC1/
> >
> > Issues fixed in this release are available here:
> >
> >
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12324094
> >
> > ...including a fix for HBASE-8242, the issue that sunk the first release
> > candidate.
> >
> > Thanks,
> > St.Ack
> >
> > 1. http://hbase.apache.org/book.html#hbase.versioning
> >
> > On Mon, Apr 1, 2013 at 4:52 PM, Stack  wrote:
> >
> > > Here is our first 0.95.0 release candidate.  Should we put this out as
> > > 0.95.0?
> > >
> > > The 0.95.x series of releases are designated "developer releases", a
> type
> > > of release we have done in the past -- see the 0.89.x series -- where
> we
> > > let out 'raw', barely-tested product so developers and those generally
> > > interested can get some early exposure to what our next stable release
> to
> > > follow 0.94.x will look like.  Putting out these 'rough cuts' also
> helps
> > > to get the feedback started earlier while the release is still baking.
> > >
> > > These "developer release" come with no guarantees.  We make no
> > > promises that the next release will be compatible with this one or even
> > > that
> > > you will be able to preserve data across the update (This is at an
> > extreme,
> > > and highly unlikely, but could be the case).  No work has been done to
> > make
> > > it so you can migrate from 0.94.x HBase.
> > >
> > > In spite of all the caveats above, we still need to vote.  Remember,
> the
> > > bar
> > > is intentionally set lower on these "developer releases"; it will have
> to
> > > be
> > > something pretty bad to block this RC going out.  There should be a new
> > > release
> > > along in a week or two and we can address the offender there.
> > >
> > > The release artifacts may be downloaded from:
> > >
> > >   http://people.apache.org/~stack/hbase-0.95.0RC0/
> > >
> > > Let the vote run for a short time, say, April 3rd.  Take it for a quick
> > > spin.
> > >
> > > Over 1500 issues have been closed against 0.95.0.  It is tough pulling
> > out
> > > the highlights but I think most will be interested in big improvements
> > > in Mean Time To Recovery, a revamped metrics, support for hadoop1 and
> > > hadoop2 the list is long.  For the full list of issues fixed see:
> > >
> > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12324094
> > >
> > > All feedback is welcome.  We'd be interested in getting feedback on
> > > everything from the packaging, layout, through documentation, UI, and
> of
> > > course,
> > > any bugs found.
> > >
> > > Thanks,
> > > St.Ack
> > >
> > >
> > >
> >
>

Re: VOTE: hbase-0.95RC1, the second "Development" Series Release is available [WAS -> VOTE: hbase-0.95.0RC0, the first "Developer Release" release candidate is available for download and vote]

[Nicolas Liochon]

I was looking at the maven dependencies in
https://repository.apache.org/content/repositories/snapshots/org/apache/hbase/hbase/0.95.0-hadoop2-SNAPSHOT/hbase-0.95.0-hadoop2-20130402.025905-1.pom

I would expect (I may be wrong) that it's the one to be used when you build
a java application on top of HBase.

But this one will reference hadoop1 as the dependency, except if you
explicitly set the value.
For example, to build ycsb benchmark against hbase 0.95 and hadoop2 you
have to:
1) Change the ycsb pom to set the right dependencies (hbase-hadoop2)
2) Recompile it with mvn package -Dhadoop.profile=2.0

If you don't set hadoop.profile when building ycsb, it will try to use
hadoop1 and the build will fail.

Nicolas


On Wed, Apr 3, 2013 at 6:37 PM, Stack  wrote:

> On Wed, Apr 3, 2013 at 9:04 AM, Nicolas Liochon  wrote:
>
> > If I'm not wrong, the pom.xml for hbase-hadoop2 gets the hadoop1-compat
> > because hadoop.profile=1.0 by default. If so, it means that a user must
> > specify it from it's maven build line as well (i.e. having the dependency
> > on hbase-hadoop2  is not enough).
> >
> >
> I do not follow N.  Can you give more detail?
>
>
> > Is this voluntary? May be it should be documented?
> >
> >
> What do you see as the product of the above Nicolas?  It looks like
> *hadoop2.tgz has the right hadoop in it (and does not have hadoop1 compat
> bundled)?
>
> Thanks N,
> St.Ack
>

Re: Please welcome our newest committer, Amitanand Aiyer

[Nicolas Liochon]

Welcome, Amit!


On Thu, Apr 4, 2013 at 7:51 PM, Jimmy Xiang  wrote:

> Congratulations!
>
> On Thu, Apr 4, 2013 at 10:40 AM, Sergey Shelukhin  >wrote:
>
> > Congrats!
> >
> > On Thu, Apr 4, 2013 at 10:21 AM, Jean-Marc Spaggiari <
> > jean-m...@spaggiari.org> wrote:
> >
> > > Congratulation Amit!
> > >
> > > > On Thu, Apr 4, 2013 at 10:18 AM, Stack  wrote:
> > > >
> > > >> Our Amit has been around with a good while now doing important, deep
> > > stuff
> > > >> in thorny areas.  Please join me welcoming him as one of the hbase
> > > >> committer team.
> > > >>
> > > >> St.Ack
> > > >>
> > >
> >
>

trunk vs. precommit dev env

[Nicolas Liochon]

Hi,

It took me ages, but I found the difference between trunk and precommit
builds.
Precommits are executed on hadoop* machines. There is a single build at a
time.
Trunks are executed on ubuntu* machines. There are two simultaneous builds.

For example, we have currently on ubuntu3:
ubuntu3  1
 
core-integration-testing-maven-3-jdk-1.6
 
#639


2
 HBase-TRUNK 
#4019



I suppose we can have two hbase builds on the same machine.

Imho, that's why we have so many failures on trunk builds.

Two solutions:
 - Use hadoop* for trunk builds. That's the best option imho, they seem
cleaner then ubuntu* and less use.
- When building trunk, use a very low parallel test factor. May be 2.

Thoughts?
Does anyone know how to do this?

Nicolas

Re: trunk vs. precommit dev env

[Nicolas Liochon]

And, from HBASE-8280, it seems we don't use the same jvm for trunk and for
precommit builds.

precommit build => *1.6*

13/04/05 14:29:02 INFO util.TestShowProperties: Property
sun.boot.library.path=/home/jenkins/tools/java/jdk1.6.0_26/jre/lib/i386
13/04/05 14:29:02 INFO util.TestShowProperties: Property
java.vm.version=20.1-b02
13/04/05 14:29:02 INFO util.TestShowProperties: Property
java.vm.vendor=Sun Microsystems Inc.

trunk build => *1.7*

13/04/05 15:22:13 INFO util.TestShowProperties: Property
sun.boot.library.path=/x1/jenkins/tools/java/jdk1.7.0_04/jre/lib/amd64
13/04/05 15:22:13 INFO util.TestShowProperties: Property
java.vm.version=23.0-b21
13/04/05 15:22:13 INFO util.TestShowProperties: Property
java.vm.vendor=Oracle Corporation





On Fri, Apr 5, 2013 at 5:29 PM, Nicolas Liochon  wrote:

> Hi,
>
> It took me ages, but I found the difference between trunk and precommit
> builds.
> Precommits are executed on hadoop* machines. There is a single build at a
> time.
> Trunks are executed on ubuntu* machines. There are two simultaneous builds.
>
> For example, we have currently on ubuntu3:
> ubuntu3 <https://builds.apache.org/computer/ubuntu3/> 1
>  
> core-integration-testing-maven-3-jdk-1.6<https://builds.apache.org/job/core-integration-testing-maven-3-jdk-1.6/>
>  
> #639<https://builds.apache.org/job/core-integration-testing-maven-3-jdk-1.6/639/>
>
>
> 2
>  HBase-TRUNK <https://builds.apache.org/job/HBase-TRUNK/> 
> #4019<https://builds.apache.org/job/HBase-TRUNK/4019/>
>
>
>
> I suppose we can have two hbase builds on the same machine.
>
> Imho, that's why we have so many failures on trunk builds.
>
> Two solutions:
>  - Use hadoop* for trunk builds. That's the best option imho, they seem
> cleaner then ubuntu* and less use.
> - When building trunk, use a very low parallel test factor. May be 2.
>
> Thoughts?
> Does anyone know how to do this?
>
> Nicolas
>
>
>

Re: trunk vs. precommit dev env

[Nicolas Liochon]

Hum, I can't see the content of the url. After the authentication I have an
unreadable page.
You're mentionning a precommit setting, but the property I got for my last
try was with a 1.6 jdk?


On Fri, Apr 5, 2013 at 6:30 PM, Ted Yu  wrote:

> On https://builds.apache.org/job/PreCommit-HBASE-Build/configure , the JDK
> has been changed to 1.7 latest.
>
> Not sure who changed that.
>
>
> On Fri, Apr 5, 2013 at 9:11 AM, Nicolas Liochon  wrote:
>
> > And, from HBASE-8280, it seems we don't use the same jvm for trunk and
> for
> > precommit builds.
> >
> > precommit build => *1.6*
> >
> > 13/04/05 14:29:02 INFO util.TestShowProperties: Property
> > sun.boot.library.path=/home/jenkins/tools/java/jdk1.6.0_26/jre/lib/i386
> > 13/04/05 14:29:02 INFO util.TestShowProperties: Property
> > java.vm.version=20.1-b02
> > 13/04/05 14:29:02 INFO util.TestShowProperties: Property
> > java.vm.vendor=Sun Microsystems Inc.
> >
> > trunk build => *1.7*
> >
> > 13/04/05 15:22:13 INFO util.TestShowProperties: Property
> > sun.boot.library.path=/x1/jenkins/tools/java/jdk1.7.0_04/jre/lib/amd64
> > 13/04/05 15:22:13 INFO util.TestShowProperties: Property
> > java.vm.version=23.0-b21
> > 13/04/05 15:22:13 INFO util.TestShowProperties: Property
> > java.vm.vendor=Oracle Corporation
> >
> >
> >
> >
> >
> > On Fri, Apr 5, 2013 at 5:29 PM, Nicolas Liochon 
> wrote:
> >
> > > Hi,
> > >
> > > It took me ages, but I found the difference between trunk and precommit
> > > builds.
> > > Precommits are executed on hadoop* machines. There is a single build
> at a
> > > time.
> > > Trunks are executed on ubuntu* machines. There are two simultaneous
> > builds.
> > >
> > > For example, we have currently on ubuntu3:
> > > ubuntu3 <https://builds.apache.org/computer/ubuntu3/> 1
> > >  core-integration-testing-maven-3-jdk-1.6<
> > https://builds.apache.org/job/core-integration-testing-maven-3-jdk-1.6/>
> > >  #639<
> >
> https://builds.apache.org/job/core-integration-testing-maven-3-jdk-1.6/639/
> > >
> > >
> > >
> > > 2
> > >  HBase-TRUNK <https://builds.apache.org/job/HBase-TRUNK/> #4019<
> > https://builds.apache.org/job/HBase-TRUNK/4019/>
> > >
> > >
> > >
> > > I suppose we can have two hbase builds on the same machine.
> > >
> > > Imho, that's why we have so many failures on trunk builds.
> > >
> > > Two solutions:
> > >  - Use hadoop* for trunk builds. That's the best option imho, they seem
> > > cleaner then ubuntu* and less use.
> > > - When building trunk, use a very low parallel test factor. May be 2.
> > >
> > > Thoughts?
> > > Does anyone know how to do this?
> > >
> > > Nicolas
> > >
> > >
> > >
> >
>

Re: trunk vs. precommit dev env

[Nicolas Liochon]

 -Dsurefire.
secondPartThreadCount=2 will make it so only two // tests

Yes. I think we should use jdk 1.6 as well.


You don't want to run the trunk on hadoop* machines? This would be safer
imho.

I'm leaving now, but I can trigger a few builds set on hadoop & jdk 1.6
tomorrow, when the bay area will be sleeping!



On Fri, Apr 5, 2013 at 8:30 PM, Stack  wrote:

> On Fri, Apr 5, 2013 at 8:29 AM, Nicolas Liochon  wrote:
>
> > Hi,
> >
> > It took me ages, but I found the difference between trunk and precommit
> > builds.
> > Precommits are executed on hadoop* machines. There is a single build at a
> > time.
> > Trunks are executed on ubuntu* machines. There are two simultaneous
> builds.
> >
> > For example, we have currently on ubuntu3:
> > ubuntu3 <https://builds.apache.org/computer/ubuntu3/> 1
> >  core-integration-testing-maven-3-jdk-1.6<
> > https://builds.apache.org/job/core-integration-testing-maven-3-jdk-1.6/>
> >  #639<
> >
> https://builds.apache.org/job/core-integration-testing-maven-3-jdk-1.6/639/
> > >
> >
> >
> > 2
> >  HBase-TRUNK <https://builds.apache.org/job/HBase-TRUNK/>
> > #4019<https://builds.apache.org/job/HBase-TRUNK/4019/>
> >
> >
> >
> > I suppose we can have two hbase builds on the same machine.
> >
> > Imho, that's why we have so many failures on trunk builds.
> >
> > Two solutions:
> >  - Use hadoop* for trunk builds. That's the best option imho, they seem
> > cleaner then ubuntu* and less use.
> > - When building trunk, use a very low parallel test factor. May be 2.
> >
> > Thoughts?
> > Does anyone know how to do this?
> >
>
> I can do this while your perms are being worked out.
>
> -Dsurefire.secondPartThreadCount=2 will make it so only two // tests
> running?
>
> Thanks N,
> St.Ack
>

Re: trunk vs. precommit dev env

[Nicolas Liochon]

Jenkins has been down for 15 hours now, so I can't change the
configuration...
If nobody objects, I will:
 - set all the builds to hadoop* and jdk 1.6
 - in parallel do a few tries on jdk 1.7. Especially, I would like to
ensure that we have a recent 64 bits version of the jdk1.7 on all machines
 - see what's flaky with 1.6 (I think Jeffrey's tool will allow us to get
this list after a few days of commit)
 - then set all the trunk builds to 1.7




On Sat, Apr 6, 2013 at 2:03 AM, Andrew Purtell  wrote:

> Also I am remiss to mention that the JDK version used on EC2 Jenkins is
> currently JDK6. I've been meaning to bring up discussion on moving to JDK7
> there. It's already installed on the AMI, the only change necessary is to
> the node configuration in Jenkins.
>
>
> On Fri, Apr 5, 2013 at 12:22 PM, Andrew Purtell 
> wrote:
>
> > It seems better to me to change precommit builds to jdk7 too. Otherwise
> > wouldn't we be relying on a EOL product for build stability? With the
> flaky
> > test category under discussion we can get green builds that way without
> > keeping one foot (or both) in the past. Thoughts?
> >
> > As for moving builds to the hadoop* machines, +1 they seem cleaner.
> >
> >
> > On Friday, April 5, 2013, Stack wrote:
> >
> >> On Fri, Apr 5, 2013 at 11:36 AM, Nicolas Liochon 
> >> wrote:
> >>
> >> >  -Dsurefire.
> >> > secondPartThreadCount=2 will make it so only two // tests
> >> >
> >> > Yes. I think we should use jdk 1.6 as well.
> >> >
> >> >
> >> >
> >> Go for it.  We had a jdk7 discussion a while back and switched trunk
> build
> >> to jdk7 as a result.  Stability is more important so feel free to change
> >> if
> >> that will help improve it.
> >>
> >>
> >> > You don't want to run the trunk on hadoop* machines? This would be
> safer
> >> > imho.
> >> >
> >> >
> >> Go for it.
> >>
> >> St.Ack
> >>
> >
> >
> > --
> > Best regards,
> >
> >- Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
> >
>
>
> --
> Best regards,
>
>- Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Re: trunk vs. precommit dev env

[Nicolas Liochon]

The jenkins settings on hadoop* are overwritten by the
${WORKSPACE}/nightly/hudsonEnv.sh
So whatever jdk settings you put in jenkins, you have a 1.6 at the end.

As well, the maven and jdk directory differ between the ubuntu and the
hadoop machines.

Today, for trunk and trunk on hadoop 2, I've set jdk=1.6 & parallel=2



On Sat, Apr 6, 2013 at 2:33 PM, Ted  wrote:

> Sounds like a nice plan.
>
> On Apr 6, 2013, at 4:38 AM, Nicolas Liochon  wrote:
>
> > Jenkins has been down for 15 hours now, so I can't change the
> > configuration...
> > If nobody objects, I will:
> > - set all the builds to hadoop* and jdk 1.6
> > - in parallel do a few tries on jdk 1.7. Especially, I would like to
> > ensure that we have a recent 64 bits version of the jdk1.7 on all
> machines
> > - see what's flaky with 1.6 (I think Jeffrey's tool will allow us to get
> > this list after a few days of commit)
> > - then set all the trunk builds to 1.7
> >
> >
> >
> >
> > On Sat, Apr 6, 2013 at 2:03 AM, Andrew Purtell 
> wrote:
> >
> >> Also I am remiss to mention that the JDK version used on EC2 Jenkins is
> >> currently JDK6. I've been meaning to bring up discussion on moving to
> JDK7
> >> there. It's already installed on the AMI, the only change necessary is
> to
> >> the node configuration in Jenkins.
> >>
> >>
> >> On Fri, Apr 5, 2013 at 12:22 PM, Andrew Purtell 
> >> wrote:
> >>
> >>> It seems better to me to change precommit builds to jdk7 too. Otherwise
> >>> wouldn't we be relying on a EOL product for build stability? With the
> >> flaky
> >>> test category under discussion we can get green builds that way without
> >>> keeping one foot (or both) in the past. Thoughts?
> >>>
> >>> As for moving builds to the hadoop* machines, +1 they seem cleaner.
> >>>
> >>>
> >>> On Friday, April 5, 2013, Stack wrote:
> >>>
> >>>> On Fri, Apr 5, 2013 at 11:36 AM, Nicolas Liochon 
> >>>> wrote:
> >>>>
> >>>>> -Dsurefire.
> >>>>> secondPartThreadCount=2 will make it so only two // tests
> >>>>>
> >>>>> Yes. I think we should use jdk 1.6 as well.
> >>>> Go for it.  We had a jdk7 discussion a while back and switched trunk
> >> build
> >>>> to jdk7 as a result.  Stability is more important so feel free to
> change
> >>>> if
> >>>> that will help improve it.
> >>>>
> >>>>
> >>>>> You don't want to run the trunk on hadoop* machines? This would be
> >> safer
> >>>>> imho.
> >>>> Go for it.
> >>>>
> >>>> St.Ack
> >>>
> >>>
> >>> --
> >>> Best regards,
> >>>
> >>>   - Andy
> >>>
> >>> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> >>> (via Tom White)
> >>
> >>
> >> --
> >> Best regards,
> >>
> >>   - Andy
> >>
> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> >> (via Tom White)
> >>
>

Re: About distributed lock service in HBase

[Nicolas Liochon]

Hi,

1. Yes. Client applications can start/read/write even when there is no
master.
2. HBase already uses ZooKeeper.

You may want to have to look at the hbase reference guide (
http://hbase.apache.org/book.html).

Nicolas


On Mon, Apr 8, 2013 at 4:39 PM, Brady Zhong wrote:

> Hi all,
>
> My name is Brady Zhong, a college student using HBase to develop our own
> project. Currently I confronted with a problem. Since we need some kind of
> high availability, we hope HBase can keep available even though the HMaster
> goes down. Here're my questions:
>
> 1. Can HBase work during the node failure of HMaster? Can users write or
> read the database before the switch and recovery of HMaster?
> 2. Why not use Zookeeper as distributed lock service for HBase, like Chubby
> for Google Big Table?
>
> Thanks very much for any help in advance.
>
>
> Best regards,
> Brady Zhong
>

Recording test results ERROR: Failed to archive test

[Nicolas Liochon]

We've got a lot of errors in precommit that seems to be due to Jenkins
needing more memory.

It seems to be a Jenkins needing more memory. It seems that we don't have
this on trunk, so it would be a difference of config between the two
environments.

Do you agree?

Who could fix this (i.e. has the access rights)?

Cheers,

Nicolas



Archiving artifacts
Recording test results ERROR: Failed to archive test reports
hudson.util.IOException2:
remote file operation failed:
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build at
hudson.remoting.Channel@55899539:hadoop3 at
hudson.FilePath.act(FilePath.java:861)at
hudson.FilePath.act(FilePath.java:838)at
hudson.tasks.junit.JUnitParser.parse(JUnitParser.java:87)at
hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:122)at
hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:134)at
hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)at
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:810)at
hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:785)at
hudson.model.Build$BuildExecution.post2(Build.java:183)at
hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:732)at
hudson.model.Run.execute(Run.java:1593)at
hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)at
hudson.model.ResourceController.execute(ResourceController.java:88)at
hudson.model.Executor.run(Executor.java:236)Caused
by:
java.io.IOException:
Remote call on hadoop3 failed at
hudson.remoting.Channel.call(Channel.java:681)at
hudson.FilePath.act(FilePath.java:854)...
13 more Caused by:
java.lang.OutOfMemoryError:
Java heap space at
java.nio.HeapCharBuffer.(HeapCharBuffer.java:39)at
java.nio.CharBuffer.allocate(CharBuffer.java:312)at
java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:760)at
java.nio.charset.Charset.decode(Charset.java:771)at
hudson.tasks.junit.SuiteResult.(SuiteResult.java:215)at
hudson.tasks.junit.SuiteResult.parseSuite(SuiteResult.java:147)at
hudson.tasks.junit.SuiteResult.parse(SuiteResult.java:132)at
hudson.tasks.junit.TestResult.parse(TestResult.java:267)at
hudson.tasks.junit.TestResult.parsePossiblyEmpty(TestResult.java:223)at
hudson.tasks.junit.TestResult.parse(TestResult.java:1

Re: Recording test results ERROR: Failed to archive test

[Nicolas Liochon]

I was under the impression that it was a hudson/jenkins stack, not a maven
one?
But it's worth trying.


On Mon, Apr 8, 2013 at 5:26 PM, Stack  wrote:

> I took a look at the jenkins config for precommit.
>
>
> We are setting this environment variable:
>
> MAVEN_OPTS=-Xmx3G
>
> ... which should be big enough.
>
> I also see
>
>  MAVEN_OPTS="-Xmx3g"
>
> in the test-patch.properties.
>
> In test-patch.sh, I see this being picked up.  Should I change
> test-patch.sh so it prints out current MAVEN_OPTS?
>
> St.Ack
>
>
>
>
>
> On Mon, Apr 8, 2013 at 8:19 AM, Nicolas Liochon  wrote:
>
> > We've got a lot of errors in precommit that seems to be due to Jenkins
> > needing more memory.
> >
> > It seems to be a Jenkins needing more memory. It seems that we don't have
> > this on trunk, so it would be a difference of config between the two
> > environments.
> >
> > Do you agree?
> >
> > Who could fix this (i.e. has the access rights)?
> >
> > Cheers,
> >
> > Nicolas
> >
> >
> >
> > Archiving artifacts
> > Recording test results ERROR: Failed to archive test reports
> > hudson.util.IOException2<
> > http://stacktrace.jenkins-ci.org/search?query=hudson.util.IOException2>:
> > remote file operation failed:
> > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build at
> > hudson.remoting.Channel@55899539:hadoop3 at
> > hudson.FilePath.act(FilePath.java:861)<
> >
> http://stacktrace.jenkins-ci.org/search/?query=hudson.FilePath.act&entity=method
> > >at
> > hudson.FilePath.act(FilePath.java:838)<
> >
> http://stacktrace.jenkins-ci.org/search/?query=hudson.FilePath.act&entity=method
> > >at
> > hudson.tasks.junit.JUnitParser.parse(JUnitParser.java:87)<
> >
> http://stacktrace.jenkins-ci.org/search/?query=hudson.tasks.junit.JUnitParser.parse&entity=method
> > >at
> >
> hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:122)<
> >
> http://stacktrace.jenkins-ci.org/search/?query=hudson.tasks.junit.JUnitResultArchiver.parse&entity=method
> > >at
> >
> >
> hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:134)<
> >
> http://stacktrace.jenkins-ci.org/search/?query=hudson.tasks.junit.JUnitResultArchiver.perform&entity=method
> > >at
> > hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)<
> >
> http://stacktrace.jenkins-ci.org/search/?query=hudson.tasks.BuildStepMonitor$1.perform&entity=method
> > >at
> >
> >
> hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:810)<
> >
> http://stacktrace.jenkins-ci.org/search/?query=hudson.model.AbstractBuild$AbstractBuildExecution.perform&entity=method
> > >at
> >
> >
> hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:785)<
> >
> http://stacktrace.jenkins-ci.org/search/?query=hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps&entity=method
> > >at
> > hudson.model.Build$BuildExecution.post2(Build.java:183)<
> >
> http://stacktrace.jenkins-ci.org/search/?query=hudson.model.Build$BuildExecution.post2&entity=method
> > >at
> >
> >
> hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:732)<
> >
> http://stacktrace.jenkins-ci.org/search/?query=hudson.model.AbstractBuild$AbstractBuildExecution.post&entity=method
> > >at
> > hudson.model.Run.execute(Run.java:1593)<
> >
> http://stacktrace.jenkins-ci.org/search/?query=hudson.model.Run.execute&entity=method
> > >at
> > hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)<
> >
> http://stacktrace.jenkins-ci.org/search/?query=hudson.model.FreeStyleBuild.run&entity=method
> > >at
> > hudson.model.ResourceController.execute(ResourceController.java:88)<
> >
> http://stacktrace.jenkins-ci.org/search/?query=hudson.model.ResourceController.execute&entity=method
> > >at
> > hudson.model.Executor.run(Executor.java:236)<
> >
> http://stacktrace.jenkins-ci.org/search/?query=hudson.model.Executor.run&entity=method
> > >Caused
> > by:
> > java.io.IOException<
> > http://stacktrace.jenkins-ci.org/search?query=java.io.IOException>:
> > Remote call on hadoop3 failed at
> > hudson.remoting.Channel.call(Channel.java:681)<
> >
> http://stacktrace.jenkins-ci.org/search/?query=hudson.remoting.Channel.call&entity=method
> > >at
>

Re: About distributed lock service in HBase

[Nicolas Liochon]

HBASE-5541 is about rowlock. The master plays no role in this.  This lock
does not have to be distributed, because, in bigtable/hbase architecture,
the rows are allocated to a single region / region server. This makes
things faster.

Nicolas


On Mon, Apr 8, 2013 at 5:27 PM, Brady Zhong wrote:

> Hi Nicolas,
>
> Thanks for your explanation. I know HBase uses ZeeKeeper as coordinated
> service but not a lock management service like Chubby. Like HBASE-5541
> states, it seems the RegionServer controls the whole write process
> including providing row lock, region lock etc.. So I wonder why we don't
> use high available service like ZooKeeper creating, managing and releasing
> lock?
>
>
> Brady
>
> On Mon, Apr 8, 2013 at 11:13 PM, Nicolas Liochon 
> wrote:
>
> > Hi,
> >
> > 1. Yes. Client applications can start/read/write even when there is no
> > master.
> > 2. HBase already uses ZooKeeper.
> >
> > You may want to have to look at the hbase reference guide (
> > http://hbase.apache.org/book.html).
> >
> > Nicolas
> >
> >
> > On Mon, Apr 8, 2013 at 4:39 PM, Brady Zhong  > >wrote:
> >
> > > Hi all,
> > >
> > > My name is Brady Zhong, a college student using HBase to develop our
> own
> > > project. Currently I confronted with a problem. Since we need some kind
> > of
> > > high availability, we hope HBase can keep available even though the
> > HMaster
> > > goes down. Here're my questions:
> > >
> > > 1. Can HBase work during the node failure of HMaster? Can users write
> or
> > > read the database before the switch and recovery of HMaster?
> > > 2. Why not use Zookeeper as distributed lock service for HBase, like
> > Chubby
> > > for Google Big Table?
> > >
> > > Thanks very much for any help in advance.
> > >
> > >
> > > Best regards,
> > > Brady Zhong
> > >
> >
>

@Test(timeout = 1000)

[Nicolas Liochon]

Hi,

I see some tests with a test timeout of 1s.
The problem with this is that it may happen with a GC or anything on the
server.

Any issue to decide to have a minimum of 60s for such settings?

Thanks,

Nicolas

Re: Recording test results ERROR: Failed to archive test

[Nicolas Liochon]

It seems random, but frequent:
https://builds.apache.org/job/PreCommit-HBASE-Build/5181/console
https://builds.apache.org/job/PreCommit-HBASE-Build/5182/console
https://builds.apache.org/job/PreCommit-HBASE-Build/5184/console

In 4955, I have errors related to maven (it seems the last version of
surefire uses more memory than before), so increasing the memory for maven
will be useful anyway I think. But as we use a 32bits jvm on hadoop*, the
limit is likely around 3.5Gb..

Cheers,

N.




On Mon, Apr 8, 2013 at 7:32 PM, Stack  wrote:

> On Mon, Apr 8, 2013 at 8:29 AM, Nicolas Liochon  wrote:
>
> > I was under the impression that it was a hudson/jenkins stack, not a
> maven
> > one?
>
>
> Pardon me.  I reread the stack trace.  Yeah, this is post-maven copying of
> results so yeah a jenkins issue.  What I found in the past is that some of
> our test output is massive (I downloaded the test result to take a look) or
> rather, can be massive on occasion; e.g. replication tests. IIRC, it was an
> in-memory realization of some xml report that was bringing on the OOME
> Workaround seemed to be fixing the test producing all the output.  Which
> test run was this?  I can take a look since I looked in the past.
>
> Thanks N,
> St.Ack
>

Re: Recording test results ERROR: Failed to archive test

[Nicolas Liochon]

It seems it's not related to the machines: on the failure above, 2 are on
hadoop2, 1 or hadoop3, and it's the only one we used recently...


On Mon, Apr 8, 2013 at 9:04 PM, Stack  wrote:

> Just saying, 3.4G to run tests is also obnoxious.  We need to fix this
> too.
> St.Ack
>
>
> On Mon, Apr 8, 2013 at 12:02 PM, Stack  wrote:
>
> > On Mon, Apr 8, 2013 at 10:40 AM, Nicolas Liochon  >wrote:
> >
> >> It seems random, but frequent:
> >> https://builds.apache.org/job/PreCommit-HBASE-Build/5181/console
> >> https://builds.apache.org/job/PreCommit-HBASE-Build/5182/console
> >> https://builds.apache.org/job/PreCommit-HBASE-Build/5184/console
> >>
> >> In 4955, I have errors related to maven (it seems the last version of
> >> surefire uses more memory than before), so increasing the memory for
> maven
> >> will be useful anyway I think. But as we use a 32bits jvm on hadoop*,
> the
> >> limit is likely around 3.5Gb..
> >>
> >> Cheers,
> >>
> >
> >
> > I looked at first job.  We have tests writing 10M of logs which is
> totally
> > obnoxious but probably not the root of this jenkins memory issue:
> >
> >
> > durruti:trunk stack$ find . -size +10M
> >
> >
> ./hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestAdmin-output.txt
> >
> >
> ./hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestCloneSnapshotFromClient-output.txt
> >
> >
> ./hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestFromClientSide-output.txt
> >
> >
> ./hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestFromClientSide3-output.txt
> >
> >
> ./hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiParallel-output.txt
> >
> >
> ./hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClient-output.txt
> >
> >
> ./hbase-server/target/surefire-reports/org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction-output.txt
> >
> >
> ./hbase-server/target/surefire-reports/org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit-output.txt
> >
> >
> ./hbase-server/target/surefire-reports/org.apache.hadoop.hbase.regionserver.wal.TestHLogSplitCompressed-output.txt
> >
> >
> ./hbase-server/target/surefire-reports/org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient-output.txt
> >
> >
> ./hbase-server/target/surefire-reports/org.apache.hadoop.hbase.TestAcidGuarantees-output.txt
> >
> >
> ./hbase-server/target/surefire-reports/org.apache.hadoop.hbase.util.TestHBaseFsck-output.txt
> >
> > Let me mess w/ upping the maven size but yeah, not sure how to get us
> more
> > jenkins memory.  Is it always on same machines that we fail?
> > St.Ack
> >
>

Re: @Test(timeout = 1000)

[Nicolas Liochon]

I've created HBASE-8003 and bumped the tests I found with a timeout
inferior to 30s to 60s.
With this, if a timeout occurs, we know it's very likely to be the test or
hbase, not the env :-)



On Tue, Apr 9, 2013 at 5:06 AM, Enis Söztutar  wrote:

> +1 on increasing the timeouts for those to at least 10sec.
>
>
> On Mon, Apr 8, 2013 at 9:19 AM, Ted Yu  wrote:
>
> > In trunk, I found 4 tests with such annotation.
> > They're all small tests.
> >
> > I guess the intention was that GC wouldn't be long in a test where
> cluster
> > is not spun up.
> >
> > I think increasing the timeout should be fine.
> >
> > Cheers
> >
> > On Mon, Apr 8, 2013 at 9:10 AM, Nicolas Liochon 
> wrote:
> >
> > > Hi,
> > >
> > > I see some tests with a test timeout of 1s.
> > > The problem with this is that it may happen with a GC or anything on
> the
> > > server.
> > >
> > > Any issue to decide to have a minimum of 60s for such settings?
> > >
> > > Thanks,
> > >
> > > Nicolas
> > >
> >
>

Re: Anyone know how to get surefire to list timed out tests?

[Nicolas Liochon]

I don't know how to do that automatically (except some workarounds such as
the one above but they don't always work).

Manually, it's not too difficult, as the report (
http://54.241.6.143/job/HBase-0.94/org.apache.hbase$hbase/71/testReport/)
tells you the number of missing tests compared to the previous build per
package.
Once you have the package, the list is usually small, and comparing with
the previous report becomes possible. In the example above, it's
TestSizeBasedThrottler  for example.

Yes, I know, it's not an industrial solution...



On Tue, Apr 9, 2013 at 5:20 AM, Ted Yu  wrote:

> A little parsing is needed:
>   ZB_STACK=`jps | grep surefirebooter | cut -d ' ' -f 1 | xargs -n 1
> jstack | grep ".test" | grep "\.java"`
>
> On Mon, Apr 8, 2013 at 8:14 PM, Andrew Purtell 
> wrote:
>
> > I just checked and the test name isn't embedded in the jps output. Will
> try
> > running with mvn -X instead.
> >
> >
> >
> > On Mon, Apr 8, 2013 at 8:12 PM, Andrew Purtell 
> > wrote:
> >
> > > Thanks Ted. Good idea.
> > >
> > >
> > > On Mon, Apr 8, 2013 at 8:06 PM, Ted Yu  wrote:
> > >
> > >> Andy:
> > >> Have you looked at how dev-support/test-patch.sh finds zombie tests ?
> > >> It starts from line 659:
> > >>
> > >>   ZOMBIE_TESTS_COUNT=`jps | grep surefirebooter | wc -l`
> > >>   if [[ $ZOMBIE_TESTS_COUNT != 0 ]] ; then
> > >>
> > >> Cheers
> > >>
> > >> On Mon, Apr 8, 2013 at 7:09 PM, Andrew Purtell 
> > >> wrote:
> > >>
> > >> > I set up a freestyle project to build by invoking maven out of a
> shell
> > >> > script. My guess is that will show the timed out tests as happens
> when
> > >> run
> > >> > locally in a terminal. See
> > http://54.241.6.143/job/HBase-0.94-Freestyle
> > >> >
> > >> >
> > >> > On Mon, Apr 8, 2013 at 5:45 PM, Andrew Purtell  >
> > >> > wrote:
> > >> >
> > >> > > See for example http://54.241.6.143/job/HBase-0.94/71/console:
> > >> > >
> > >> > > [...]
> > >> > > [JENKINS] Recording test results
> > >> > > [INFO]
> > >> > >
> > >>
> 
> > >> > > [INFO] BUILD FAILURE
> > >> > > [INFO]
> > >> > >
> > >>
> 
> > >> > > [INFO] Total time: 1:18:06.862s
> > >> > > [INFO] Finished at: Mon Apr 08 08:10:20 UTC 2013
> > >> > > [INFO] Final Memory: 32M/501M
> > >> > > [INFO]
> > >> > >
> > >>
> 
> > >> > > [JENKINS] Archiving
> > >> /home/ec2-user/jenkins/workspace/HBase-0.94/pom.xml
> > >> > to
> > >> > >
> > >> >
> > >>
> >
> /var/lib/jenkins/jobs/HBase-0.94/modules/org.apache.hbase$hbase/builds/2013-04-08_06-52-05/archive/org.apache.hbase/hbase/0.94.6/hbase-0.94.6.pom
> > >> > > Waiting for Jenkins to finish collecting data
> > >> > > mavenExecutionResult exceptions not empty
> > >> > > message : Failed to execute goal
> > >> > >
> > org.apache.maven.plugins:maven-surefire-plugin:2.12-TRUNK-HBASE-2:test
> > >> > > (secondPartTestsExecution) on project hbase: Failure or timeout
> > >> > > cause : Failure or timeout
> > >> > > Stack trace :
> > >> > > org.apache.maven.lifecycle.LifecycleExecutionException: Failed to
> > >> execute
> > >> > > goal
> > >> >
> org.apache.maven.plugins:maven-surefire-plugin:2.12-TRUNK-HBASE-2:test
> > >> > > (secondPartTestsExecution) on project hbase: Failure or timeout
> > >> > > [unhelpful big stacktrace]
> > >> > > channel stopped
> > >> > > Sending e-mails to: dev@hbase.apache.org
> > >> > > Finished: FAILURE
> > >> > >
> > >> > > It would be fantastic if the test(s) which timed out were listed
> > >> somehow
> > >> > > here.
> > >> > >
> > >> > > --
> > >> > > Best regards,
> > >> > >
> > >> > >- Andy
> > >> > >
> > >> > > Problems worthy of attack prove their worth by hitting back. -
> Piet
> > >> Hein
> > >> > > (via Tom White)
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Best regards,
> > >> >
> > >> >- Andy
> > >> >
> > >> > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein
> > >> > (via Tom White)
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > >- Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> >- Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>

Re: Build failed in Jenkins: HBase-TRUNK #4047

[Nicolas Liochon]

I don't know who added them actually. They're executed in 0.95 as well I
think: they are executed with mvn install.
They seem to need ~1 hour to run and failed 2 of the last 10 builds (so
they fail 33% of the time when they are executed).

For now, on trunk, I removed them by setting a dummy value to it.test.

On the long term, the best option I see is to set another build, depending
on trunk, to execute them. This would save some time on the main build,
make the main build consistent with the precommit one, while still having
the ITests running often


On Wed, Apr 10, 2013 at 8:35 AM, Stack  wrote:

> On Tue, Apr 9, 2013 at 5:01 PM, Sergey Shelukhin  >wrote:
>
> > I noticed that integration tests ran as part of the build... one of them
> > has 30-minutes runtime by default and so it failed.
> > Did anything change recently w/the setup? I thought the tests didn't run
> as
> > part of the build before.
> >
> >
> Hmm. They should not be running as part of regular build.
>
> [INFO] HBase - Integration Tests . FAILURE
> [35:19.292s]
>
>
> Will look in morning.  Maybe nkeywal has some input?  (He just came
> online).
>
>
> St.Ack
>

Re: Build failed in Jenkins: HBase-TRUNK #4047

[Nicolas Liochon]

When you do install, it implies verify.
And using 'mvn install' makes sense as it more or less needed when using
maven multimodules.
What we could do is something like "mvn clean install -DskipTests && mvn
test"


On Wed, Apr 10, 2013 at 8:43 PM, Sergey Shelukhin wrote:

> I think it.test is needed to package them properly.
> What causes them to be executed in the first place? Do we now have verify
> step? Wasn't the case before.
>
> On Wed, Apr 10, 2013 at 9:38 AM, Stack  wrote:
>
> > On Wed, Apr 10, 2013 at 12:16 AM, Nicolas Liochon 
> > wrote:
> >
> > > I don't know who added them actually. They're executed in 0.95 as well
> I
> > > think: they are executed with mvn install.
> > > They seem to need ~1 hour to run and failed 2 of the last 10 builds (so
> > > they fail 33% of the time when they are executed).
> > >
> > > For now, on trunk, I removed them by setting a dummy value to it.test.
> > >
> > >
> > Thanks N.  I did the same in 0.95.  I filed
> > https://issues.apache.org/jira/browse/HBASE-8319 to figure how to turn
> > them
> > off again so they are not inline w/ general unit test suite.
> >
> >
> >
> > > On the long term, the best option I see is to set another build,
> > depending
> > > on trunk, to execute them. This would save some time on the main build,
> > > make the main build consistent with the precommit one, while still
> having
> > > the ITests running often
> > >
> > >
> > Yes.
> >
> > St.Ack
> >
>

trunk build

[Nicolas Liochon]

Hi,

HBase-trunk build starts to be in a better shape now, we're likely other
50% success rate now. I've got a few questions however.

Issues were:
 - Trunk built on ubuntu* machines where using jdk 1.7 while the precommit,
on hadoop*, was on 1.6
 - precommit can't be set to use 1.7. It uses 1.6 32bits, as set by
hudsonEnv.sh
 - Trunk builds are executed in parallel with other builds, while for
precommit, hbase is alone on is machine.
 - jenkins, on precommit, often fails to parse the build results (jenkins
memory issue, still open, see below).

I've:
 - changed the jdk to 1.6
 - set the test parallelisation to 2 on trunk/ubuntu
 - fixed various test flakiness
 - increased the test timeout of a few tests to 60s. We should not set the
timeout to lower values, it makes them dependant to the machine env. When
in doubt, don't use them imho, or set them to 10 minutes or more.
 - changes some logs from debug to trace or even remove some.

Still to be done:
 - install the jdk 1.7 64 bits on the hadoop* machines. Someone knows how
to do that?
 - find a way to make it used by the test even if we use the script
'hudsonEnv.sh'.
 - increase the memory of the Jenkins JVM. Again, someone knows how to do
that?
 - use all the hadoop* to build. We're using 3 out of the 9 available
today. Any reason for this?
 - there are still some flaky tests...

Cheers,

Nicolas

Re: Pre-commit build broken?

[Nicolas Liochon]

May be the infra team is working on the machine. But the error you quoted
can be misleading as the real settings are set later in the hudsonEnv
script I mentionned yesterday.

The message says

-1 hadoop2.0. The patch failed to compile against the hadoop 2.0 profile.

Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5265//console


Does it work for you locally?



On Thu, Apr 11, 2013 at 6:15 PM, Ted Yu  wrote:

> This happened on hadoop2 while QA run on hadoop1 proceeded normally.
>
> On Thu, Apr 11, 2013 at 9:07 AM, Jean-Marc Spaggiari <
> jean-m...@spaggiari.org> wrote:
>
> > Hi,
> >
> > Do we have any issue with the pre-commit builds?
> >
> > https://builds.apache.org/job/PreCommit-HBASE-Build/5265//console
> >
> >
> > OpenJDK Runtime Environment (IcedTea6 1.9.9)
> (6b20-1.9.9-0ubuntu1~10.04.2)
> > OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> > /home/hudson/tools/java/latest1.6
> > /tmp/hudson220092556573598577.sh: line 7:
> > /home/hudson/tools/java/latest1.6/bin/java: No such file or directory
> >
> > JM
> >
>

precommit machines

[Nicolas Liochon]

I've added hadoop4 and hadoop5 to the machines to be used for precommit
tests/
May be they were not in the list for a good reason. Remove them in
https://builds.apache.org/job/PreCommit-HBASE-Build/configure if it breaks
too much.

If it works well during a few days, I will add hadoop6+ as well.

Cheers,

Nicolas

Re: precommit machines

[Nicolas Liochon]

Done. Is this documented somewhere? I had no answer to my mail to the dev
list when I asked why we were not using these machines.



On Fri, Apr 12, 2013 at 6:55 PM, Giridharan Kesavan <
gkesa...@hortonworks.com> wrote:

> Nicolas,
>
> Please do not add more machins to hbase pool. There are other projects that
> uses those machines like Pig, hive, hadoop, zookeeper, hcatalog and couple
> of other projects as well.
>
> Please remove hadoop4 and hadoop5 from the pool.
>
> thanks,
> Giridharan
> Apache Jenkins Admin
>
>
> >
> > ------ Forwarded message --
> > From: Nicolas Liochon 
> > Date: Fri, Apr 12, 2013 at 8:43 AM
> > Subject: precommit machines
> > To: dev@hbase.apache.org
> >
> >
> > I've added hadoop4 and hadoop5 to the machines to be used for precommit
> > tests/
> > May be they were not in the list for a good reason. Remove them in
> > https://builds.apache.org/job/PreCommit-HBASE-Build/configure if it
> breaks
> > too much.
> >
> > If it works well during a few days, I will add hadoop6+ as well.
> >
> > Cheers,
> >
> > Nicolas
> >
> >
>

Re: precommit machines

[Nicolas Liochon]

You can enhance TestShowProperties to add the info you want to check.
Then
https://builds.apache.org/job/HBase-TRUNK/4062/artifact/trunk/hbase-common/target/surefire-reports/org.apache.hadoop.hbase.util.TestShowProperties-output.txt

I see that ubuntu3 has a newer java version than ubuntu6:

java.version=1.6.0_32 vs. java.version=1.6.0_27

As well
MaxFileDescriptor=4 (was 4), vs.  MaxFileDescriptor=4096 (was 4096)

could explain some problems. I'm going to create an infra call for this.

But for trunk, on the last 5 failures, 2 were on ubuntu3, so it's not the
only issue (and some issues are in HBase, not the env :-).


On Mon, Apr 15, 2013 at 5:29 AM, lars hofhansl  wrote:

> On a related note, I have limited the 0.94 build to Ubuntu3 only, after I
> noticed that almost all transient failures were on Ubuntu{1|2|4|5}.
> Did that with build #950, not that a single failure since (hope I did not
> jinx it now!!).
>
> Might be worth figuring out how that machine differs from the other
> Ubunutu environments.
>
>
> Now I need to check the EC2 builds, which have been failing for a while
> now.
>
>
> -- Lars
>
>
>
> 
>  From: Stack 
> To: HBase Dev List 
> Cc: gkesa...@apache.org
> Sent: Sunday, April 14, 2013 6:38 PM
> Subject: Re: precommit machines
>
>
> Nkeywal:
>
> Would suggest putting hadoop4 and hadoop5 back in rotation.  Maybe then
> we'd get a response from Giri (smile).
>
> https://issues.apache.org/jira/browse/INFRA-6140 says that we should write
> the builds@ list to ask folks like Giri what the story is on the OOMEs,
> etc., since Apache INFRA does not run the hadoop* boxes.
>
> St.Ack
>
>
> On Fri, Apr 12, 2013 at 8:30 PM, Stack  wrote:
>
> > Yeah, can you give us some background Giri?
> > Thanks,
> > St.Ack
> >
> >
> > On Fri, Apr 12, 2013 at 10:45 AM, Nicolas Liochon  >wrote:
> >
> >> Done. Is this documented somewhere? I had no answer to my mail to the
> dev
> >> list when I asked why we were not using these machines.
> >>
> >>
> >>
> >> On Fri, Apr 12, 2013 at 6:55 PM, Giridharan Kesavan <
> >> gkesa...@hortonworks.com> wrote:
> >>
> >> > Nicolas,
> >> >
> >> > Please do not add more machins to hbase pool. There are other projects
> >> that
> >> > uses those machines like Pig, hive, hadoop, zookeeper, hcatalog and
> >> couple
> >> > of other projects as well.
> >> >
> >> > Please remove hadoop4 and hadoop5 from the pool.
> >> >
> >> > thanks,
> >> > Giridharan
> >> > Apache Jenkins Admin
> >> >
> >> >
> >> > >
> >> > > -- Forwarded message --
> >> > > From: Nicolas Liochon 
> >> > > Date: Fri, Apr 12, 2013 at 8:43 AM
> >> > > Subject: precommit machines
> >> > > To: dev@hbase.apache.org
> >> > >
> >> > >
> >> > > I've added hadoop4 and hadoop5 to the machines to be used for
> >> precommit
> >> > > tests/
> >> > > May be they were not in the list for a good reason. Remove them in
> >> > > https://builds.apache.org/job/PreCommit-HBASE-Build/configure if it
> >> > breaks
> >> > > too much.
> >> > >
> >> > > If it works well during a few days, I will add hadoop6+ as well.
> >> > >
> >> > > Cheers,
> >> > >
> >> > > Nicolas
> >> > >
> >> > >
> >> >
> >>
> >
> >
>

hadoop* machine: jenkins OOM, JDK 1.7 and config description

[Nicolas Liochon]

Hello,

We have a few needs related to the HBase project, mentioned in INFRA-6140.
 - We have very often (but not always) OutOfMemoryError in Jenkins. Here is
an example on the tail of this console output:
https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/5310/console.
We don't have this on the builds on ubuntu* machines.
 - We would like to use the JDK 1.7 64 bits on these machine.
 - We would like to know the configuration of these machines: we run the
tests in parallel; this would help us to maximize our usage and understand
possible random test failures.

Would someone be willing to help us on this?

Thanks,

Nicolas

Re: precommit machines

[Nicolas Liochon]

That's what I asked for in INFRA-6154. It doesn't mean I will get it.


On Mon, Apr 15, 2013 at 3:17 PM, Jean-Marc Spaggiari <
jean-m...@spaggiari.org> wrote:

> The last JDK 1.6 update is the 43...
>
> Is it possible to ask them to align all the servers with this latest
> version? Consistency might help to troubleshot the issues, if any.
>
> JM
>
> 2013/4/15 Nicolas Liochon 
>
> > You can enhance TestShowProperties to add the info you want to check.
> > Then
> >
> >
> https://builds.apache.org/job/HBase-TRUNK/4062/artifact/trunk/hbase-common/target/surefire-reports/org.apache.hadoop.hbase.util.TestShowProperties-output.txt
> >
> > I see that ubuntu3 has a newer java version than ubuntu6:
> >
> > java.version=1.6.0_32 vs. java.version=1.6.0_27
> >
> > As well
> > MaxFileDescriptor=4 (was 4), vs.  MaxFileDescriptor=4096 (was
> 4096)
> >
> > could explain some problems. I'm going to create an infra call for this.
> >
> > But for trunk, on the last 5 failures, 2 were on ubuntu3, so it's not the
> > only issue (and some issues are in HBase, not the env :-).
> >
> >
> > On Mon, Apr 15, 2013 at 5:29 AM, lars hofhansl  wrote:
> >
> > > On a related note, I have limited the 0.94 build to Ubuntu3 only,
> after I
> > > noticed that almost all transient failures were on Ubuntu{1|2|4|5}.
> > > Did that with build #950, not that a single failure since (hope I did
> not
> > > jinx it now!!).
> > >
> > > Might be worth figuring out how that machine differs from the other
> > > Ubunutu environments.
> > >
> > >
> > > Now I need to check the EC2 builds, which have been failing for a while
> > > now.
> > >
> > >
> > > -- Lars
> > >
> > >
> > >
> > > 
> > >  From: Stack 
> > > To: HBase Dev List 
> > > Cc: gkesa...@apache.org
> > > Sent: Sunday, April 14, 2013 6:38 PM
> > > Subject: Re: precommit machines
> > >
> > >
> > > Nkeywal:
> > >
> > > Would suggest putting hadoop4 and hadoop5 back in rotation.  Maybe then
> > > we'd get a response from Giri (smile).
> > >
> > > https://issues.apache.org/jira/browse/INFRA-6140 says that we should
> > write
> > > the builds@ list to ask folks like Giri what the story is on the
> OOMEs,
> > > etc., since Apache INFRA does not run the hadoop* boxes.
> > >
> > > St.Ack
> > >
> > >
> > > On Fri, Apr 12, 2013 at 8:30 PM, Stack  wrote:
> > >
> > > > Yeah, can you give us some background Giri?
> > > > Thanks,
> > > > St.Ack
> > > >
> > > >
> > > > On Fri, Apr 12, 2013 at 10:45 AM, Nicolas Liochon  > > >wrote:
> > > >
> > > >> Done. Is this documented somewhere? I had no answer to my mail to
> the
> > > dev
> > > >> list when I asked why we were not using these machines.
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Apr 12, 2013 at 6:55 PM, Giridharan Kesavan <
> > > >> gkesa...@hortonworks.com> wrote:
> > > >>
> > > >> > Nicolas,
> > > >> >
> > > >> > Please do not add more machins to hbase pool. There are other
> > projects
> > > >> that
> > > >> > uses those machines like Pig, hive, hadoop, zookeeper, hcatalog
> and
> > > >> couple
> > > >> > of other projects as well.
> > > >> >
> > > >> > Please remove hadoop4 and hadoop5 from the pool.
> > > >> >
> > > >> > thanks,
> > > >> > Giridharan
> > > >> > Apache Jenkins Admin
> > > >> >
> > > >> >
> > > >> > >
> > > >> > > -- Forwarded message --
> > > >> > > From: Nicolas Liochon 
> > > >> > > Date: Fri, Apr 12, 2013 at 8:43 AM
> > > >> > > Subject: precommit machines
> > > >> > > To: dev@hbase.apache.org
> > > >> > >
> > > >> > >
> > > >> > > I've added hadoop4 and hadoop5 to the machines to be used for
> > > >> precommit
> > > >> > > tests/
> > > >> > > May be they were not in the list for a good reason. Remove them
> in
> > > >> > > https://builds.apache.org/job/PreCommit-HBASE-Build/configureif
> > it
> > > >> > breaks
> > > >> > > too much.
> > > >> > >
> > > >> > > If it works well during a few days, I will add hadoop6+ as well.
> > > >> > >
> > > >> > > Cheers,
> > > >> > >
> > > >> > > Nicolas
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Re: hadoop* machine: jenkins OOM, JDK 1.7 and config description

[Nicolas Liochon]

 As far as I know, we
don't have the access rights to fix this ourselves (and it's a shared
environment, so it would be risky anyway).

All the points on hadoop* machines mentioned in the previous mail are still
open as well. Migrating our trunk build to JDK 1.7 would be nice for us.

Lastly, we've noticed some differences in config on ubuntu* machines. Are
the configs documented somewhere?

Cheers,

Nicolas (from HBase project).



On Mon, Apr 15, 2013 at 1:12 PM, Nicolas Liochon  wrote:

> Hello,
>
> We have a few needs related to the HBase project, mentioned in INFRA-6140.
>  - We have very often (but not always) OutOfMemoryError in Jenkins. Here
> is an example on the tail of this console output:
> https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/5310/console<https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/5247/console>.
>  We don't have this on the builds on ubuntu* machines.
>  - We would like to use the JDK 1.7 64 bits on these machine.
>  - We would like to know the configuration of these machines: we run the
> tests in parallel; this would help us to maximize our usage and understand
> possible random test failures.
>
> Would someone be willing to help us on this?
>
> Thanks,
>
> Nicolas
>