Re: DISCUSS: How can we have less branches?

[Andrew Purtell]

Branch 2 doesn't have a release yet never mind enough production experience for 
anyone other than bleeding edge to consider it. Some of us are rebasing 
production on 1.x and once there intend for mid to long term stable operations. 
So I think it is very likely we as a project will be maintaining branch-1 for a 
long time. Perhaps as long as 3 years. (The lifetime of 0.98). Maybe it ends 
with 1.4 but that's probably unlikely. 

> On Aug 8, 2017, at 8:45 PM, Phil Yang  wrote:
> 
> Another option is no 1.5/branch-1 any more. What new features we are going
> to have in HBase 1.5 (if it will be released)? Backporting from branch-2 to
> branch-1 is not easy, so maybe we will not have any big feature in branch-1
> after releasing 1.4. And if we release 1.5 after releasing 2.0, it may
> confuse users. So IMO we can focus on branch-2 for new features.
> 
> I think it will be good if we have fixed logic for EOL branches, for
> example(just an example, can discuss):
> 1:Release a final version and EOL 1.x after we think 1.x+1 is stable.
> 2:Cut off new 1.x+2 branch after 1.x+1 is stable, don't cut too early.
> 
> Then we will not need discussing each time for each branch EOL and we will
> have a fixed number of maintaining branches.
> 
> Thanks,
> Phil
> 
> 
> 2017-08-09 1:44 GMT+08:00 Andrew Purtell :
> 
>> Well you are not wrong that branching was premature, it turns out. But I'll
>> roll with it...
>> 
>> 
>> On Tue, Aug 8, 2017 at 10:43 AM, Zach York 
>> wrote:
>> 
 I made branch-1.4 a few weeks ago only.
>>> 
>>> Whoops, sorry for that! For some reason I thought I had seen it months
>> ago.
>>> 
>>> On Tue, Aug 8, 2017 at 10:40 AM, Andrew Purtell 
>>> wrote:
>>> 
 +1 from me on making 1.1 our LTS. Either 1.1 or 1.2 are candidates. I
>>> think
 1.1 has the edge because it lacks locking changes introduced into 1.2,
>>> just
 like 1.2 lacks locking changes introduced in 1.3 - the latter of which
>>> has
 had far reaching consequences, and the former not an insignificant
>> change
 either.
 
 
 
 On Tue, Aug 8, 2017 at 10:31 AM, Nick Dimiduk 
>>> wrote:
 
> On Tue, Aug 8, 2017 at 9:15 AM Mike Drob  wrote:
> 
>>> The discussion also brought up the notion of an LTS release line.
>>> I'm
> not
>>> sure how this jives with the more fequent minors, but would
>> require
> some
>>> branch that's so stable that an RM can effectively spin releases
 blind.
>> 
>> Seems to me like this branch would necessarily need to be very
>> backport-light? Only the top of the highest priority issues would
>> be
>> backportable to it, no?
> 
> 
> The LTS is as 1.1 is today -- bug fixes only. The difference here is
>>> we'd
> "formally" recognize the LTS designation somehow, perhaps with a
>>> symlink
> marker as we do for the "stable" designation.
> 
> On Tue, Aug 8, 2017 at 11:09 AM, Nick Dimiduk 
 wrote:
>> 
>>> Last time we DISCUSSed EOL of 1.1 was back in November. At that
 time, a
>>> litany of issues were raised re: 1.2. Have those concerns been
> addressed?
>>> It seems to me that making this one the last release is too
>> abrupt
>>> to
>> folks
>>> tracking Apache. Would be better to give some notice.
>>> 
>>> Had a nice hallway conversation a couple months back (at
>>> PhoenixCon,
 as
>> it
>>> happens; they feel the pain as well) about our branch situation.
>>> I'll
> let
>>> the others chime in with more details, but the gist as I recall
>> is
 that
>> we
>>> should be doing more frequent minor releases with fewer patch
 releases.
>>> This pushes stabilization efforts closer to master and also
>> imposes
> more
>>> strict stability requirements on big new features before they can
>>> be
>> merged
>>> off the feature branch.
>>> 
>>> The discussion also brought up the notion of an LTS release line.
>>> I'm
> not
>>> sure how this jives with the more fequent minors, but would
>> require
> some
>>> branch that's so stable that an RM can effectively spin releases
 blind.
>>> 
 On Tue, Aug 8, 2017 at 12:14 AM Stack  wrote:
 
 (This came up during dev meeting in Shenzhen) We are running
>> too
 many
 branches and/or when applying patches, we do not do a good job
>>> backporting
 to all active branches, especially fixes.
 
 We have master, branch-2, branch-1, branch-1.4, branch-1.3,
> branch-1.2,
>>> and
 branch-1.1 active currently. If a dirty bug fix, the applier is
>>> backporting
 to 7 branches. It takes a while applying to all especially if
>> the
>>> backport
 doesn't go in clean. I suppose the RM could monitor 

Re: DISCUSS: How can we have less branches?

[Phil Yang]

Another option is no 1.5/branch-1 any more. What new features we are going
to have in HBase 1.5 (if it will be released)? Backporting from branch-2 to
branch-1 is not easy, so maybe we will not have any big feature in branch-1
after releasing 1.4. And if we release 1.5 after releasing 2.0, it may
confuse users. So IMO we can focus on branch-2 for new features.

I think it will be good if we have fixed logic for EOL branches, for
example(just an example, can discuss):
1:Release a final version and EOL 1.x after we think 1.x+1 is stable.
2:Cut off new 1.x+2 branch after 1.x+1 is stable, don't cut too early.

Then we will not need discussing each time for each branch EOL and we will
have a fixed number of maintaining branches.

Thanks,
Phil


2017-08-09 1:44 GMT+08:00 Andrew Purtell :

> Well you are not wrong that branching was premature, it turns out. But I'll
> roll with it...
>
>
> On Tue, Aug 8, 2017 at 10:43 AM, Zach York 
> wrote:
>
> > > I made branch-1.4 a few weeks ago only.
> >
> > Whoops, sorry for that! For some reason I thought I had seen it months
> ago.
> >
> > On Tue, Aug 8, 2017 at 10:40 AM, Andrew Purtell 
> > wrote:
> >
> > > +1 from me on making 1.1 our LTS. Either 1.1 or 1.2 are candidates. I
> > think
> > > 1.1 has the edge because it lacks locking changes introduced into 1.2,
> > just
> > > like 1.2 lacks locking changes introduced in 1.3 - the latter of which
> > has
> > > had far reaching consequences, and the former not an insignificant
> change
> > > either.
> > >
> > >
> > >
> > > On Tue, Aug 8, 2017 at 10:31 AM, Nick Dimiduk 
> > wrote:
> > >
> > > > On Tue, Aug 8, 2017 at 9:15 AM Mike Drob  wrote:
> > > >
> > > > > > The discussion also brought up the notion of an LTS release line.
> > I'm
> > > > not
> > > > > > sure how this jives with the more fequent minors, but would
> require
> > > > some
> > > > > > branch that's so stable that an RM can effectively spin releases
> > > blind.
> > > > >
> > > > > Seems to me like this branch would necessarily need to be very
> > > > > backport-light? Only the top of the highest priority issues would
> be
> > > > > backportable to it, no?
> > > >
> > > >
> > > > The LTS is as 1.1 is today -- bug fixes only. The difference here is
> > we'd
> > > > "formally" recognize the LTS designation somehow, perhaps with a
> > symlink
> > > > marker as we do for the "stable" designation.
> > > >
> > > > On Tue, Aug 8, 2017 at 11:09 AM, Nick Dimiduk 
> > > wrote:
> > > > >
> > > > > > Last time we DISCUSSed EOL of 1.1 was back in November. At that
> > > time, a
> > > > > > litany of issues were raised re: 1.2. Have those concerns been
> > > > addressed?
> > > > > > It seems to me that making this one the last release is too
> abrupt
> > to
> > > > > folks
> > > > > > tracking Apache. Would be better to give some notice.
> > > > > >
> > > > > > Had a nice hallway conversation a couple months back (at
> > PhoenixCon,
> > > as
> > > > > it
> > > > > > happens; they feel the pain as well) about our branch situation.
> > I'll
> > > > let
> > > > > > the others chime in with more details, but the gist as I recall
> is
> > > that
> > > > > we
> > > > > > should be doing more frequent minor releases with fewer patch
> > > releases.
> > > > > > This pushes stabilization efforts closer to master and also
> imposes
> > > > more
> > > > > > strict stability requirements on big new features before they can
> > be
> > > > > merged
> > > > > > off the feature branch.
> > > > > >
> > > > > > The discussion also brought up the notion of an LTS release line.
> > I'm
> > > > not
> > > > > > sure how this jives with the more fequent minors, but would
> require
> > > > some
> > > > > > branch that's so stable that an RM can effectively spin releases
> > > blind.
> > > > > >
> > > > > > On Tue, Aug 8, 2017 at 12:14 AM Stack  wrote:
> > > > > >
> > > > > > > (This came up during dev meeting in Shenzhen) We are running
> too
> > > many
> > > > > > > branches and/or when applying patches, we do not do a good job
> > > > > > backporting
> > > > > > > to all active branches, especially fixes.
> > > > > > >
> > > > > > > We have master, branch-2, branch-1, branch-1.4, branch-1.3,
> > > > branch-1.2,
> > > > > > and
> > > > > > > branch-1.1 active currently. If a dirty bug fix, the applier is
> > > > > > backporting
> > > > > > > to 7 branches. It takes a while applying to all especially if
> the
> > > > > > backport
> > > > > > > doesn't go in clean. I suppose the RM could monitor all
> upstream
> > of
> > > > > them
> > > > > > > and then pull wanted patches back (we could do that too) but
> > seems
> > > > > like a
> > > > > > > burden on an RMer.
> > > > > > >
> > > > > > > We should EOL a few?
> > > > > > >
> > > > > > > Nick is on branch-1.1 release at the moment. Perhaps this could
> > be
> > > > the
> > > > > > last
> > > > > > 

[jira] [Created] (HBASE-18543) master.TestMasterFailover fails on master

[Umesh Agashe (JIRA)]

Umesh Agashe created HBASE-18543:


 Summary: master.TestMasterFailover fails on master
 Key: HBASE-18543
 URL: https://issues.apache.org/jira/browse/HBASE-18543
 Project: HBase
  Issue Type: Bug
  Components: amv2
Affects Versions: 2.0.0
Reporter: Umesh Agashe
Assignee: Umesh Agashe
 Fix For: 2.0.0






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (HBASE-18020) Update API Compliance Checker to Incorporate Improvements Done in Hadoop

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell reopened HBASE-18020:


Does there need to be a documentation update too. This didn't work for me. On 
branch-1.4 

./dev-support/checkcompatibility.py --annotation 
org.apache.hadoop.hbase.classification.InterfaceAudience.Public --annotation 
org.apache.hadoop.hbase.classification.InterfaceAudience.LimitedPrivate  
--include-file "hbase-*" rel/1.3.1 branch-1.4

INFO:root:Source revision: rel/1.3.1
INFO:root:Destination revision: branch-1.4
INFO:root:Applying JAR filename include filter: hbase-*
INFO:root:Filtering classes using 2 annotation(s):
INFO:root:  org.apache.hadoop.hbase.classification.InterfaceAudience.Public
INFO:root:  
org.apache.hadoop.hbase.classification.InterfaceAudience.LimitedPrivate
INFO:root:Downloading Java ACC...
INFO:root:Removing scratch dir /data/src/hbase/target/compat-check 
INFO:root:Creating empty scratch dir /data/src/hbase/target/compat-check 
INFO:root:Checking out 1b8ca49af872b63c7b1fbc20a6fdbe71516dd04c in 
/data/src/hbase/target/compat-check/src
INFO:root:Checking out e9e16b59420c5d9a47b9b014b3bb3cb3421b1de9 in 
/data/src/hbase/target/compat-check/dst
INFO:root:Building in /data/src/hbase/target/compat-check/src 

...

INFO:root:Will check compatibility between original jars:

/data/src/hbase/target/compat-check/src/hbase-archetypes/hbase-client-project/target/hbase-client-project-1.3.1.jar

...

and new jars:

/data/src/hbase/target/compat-check/dst/hbase-archetypes/hbase-shaded-client-project/target/hbase-shaded-client-project-1.4.0-SNAPSHOT.jar

...

Traceback (most recent call last):
  File "./dev-support/checkcompatibility.py", line 514, in 
main()
  File "./dev-support/checkcompatibility.py", line 508, in main
dst_jars, args.annotations, skip_annotations)
  File "./dev-support/checkcompatibility.py", line 283, in run_java_acc
"-l", get_repo_name(),
  File "./dev-support/checkcompatibility.py", line 126, in get_repo_name
cwd=get_repo_dir()).strip()
  File "./dev-support/checkcompatibility.py", line 71, in check_output
raise error
subprocess.CalledProcessError: Command '['git', 'config', '--get', 
'remote.origin.url']' returned non-zero exit status 1

[apurtell@ip-172-31-47-35 hbase]$ git --version
git version 1.7.1


> Update API Compliance Checker to Incorporate Improvements Done in Hadoop
> 
>
> Key: HBASE-18020
> URL: https://issues.apache.org/jira/browse/HBASE-18020
> Project: HBase
>  Issue Type: Improvement
>  Components: API, community
>Reporter: Alex Leblang
>Assignee: Alex Leblang
> Fix For: 2.0.0, 3.0.0, 1.4.0, 1.3.2, 1.5.0, 1.2.7, 1.1.12
>
> Attachments: HBASE-18020.0.patch, HBASE-18020.branch-1.2.001.patch, 
> HBASE-18020.branch-1.2.002.patch, HBASE-18020.branch-1.2.003.patch, 
> HBASE-18020.branch-1.2.004.patch
>
>
> Recently the Hadoop community has made a number of improvements in their api 
> compliance checker based on feedback from the hbase and kudu community. We 
> should adopt these changes ourselves.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18542) [HLC] Performance microbenchmarks

[Appy (JIRA)]

Appy created HBASE-18542:


 Summary: [HLC] Performance microbenchmarks
 Key: HBASE-18542
 URL: https://issues.apache.org/jira/browse/HBASE-18542
 Project: HBase
  Issue Type: Sub-task
Reporter: Appy


Need tests to benchmark performance of Clock#now() and update() functions (for 
all types of clocks).
If update() is too costly, we can do optimizations in 
ExecuteProceduresRemoteCall#call() and other places where we call update() in 
loop. Instead, it might be faster to calculate max timestamp in loop and call 
update() just once.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18541) [C++] Segfaults from JNI

[Enis Soztutar (JIRA)]

Enis Soztutar created HBASE-18541:
-

 Summary: [C++] Segfaults from JNI
 Key: HBASE-18541
 URL: https://issues.apache.org/jira/browse/HBASE-18541
 Project: HBase
  Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Ted Yu


retry-test and multi-retry-test fails flakily when run with 
{code}
buck test --all --no-results-cache
{code}
or when run in a loop:
{code}
for i in `seq 1 10`; do buck test --no-results-cache core:retry-test || break 
1; done
{code}

The problem seems to be within the JNI internals and usually happens at the 
create table method call. I was not able to inspect much, but the comments in 
our mini-cluster indicate that we may need to use global references instead of 
local ones. I suspect the problem happens when there is a GC run for the test 
since the failure happens usually after some time (but almost always in create 
table method). 

[~ted_yu] do you mind taking a look at this. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HBASE-18530) precommit should check validity of changes to nightly jenkinsfile

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey resolved HBASE-18530.
-
Resolution: Duplicate

hopefully I won't file this ticket again next week. ;p

> precommit should check validity of changes to nightly jenkinsfile
> -
>
> Key: HBASE-18530
> URL: https://issues.apache.org/jira/browse/HBASE-18530
> Project: HBase
>  Issue Type: New Feature
>  Components: community, test
>Reporter: Sean Busbey
>
> It'd be nice if our precommit job could check changes to the nightly 
> jenkinsfile. Even if it's just a simple syntax check.
> I believe there's a plugin for jenkins that you can curl with a proposed 
> Jenkinsfile, but someone will need to chase down specifics.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18540) Ad-hoc job for checking if a test is flaky

[Sean Busbey (JIRA)]

Sean Busbey created HBASE-18540:
---

 Summary: Ad-hoc job for checking if a test is flaky
 Key: HBASE-18540
 URL: https://issues.apache.org/jira/browse/HBASE-18540
 Project: HBase
  Issue Type: New Feature
  Components: test
Reporter: Sean Busbey


It would be great if we had an ad-hoc job that could run a given test a bunch 
of times and tell me if it is flaky.

I'm trying to evaluate the state of our new nightly jobs, and I'm not sure if 
hte failures I'm seeing are a) things I need to get fixed soon, b) things that 
are flaky but previously unrecognized, c) things that are flaky but missed 
because our current flaky list is only based on the behavior of the master 
branch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: DISCUSS: How can we have less branches?

[Andrew Purtell]

Well you are not wrong that branching was premature, it turns out. But I'll
roll with it...


On Tue, Aug 8, 2017 at 10:43 AM, Zach York 
wrote:

> > I made branch-1.4 a few weeks ago only.
>
> Whoops, sorry for that! For some reason I thought I had seen it months ago.
>
> On Tue, Aug 8, 2017 at 10:40 AM, Andrew Purtell 
> wrote:
>
> > +1 from me on making 1.1 our LTS. Either 1.1 or 1.2 are candidates. I
> think
> > 1.1 has the edge because it lacks locking changes introduced into 1.2,
> just
> > like 1.2 lacks locking changes introduced in 1.3 - the latter of which
> has
> > had far reaching consequences, and the former not an insignificant change
> > either.
> >
> >
> >
> > On Tue, Aug 8, 2017 at 10:31 AM, Nick Dimiduk 
> wrote:
> >
> > > On Tue, Aug 8, 2017 at 9:15 AM Mike Drob  wrote:
> > >
> > > > > The discussion also brought up the notion of an LTS release line.
> I'm
> > > not
> > > > > sure how this jives with the more fequent minors, but would require
> > > some
> > > > > branch that's so stable that an RM can effectively spin releases
> > blind.
> > > >
> > > > Seems to me like this branch would necessarily need to be very
> > > > backport-light? Only the top of the highest priority issues would be
> > > > backportable to it, no?
> > >
> > >
> > > The LTS is as 1.1 is today -- bug fixes only. The difference here is
> we'd
> > > "formally" recognize the LTS designation somehow, perhaps with a
> symlink
> > > marker as we do for the "stable" designation.
> > >
> > > On Tue, Aug 8, 2017 at 11:09 AM, Nick Dimiduk 
> > wrote:
> > > >
> > > > > Last time we DISCUSSed EOL of 1.1 was back in November. At that
> > time, a
> > > > > litany of issues were raised re: 1.2. Have those concerns been
> > > addressed?
> > > > > It seems to me that making this one the last release is too abrupt
> to
> > > > folks
> > > > > tracking Apache. Would be better to give some notice.
> > > > >
> > > > > Had a nice hallway conversation a couple months back (at
> PhoenixCon,
> > as
> > > > it
> > > > > happens; they feel the pain as well) about our branch situation.
> I'll
> > > let
> > > > > the others chime in with more details, but the gist as I recall is
> > that
> > > > we
> > > > > should be doing more frequent minor releases with fewer patch
> > releases.
> > > > > This pushes stabilization efforts closer to master and also imposes
> > > more
> > > > > strict stability requirements on big new features before they can
> be
> > > > merged
> > > > > off the feature branch.
> > > > >
> > > > > The discussion also brought up the notion of an LTS release line.
> I'm
> > > not
> > > > > sure how this jives with the more fequent minors, but would require
> > > some
> > > > > branch that's so stable that an RM can effectively spin releases
> > blind.
> > > > >
> > > > > On Tue, Aug 8, 2017 at 12:14 AM Stack  wrote:
> > > > >
> > > > > > (This came up during dev meeting in Shenzhen) We are running too
> > many
> > > > > > branches and/or when applying patches, we do not do a good job
> > > > > backporting
> > > > > > to all active branches, especially fixes.
> > > > > >
> > > > > > We have master, branch-2, branch-1, branch-1.4, branch-1.3,
> > > branch-1.2,
> > > > > and
> > > > > > branch-1.1 active currently. If a dirty bug fix, the applier is
> > > > > backporting
> > > > > > to 7 branches. It takes a while applying to all especially if the
> > > > > backport
> > > > > > doesn't go in clean. I suppose the RM could monitor all upstream
> of
> > > > them
> > > > > > and then pull wanted patches back (we could do that too) but
> seems
> > > > like a
> > > > > > burden on an RMer.
> > > > > >
> > > > > > We should EOL a few?
> > > > > >
> > > > > > Nick is on branch-1.1 release at the moment. Perhaps this could
> be
> > > the
> > > > > last
> > > > > > off branch-1.1.
> > > > > >
> > > > > > 1.2 hosts our current stable release though 1.3 is out. How about
> > we
> > > > cut
> > > > > a
> > > > > > release off 1.3, call it stable, and then EOL 1.2 after another
> > > release
> > > > > or
> > > > > > so?
> > > > > >
> > > > > > What you all think?
> > > > > >
> > > > > > Thanks,
> > > > > > S
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> >- A23, Crosstalk
> >
>



-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

Re: DISCUSS: How can we have less branches?

[Zach York]

> I made branch-1.4 a few weeks ago only.

Whoops, sorry for that! For some reason I thought I had seen it months ago.

On Tue, Aug 8, 2017 at 10:40 AM, Andrew Purtell  wrote:

> +1 from me on making 1.1 our LTS. Either 1.1 or 1.2 are candidates. I think
> 1.1 has the edge because it lacks locking changes introduced into 1.2, just
> like 1.2 lacks locking changes introduced in 1.3 - the latter of which has
> had far reaching consequences, and the former not an insignificant change
> either.
>
>
>
> On Tue, Aug 8, 2017 at 10:31 AM, Nick Dimiduk  wrote:
>
> > On Tue, Aug 8, 2017 at 9:15 AM Mike Drob  wrote:
> >
> > > > The discussion also brought up the notion of an LTS release line. I'm
> > not
> > > > sure how this jives with the more fequent minors, but would require
> > some
> > > > branch that's so stable that an RM can effectively spin releases
> blind.
> > >
> > > Seems to me like this branch would necessarily need to be very
> > > backport-light? Only the top of the highest priority issues would be
> > > backportable to it, no?
> >
> >
> > The LTS is as 1.1 is today -- bug fixes only. The difference here is we'd
> > "formally" recognize the LTS designation somehow, perhaps with a symlink
> > marker as we do for the "stable" designation.
> >
> > On Tue, Aug 8, 2017 at 11:09 AM, Nick Dimiduk 
> wrote:
> > >
> > > > Last time we DISCUSSed EOL of 1.1 was back in November. At that
> time, a
> > > > litany of issues were raised re: 1.2. Have those concerns been
> > addressed?
> > > > It seems to me that making this one the last release is too abrupt to
> > > folks
> > > > tracking Apache. Would be better to give some notice.
> > > >
> > > > Had a nice hallway conversation a couple months back (at PhoenixCon,
> as
> > > it
> > > > happens; they feel the pain as well) about our branch situation. I'll
> > let
> > > > the others chime in with more details, but the gist as I recall is
> that
> > > we
> > > > should be doing more frequent minor releases with fewer patch
> releases.
> > > > This pushes stabilization efforts closer to master and also imposes
> > more
> > > > strict stability requirements on big new features before they can be
> > > merged
> > > > off the feature branch.
> > > >
> > > > The discussion also brought up the notion of an LTS release line. I'm
> > not
> > > > sure how this jives with the more fequent minors, but would require
> > some
> > > > branch that's so stable that an RM can effectively spin releases
> blind.
> > > >
> > > > On Tue, Aug 8, 2017 at 12:14 AM Stack  wrote:
> > > >
> > > > > (This came up during dev meeting in Shenzhen) We are running too
> many
> > > > > branches and/or when applying patches, we do not do a good job
> > > > backporting
> > > > > to all active branches, especially fixes.
> > > > >
> > > > > We have master, branch-2, branch-1, branch-1.4, branch-1.3,
> > branch-1.2,
> > > > and
> > > > > branch-1.1 active currently. If a dirty bug fix, the applier is
> > > > backporting
> > > > > to 7 branches. It takes a while applying to all especially if the
> > > > backport
> > > > > doesn't go in clean. I suppose the RM could monitor all upstream of
> > > them
> > > > > and then pull wanted patches back (we could do that too) but seems
> > > like a
> > > > > burden on an RMer.
> > > > >
> > > > > We should EOL a few?
> > > > >
> > > > > Nick is on branch-1.1 release at the moment. Perhaps this could be
> > the
> > > > last
> > > > > off branch-1.1.
> > > > >
> > > > > 1.2 hosts our current stable release though 1.3 is out. How about
> we
> > > cut
> > > > a
> > > > > release off 1.3, call it stable, and then EOL 1.2 after another
> > release
> > > > or
> > > > > so?
> > > > >
> > > > > What you all think?
> > > > >
> > > > > Thanks,
> > > > > S
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>

Re: DISCUSS: How can we have less branches?

[Andrew Purtell]

+1 from me on making 1.1 our LTS. Either 1.1 or 1.2 are candidates. I think
1.1 has the edge because it lacks locking changes introduced into 1.2, just
like 1.2 lacks locking changes introduced in 1.3 - the latter of which has
had far reaching consequences, and the former not an insignificant change
either.



On Tue, Aug 8, 2017 at 10:31 AM, Nick Dimiduk  wrote:

> On Tue, Aug 8, 2017 at 9:15 AM Mike Drob  wrote:
>
> > > The discussion also brought up the notion of an LTS release line. I'm
> not
> > > sure how this jives with the more fequent minors, but would require
> some
> > > branch that's so stable that an RM can effectively spin releases blind.
> >
> > Seems to me like this branch would necessarily need to be very
> > backport-light? Only the top of the highest priority issues would be
> > backportable to it, no?
>
>
> The LTS is as 1.1 is today -- bug fixes only. The difference here is we'd
> "formally" recognize the LTS designation somehow, perhaps with a symlink
> marker as we do for the "stable" designation.
>
> On Tue, Aug 8, 2017 at 11:09 AM, Nick Dimiduk  wrote:
> >
> > > Last time we DISCUSSed EOL of 1.1 was back in November. At that time, a
> > > litany of issues were raised re: 1.2. Have those concerns been
> addressed?
> > > It seems to me that making this one the last release is too abrupt to
> > folks
> > > tracking Apache. Would be better to give some notice.
> > >
> > > Had a nice hallway conversation a couple months back (at PhoenixCon, as
> > it
> > > happens; they feel the pain as well) about our branch situation. I'll
> let
> > > the others chime in with more details, but the gist as I recall is that
> > we
> > > should be doing more frequent minor releases with fewer patch releases.
> > > This pushes stabilization efforts closer to master and also imposes
> more
> > > strict stability requirements on big new features before they can be
> > merged
> > > off the feature branch.
> > >
> > > The discussion also brought up the notion of an LTS release line. I'm
> not
> > > sure how this jives with the more fequent minors, but would require
> some
> > > branch that's so stable that an RM can effectively spin releases blind.
> > >
> > > On Tue, Aug 8, 2017 at 12:14 AM Stack  wrote:
> > >
> > > > (This came up during dev meeting in Shenzhen) We are running too many
> > > > branches and/or when applying patches, we do not do a good job
> > > backporting
> > > > to all active branches, especially fixes.
> > > >
> > > > We have master, branch-2, branch-1, branch-1.4, branch-1.3,
> branch-1.2,
> > > and
> > > > branch-1.1 active currently. If a dirty bug fix, the applier is
> > > backporting
> > > > to 7 branches. It takes a while applying to all especially if the
> > > backport
> > > > doesn't go in clean. I suppose the RM could monitor all upstream of
> > them
> > > > and then pull wanted patches back (we could do that too) but seems
> > like a
> > > > burden on an RMer.
> > > >
> > > > We should EOL a few?
> > > >
> > > > Nick is on branch-1.1 release at the moment. Perhaps this could be
> the
> > > last
> > > > off branch-1.1.
> > > >
> > > > 1.2 hosts our current stable release though 1.3 is out. How about we
> > cut
> > > a
> > > > release off 1.3, call it stable, and then EOL 1.2 after another
> release
> > > or
> > > > so?
> > > >
> > > > What you all think?
> > > >
> > > > Thanks,
> > > > S
> > > >
> > >
> >
>



-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

Re: DISCUSS: How can we have less branches?

[Andrew Purtell]

I made branch-1.4 a few weeks ago only. I did it because I had hoped to RC
quite quickly. Then the reality of compatibility breaks and failing unit
tests hit, as well as insufficient time for me to make progress on my
must-have for the release, which is a backport to RSGroups. I will get to
all of this as soon as I can given work commitments. Maybe we can have a RC
of 1.4.0 by the end of next month.

I don't disagree with you, though. If there were more active scrutiny of
branch-1 then we would have found the compat breaks and addressed the test
problems near when they happened and my work as RM of 1.4 will have been
lessened considerably.

I would like to see us move to a model where we have three main code lines:

master: 3.x
branch-2: 2.x
branch-1: 1.x

and we migrate our RM model away from RMs curating patch branches to a
model where there are three people actively monitoring the state of the
above three branches, and, when considering releasing, we prefer to cut new
minors from these three branches (modulo master) over patch release work.
There will be times when patch releases are necessary. I don't think a
dedicated RM for a patch branch is necessary, though. If there is a
critical bug fix for branch-1, the "RM" for branch-1 can spin patch
releases of all 1.x code lines we have decided to keep alive and run votes
on all of them. This is incremental additional effort after getting set up
to make releases in the first place.

There are two huge benefits:

1. We split precious RM resources in fewer ways.

2. We have fewer code lines receiving the majority of commits, so
committers do less work.


On Tue, Aug 8, 2017 at 10:19 AM, Zach York 
wrote:

> One problem that I see is that it appears we are cutting branches extremely
> early.
> I'm going to pick on branch-1.4 for a moment. This branch has existed (as
> far as I can remember) for at least 3-6 months, yet we still have not had a
> 1.4.0 release. I understand that releases can take time, but this adds pain
> for developers (If I'm developing a new feature or bug fix, do I put it on
> branch-1 and it will be pulled into branch-1.4 or do I need to explicitly
> put it on branch-1.4?). It seems that this branch should be the same as
> branch-1 at the moment.
>
> So my suggestion would be use branch-1 (or branch-2 now) for all
> development and only create a branch when the first release of a line is
> imminent.
>
> Perhaps I am missing some context, but that is my 2 cents.
>
> Thanks,
> Zach
>
> On Tue, Aug 8, 2017 at 9:14 AM, Mike Drob  wrote:
>
> > > The discussion also brought up the notion of an LTS release line. I'm
> not
> > > sure how this jives with the more fequent minors, but would require
> some
> > > branch that's so stable that an RM can effectively spin releases blind.
> >
> > Seems to me like this branch would necessarily need to be very
> > backport-light? Only the top of the highest priority issues would be
> > backportable to it, no?
> >
> > On Tue, Aug 8, 2017 at 11:09 AM, Nick Dimiduk 
> wrote:
> >
> > > Last time we DISCUSSed EOL of 1.1 was back in November. At that time, a
> > > litany of issues were raised re: 1.2. Have those concerns been
> addressed?
> > > It seems to me that making this one the last release is too abrupt to
> > folks
> > > tracking Apache. Would be better to give some notice.
> > >
> > > Had a nice hallway conversation a couple months back (at PhoenixCon, as
> > it
> > > happens; they feel the pain as well) about our branch situation. I'll
> let
> > > the others chime in with more details, but the gist as I recall is that
> > we
> > > should be doing more frequent minor releases with fewer patch releases.
> > > This pushes stabilization efforts closer to master and also imposes
> more
> > > strict stability requirements on big new features before they can be
> > merged
> > > off the feature branch.
> > >
> > > The discussion also brought up the notion of an LTS release line. I'm
> not
> > > sure how this jives with the more fequent minors, but would require
> some
> > > branch that's so stable that an RM can effectively spin releases blind.
> > >
> > > On Tue, Aug 8, 2017 at 12:14 AM Stack  wrote:
> > >
> > > > (This came up during dev meeting in Shenzhen) We are running too many
> > > > branches and/or when applying patches, we do not do a good job
> > > backporting
> > > > to all active branches, especially fixes.
> > > >
> > > > We have master, branch-2, branch-1, branch-1.4, branch-1.3,
> branch-1.2,
> > > and
> > > > branch-1.1 active currently. If a dirty bug fix, the applier is
> > > backporting
> > > > to 7 branches. It takes a while applying to all especially if the
> > > backport
> > > > doesn't go in clean. I suppose the RM could monitor all upstream of
> > them
> > > > and then pull wanted patches back (we could do that too) but seems
> > like a
> > > > burden on an RMer.
> > > >
> > > > We should EOL a few?
> > 

Re: DISCUSS: How can we have less branches?

[Nick Dimiduk]

On Tue, Aug 8, 2017 at 9:15 AM Mike Drob  wrote:

> > The discussion also brought up the notion of an LTS release line. I'm not
> > sure how this jives with the more fequent minors, but would require some
> > branch that's so stable that an RM can effectively spin releases blind.
>
> Seems to me like this branch would necessarily need to be very
> backport-light? Only the top of the highest priority issues would be
> backportable to it, no?


The LTS is as 1.1 is today -- bug fixes only. The difference here is we'd
"formally" recognize the LTS designation somehow, perhaps with a symlink
marker as we do for the "stable" designation.

On Tue, Aug 8, 2017 at 11:09 AM, Nick Dimiduk  wrote:
>
> > Last time we DISCUSSed EOL of 1.1 was back in November. At that time, a
> > litany of issues were raised re: 1.2. Have those concerns been addressed?
> > It seems to me that making this one the last release is too abrupt to
> folks
> > tracking Apache. Would be better to give some notice.
> >
> > Had a nice hallway conversation a couple months back (at PhoenixCon, as
> it
> > happens; they feel the pain as well) about our branch situation. I'll let
> > the others chime in with more details, but the gist as I recall is that
> we
> > should be doing more frequent minor releases with fewer patch releases.
> > This pushes stabilization efforts closer to master and also imposes more
> > strict stability requirements on big new features before they can be
> merged
> > off the feature branch.
> >
> > The discussion also brought up the notion of an LTS release line. I'm not
> > sure how this jives with the more fequent minors, but would require some
> > branch that's so stable that an RM can effectively spin releases blind.
> >
> > On Tue, Aug 8, 2017 at 12:14 AM Stack  wrote:
> >
> > > (This came up during dev meeting in Shenzhen) We are running too many
> > > branches and/or when applying patches, we do not do a good job
> > backporting
> > > to all active branches, especially fixes.
> > >
> > > We have master, branch-2, branch-1, branch-1.4, branch-1.3, branch-1.2,
> > and
> > > branch-1.1 active currently. If a dirty bug fix, the applier is
> > backporting
> > > to 7 branches. It takes a while applying to all especially if the
> > backport
> > > doesn't go in clean. I suppose the RM could monitor all upstream of
> them
> > > and then pull wanted patches back (we could do that too) but seems
> like a
> > > burden on an RMer.
> > >
> > > We should EOL a few?
> > >
> > > Nick is on branch-1.1 release at the moment. Perhaps this could be the
> > last
> > > off branch-1.1.
> > >
> > > 1.2 hosts our current stable release though 1.3 is out. How about we
> cut
> > a
> > > release off 1.3, call it stable, and then EOL 1.2 after another release
> > or
> > > so?
> > >
> > > What you all think?
> > >
> > > Thanks,
> > > S
> > >
> >
>

[jira] [Created] (HBASE-18539) Remove usages of masterSystemTime

[Amit Patel (JIRA)]

Amit Patel created HBASE-18539:
--

 Summary: Remove usages of masterSystemTime
 Key: HBASE-18539
 URL: https://issues.apache.org/jira/browse/HBASE-18539
 Project: HBase
  Issue Type: Sub-task
Reporter: Amit Patel
Assignee: Amit Patel
Priority: Trivial


After removing the use of the master's system timestamp for meta updates in 
[HBASE-18328|https://issues.apache.org/jira/browse/HBASE-18328] and now that we 
pass the timestamps during region open/close in 
[HBASE-18395|https://issues.apache.org/jira/browse/HBASE-18395], we can also 
remove all legacy usages of masterSystemTime.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: DISCUSS: How can we have less branches?

[Zach York]

One problem that I see is that it appears we are cutting branches extremely
early.
I'm going to pick on branch-1.4 for a moment. This branch has existed (as
far as I can remember) for at least 3-6 months, yet we still have not had a
1.4.0 release. I understand that releases can take time, but this adds pain
for developers (If I'm developing a new feature or bug fix, do I put it on
branch-1 and it will be pulled into branch-1.4 or do I need to explicitly
put it on branch-1.4?). It seems that this branch should be the same as
branch-1 at the moment.

So my suggestion would be use branch-1 (or branch-2 now) for all
development and only create a branch when the first release of a line is
imminent.

Perhaps I am missing some context, but that is my 2 cents.

Thanks,
Zach

On Tue, Aug 8, 2017 at 9:14 AM, Mike Drob  wrote:

> > The discussion also brought up the notion of an LTS release line. I'm not
> > sure how this jives with the more fequent minors, but would require some
> > branch that's so stable that an RM can effectively spin releases blind.
>
> Seems to me like this branch would necessarily need to be very
> backport-light? Only the top of the highest priority issues would be
> backportable to it, no?
>
> On Tue, Aug 8, 2017 at 11:09 AM, Nick Dimiduk  wrote:
>
> > Last time we DISCUSSed EOL of 1.1 was back in November. At that time, a
> > litany of issues were raised re: 1.2. Have those concerns been addressed?
> > It seems to me that making this one the last release is too abrupt to
> folks
> > tracking Apache. Would be better to give some notice.
> >
> > Had a nice hallway conversation a couple months back (at PhoenixCon, as
> it
> > happens; they feel the pain as well) about our branch situation. I'll let
> > the others chime in with more details, but the gist as I recall is that
> we
> > should be doing more frequent minor releases with fewer patch releases.
> > This pushes stabilization efforts closer to master and also imposes more
> > strict stability requirements on big new features before they can be
> merged
> > off the feature branch.
> >
> > The discussion also brought up the notion of an LTS release line. I'm not
> > sure how this jives with the more fequent minors, but would require some
> > branch that's so stable that an RM can effectively spin releases blind.
> >
> > On Tue, Aug 8, 2017 at 12:14 AM Stack  wrote:
> >
> > > (This came up during dev meeting in Shenzhen) We are running too many
> > > branches and/or when applying patches, we do not do a good job
> > backporting
> > > to all active branches, especially fixes.
> > >
> > > We have master, branch-2, branch-1, branch-1.4, branch-1.3, branch-1.2,
> > and
> > > branch-1.1 active currently. If a dirty bug fix, the applier is
> > backporting
> > > to 7 branches. It takes a while applying to all especially if the
> > backport
> > > doesn't go in clean. I suppose the RM could monitor all upstream of
> them
> > > and then pull wanted patches back (we could do that too) but seems
> like a
> > > burden on an RMer.
> > >
> > > We should EOL a few?
> > >
> > > Nick is on branch-1.1 release at the moment. Perhaps this could be the
> > last
> > > off branch-1.1.
> > >
> > > 1.2 hosts our current stable release though 1.3 is out. How about we
> cut
> > a
> > > release off 1.3, call it stable, and then EOL 1.2 after another release
> > or
> > > so?
> > >
> > > What you all think?
> > >
> > > Thanks,
> > > S
> > >
> >
>

Re: DISCUSS: How can we have less branches?

[Mike Drob]

> The discussion also brought up the notion of an LTS release line. I'm not
> sure how this jives with the more fequent minors, but would require some
> branch that's so stable that an RM can effectively spin releases blind.

Seems to me like this branch would necessarily need to be very
backport-light? Only the top of the highest priority issues would be
backportable to it, no?

On Tue, Aug 8, 2017 at 11:09 AM, Nick Dimiduk  wrote:

> Last time we DISCUSSed EOL of 1.1 was back in November. At that time, a
> litany of issues were raised re: 1.2. Have those concerns been addressed?
> It seems to me that making this one the last release is too abrupt to folks
> tracking Apache. Would be better to give some notice.
>
> Had a nice hallway conversation a couple months back (at PhoenixCon, as it
> happens; they feel the pain as well) about our branch situation. I'll let
> the others chime in with more details, but the gist as I recall is that we
> should be doing more frequent minor releases with fewer patch releases.
> This pushes stabilization efforts closer to master and also imposes more
> strict stability requirements on big new features before they can be merged
> off the feature branch.
>
> The discussion also brought up the notion of an LTS release line. I'm not
> sure how this jives with the more fequent minors, but would require some
> branch that's so stable that an RM can effectively spin releases blind.
>
> On Tue, Aug 8, 2017 at 12:14 AM Stack  wrote:
>
> > (This came up during dev meeting in Shenzhen) We are running too many
> > branches and/or when applying patches, we do not do a good job
> backporting
> > to all active branches, especially fixes.
> >
> > We have master, branch-2, branch-1, branch-1.4, branch-1.3, branch-1.2,
> and
> > branch-1.1 active currently. If a dirty bug fix, the applier is
> backporting
> > to 7 branches. It takes a while applying to all especially if the
> backport
> > doesn't go in clean. I suppose the RM could monitor all upstream of them
> > and then pull wanted patches back (we could do that too) but seems like a
> > burden on an RMer.
> >
> > We should EOL a few?
> >
> > Nick is on branch-1.1 release at the moment. Perhaps this could be the
> last
> > off branch-1.1.
> >
> > 1.2 hosts our current stable release though 1.3 is out. How about we cut
> a
> > release off 1.3, call it stable, and then EOL 1.2 after another release
> or
> > so?
> >
> > What you all think?
> >
> > Thanks,
> > S
> >
>

Re: DISCUSS: How can we have less branches?

[Nick Dimiduk]

Last time we DISCUSSed EOL of 1.1 was back in November. At that time, a
litany of issues were raised re: 1.2. Have those concerns been addressed?
It seems to me that making this one the last release is too abrupt to folks
tracking Apache. Would be better to give some notice.

Had a nice hallway conversation a couple months back (at PhoenixCon, as it
happens; they feel the pain as well) about our branch situation. I'll let
the others chime in with more details, but the gist as I recall is that we
should be doing more frequent minor releases with fewer patch releases.
This pushes stabilization efforts closer to master and also imposes more
strict stability requirements on big new features before they can be merged
off the feature branch.

The discussion also brought up the notion of an LTS release line. I'm not
sure how this jives with the more fequent minors, but would require some
branch that's so stable that an RM can effectively spin releases blind.

On Tue, Aug 8, 2017 at 12:14 AM Stack  wrote:

> (This came up during dev meeting in Shenzhen) We are running too many
> branches and/or when applying patches, we do not do a good job backporting
> to all active branches, especially fixes.
>
> We have master, branch-2, branch-1, branch-1.4, branch-1.3, branch-1.2, and
> branch-1.1 active currently. If a dirty bug fix, the applier is backporting
> to 7 branches. It takes a while applying to all especially if the backport
> doesn't go in clean. I suppose the RM could monitor all upstream of them
> and then pull wanted patches back (we could do that too) but seems like a
> burden on an RMer.
>
> We should EOL a few?
>
> Nick is on branch-1.1 release at the moment. Perhaps this could be the last
> off branch-1.1.
>
> 1.2 hosts our current stable release though 1.3 is out. How about we cut a
> release off 1.3, call it stable, and then EOL 1.2 after another release or
> so?
>
> What you all think?
>
> Thanks,
> S
>

Re: DISCUSS: How can we have less branches?

[Mikhail Antonov]

I can also start separate discussion thread on what prevents 1.3 (and 1.4
subsequently, though Andrew would be the right person here) from being
marked stable in our books, so this issue gets more focused attention.

-Mikhail

On Tue, Aug 8, 2017 at 8:10 AM, Mikhail Antonov 
wrote:

> I think in the dev community interest and ultimately in interest of our
> users to have a fewer maintained branches.
>
> For a variety of reasons.
>
> Efforts put in the maintenance of old release lines is the effort not put
> into releasing new ones, since community resources are
> finite. Active backporting (beyond critical security fixes and such) of
> new things somewhat reduces users incentive to upgrade often. Then we can
> get in the loop situation when new major features aren't deemed very stable
> - but it's really hard to get features of that complexity stable and robust
> if they're not used beyond integration tests.
>
> So before we go too deeply into discussion on how to make backporting
> easier, I'd rather see us focusing on releasing new branches and making
> them stable.
>
> -Mikhail
>
>
>
> On Tue, Aug 8, 2017 at 7:10 AM, Sean Busbey  wrote:
>
>> Are we approaching this problem wrong though?
>>
>> Most of the cherry picks I have to do to the maintenance release lines are
>> clean.
>>
>> What if we get more tooling to help with the everything-goes-right case?
>>
>> My last read of asf policy and infra folks (from back in the website
>> automation) is they won't look favorably on Jenkins pushing back ports
>> into
>> our repo.
>>
>> But! We could make something similar to the website job from back when a
>> committer still had to do the git push. A nightly job that tries to cherry
>> pick and gives us a set of back ports that worked.
>>
>> We could either have it look to jira for fix versions to try backporting
>> to, or we could have it try everything and update jira with what worked.
>>
>> Thoughts?
>>
>> On Aug 8, 2017 02:14, "Stack"  wrote:
>>
>> > (This came up during dev meeting in Shenzhen) We are running too many
>> > branches and/or when applying patches, we do not do a good job
>> backporting
>> > to all active branches, especially fixes.
>> >
>> > We have master, branch-2, branch-1, branch-1.4, branch-1.3, branch-1.2,
>> and
>> > branch-1.1 active currently. If a dirty bug fix, the applier is
>> backporting
>> > to 7 branches. It takes a while applying to all especially if the
>> backport
>> > doesn't go in clean. I suppose the RM could monitor all upstream of them
>> > and then pull wanted patches back (we could do that too) but seems like
>> a
>> > burden on an RMer.
>> >
>> > We should EOL a few?
>> >
>> > Nick is on branch-1.1 release at the moment. Perhaps this could be the
>> last
>> > off branch-1.1.
>> >
>> > 1.2 hosts our current stable release though 1.3 is out. How about we
>> cut a
>> > release off 1.3, call it stable, and then EOL 1.2 after another release
>> or
>> > so?
>> >
>> > What you all think?
>> >
>> > Thanks,
>> > S
>> >
>>
>
>
>
> --
> Thanks,
> Michael Antonov
>



-- 
Thanks,
Michael Antonov

Re: DISCUSS: How can we have less branches?

[Mikhail Antonov]

I think in the dev community interest and ultimately in interest of our
users to have a fewer maintained branches.

For a variety of reasons.

Efforts put in the maintenance of old release lines is the effort not put
into releasing new ones, since community resources are
finite. Active backporting (beyond critical security fixes and such) of new
things somewhat reduces users incentive to upgrade often. Then we can get
in the loop situation when new major features aren't deemed very stable -
but it's really hard to get features of that complexity stable and robust
if they're not used beyond integration tests.

So before we go too deeply into discussion on how to make backporting
easier, I'd rather see us focusing on releasing new branches and making
them stable.

-Mikhail



On Tue, Aug 8, 2017 at 7:10 AM, Sean Busbey  wrote:

> Are we approaching this problem wrong though?
>
> Most of the cherry picks I have to do to the maintenance release lines are
> clean.
>
> What if we get more tooling to help with the everything-goes-right case?
>
> My last read of asf policy and infra folks (from back in the website
> automation) is they won't look favorably on Jenkins pushing back ports into
> our repo.
>
> But! We could make something similar to the website job from back when a
> committer still had to do the git push. A nightly job that tries to cherry
> pick and gives us a set of back ports that worked.
>
> We could either have it look to jira for fix versions to try backporting
> to, or we could have it try everything and update jira with what worked.
>
> Thoughts?
>
> On Aug 8, 2017 02:14, "Stack"  wrote:
>
> > (This came up during dev meeting in Shenzhen) We are running too many
> > branches and/or when applying patches, we do not do a good job
> backporting
> > to all active branches, especially fixes.
> >
> > We have master, branch-2, branch-1, branch-1.4, branch-1.3, branch-1.2,
> and
> > branch-1.1 active currently. If a dirty bug fix, the applier is
> backporting
> > to 7 branches. It takes a while applying to all especially if the
> backport
> > doesn't go in clean. I suppose the RM could monitor all upstream of them
> > and then pull wanted patches back (we could do that too) but seems like a
> > burden on an RMer.
> >
> > We should EOL a few?
> >
> > Nick is on branch-1.1 release at the moment. Perhaps this could be the
> last
> > off branch-1.1.
> >
> > 1.2 hosts our current stable release though 1.3 is out. How about we cut
> a
> > release off 1.3, call it stable, and then EOL 1.2 after another release
> or
> > so?
> >
> > What you all think?
> >
> > Thanks,
> > S
> >
>



-- 
Thanks,
Michael Antonov

Still Failing: HBase Generate Website

[Apache Jenkins Server]

Build status: Still Failing

The HBase website has not been updated to incorporate HBase commit 
${HBASE_GIT_SHA}.

See https://builds.apache.org/job/hbase_generate_website/1074/console

Re: DISCUSS: How can we have less branches?

[Sean Busbey]

Are we approaching this problem wrong though?

Most of the cherry picks I have to do to the maintenance release lines are
clean.

What if we get more tooling to help with the everything-goes-right case?

My last read of asf policy and infra folks (from back in the website
automation) is they won't look favorably on Jenkins pushing back ports into
our repo.

But! We could make something similar to the website job from back when a
committer still had to do the git push. A nightly job that tries to cherry
pick and gives us a set of back ports that worked.

We could either have it look to jira for fix versions to try backporting
to, or we could have it try everything and update jira with what worked.

Thoughts?

On Aug 8, 2017 02:14, "Stack"  wrote:

> (This came up during dev meeting in Shenzhen) We are running too many
> branches and/or when applying patches, we do not do a good job backporting
> to all active branches, especially fixes.
>
> We have master, branch-2, branch-1, branch-1.4, branch-1.3, branch-1.2, and
> branch-1.1 active currently. If a dirty bug fix, the applier is backporting
> to 7 branches. It takes a while applying to all especially if the backport
> doesn't go in clean. I suppose the RM could monitor all upstream of them
> and then pull wanted patches back (we could do that too) but seems like a
> burden on an RMer.
>
> We should EOL a few?
>
> Nick is on branch-1.1 release at the moment. Perhaps this could be the last
> off branch-1.1.
>
> 1.2 hosts our current stable release though 1.3 is out. How about we cut a
> release off 1.3, call it stable, and then EOL 1.2 after another release or
> so?
>
> What you all think?
>
> Thanks,
> S
>

Re: DISCUSS: How can we have less branches?

[Sean Busbey]

I'd love to see us more aggressively updating the stable pointer.

I spent a but of time at the last hbasecon west socializing more EOL, but
as Mikhail mentions we can't make 1.3 the stable line while we know it's
broken. AFAIK, that same problem is in brach-1.4.

We could immediately EOL 1.3 and then declare 1.4.1 stable once the problem
is fixed. (Aiming the fix to 1.4.1 so that Andrew doesn't need to stop
before making a 1.4.0 if his other stabilization efforts finish first.)

This gets us fewer branches now and a plan to move the stable pointer.


On Aug 8, 2017 05:17, "Mikhail Antonov"  wrote:

I totally agree we have too many branches to maintain.

I think we should EOL 1.1. 1.2 has been out and stable for a long time,
doesn't have any outstanding major problems preventing upgrade that I'm
aware of, though obviously Sean would know better.

Speaking of 1.3, there's one issue that I feel needs to be addressed before
we move stable pointer to 1.3 and start EOL-ing 1.2

HBASE-18397, TL;DR - major optimization changes in scanners locking schema
in 1.3 broke store file accounting, which could cause failed long scans,
failed splits and failed snapshots. Many people spent lots and lots of time
fixing various aspects of that, but there are still issues not yet fixed.
Any help in that area would be amazing. This specific issue probably
deserves separate discussion.

Thanks,
Mikhail

On Tue, Aug 8, 2017 at 12:14 AM, Stack  wrote:

> (This came up during dev meeting in Shenzhen) We are running too many
> branches and/or when applying patches, we do not do a good job backporting
> to all active branches, especially fixes.
>
> We have master, branch-2, branch-1, branch-1.4, branch-1.3, branch-1.2,
and
> branch-1.1 active currently. If a dirty bug fix, the applier is
backporting
> to 7 branches. It takes a while applying to all especially if the backport
> doesn't go in clean. I suppose the RM could monitor all upstream of them
> and then pull wanted patches back (we could do that too) but seems like a
> burden on an RMer.
>
> We should EOL a few?
>
> Nick is on branch-1.1 release at the moment. Perhaps this could be the
last
> off branch-1.1.
>
> 1.2 hosts our current stable release though 1.3 is out. How about we cut a
> release off 1.3, call it stable, and then EOL 1.2 after another release or
> so?
>
> What you all think?
>
> Thanks,
> S
>



--
Thanks,
Michael Antonov

[jira] [Created] (HBASE-18538) deduplicate copies of jquery files

[Tamas Penzes (JIRA)]

Tamas Penzes created HBASE-18538:


 Summary: deduplicate copies of jquery files
 Key: HBASE-18538
 URL: https://issues.apache.org/jira/browse/HBASE-18538
 Project: HBase
  Issue Type: Improvement
Reporter: Tamas Penzes
Priority: Minor
 Fix For: 2.0.0


In HBASE-14093 we started using a webjar as a source of Bootstrap.
Based on that solution it would be very simple to use jQuery from webjar too.
This way only the version number should be changed at later version updates and 
we would not store the JS code of jQuery.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: DISCUSS: How can we have less branches?

[Mikhail Antonov]

I totally agree we have too many branches to maintain.

I think we should EOL 1.1. 1.2 has been out and stable for a long time,
doesn't have any outstanding major problems preventing upgrade that I'm
aware of, though obviously Sean would know better.

Speaking of 1.3, there's one issue that I feel needs to be addressed before
we move stable pointer to 1.3 and start EOL-ing 1.2

HBASE-18397, TL;DR - major optimization changes in scanners locking schema
in 1.3 broke store file accounting, which could cause failed long scans,
failed splits and failed snapshots. Many people spent lots and lots of time
fixing various aspects of that, but there are still issues not yet fixed.
Any help in that area would be amazing. This specific issue probably
deserves separate discussion.

Thanks,
Mikhail

On Tue, Aug 8, 2017 at 12:14 AM, Stack  wrote:

> (This came up during dev meeting in Shenzhen) We are running too many
> branches and/or when applying patches, we do not do a good job backporting
> to all active branches, especially fixes.
>
> We have master, branch-2, branch-1, branch-1.4, branch-1.3, branch-1.2, and
> branch-1.1 active currently. If a dirty bug fix, the applier is backporting
> to 7 branches. It takes a while applying to all especially if the backport
> doesn't go in clean. I suppose the RM could monitor all upstream of them
> and then pull wanted patches back (we could do that too) but seems like a
> burden on an RMer.
>
> We should EOL a few?
>
> Nick is on branch-1.1 release at the moment. Perhaps this could be the last
> off branch-1.1.
>
> 1.2 hosts our current stable release though 1.3 is out. How about we cut a
> release off 1.3, call it stable, and then EOL 1.2 after another release or
> so?
>
> What you all think?
>
> Thanks,
> S
>



-- 
Thanks,
Michael Antonov

Re: [DISCUSS] Should flush decisions be made based on data size (key-value only) or based on heap size (including metadata overhead)?

[Eshcar Hillel]

We can change the AtomicLong to be AtomicReference which is 
updated atomically.MemStoreSize can be changed to hold on-heap memory, off-heap 
memory and data size, or just on-heap memory and off-heap memory if data size 
is not required anymore.



On Monday, August 7, 2017, 10:23:12 AM GMT+3, Anoop John 
 wrote:

Sorry for being later to reply.

So u mean we should track both sizes even at Region level?  This was
considered at that time but did not do as that will add more overhead.
We have to deal with 2 AtomicLongs in every Region.  Right now we
handle this double check at RS level only so that added just one more
variable dealing.

-Anoop-

On Mon, Jul 10, 2017 at 7:34 PM, Eshcar Hillel
 wrote:
> Here is a suggestion:We can track both heap and off-heap sizes and have 2 
> thresholds one for limiting heap size and one for limiting off-heap size.And 
> in all decision making junctions we check whether one of the thresholds is 
> exceeded and if it is we trigger a flush. We can choose which entity to flush 
> based on the cause.For example, if we decided to flush since the heap size 
> exceeds the heap threshold than we flush the region/store with greatest heap 
> size. and likewise for off-heap flush.
>
> I can prepare a patch.
>
> This is not rolling back HBASE-18294 simply refining it to have different 
> decision making for the on and off heap cases.
>
> On Monday, July 10, 2017, 8:25:12 AM GMT+3, Anoop John 
>  wrote:
>
> Stack and others..
> We wont do any OOM or FullGC issues.  Because globally at RS level we
> will track both the data size (of all the memstores) and the heap
> size.  The decision there accounts both. In fact in case of normal on
> heap memstores, the accounting is like the old way of heap size based.
>
> At region level (and at Segments level)  we track data size only.  The
> decisions are based on data size.
>
> So in the past region flush size of 128 MB means we will flush when
> the heap size of that region crosses 128 MB.  But now it is data size
> alone.  What I feel is that is more inclined to a normal user
> thinking.  He say flush size of 128 MB and then the thinking can be
> 128 MB of data.
>
> The background of this change is the off heap memstores where we need
> separate tracking of both data and heap overhead sizes.  But at
> region level this behave change was done thinking that is more user
> oriented
>
> I agree with Yu that it is a surprising behave change. Ya if not tuned
> accordingly one might see more blocked writes. Because the per region
> flushes are more delayed now and so chances of reaching the global
> memstore upper barrier chances are more.  And then we will block
> writes and force flushes.  (But off heap memstores will do better job
> here).  But this would NOT cause any OOME or FullGC.
>
> I guess we should have reduced the 128 MB default flush size then?  I
> asked this Q in that jira and then we did not discuss further.
>
> I hope I explained the background and the change and the impacts.  Thanks.
>
> -Anoop-
>
> On Thu, Jul 6, 2017 at 11:43 AM, 宾莉金（binlijin）  wrote:
>> I like to use the former, heap occupancy, so we not need to worry about the
>> OOM and FullGc，and change configuration to adapted to new policy.
>>
>> 2017-07-06 14:03 GMT+08:00 Stack :
>>
>>> On Wed, Jul 5, 2017 at 9:59 PM, ramkrishna vasudevan <
>>> ramkrishna.s.vasude...@gmail.com> wrote:
>>>
>>> >
>>> > >>Sounds like we should be doing the former, heap occupancy
>>> > Stack, so do you mean we need to roll back this new change in trunk? The
>>> > background is https://issues.apache.org/jira/browse/HBASE-16747.
>>> >
>>> >
>>> I remember that issue. It seems good to me (as it did then) where we have
>>> the global tracking in RS of all data and overhead so we shouldn't OOME and
>>> we keep accounting of overhead and data distinct because now data can be
>>> onheap or offheap.
>>>
>>> We shouldn't be doing blocking updates -- not when there is probably loads
>>> of memory still available -- but that is a different (critical) issue.
>>> Sounds like current configs can 'surprise' -- see Yu Li note -- given the
>>> new accounting.
>>>
>>> Looks like I need to read HBASE-18294
>>>  to figure what the
>>> pivot/problem w/ the new policy is.
>>>
>>> Thanks,
>>> St.Ack
>>>
>>>
>>>
>>>
>>>
>>> > Regards
>>> > Ram
>>> >
>>> >
>>> > On Thu, Jul 6, 2017 at 8:40 AM, Yu Li  wrote:
>>> >
>>> > > We've also observed more blocking updates happening with the new policy
>>> > > (flush decision made on data size), but could work-around it by
>>> reducing
>>> > > the hbase.hregion.memstore.flush.size setting. The advantage of
>>> current
>>> > > policy is we could control the flushed file size more accurately, but
>>> > > meanwhile losing some "compatibility" (requires configuration updating
>>> > > during rolling 

DISCUSS: How can we have less branches?

[Stack]

(This came up during dev meeting in Shenzhen) We are running too many
branches and/or when applying patches, we do not do a good job backporting
to all active branches, especially fixes.

We have master, branch-2, branch-1, branch-1.4, branch-1.3, branch-1.2, and
branch-1.1 active currently. If a dirty bug fix, the applier is backporting
to 7 branches. It takes a while applying to all especially if the backport
doesn't go in clean. I suppose the RM could monitor all upstream of them
and then pull wanted patches back (we could do that too) but seems like a
burden on an RMer.

We should EOL a few?

Nick is on branch-1.1 release at the moment. Perhaps this could be the last
off branch-1.1.

1.2 hosts our current stable release though 1.3 is out. How about we cut a
release off 1.3, call it stable, and then EOL 1.2 after another release or
so?

What you all think?

Thanks,
S

Re: Notes from dev meetup in Shenzhen, August 5th, 2017

[Stack]

On Tue, Aug 8, 2017 at 12:01 PM, Yu Li  wrote:
...

> Maybe we could open several umbrellas in JIRA to follow up the
> implement-able topics and do some prioritizing? Thanks.
>

Agree.

Let me start up the DISCUSS suggested by Ashish; it just came up again on
the tail of an issue

St.Ack




> Best Regards,
> Yu
>
> On 8 August 2017 at 10:54, ashish singhi  wrote:
>
> > Great write up, Stack. Covering everything what we all discussed.
> > It was very nice meeting you all and hope we can continue this HBaseCon
> > Asia.
> >
> > Regards,
> > Ashish
> >
> > From: saint@gmail.com [mailto:saint@gmail.com] On Behalf Of
> Stack
> > Sent: 08 August 2017 00:07
> > To: HBase Dev List 
> > Subject: Notes from dev meetup in Shenzhen, August 5th, 2017
> >
> > At fancy Huawei headquarters, 10:00-12:00AM or so (with nice coffee and
> > fancy little cake squares provided about half way through the session).
> >
> > For list of attendees, see picture at end of this email.
> >
> > Discussion was mostly in Chinese with about 25% in English plus some
> > gracious sideline translation so the below is patchy. Hopefully you get
> the
> > gist.
> >
> > For client-side scanner going against hfiles directly; is there a means
> of
> > being able to pass the permissions from hbase to hdfs?
> >
> > Issues w/ the hbase 99th percentile were brought up. "DynamoDB can do
> > 10ms". How to do better?
> >
> > SSD is not enough.
> >
> > GC messes us up.
> >
> > Will the Distributed Log Replay come back to help improve MTTR? We could
> > redo on new ProcedureV2 basis. ZK timeout is the biggest issue. Do as we
> > used to and just rely on the regionserver heartbeating...
> >
> > Read replica helps w/ MTTR.
> >
> > Ratis incubator project to do a quorum based hbase?
> >
> > Digression on licensing issues around fb wangle and folly.
> >
> > Redo of hbase but quorum based would be another project altogether.
> >
> > Decided to go around the table to talk about concerns and what people are
> > working on.
> >
> > Jieshan wondered what could be done to improve OLAP over hbase.
> >
> > Client side scanner was brought up again as means of skipping RS overhead
> > and doing better OLAP.
> >
> > Have HBase compact to parquet files. Query parquet and hbase.
> >
> > At Huawei, they are using 1.0 hbase. Most problems are assignment. They
> > have .5M regions. RIT is a killer. Double assignment issues. And RIT.
> They
> > run their own services. Suggested they upgrade to get fixes at least.
> Then
> > 2.0.
> >
> > Will HBase federate like HDFS? Can Master handle load at large scale? It
> > needs to do federation too?
> >
> > Anyone using Bulk loaded replication? (Yes, it just works so no one talks
> > about it...)
> >
> > Request that fixes be backported to all active branches, not just most
> > current.
> >
> > Andrew was good at backporting... not all RMs are.
> >
> > Too many branches. What should we do?
> >
> > Proliferation of branches makes for too much work.
> >
> > Need to cleanup bugs in 1.3. Make it stable release now.
> >
> > Lets do more active EOL'ing of branches. 1.1?.
> >
> > Hubert asked if we can have clusters where RS are differently capable?
> > i.e. several generations of HW all running in the same cluster.
> >
> > What if fat server goes down.
> >
> > Balancer could take of it all. RS Capacity. Balancer can take it into
> > account.
> > Regionserver labels like YARN labels. Characteristics.
> >
> > Or run it all in docker when heterogeneous cluster. The K8 talk from day
> > before was mentioned; we should all look at being able to deploy in k8
> and
> > docker.
> >
> > Lets put out kubernetes blog...(Doing).
> >
> > Alibaba looking at HBase as native YARN app.
> >
> > i/o is hard even when containers.
> >
> > Use autoscaler of K8 when heavy user.
> >
> > Limit i/o use w/ CP. Throttle.
> >
> > Spark and client-side scanner came up again.
> >
> > Snapshot input format in spark.
> >
> > HBase federation came up again. jd.com talking of 3k to
> 4k
> > nodes in a cluster. Millions of regions. Region assignment is messing
> them
> > up.
> >
> > Maybe federation is good idea? Argument that it is too much operational
> > conplexity. Can we fix master load w/ splittable meta, etc?
> >
> > Was brought up that even w/ 100s of RS there is scale issue, nvm
> thousands.
> >
> > Alibaba talked about disaster recovery. Described issue where HDFS has
> > fencing problem during an upgrade. There was no active NN. All RS went
> down.
> > ZK is another POF. If ZK is not available. Operators were being asked how
> > much longer the cluster was going to be down but they could not answer
> the
> > question. No indicators from HBase on how much longer it will be down or
> > how many WALs its processed and how many more to go. Operator unable to
> > tell his org how long it would be before it all came back on line. Should
> > say how many regions are online and how