Re: [DISCUSS] 2.1.1 Release Plans?

2018-10-24 Thread Stack 
On Tue, Oct 23, 2018 at 10:36 AM Stack  wrote:

> Heads-up. I'm going to cut an RC0 this evening unless objection. HBASE-21073
> landed this morning as did some other sweet fixes over night.
>
> This didn't happen. A bunch of other nice fixes showed up last night --
thanks for all the hard work -- and today our Drob is working on failing
unit tests, etc. Hopefully an RC tomorrow.
Thanks,
S


> Mildly related, I put up first cut at hbck2 doc over at
> https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2.
> There is way more to do -- including figuring gitbox vs gh-pages -- but
> feedback/patches appreciated.
>
> Thanks,
> S
>
> On Mon, Oct 22, 2018 at 11:33 AM Stack  wrote:
>
>> On Mon, Oct 22, 2018 at 10:21 AM Josh Elser  wrote:
>>
>>> Orthogonal question: you codifying the "damage" you're doing?
>>>
>>>
>> No.
>>
>> What you thinking? Writing unit tests that launch substantial clusters
>> that get damaged in various ways? We'd then use tools to fix and confirm
>> wholesomeness?
>>
>> If so, tooling is currently inadequate or absent as is understanding of
>> the various failure types and then what the subsequent fixes should be (For
>> general travails, see [1]).
>>
>>
>>
>>> I remember trying to debug hbck1 unit test failures (and it was a pain).
>>> Coming back with fresh/high-level test scenarios sounds like a great way
>>> to keep quality on hbck2 up.
>>>
>>>
>> Agree. hbck2 has some tests of its basic functionality but I at least
>> have done no work beyond that.
>>
>> S
>>
>> 1.
>> https://docs.google.com/document/d/1Y0HIo5yRGXi7nl-JWc69JtxB87fYE-jXe8nBe7HWKe0/edit#heading=h.8e0m4iclfn0g
>>
>>
>>
>>> On 10/19/18 1:32 PM, Stack wrote:
>>> >   * Lots of progress on an hbck2. It has some basic utility (see
>>> below) that
>>> > has been useful to me at least hacking on a test cluster I've been
>>> doing
>>> > damage too this last week or so. It exits with complaint if run
>>> against an
>>> > hbase that doesn't have support for hbck2 ops (i.e. < 2.0.3 or <
>>> 2.1.0) and
>>> > it is itself versioned. I'll work on a bit of doc and our Sean is
>>> working
>>> > on making it easy to find and run over in HBASE-21215
>>> > . We could cut a
>>> 1.0.0RC
>>> > inside the next week or so I'd say.
>>>
>>

Re: [DISCUSS] 2.1.1 Release Plans?

2018-10-23 Thread Stack 
Heads-up. I'm going to cut an RC0 this evening unless objection. HBASE-21073
landed this morning as did some other sweet fixes over night.

Mildly related, I put up first cut at hbck2 doc over at
https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2.
There is way more to do -- including figuring gitbox vs gh-pages -- but
feedback/patches appreciated.

Thanks,
S

On Mon, Oct 22, 2018 at 11:33 AM Stack  wrote:

> On Mon, Oct 22, 2018 at 10:21 AM Josh Elser  wrote:
>
>> Orthogonal question: you codifying the "damage" you're doing?
>>
>>
> No.
>
> What you thinking? Writing unit tests that launch substantial clusters
> that get damaged in various ways? We'd then use tools to fix and confirm
> wholesomeness?
>
> If so, tooling is currently inadequate or absent as is understanding of
> the various failure types and then what the subsequent fixes should be (For
> general travails, see [1]).
>
>
>
>> I remember trying to debug hbck1 unit test failures (and it was a pain).
>> Coming back with fresh/high-level test scenarios sounds like a great way
>> to keep quality on hbck2 up.
>>
>>
> Agree. hbck2 has some tests of its basic functionality but I at least have
> done no work beyond that.
>
> S
>
> 1.
> https://docs.google.com/document/d/1Y0HIo5yRGXi7nl-JWc69JtxB87fYE-jXe8nBe7HWKe0/edit#heading=h.8e0m4iclfn0g
>
>
>
>> On 10/19/18 1:32 PM, Stack wrote:
>> >   * Lots of progress on an hbck2. It has some basic utility (see below)
>> that
>> > has been useful to me at least hacking on a test cluster I've been doing
>> > damage too this last week or so. It exits with complaint if run against
>> an
>> > hbase that doesn't have support for hbck2 ops (i.e. < 2.0.3 or < 2.1.0)
>> and
>> > it is itself versioned. I'll work on a bit of doc and our Sean is
>> working
>> > on making it easy to find and run over in HBASE-21215
>> > . We could cut a
>> 1.0.0RC
>> > inside the next week or so I'd say.
>>
>

Re: [DISCUSS] 2.1.1 Release Plans?

2018-10-22 Thread Stack 
On Mon, Oct 22, 2018 at 10:21 AM Josh Elser  wrote:

> Orthogonal question: you codifying the "damage" you're doing?
>
>
No.

What you thinking? Writing unit tests that launch substantial clusters that
get damaged in various ways? We'd then use tools to fix and confirm
wholesomeness?

If so, tooling is currently inadequate or absent as is understanding of the
various failure types and then what the subsequent fixes should be (For
general travails, see [1]).



> I remember trying to debug hbck1 unit test failures (and it was a pain).
> Coming back with fresh/high-level test scenarios sounds like a great way
> to keep quality on hbck2 up.
>
>
Agree. hbck2 has some tests of its basic functionality but I at least have
done no work beyond that.

S

1.
https://docs.google.com/document/d/1Y0HIo5yRGXi7nl-JWc69JtxB87fYE-jXe8nBe7HWKe0/edit#heading=h.8e0m4iclfn0g



> On 10/19/18 1:32 PM, Stack wrote:
> >   * Lots of progress on an hbck2. It has some basic utility (see below)
> that
> > has been useful to me at least hacking on a test cluster I've been doing
> > damage too this last week or so. It exits with complaint if run against
> an
> > hbase that doesn't have support for hbck2 ops (i.e. < 2.0.3 or < 2.1.0)
> and
> > it is itself versioned. I'll work on a bit of doc and our Sean is working
> > on making it easy to find and run over in HBASE-21215
> > . We could cut a
> 1.0.0RC
> > inside the next week or so I'd say.
>

Re: [DISCUSS] 2.1.1 Release Plans?

2018-10-22 Thread Stack 
On Mon, Oct 22, 2018 at 10:00 AM Mike Drob  wrote:

> Stack - I'm still working on HBASE-21073, there were tests that failed in
> precommit so I launched a retry. Please don't cut an RC without it.
>
>
NP. HBASE-21073 is not a blocker though. Hopefully lands in next day or so.
Thanks,
S


> On Mon, Oct 22, 2018 at 11:56 AM Stack  wrote:
>
> > Back again
> >
> > Lets push out a 2.1.1RC0.
> >
> > Here are the list of outstanding issues:
> > https://issues.apache.org/jira/projects/HBASE/versions/12343470
> >
> > Nightlies are gettting better [1] but still some flakies in the mix[2].
> >
> > If you are up for helping or you own any of the above outstanding issues,
> > please take a look and help resolve them over the next day or so.
> >
> > Meantime, lets call a halt on branch-2.1 commits, or at least, lets at
> > least chat before commit (Some great fixes have come in over the last few
> > days).
> >
> > I've been testing the tip of branch-2.1 and its looking pretty good. Will
> > keep at it.
> >
> > Thanks,
> > S
> >
> > 1.
> >
> >
> https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-2.1/
> > 2.
> >
> >
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.1/lastSuccessfulBuild/artifact/dashboard.html
> >
> > On Fri, Oct 19, 2018 at 10:32 AM Stack  wrote:
> >
> > > Kicking this thread
> > >
> > >  * Lots of progress on an hbck2. It has some basic utility (see below)
> > > that has been useful to me at least hacking on a test cluster I've been
> > > doing damage too this last week or so. It exits with complaint if run
> > > against an hbase that doesn't have support for hbck2 ops (i.e. < 2.0.3
> > or <
> > > 2.1.0) and it is itself versioned. I'll work on a bit of doc and our
> Sean
> > > is working on making it easy to find and run over in HBASE-21215
> > > . We could cut a
> > > 1.0.0RC inside the next week or so I'd say.
> > >  * A bunch of messy stuff has been fixed over the last few weeks on the
> > > tip of branch-2.1 thanks to our Duo, Allan, JIngyun,among others (and
> > > backported to branch-2.0 <= Look for a 2.0.3RC soon after the
> > 2.1.1RC...).
> > > In cluster testing, we're not looking bad.
> > >
> > > So, I think a 2.1.1RC0 is not far off. If you want to help out, there's
> > > just a few outstanding issues [1]. If any are yours, please do an
> update
> > > (including moving out of 2.1.1 if you don't think it will make it ).
> The
> > > other area that needs love is failing unit tests. There are just a few.
> > > Pick one and have a go at it [2].
> > >
> > > Lets try and get an RC0 up next week or so?
> > > Thanks,
> > > S
> > >
> > > 1. https://issues.apache.org/jira/projects/HBASE/versions/12343470
> > > 2.
> > >
> >
> https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-2.1/
> > > and
> > >
> >
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.1/lastSuccessfulBuild/artifact/dashboard.html
> > >
> > > Below is usage for HBCK2 as of today:
> > >
> > > $
> > >
> >
> HBASE_CLASSPATH_PREFIX=~/checkouts/hbase-operator-tools/hbase-hbck2/target/hbase-hbck2-1.0.0-SNAPSHOT.jar
> > > ./bin/hbase org.apache.hbase.HBCK2
> > > usage: HBCK2 [OPTIONS] COMMAND 
> > >
> > > Options:
> > >  -d,--debug run with debug output
> > >  -h,--help  output this help message
> > >  -p,--hbase.zookeeper.property.clientPort   port of target hbase
> ensemble
> > >  -q,--hbase.zookeeper.quorum   ensemble of target hbase
> > >  -v,--version   this hbck2 version
> > >  -z,--zookeeper.znode.parentparent znode of target
> hbase
> > >
> > > Commands:
> > >  assigns [OPTIONS] ...
> > >Options:
> > > -o,--override  override ownership by another procedure
> > >A 'raw' assign that can be used even during Master initialization.
> > >Skirts Coprocessors. Pass one or more encoded RegionNames.
> > >1588230740 is the hard-coded name for the hbase:meta region and
> > >de00010733901a05f5a2a3a382e27dd4 is an example of what a user-space
> > >encoded Region name looks like. For example:
> > >  $ HBCK2 assign 1588230740 de00010733901a05f5a2a3a382e27dd4
> > >Returns the pid(s) of the created AssignProcedure(s) or -1 if none.
> > >
> > >  bypass [OPTIONS] ...
> > >Options:
> > > -o,--override   override if procedure is running/stuck
> > > -r,--recursive  bypass parent and its children. SLOW! EXPENSIVE!
> > > -w,--lockWait   milliseconds to wait on lock before giving up;
> > > default=1
> > >Pass one (or more) procedure 'pid's to skip to procedure finish.
> > >Parent of bypassed procedure will also be skipped to the finish.
> > >Entities will be left in an inconsistent state and will require
> > >manual fixup. May need Master restart to clear locks still held.
> > >Bypass fails

Re: [DISCUSS] 2.1.1 Release Plans?

2018-10-22 Thread Josh Elser 

Orthogonal question: you codifying the "damage" you're doing?

I remember trying to debug hbck1 unit test failures (and it was a pain). 
Coming back with fresh/high-level test scenarios sounds like a great way 
to keep quality on hbck2 up.


On 10/19/18 1:32 PM, Stack wrote:

  * Lots of progress on an hbck2. It has some basic utility (see below) that
has been useful to me at least hacking on a test cluster I've been doing
damage too this last week or so. It exits with complaint if run against an
hbase that doesn't have support for hbck2 ops (i.e. < 2.0.3 or < 2.1.0) and
it is itself versioned. I'll work on a bit of doc and our Sean is working
on making it easy to find and run over in HBASE-21215
. We could cut a 1.0.0RC
inside the next week or so I'd say.

Re: [DISCUSS] 2.1.1 Release Plans?

2018-10-22 Thread Mike Drob 
Stack - I'm still working on HBASE-21073, there were tests that failed in
precommit so I launched a retry. Please don't cut an RC without it.

On Mon, Oct 22, 2018 at 11:56 AM Stack  wrote:

> Back again
>
> Lets push out a 2.1.1RC0.
>
> Here are the list of outstanding issues:
> https://issues.apache.org/jira/projects/HBASE/versions/12343470
>
> Nightlies are gettting better [1] but still some flakies in the mix[2].
>
> If you are up for helping or you own any of the above outstanding issues,
> please take a look and help resolve them over the next day or so.
>
> Meantime, lets call a halt on branch-2.1 commits, or at least, lets at
> least chat before commit (Some great fixes have come in over the last few
> days).
>
> I've been testing the tip of branch-2.1 and its looking pretty good. Will
> keep at it.
>
> Thanks,
> S
>
> 1.
>
> https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-2.1/
> 2.
>
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.1/lastSuccessfulBuild/artifact/dashboard.html
>
> On Fri, Oct 19, 2018 at 10:32 AM Stack  wrote:
>
> > Kicking this thread
> >
> >  * Lots of progress on an hbck2. It has some basic utility (see below)
> > that has been useful to me at least hacking on a test cluster I've been
> > doing damage too this last week or so. It exits with complaint if run
> > against an hbase that doesn't have support for hbck2 ops (i.e. < 2.0.3
> or <
> > 2.1.0) and it is itself versioned. I'll work on a bit of doc and our Sean
> > is working on making it easy to find and run over in HBASE-21215
> > . We could cut a
> > 1.0.0RC inside the next week or so I'd say.
> >  * A bunch of messy stuff has been fixed over the last few weeks on the
> > tip of branch-2.1 thanks to our Duo, Allan, JIngyun,among others (and
> > backported to branch-2.0 <= Look for a 2.0.3RC soon after the
> 2.1.1RC...).
> > In cluster testing, we're not looking bad.
> >
> > So, I think a 2.1.1RC0 is not far off. If you want to help out, there's
> > just a few outstanding issues [1]. If any are yours, please do an update
> > (including moving out of 2.1.1 if you don't think it will make it ). The
> > other area that needs love is failing unit tests. There are just a few.
> > Pick one and have a go at it [2].
> >
> > Lets try and get an RC0 up next week or so?
> > Thanks,
> > S
> >
> > 1. https://issues.apache.org/jira/projects/HBASE/versions/12343470
> > 2.
> >
> https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-2.1/
> > and
> >
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.1/lastSuccessfulBuild/artifact/dashboard.html
> >
> > Below is usage for HBCK2 as of today:
> >
> > $
> >
> HBASE_CLASSPATH_PREFIX=~/checkouts/hbase-operator-tools/hbase-hbck2/target/hbase-hbck2-1.0.0-SNAPSHOT.jar
> > ./bin/hbase org.apache.hbase.HBCK2
> > usage: HBCK2 [OPTIONS] COMMAND 
> >
> > Options:
> >  -d,--debug run with debug output
> >  -h,--help  output this help message
> >  -p,--hbase.zookeeper.property.clientPort   port of target hbase ensemble
> >  -q,--hbase.zookeeper.quorum   ensemble of target hbase
> >  -v,--version   this hbck2 version
> >  -z,--zookeeper.znode.parentparent znode of target hbase
> >
> > Commands:
> >  assigns [OPTIONS] ...
> >Options:
> > -o,--override  override ownership by another procedure
> >A 'raw' assign that can be used even during Master initialization.
> >Skirts Coprocessors. Pass one or more encoded RegionNames.
> >1588230740 is the hard-coded name for the hbase:meta region and
> >de00010733901a05f5a2a3a382e27dd4 is an example of what a user-space
> >encoded Region name looks like. For example:
> >  $ HBCK2 assign 1588230740 de00010733901a05f5a2a3a382e27dd4
> >Returns the pid(s) of the created AssignProcedure(s) or -1 if none.
> >
> >  bypass [OPTIONS] ...
> >Options:
> > -o,--override   override if procedure is running/stuck
> > -r,--recursive  bypass parent and its children. SLOW! EXPENSIVE!
> > -w,--lockWait   milliseconds to wait on lock before giving up;
> > default=1
> >Pass one (or more) procedure 'pid's to skip to procedure finish.
> >Parent of bypassed procedure will also be skipped to the finish.
> >Entities will be left in an inconsistent state and will require
> >manual fixup. May need Master restart to clear locks still held.
> >Bypass fails if procedure has children. Add 'recursive' if all
> >you have is a parent pid to finish parent and children. This
> >is SLOW, and dangerous so use selectively. Does not always work.
> >
> >  unassigns ...
> >Options:
> > -o,--override  override ownership by another procedure
> >A 'raw' unassign that can be used even during Master initialization.
> >Skir

Re: [DISCUSS] 2.1.1 Release Plans?

2018-10-22 Thread Stack 
Back again

Lets push out a 2.1.1RC0.

Here are the list of outstanding issues:
https://issues.apache.org/jira/projects/HBASE/versions/12343470

Nightlies are gettting better [1] but still some flakies in the mix[2].

If you are up for helping or you own any of the above outstanding issues,
please take a look and help resolve them over the next day or so.

Meantime, lets call a halt on branch-2.1 commits, or at least, lets at
least chat before commit (Some great fixes have come in over the last few
days).

I've been testing the tip of branch-2.1 and its looking pretty good. Will
keep at it.

Thanks,
S

1.
https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-2.1/
2.
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.1/lastSuccessfulBuild/artifact/dashboard.html

On Fri, Oct 19, 2018 at 10:32 AM Stack  wrote:

> Kicking this thread
>
>  * Lots of progress on an hbck2. It has some basic utility (see below)
> that has been useful to me at least hacking on a test cluster I've been
> doing damage too this last week or so. It exits with complaint if run
> against an hbase that doesn't have support for hbck2 ops (i.e. < 2.0.3 or <
> 2.1.0) and it is itself versioned. I'll work on a bit of doc and our Sean
> is working on making it easy to find and run over in HBASE-21215
> . We could cut a
> 1.0.0RC inside the next week or so I'd say.
>  * A bunch of messy stuff has been fixed over the last few weeks on the
> tip of branch-2.1 thanks to our Duo, Allan, JIngyun,among others (and
> backported to branch-2.0 <= Look for a 2.0.3RC soon after the 2.1.1RC...).
> In cluster testing, we're not looking bad.
>
> So, I think a 2.1.1RC0 is not far off. If you want to help out, there's
> just a few outstanding issues [1]. If any are yours, please do an update
> (including moving out of 2.1.1 if you don't think it will make it ). The
> other area that needs love is failing unit tests. There are just a few.
> Pick one and have a go at it [2].
>
> Lets try and get an RC0 up next week or so?
> Thanks,
> S
>
> 1. https://issues.apache.org/jira/projects/HBASE/versions/12343470
> 2.
> https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-2.1/
> and
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.1/lastSuccessfulBuild/artifact/dashboard.html
>
> Below is usage for HBCK2 as of today:
>
> $
> HBASE_CLASSPATH_PREFIX=~/checkouts/hbase-operator-tools/hbase-hbck2/target/hbase-hbck2-1.0.0-SNAPSHOT.jar
> ./bin/hbase org.apache.hbase.HBCK2
> usage: HBCK2 [OPTIONS] COMMAND 
>
> Options:
>  -d,--debug run with debug output
>  -h,--help  output this help message
>  -p,--hbase.zookeeper.property.clientPort   port of target hbase ensemble
>  -q,--hbase.zookeeper.quorum   ensemble of target hbase
>  -v,--version   this hbck2 version
>  -z,--zookeeper.znode.parentparent znode of target hbase
>
> Commands:
>  assigns [OPTIONS] ...
>Options:
> -o,--override  override ownership by another procedure
>A 'raw' assign that can be used even during Master initialization.
>Skirts Coprocessors. Pass one or more encoded RegionNames.
>1588230740 is the hard-coded name for the hbase:meta region and
>de00010733901a05f5a2a3a382e27dd4 is an example of what a user-space
>encoded Region name looks like. For example:
>  $ HBCK2 assign 1588230740 de00010733901a05f5a2a3a382e27dd4
>Returns the pid(s) of the created AssignProcedure(s) or -1 if none.
>
>  bypass [OPTIONS] ...
>Options:
> -o,--override   override if procedure is running/stuck
> -r,--recursive  bypass parent and its children. SLOW! EXPENSIVE!
> -w,--lockWait   milliseconds to wait on lock before giving up;
> default=1
>Pass one (or more) procedure 'pid's to skip to procedure finish.
>Parent of bypassed procedure will also be skipped to the finish.
>Entities will be left in an inconsistent state and will require
>manual fixup. May need Master restart to clear locks still held.
>Bypass fails if procedure has children. Add 'recursive' if all
>you have is a parent pid to finish parent and children. This
>is SLOW, and dangerous so use selectively. Does not always work.
>
>  unassigns ...
>Options:
> -o,--override  override ownership by another procedure
>A 'raw' unassign that can be used even during Master initialization.
>Skirts Coprocessors. Pass one or more encoded RegionNames:
>1588230740 is the hard-coded name for the hbase:meta region and
>de00010733901a05f5a2a3a382e27dd4 is an example of what a user-space
>encoded Region name looks like. For example:
>  $ HBCK2 unassign 1588230740 de00010733901a05f5a2a3a382e27dd4
>Returns the pid(s) of the created UnassignProcedure(s) or -1 if none.
>
>  setTableStat

Re: [DISCUSS] 2.1.1 Release Plans?

2018-10-19 Thread Stack 
Kicking this thread

 * Lots of progress on an hbck2. It has some basic utility (see below) that
has been useful to me at least hacking on a test cluster I've been doing
damage too this last week or so. It exits with complaint if run against an
hbase that doesn't have support for hbck2 ops (i.e. < 2.0.3 or < 2.1.0) and
it is itself versioned. I'll work on a bit of doc and our Sean is working
on making it easy to find and run over in HBASE-21215
. We could cut a 1.0.0RC
inside the next week or so I'd say.
 * A bunch of messy stuff has been fixed over the last few weeks on the tip
of branch-2.1 thanks to our Duo, Allan, JIngyun,among others (and
backported to branch-2.0 <= Look for a 2.0.3RC soon after the 2.1.1RC...).
In cluster testing, we're not looking bad.

So, I think a 2.1.1RC0 is not far off. If you want to help out, there's
just a few outstanding issues [1]. If any are yours, please do an update
(including moving out of 2.1.1 if you don't think it will make it ). The
other area that needs love is failing unit tests. There are just a few.
Pick one and have a go at it [2].

Lets try and get an RC0 up next week or so?
Thanks,
S

1. https://issues.apache.org/jira/projects/HBASE/versions/12343470
2.
https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-2.1/
and
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.1/lastSuccessfulBuild/artifact/dashboard.html

Below is usage for HBCK2 as of today:

$
HBASE_CLASSPATH_PREFIX=~/checkouts/hbase-operator-tools/hbase-hbck2/target/hbase-hbck2-1.0.0-SNAPSHOT.jar
./bin/hbase org.apache.hbase.HBCK2
usage: HBCK2 [OPTIONS] COMMAND 

Options:
 -d,--debug run with debug output
 -h,--help  output this help message
 -p,--hbase.zookeeper.property.clientPort   port of target hbase ensemble
 -q,--hbase.zookeeper.quorum   ensemble of target hbase
 -v,--version   this hbck2 version
 -z,--zookeeper.znode.parentparent znode of target hbase

Commands:
 assigns [OPTIONS] ...
   Options:
-o,--override  override ownership by another procedure
   A 'raw' assign that can be used even during Master initialization.
   Skirts Coprocessors. Pass one or more encoded RegionNames.
   1588230740 is the hard-coded name for the hbase:meta region and
   de00010733901a05f5a2a3a382e27dd4 is an example of what a user-space
   encoded Region name looks like. For example:
 $ HBCK2 assign 1588230740 de00010733901a05f5a2a3a382e27dd4
   Returns the pid(s) of the created AssignProcedure(s) or -1 if none.

 bypass [OPTIONS] ...
   Options:
-o,--override   override if procedure is running/stuck
-r,--recursive  bypass parent and its children. SLOW! EXPENSIVE!
-w,--lockWait   milliseconds to wait on lock before giving up;
default=1
   Pass one (or more) procedure 'pid's to skip to procedure finish.
   Parent of bypassed procedure will also be skipped to the finish.
   Entities will be left in an inconsistent state and will require
   manual fixup. May need Master restart to clear locks still held.
   Bypass fails if procedure has children. Add 'recursive' if all
   you have is a parent pid to finish parent and children. This
   is SLOW, and dangerous so use selectively. Does not always work.

 unassigns ...
   Options:
-o,--override  override ownership by another procedure
   A 'raw' unassign that can be used even during Master initialization.
   Skirts Coprocessors. Pass one or more encoded RegionNames:
   1588230740 is the hard-coded name for the hbase:meta region and
   de00010733901a05f5a2a3a382e27dd4 is an example of what a user-space
   encoded Region name looks like. For example:
 $ HBCK2 unassign 1588230740 de00010733901a05f5a2a3a382e27dd4
   Returns the pid(s) of the created UnassignProcedure(s) or -1 if none.

 setTableState  
   Possible table states: ENABLED, DISABLED, DISABLING, ENABLING
   To read current table state, in the hbase shell run:
 hbase> get 'hbase:meta', '', 'table:state'
   A value of \x08\x00 == ENABLED, \x08\x01 == DISABLED, etc.
   An example making table name 'user' ENABLED:
 $ HBCK2 setTableState users ENABLED
   Returns whatever the previous table state was.




On Mon, Oct 8, 2018 at 4:34 PM Stack  wrote:

> On Mon, Oct 8, 2018 at 4:01 PM Josh Elser  wrote:
>
>> Best place to find hbck2 issue needing review is off of HBASE-19121 or
>> somewhere else?
>>
>>
> For 2.1.1 issues, see the 2.1.1 release listing:
> https://issues.apache.org/jira/projects/HBASE/versions/12343470 Half
> these items are items turned up testing branch-2.1 and trying to use hbck2.
> Will link a few others.
>
>
>> All: please feel free to ping directly if you want/need reviews.
>>
>> Will do.
>
> Thanks,
> S
>
>
>
>> On 10/5/18 7:41 PM, 张铎(Duo Zhang) wrote:
>> > Stack has a plan on the 2.1.1 release where we want to finish the first
>> > versio

Re: [DISCUSS] 2.1.1 Release Plans?

2018-10-08 Thread Stack 
On Mon, Oct 8, 2018 at 4:01 PM Josh Elser  wrote:

> Best place to find hbck2 issue needing review is off of HBASE-19121 or
> somewhere else?
>
>
For 2.1.1 issues, see the 2.1.1 release listing:
https://issues.apache.org/jira/projects/HBASE/versions/12343470 Half these
items are items turned up testing branch-2.1 and trying to use hbck2. Will
link a few others.


> All: please feel free to ping directly if you want/need reviews.
>
> Will do.

Thanks,
S



> On 10/5/18 7:41 PM, 张铎(Duo Zhang) wrote:
> > Stack has a plan on the 2.1.1 release where we want to finish the first
> > version on hbck2. In the real deploy we have met a stuck cluster several
> > times, and lots of users have asked that why hbck can not work any
> more...
> >
> > So the current opening issue is not important, please help reviewing the
> > patches for hbck2 to speed up the release...
> >
> > Thanks for bringing this up
> >
> > Mike Drob 于2018年10月5日 周五23:53写道：
> >
> >> Devs,
> >>
> >> It's been almost 3 months since 2.1.0 was released (Jul 19) and we have
> 150
> >> commits on branch-2.1 in that time. What do folks think of getting a
> >> release going? I know that there's been some discussion around the HBCK2
> >> stuff landing, but I feel like the conversation has gotten a bit lost
> >> without an actual release to relate to.
> >>
> >> Duo, as the 2.1.0 release manager, are you interested in maintaining the
> >> 2.1 branch release cadence? If you've gotten busy, then let's find
> another
> >> volunteer.
> >>
> >> There are 18 issues open or in progress currently. Only one is labelled
> >> blocker, and five more are critical -- let's evaluate these and the
> rest to
> >> figure out what we need for a release to happen. I went ahead and
> created a
> >> 2.1.2 version in Jira so that we have somewhere to move issues that
> aren't
> >> getting done soon.
> >>
> >> Meanwhile, I think we also need to look at test stabilization --
> there's 15
> >> tests on the dashboard that might need attention.
> >>
> >> Mike
> >>
> >
>

Re: [DISCUSS] 2.1.1 Release Plans?

2018-10-08 Thread Josh Elser 
Best place to find hbck2 issue needing review is off of HBASE-19121 or 
somewhere else?


All: please feel free to ping directly if you want/need reviews.

On 10/5/18 7:41 PM, 张铎(Duo Zhang) wrote:

Stack has a plan on the 2.1.1 release where we want to finish the first
version on hbck2. In the real deploy we have met a stuck cluster several
times, and lots of users have asked that why hbck can not work any more...

So the current opening issue is not important, please help reviewing the
patches for hbck2 to speed up the release...

Thanks for bringing this up

Mike Drob 于2018年10月5日 周五23:53写道：


Devs,

It's been almost 3 months since 2.1.0 was released (Jul 19) and we have 150
commits on branch-2.1 in that time. What do folks think of getting a
release going? I know that there's been some discussion around the HBCK2
stuff landing, but I feel like the conversation has gotten a bit lost
without an actual release to relate to.

Duo, as the 2.1.0 release manager, are you interested in maintaining the
2.1 branch release cadence? If you've gotten busy, then let's find another
volunteer.

There are 18 issues open or in progress currently. Only one is labelled
blocker, and five more are critical -- let's evaluate these and the rest to
figure out what we need for a release to happen. I went ahead and created a
2.1.2 version in Jira so that we have somewhere to move issues that aren't
getting done soon.

Meanwhile, I think we also need to look at test stabilization -- there's 15
tests on the dashboard that might need attention.

Mike

Re: [DISCUSS] 2.1.1 Release Plans?

2018-10-08 Thread Stack 
On Mon, Oct 8, 2018 at 8:19 AM Mike Drob  wrote:

> Thanks for pointing me to that discussion, Duo. I've updated HBASE-19121 to
> have fix version 2.1.1 and 2.0.3 so that it is easier (at least for me) to
> find via Jira.
>
>
I was thinking HBASE-19121 more an umbrella to hang hbck2 stuff off. Would
suggest we unschedule 2.1.1/2.0.3 as target.


> I'll start looking at the hbck2 patches and will leave comments there.
>
> There are a lot of subtasks against that issue, it is unclear to me which
> ones are critical and which ones are not for a release.
>
>
They usually have a target fix but let me give them another pass. Not all
have to land in 2.1.1/2.0.3.


Since hbck2 tool is in a different repo, have we thought about what the
> release actions would look like? Do we need to have two votes going at the
> same time? Can we release core repo with an hbck service in master, but
> release the hbck tool sometime later? This first release is going to be
> more complex I think; would be good if somebody can figure that out in
> parallel so we're not blocked on it later.
>
>
Yeah, the point of hbck2 being a separate repo was so they could have
different release cadences. It would be good if the first hbck2 release
happened around 2.1.1 release time. I can work on this part.

S



> Mike
>
> On Fri, Oct 5, 2018 at 6:41 PM 张铎(Duo Zhang) 
> wrote:
>
> > Stack has a plan on the 2.1.1 release where we want to finish the first
> > version on hbck2. In the real deploy we have met a stuck cluster several
> > times, and lots of users have asked that why hbck can not work any
> more...
> >
> > So the current opening issue is not important, please help reviewing the
> > patches for hbck2 to speed up the release...
> >
> > Thanks for bringing this up
> >
> > Mike Drob 于2018年10月5日 周五23:53写道：
> >
> > > Devs,
> > >
> > > It's been almost 3 months since 2.1.0 was released (Jul 19) and we have
> > 150
> > > commits on branch-2.1 in that time. What do folks think of getting a
> > > release going? I know that there's been some discussion around the
> HBCK2
> > > stuff landing, but I feel like the conversation has gotten a bit lost
> > > without an actual release to relate to.
> > >
> > > Duo, as the 2.1.0 release manager, are you interested in maintaining
> the
> > > 2.1 branch release cadence? If you've gotten busy, then let's find
> > another
> > > volunteer.
> > >
> > > There are 18 issues open or in progress currently. Only one is labelled
> > > blocker, and five more are critical -- let's evaluate these and the
> rest
> > to
> > > figure out what we need for a release to happen. I went ahead and
> > created a
> > > 2.1.2 version in Jira so that we have somewhere to move issues that
> > aren't
> > > getting done soon.
> > >
> > > Meanwhile, I think we also need to look at test stabilization --
> there's
> > 15
> > > tests on the dashboard that might need attention.
> > >
> > > Mike
> > >
> >
>

Re: [DISCUSS] 2.1.1 Release Plans?

2018-10-08 Thread Stack 
On Fri, Oct 5, 2018 at 8:53 AM Mike Drob  wrote:

> Devs,
>
> It's been almost 3 months since 2.1.0 was released (Jul 19) and we have 150
> commits on branch-2.1 in that time. What do folks think of getting a
> release going? I know that there's been some discussion around the HBCK2
> stuff landing, but I feel like the conversation has gotten a bit lost
> without an actual release to relate to.
>

Sounds good. I was under the delusion that I was going to help push 2.1.1
out but I went off to work on hbck2 and its been taking a while figuring
what would comprise a first cut at server-side support for a useable hbck2.
I was thinking that 2.1.1 would have our first cut of APIs for a useable
HBCK2.

This project has been coming along with distractions as I run into fun,
dirty bugs in branch-2.1.


>
> There are 18 issues open or in progress currently. Only one is labelled
> blocker, and five more are critical -- let's evaluate these and the rest to
> figure out what we need for a release to happen. I went ahead and created a
> 2.1.2 version in Jira so that we have somewhere to move issues that aren't
> getting done soon.
>
> Meanwhile, I think we also need to look at test stabilization -- there's 15
> tests on the dashboard that might need attention.
>
>
Thanks for doing the above. This is work that needs to be done regardless.
S


> Mike
>

Re: [DISCUSS] 2.1.1 Release Plans?

2018-10-08 Thread Andrew Purtell 
If you need a RM to do ~monthly cadence releases of _one_ 2.x line, I can
volunteer for that. I intend to continue releasing from branch-1 but seeing
how I'm set up to do monthly releases of one set of binaries I can do
another with only a modest increment of effort.

That said, it would be great if we had more people acting as RMs. You only
need commit rights to be a RM, so any committer is welcome to try
releasing. If you are interested, one of us, such as myself, can mentor you
and walk you through the process for the first couple of releases.


On Mon, Oct 8, 2018 at 8:31 AM Mike Drob  wrote:

> Another thought that I realized I forgot to mention -
>
> Previously on branch-1 we've been fairly consistent with time-bound
> maintenance releases happening almost monthly, looking at the great work
> done by our Andrew and Nick especially.
>
> We've fallen off track on branch-2 with maintenance releases on 2.0.x being
> fairly high in patch count. Although not perfect, I think this is a
> reasonable first order approximation for the amount of change going in. If
> we're not releasing often enough (or at all) in this situation, then folks
> that are trying to use it on real clusters don't have access to the fixes.
> Agree that it is bad they don't have hbck to fix some problems, but as they
> run they will keep finding more and more new issues and even running into
> the same issues because they have no better release to upgrade to.
>
> So theoretically we could get some fixes out to folks now to make their
> lives a little bit easier, and then get them another release with hbck2
> when it is ready.
>
> Mike
>
> On Mon, Oct 8, 2018 at 10:19 AM Mike Drob  wrote:
>
> > Thanks for pointing me to that discussion, Duo. I've updated HBASE-19121
> > to have fix version 2.1.1 and 2.0.3 so that it is easier (at least for
> me)
> > to find via Jira.
> >
> > I'll start looking at the hbck2 patches and will leave comments there.
> >
> > There are a lot of subtasks against that issue, it is unclear to me which
> > ones are critical and which ones are not for a release.
> >
> > Since hbck2 tool is in a different repo, have we thought about what the
> > release actions would look like? Do we need to have two votes going at
> the
> > same time? Can we release core repo with an hbck service in master, but
> > release the hbck tool sometime later? This first release is going to be
> > more complex I think; would be good if somebody can figure that out in
> > parallel so we're not blocked on it later.
> >
> > Mike
> >
> > On Fri, Oct 5, 2018 at 6:41 PM 张铎(Duo Zhang) 
> > wrote:
> >
> >> Stack has a plan on the 2.1.1 release where we want to finish the first
> >> version on hbck2. In the real deploy we have met a stuck cluster several
> >> times, and lots of users have asked that why hbck can not work any
> more...
> >>
> >> So the current opening issue is not important, please help reviewing the
> >> patches for hbck2 to speed up the release...
> >>
> >> Thanks for bringing this up
> >>
> >> Mike Drob 于2018年10月5日 周五23:53写道：
> >>
> >> > Devs,
> >> >
> >> > It's been almost 3 months since 2.1.0 was released (Jul 19) and we
> have
> >> 150
> >> > commits on branch-2.1 in that time. What do folks think of getting a
> >> > release going? I know that there's been some discussion around the
> HBCK2
> >> > stuff landing, but I feel like the conversation has gotten a bit lost
> >> > without an actual release to relate to.
> >> >
> >> > Duo, as the 2.1.0 release manager, are you interested in maintaining
> the
> >> > 2.1 branch release cadence? If you've gotten busy, then let's find
> >> another
> >> > volunteer.
> >> >
> >> > There are 18 issues open or in progress currently. Only one is
> labelled
> >> > blocker, and five more are critical -- let's evaluate these and the
> >> rest to
> >> > figure out what we need for a release to happen. I went ahead and
> >> created a
> >> > 2.1.2 version in Jira so that we have somewhere to move issues that
> >> aren't
> >> > getting done soon.
> >> >
> >> > Meanwhile, I think we also need to look at test stabilization --
> >> there's 15
> >> > tests on the dashboard that might need attention.
> >> >
> >> > Mike
> >> >
> >>
> >
>


-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

Re: [DISCUSS] 2.1.1 Release Plans?

2018-10-08 Thread Mike Drob 
Another thought that I realized I forgot to mention -

Previously on branch-1 we've been fairly consistent with time-bound
maintenance releases happening almost monthly, looking at the great work
done by our Andrew and Nick especially.

We've fallen off track on branch-2 with maintenance releases on 2.0.x being
fairly high in patch count. Although not perfect, I think this is a
reasonable first order approximation for the amount of change going in. If
we're not releasing often enough (or at all) in this situation, then folks
that are trying to use it on real clusters don't have access to the fixes.
Agree that it is bad they don't have hbck to fix some problems, but as they
run they will keep finding more and more new issues and even running into
the same issues because they have no better release to upgrade to.

So theoretically we could get some fixes out to folks now to make their
lives a little bit easier, and then get them another release with hbck2
when it is ready.

Mike

On Mon, Oct 8, 2018 at 10:19 AM Mike Drob  wrote:

> Thanks for pointing me to that discussion, Duo. I've updated HBASE-19121
> to have fix version 2.1.1 and 2.0.3 so that it is easier (at least for me)
> to find via Jira.
>
> I'll start looking at the hbck2 patches and will leave comments there.
>
> There are a lot of subtasks against that issue, it is unclear to me which
> ones are critical and which ones are not for a release.
>
> Since hbck2 tool is in a different repo, have we thought about what the
> release actions would look like? Do we need to have two votes going at the
> same time? Can we release core repo with an hbck service in master, but
> release the hbck tool sometime later? This first release is going to be
> more complex I think; would be good if somebody can figure that out in
> parallel so we're not blocked on it later.
>
> Mike
>
> On Fri, Oct 5, 2018 at 6:41 PM 张铎(Duo Zhang) 
> wrote:
>
>> Stack has a plan on the 2.1.1 release where we want to finish the first
>> version on hbck2. In the real deploy we have met a stuck cluster several
>> times, and lots of users have asked that why hbck can not work any more...
>>
>> So the current opening issue is not important, please help reviewing the
>> patches for hbck2 to speed up the release...
>>
>> Thanks for bringing this up
>>
>> Mike Drob 于2018年10月5日 周五23:53写道：
>>
>> > Devs,
>> >
>> > It's been almost 3 months since 2.1.0 was released (Jul 19) and we have
>> 150
>> > commits on branch-2.1 in that time. What do folks think of getting a
>> > release going? I know that there's been some discussion around the HBCK2
>> > stuff landing, but I feel like the conversation has gotten a bit lost
>> > without an actual release to relate to.
>> >
>> > Duo, as the 2.1.0 release manager, are you interested in maintaining the
>> > 2.1 branch release cadence? If you've gotten busy, then let's find
>> another
>> > volunteer.
>> >
>> > There are 18 issues open or in progress currently. Only one is labelled
>> > blocker, and five more are critical -- let's evaluate these and the
>> rest to
>> > figure out what we need for a release to happen. I went ahead and
>> created a
>> > 2.1.2 version in Jira so that we have somewhere to move issues that
>> aren't
>> > getting done soon.
>> >
>> > Meanwhile, I think we also need to look at test stabilization --
>> there's 15
>> > tests on the dashboard that might need attention.
>> >
>> > Mike
>> >
>>
>

Re: [DISCUSS] 2.1.1 Release Plans?

2018-10-08 Thread Mike Drob 
Thanks for pointing me to that discussion, Duo. I've updated HBASE-19121 to
have fix version 2.1.1 and 2.0.3 so that it is easier (at least for me) to
find via Jira.

I'll start looking at the hbck2 patches and will leave comments there.

There are a lot of subtasks against that issue, it is unclear to me which
ones are critical and which ones are not for a release.

Since hbck2 tool is in a different repo, have we thought about what the
release actions would look like? Do we need to have two votes going at the
same time? Can we release core repo with an hbck service in master, but
release the hbck tool sometime later? This first release is going to be
more complex I think; would be good if somebody can figure that out in
parallel so we're not blocked on it later.

Mike

On Fri, Oct 5, 2018 at 6:41 PM 张铎(Duo Zhang)  wrote:

> Stack has a plan on the 2.1.1 release where we want to finish the first
> version on hbck2. In the real deploy we have met a stuck cluster several
> times, and lots of users have asked that why hbck can not work any more...
>
> So the current opening issue is not important, please help reviewing the
> patches for hbck2 to speed up the release...
>
> Thanks for bringing this up
>
> Mike Drob 于2018年10月5日 周五23:53写道：
>
> > Devs,
> >
> > It's been almost 3 months since 2.1.0 was released (Jul 19) and we have
> 150
> > commits on branch-2.1 in that time. What do folks think of getting a
> > release going? I know that there's been some discussion around the HBCK2
> > stuff landing, but I feel like the conversation has gotten a bit lost
> > without an actual release to relate to.
> >
> > Duo, as the 2.1.0 release manager, are you interested in maintaining the
> > 2.1 branch release cadence? If you've gotten busy, then let's find
> another
> > volunteer.
> >
> > There are 18 issues open or in progress currently. Only one is labelled
> > blocker, and five more are critical -- let's evaluate these and the rest
> to
> > figure out what we need for a release to happen. I went ahead and
> created a
> > 2.1.2 version in Jira so that we have somewhere to move issues that
> aren't
> > getting done soon.
> >
> > Meanwhile, I think we also need to look at test stabilization -- there's
> 15
> > tests on the dashboard that might need attention.
> >
> > Mike
> >
>

Re: [DISCUSS] 2.1.1 Release Plans?

2018-10-05 Thread Duo Zhang 
Stack has a plan on the 2.1.1 release where we want to finish the first
version on hbck2. In the real deploy we have met a stuck cluster several
times, and lots of users have asked that why hbck can not work any more...

So the current opening issue is not important, please help reviewing the
patches for hbck2 to speed up the release...

Thanks for bringing this up

Mike Drob 于2018年10月5日 周五23:53写道：

> Devs,
>
> It's been almost 3 months since 2.1.0 was released (Jul 19) and we have 150
> commits on branch-2.1 in that time. What do folks think of getting a
> release going? I know that there's been some discussion around the HBCK2
> stuff landing, but I feel like the conversation has gotten a bit lost
> without an actual release to relate to.
>
> Duo, as the 2.1.0 release manager, are you interested in maintaining the
> 2.1 branch release cadence? If you've gotten busy, then let's find another
> volunteer.
>
> There are 18 issues open or in progress currently. Only one is labelled
> blocker, and five more are critical -- let's evaluate these and the rest to
> figure out what we need for a release to happen. I went ahead and created a
> 2.1.2 version in Jira so that we have somewhere to move issues that aren't
> getting done soon.
>
> Meanwhile, I think we also need to look at test stabilization -- there's 15
> tests on the dashboard that might need attention.
>
> Mike
>