[jira] [Created] (HBASE-24772) Use GetoptLong or OptionParser in hbase-shell

[Elliot Miller (Jira)]

Elliot Miller created HBASE-24772:
-

 Summary: Use GetoptLong or OptionParser in hbase-shell
 Key: HBASE-24772
 URL: https://issues.apache.org/jira/browse/HBASE-24772
 Project: HBase
  Issue Type: Task
  Components: shell
Affects Versions: 2.3.0, 3.0.0-alpha-1
Reporter: Elliot Miller


Currently, our hbase-shell command line argument parser is custom-rolled. It 
would be awesome to instead use Ruby's GetoptLong or OptionParser.
 * [https://ruby-doc.org/stdlib-2.3.0/libdoc/getoptlong/rdoc/GetoptLong.html]
 * [https://ruby-doc.org/stdlib-2.3.0/libdoc/optparse/rdoc/OptionParser.html]

There is a long-standing {{FIXME}} comment in {{bin/hirb.rb}} to address this: 
[https://github.com/apache/hbase/blob/975cdf7b88f001aa51d83395c663892f3b5ffbdb/bin/hirb.rb#L50]

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24771) Investigate Potential to Use Ruby's Help Functionality in Shell

[Elliot Miller (Jira)]

Elliot Miller created HBASE-24771:
-

 Summary: Investigate Potential to Use Ruby's Help Functionality in 
Shell
 Key: HBASE-24771
 URL: https://issues.apache.org/jira/browse/HBASE-24771
 Project: HBase
  Issue Type: Task
  Components: shell
Affects Versions: 2.3.0, 3.0.0-alpha-1
Reporter: Elliot Miller


Ruby and Interactive Ruby (irb) have functionality for helping users (via 
introspection). It would be interesting to see if we can hook into this for 
hbase-shell.
 * Ruby has a built in method called "ri" which lets users explore help text 
for modules, classes, and methods from RDocs.
 ** [https://ruby.github.io/rdoc/RI_rdoc.html]
 * Interactive ruby has a method called "irb_help" which gets patched onto the 
main Object as "help" (except in our shell, where we have our own help method). 
Under the hood, it uses {{Ri}}.
 ** [https://docs.ruby-lang.org/en/2.3.0/IRB/ExtendCommandBundle.html]
 ** 
[https://github.com/ruby/irb/blob/66c4eab4e71b47f3c78ed68727c0bd1c3897591d/lib/irb/cmd/help.rb]

At the moment, {{irb_help}}/{{ri}} knows nothing about the HBase shell commands
{code:java}
hbase> irb_help

Enter the method name you want to look up.
You can use tab to autocomplete.
Enter a blank line to exit.

>> list
Nothing known about .list
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24770) Reimplement the Constraints API and revisit the IA annotations on related classes

[Duo Zhang (Jira)]

Duo Zhang created HBASE-24770:
-

 Summary: Reimplement the Constraints API and revisit the IA 
annotations on related classes
 Key: HBASE-24770
 URL: https://issues.apache.org/jira/browse/HBASE-24770
 Project: HBase
  Issue Type: Task
  Components: Client, Coprocessors
Reporter: Duo Zhang
Assignee: Duo Zhang
 Fix For: 3.0.0-alpha-1






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: On org.apache.hadoop.hbase.constraint

[Duo Zhang]

OK, seems no response. Will file an issue to redefine the API and change
the IA annotations.

张铎(Duo Zhang)  于2020年6月8日周一 上午10:19写道：

> Let's open an issue for this feature?
>
> Let's wait for a week, if still no use cases from users, I think we can do
> some breaking changes on 3.0.0 to open the feature to end users. On 2.x,
> let's still keep the old classes.
>
> Thanks.
>
> Sean Busbey  于2020年6月8日周一 上午9:39写道：
>
>> I think it being labeled IA.Private is incorrect.
>>
>> The red guide talks about the feature and directly points folks to the
>> javadoc of one of the IA.Private classes.
>>
>> http://hbase.apache.org/book.html#_constraints
>>
>> I'm all for us figuring out what the public surface should be and
>> correcting this gap, but we need to be mindful as though it were more
>> public than that annotation claims.
>>
>> On Sun, Jun 7, 2020, 19:30 Stack  wrote:
>>
>> > If IA.Private, it was for internal use only? Doesn't need a deprecation
>> to
>> > change it?
>> >
>> > Please speak up if you are using the Constraint feature!
>> >
>> > Thanks,
>> > S
>> >
>> > On Sat, Jun 6, 2020 at 12:40 AM 张铎(Duo Zhang) 
>> > wrote:
>> >
>> > > The related classes are marked as IA.Private which means it is not
>> part
>> > of
>> > > our public API...
>> > >
>> > > That's why I check for shell support, as if there is no shell support,
>> > then
>> > > users have no way to make use of it without breaking the
>> > InterfaceAudience
>> > > rule...
>> > >
>> > > Jesse Yates  于2020年6月6日周六 上午1:04写道：
>> > >
>> > > > Not particularly. Just because there is no shell integration though,
>> > > > doesn't mean it isn't used -  it has been around for 5 years, which
>> > means
>> > > > someone likely has picked it up. You should probably ask on the user
>> > list
>> > > > and/or do a deprecation cycle to before just removing.
>> > > > ---
>> > > > Jesse Yates
>> > > > @jesse_yates
>> > > > jesseyates.com 
>> > > >
>> > > >
>> > > > On Fri, Jun 5, 2020 at 8:50 AM 张铎(Duo Zhang) > >
>> > > > wrote:
>> > > >
>> > > > > Seems only this issue has been finished.
>> > > > >
>> > > > > https://issues.apache.org/jira/browse/HBASE-4605
>> > > > >
>> > > > > Which brought in these classes, but the later approach on adding
>> > shell
>> > > > > support had been resolved as incomplete.
>> > > > >
>> > > > > https://issues.apache.org/jira/browse/HBASE-4879
>> > > > >
>> > > > > So I guess there is no actual use in HBase yet.
>> > > > >
>> > > > > Do you still want to finish this feature?
>> > > > >
>> > > > > Thanks.
>> > > > >
>> > > > > Jesse Yates  于2020年6月5日周五 下午11:29写道：
>> > > > >
>> > > > > > Here is the original JIRA for the constraint work:
>> > > > > > https://issues.apache.org/jira/browse/HBASE-4999
>> > > > > >
>> > > > > > Its a common DB feature, but not sure if folks actually use it
>> in
>> > > > HBase.
>> > > > > > ---
>> > > > > > Jesse Yates
>> > > > > > @jesse_yates
>> > > > > > jesseyates.com 
>> > > > > >
>> > > > > >
>> > > > > > On Fri, Jun 5, 2020 at 4:06 AM 张铎(Duo Zhang) <
>> > palomino...@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > When removing HTableDescriptor on master branch, I found that
>> it
>> > > has
>> > > > > been
>> > > > > > > referenced by org.apache.hadoop.hbase.constraint package.
>> > > > > > >
>> > > > > > > The problem here is that, the API design is to pass in an
>> > > > > > HTableDescriptor
>> > > > > > > and modify it directly, but now, TableDescriptor is
>> immutable, so
>> > > we
>> > > > > need
>> > > > > > > to redesign the API.
>> > > > > > >
>> > > > > > > But the problem is that, all the classes are marked as
>> > IA.Private,
>> > > > and
>> > > > > > the
>> > > > > > > only references to these classes are in the test code. And I
>> can
>> > > not
>> > > > > find
>> > > > > > > any useful information through the git log, the earliest one
>> is
>> > > > > > >
>> > > > > > > commit 390f32d79fd0c0464fcab008150ad182f4c0abef
>> > > > > > > Author: Michael Stack 
>> > > > > > > Date:   Sat May 26 05:56:04 2012 +
>> > > > > > >
>> > > > > > > HBASE-4336 Convert source tree into maven modules
>> > > > > > >
>> > > > > > > git-svn-id:
>> > > https://svn.apache.org/repos/asf/hbase/trunk@1342856
>> > > > > > > 13f79535-47bb-0310-9956-ffa450edef68
>> > > > > > > <
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://svn.apache.org/repos/asf/hbase/trunk@134285613f79535-47bb-0310-9956-ffa450edef68
>> > > > > > >
>> > > > > > >
>> > > > > > > Does anyone still use this feature? Or does anyone have some
>> > > > background
>> > > > > > on
>> > > > > > > how this feature works?
>> > > > > > >
>> > > > > > > Thanks.
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

[jira] [Resolved] (HBASE-20835) Document how to get replication reporting

[Duo Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-20835.
---
Fix Version/s: (was: 3.0.0-alpha-1)
   Resolution: Later

Can reopen later if this is still a problem.

> Document how to get replication reporting
> -
>
> Key: HBASE-20835
> URL: https://issues.apache.org/jira/browse/HBASE-20835
> Project: HBase
>  Issue Type: Task
>  Components: Replication
>Affects Versions: 2.1.0
>Reporter: Mike Drob
>Assignee: Duo Zhang
>Priority: Critical
>
> Based on my questions at the tail end of HBASE-19543
> bq. We have some tooling that checks on replication queues and reads the 
> znode as the source of truth. When replication is disabled, it's expected 
> that the node was still there, but just empty. Is there a better way to get 
> this same information?
> I understand that with table based replication it doesn't make sense to check 
> ZK for status. However, losing the ability to inspect the data and get 
> information is a tough hit for operators. Do we have APIs that expose the 
> same sort of metrics?
> bq. how many peers/queues, queue size, position in the queue, and age of last 
> op
> Assigning to you for now, Duo, since you were both primary implementor and RM 
> for 2.1.0 and I'm not sure who else would know the answers. If the docs 
> already exist, then nothing to do but we should include them in the RN. Maybe 
> this will need additional code, but I hope it's already there and is 
> something we can write a workaround for.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-21788) OpenRegionProcedure (after recovery?) is unreliable and needs to be improved

[Duo Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-21788.
---
Resolution: Incomplete

We have done a bunch of fixes on AMv2 and PV2, resolve for now as no new 
findings for a long time.

> OpenRegionProcedure (after recovery?) is unreliable and needs to be improved
> 
>
> Key: HBASE-21788
> URL: https://issues.apache.org/jira/browse/HBASE-21788
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1
>Reporter: Sergey Shelukhin
>Assignee: Michael Stack
>Priority: Critical
> Attachments: WAL-Orphan.log
>
>
> Not much for this one yet.
> I repeatedly see the cases when the region is stuck in OPENING, and after 
> master restart RIT is recovered, and stays WAITING; its OpenRegionProcedure 
> (also recovered) is stuck in Runnable and never does anything for hours. I 
> cannot find logs on the target server indicating that it ever tried to do 
> anything after master restart.
> This procedure needs at the very least logging of what it's trying to do, and 
> maybe a timeout so it unconditionally fails after a configurable period (1 
> hour?).
> I may also investigate why it doesn't do anything and file a separate bug. I 
> wonder if it's somehow related to the region status check, but this is just a 
> hunch.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-19952) Find tests which are declared with wrong category

[Duo Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-19952.
---
Resolution: Later

Now we set the timeout to 13 min for all categories. Can open later if we still 
want to have different timeouts for them.

> Find tests which are declared with wrong category
> -
>
> Key: HBASE-19952
> URL: https://issues.apache.org/jira/browse/HBASE-19952
> Project: HBase
>  Issue Type: Bug
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Attachments: category-report
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-22175) Unify the old PreCommit-HBase-Build and the new GitHub PR based PreCommit

[Duo Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-22175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-22175.
---
Resolution: Not A Problem

We only support PR now.

> Unify the old PreCommit-HBase-Build and the new GitHub PR based PreCommit
> -
>
> Key: HBASE-22175
> URL: https://issues.apache.org/jira/browse/HBASE-22175
> Project: HBase
>  Issue Type: Sub-task
>  Components: build
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HBASE-22175-HBASE-22175.patch
>
>
> We should do this to keep the pre commit behavior the same between GitHub PR 
> and the old Patch Available on jira.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-23889) Switch back to ZKConnectionRegistry by default at least in test

[Duo Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-23889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-23889.
---
Resolution: Won't Fix

> Switch back to ZKConnectionRegistry by default at least in test
> ---
>
> Key: HBASE-23889
> URL: https://issues.apache.org/jira/browse/HBASE-23889
> Project: HBase
>  Issue Type: Bug
>  Components: Client, rpc, test
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> For now, MasterRegistry can not deal with master restart, as it can not load 
> the new master address automatically.
> I see there is a invalidateConnection method in HBaseTestingUtilities but it 
> needs a very big refactoring to make all the UTs work like this.
> So here I suggest we switch back to ZKConnectionRegistry by default, and open 
> a new feature branch to finish the TODOs and the refactoring on UTs.
> As now it is already a big problem for me as I want to merge a feature branch 
> back to master but the state of UTs are a mess.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-23921) Findbugs is OOM on master

[Duo Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-23921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-23921.
---
Resolution: Fixed

The problem has been solved by later jenkins pipeline changes. And since we 
haven't revert the PR here, set it as fixed...

> Findbugs is OOM on master
> -
>
> Key: HBASE-23921
> URL: https://issues.apache.org/jira/browse/HBASE-23921
> Project: HBase
>  Issue Type: Bug
>  Components: build, findbugs
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0-alpha-1
>
>
> Happens after merging of HBASE-22514.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-21994) Expose TableDescriptors to master coprocessor

[Duo Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-21994.
---
Resolution: Later

Can reopen later if we want to use TableDescriptors in CP in the future.

> Expose TableDescriptors to master coprocessor
> -
>
> Key: HBASE-21994
> URL: https://issues.apache.org/jira/browse/HBASE-21994
> Project: HBase
>  Issue Type: Improvement
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HBASE-21994-v1.patch, HBASE-21994-v2.patch, 
> HBASE-21994.patch
>
>
> When implementing HBASE-21717, I found that AccessController will use the 
> Admin interface to get table descriptors on master. We can expose the read 
> only methods in TableDescritors to CPs so user can get the table descriptor 
> directly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] we need to take action if we want asf jenkins managed tests after Aug 15 2020.

[Duo Zhang]

The problem seems because of this:

https://issues.jenkins-ci.org/browse/JENKINS-48556

I triggered the job again, it passed the timestamps call, and will keep an
eye on it.

张铎(Duo Zhang)  于2020年7月21日周二 上午11:18写道：

> On the sponsors, we could have a try.
>
> The problem here is the process of the donation? IIRC there is a thread on
> the infra mailing list about how to donate machines to a specific project
> and the discussion did not go well...
>
> Sean Busbey  于2020年7月21日周二 上午11:13写道：
>
>> We could check with ASF infra for the current state of things wrt GitHub
>> actions. I believe there is a queue set up across ASF projects.
>>
>> It has the same resource issue Travis had; things are fine until some
>> critical mass of projects seeking better perf realize some new option is
>> available and then quickly all available resources are consumed.
>>
>> AFAICT the only option that gets us the same or better as the H* nodes
>> will
>> be finding sponsors and running our own.
>>
>> On Mon, Jul 20, 2020, 21:55 张铎(Duo Zhang)  wrote:
>>
>> > I think our nightly, flakey, and pre commit jobs should be transferred
>> as a
>> > whole? They depend on each other.
>> >
>> > I offer my help on the transition.
>> >
>> > And on github CI, does ASF have a special deal with github? If not, I do
>> > not think the default resource can fit our requirements...
>> >
>> >
>> >
>> > Sean Busbey  于2020年7月21日周二 上午1:49写道：
>> >
>> > > Hi folks!
>> > >
>> > > Back in April there was a brief discussion[1] about ASF Infra's
>> > > notification that builds.a.o is going away and we are currently slated
>> > > to migrate to a set of CI servers for "Hadoop and related projects".
>> > > This is the ci farm that will contain the bulk of the H* worker nodes
>> > > that are donated by Yahoo!, which are the nodes we've been running on
>> > > for ages[2].
>> > >
>> > > Migration discussion still happens on the hadoop-migrations@i.a.o
>> > > list[3] and recently ASF Infra set a target date of August 15th for
>> > > turning off the existing builds.a.o server[4].
>> > >
>> > > That gives us a little under 4 weeks to have things up and working on
>> > > the new ci-hadoop.a.o jenkins coordinator[5]. it’s not clear to me
>> > > that the level of effort we’ll need to spend is worth what we get out
>> > > of a continuation of the status quo on builds.a.o. I did a quick test
>> > > by updating the nightly job on ci-hadoop.a.o to run just branch-2,
>> > > since that has been stable on builds.a.o. It failed with a Jenkins
>> > > pipeline DSL syntax error[6] so I'm assuming migrating will be a slog.
>> > >
>> > > As far as I can see our options are:
>> > >
>> > > * Do nothing. Have no testing or automated website publication in mid
>> > > August.
>> > > * Transition website publication and nothing else (probably can be
>> > > done in a day)
>> > > * Transition just precommit testing for various repos (probably can be
>> > > done in a few days)
>> > > * Transition everything (no idea how long it takes due to nightly,
>> > > flaky stuff, etc)
>> > >
>> > > The alternatives if we do not transition any given job to ci-hadoop:
>> > >
>> > > * Try to move to GitHub Actions
>> > > * Try to move to Travis CI
>> > > * Try to move to Jenkins infra we maintain ourselves (presumably by
>> > > soliciting project specific donations for worker nodes on cloud
>> > > vendors)
>> > >
>> > > It's important to remember that as a project we have a heavy footprint
>> > > wherever our nightly tests run. For context, a given branch's nightly
>> > > can keep 3-4 executors busy for 6+ hours on the current builds.a.o
>> > > setup. There's been a bunch of great work lately on bringing down what
>> > > it takes to run the full test suite, but applying that work to nightly
>> > > is itself a significant undertaking.
>> > >
>> > > What are folks thinking? Most importantly who is ready to work towards
>> > > any given approach?
>> > >
>> > > [1] [DISCUSS] Migrating HBase to new CI Master
>> > > https://s.apache.org/fux1o
>> > >
>> > > [2] https://builds.apache.org/view/H-L/view/HBase/
>> > >
>> > > [3]
>> > https://lists.apache.org/list.html?hadoop-migrati...@infra.apache.org
>> > >
>> > > [4] [IMPORTANT] - 2 more HADOOP nodes migrated over to ci-hadoop
>> > > https://s.apache.org/7e1nq
>> > >
>> > > [5] https://ci-hadoop.apache.org/job/HBase/
>> > >
>> > > [6]
>> > >
>> >
>> https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/2/console
>> > >
>> >
>>
>

[jira] [Created] (HBASE-24769) Auto scale RSGroup

[Sun Xin (Jira)]

Sun Xin created HBASE-24769:
---

 Summary: Auto scale RSGroup
 Key: HBASE-24769
 URL: https://issues.apache.org/jira/browse/HBASE-24769
 Project: HBase
  Issue Type: New Feature
  Components: rsgroup
Affects Versions: 3.0.0-alpha-1
Reporter: Sun Xin
Assignee: Sun Xin
 Fix For: 3.0.0-alpha-1


In current use, if RSs go offline or online, we must manually move RSs in or 
out RSGroups.

Now we can configure how many servers rsgroups need base on HBASE-24431 , and 
then add an AutoScaleChore to periodically check and move servers.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)