[jira] [Created] (HBASE-13688) Procedure v2 - Reimplement label and auth cache updates

[Andrew Purtell (JIRA)]

Andrew Purtell created HBASE-13688:
--

 Summary: Procedure v2 - Reimplement label and auth cache updates
 Key: HBASE-13688
 URL: https://issues.apache.org/jira/browse/HBASE-13688
 Project: HBase
  Issue Type: Sub-task
Reporter: Andrew Purtell


Use Procedure v2 to ensure that caches in the VisibilityControllers are 
consistent after addLabels, setAuths, and clearAuths



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-13687) Procedure V2 - Reimplement permissions caching on grant/revoke

[Andrew Purtell (JIRA)]

Andrew Purtell created HBASE-13687:
--

 Summary: Procedure V2 - Reimplement permissions caching on 
grant/revoke
 Key: HBASE-13687
 URL: https://issues.apache.org/jira/browse/HBASE-13687
 Project: HBase
  Issue Type: Sub-task
Reporter: Andrew Purtell


Use Procedure V2 to ensure cached perms are consistent after grant and revoke.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Hbase client create many threads

[龚俊衡]

Hi,
When batch get some data from cluster, the client will block some threads base 
on region server amount
In code client use Future.get block a thread for each related region servers 
and wait them finish one by one.
In our case we have 12 region servers, so most of time the client will cost 12 
threads for a batch get, if we have many concurrent request it will OOM by can 
not create native threads.
I suggest we use some kind async or call back to process region server's 
response.

[jira] [Created] (HBASE-13686) Fail to limit rate in RateLimiter

[Guanghao Zhang (JIRA)]

Guanghao Zhang created HBASE-13686:
--

 Summary: Fail to limit rate in RateLimiter
 Key: HBASE-13686
 URL: https://issues.apache.org/jira/browse/HBASE-13686
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.0.0, 1.1.0
Reporter: Guanghao Zhang
Priority: Minor


While using the patch in HBASE-11598 , I found that RateLimiter can't to limit 
the rate right.
{code} 
 /**
   * given the time interval, are there enough available resources to allow 
execution?
   * @param now the current timestamp
   * @param lastTs the timestamp of the last update
   * @param amount the number of required resources
   * @return true if there are enough available resources, otherwise false
   */
  public synchronized boolean canExecute(final long now, final long lastTs, 
final long amount) {
return avail >= amount ? true : refill(now, lastTs) >= amount;
  }
{code}
When avail >= amount, avail can't be refill. But in the next time to call 
canExecute, lastTs maybe update. So avail will waste some time to refill. Even 
we use smaller rate than the limit, the canExecute will return false. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Troubles with HBase 1.1.0 RC2

[Enis Söztutar]

Yeah, for coprocessors, what Andrew said. You have to make minor changes.

>From your repo, I was able to build:

HW10676:hbase-deps-test$ ./build.sh

:compileJava

Download
https://repository.apache.org/content/repositories/orgapachehbase-1078/org/apache/hbase/hbase/1.1.0/hbase-1.1.0.pom

Download
https://repository.apache.org/content/repositories/orgapachehbase-1078/org/apache/hbase/hbase-server/1.1.0/hbase-server-1.1.0.jar

Download
https://repository.apache.org/content/repositories/orgapachehbase-1078/org/apache/hbase/hbase-common/1.1.0/hbase-common-1.1.0.jar

Download
https://repository.apache.org/content/repositories/orgapachehbase-1078/org/apache/hbase/hbase-common/1.1.0/hbase-common-1.1.0-tests.jar

Download
https://repository.apache.org/content/repositories/orgapachehbase-1078/org/apache/hbase/hbase-protocol/1.1.0/hbase-protocol-1.1.0.jar

Download
https://repository.apache.org/content/repositories/orgapachehbase-1078/org/apache/hbase/hbase-procedure/1.1.0/hbase-procedure-1.1.0.jar

Download
https://repository.apache.org/content/repositories/orgapachehbase-1078/org/apache/hbase/hbase-client/1.1.0/hbase-client-1.1.0.jar

Download
https://repository.apache.org/content/repositories/orgapachehbase-1078/org/apache/hbase/hbase-prefix-tree/1.1.0/hbase-prefix-tree-1.1.0.jar

Download
https://repository.apache.org/content/repositories/orgapachehbase-1078/org/apache/hbase/hbase-hadoop-compat/1.1.0/hbase-hadoop-compat-1.1.0.jar

Download
https://repository.apache.org/content/repositories/orgapachehbase-1078/org/apache/hbase/hbase-hadoop2-compat/1.1.0/hbase-hadoop2-compat-1.1.0.jar

Download
https://repository.apache.org/content/repositories/orgapachehbase-1078/org/apache/hbase/hbase-annotations/1.1.0/hbase-annotations-1.1.0.jar

:processResources UP-TO-DATE

:classes

:jar

:assemble

:compileTestJava UP-TO-DATE

:processTestResources UP-TO-DATE

:testClasses UP-TO-DATE

:test UP-TO-DATE

:check UP-TO-DATE

:build


BUILD SUCCESSFUL


Total time: 1 mins 8.182 secs


Also you should not need to pass -Dcompat.module=hbase-hadoop2-compat.

Enis

On Wed, May 13, 2015 at 3:21 PM, Andrew Purtell  wrote:

> > So, it looks like RegionCoprocessorEnvironment.getRegion() has been
> removed?
>
> No, the signature has changed, basically s/HRegion/Region/. HRegion is an
> internal, low level implementation type. Has always been. We have replaced
> it with Region, an interface that contains a subset of HRegion we feel we
> can support for coprocessor source and binary compatibility longer term.
> This work was done on HBASE-12972 if you're curious to know more about it.
>
> > I guess I deploy a new coproc that uses whatever the new method is, and
> then in my client, detect at runtime which HBase version I'm talking to and
> use that to determine which coprocessor to hit?
>
> Coprocessors are server side extensions. These API changes will require you
> to modify the code you plan to deploy on the server. I don't think any
> client side changes are needed. Unless your coprocessor implements an
> Endpoint and _you_ are changing your RPC message formats, a 1.0.x client
> shouldn't care whether it is talking to a 1.0.x server or a 1.1.x server,
> running your coprocessor or not.
>
>
>
> On Wed, May 13, 2015 at 3:00 PM, James Estes 
> wrote:
>
> > I saw the vote thread for RC2, so tried to build my project against it.
> >
> > My build fails when I depend on 1.1.0. I created a bare bones project
> > to show the issue I'm running into:
> > https://github.com/housejester/hbase-deps-test
> >
> > To be clear, it works in 1.0.0 (and I did add the repository).
> >
> > Further, we have a coprocessor and when I stand up a 1.1.0 HBase and
> > call my endpoint, I get:
> >
> > ! Caused by: java.lang.NoSuchMethodError:
> >
> >
> org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment.getRegion()Lorg/apache/hadoop/hbase/regionserver/HRegion;
> >
> > The same coprocessor works under 1.0.0.
> >
> > So, it looks like RegionCoprocessorEnvironment.getRegion() has been
> > removed?
> >
> > The Audience annotation is:
> > @InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.COPROC)
> > @InterfaceStability.Evolving
> >
> > Since it is Evolving it is allowed to change in a breaking way. I'm
> > trying to think about how I migrate. I guess I deploy a new coproc
> > that uses whatever the new method is, and then in my client, detect at
> > runtime which HBase version I'm talking to and use that to determine
> > which coprocessor to hit?
> >
> > Thanks,
> > James
> >
>
>
>
> --
> Best regards,
>
>- Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

[jira] [Created] (HBASE-13685) Procedure v2 - Add maxProcId to the wal header

[Matteo Bertozzi (JIRA)]

Matteo Bertozzi created HBASE-13685:
---

 Summary: Procedure v2 - Add maxProcId to the wal header
 Key: HBASE-13685
 URL: https://issues.apache.org/jira/browse/HBASE-13685
 Project: HBase
  Issue Type: Sub-task
  Components: proc-v2
Affects Versions: 1.1.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Blocker
 Fix For: 1.1.0
 Attachments: HBASE-13685-v0.patch

while working on HBASE-13476 I found that having the max-proc-id in the wal 
header, allows some nice optimizations.

[~ndimiduk] since 1.1 is not released yet, can we get this in so we can avoid 
all extra handling?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HBASE-13035) [0.98] Backport HBASE-12867 - Shell does not support custom replication endpoint specification

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell reopened HBASE-13035:


Yeah, not going to be so easy.

> [0.98] Backport HBASE-12867 - Shell does not support custom replication 
> endpoint specification
> --
>
> Key: HBASE-13035
> URL: https://issues.apache.org/jira/browse/HBASE-13035
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Enis Soztutar
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-13684) Allow mlockagent to be used when not starting as root

[Patrick White (JIRA)]

Patrick White created HBASE-13684:
-

 Summary: Allow mlockagent to be used when not starting as root
 Key: HBASE-13684
 URL: https://issues.apache.org/jira/browse/HBASE-13684
 Project: HBase
  Issue Type: Improvement
Reporter: Patrick White
Priority: Minor


This patch allows us to mlock all the things without having to start as root 
and drop privs.

The old way of using the user=xyz argument to drop privs if it was started as 
root still works.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-13655) Deprecate duplicate getCompression methods in HColumnDescriptor

[Lars Francke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Francke resolved HBASE-13655.
--
Resolution: Fixed

> Deprecate duplicate getCompression methods in HColumnDescriptor
> ---
>
> Key: HBASE-13655
> URL: https://issues.apache.org/jira/browse/HBASE-13655
> Project: HBase
>  Issue Type: Improvement
>Reporter: Lars Francke
>Assignee: Lars Francke
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-13655-v1.patch, HBASE-13655.patch
>
>
> {{HColumnDescriptor}} has {{getCompression}} and {{getCompressionType}} 
> methods returning the same thing. The same goes for 
> {{getCompactionCompression}} and {{getCompactionCompressionType}}.
> This deprecates the methods without {{Type}} in 2.0.0 so they can be removed 
> in 3.0.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Ruby shell versions for HBase 2.0

[Andrew Purtell]

> Only shell we could
​> ​
swap in w/o annoying users would be SQL (ducks);
​[...]
 You would
​> ​
also have to manage expectations: i.e. that the SQL would be extremely
​> ​
basic and that the REPL does not come with an idling 100 node  cluster
​> ​
ready to take up any involved queries.

If we are talking about adding a SQL shell, it doesn't have to be extremely
basic, we can revisit my proposal from the other month about embedding
Phoenix.


On Wed, May 13, 2015 at 4:50 PM, Stack  wrote:

> On Wed, May 13, 2015 at 1:38 PM, Andrew Purtell 
> wrote:
>
> > > That's only if you assume HBase users actually understand Ruby though,
> > no?
> >
> > Yes, and this is a fair point.
> >
> > The language REPL in which we're embedding the shell DSL doesn't have to
> be
> > Ruby. JavaScript would work, maybe vi
> ​​
> a Nashorn. Python would work, via
> > Jython. Or Scala's REPL, even.
> >
> >
> >
> JRuby had best interpreter and the others considered had extra baggage
> doing a DSL (For history, see HBASE-487). Having to put tabs in your shell
> doing loops and no jline at the time made jython a non-runner though I
> wanted it to win (Has anything changed here?) JS is probably way better
> than when I looked at it back then?
>
> Regardless, whether jython, JS, scala, or custom, IMO, we'd just be pissing
> of users if we swap one idiosyncratic for another.  Only shell we could
> swap in w/o annoying users would be SQL (ducks); and even here you'd
> probably have to implement a bunch of operators on the other side of
> sqllines bang operator (or equivalent) to do the hbase specifics. You would
> also have to manage expectations: i.e. that the SQL would be extremely
> basic and that the REPL does not come with an idling 100 node  cluster
> ready to take up any involved queries.
>
> St.Ack
>
>
>
>
> > On Wed, May 13, 2015 at 1:10 PM, Josh Elser 
> wrote:
> >
> > > Andrew Purtell wrote:
> > >
> > >> This is because the Accumulo shell is a custom built shell? If so, we
> > had
> > >> one of those once and replaced it with the IRB based one. We didn't
> > settle
> > >> on JRuby right away but, at the time, the consensus was we didn't want
> > to
> > >> be in the business of maintaining yet another REPL when specialist
> open
> > >> source communities have already done that. Why has this changed? Has
> it?
> > >>
> > >
> > > Accumulo shell is just a Java program that uses JLine. It has no
> control
> > > flow structures, variables, or anything like that.
> > >
> > >  Why is the Accumulo shell easier to script with? Does it have control
> > flow
> > >> constructs? Variable assignment? Or is it easier somehow because it
> does
> > >> not have those things? Even the system shell has those. Hmm... Are we
> > >> saying that any control flow or variables/substitution should be done
> by
> > >> calling our shell from a bash or sh script? Wouldn't that be super
> slow
> > >> given Java's startup time for every invocation of our shell? A bit
> like
> > >> trying to script the AWS command line utilities, which is
> excruciating.
> > >>
> > >
> > > This is what I was trying to get at: the lack of typical language
> > > constructs is both beneficial and irritating. I think it depends on
> what
> > > you really want to do. Does that mean trying to maintain two shells is
> > > worth it? I'm not sure, but I'm also not sure where the happy medium
> is.
> > >
> > > If you exec the shell to just run one command at a time, yes. It is
> slow.
> > > You can redirect input or provide a file of commands to run at one time
> > > which don't suffer from the JVM startup problem.
> > >
> > >  I can see why a custom shell built with Java would easier to test
> where
> > >> the
> > >> rest of the project is using Surefire/JUnit. That's a developer
> > >> convenience
> > >> concern though, not a user convenience one.
> > >>
> > >
> > > That's only if you assume HBase users actually understand Ruby though,
> > no?
> > > I would think that something which acts like your normal unix shell
> would
> > > be familiar to users w/o the need to understand some other language.
> > >
> > >
> > >
> > >> On Wed, May 13, 2015 at 10:41 AM, Sean Busbey
> > >> wrote:
> > >>
> > >>  Pros:
> > >>>
> > >>> * It's easier to test and maintain
> > >>> * it's easier to script with
> > >>> * its interactive mode "feels" mode focused on the task of
> interacting
> > >>> with
> > >>> a cluster to me (maybe this is just acting like a psql or mysql
> shell)
> > >>>
> > >>> Cons
> > >>>
> > >>> * adding custom commands requires knowing java
> > >>>
> > >>> --
> > >>> Sean
> > >>> On May 13, 2015 12:31 PM, "Andrew Purtell"
> > wrote:
> > >>>
> > >>>  Why is the Accumulo shell superior?
> > 
> >  Is it scriptable?
> > 
> > 
> >  On Wed, May 13, 2015 at 10:28 AM, Sean Busbey
> > 
> > >>> wrote:
> > >>>
> >  I would love to rip out the JRuby shell entirely and make something
> > >
> >  closer
> > 
> > > to the Accumulo shell, but I expect that 

Re: Ruby shell versions for HBase 2.0

[Lars Francke]

I'm also in favor of a rewrite in another language (but it doesn't have to
be now). This is very subjective but I never got the hang of the Ruby
stuff, it's always annoying to use and I end up googling Ruby stuff almost
every time I want to do something that's not covered in the help command.
Scala would make way more sense to me. If no rewrite I have no problem
moving to Ruby 2.2.

I know it's at least a few years out but a Java REPL might be coming in
Java 9 () so if we're still
discussing by the time we switch to JDK9 that might be an option ;-)


On Wed, May 13, 2015 at 10:38 PM, Andrew Purtell 
wrote:

> > That's only if you assume HBase users actually understand Ruby though,
> no?
>
> Yes, and this is a fair point.
>
> The language REPL in which we're embedding the shell DSL doesn't have to be
> Ruby. JavaScript would work, maybe via Nashorn. Python would work, via
> Jython. Or Scala's REPL, even.
>
>
> On Wed, May 13, 2015 at 1:10 PM, Josh Elser  wrote:
>
> > Andrew Purtell wrote:
> >
> >> This is because the Accumulo shell is a custom built shell? If so, we
> had
> >> one of those once and replaced it with the IRB based one. We didn't
> settle
> >> on JRuby right away but, at the time, the consensus was we didn't want
> to
> >> be in the business of maintaining yet another REPL when specialist open
> >> source communities have already done that. Why has this changed? Has it?
> >>
> >
> > Accumulo shell is just a Java program that uses JLine. It has no control
> > flow structures, variables, or anything like that.
> >
> >  Why is the Accumulo shell easier to script with? Does it have control
> flow
> >> constructs? Variable assignment? Or is it easier somehow because it does
> >> not have those things? Even the system shell has those. Hmm... Are we
> >> saying that any control flow or variables/substitution should be done by
> >> calling our shell from a bash or sh script? Wouldn't that be super slow
> >> given Java's startup time for every invocation of our shell? A bit like
> >> trying to script the AWS command line utilities, which is excruciating.
> >>
> >
> > This is what I was trying to get at: the lack of typical language
> > constructs is both beneficial and irritating. I think it depends on what
> > you really want to do. Does that mean trying to maintain two shells is
> > worth it? I'm not sure, but I'm also not sure where the happy medium is.
> >
> > If you exec the shell to just run one command at a time, yes. It is slow.
> > You can redirect input or provide a file of commands to run at one time
> > which don't suffer from the JVM startup problem.
> >
> >  I can see why a custom shell built with Java would easier to test where
> >> the
> >> rest of the project is using Surefire/JUnit. That's a developer
> >> convenience
> >> concern though, not a user convenience one.
> >>
> >
> > That's only if you assume HBase users actually understand Ruby though,
> no?
> > I would think that something which acts like your normal unix shell would
> > be familiar to users w/o the need to understand some other language.
> >
> >
> >
> >> On Wed, May 13, 2015 at 10:41 AM, Sean Busbey
> >> wrote:
> >>
> >>  Pros:
> >>>
> >>> * It's easier to test and maintain
> >>> * it's easier to script with
> >>> * its interactive mode "feels" mode focused on the task of interacting
> >>> with
> >>> a cluster to me (maybe this is just acting like a psql or mysql shell)
> >>>
> >>> Cons
> >>>
> >>> * adding custom commands requires knowing java
> >>>
> >>> --
> >>> Sean
> >>> On May 13, 2015 12:31 PM, "Andrew Purtell"
> wrote:
> >>>
> >>>  Why is the Accumulo shell superior?
> 
>  Is it scriptable?
> 
> 
>  On Wed, May 13, 2015 at 10:28 AM, Sean Busbey
> 
> >>> wrote:
> >>>
>  I would love to rip out the JRuby shell entirely and make something
> >
>  closer
> 
> > to the Accumulo shell, but I expect that will
> >
> > 1) be way more work
> >
> > 2) be even less compatible for those that rely on customizations.
> >
> > I figured given time we could get a preview "user shell" (rather than
> > "power shell" via irb) together in 2.0 and aim for default in 3.0.
> >
> > --
> > Sean
> > On May 13, 2015 12:19 PM, "Stack"  wrote:
> >
> >  Nice writeup Sean.
> >>
> >> Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license
> >>
> > is
> >>>
>  still amenable and hopefully jruby 9k will be slimmer than jruby
> >>
> > 1.7+.
> >>>
>  But if we are going to do a significant shell refactor for hbase 2.0,
> >> should we consider doing something more radical; e.g. a new shell?
> If
> >> interest, could start up a new thread so don't distract from this
> >>
> > one.
> >>>
>  St.Ack
> >>
> >>
> >>
> >>
> >> On Wed, May 13, 2015 at 9:22 AM, Sean Busbey
> >>
> > wrote:
> >
> >> Hi folks!
> >>>
> >>> I

Re: Ruby shell versions for HBase 2.0

[Stack]

On Wed, May 13, 2015 at 1:38 PM, Andrew Purtell  wrote:

> > That's only if you assume HBase users actually understand Ruby though,
> no?
>
> Yes, and this is a fair point.
>
> The language REPL in which we're embedding the shell DSL doesn't have to be
> Ruby. JavaScript would work, maybe via Nashorn. Python would work, via
> Jython. Or Scala's REPL, even.
>
>
>
JRuby had best interpreter and the others considered had extra baggage
doing a DSL (For history, see HBASE-487). Having to put tabs in your shell
doing loops and no jline at the time made jython a non-runner though I
wanted it to win (Has anything changed here?) JS is probably way better
than when I looked at it back then?

Regardless, whether jython, JS, scala, or custom, IMO, we'd just be pissing
of users if we swap one idiosyncratic for another.  Only shell we could
swap in w/o annoying users would be SQL (ducks); and even here you'd
probably have to implement a bunch of operators on the other side of
sqllines bang operator (or equivalent) to do the hbase specifics. You would
also have to manage expectations: i.e. that the SQL would be extremely
basic and that the REPL does not come with an idling 100 node  cluster
ready to take up any involved queries.

St.Ack




> On Wed, May 13, 2015 at 1:10 PM, Josh Elser  wrote:
>
> > Andrew Purtell wrote:
> >
> >> This is because the Accumulo shell is a custom built shell? If so, we
> had
> >> one of those once and replaced it with the IRB based one. We didn't
> settle
> >> on JRuby right away but, at the time, the consensus was we didn't want
> to
> >> be in the business of maintaining yet another REPL when specialist open
> >> source communities have already done that. Why has this changed? Has it?
> >>
> >
> > Accumulo shell is just a Java program that uses JLine. It has no control
> > flow structures, variables, or anything like that.
> >
> >  Why is the Accumulo shell easier to script with? Does it have control
> flow
> >> constructs? Variable assignment? Or is it easier somehow because it does
> >> not have those things? Even the system shell has those. Hmm... Are we
> >> saying that any control flow or variables/substitution should be done by
> >> calling our shell from a bash or sh script? Wouldn't that be super slow
> >> given Java's startup time for every invocation of our shell? A bit like
> >> trying to script the AWS command line utilities, which is excruciating.
> >>
> >
> > This is what I was trying to get at: the lack of typical language
> > constructs is both beneficial and irritating. I think it depends on what
> > you really want to do. Does that mean trying to maintain two shells is
> > worth it? I'm not sure, but I'm also not sure where the happy medium is.
> >
> > If you exec the shell to just run one command at a time, yes. It is slow.
> > You can redirect input or provide a file of commands to run at one time
> > which don't suffer from the JVM startup problem.
> >
> >  I can see why a custom shell built with Java would easier to test where
> >> the
> >> rest of the project is using Surefire/JUnit. That's a developer
> >> convenience
> >> concern though, not a user convenience one.
> >>
> >
> > That's only if you assume HBase users actually understand Ruby though,
> no?
> > I would think that something which acts like your normal unix shell would
> > be familiar to users w/o the need to understand some other language.
> >
> >
> >
> >> On Wed, May 13, 2015 at 10:41 AM, Sean Busbey
> >> wrote:
> >>
> >>  Pros:
> >>>
> >>> * It's easier to test and maintain
> >>> * it's easier to script with
> >>> * its interactive mode "feels" mode focused on the task of interacting
> >>> with
> >>> a cluster to me (maybe this is just acting like a psql or mysql shell)
> >>>
> >>> Cons
> >>>
> >>> * adding custom commands requires knowing java
> >>>
> >>> --
> >>> Sean
> >>> On May 13, 2015 12:31 PM, "Andrew Purtell"
> wrote:
> >>>
> >>>  Why is the Accumulo shell superior?
> 
>  Is it scriptable?
> 
> 
>  On Wed, May 13, 2015 at 10:28 AM, Sean Busbey
> 
> >>> wrote:
> >>>
>  I would love to rip out the JRuby shell entirely and make something
> >
>  closer
> 
> > to the Accumulo shell, but I expect that will
> >
> > 1) be way more work
> >
> > 2) be even less compatible for those that rely on customizations.
> >
> > I figured given time we could get a preview "user shell" (rather than
> > "power shell" via irb) together in 2.0 and aim for default in 3.0.
> >
> > --
> > Sean
> > On May 13, 2015 12:19 PM, "Stack"  wrote:
> >
> >  Nice writeup Sean.
> >>
> >> Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license
> >>
> > is
> >>>
>  still amenable and hopefully jruby 9k will be slimmer than jruby
> >>
> > 1.7+.
> >>>
>  But if we are going to do a significant shell refactor for hbase 2.0,
> >> should we consider doing something more radical; e.g. a new shell?

Re: Ruby shell versions for HBase 2.0

[Michael Segel]

If you were going to replace Ruby, Python (Jython) and Scala would make more 
sense. 

Then you could go to play (the Scala GUI) for a GUI to HBase reusing the code. 


> On May 13, 2015, at 3:38 PM, Andrew Purtell  wrote:
> 
>> That's only if you assume HBase users actually understand Ruby though, no?
> 
> Yes, and this is a fair point.
> 
> The language REPL in which we're embedding the shell DSL doesn't have to be
> Ruby. JavaScript would work, maybe via Nashorn. Python would work, via
> Jython. Or Scala's REPL, even.
> 
> 
> On Wed, May 13, 2015 at 1:10 PM, Josh Elser  wrote:
> 
>> Andrew Purtell wrote:
>> 
>>> This is because the Accumulo shell is a custom built shell? If so, we had
>>> one of those once and replaced it with the IRB based one. We didn't settle
>>> on JRuby right away but, at the time, the consensus was we didn't want to
>>> be in the business of maintaining yet another REPL when specialist open
>>> source communities have already done that. Why has this changed? Has it?
>>> 
>> 
>> Accumulo shell is just a Java program that uses JLine. It has no control
>> flow structures, variables, or anything like that.
>> 
>> Why is the Accumulo shell easier to script with? Does it have control flow
>>> constructs? Variable assignment? Or is it easier somehow because it does
>>> not have those things? Even the system shell has those. Hmm... Are we
>>> saying that any control flow or variables/substitution should be done by
>>> calling our shell from a bash or sh script? Wouldn't that be super slow
>>> given Java's startup time for every invocation of our shell? A bit like
>>> trying to script the AWS command line utilities, which is excruciating.
>>> 
>> 
>> This is what I was trying to get at: the lack of typical language
>> constructs is both beneficial and irritating. I think it depends on what
>> you really want to do. Does that mean trying to maintain two shells is
>> worth it? I'm not sure, but I'm also not sure where the happy medium is.
>> 
>> If you exec the shell to just run one command at a time, yes. It is slow.
>> You can redirect input or provide a file of commands to run at one time
>> which don't suffer from the JVM startup problem.
>> 
>> I can see why a custom shell built with Java would easier to test where
>>> the
>>> rest of the project is using Surefire/JUnit. That's a developer
>>> convenience
>>> concern though, not a user convenience one.
>>> 
>> 
>> That's only if you assume HBase users actually understand Ruby though, no?
>> I would think that something which acts like your normal unix shell would
>> be familiar to users w/o the need to understand some other language.
>> 
>> 
>> 
>>> On Wed, May 13, 2015 at 10:41 AM, Sean Busbey
>>> wrote:
>>> 
>>> Pros:
 
 * It's easier to test and maintain
 * it's easier to script with
 * its interactive mode "feels" mode focused on the task of interacting
 with
 a cluster to me (maybe this is just acting like a psql or mysql shell)
 
 Cons
 
 * adding custom commands requires knowing java
 
 --
 Sean
 On May 13, 2015 12:31 PM, "Andrew Purtell"  wrote:
 
 Why is the Accumulo shell superior?
> 
> Is it scriptable?
> 
> 
> On Wed, May 13, 2015 at 10:28 AM, Sean Busbey
> 
 wrote:
 
> I would love to rip out the JRuby shell entirely and make something
>> 
> closer
> 
>> to the Accumulo shell, but I expect that will
>> 
>> 1) be way more work
>> 
>> 2) be even less compatible for those that rely on customizations.
>> 
>> I figured given time we could get a preview "user shell" (rather than
>> "power shell" via irb) together in 2.0 and aim for default in 3.0.
>> 
>> --
>> Sean
>> On May 13, 2015 12:19 PM, "Stack"  wrote:
>> 
>> Nice writeup Sean.
>>> 
>>> Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license
>>> 
>> is
 
> still amenable and hopefully jruby 9k will be slimmer than jruby
>>> 
>> 1.7+.
 
> But if we are going to do a significant shell refactor for hbase 2.0,
>>> should we consider doing something more radical; e.g. a new shell? If
>>> interest, could start up a new thread so don't distract from this
>>> 
>> one.
 
> St.Ack
>>> 
>>> 
>>> 
>>> 
>>> On Wed, May 13, 2015 at 9:22 AM, Sean Busbey
>>> 
>> wrote:
>> 
>>> Hi folks!
 
 If you weren't aware, our current shell relies on Ruby,
 
>>> specifically
 
> the
>> 
>>> REPL program IRB from JRuby. When we launched 1.0 we were on JRuby
 
>>> 1.6
> 
>> with
>>> 
 defaults, which means we're stuck on Ruby 1.8.
 
 For those that don't already know, Ruby 1.8 is super old and has
 
>>> been
 
> walking off into the sunset for a few years now. Most (but not
 
>>> all!)
 
> formal
>

[jira] [Created] (HBASE-13683) Doc HBase and G1GC

[stack (JIRA)]

stack created HBASE-13683:
-

 Summary: Doc HBase and G1GC
 Key: HBASE-13683
 URL: https://issues.apache.org/jira/browse/HBASE-13683
 Project: HBase
  Issue Type: Task
  Components: documentation
Reporter: stack


Add section to refguide on running G1GC with HBase. There is the intel talk at 
hbasecon with nice pictures and healthy recommendations, there is our 
[~esteban]'s recent experience running G1GC, and the mighty [~bbeaudreault] 
dumped a bunch of helpful advice in the mailing list just now: 
http://search-hadoop.com/m/YGbbupEDoKTrDo/%2522How+to+know+the+root+reason+to+cause+RegionServer+OOM%253F%2522&subj=Re+How+to+know+the+root+reason+to+cause+RegionServer+OOM+



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-13035) [0.98] Backport HBASE-12867 - Shell does not support custom replication endpoint specification

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell resolved HBASE-13035.

Resolution: Not A Problem

> [0.98] Backport HBASE-12867 - Shell does not support custom replication 
> endpoint specification
> --
>
> Key: HBASE-13035
> URL: https://issues.apache.org/jira/browse/HBASE-13035
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Enis Soztutar
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Troubles with HBase 1.1.0 RC2

[Andrew Purtell]

> So, it looks like RegionCoprocessorEnvironment.getRegion() has been
removed?

No, the signature has changed, basically s/HRegion/Region/. HRegion is an
internal, low level implementation type. Has always been. We have replaced
it with Region, an interface that contains a subset of HRegion we feel we
can support for coprocessor source and binary compatibility longer term.
This work was done on HBASE-12972 if you're curious to know more about it.

> I guess I deploy a new coproc that uses whatever the new method is, and
then in my client, detect at runtime which HBase version I'm talking to and
use that to determine which coprocessor to hit?

Coprocessors are server side extensions. These API changes will require you
to modify the code you plan to deploy on the server. I don't think any
client side changes are needed. Unless your coprocessor implements an
Endpoint and _you_ are changing your RPC message formats, a 1.0.x client
shouldn't care whether it is talking to a 1.0.x server or a 1.1.x server,
running your coprocessor or not.



On Wed, May 13, 2015 at 3:00 PM, James Estes  wrote:

> I saw the vote thread for RC2, so tried to build my project against it.
>
> My build fails when I depend on 1.1.0. I created a bare bones project
> to show the issue I'm running into:
> https://github.com/housejester/hbase-deps-test
>
> To be clear, it works in 1.0.0 (and I did add the repository).
>
> Further, we have a coprocessor and when I stand up a 1.1.0 HBase and
> call my endpoint, I get:
>
> ! Caused by: java.lang.NoSuchMethodError:
>
> org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment.getRegion()Lorg/apache/hadoop/hbase/regionserver/HRegion;
>
> The same coprocessor works under 1.0.0.
>
> So, it looks like RegionCoprocessorEnvironment.getRegion() has been
> removed?
>
> The Audience annotation is:
> @InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.COPROC)
> @InterfaceStability.Evolving
>
> Since it is Evolving it is allowed to change in a breaking way. I'm
> trying to think about how I migrate. I guess I deploy a new coproc
> that uses whatever the new method is, and then in my client, detect at
> runtime which HBase version I'm talking to and use that to determine
> which coprocessor to hit?
>
> Thanks,
> James
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: Ruby shell versions for HBase 2.0

[Andrew Purtell]

> That's only if you assume HBase users actually understand Ruby though, no?

Yes, and this is a fair point.

The language REPL in which we're embedding the shell DSL doesn't have to be
Ruby. JavaScript would work, maybe via Nashorn. Python would work, via
Jython. Or Scala's REPL, even.


On Wed, May 13, 2015 at 1:10 PM, Josh Elser  wrote:

> Andrew Purtell wrote:
>
>> This is because the Accumulo shell is a custom built shell? If so, we had
>> one of those once and replaced it with the IRB based one. We didn't settle
>> on JRuby right away but, at the time, the consensus was we didn't want to
>> be in the business of maintaining yet another REPL when specialist open
>> source communities have already done that. Why has this changed? Has it?
>>
>
> Accumulo shell is just a Java program that uses JLine. It has no control
> flow structures, variables, or anything like that.
>
>  Why is the Accumulo shell easier to script with? Does it have control flow
>> constructs? Variable assignment? Or is it easier somehow because it does
>> not have those things? Even the system shell has those. Hmm... Are we
>> saying that any control flow or variables/substitution should be done by
>> calling our shell from a bash or sh script? Wouldn't that be super slow
>> given Java's startup time for every invocation of our shell? A bit like
>> trying to script the AWS command line utilities, which is excruciating.
>>
>
> This is what I was trying to get at: the lack of typical language
> constructs is both beneficial and irritating. I think it depends on what
> you really want to do. Does that mean trying to maintain two shells is
> worth it? I'm not sure, but I'm also not sure where the happy medium is.
>
> If you exec the shell to just run one command at a time, yes. It is slow.
> You can redirect input or provide a file of commands to run at one time
> which don't suffer from the JVM startup problem.
>
>  I can see why a custom shell built with Java would easier to test where
>> the
>> rest of the project is using Surefire/JUnit. That's a developer
>> convenience
>> concern though, not a user convenience one.
>>
>
> That's only if you assume HBase users actually understand Ruby though, no?
> I would think that something which acts like your normal unix shell would
> be familiar to users w/o the need to understand some other language.
>
>
>
>> On Wed, May 13, 2015 at 10:41 AM, Sean Busbey
>> wrote:
>>
>>  Pros:
>>>
>>> * It's easier to test and maintain
>>> * it's easier to script with
>>> * its interactive mode "feels" mode focused on the task of interacting
>>> with
>>> a cluster to me (maybe this is just acting like a psql or mysql shell)
>>>
>>> Cons
>>>
>>> * adding custom commands requires knowing java
>>>
>>> --
>>> Sean
>>> On May 13, 2015 12:31 PM, "Andrew Purtell"  wrote:
>>>
>>>  Why is the Accumulo shell superior?

 Is it scriptable?


 On Wed, May 13, 2015 at 10:28 AM, Sean Busbey

>>> wrote:
>>>
 I would love to rip out the JRuby shell entirely and make something
>
 closer

> to the Accumulo shell, but I expect that will
>
> 1) be way more work
>
> 2) be even less compatible for those that rely on customizations.
>
> I figured given time we could get a preview "user shell" (rather than
> "power shell" via irb) together in 2.0 and aim for default in 3.0.
>
> --
> Sean
> On May 13, 2015 12:19 PM, "Stack"  wrote:
>
>  Nice writeup Sean.
>>
>> Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license
>>
> is
>>>
 still amenable and hopefully jruby 9k will be slimmer than jruby
>>
> 1.7+.
>>>
 But if we are going to do a significant shell refactor for hbase 2.0,
>> should we consider doing something more radical; e.g. a new shell? If
>> interest, could start up a new thread so don't distract from this
>>
> one.
>>>
 St.Ack
>>
>>
>>
>>
>> On Wed, May 13, 2015 at 9:22 AM, Sean Busbey
>>
> wrote:
>
>> Hi folks!
>>>
>>> If you weren't aware, our current shell relies on Ruby,
>>>
>> specifically
>>>
 the
>
>> REPL program IRB from JRuby. When we launched 1.0 we were on JRuby
>>>
>> 1.6

> with
>>
>>> defaults, which means we're stuck on Ruby 1.8.
>>>
>>> For those that don't already know, Ruby 1.8 is super old and has
>>>
>> been
>>>
 walking off into the sunset for a few years now. Most (but not
>>>
>> all!)
>>>
 formal
>>
>>> support systems for running Ruby have EOLed 1.8 and there are
>>>
>> numerous

> known issues with it.
>>>
>>> Right now there's an open ticket to get us on JRuby 1.7 so that our
>>>
>> shell
>
>> can work on PPC systems[1]. That version of JRuby defaults to Ruby
>>>
>> 1.9

> but
>>
>>> can be run in Ruby 1.8 mode. There are some implementation details
>>

Re: Ruby shell versions for HBase 2.0

[Josh Elser]

Andrew Purtell wrote:

This is because the Accumulo shell is a custom built shell? If so, we had
one of those once and replaced it with the IRB based one. We didn't settle
on JRuby right away but, at the time, the consensus was we didn't want to
be in the business of maintaining yet another REPL when specialist open
source communities have already done that. Why has this changed? Has it?


Accumulo shell is just a Java program that uses JLine. It has no control 
flow structures, variables, or anything like that.



Why is the Accumulo shell easier to script with? Does it have control flow
constructs? Variable assignment? Or is it easier somehow because it does
not have those things? Even the system shell has those. Hmm... Are we
saying that any control flow or variables/substitution should be done by
calling our shell from a bash or sh script? Wouldn't that be super slow
given Java's startup time for every invocation of our shell? A bit like
trying to script the AWS command line utilities, which is excruciating.


This is what I was trying to get at: the lack of typical language 
constructs is both beneficial and irritating. I think it depends on what 
you really want to do. Does that mean trying to maintain two shells is 
worth it? I'm not sure, but I'm also not sure where the happy medium is.


If you exec the shell to just run one command at a time, yes. It is 
slow. You can redirect input or provide a file of commands to run at one 
time which don't suffer from the JVM startup problem.



I can see why a custom shell built with Java would easier to test where the
rest of the project is using Surefire/JUnit. That's a developer convenience
concern though, not a user convenience one.


That's only if you assume HBase users actually understand Ruby though, 
no? I would think that something which acts like your normal unix shell 
would be familiar to users w/o the need to understand some other language.




On Wed, May 13, 2015 at 10:41 AM, Sean Busbey  wrote:


Pros:

* It's easier to test and maintain
* it's easier to script with
* its interactive mode "feels" mode focused on the task of interacting with
a cluster to me (maybe this is just acting like a psql or mysql shell)

Cons

* adding custom commands requires knowing java

--
Sean
On May 13, 2015 12:31 PM, "Andrew Purtell"  wrote:


Why is the Accumulo shell superior?

Is it scriptable?


On Wed, May 13, 2015 at 10:28 AM, Sean Busbey

wrote:

I would love to rip out the JRuby shell entirely and make something

closer

to the Accumulo shell, but I expect that will

1) be way more work

2) be even less compatible for those that rely on customizations.

I figured given time we could get a preview "user shell" (rather than
"power shell" via irb) together in 2.0 and aim for default in 3.0.

--
Sean
On May 13, 2015 12:19 PM, "Stack"  wrote:


Nice writeup Sean.

Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license

is

still amenable and hopefully jruby 9k will be slimmer than jruby

1.7+.

But if we are going to do a significant shell refactor for hbase 2.0,
should we consider doing something more radical; e.g. a new shell? If
interest, could start up a new thread so don't distract from this

one.

St.Ack




On Wed, May 13, 2015 at 9:22 AM, Sean Busbey

wrote:

Hi folks!

If you weren't aware, our current shell relies on Ruby,

specifically

the

REPL program IRB from JRuby. When we launched 1.0 we were on JRuby

1.6

with

defaults, which means we're stuck on Ruby 1.8.

For those that don't already know, Ruby 1.8 is super old and has

been

walking off into the sunset for a few years now. Most (but not

all!)

formal

support systems for running Ruby have EOLed 1.8 and there are

numerous

known issues with it.

Right now there's an open ticket to get us on JRuby 1.7 so that our

shell

can work on PPC systems[1]. That version of JRuby defaults to Ruby

1.9

but

can be run in Ruby 1.8 mode. There are some implementation details
outstanding, but I'm hoping that ticket can work out such that it

can

land

in branch-1.

For HBase 2.0, I'd like us to plan for a little farther out in the

future

than just updating to Ruby 1.9 (though that would be a fine

incremental

step with some non-trivial work attached). The "current" version of

Ruby

is

2.2. Much like the move from 1.8 ->  1.9 it is not backwards

compatible.

JRuby's next major maintenance release line is "version 9000"[2]

and

it

will start out *only* supporting Ruby 2.2. Right now JRuby 9000 is

in

its

second "preview" release. They still have a few feature complete

items

to

address before they hit their first GA release.

I'd like us to move to Ruby 2.2 via JRuby 9000 for HBase 2.0.  This

will

cause operator pain to folks with advanced scripts based on Ruby

1.8,

but

it should allow us to update versions to avoid e.g. perf,

correctness,

and

security issues much more easily over the lifetime of 2.0.

What do folks think? Would JRuby 9000 need to hit a GA release

Re: JIRA default resolution change

[Nick Dimiduk]

Thanks Ted.

On Wed, May 13, 2015 at 11:51 AM, Ted Yu  wrote:

> See https://issues.apache.org/jira/browse/INFRA-9646
>
> Before the above is resolved, please pick 'Fixed' as resolution.
>
> Cheers
>
> On Wed, May 13, 2015 at 11:42 AM, Nick Dimiduk  wrote:
>
> > What's the "pending fixed" default for resolving issues now? Was that
> > change intentional? Is there a new workflow to follow now?
> >
> > Thanks,
> > Nick
> >
>

[jira] [Resolved] (HBASE-13664) Use HBase 1.0 interfaces in ConnectionCache

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HBASE-13664.

Resolution: Fixed

> Use HBase 1.0 interfaces in ConnectionCache
> ---
>
> Key: HBASE-13664
> URL: https://issues.apache.org/jira/browse/HBASE-13664
> Project: HBase
>  Issue Type: Bug
>Reporter: Solomon Duskis
>Assignee: Solomon Duskis
> Fix For: 1.0.2, 1.2.0, 1.1.1
>
> Attachments: HBASE-13664.branch-1.patch, HBASE-13664.branch-1.patch
>
>
> ConnectionCache uses the new HBase interfaces in master.  Update  branch-1* 
> to use them as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: JIRA default resolution change

[Ted Yu]

See https://issues.apache.org/jira/browse/INFRA-9646

Before the above is resolved, please pick 'Fixed' as resolution.

Cheers

On Wed, May 13, 2015 at 11:42 AM, Nick Dimiduk  wrote:

> What's the "pending fixed" default for resolving issues now? Was that
> change intentional? Is there a new workflow to follow now?
>
> Thanks,
> Nick
>

[jira] [Resolved] (HBASE-13678) Hbase client block on region call back

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell resolved HBASE-13678.

Resolution: Invalid

This is the project development issue tracker. Please make suggestions or ask 
questions on the mailing lists:
dev@hbase.apache.org for developers
u...@hbase.apache.org for users
or both if you are unsure which is the better audience.

> Hbase client block on region call back
> --
>
> Key: HBASE-13678
> URL: https://issues.apache.org/jira/browse/HBASE-13678
> Project: HBase
>  Issue Type: Bug
>  Components: Client
>Affects Versions: 0.94.15
> Environment: CentOS 6.4 scala 2.11/java 1.7
>Reporter: Junheng Gong
>  Labels: easyfix, performance
>
> When batch get some data from cluster, the client will block some threads 
> base on region server amount
> In code client use Future.get block a thread for each related region servers 
> and wait them finish one by one.
> In our case we have 12 region servers, so most of time the client will cost 
> 12 threads for a batch get, if we have many concurrent request it will OOM by 
> can not create native threads.
> I suggest we use some kind async or call back to process region server's 
> response.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

JIRA default resolution change

[Nick Dimiduk]

What's the "pending fixed" default for resolving issues now? Was that
change intentional? Is there a new workflow to follow now?

Thanks,
Nick

[jira] [Created] (HBASE-13682) Compute HDFS locality in parallel.

[Elliott Clark (JIRA)]

Elliott Clark created HBASE-13682:
-

 Summary: Compute HDFS locality in parallel.
 Key: HBASE-13682
 URL: https://issues.apache.org/jira/browse/HBASE-13682
 Project: HBase
  Issue Type: Bug
Reporter: Elliott Clark


Right now when the balancer needs to know about region locality it asks the 
cache in serial. When the cache is empty or expired it goes to the NN.

On larger clusters with lots of blocks this can be really slow. That means that 
balancer is un-usable while masters are being restarted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-13201) Remove HTablePool from thrift-server

[Nick Dimiduk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-13201.
--
   Resolution: Fixed
Fix Version/s: 1.0.2

Cherry-picked {{242f62f}} from branch-1. Ran locally {{mvn test 
-Dtest=Test*Thrift*}}, ~65 tests, all passed.

FYI [~enis].

> Remove HTablePool from thrift-server
> 
>
> Key: HBASE-13201
> URL: https://issues.apache.org/jira/browse/HBASE-13201
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Solomon Duskis
>Assignee: Solomon Duskis
> Fix For: 2.0.0, 1.1.0, 1.0.2
>
> Attachments: HBASE-13201-1-branch-1.patch, HBASE-13201-1.patch, 
> HBASE-13201-2-branch-1.patch, HBASE-13201.patch
>
>
> HTablePool is an old concept that was superseded by Connection.  
> thrift-server doesn't need HTablePool anymore and can use the ConnectionCache 
> directly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HBASE-13201) Remove HTablePool from thrift-server

[Nick Dimiduk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk reopened HBASE-13201:
--

Oops. Sorry about that Solomon.

> Remove HTablePool from thrift-server
> 
>
> Key: HBASE-13201
> URL: https://issues.apache.org/jira/browse/HBASE-13201
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Solomon Duskis
>Assignee: Solomon Duskis
> Fix For: 2.0.0, 1.1.0
>
> Attachments: HBASE-13201-1-branch-1.patch, HBASE-13201-1.patch, 
> HBASE-13201-2-branch-1.patch, HBASE-13201.patch
>
>
> HTablePool is an old concept that was superseded by Connection.  
> thrift-server doesn't need HTablePool anymore and can use the ConnectionCache 
> directly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Ruby shell versions for HBase 2.0

[Michael Segel]

Hmmm. 

So if we move to a different tech, the modifications / customizations people 
have done are going to be useless. 
If we upgrade to a new version of JRuby, some scripts may not have issues, 
others may have some , and a third group would have major rewrites. 

The interesting / downside to JRuby is that within the entire eco system… no 
other use of JRuby. 

If you were going to switch… maybe Python? (Jython) 



> On May 13, 2015, at 12:37 PM, Sean Busbey  wrote:
> 
> On May 13, 2015 12:06 PM, "Michael Segel"  wrote:
>> 
>> So…
>> Silly question…
>> Do you really need to worry about backward’s compatibility?
>> 
>> How many people have customized HBaseShell ?
>> 
>> What are the common customizations and if you port HBase shell, how much
> work would filter through to the custom code?
>> 
>> 
> 
> These are excellent questions to which I have poor answers.


> The mechanics of the interactive repl via IRB will probably not be that
> different. However, the ref guide calls out the ability to customize things
> via hooking in your own ruby script so there's little bound on what we've
> set for expectations. Given ruby 1.8's age, I'm not even sure there are 1.8
> to 2.2 guides; likely folks would need to follow 1.8 to 1.9 and 1.9 to 2.2
> guides (or we could work to provide some combined guidance).
> 
> Customizations are just as likely to be "call this sequence of commands"
> (little to no work for update) as a daemon that does health and recovery
> checks (lots of work, approaching "rewrite").
> 
> Personally, I avoid customizations at least in part because it's ruby 1.8.
> I've also discouraged those who ask me. Instead I rely on the
> non-interactive flag and bash.  But given the ref guide positioning, it's
> likely we'll break a nontrivial number of folks.

Re: Ruby shell versions for HBase 2.0

[Andrew Purtell]

Having two shells - a simple user shell for tinkering and an advanced one
for real ops and developers - would be fine, as long as one is not
deprecated. Can't trust something deprecated for tooling.


On Wed, May 13, 2015 at 10:52 AM, Esteban Gutierrez 
wrote:

> +1 for new JRuby for now and having a JIRA to discuss what features should
> this shell have would be great.
>
> However, I think that during the last few years many of us have used the
> hbase shell for major surgery as Andrew said and it has added some
> confusion to users about what the hbase shell is used for and I think that
> functionality shouldn't be longer exposed to regular users. One way we
> could probably do this is to have a user facing CLI that will be more for
> DDL, permissions and data interaction for users (sort of psql or phoenix
> /me runs too) and deprecate/update the old shell just leaving table/region
> management functionality for admins, also since most of advanced scripting
> is done by interacting with Admin directly (see region_mover.rb or
> copy_tables_desc.rb) I don't think we should worry too much about backwards
> compatibility if that gets properly documented.
>
> thanks,
> esteban.
>
>
>
> --
> Cloudera, Inc.
>
>
> On Wed, May 13, 2015 at 10:25 AM, Andrew Purtell 
> wrote:
>
> > I'd be curious to hear proposals for a new shell. Wondering what the
> > arguments in favor would be and arguments against current.
> >
> > JRuby has served us well. Recently it personally saved me hassle by
> > allowing scripted surgery (advanced ops) rather than dev of a one off
> Java
> > utility. OTOH, if what we had available for scripting was the Nashorn JS
> > engine (Java 8+) instead, that would have worked well also.
> >
> > And if we're talking about new shells, what about a SQL shell? /me ducks
> > and runs for cover
> >
> >
> > On Wed, May 13, 2015 at 10:19 AM, Stack  wrote:
> >
> > > Nice writeup Sean.
> > >
> > > Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license is
> > > still amenable and hopefully jruby 9k will be slimmer than jruby 1.7+.
> > >
> > > But if we are going to do a significant shell refactor for hbase 2.0,
> > > should we consider doing something more radical; e.g. a new shell? If
> > > interest, could start up a new thread so don't distract from this one.
> > >
> > > St.Ack
> > >
> > >
> > >
> > >
> > > On Wed, May 13, 2015 at 9:22 AM, Sean Busbey 
> > wrote:
> > >
> > > > Hi folks!
> > > >
> > > > If you weren't aware, our current shell relies on Ruby, specifically
> > the
> > > > REPL program IRB from JRuby. When we launched 1.0 we were on JRuby
> 1.6
> > > with
> > > > defaults, which means we're stuck on Ruby 1.8.
> > > >
> > > > For those that don't already know, Ruby 1.8 is super old and has been
> > > > walking off into the sunset for a few years now. Most (but not all!)
> > > formal
> > > > support systems for running Ruby have EOLed 1.8 and there are
> numerous
> > > > known issues with it.
> > > >
> > > > Right now there's an open ticket to get us on JRuby 1.7 so that our
> > shell
> > > > can work on PPC systems[1]. That version of JRuby defaults to Ruby
> 1.9
> > > but
> > > > can be run in Ruby 1.8 mode. There are some implementation details
> > > > outstanding, but I'm hoping that ticket can work out such that it can
> > > land
> > > > in branch-1.
> > > >
> > > > For HBase 2.0, I'd like us to plan for a little farther out in the
> > future
> > > > than just updating to Ruby 1.9 (though that would be a fine
> incremental
> > > > step with some non-trivial work attached). The "current" version of
> > Ruby
> > > is
> > > > 2.2. Much like the move from 1.8 -> 1.9 it is not backwards
> compatible.
> > > >
> > > > JRuby's next major maintenance release line is "version 9000"[2] and
> it
> > > > will start out *only* supporting Ruby 2.2. Right now JRuby 9000 is in
> > its
> > > > second "preview" release. They still have a few feature complete
> items
> > to
> > > > address before they hit their first GA release.
> > > >
> > > > I'd like us to move to Ruby 2.2 via JRuby 9000 for HBase 2.0.  This
> > will
> > > > cause operator pain to folks with advanced scripts based on Ruby 1.8,
> > but
> > > > it should allow us to update versions to avoid e.g. perf,
> correctness,
> > > and
> > > > security issues much more easily over the lifetime of 2.0.
> > > >
> > > > What do folks think? Would JRuby 9000 need to hit a GA release prior
> to
> > > > HBase 2.0 getting released for us to adopt it? Or would it only need
> > > enough
> > > > of Ruby 2.2 to run our current shell?
> > > >
> > > >
> > > > [1]: https://issues.apache.org/jira/browse/HBASE-13338
> > > > [2]: http://www.slideshare.net/CharlesNutter/over-9000-jruby-in-2015
> > > >
> > > > --
> > > > Sean
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> >- Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>



-- 
Best regards,

   - Andy

Problems worthy of attack pro

Re: Ruby shell versions for HBase 2.0

[Andrew Purtell]

This is because the Accumulo shell is a custom built shell? If so, we had
one of those once and replaced it with the IRB based one. We didn't settle
on JRuby right away but, at the time, the consensus was we didn't want to
be in the business of maintaining yet another REPL when specialist open
source communities have already done that. Why has this changed? Has it?

Why is the Accumulo shell easier to script with? Does it have control flow
constructs? Variable assignment? Or is it easier somehow because it does
not have those things? Even the system shell has those. Hmm... Are we
saying that any control flow or variables/substitution should be done by
calling our shell from a bash or sh script? Wouldn't that be super slow
given Java's startup time for every invocation of our shell? A bit like
trying to script the AWS command line utilities, which is excruciating.

I can see why a custom shell built with Java would easier to test where the
rest of the project is using Surefire/JUnit. That's a developer convenience
concern though, not a user convenience one.


On Wed, May 13, 2015 at 10:41 AM, Sean Busbey  wrote:

> Pros:
>
> * It's easier to test and maintain
> * it's easier to script with
> * its interactive mode "feels" mode focused on the task of interacting with
> a cluster to me (maybe this is just acting like a psql or mysql shell)
>
> Cons
>
> * adding custom commands requires knowing java
>
> --
> Sean
> On May 13, 2015 12:31 PM, "Andrew Purtell"  wrote:
>
> > Why is the Accumulo shell superior?
> >
> > Is it scriptable?
> >
> >
> > On Wed, May 13, 2015 at 10:28 AM, Sean Busbey 
> wrote:
> >
> > > I would love to rip out the JRuby shell entirely and make something
> > closer
> > > to the Accumulo shell, but I expect that will
> > >
> > > 1) be way more work
> > >
> > > 2) be even less compatible for those that rely on customizations.
> > >
> > > I figured given time we could get a preview "user shell" (rather than
> > > "power shell" via irb) together in 2.0 and aim for default in 3.0.
> > >
> > > --
> > > Sean
> > > On May 13, 2015 12:19 PM, "Stack"  wrote:
> > >
> > > > Nice writeup Sean.
> > > >
> > > > Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license
> is
> > > > still amenable and hopefully jruby 9k will be slimmer than jruby
> 1.7+.
> > > >
> > > > But if we are going to do a significant shell refactor for hbase 2.0,
> > > > should we consider doing something more radical; e.g. a new shell? If
> > > > interest, could start up a new thread so don't distract from this
> one.
> > > >
> > > > St.Ack
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, May 13, 2015 at 9:22 AM, Sean Busbey 
> > > wrote:
> > > >
> > > > > Hi folks!
> > > > >
> > > > > If you weren't aware, our current shell relies on Ruby,
> specifically
> > > the
> > > > > REPL program IRB from JRuby. When we launched 1.0 we were on JRuby
> > 1.6
> > > > with
> > > > > defaults, which means we're stuck on Ruby 1.8.
> > > > >
> > > > > For those that don't already know, Ruby 1.8 is super old and has
> been
> > > > > walking off into the sunset for a few years now. Most (but not
> all!)
> > > > formal
> > > > > support systems for running Ruby have EOLed 1.8 and there are
> > numerous
> > > > > known issues with it.
> > > > >
> > > > > Right now there's an open ticket to get us on JRuby 1.7 so that our
> > > shell
> > > > > can work on PPC systems[1]. That version of JRuby defaults to Ruby
> > 1.9
> > > > but
> > > > > can be run in Ruby 1.8 mode. There are some implementation details
> > > > > outstanding, but I'm hoping that ticket can work out such that it
> can
> > > > land
> > > > > in branch-1.
> > > > >
> > > > > For HBase 2.0, I'd like us to plan for a little farther out in the
> > > future
> > > > > than just updating to Ruby 1.9 (though that would be a fine
> > incremental
> > > > > step with some non-trivial work attached). The "current" version of
> > > Ruby
> > > > is
> > > > > 2.2. Much like the move from 1.8 -> 1.9 it is not backwards
> > compatible.
> > > > >
> > > > > JRuby's next major maintenance release line is "version 9000"[2]
> and
> > it
> > > > > will start out *only* supporting Ruby 2.2. Right now JRuby 9000 is
> in
> > > its
> > > > > second "preview" release. They still have a few feature complete
> > items
> > > to
> > > > > address before they hit their first GA release.
> > > > >
> > > > > I'd like us to move to Ruby 2.2 via JRuby 9000 for HBase 2.0.  This
> > > will
> > > > > cause operator pain to folks with advanced scripts based on Ruby
> 1.8,
> > > but
> > > > > it should allow us to update versions to avoid e.g. perf,
> > correctness,
> > > > and
> > > > > security issues much more easily over the lifetime of 2.0.
> > > > >
> > > > > What do folks think? Would JRuby 9000 need to hit a GA release
> prior
> > to
> > > > > HBase 2.0 getting released for us to adopt it? Or would it only
> need
> > > > enough
> > > > > of Ruby 2.2 to run our current shell?
> > > > >
> > > > >
> > > > > [1]

Re: Ruby shell versions for HBase 2.0

[Josh Elser]

Agreed on these points, Sean. Having used both, I think both approaches 
have their value and their drawbacks.


The ruby shell is _wonderful_ from having a full programming language to 
interact with. Accumulo's shell would force you to use your standard 
unix-toolbelt if you want to do any extra parsing/logic.


OTOH, general introspection and interaction with the system feels much 
more natural to me w/ Accumulo's shell. I know that's not a good way to 
quantify my feelings. I think it's mostly due to not having to write 
Ruby when I'm not actually scripting things. Another point is that 
Accumulo uses JLine under the hood which always comes with its own 
burden (things like C^c nuked the entire shell for the longest time 
because JLine didn't support it -- we sometimes have to fix JLine to get 
the functionality we want).


I also caught wind of someone actually using JavaScript via JSR-223 w/ 
the Accumulo shell, but it's not widely advertised at this point.


Finding a the right mix between ease of scripting and simplicity of 
administrative interactions is key if we want to move beyond what the 
HBase shell is now. My $0.02.


(and, to not completely derail the original conversation, moving to Ruby 
2.2 + JRuby 9000 would be what I think the right move to make is)


Sean Busbey wrote:

Pros:

* It's easier to test and maintain
* it's easier to script with
* its interactive mode "feels" mode focused on the task of interacting with
a cluster to me (maybe this is just acting like a psql or mysql shell)

Cons

* adding custom commands requires knowing java

Re: Problems with Phoenix and HBase

[Alicia Shu]

Please follow the doc
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.3/bk_installing_manua
lly_book/content/rpm-chap-phoenix.html and make sure your setup is all
right.

Alicia

On 5/13/15, 10:05 AM, "Asfare"  wrote:

>Hi, the shell problem is solved.
>
>Now the problem is this:
>sqlline.py localhost
>Setting property: [isolation, TRANSACTION_READ_COMMITTED]
>issuing: !connect jdbc:phoenix:localhost none none
>org.apache.phoenix.jdbc.PhoenixDriver
>Connecting to jdbc:phoenix:localhost
>SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>SLF4J: Defaulting to no-operation (NOP) logger implementation
>SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
>details.
>15/05/13 19:04:11 WARN util.NativeCodeLoader: Unable to load native-hadoop
>library for your platform... using builtin-java classes where applicable
>
>And no more
>
>
>In the other hand the master log is:
>2015-05-13 18:38:16,016 INFO  [AM.ZK.Worker-pool2-t13]
>master.RegionStates:
>Transition {3c89ac6db5c278c3d74e3801fd2b0005 state=OPENING,
>ts=1431535095997, server=myname.mycompany.com,16201,1431535081143} to
>{3c89ac6db5c278c3d74e3801fd2b0005 state=CLOSED, ts=1431535096016,
>server=myname.mycompany.com,16201,1431535081143}
>2015-05-13 18:38:16,016 WARN  [AM.ZK.Worker-pool2-t13]
>master.AssignmentManager: Can't move 3c89ac6db5c278c3d74e3801fd2b0005,
>there
>is no destination server available.
>2015-05-13 18:38:16,017 INFO  [AM.ZK.Worker-pool2-t13]
>master.RegionStates:
>Transition {3c89ac6db5c278c3d74e3801fd2b0005 state=CLOSED,
>ts=1431535096016,
>server=myname.mycompany.com,16201,1431535081143} to
>{3c89ac6db5c278c3d74e3801fd2b0005 state=OFFLINE, ts=1431535096017,
>server=myname.mycompany.com,16201,1431535081143}
>2015-05-13 18:38:16,017 INFO  [AM.ZK.Worker-pool2-t17]
>master.RegionStates:
>Transition {c76be7ccc0d249d9e9f31e6f45a9f7ab state=PENDING_OPEN,
>ts=1431535095984, server=myname.mycompany.com,16201,1431535081143} to
>{c76be7ccc0d249d9e9f31e6f45a9f7ab state=OPENING, ts=1431535096017,
>server=myname.mycompany.com,16201,1431535081143}
>2015-05-13 18:38:16,021 INFO  [AM.ZK.Worker-pool2-t13]
>master.AssignmentManager: Assigning
>dummy,,1430393911021.3c89ac6db5c278c3d74e3801fd2b0005. to
>myname.mycompany.com,16201,1431535081143
>2015-05-13 18:38:16,022 INFO  [AM.ZK.Worker-pool2-t13]
>master.RegionStates:
>Transition {3c89ac6db5c278c3d74e3801fd2b0005 state=OFFLINE,
>ts=1431535096017, server=myname.mycompany.com,16201,1431535081143} to
>{3c89ac6db5c278c3d74e3801fd2b0005 state=PENDING_OPEN, ts=1431535096022,
>server=myname.mycompany.com,16201,1431535081143}
>2015-05-13 18:38:16,026 INFO  [AM.ZK.Worker-pool2-t15]
>master.RegionStates:
>Transition {c76be7ccc0d249d9e9f31e6f45a9f7ab state=OPENING,
>ts=1431535096017, server=myname.mycompany.com,16201,1431535081143} to
>{c76be7ccc0d249d9e9f31e6f45a9f7ab state=CLOSED, ts=1431535096026,
>server=myname.mycompany.com,16201,1431535081143}
>2015-05-13 18:38:16,026 WARN  [AM.ZK.Worker-pool2-t15]
>master.AssignmentManager: Can't move c76be7ccc0d249d9e9f31e6f45a9f7ab,
>there
>is no destination server available.
>2015-05-13 18:38:16,026 INFO  [AM.ZK.Worker-pool2-t15]
>master.RegionStates:
>Transition {c76be7ccc0d249d9e9f31e6f45a9f7ab state=CLOSED,
>ts=1431535096026,
>server=myname.mycompany.com,16201,1431535081143} to
>{c76be7ccc0d249d9e9f31e6f45a9f7ab state=OFFLINE, ts=1431535096026,
>server=myname.mycompany.com,16201,1431535081143}
>2015-05-13 18:38:16,035 INFO  [AM.ZK.Worker-pool2-t15]
>master.AssignmentManager: Assigning
>dummy2,,1430396515524.c76be7ccc0d249d9e9f31e6f45a9f7ab. to
>myname.mycompany.com,16201,1431535081143
>2015-05-13 18:38:16,035 INFO  [AM.ZK.Worker-pool2-t15]
>master.RegionStates:
>Transition {c76be7ccc0d249d9e9f31e6f45a9f7ab state=OFFLINE,
>ts=1431535096026, server=myname.mycompany.com,16201,1431535081143} to
>{c76be7ccc0d249d9e9f31e6f45a9f7ab state=PENDING_OPEN, ts=1431535096035,
>server=myname.mycompany.com,16201,1431535081143}
>2015-05-13 18:38:16,040 INFO  [AM.ZK.Worker-pool2-t19]
>master.RegionStates:
>Transition {3c89ac6db5c278c3d74e3801fd2b0005 state=PENDING_OPEN,
>ts=1431535096022, server=myname.mycompany.com,16201,1431535081143} to
>{3c89ac6db5c278c3d74e3801fd2b0005 state=OPENING, ts=1431535096040,
>server=myname.mycompany.com,16201,1431535081143}
>2015-05-13 18:38:16,048 WARN  [AM.ZK.Worker-pool2-t10]
>master.RegionStates:
>Failed to open/close 3c89ac6db5c278c3d74e3801fd2b0005 on
>myname.mycompany.com,16201,1431535081143, set to FAILED_OPEN
>2015-05-13 18:38:16,048 INFO  [AM.ZK.Worker-pool2-t10]
>master.RegionStates:
>Transition {3c89ac6db5c278c3d74e3801fd2b0005 state=OPENING,
>ts=1431535096040, server=myname.mycompany.com,16201,1431535081143} to
>{3c89ac6db5c278c3d74e3801fd2b0005 state=FAILED_OPEN, ts=1431535096048,
>server=myname.mycompany.com,16201,1431535081143}
>2015-05-13 18:38:16,058 INFO  [AM.ZK.Worker-pool2-t15]
>master.RegionStates:
>Transition {c76be7ccc0d24

Re: Ruby shell versions for HBase 2.0

[Esteban Gutierrez]

+1 for new JRuby for now and having a JIRA to discuss what features should
this shell have would be great.

However, I think that during the last few years many of us have used the
hbase shell for major surgery as Andrew said and it has added some
confusion to users about what the hbase shell is used for and I think that
functionality shouldn't be longer exposed to regular users. One way we
could probably do this is to have a user facing CLI that will be more for
DDL, permissions and data interaction for users (sort of psql or phoenix
/me runs too) and deprecate/update the old shell just leaving table/region
management functionality for admins, also since most of advanced scripting
is done by interacting with Admin directly (see region_mover.rb or
copy_tables_desc.rb) I don't think we should worry too much about backwards
compatibility if that gets properly documented.

thanks,
esteban.



--
Cloudera, Inc.


On Wed, May 13, 2015 at 10:25 AM, Andrew Purtell 
wrote:

> I'd be curious to hear proposals for a new shell. Wondering what the
> arguments in favor would be and arguments against current.
>
> JRuby has served us well. Recently it personally saved me hassle by
> allowing scripted surgery (advanced ops) rather than dev of a one off Java
> utility. OTOH, if what we had available for scripting was the Nashorn JS
> engine (Java 8+) instead, that would have worked well also.
>
> And if we're talking about new shells, what about a SQL shell? /me ducks
> and runs for cover
>
>
> On Wed, May 13, 2015 at 10:19 AM, Stack  wrote:
>
> > Nice writeup Sean.
> >
> > Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license is
> > still amenable and hopefully jruby 9k will be slimmer than jruby 1.7+.
> >
> > But if we are going to do a significant shell refactor for hbase 2.0,
> > should we consider doing something more radical; e.g. a new shell? If
> > interest, could start up a new thread so don't distract from this one.
> >
> > St.Ack
> >
> >
> >
> >
> > On Wed, May 13, 2015 at 9:22 AM, Sean Busbey 
> wrote:
> >
> > > Hi folks!
> > >
> > > If you weren't aware, our current shell relies on Ruby, specifically
> the
> > > REPL program IRB from JRuby. When we launched 1.0 we were on JRuby 1.6
> > with
> > > defaults, which means we're stuck on Ruby 1.8.
> > >
> > > For those that don't already know, Ruby 1.8 is super old and has been
> > > walking off into the sunset for a few years now. Most (but not all!)
> > formal
> > > support systems for running Ruby have EOLed 1.8 and there are numerous
> > > known issues with it.
> > >
> > > Right now there's an open ticket to get us on JRuby 1.7 so that our
> shell
> > > can work on PPC systems[1]. That version of JRuby defaults to Ruby 1.9
> > but
> > > can be run in Ruby 1.8 mode. There are some implementation details
> > > outstanding, but I'm hoping that ticket can work out such that it can
> > land
> > > in branch-1.
> > >
> > > For HBase 2.0, I'd like us to plan for a little farther out in the
> future
> > > than just updating to Ruby 1.9 (though that would be a fine incremental
> > > step with some non-trivial work attached). The "current" version of
> Ruby
> > is
> > > 2.2. Much like the move from 1.8 -> 1.9 it is not backwards compatible.
> > >
> > > JRuby's next major maintenance release line is "version 9000"[2] and it
> > > will start out *only* supporting Ruby 2.2. Right now JRuby 9000 is in
> its
> > > second "preview" release. They still have a few feature complete items
> to
> > > address before they hit their first GA release.
> > >
> > > I'd like us to move to Ruby 2.2 via JRuby 9000 for HBase 2.0.  This
> will
> > > cause operator pain to folks with advanced scripts based on Ruby 1.8,
> but
> > > it should allow us to update versions to avoid e.g. perf, correctness,
> > and
> > > security issues much more easily over the lifetime of 2.0.
> > >
> > > What do folks think? Would JRuby 9000 need to hit a GA release prior to
> > > HBase 2.0 getting released for us to adopt it? Or would it only need
> > enough
> > > of Ruby 2.2 to run our current shell?
> > >
> > >
> > > [1]: https://issues.apache.org/jira/browse/HBASE-13338
> > > [2]: http://www.slideshare.net/CharlesNutter/over-9000-jruby-in-2015
> > >
> > > --
> > > Sean
> > >
> >
>
>
>
> --
> Best regards,
>
>- Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Re: Ruby shell versions for HBase 2.0

[Sean Busbey]

Pros:

* It's easier to test and maintain
* it's easier to script with
* its interactive mode "feels" mode focused on the task of interacting with
a cluster to me (maybe this is just acting like a psql or mysql shell)

Cons

* adding custom commands requires knowing java

-- 
Sean
On May 13, 2015 12:31 PM, "Andrew Purtell"  wrote:

> Why is the Accumulo shell superior?
>
> Is it scriptable?
>
>
> On Wed, May 13, 2015 at 10:28 AM, Sean Busbey  wrote:
>
> > I would love to rip out the JRuby shell entirely and make something
> closer
> > to the Accumulo shell, but I expect that will
> >
> > 1) be way more work
> >
> > 2) be even less compatible for those that rely on customizations.
> >
> > I figured given time we could get a preview "user shell" (rather than
> > "power shell" via irb) together in 2.0 and aim for default in 3.0.
> >
> > --
> > Sean
> > On May 13, 2015 12:19 PM, "Stack"  wrote:
> >
> > > Nice writeup Sean.
> > >
> > > Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license is
> > > still amenable and hopefully jruby 9k will be slimmer than jruby 1.7+.
> > >
> > > But if we are going to do a significant shell refactor for hbase 2.0,
> > > should we consider doing something more radical; e.g. a new shell? If
> > > interest, could start up a new thread so don't distract from this one.
> > >
> > > St.Ack
> > >
> > >
> > >
> > >
> > > On Wed, May 13, 2015 at 9:22 AM, Sean Busbey 
> > wrote:
> > >
> > > > Hi folks!
> > > >
> > > > If you weren't aware, our current shell relies on Ruby, specifically
> > the
> > > > REPL program IRB from JRuby. When we launched 1.0 we were on JRuby
> 1.6
> > > with
> > > > defaults, which means we're stuck on Ruby 1.8.
> > > >
> > > > For those that don't already know, Ruby 1.8 is super old and has been
> > > > walking off into the sunset for a few years now. Most (but not all!)
> > > formal
> > > > support systems for running Ruby have EOLed 1.8 and there are
> numerous
> > > > known issues with it.
> > > >
> > > > Right now there's an open ticket to get us on JRuby 1.7 so that our
> > shell
> > > > can work on PPC systems[1]. That version of JRuby defaults to Ruby
> 1.9
> > > but
> > > > can be run in Ruby 1.8 mode. There are some implementation details
> > > > outstanding, but I'm hoping that ticket can work out such that it can
> > > land
> > > > in branch-1.
> > > >
> > > > For HBase 2.0, I'd like us to plan for a little farther out in the
> > future
> > > > than just updating to Ruby 1.9 (though that would be a fine
> incremental
> > > > step with some non-trivial work attached). The "current" version of
> > Ruby
> > > is
> > > > 2.2. Much like the move from 1.8 -> 1.9 it is not backwards
> compatible.
> > > >
> > > > JRuby's next major maintenance release line is "version 9000"[2] and
> it
> > > > will start out *only* supporting Ruby 2.2. Right now JRuby 9000 is in
> > its
> > > > second "preview" release. They still have a few feature complete
> items
> > to
> > > > address before they hit their first GA release.
> > > >
> > > > I'd like us to move to Ruby 2.2 via JRuby 9000 for HBase 2.0.  This
> > will
> > > > cause operator pain to folks with advanced scripts based on Ruby 1.8,
> > but
> > > > it should allow us to update versions to avoid e.g. perf,
> correctness,
> > > and
> > > > security issues much more easily over the lifetime of 2.0.
> > > >
> > > > What do folks think? Would JRuby 9000 need to hit a GA release prior
> to
> > > > HBase 2.0 getting released for us to adopt it? Or would it only need
> > > enough
> > > > of Ruby 2.2 to run our current shell?
> > > >
> > > >
> > > > [1]: https://issues.apache.org/jira/browse/HBASE-13338
> > > > [2]: http://www.slideshare.net/CharlesNutter/over-9000-jruby-in-2015
> > > >
> > > > --
> > > > Sean
> > > >
> > >
> >
>
>
>
> --
> Best regards,
>
>- Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Re: Ruby shell versions for HBase 2.0

[Sean Busbey]

On May 13, 2015 12:06 PM, "Michael Segel"  wrote:
>
> So…
> Silly question…
> Do you really need to worry about backward’s compatibility?
>
> How many people have customized HBaseShell ?
>
> What are the common customizations and if you port HBase shell, how much
work would filter through to the custom code?
>
>

These are excellent questions to which I have poor answers.

The mechanics of the interactive repl via IRB will probably not be that
different. However, the ref guide calls out the ability to customize things
via hooking in your own ruby script so there's little bound on what we've
set for expectations. Given ruby 1.8's age, I'm not even sure there are 1.8
to 2.2 guides; likely folks would need to follow 1.8 to 1.9 and 1.9 to 2.2
guides (or we could work to provide some combined guidance).

Customizations are just as likely to be "call this sequence of commands"
(little to no work for update) as a daemon that does health and recovery
checks (lots of work, approaching "rewrite").

Personally, I avoid customizations at least in part because it's ruby 1.8.
I've also discouraged those who ask me. Instead I rely on the
non-interactive flag and bash.  But given the ref guide positioning, it's
likely we'll break a nontrivial number of folks.

Re: Ruby shell versions for HBase 2.0

[Andrew Purtell]

Why is the Accumulo shell superior?

Is it scriptable?


On Wed, May 13, 2015 at 10:28 AM, Sean Busbey  wrote:

> I would love to rip out the JRuby shell entirely and make something closer
> to the Accumulo shell, but I expect that will
>
> 1) be way more work
>
> 2) be even less compatible for those that rely on customizations.
>
> I figured given time we could get a preview "user shell" (rather than
> "power shell" via irb) together in 2.0 and aim for default in 3.0.
>
> --
> Sean
> On May 13, 2015 12:19 PM, "Stack"  wrote:
>
> > Nice writeup Sean.
> >
> > Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license is
> > still amenable and hopefully jruby 9k will be slimmer than jruby 1.7+.
> >
> > But if we are going to do a significant shell refactor for hbase 2.0,
> > should we consider doing something more radical; e.g. a new shell? If
> > interest, could start up a new thread so don't distract from this one.
> >
> > St.Ack
> >
> >
> >
> >
> > On Wed, May 13, 2015 at 9:22 AM, Sean Busbey 
> wrote:
> >
> > > Hi folks!
> > >
> > > If you weren't aware, our current shell relies on Ruby, specifically
> the
> > > REPL program IRB from JRuby. When we launched 1.0 we were on JRuby 1.6
> > with
> > > defaults, which means we're stuck on Ruby 1.8.
> > >
> > > For those that don't already know, Ruby 1.8 is super old and has been
> > > walking off into the sunset for a few years now. Most (but not all!)
> > formal
> > > support systems for running Ruby have EOLed 1.8 and there are numerous
> > > known issues with it.
> > >
> > > Right now there's an open ticket to get us on JRuby 1.7 so that our
> shell
> > > can work on PPC systems[1]. That version of JRuby defaults to Ruby 1.9
> > but
> > > can be run in Ruby 1.8 mode. There are some implementation details
> > > outstanding, but I'm hoping that ticket can work out such that it can
> > land
> > > in branch-1.
> > >
> > > For HBase 2.0, I'd like us to plan for a little farther out in the
> future
> > > than just updating to Ruby 1.9 (though that would be a fine incremental
> > > step with some non-trivial work attached). The "current" version of
> Ruby
> > is
> > > 2.2. Much like the move from 1.8 -> 1.9 it is not backwards compatible.
> > >
> > > JRuby's next major maintenance release line is "version 9000"[2] and it
> > > will start out *only* supporting Ruby 2.2. Right now JRuby 9000 is in
> its
> > > second "preview" release. They still have a few feature complete items
> to
> > > address before they hit their first GA release.
> > >
> > > I'd like us to move to Ruby 2.2 via JRuby 9000 for HBase 2.0.  This
> will
> > > cause operator pain to folks with advanced scripts based on Ruby 1.8,
> but
> > > it should allow us to update versions to avoid e.g. perf, correctness,
> > and
> > > security issues much more easily over the lifetime of 2.0.
> > >
> > > What do folks think? Would JRuby 9000 need to hit a GA release prior to
> > > HBase 2.0 getting released for us to adopt it? Or would it only need
> > enough
> > > of Ruby 2.2 to run our current shell?
> > >
> > >
> > > [1]: https://issues.apache.org/jira/browse/HBASE-13338
> > > [2]: http://www.slideshare.net/CharlesNutter/over-9000-jruby-in-2015
> > >
> > > --
> > > Sean
> > >
> >
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: Ruby shell versions for HBase 2.0

[Sean Busbey]

I would love to rip out the JRuby shell entirely and make something closer
to the Accumulo shell, but I expect that will

1) be way more work

2) be even less compatible for those that rely on customizations.

I figured given time we could get a preview "user shell" (rather than
"power shell" via irb) together in 2.0 and aim for default in 3.0.

-- 
Sean
On May 13, 2015 12:19 PM, "Stack"  wrote:

> Nice writeup Sean.
>
> Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license is
> still amenable and hopefully jruby 9k will be slimmer than jruby 1.7+.
>
> But if we are going to do a significant shell refactor for hbase 2.0,
> should we consider doing something more radical; e.g. a new shell? If
> interest, could start up a new thread so don't distract from this one.
>
> St.Ack
>
>
>
>
> On Wed, May 13, 2015 at 9:22 AM, Sean Busbey  wrote:
>
> > Hi folks!
> >
> > If you weren't aware, our current shell relies on Ruby, specifically the
> > REPL program IRB from JRuby. When we launched 1.0 we were on JRuby 1.6
> with
> > defaults, which means we're stuck on Ruby 1.8.
> >
> > For those that don't already know, Ruby 1.8 is super old and has been
> > walking off into the sunset for a few years now. Most (but not all!)
> formal
> > support systems for running Ruby have EOLed 1.8 and there are numerous
> > known issues with it.
> >
> > Right now there's an open ticket to get us on JRuby 1.7 so that our shell
> > can work on PPC systems[1]. That version of JRuby defaults to Ruby 1.9
> but
> > can be run in Ruby 1.8 mode. There are some implementation details
> > outstanding, but I'm hoping that ticket can work out such that it can
> land
> > in branch-1.
> >
> > For HBase 2.0, I'd like us to plan for a little farther out in the future
> > than just updating to Ruby 1.9 (though that would be a fine incremental
> > step with some non-trivial work attached). The "current" version of Ruby
> is
> > 2.2. Much like the move from 1.8 -> 1.9 it is not backwards compatible.
> >
> > JRuby's next major maintenance release line is "version 9000"[2] and it
> > will start out *only* supporting Ruby 2.2. Right now JRuby 9000 is in its
> > second "preview" release. They still have a few feature complete items to
> > address before they hit their first GA release.
> >
> > I'd like us to move to Ruby 2.2 via JRuby 9000 for HBase 2.0.  This will
> > cause operator pain to folks with advanced scripts based on Ruby 1.8, but
> > it should allow us to update versions to avoid e.g. perf, correctness,
> and
> > security issues much more easily over the lifetime of 2.0.
> >
> > What do folks think? Would JRuby 9000 need to hit a GA release prior to
> > HBase 2.0 getting released for us to adopt it? Or would it only need
> enough
> > of Ruby 2.2 to run our current shell?
> >
> >
> > [1]: https://issues.apache.org/jira/browse/HBASE-13338
> > [2]: http://www.slideshare.net/CharlesNutter/over-9000-jruby-in-2015
> >
> > --
> > Sean
> >
>

Re: Ruby shell versions for HBase 2.0

[Andrew Purtell]

I'd be curious to hear proposals for a new shell. Wondering what the
arguments in favor would be and arguments against current.

JRuby has served us well. Recently it personally saved me hassle by
allowing scripted surgery (advanced ops) rather than dev of a one off Java
utility. OTOH, if what we had available for scripting was the Nashorn JS
engine (Java 8+) instead, that would have worked well also.

And if we're talking about new shells, what about a SQL shell? /me ducks
and runs for cover


On Wed, May 13, 2015 at 10:19 AM, Stack  wrote:

> Nice writeup Sean.
>
> Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license is
> still amenable and hopefully jruby 9k will be slimmer than jruby 1.7+.
>
> But if we are going to do a significant shell refactor for hbase 2.0,
> should we consider doing something more radical; e.g. a new shell? If
> interest, could start up a new thread so don't distract from this one.
>
> St.Ack
>
>
>
>
> On Wed, May 13, 2015 at 9:22 AM, Sean Busbey  wrote:
>
> > Hi folks!
> >
> > If you weren't aware, our current shell relies on Ruby, specifically the
> > REPL program IRB from JRuby. When we launched 1.0 we were on JRuby 1.6
> with
> > defaults, which means we're stuck on Ruby 1.8.
> >
> > For those that don't already know, Ruby 1.8 is super old and has been
> > walking off into the sunset for a few years now. Most (but not all!)
> formal
> > support systems for running Ruby have EOLed 1.8 and there are numerous
> > known issues with it.
> >
> > Right now there's an open ticket to get us on JRuby 1.7 so that our shell
> > can work on PPC systems[1]. That version of JRuby defaults to Ruby 1.9
> but
> > can be run in Ruby 1.8 mode. There are some implementation details
> > outstanding, but I'm hoping that ticket can work out such that it can
> land
> > in branch-1.
> >
> > For HBase 2.0, I'd like us to plan for a little farther out in the future
> > than just updating to Ruby 1.9 (though that would be a fine incremental
> > step with some non-trivial work attached). The "current" version of Ruby
> is
> > 2.2. Much like the move from 1.8 -> 1.9 it is not backwards compatible.
> >
> > JRuby's next major maintenance release line is "version 9000"[2] and it
> > will start out *only* supporting Ruby 2.2. Right now JRuby 9000 is in its
> > second "preview" release. They still have a few feature complete items to
> > address before they hit their first GA release.
> >
> > I'd like us to move to Ruby 2.2 via JRuby 9000 for HBase 2.0.  This will
> > cause operator pain to folks with advanced scripts based on Ruby 1.8, but
> > it should allow us to update versions to avoid e.g. perf, correctness,
> and
> > security issues much more easily over the lifetime of 2.0.
> >
> > What do folks think? Would JRuby 9000 need to hit a GA release prior to
> > HBase 2.0 getting released for us to adopt it? Or would it only need
> enough
> > of Ruby 2.2 to run our current shell?
> >
> >
> > [1]: https://issues.apache.org/jira/browse/HBASE-13338
> > [2]: http://www.slideshare.net/CharlesNutter/over-9000-jruby-in-2015
> >
> > --
> > Sean
> >
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: Ruby shell versions for HBase 2.0

[Darion Yaphet]

sorry to say I don't good at Ruby .
HBase Shell may should upgrade :)

2015-05-14 1:06 GMT+08:00 Michael Segel :

> So…
> Silly question…
> Do you really need to worry about backward’s compatibility?
>
> How many people have customized HBaseShell ?
>
> What are the common customizations and if you port HBase shell, how much
> work would filter through to the custom code?
>
>
> > On May 13, 2015, at 11:22 AM, Sean Busbey  wrote:
> >
> > Hi folks!
> >
> > If you weren't aware, our current shell relies on Ruby, specifically the
> > REPL program IRB from JRuby. When we launched 1.0 we were on JRuby 1.6
> with
> > defaults, which means we're stuck on Ruby 1.8.
> >
> > For those that don't already know, Ruby 1.8 is super old and has been
> > walking off into the sunset for a few years now. Most (but not all!)
> formal
> > support systems for running Ruby have EOLed 1.8 and there are numerous
> > known issues with it.
> >
> > Right now there's an open ticket to get us on JRuby 1.7 so that our shell
> > can work on PPC systems[1]. That version of JRuby defaults to Ruby 1.9
> but
> > can be run in Ruby 1.8 mode. There are some implementation details
> > outstanding, but I'm hoping that ticket can work out such that it can
> land
> > in branch-1.
> >
> > For HBase 2.0, I'd like us to plan for a little farther out in the future
> > than just updating to Ruby 1.9 (though that would be a fine incremental
> > step with some non-trivial work attached). The "current" version of Ruby
> is
> > 2.2. Much like the move from 1.8 -> 1.9 it is not backwards compatible.
> >
> > JRuby's next major maintenance release line is "version 9000"[2] and it
> > will start out *only* supporting Ruby 2.2. Right now JRuby 9000 is in its
> > second "preview" release. They still have a few feature complete items to
> > address before they hit their first GA release.
> >
> > I'd like us to move to Ruby 2.2 via JRuby 9000 for HBase 2.0.  This will
> > cause operator pain to folks with advanced scripts based on Ruby 1.8, but
> > it should allow us to update versions to avoid e.g. perf, correctness,
> and
> > security issues much more easily over the lifetime of 2.0.
> >
> > What do folks think? Would JRuby 9000 need to hit a GA release prior to
> > HBase 2.0 getting released for us to adopt it? Or would it only need
> enough
> > of Ruby 2.2 to run our current shell?
> >
> >
> > [1]: https://issues.apache.org/jira/browse/HBASE-13338
> > [2]: http://www.slideshare.net/CharlesNutter/over-9000-jruby-in-2015
> >
> > --
> > Sean
>
>


-- 

long is the way and hard  that out of Hell leads up to light

Re: Ruby shell versions for HBase 2.0

[Stack]

Nice writeup Sean.

Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license is
still amenable and hopefully jruby 9k will be slimmer than jruby 1.7+.

But if we are going to do a significant shell refactor for hbase 2.0,
should we consider doing something more radical; e.g. a new shell? If
interest, could start up a new thread so don't distract from this one.

St.Ack




On Wed, May 13, 2015 at 9:22 AM, Sean Busbey  wrote:

> Hi folks!
>
> If you weren't aware, our current shell relies on Ruby, specifically the
> REPL program IRB from JRuby. When we launched 1.0 we were on JRuby 1.6 with
> defaults, which means we're stuck on Ruby 1.8.
>
> For those that don't already know, Ruby 1.8 is super old and has been
> walking off into the sunset for a few years now. Most (but not all!) formal
> support systems for running Ruby have EOLed 1.8 and there are numerous
> known issues with it.
>
> Right now there's an open ticket to get us on JRuby 1.7 so that our shell
> can work on PPC systems[1]. That version of JRuby defaults to Ruby 1.9 but
> can be run in Ruby 1.8 mode. There are some implementation details
> outstanding, but I'm hoping that ticket can work out such that it can land
> in branch-1.
>
> For HBase 2.0, I'd like us to plan for a little farther out in the future
> than just updating to Ruby 1.9 (though that would be a fine incremental
> step with some non-trivial work attached). The "current" version of Ruby is
> 2.2. Much like the move from 1.8 -> 1.9 it is not backwards compatible.
>
> JRuby's next major maintenance release line is "version 9000"[2] and it
> will start out *only* supporting Ruby 2.2. Right now JRuby 9000 is in its
> second "preview" release. They still have a few feature complete items to
> address before they hit their first GA release.
>
> I'd like us to move to Ruby 2.2 via JRuby 9000 for HBase 2.0.  This will
> cause operator pain to folks with advanced scripts based on Ruby 1.8, but
> it should allow us to update versions to avoid e.g. perf, correctness, and
> security issues much more easily over the lifetime of 2.0.
>
> What do folks think? Would JRuby 9000 need to hit a GA release prior to
> HBase 2.0 getting released for us to adopt it? Or would it only need enough
> of Ruby 2.2 to run our current shell?
>
>
> [1]: https://issues.apache.org/jira/browse/HBASE-13338
> [2]: http://www.slideshare.net/CharlesNutter/over-9000-jruby-in-2015
>
> --
> Sean
>

Re: Problems with Phoenix and HBase

[Asfare]

Hi, the shell problem is solved. 

Now the problem is this:
sqlline.py localhost
Setting property: [isolation, TRANSACTION_READ_COMMITTED]
issuing: !connect jdbc:phoenix:localhost none none
org.apache.phoenix.jdbc.PhoenixDriver
Connecting to jdbc:phoenix:localhost
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
15/05/13 19:04:11 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable

And no more


In the other hand the master log is:
2015-05-13 18:38:16,016 INFO  [AM.ZK.Worker-pool2-t13] master.RegionStates:
Transition {3c89ac6db5c278c3d74e3801fd2b0005 state=OPENING,
ts=1431535095997, server=myname.mycompany.com,16201,1431535081143} to
{3c89ac6db5c278c3d74e3801fd2b0005 state=CLOSED, ts=1431535096016,
server=myname.mycompany.com,16201,1431535081143}
2015-05-13 18:38:16,016 WARN  [AM.ZK.Worker-pool2-t13]
master.AssignmentManager: Can't move 3c89ac6db5c278c3d74e3801fd2b0005, there
is no destination server available.
2015-05-13 18:38:16,017 INFO  [AM.ZK.Worker-pool2-t13] master.RegionStates:
Transition {3c89ac6db5c278c3d74e3801fd2b0005 state=CLOSED, ts=1431535096016,
server=myname.mycompany.com,16201,1431535081143} to
{3c89ac6db5c278c3d74e3801fd2b0005 state=OFFLINE, ts=1431535096017,
server=myname.mycompany.com,16201,1431535081143}
2015-05-13 18:38:16,017 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates:
Transition {c76be7ccc0d249d9e9f31e6f45a9f7ab state=PENDING_OPEN,
ts=1431535095984, server=myname.mycompany.com,16201,1431535081143} to
{c76be7ccc0d249d9e9f31e6f45a9f7ab state=OPENING, ts=1431535096017,
server=myname.mycompany.com,16201,1431535081143}
2015-05-13 18:38:16,021 INFO  [AM.ZK.Worker-pool2-t13]
master.AssignmentManager: Assigning
dummy,,1430393911021.3c89ac6db5c278c3d74e3801fd2b0005. to
myname.mycompany.com,16201,1431535081143
2015-05-13 18:38:16,022 INFO  [AM.ZK.Worker-pool2-t13] master.RegionStates:
Transition {3c89ac6db5c278c3d74e3801fd2b0005 state=OFFLINE,
ts=1431535096017, server=myname.mycompany.com,16201,1431535081143} to
{3c89ac6db5c278c3d74e3801fd2b0005 state=PENDING_OPEN, ts=1431535096022,
server=myname.mycompany.com,16201,1431535081143}
2015-05-13 18:38:16,026 INFO  [AM.ZK.Worker-pool2-t15] master.RegionStates:
Transition {c76be7ccc0d249d9e9f31e6f45a9f7ab state=OPENING,
ts=1431535096017, server=myname.mycompany.com,16201,1431535081143} to
{c76be7ccc0d249d9e9f31e6f45a9f7ab state=CLOSED, ts=1431535096026,
server=myname.mycompany.com,16201,1431535081143}
2015-05-13 18:38:16,026 WARN  [AM.ZK.Worker-pool2-t15]
master.AssignmentManager: Can't move c76be7ccc0d249d9e9f31e6f45a9f7ab, there
is no destination server available.
2015-05-13 18:38:16,026 INFO  [AM.ZK.Worker-pool2-t15] master.RegionStates:
Transition {c76be7ccc0d249d9e9f31e6f45a9f7ab state=CLOSED, ts=1431535096026,
server=myname.mycompany.com,16201,1431535081143} to
{c76be7ccc0d249d9e9f31e6f45a9f7ab state=OFFLINE, ts=1431535096026,
server=myname.mycompany.com,16201,1431535081143}
2015-05-13 18:38:16,035 INFO  [AM.ZK.Worker-pool2-t15]
master.AssignmentManager: Assigning
dummy2,,1430396515524.c76be7ccc0d249d9e9f31e6f45a9f7ab. to
myname.mycompany.com,16201,1431535081143
2015-05-13 18:38:16,035 INFO  [AM.ZK.Worker-pool2-t15] master.RegionStates:
Transition {c76be7ccc0d249d9e9f31e6f45a9f7ab state=OFFLINE,
ts=1431535096026, server=myname.mycompany.com,16201,1431535081143} to
{c76be7ccc0d249d9e9f31e6f45a9f7ab state=PENDING_OPEN, ts=1431535096035,
server=myname.mycompany.com,16201,1431535081143}
2015-05-13 18:38:16,040 INFO  [AM.ZK.Worker-pool2-t19] master.RegionStates:
Transition {3c89ac6db5c278c3d74e3801fd2b0005 state=PENDING_OPEN,
ts=1431535096022, server=myname.mycompany.com,16201,1431535081143} to
{3c89ac6db5c278c3d74e3801fd2b0005 state=OPENING, ts=1431535096040,
server=myname.mycompany.com,16201,1431535081143}
2015-05-13 18:38:16,048 WARN  [AM.ZK.Worker-pool2-t10] master.RegionStates:
Failed to open/close 3c89ac6db5c278c3d74e3801fd2b0005 on
myname.mycompany.com,16201,1431535081143, set to FAILED_OPEN
2015-05-13 18:38:16,048 INFO  [AM.ZK.Worker-pool2-t10] master.RegionStates:
Transition {3c89ac6db5c278c3d74e3801fd2b0005 state=OPENING,
ts=1431535096040, server=myname.mycompany.com,16201,1431535081143} to
{3c89ac6db5c278c3d74e3801fd2b0005 state=FAILED_OPEN, ts=1431535096048,
server=myname.mycompany.com,16201,1431535081143}
2015-05-13 18:38:16,058 INFO  [AM.ZK.Worker-pool2-t15] master.RegionStates:
Transition {c76be7ccc0d249d9e9f31e6f45a9f7ab state=PENDING_OPEN,
ts=1431535096035, server=myname.mycompany.com,16201,1431535081143} to
{c76be7ccc0d249d9e9f31e6f45a9f7ab state=OPENING, ts=1431535096058,
server=myname.mycompany.com,16201,1431535081143}
2015-05-13 18:38:16,083 INFO  [AM.ZK.Worker-pool2-t2] master.RegionStates:
Transition {c76be7ccc

Re: Ruby shell versions for HBase 2.0

[Michael Segel]

So… 
Silly question… 
Do you really need to worry about backward’s compatibility? 

How many people have customized HBaseShell ? 

What are the common customizations and if you port HBase shell, how much work 
would filter through to the custom code? 


> On May 13, 2015, at 11:22 AM, Sean Busbey  wrote:
> 
> Hi folks!
> 
> If you weren't aware, our current shell relies on Ruby, specifically the
> REPL program IRB from JRuby. When we launched 1.0 we were on JRuby 1.6 with
> defaults, which means we're stuck on Ruby 1.8.
> 
> For those that don't already know, Ruby 1.8 is super old and has been
> walking off into the sunset for a few years now. Most (but not all!) formal
> support systems for running Ruby have EOLed 1.8 and there are numerous
> known issues with it.
> 
> Right now there's an open ticket to get us on JRuby 1.7 so that our shell
> can work on PPC systems[1]. That version of JRuby defaults to Ruby 1.9 but
> can be run in Ruby 1.8 mode. There are some implementation details
> outstanding, but I'm hoping that ticket can work out such that it can land
> in branch-1.
> 
> For HBase 2.0, I'd like us to plan for a little farther out in the future
> than just updating to Ruby 1.9 (though that would be a fine incremental
> step with some non-trivial work attached). The "current" version of Ruby is
> 2.2. Much like the move from 1.8 -> 1.9 it is not backwards compatible.
> 
> JRuby's next major maintenance release line is "version 9000"[2] and it
> will start out *only* supporting Ruby 2.2. Right now JRuby 9000 is in its
> second "preview" release. They still have a few feature complete items to
> address before they hit their first GA release.
> 
> I'd like us to move to Ruby 2.2 via JRuby 9000 for HBase 2.0.  This will
> cause operator pain to folks with advanced scripts based on Ruby 1.8, but
> it should allow us to update versions to avoid e.g. perf, correctness, and
> security issues much more easily over the lifetime of 2.0.
> 
> What do folks think? Would JRuby 9000 need to hit a GA release prior to
> HBase 2.0 getting released for us to adopt it? Or would it only need enough
> of Ruby 2.2 to run our current shell?
> 
> 
> [1]: https://issues.apache.org/jira/browse/HBASE-13338
> [2]: http://www.slideshare.net/CharlesNutter/over-9000-jruby-in-2015
> 
> -- 
> Sean

Ruby shell versions for HBase 2.0

[Sean Busbey]

Hi folks!

If you weren't aware, our current shell relies on Ruby, specifically the
REPL program IRB from JRuby. When we launched 1.0 we were on JRuby 1.6 with
defaults, which means we're stuck on Ruby 1.8.

For those that don't already know, Ruby 1.8 is super old and has been
walking off into the sunset for a few years now. Most (but not all!) formal
support systems for running Ruby have EOLed 1.8 and there are numerous
known issues with it.

Right now there's an open ticket to get us on JRuby 1.7 so that our shell
can work on PPC systems[1]. That version of JRuby defaults to Ruby 1.9 but
can be run in Ruby 1.8 mode. There are some implementation details
outstanding, but I'm hoping that ticket can work out such that it can land
in branch-1.

For HBase 2.0, I'd like us to plan for a little farther out in the future
than just updating to Ruby 1.9 (though that would be a fine incremental
step with some non-trivial work attached). The "current" version of Ruby is
2.2. Much like the move from 1.8 -> 1.9 it is not backwards compatible.

JRuby's next major maintenance release line is "version 9000"[2] and it
will start out *only* supporting Ruby 2.2. Right now JRuby 9000 is in its
second "preview" release. They still have a few feature complete items to
address before they hit their first GA release.

I'd like us to move to Ruby 2.2 via JRuby 9000 for HBase 2.0.  This will
cause operator pain to folks with advanced scripts based on Ruby 1.8, but
it should allow us to update versions to avoid e.g. perf, correctness, and
security issues much more easily over the lifetime of 2.0.

What do folks think? Would JRuby 9000 need to hit a GA release prior to
HBase 2.0 getting released for us to adopt it? Or would it only need enough
of Ruby 2.2 to run our current shell?


[1]: https://issues.apache.org/jira/browse/HBASE-13338
[2]: http://www.slideshare.net/CharlesNutter/over-9000-jruby-in-2015

-- 
Sean

Re: Problems with Phoenix and HBase

[Ted Yu]

Putting dev@ to bcc

The question w.r.t. sqlline should be posted to Phoenix user mailing list.

w.r.t. question on hbase shell, can you give us more information ?
release of hbase you use
was there exception from hbase shell when no tables were returned ?
master log snippet when above happened

On Wed, May 13, 2015 at 3:30 AM, Asfare  wrote:

> Hi!
>
> I'm very new in HBase and newer in Phoenix.
> My mainly problem currently is that:
>
> When I try to lanch sqlline the execution crash and the client can't
> connect
> to the server. Moreover, I think the hbase have another problem, because
> when I try to use the command list in the shell sometimes recognize my
> tables, sometimes not. Can you give me some advices or tips?
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Problems-with-Phoenix-and-HBase-tp4071362.html
> Sent from the HBase Developer mailing list archive at Nabble.com.
>

[jira] [Created] (HBASE-13681) Refactor Scan section in refguide to take account of scanner chunking, heartbeating, prefetch

[stack (JIRA)]

stack created HBASE-13681:
-

 Summary: Refactor Scan section in refguide to take account of 
scanner chunking, heartbeating, prefetch
 Key: HBASE-13681
 URL: https://issues.apache.org/jira/browse/HBASE-13681
 Project: HBase
  Issue Type: Task
  Components: documentation
Affects Versions: 1.1.0
Reporter: stack


Scanners got a revamp courtesy of [~jonathan.lawlor] Our Scan section in perf 
section, http://hbase.apache.org/book.html#perf.reading, doesn't jibe with his 
redo. Fix. His blog post is good source material: 
https://blogs.apache.org/hbase/entry/scan_improvements_in_hbase_1

While at it, include note on HBASE-13071. It has a fat release note to use as 
input.

[~misty] This one for you?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Problems with Phoenix and HBase

[Asfare]

Hi!

I'm very new in HBase and newer in Phoenix. 
My mainly problem currently is that:

When I try to lanch sqlline the execution crash and the client can't connect
to the server. Moreover, I think the hbase have another problem, because
when I try to use the command list in the shell sometimes recognize my
tables, sometimes not. Can you give me some advices or tips?



--
View this message in context: 
http://apache-hbase.679495.n3.nabble.com/Problems-with-Phoenix-and-HBase-tp4071362.html
Sent from the HBase Developer mailing list archive at Nabble.com.

[jira] [Created] (HBASE-13679) Change ColumnTracker and SQM to deal with Cell instead of byte[], int, int

[Anoop Sam John (JIRA)]

Anoop Sam John created HBASE-13679:
--

 Summary: Change ColumnTracker and SQM to deal with Cell instead of 
byte[], int, int
 Key: HBASE-13679
 URL: https://issues.apache.org/jira/browse/HBASE-13679
 Project: HBase
  Issue Type: Sub-task
Reporter: Anoop Sam John
Assignee: Anoop Sam John


We deal with rowArray and qualifierArray in these classes.  Instead keep ref to 
current/previous cells and use Cell based compares/matching.

Again this will increase the number of calls to getXXXOffset/Length. 
HBASE-13448 has to go in before this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)