Re: Ruby shell versions for HBase 2.0
Andrew Purtell wrote: This is because the Accumulo shell is a custom built shell? If so, we had one of those once and replaced it with the IRB based one. We didn't settle on JRuby right away but, at the time, the consensus was we didn't want to be in the business of maintaining yet another REPL when specialist open source communities have already done that. Why has this changed? Has it? Accumulo shell is just a Java program that uses JLine. It has no control flow structures, variables, or anything like that. Why is the Accumulo shell easier to script with? Does it have control flow constructs? Variable assignment? Or is it easier somehow because it does not have those things? Even the system shell has those. Hmm... Are we saying that any control flow or variables/substitution should be done by calling our shell from a bash or sh script? Wouldn't that be super slow given Java's startup time for every invocation of our shell? A bit like trying to script the AWS command line utilities, which is excruciating. This is what I was trying to get at: the lack of typical language constructs is both beneficial and irritating. I think it depends on what you really want to do. Does that mean trying to maintain two shells is worth it? I'm not sure, but I'm also not sure where the happy medium is. If you exec the shell to just run one command at a time, yes. It is slow. You can redirect input or provide a file of commands to run at one time which don't suffer from the JVM startup problem. I can see why a custom shell built with Java would easier to test where the rest of the project is using Surefire/JUnit. That's a developer convenience concern though, not a user convenience one. That's only if you assume HBase users actually understand Ruby though, no? I would think that something which acts like your normal unix shell would be familiar to users w/o the need to understand some other language. On Wed, May 13, 2015 at 10:41 AM, Sean Busbeybus...@cloudera.com wrote: Pros: * It's easier to test and maintain * it's easier to script with * its interactive mode feels mode focused on the task of interacting with a cluster to me (maybe this is just acting like a psql or mysql shell) Cons * adding custom commands requires knowing java -- Sean On May 13, 2015 12:31 PM, Andrew Purtellapurt...@apache.org wrote: Why is the Accumulo shell superior? Is it scriptable? On Wed, May 13, 2015 at 10:28 AM, Sean Busbeybus...@cloudera.com wrote: I would love to rip out the JRuby shell entirely and make something closer to the Accumulo shell, but I expect that will 1) be way more work 2) be even less compatible for those that rely on customizations. I figured given time we could get a preview user shell (rather than power shell via irb) together in 2.0 and aim for default in 3.0. -- Sean On May 13, 2015 12:19 PM, Stackst...@duboce.net wrote: Nice writeup Sean. Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license is still amenable and hopefully jruby 9k will be slimmer than jruby 1.7+. But if we are going to do a significant shell refactor for hbase 2.0, should we consider doing something more radical; e.g. a new shell? If interest, could start up a new thread so don't distract from this one. St.Ack On Wed, May 13, 2015 at 9:22 AM, Sean Busbeybus...@cloudera.com wrote: Hi folks! If you weren't aware, our current shell relies on Ruby, specifically the REPL program IRB from JRuby. When we launched 1.0 we were on JRuby 1.6 with defaults, which means we're stuck on Ruby 1.8. For those that don't already know, Ruby 1.8 is super old and has been walking off into the sunset for a few years now. Most (but not all!) formal support systems for running Ruby have EOLed 1.8 and there are numerous known issues with it. Right now there's an open ticket to get us on JRuby 1.7 so that our shell can work on PPC systems[1]. That version of JRuby defaults to Ruby 1.9 but can be run in Ruby 1.8 mode. There are some implementation details outstanding, but I'm hoping that ticket can work out such that it can land in branch-1. For HBase 2.0, I'd like us to plan for a little farther out in the future than just updating to Ruby 1.9 (though that would be a fine incremental step with some non-trivial work attached). The current version of Ruby is 2.2. Much like the move from 1.8 - 1.9 it is not backwards compatible. JRuby's next major maintenance release line is version 9000[2] and it will start out *only* supporting Ruby 2.2. Right now JRuby 9000 is in its second preview release. They still have a few feature complete items to address before they hit their first GA release. I'd like us to move to Ruby 2.2 via JRuby 9000 for HBase 2.0. This will cause operator pain to folks with advanced scripts based on Ruby 1.8, but it should allow us to update versions to avoid e.g. perf, correctness, and security issues much more easily over the
Re: Ruby shell versions for HBase 2.0
That's only if you assume HBase users actually understand Ruby though, no? Yes, and this is a fair point. The language REPL in which we're embedding the shell DSL doesn't have to be Ruby. JavaScript would work, maybe via Nashorn. Python would work, via Jython. Or Scala's REPL, even. On Wed, May 13, 2015 at 1:10 PM, Josh Elser josh.el...@gmail.com wrote: Andrew Purtell wrote: This is because the Accumulo shell is a custom built shell? If so, we had one of those once and replaced it with the IRB based one. We didn't settle on JRuby right away but, at the time, the consensus was we didn't want to be in the business of maintaining yet another REPL when specialist open source communities have already done that. Why has this changed? Has it? Accumulo shell is just a Java program that uses JLine. It has no control flow structures, variables, or anything like that. Why is the Accumulo shell easier to script with? Does it have control flow constructs? Variable assignment? Or is it easier somehow because it does not have those things? Even the system shell has those. Hmm... Are we saying that any control flow or variables/substitution should be done by calling our shell from a bash or sh script? Wouldn't that be super slow given Java's startup time for every invocation of our shell? A bit like trying to script the AWS command line utilities, which is excruciating. This is what I was trying to get at: the lack of typical language constructs is both beneficial and irritating. I think it depends on what you really want to do. Does that mean trying to maintain two shells is worth it? I'm not sure, but I'm also not sure where the happy medium is. If you exec the shell to just run one command at a time, yes. It is slow. You can redirect input or provide a file of commands to run at one time which don't suffer from the JVM startup problem. I can see why a custom shell built with Java would easier to test where the rest of the project is using Surefire/JUnit. That's a developer convenience concern though, not a user convenience one. That's only if you assume HBase users actually understand Ruby though, no? I would think that something which acts like your normal unix shell would be familiar to users w/o the need to understand some other language. On Wed, May 13, 2015 at 10:41 AM, Sean Busbeybus...@cloudera.com wrote: Pros: * It's easier to test and maintain * it's easier to script with * its interactive mode feels mode focused on the task of interacting with a cluster to me (maybe this is just acting like a psql or mysql shell) Cons * adding custom commands requires knowing java -- Sean On May 13, 2015 12:31 PM, Andrew Purtellapurt...@apache.org wrote: Why is the Accumulo shell superior? Is it scriptable? On Wed, May 13, 2015 at 10:28 AM, Sean Busbeybus...@cloudera.com wrote: I would love to rip out the JRuby shell entirely and make something closer to the Accumulo shell, but I expect that will 1) be way more work 2) be even less compatible for those that rely on customizations. I figured given time we could get a preview user shell (rather than power shell via irb) together in 2.0 and aim for default in 3.0. -- Sean On May 13, 2015 12:19 PM, Stackst...@duboce.net wrote: Nice writeup Sean. Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license is still amenable and hopefully jruby 9k will be slimmer than jruby 1.7+. But if we are going to do a significant shell refactor for hbase 2.0, should we consider doing something more radical; e.g. a new shell? If interest, could start up a new thread so don't distract from this one. St.Ack On Wed, May 13, 2015 at 9:22 AM, Sean Busbeybus...@cloudera.com wrote: Hi folks! If you weren't aware, our current shell relies on Ruby, specifically the REPL program IRB from JRuby. When we launched 1.0 we were on JRuby 1.6 with defaults, which means we're stuck on Ruby 1.8. For those that don't already know, Ruby 1.8 is super old and has been walking off into the sunset for a few years now. Most (but not all!) formal support systems for running Ruby have EOLed 1.8 and there are numerous known issues with it. Right now there's an open ticket to get us on JRuby 1.7 so that our shell can work on PPC systems[1]. That version of JRuby defaults to Ruby 1.9 but can be run in Ruby 1.8 mode. There are some implementation details outstanding, but I'm hoping that ticket can work out such that it can land in branch-1. For HBase 2.0, I'd like us to plan for a little farther out in the future than just updating to Ruby 1.9 (though that would be a fine incremental step with some non-trivial work attached). The current version of Ruby is 2.2. Much like the move from 1.8 - 1.9 it is not backwards compatible. JRuby's next major maintenance release line is version 9000[2] and it will start out *only*
Re: Ruby shell versions for HBase 2.0
Only shell we could swap in w/o annoying users would be SQL (ducks); [...] You would also have to manage expectations: i.e. that the SQL would be extremely basic and that the REPL does not come with an idling 100 node cluster ready to take up any involved queries. If we are talking about adding a SQL shell, it doesn't have to be extremely basic, we can revisit my proposal from the other month about embedding Phoenix. On Wed, May 13, 2015 at 4:50 PM, Stack st...@duboce.net wrote: On Wed, May 13, 2015 at 1:38 PM, Andrew Purtell apurt...@apache.org wrote: That's only if you assume HBase users actually understand Ruby though, no? Yes, and this is a fair point. The language REPL in which we're embedding the shell DSL doesn't have to be Ruby. JavaScript would work, maybe vi a Nashorn. Python would work, via Jython. Or Scala's REPL, even. JRuby had best interpreter and the others considered had extra baggage doing a DSL (For history, see HBASE-487). Having to put tabs in your shell doing loops and no jline at the time made jython a non-runner though I wanted it to win (Has anything changed here?) JS is probably way better than when I looked at it back then? Regardless, whether jython, JS, scala, or custom, IMO, we'd just be pissing of users if we swap one idiosyncratic for another. Only shell we could swap in w/o annoying users would be SQL (ducks); and even here you'd probably have to implement a bunch of operators on the other side of sqllines bang operator (or equivalent) to do the hbase specifics. You would also have to manage expectations: i.e. that the SQL would be extremely basic and that the REPL does not come with an idling 100 node cluster ready to take up any involved queries. St.Ack On Wed, May 13, 2015 at 1:10 PM, Josh Elser josh.el...@gmail.com wrote: Andrew Purtell wrote: This is because the Accumulo shell is a custom built shell? If so, we had one of those once and replaced it with the IRB based one. We didn't settle on JRuby right away but, at the time, the consensus was we didn't want to be in the business of maintaining yet another REPL when specialist open source communities have already done that. Why has this changed? Has it? Accumulo shell is just a Java program that uses JLine. It has no control flow structures, variables, or anything like that. Why is the Accumulo shell easier to script with? Does it have control flow constructs? Variable assignment? Or is it easier somehow because it does not have those things? Even the system shell has those. Hmm... Are we saying that any control flow or variables/substitution should be done by calling our shell from a bash or sh script? Wouldn't that be super slow given Java's startup time for every invocation of our shell? A bit like trying to script the AWS command line utilities, which is excruciating. This is what I was trying to get at: the lack of typical language constructs is both beneficial and irritating. I think it depends on what you really want to do. Does that mean trying to maintain two shells is worth it? I'm not sure, but I'm also not sure where the happy medium is. If you exec the shell to just run one command at a time, yes. It is slow. You can redirect input or provide a file of commands to run at one time which don't suffer from the JVM startup problem. I can see why a custom shell built with Java would easier to test where the rest of the project is using Surefire/JUnit. That's a developer convenience concern though, not a user convenience one. That's only if you assume HBase users actually understand Ruby though, no? I would think that something which acts like your normal unix shell would be familiar to users w/o the need to understand some other language. On Wed, May 13, 2015 at 10:41 AM, Sean Busbeybus...@cloudera.com wrote: Pros: * It's easier to test and maintain * it's easier to script with * its interactive mode feels mode focused on the task of interacting with a cluster to me (maybe this is just acting like a psql or mysql shell) Cons * adding custom commands requires knowing java -- Sean On May 13, 2015 12:31 PM, Andrew Purtellapurt...@apache.org wrote: Why is the Accumulo shell superior? Is it scriptable? On Wed, May 13, 2015 at 10:28 AM, Sean Busbeybus...@cloudera.com wrote: I would love to rip out the JRuby shell entirely and make something closer to the Accumulo shell, but I expect that will 1) be way more work 2) be even less compatible for those that rely on customizations. I figured given time we could get a preview user shell (rather than power shell via irb) together in 2.0 and aim for default in 3.0. -- Sean On May 13, 2015 12:19 PM,
Re: Ruby shell versions for HBase 2.0
If you were going to replace Ruby, Python (Jython) and Scala would make more sense. Then you could go to play (the Scala GUI) for a GUI to HBase reusing the code. On May 13, 2015, at 3:38 PM, Andrew Purtell apurt...@apache.org wrote: That's only if you assume HBase users actually understand Ruby though, no? Yes, and this is a fair point. The language REPL in which we're embedding the shell DSL doesn't have to be Ruby. JavaScript would work, maybe via Nashorn. Python would work, via Jython. Or Scala's REPL, even. On Wed, May 13, 2015 at 1:10 PM, Josh Elser josh.el...@gmail.com wrote: Andrew Purtell wrote: This is because the Accumulo shell is a custom built shell? If so, we had one of those once and replaced it with the IRB based one. We didn't settle on JRuby right away but, at the time, the consensus was we didn't want to be in the business of maintaining yet another REPL when specialist open source communities have already done that. Why has this changed? Has it? Accumulo shell is just a Java program that uses JLine. It has no control flow structures, variables, or anything like that. Why is the Accumulo shell easier to script with? Does it have control flow constructs? Variable assignment? Or is it easier somehow because it does not have those things? Even the system shell has those. Hmm... Are we saying that any control flow or variables/substitution should be done by calling our shell from a bash or sh script? Wouldn't that be super slow given Java's startup time for every invocation of our shell? A bit like trying to script the AWS command line utilities, which is excruciating. This is what I was trying to get at: the lack of typical language constructs is both beneficial and irritating. I think it depends on what you really want to do. Does that mean trying to maintain two shells is worth it? I'm not sure, but I'm also not sure where the happy medium is. If you exec the shell to just run one command at a time, yes. It is slow. You can redirect input or provide a file of commands to run at one time which don't suffer from the JVM startup problem. I can see why a custom shell built with Java would easier to test where the rest of the project is using Surefire/JUnit. That's a developer convenience concern though, not a user convenience one. That's only if you assume HBase users actually understand Ruby though, no? I would think that something which acts like your normal unix shell would be familiar to users w/o the need to understand some other language. On Wed, May 13, 2015 at 10:41 AM, Sean Busbeybus...@cloudera.com wrote: Pros: * It's easier to test and maintain * it's easier to script with * its interactive mode feels mode focused on the task of interacting with a cluster to me (maybe this is just acting like a psql or mysql shell) Cons * adding custom commands requires knowing java -- Sean On May 13, 2015 12:31 PM, Andrew Purtellapurt...@apache.org wrote: Why is the Accumulo shell superior? Is it scriptable? On Wed, May 13, 2015 at 10:28 AM, Sean Busbeybus...@cloudera.com wrote: I would love to rip out the JRuby shell entirely and make something closer to the Accumulo shell, but I expect that will 1) be way more work 2) be even less compatible for those that rely on customizations. I figured given time we could get a preview user shell (rather than power shell via irb) together in 2.0 and aim for default in 3.0. -- Sean On May 13, 2015 12:19 PM, Stackst...@duboce.net wrote: Nice writeup Sean. Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license is still amenable and hopefully jruby 9k will be slimmer than jruby 1.7+. But if we are going to do a significant shell refactor for hbase 2.0, should we consider doing something more radical; e.g. a new shell? If interest, could start up a new thread so don't distract from this one. St.Ack On Wed, May 13, 2015 at 9:22 AM, Sean Busbeybus...@cloudera.com wrote: Hi folks! If you weren't aware, our current shell relies on Ruby, specifically the REPL program IRB from JRuby. When we launched 1.0 we were on JRuby 1.6 with defaults, which means we're stuck on Ruby 1.8. For those that don't already know, Ruby 1.8 is super old and has been walking off into the sunset for a few years now. Most (but not all!) formal support systems for running Ruby have EOLed 1.8 and there are numerous known issues with it. Right now there's an open ticket to get us on JRuby 1.7 so that our shell can work on PPC systems[1]. That version of JRuby defaults to Ruby 1.9 but can be run in Ruby 1.8 mode. There are some implementation details outstanding, but I'm hoping that ticket can work out such that it can land in branch-1. For HBase 2.0, I'd like us to plan for a little farther out in the future than just
Re: Ruby shell versions for HBase 2.0
I'm also in favor of a rewrite in another language (but it doesn't have to be now). This is very subjective but I never got the hang of the Ruby stuff, it's always annoying to use and I end up googling Ruby stuff almost every time I want to do something that's not covered in the help command. Scala would make way more sense to me. If no rewrite I have no problem moving to Ruby 2.2. I know it's at least a few years out but a Java REPL might be coming in Java 9 (http://openjdk.java.net/projects/kulla/) so if we're still discussing by the time we switch to JDK9 that might be an option ;-) On Wed, May 13, 2015 at 10:38 PM, Andrew Purtell apurt...@apache.org wrote: That's only if you assume HBase users actually understand Ruby though, no? Yes, and this is a fair point. The language REPL in which we're embedding the shell DSL doesn't have to be Ruby. JavaScript would work, maybe via Nashorn. Python would work, via Jython. Or Scala's REPL, even. On Wed, May 13, 2015 at 1:10 PM, Josh Elser josh.el...@gmail.com wrote: Andrew Purtell wrote: This is because the Accumulo shell is a custom built shell? If so, we had one of those once and replaced it with the IRB based one. We didn't settle on JRuby right away but, at the time, the consensus was we didn't want to be in the business of maintaining yet another REPL when specialist open source communities have already done that. Why has this changed? Has it? Accumulo shell is just a Java program that uses JLine. It has no control flow structures, variables, or anything like that. Why is the Accumulo shell easier to script with? Does it have control flow constructs? Variable assignment? Or is it easier somehow because it does not have those things? Even the system shell has those. Hmm... Are we saying that any control flow or variables/substitution should be done by calling our shell from a bash or sh script? Wouldn't that be super slow given Java's startup time for every invocation of our shell? A bit like trying to script the AWS command line utilities, which is excruciating. This is what I was trying to get at: the lack of typical language constructs is both beneficial and irritating. I think it depends on what you really want to do. Does that mean trying to maintain two shells is worth it? I'm not sure, but I'm also not sure where the happy medium is. If you exec the shell to just run one command at a time, yes. It is slow. You can redirect input or provide a file of commands to run at one time which don't suffer from the JVM startup problem. I can see why a custom shell built with Java would easier to test where the rest of the project is using Surefire/JUnit. That's a developer convenience concern though, not a user convenience one. That's only if you assume HBase users actually understand Ruby though, no? I would think that something which acts like your normal unix shell would be familiar to users w/o the need to understand some other language. On Wed, May 13, 2015 at 10:41 AM, Sean Busbeybus...@cloudera.com wrote: Pros: * It's easier to test and maintain * it's easier to script with * its interactive mode feels mode focused on the task of interacting with a cluster to me (maybe this is just acting like a psql or mysql shell) Cons * adding custom commands requires knowing java -- Sean On May 13, 2015 12:31 PM, Andrew Purtellapurt...@apache.org wrote: Why is the Accumulo shell superior? Is it scriptable? On Wed, May 13, 2015 at 10:28 AM, Sean Busbeybus...@cloudera.com wrote: I would love to rip out the JRuby shell entirely and make something closer to the Accumulo shell, but I expect that will 1) be way more work 2) be even less compatible for those that rely on customizations. I figured given time we could get a preview user shell (rather than power shell via irb) together in 2.0 and aim for default in 3.0. -- Sean On May 13, 2015 12:19 PM, Stackst...@duboce.net wrote: Nice writeup Sean. Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license is still amenable and hopefully jruby 9k will be slimmer than jruby 1.7+. But if we are going to do a significant shell refactor for hbase 2.0, should we consider doing something more radical; e.g. a new shell? If interest, could start up a new thread so don't distract from this one. St.Ack On Wed, May 13, 2015 at 9:22 AM, Sean Busbeybus...@cloudera.com wrote: Hi folks! If you weren't aware, our current shell relies on Ruby, specifically the REPL program IRB from JRuby. When we launched 1.0 we were on JRuby 1.6 with defaults, which means we're stuck on Ruby 1.8. For those that don't already know, Ruby 1.8 is super old and has been walking off into the sunset for a few years now. Most (but not all!) formal
Re: Ruby shell versions for HBase 2.0
On Wed, May 13, 2015 at 1:38 PM, Andrew Purtell apurt...@apache.org wrote: That's only if you assume HBase users actually understand Ruby though, no? Yes, and this is a fair point. The language REPL in which we're embedding the shell DSL doesn't have to be Ruby. JavaScript would work, maybe via Nashorn. Python would work, via Jython. Or Scala's REPL, even. JRuby had best interpreter and the others considered had extra baggage doing a DSL (For history, see HBASE-487). Having to put tabs in your shell doing loops and no jline at the time made jython a non-runner though I wanted it to win (Has anything changed here?) JS is probably way better than when I looked at it back then? Regardless, whether jython, JS, scala, or custom, IMO, we'd just be pissing of users if we swap one idiosyncratic for another. Only shell we could swap in w/o annoying users would be SQL (ducks); and even here you'd probably have to implement a bunch of operators on the other side of sqllines bang operator (or equivalent) to do the hbase specifics. You would also have to manage expectations: i.e. that the SQL would be extremely basic and that the REPL does not come with an idling 100 node cluster ready to take up any involved queries. St.Ack On Wed, May 13, 2015 at 1:10 PM, Josh Elser josh.el...@gmail.com wrote: Andrew Purtell wrote: This is because the Accumulo shell is a custom built shell? If so, we had one of those once and replaced it with the IRB based one. We didn't settle on JRuby right away but, at the time, the consensus was we didn't want to be in the business of maintaining yet another REPL when specialist open source communities have already done that. Why has this changed? Has it? Accumulo shell is just a Java program that uses JLine. It has no control flow structures, variables, or anything like that. Why is the Accumulo shell easier to script with? Does it have control flow constructs? Variable assignment? Or is it easier somehow because it does not have those things? Even the system shell has those. Hmm... Are we saying that any control flow or variables/substitution should be done by calling our shell from a bash or sh script? Wouldn't that be super slow given Java's startup time for every invocation of our shell? A bit like trying to script the AWS command line utilities, which is excruciating. This is what I was trying to get at: the lack of typical language constructs is both beneficial and irritating. I think it depends on what you really want to do. Does that mean trying to maintain two shells is worth it? I'm not sure, but I'm also not sure where the happy medium is. If you exec the shell to just run one command at a time, yes. It is slow. You can redirect input or provide a file of commands to run at one time which don't suffer from the JVM startup problem. I can see why a custom shell built with Java would easier to test where the rest of the project is using Surefire/JUnit. That's a developer convenience concern though, not a user convenience one. That's only if you assume HBase users actually understand Ruby though, no? I would think that something which acts like your normal unix shell would be familiar to users w/o the need to understand some other language. On Wed, May 13, 2015 at 10:41 AM, Sean Busbeybus...@cloudera.com wrote: Pros: * It's easier to test and maintain * it's easier to script with * its interactive mode feels mode focused on the task of interacting with a cluster to me (maybe this is just acting like a psql or mysql shell) Cons * adding custom commands requires knowing java -- Sean On May 13, 2015 12:31 PM, Andrew Purtellapurt...@apache.org wrote: Why is the Accumulo shell superior? Is it scriptable? On Wed, May 13, 2015 at 10:28 AM, Sean Busbeybus...@cloudera.com wrote: I would love to rip out the JRuby shell entirely and make something closer to the Accumulo shell, but I expect that will 1) be way more work 2) be even less compatible for those that rely on customizations. I figured given time we could get a preview user shell (rather than power shell via irb) together in 2.0 and aim for default in 3.0. -- Sean On May 13, 2015 12:19 PM, Stackst...@duboce.net wrote: Nice writeup Sean. Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license is still amenable and hopefully jruby 9k will be slimmer than jruby 1.7+. But if we are going to do a significant shell refactor for hbase 2.0, should we consider doing something more radical; e.g. a new shell? If interest, could start up a new thread so don't distract from this one. St.Ack On Wed, May 13, 2015 at 9:22 AM, Sean Busbeybus...@cloudera.com wrote: Hi folks! If you weren't aware, our current shell relies on Ruby, specifically the REPL program IRB from
Ruby shell versions for HBase 2.0
Hi folks! If you weren't aware, our current shell relies on Ruby, specifically the REPL program IRB from JRuby. When we launched 1.0 we were on JRuby 1.6 with defaults, which means we're stuck on Ruby 1.8. For those that don't already know, Ruby 1.8 is super old and has been walking off into the sunset for a few years now. Most (but not all!) formal support systems for running Ruby have EOLed 1.8 and there are numerous known issues with it. Right now there's an open ticket to get us on JRuby 1.7 so that our shell can work on PPC systems[1]. That version of JRuby defaults to Ruby 1.9 but can be run in Ruby 1.8 mode. There are some implementation details outstanding, but I'm hoping that ticket can work out such that it can land in branch-1. For HBase 2.0, I'd like us to plan for a little farther out in the future than just updating to Ruby 1.9 (though that would be a fine incremental step with some non-trivial work attached). The current version of Ruby is 2.2. Much like the move from 1.8 - 1.9 it is not backwards compatible. JRuby's next major maintenance release line is version 9000[2] and it will start out *only* supporting Ruby 2.2. Right now JRuby 9000 is in its second preview release. They still have a few feature complete items to address before they hit their first GA release. I'd like us to move to Ruby 2.2 via JRuby 9000 for HBase 2.0. This will cause operator pain to folks with advanced scripts based on Ruby 1.8, but it should allow us to update versions to avoid e.g. perf, correctness, and security issues much more easily over the lifetime of 2.0. What do folks think? Would JRuby 9000 need to hit a GA release prior to HBase 2.0 getting released for us to adopt it? Or would it only need enough of Ruby 2.2 to run our current shell? [1]: https://issues.apache.org/jira/browse/HBASE-13338 [2]: http://www.slideshare.net/CharlesNutter/over-9000-jruby-in-2015 -- Sean
Re: Ruby shell versions for HBase 2.0
So… Silly question… Do you really need to worry about backward’s compatibility? How many people have customized HBaseShell ? What are the common customizations and if you port HBase shell, how much work would filter through to the custom code? On May 13, 2015, at 11:22 AM, Sean Busbey bus...@cloudera.com wrote: Hi folks! If you weren't aware, our current shell relies on Ruby, specifically the REPL program IRB from JRuby. When we launched 1.0 we were on JRuby 1.6 with defaults, which means we're stuck on Ruby 1.8. For those that don't already know, Ruby 1.8 is super old and has been walking off into the sunset for a few years now. Most (but not all!) formal support systems for running Ruby have EOLed 1.8 and there are numerous known issues with it. Right now there's an open ticket to get us on JRuby 1.7 so that our shell can work on PPC systems[1]. That version of JRuby defaults to Ruby 1.9 but can be run in Ruby 1.8 mode. There are some implementation details outstanding, but I'm hoping that ticket can work out such that it can land in branch-1. For HBase 2.0, I'd like us to plan for a little farther out in the future than just updating to Ruby 1.9 (though that would be a fine incremental step with some non-trivial work attached). The current version of Ruby is 2.2. Much like the move from 1.8 - 1.9 it is not backwards compatible. JRuby's next major maintenance release line is version 9000[2] and it will start out *only* supporting Ruby 2.2. Right now JRuby 9000 is in its second preview release. They still have a few feature complete items to address before they hit their first GA release. I'd like us to move to Ruby 2.2 via JRuby 9000 for HBase 2.0. This will cause operator pain to folks with advanced scripts based on Ruby 1.8, but it should allow us to update versions to avoid e.g. perf, correctness, and security issues much more easily over the lifetime of 2.0. What do folks think? Would JRuby 9000 need to hit a GA release prior to HBase 2.0 getting released for us to adopt it? Or would it only need enough of Ruby 2.2 to run our current shell? [1]: https://issues.apache.org/jira/browse/HBASE-13338 [2]: http://www.slideshare.net/CharlesNutter/over-9000-jruby-in-2015 -- Sean
Re: Ruby shell versions for HBase 2.0
Nice writeup Sean. Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license is still amenable and hopefully jruby 9k will be slimmer than jruby 1.7+. But if we are going to do a significant shell refactor for hbase 2.0, should we consider doing something more radical; e.g. a new shell? If interest, could start up a new thread so don't distract from this one. St.Ack On Wed, May 13, 2015 at 9:22 AM, Sean Busbey bus...@cloudera.com wrote: Hi folks! If you weren't aware, our current shell relies on Ruby, specifically the REPL program IRB from JRuby. When we launched 1.0 we were on JRuby 1.6 with defaults, which means we're stuck on Ruby 1.8. For those that don't already know, Ruby 1.8 is super old and has been walking off into the sunset for a few years now. Most (but not all!) formal support systems for running Ruby have EOLed 1.8 and there are numerous known issues with it. Right now there's an open ticket to get us on JRuby 1.7 so that our shell can work on PPC systems[1]. That version of JRuby defaults to Ruby 1.9 but can be run in Ruby 1.8 mode. There are some implementation details outstanding, but I'm hoping that ticket can work out such that it can land in branch-1. For HBase 2.0, I'd like us to plan for a little farther out in the future than just updating to Ruby 1.9 (though that would be a fine incremental step with some non-trivial work attached). The current version of Ruby is 2.2. Much like the move from 1.8 - 1.9 it is not backwards compatible. JRuby's next major maintenance release line is version 9000[2] and it will start out *only* supporting Ruby 2.2. Right now JRuby 9000 is in its second preview release. They still have a few feature complete items to address before they hit their first GA release. I'd like us to move to Ruby 2.2 via JRuby 9000 for HBase 2.0. This will cause operator pain to folks with advanced scripts based on Ruby 1.8, but it should allow us to update versions to avoid e.g. perf, correctness, and security issues much more easily over the lifetime of 2.0. What do folks think? Would JRuby 9000 need to hit a GA release prior to HBase 2.0 getting released for us to adopt it? Or would it only need enough of Ruby 2.2 to run our current shell? [1]: https://issues.apache.org/jira/browse/HBASE-13338 [2]: http://www.slideshare.net/CharlesNutter/over-9000-jruby-in-2015 -- Sean
Re: Ruby shell versions for HBase 2.0
I'd be curious to hear proposals for a new shell. Wondering what the arguments in favor would be and arguments against current. JRuby has served us well. Recently it personally saved me hassle by allowing scripted surgery (advanced ops) rather than dev of a one off Java utility. OTOH, if what we had available for scripting was the Nashorn JS engine (Java 8+) instead, that would have worked well also. And if we're talking about new shells, what about a SQL shell? /me ducks and runs for cover On Wed, May 13, 2015 at 10:19 AM, Stack st...@duboce.net wrote: Nice writeup Sean. Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license is still amenable and hopefully jruby 9k will be slimmer than jruby 1.7+. But if we are going to do a significant shell refactor for hbase 2.0, should we consider doing something more radical; e.g. a new shell? If interest, could start up a new thread so don't distract from this one. St.Ack On Wed, May 13, 2015 at 9:22 AM, Sean Busbey bus...@cloudera.com wrote: Hi folks! If you weren't aware, our current shell relies on Ruby, specifically the REPL program IRB from JRuby. When we launched 1.0 we were on JRuby 1.6 with defaults, which means we're stuck on Ruby 1.8. For those that don't already know, Ruby 1.8 is super old and has been walking off into the sunset for a few years now. Most (but not all!) formal support systems for running Ruby have EOLed 1.8 and there are numerous known issues with it. Right now there's an open ticket to get us on JRuby 1.7 so that our shell can work on PPC systems[1]. That version of JRuby defaults to Ruby 1.9 but can be run in Ruby 1.8 mode. There are some implementation details outstanding, but I'm hoping that ticket can work out such that it can land in branch-1. For HBase 2.0, I'd like us to plan for a little farther out in the future than just updating to Ruby 1.9 (though that would be a fine incremental step with some non-trivial work attached). The current version of Ruby is 2.2. Much like the move from 1.8 - 1.9 it is not backwards compatible. JRuby's next major maintenance release line is version 9000[2] and it will start out *only* supporting Ruby 2.2. Right now JRuby 9000 is in its second preview release. They still have a few feature complete items to address before they hit their first GA release. I'd like us to move to Ruby 2.2 via JRuby 9000 for HBase 2.0. This will cause operator pain to folks with advanced scripts based on Ruby 1.8, but it should allow us to update versions to avoid e.g. perf, correctness, and security issues much more easily over the lifetime of 2.0. What do folks think? Would JRuby 9000 need to hit a GA release prior to HBase 2.0 getting released for us to adopt it? Or would it only need enough of Ruby 2.2 to run our current shell? [1]: https://issues.apache.org/jira/browse/HBASE-13338 [2]: http://www.slideshare.net/CharlesNutter/over-9000-jruby-in-2015 -- Sean -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: Ruby shell versions for HBase 2.0
I would love to rip out the JRuby shell entirely and make something closer to the Accumulo shell, but I expect that will 1) be way more work 2) be even less compatible for those that rely on customizations. I figured given time we could get a preview user shell (rather than power shell via irb) together in 2.0 and aim for default in 3.0. -- Sean On May 13, 2015 12:19 PM, Stack st...@duboce.net wrote: Nice writeup Sean. Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license is still amenable and hopefully jruby 9k will be slimmer than jruby 1.7+. But if we are going to do a significant shell refactor for hbase 2.0, should we consider doing something more radical; e.g. a new shell? If interest, could start up a new thread so don't distract from this one. St.Ack On Wed, May 13, 2015 at 9:22 AM, Sean Busbey bus...@cloudera.com wrote: Hi folks! If you weren't aware, our current shell relies on Ruby, specifically the REPL program IRB from JRuby. When we launched 1.0 we were on JRuby 1.6 with defaults, which means we're stuck on Ruby 1.8. For those that don't already know, Ruby 1.8 is super old and has been walking off into the sunset for a few years now. Most (but not all!) formal support systems for running Ruby have EOLed 1.8 and there are numerous known issues with it. Right now there's an open ticket to get us on JRuby 1.7 so that our shell can work on PPC systems[1]. That version of JRuby defaults to Ruby 1.9 but can be run in Ruby 1.8 mode. There are some implementation details outstanding, but I'm hoping that ticket can work out such that it can land in branch-1. For HBase 2.0, I'd like us to plan for a little farther out in the future than just updating to Ruby 1.9 (though that would be a fine incremental step with some non-trivial work attached). The current version of Ruby is 2.2. Much like the move from 1.8 - 1.9 it is not backwards compatible. JRuby's next major maintenance release line is version 9000[2] and it will start out *only* supporting Ruby 2.2. Right now JRuby 9000 is in its second preview release. They still have a few feature complete items to address before they hit their first GA release. I'd like us to move to Ruby 2.2 via JRuby 9000 for HBase 2.0. This will cause operator pain to folks with advanced scripts based on Ruby 1.8, but it should allow us to update versions to avoid e.g. perf, correctness, and security issues much more easily over the lifetime of 2.0. What do folks think? Would JRuby 9000 need to hit a GA release prior to HBase 2.0 getting released for us to adopt it? Or would it only need enough of Ruby 2.2 to run our current shell? [1]: https://issues.apache.org/jira/browse/HBASE-13338 [2]: http://www.slideshare.net/CharlesNutter/over-9000-jruby-in-2015 -- Sean
Re: Ruby shell versions for HBase 2.0
On May 13, 2015 12:06 PM, Michael Segel michael_se...@hotmail.com wrote: So… Silly question… Do you really need to worry about backward’s compatibility? How many people have customized HBaseShell ? What are the common customizations and if you port HBase shell, how much work would filter through to the custom code? These are excellent questions to which I have poor answers. The mechanics of the interactive repl via IRB will probably not be that different. However, the ref guide calls out the ability to customize things via hooking in your own ruby script so there's little bound on what we've set for expectations. Given ruby 1.8's age, I'm not even sure there are 1.8 to 2.2 guides; likely folks would need to follow 1.8 to 1.9 and 1.9 to 2.2 guides (or we could work to provide some combined guidance). Customizations are just as likely to be call this sequence of commands (little to no work for update) as a daemon that does health and recovery checks (lots of work, approaching rewrite). Personally, I avoid customizations at least in part because it's ruby 1.8. I've also discouraged those who ask me. Instead I rely on the non-interactive flag and bash. But given the ref guide positioning, it's likely we'll break a nontrivial number of folks.
Re: Ruby shell versions for HBase 2.0
sorry to say I don't good at Ruby . HBase Shell may should upgrade :) 2015-05-14 1:06 GMT+08:00 Michael Segel michael_se...@hotmail.com: So… Silly question… Do you really need to worry about backward’s compatibility? How many people have customized HBaseShell ? What are the common customizations and if you port HBase shell, how much work would filter through to the custom code? On May 13, 2015, at 11:22 AM, Sean Busbey bus...@cloudera.com wrote: Hi folks! If you weren't aware, our current shell relies on Ruby, specifically the REPL program IRB from JRuby. When we launched 1.0 we were on JRuby 1.6 with defaults, which means we're stuck on Ruby 1.8. For those that don't already know, Ruby 1.8 is super old and has been walking off into the sunset for a few years now. Most (but not all!) formal support systems for running Ruby have EOLed 1.8 and there are numerous known issues with it. Right now there's an open ticket to get us on JRuby 1.7 so that our shell can work on PPC systems[1]. That version of JRuby defaults to Ruby 1.9 but can be run in Ruby 1.8 mode. There are some implementation details outstanding, but I'm hoping that ticket can work out such that it can land in branch-1. For HBase 2.0, I'd like us to plan for a little farther out in the future than just updating to Ruby 1.9 (though that would be a fine incremental step with some non-trivial work attached). The current version of Ruby is 2.2. Much like the move from 1.8 - 1.9 it is not backwards compatible. JRuby's next major maintenance release line is version 9000[2] and it will start out *only* supporting Ruby 2.2. Right now JRuby 9000 is in its second preview release. They still have a few feature complete items to address before they hit their first GA release. I'd like us to move to Ruby 2.2 via JRuby 9000 for HBase 2.0. This will cause operator pain to folks with advanced scripts based on Ruby 1.8, but it should allow us to update versions to avoid e.g. perf, correctness, and security issues much more easily over the lifetime of 2.0. What do folks think? Would JRuby 9000 need to hit a GA release prior to HBase 2.0 getting released for us to adopt it? Or would it only need enough of Ruby 2.2 to run our current shell? [1]: https://issues.apache.org/jira/browse/HBASE-13338 [2]: http://www.slideshare.net/CharlesNutter/over-9000-jruby-in-2015 -- Sean -- long is the way and hard that out of Hell leads up to light
Re: Ruby shell versions for HBase 2.0
Why is the Accumulo shell superior? Is it scriptable? On Wed, May 13, 2015 at 10:28 AM, Sean Busbey bus...@cloudera.com wrote: I would love to rip out the JRuby shell entirely and make something closer to the Accumulo shell, but I expect that will 1) be way more work 2) be even less compatible for those that rely on customizations. I figured given time we could get a preview user shell (rather than power shell via irb) together in 2.0 and aim for default in 3.0. -- Sean On May 13, 2015 12:19 PM, Stack st...@duboce.net wrote: Nice writeup Sean. Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license is still amenable and hopefully jruby 9k will be slimmer than jruby 1.7+. But if we are going to do a significant shell refactor for hbase 2.0, should we consider doing something more radical; e.g. a new shell? If interest, could start up a new thread so don't distract from this one. St.Ack On Wed, May 13, 2015 at 9:22 AM, Sean Busbey bus...@cloudera.com wrote: Hi folks! If you weren't aware, our current shell relies on Ruby, specifically the REPL program IRB from JRuby. When we launched 1.0 we were on JRuby 1.6 with defaults, which means we're stuck on Ruby 1.8. For those that don't already know, Ruby 1.8 is super old and has been walking off into the sunset for a few years now. Most (but not all!) formal support systems for running Ruby have EOLed 1.8 and there are numerous known issues with it. Right now there's an open ticket to get us on JRuby 1.7 so that our shell can work on PPC systems[1]. That version of JRuby defaults to Ruby 1.9 but can be run in Ruby 1.8 mode. There are some implementation details outstanding, but I'm hoping that ticket can work out such that it can land in branch-1. For HBase 2.0, I'd like us to plan for a little farther out in the future than just updating to Ruby 1.9 (though that would be a fine incremental step with some non-trivial work attached). The current version of Ruby is 2.2. Much like the move from 1.8 - 1.9 it is not backwards compatible. JRuby's next major maintenance release line is version 9000[2] and it will start out *only* supporting Ruby 2.2. Right now JRuby 9000 is in its second preview release. They still have a few feature complete items to address before they hit their first GA release. I'd like us to move to Ruby 2.2 via JRuby 9000 for HBase 2.0. This will cause operator pain to folks with advanced scripts based on Ruby 1.8, but it should allow us to update versions to avoid e.g. perf, correctness, and security issues much more easily over the lifetime of 2.0. What do folks think? Would JRuby 9000 need to hit a GA release prior to HBase 2.0 getting released for us to adopt it? Or would it only need enough of Ruby 2.2 to run our current shell? [1]: https://issues.apache.org/jira/browse/HBASE-13338 [2]: http://www.slideshare.net/CharlesNutter/over-9000-jruby-in-2015 -- Sean -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: Ruby shell versions for HBase 2.0
Pros: * It's easier to test and maintain * it's easier to script with * its interactive mode feels mode focused on the task of interacting with a cluster to me (maybe this is just acting like a psql or mysql shell) Cons * adding custom commands requires knowing java -- Sean On May 13, 2015 12:31 PM, Andrew Purtell apurt...@apache.org wrote: Why is the Accumulo shell superior? Is it scriptable? On Wed, May 13, 2015 at 10:28 AM, Sean Busbey bus...@cloudera.com wrote: I would love to rip out the JRuby shell entirely and make something closer to the Accumulo shell, but I expect that will 1) be way more work 2) be even less compatible for those that rely on customizations. I figured given time we could get a preview user shell (rather than power shell via irb) together in 2.0 and aim for default in 3.0. -- Sean On May 13, 2015 12:19 PM, Stack st...@duboce.net wrote: Nice writeup Sean. Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license is still amenable and hopefully jruby 9k will be slimmer than jruby 1.7+. But if we are going to do a significant shell refactor for hbase 2.0, should we consider doing something more radical; e.g. a new shell? If interest, could start up a new thread so don't distract from this one. St.Ack On Wed, May 13, 2015 at 9:22 AM, Sean Busbey bus...@cloudera.com wrote: Hi folks! If you weren't aware, our current shell relies on Ruby, specifically the REPL program IRB from JRuby. When we launched 1.0 we were on JRuby 1.6 with defaults, which means we're stuck on Ruby 1.8. For those that don't already know, Ruby 1.8 is super old and has been walking off into the sunset for a few years now. Most (but not all!) formal support systems for running Ruby have EOLed 1.8 and there are numerous known issues with it. Right now there's an open ticket to get us on JRuby 1.7 so that our shell can work on PPC systems[1]. That version of JRuby defaults to Ruby 1.9 but can be run in Ruby 1.8 mode. There are some implementation details outstanding, but I'm hoping that ticket can work out such that it can land in branch-1. For HBase 2.0, I'd like us to plan for a little farther out in the future than just updating to Ruby 1.9 (though that would be a fine incremental step with some non-trivial work attached). The current version of Ruby is 2.2. Much like the move from 1.8 - 1.9 it is not backwards compatible. JRuby's next major maintenance release line is version 9000[2] and it will start out *only* supporting Ruby 2.2. Right now JRuby 9000 is in its second preview release. They still have a few feature complete items to address before they hit their first GA release. I'd like us to move to Ruby 2.2 via JRuby 9000 for HBase 2.0. This will cause operator pain to folks with advanced scripts based on Ruby 1.8, but it should allow us to update versions to avoid e.g. perf, correctness, and security issues much more easily over the lifetime of 2.0. What do folks think? Would JRuby 9000 need to hit a GA release prior to HBase 2.0 getting released for us to adopt it? Or would it only need enough of Ruby 2.2 to run our current shell? [1]: https://issues.apache.org/jira/browse/HBASE-13338 [2]: http://www.slideshare.net/CharlesNutter/over-9000-jruby-in-2015 -- Sean -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: Ruby shell versions for HBase 2.0
Agreed on these points, Sean. Having used both, I think both approaches have their value and their drawbacks. The ruby shell is _wonderful_ from having a full programming language to interact with. Accumulo's shell would force you to use your standard unix-toolbelt if you want to do any extra parsing/logic. OTOH, general introspection and interaction with the system feels much more natural to me w/ Accumulo's shell. I know that's not a good way to quantify my feelings. I think it's mostly due to not having to write Ruby when I'm not actually scripting things. Another point is that Accumulo uses JLine under the hood which always comes with its own burden (things like C^c nuked the entire shell for the longest time because JLine didn't support it -- we sometimes have to fix JLine to get the functionality we want). I also caught wind of someone actually using JavaScript via JSR-223 w/ the Accumulo shell, but it's not widely advertised at this point. Finding a the right mix between ease of scripting and simplicity of administrative interactions is key if we want to move beyond what the HBase shell is now. My $0.02. (and, to not completely derail the original conversation, moving to Ruby 2.2 + JRuby 9000 would be what I think the right move to make is) Sean Busbey wrote: Pros: * It's easier to test and maintain * it's easier to script with * its interactive mode feels mode focused on the task of interacting with a cluster to me (maybe this is just acting like a psql or mysql shell) Cons * adding custom commands requires knowing java
Re: Ruby shell versions for HBase 2.0
Having two shells - a simple user shell for tinkering and an advanced one for real ops and developers - would be fine, as long as one is not deprecated. Can't trust something deprecated for tooling. On Wed, May 13, 2015 at 10:52 AM, Esteban Gutierrez este...@cloudera.com wrote: +1 for new JRuby for now and having a JIRA to discuss what features should this shell have would be great. However, I think that during the last few years many of us have used the hbase shell for major surgery as Andrew said and it has added some confusion to users about what the hbase shell is used for and I think that functionality shouldn't be longer exposed to regular users. One way we could probably do this is to have a user facing CLI that will be more for DDL, permissions and data interaction for users (sort of psql or phoenix /me runs too) and deprecate/update the old shell just leaving table/region management functionality for admins, also since most of advanced scripting is done by interacting with Admin directly (see region_mover.rb or copy_tables_desc.rb) I don't think we should worry too much about backwards compatibility if that gets properly documented. thanks, esteban. -- Cloudera, Inc. On Wed, May 13, 2015 at 10:25 AM, Andrew Purtell apurt...@apache.org wrote: I'd be curious to hear proposals for a new shell. Wondering what the arguments in favor would be and arguments against current. JRuby has served us well. Recently it personally saved me hassle by allowing scripted surgery (advanced ops) rather than dev of a one off Java utility. OTOH, if what we had available for scripting was the Nashorn JS engine (Java 8+) instead, that would have worked well also. And if we're talking about new shells, what about a SQL shell? /me ducks and runs for cover On Wed, May 13, 2015 at 10:19 AM, Stack st...@duboce.net wrote: Nice writeup Sean. Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license is still amenable and hopefully jruby 9k will be slimmer than jruby 1.7+. But if we are going to do a significant shell refactor for hbase 2.0, should we consider doing something more radical; e.g. a new shell? If interest, could start up a new thread so don't distract from this one. St.Ack On Wed, May 13, 2015 at 9:22 AM, Sean Busbey bus...@cloudera.com wrote: Hi folks! If you weren't aware, our current shell relies on Ruby, specifically the REPL program IRB from JRuby. When we launched 1.0 we were on JRuby 1.6 with defaults, which means we're stuck on Ruby 1.8. For those that don't already know, Ruby 1.8 is super old and has been walking off into the sunset for a few years now. Most (but not all!) formal support systems for running Ruby have EOLed 1.8 and there are numerous known issues with it. Right now there's an open ticket to get us on JRuby 1.7 so that our shell can work on PPC systems[1]. That version of JRuby defaults to Ruby 1.9 but can be run in Ruby 1.8 mode. There are some implementation details outstanding, but I'm hoping that ticket can work out such that it can land in branch-1. For HBase 2.0, I'd like us to plan for a little farther out in the future than just updating to Ruby 1.9 (though that would be a fine incremental step with some non-trivial work attached). The current version of Ruby is 2.2. Much like the move from 1.8 - 1.9 it is not backwards compatible. JRuby's next major maintenance release line is version 9000[2] and it will start out *only* supporting Ruby 2.2. Right now JRuby 9000 is in its second preview release. They still have a few feature complete items to address before they hit their first GA release. I'd like us to move to Ruby 2.2 via JRuby 9000 for HBase 2.0. This will cause operator pain to folks with advanced scripts based on Ruby 1.8, but it should allow us to update versions to avoid e.g. perf, correctness, and security issues much more easily over the lifetime of 2.0. What do folks think? Would JRuby 9000 need to hit a GA release prior to HBase 2.0 getting released for us to adopt it? Or would it only need enough of Ruby 2.2 to run our current shell? [1]: https://issues.apache.org/jira/browse/HBASE-13338 [2]: http://www.slideshare.net/CharlesNutter/over-9000-jruby-in-2015 -- Sean -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: Ruby shell versions for HBase 2.0
+1 for new JRuby for now and having a JIRA to discuss what features should this shell have would be great. However, I think that during the last few years many of us have used the hbase shell for major surgery as Andrew said and it has added some confusion to users about what the hbase shell is used for and I think that functionality shouldn't be longer exposed to regular users. One way we could probably do this is to have a user facing CLI that will be more for DDL, permissions and data interaction for users (sort of psql or phoenix /me runs too) and deprecate/update the old shell just leaving table/region management functionality for admins, also since most of advanced scripting is done by interacting with Admin directly (see region_mover.rb or copy_tables_desc.rb) I don't think we should worry too much about backwards compatibility if that gets properly documented. thanks, esteban. -- Cloudera, Inc. On Wed, May 13, 2015 at 10:25 AM, Andrew Purtell apurt...@apache.org wrote: I'd be curious to hear proposals for a new shell. Wondering what the arguments in favor would be and arguments against current. JRuby has served us well. Recently it personally saved me hassle by allowing scripted surgery (advanced ops) rather than dev of a one off Java utility. OTOH, if what we had available for scripting was the Nashorn JS engine (Java 8+) instead, that would have worked well also. And if we're talking about new shells, what about a SQL shell? /me ducks and runs for cover On Wed, May 13, 2015 at 10:19 AM, Stack st...@duboce.net wrote: Nice writeup Sean. Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license is still amenable and hopefully jruby 9k will be slimmer than jruby 1.7+. But if we are going to do a significant shell refactor for hbase 2.0, should we consider doing something more radical; e.g. a new shell? If interest, could start up a new thread so don't distract from this one. St.Ack On Wed, May 13, 2015 at 9:22 AM, Sean Busbey bus...@cloudera.com wrote: Hi folks! If you weren't aware, our current shell relies on Ruby, specifically the REPL program IRB from JRuby. When we launched 1.0 we were on JRuby 1.6 with defaults, which means we're stuck on Ruby 1.8. For those that don't already know, Ruby 1.8 is super old and has been walking off into the sunset for a few years now. Most (but not all!) formal support systems for running Ruby have EOLed 1.8 and there are numerous known issues with it. Right now there's an open ticket to get us on JRuby 1.7 so that our shell can work on PPC systems[1]. That version of JRuby defaults to Ruby 1.9 but can be run in Ruby 1.8 mode. There are some implementation details outstanding, but I'm hoping that ticket can work out such that it can land in branch-1. For HBase 2.0, I'd like us to plan for a little farther out in the future than just updating to Ruby 1.9 (though that would be a fine incremental step with some non-trivial work attached). The current version of Ruby is 2.2. Much like the move from 1.8 - 1.9 it is not backwards compatible. JRuby's next major maintenance release line is version 9000[2] and it will start out *only* supporting Ruby 2.2. Right now JRuby 9000 is in its second preview release. They still have a few feature complete items to address before they hit their first GA release. I'd like us to move to Ruby 2.2 via JRuby 9000 for HBase 2.0. This will cause operator pain to folks with advanced scripts based on Ruby 1.8, but it should allow us to update versions to avoid e.g. perf, correctness, and security issues much more easily over the lifetime of 2.0. What do folks think? Would JRuby 9000 need to hit a GA release prior to HBase 2.0 getting released for us to adopt it? Or would it only need enough of Ruby 2.2 to run our current shell? [1]: https://issues.apache.org/jira/browse/HBASE-13338 [2]: http://www.slideshare.net/CharlesNutter/over-9000-jruby-in-2015 -- Sean -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: Ruby shell versions for HBase 2.0
This is because the Accumulo shell is a custom built shell? If so, we had one of those once and replaced it with the IRB based one. We didn't settle on JRuby right away but, at the time, the consensus was we didn't want to be in the business of maintaining yet another REPL when specialist open source communities have already done that. Why has this changed? Has it? Why is the Accumulo shell easier to script with? Does it have control flow constructs? Variable assignment? Or is it easier somehow because it does not have those things? Even the system shell has those. Hmm... Are we saying that any control flow or variables/substitution should be done by calling our shell from a bash or sh script? Wouldn't that be super slow given Java's startup time for every invocation of our shell? A bit like trying to script the AWS command line utilities, which is excruciating. I can see why a custom shell built with Java would easier to test where the rest of the project is using Surefire/JUnit. That's a developer convenience concern though, not a user convenience one. On Wed, May 13, 2015 at 10:41 AM, Sean Busbey bus...@cloudera.com wrote: Pros: * It's easier to test and maintain * it's easier to script with * its interactive mode feels mode focused on the task of interacting with a cluster to me (maybe this is just acting like a psql or mysql shell) Cons * adding custom commands requires knowing java -- Sean On May 13, 2015 12:31 PM, Andrew Purtell apurt...@apache.org wrote: Why is the Accumulo shell superior? Is it scriptable? On Wed, May 13, 2015 at 10:28 AM, Sean Busbey bus...@cloudera.com wrote: I would love to rip out the JRuby shell entirely and make something closer to the Accumulo shell, but I expect that will 1) be way more work 2) be even less compatible for those that rely on customizations. I figured given time we could get a preview user shell (rather than power shell via irb) together in 2.0 and aim for default in 3.0. -- Sean On May 13, 2015 12:19 PM, Stack st...@duboce.net wrote: Nice writeup Sean. Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license is still amenable and hopefully jruby 9k will be slimmer than jruby 1.7+. But if we are going to do a significant shell refactor for hbase 2.0, should we consider doing something more radical; e.g. a new shell? If interest, could start up a new thread so don't distract from this one. St.Ack On Wed, May 13, 2015 at 9:22 AM, Sean Busbey bus...@cloudera.com wrote: Hi folks! If you weren't aware, our current shell relies on Ruby, specifically the REPL program IRB from JRuby. When we launched 1.0 we were on JRuby 1.6 with defaults, which means we're stuck on Ruby 1.8. For those that don't already know, Ruby 1.8 is super old and has been walking off into the sunset for a few years now. Most (but not all!) formal support systems for running Ruby have EOLed 1.8 and there are numerous known issues with it. Right now there's an open ticket to get us on JRuby 1.7 so that our shell can work on PPC systems[1]. That version of JRuby defaults to Ruby 1.9 but can be run in Ruby 1.8 mode. There are some implementation details outstanding, but I'm hoping that ticket can work out such that it can land in branch-1. For HBase 2.0, I'd like us to plan for a little farther out in the future than just updating to Ruby 1.9 (though that would be a fine incremental step with some non-trivial work attached). The current version of Ruby is 2.2. Much like the move from 1.8 - 1.9 it is not backwards compatible. JRuby's next major maintenance release line is version 9000[2] and it will start out *only* supporting Ruby 2.2. Right now JRuby 9000 is in its second preview release. They still have a few feature complete items to address before they hit their first GA release. I'd like us to move to Ruby 2.2 via JRuby 9000 for HBase 2.0. This will cause operator pain to folks with advanced scripts based on Ruby 1.8, but it should allow us to update versions to avoid e.g. perf, correctness, and security issues much more easily over the lifetime of 2.0. What do folks think? Would JRuby 9000 need to hit a GA release prior to HBase 2.0 getting released for us to adopt it? Or would it only need enough of Ruby 2.2 to run our current shell? [1]: https://issues.apache.org/jira/browse/HBASE-13338 [2]: http://www.slideshare.net/CharlesNutter/over-9000-jruby-in-2015 -- Sean -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) -- Best regards, - Andy
Re: Ruby shell versions for HBase 2.0
Hmmm. So if we move to a different tech, the modifications / customizations people have done are going to be useless. If we upgrade to a new version of JRuby, some scripts may not have issues, others may have some , and a third group would have major rewrites. The interesting / downside to JRuby is that within the entire eco system… no other use of JRuby. If you were going to switch… maybe Python? (Jython) On May 13, 2015, at 12:37 PM, Sean Busbey bus...@cloudera.com wrote: On May 13, 2015 12:06 PM, Michael Segel michael_se...@hotmail.com wrote: So… Silly question… Do you really need to worry about backward’s compatibility? How many people have customized HBaseShell ? What are the common customizations and if you port HBase shell, how much work would filter through to the custom code? These are excellent questions to which I have poor answers. The mechanics of the interactive repl via IRB will probably not be that different. However, the ref guide calls out the ability to customize things via hooking in your own ruby script so there's little bound on what we've set for expectations. Given ruby 1.8's age, I'm not even sure there are 1.8 to 2.2 guides; likely folks would need to follow 1.8 to 1.9 and 1.9 to 2.2 guides (or we could work to provide some combined guidance). Customizations are just as likely to be call this sequence of commands (little to no work for update) as a daemon that does health and recovery checks (lots of work, approaching rewrite). Personally, I avoid customizations at least in part because it's ruby 1.8. I've also discouraged those who ask me. Instead I rely on the non-interactive flag and bash. But given the ref guide positioning, it's likely we'll break a nontrivial number of folks.
