I can summarize the bulk of the responses as "it's problematic (for varying reasons)". Thanks everyone for the feedback, I don't plan to take this further.
> I'm wondering if this is something that more naturally lies in the various distributions' hands (CDH, Horton, etc). > [...] > I know a bunch of you already work at the various companies that make > distributions, so the work might eventually fall to you anyway. You probably didn't mean it that way, but affiliation and what employers may or may not have planned commercially can't be a consideration in Apache project decisionmaking. > but I wonder if it sets a bad precedent. What sets Phoenix apart from other popular coprocessors in the future? Should they be included too? I find this frustrating, again, because I don't see where I proposed to include other popular coprocessors in the future. I had a specific proposal involving the binary artifacts of one other Apache project. I don't think the appropriate first reaction to a novel proposal is establishment of an all-encompassing policy. This does a disservice to the proposal at hand, which isn't suggesting any of the future hypotheticals we might imagine. I think we are capable of reasoned case-by-case decisionmaking. My experience is preemptive red tape only benefits the status quo, and status quo, at least as a long term condition, isn't healthy for an open source project. On Thu, Mar 19, 2015 at 12:30 PM, Bryan Beaudreault < bbeaudrea...@hubspot.com> wrote: > As a user I do think this would be pretty convenient, but I wonder if it > sets a bad precedent. What sets Phoenix apart from other popular > coprocessors in the future? Should they be included too? If so it then adds > stress on you guys to come up with guidelines that must be met to be > considered "production ready", and then vet each of the pre-packaged > plugins against that at each release. > > > I'm wondering if this is something that more naturally lies in the various > distributions' hands (CDH, Horton, etc). They already do the work to make > sure X works with Y at a particular version, backporting important > security/performance fixes, etc. It seems reasonable for them to package a > particular version of phoenix that has been tested with a particular > version of the entire stack. I'm not sure what the process would be to > suggest this to them. > > I know a bunch of you already work at the various companies that make > distributions, so the work might eventually fall to you anyway. But it at > least sets a clear separation that more closely maps to how things have > historically seemed to work for other parts of the apache/hadoop stack. > > On Thu, Mar 19, 2015 at 3:08 PM, Andrew Purtell <apurt...@apache.org> > wrote: > > > The idea I am proposing now is the inclusion of Phoenix bits into a > > convenience artifact that results in a SQL access skin for HBase. What we > > have available today for that are the coprocessors, the JDBC driver, and > > sqlline. Whether or not we'd want to include a query server as another > type > > of gateway and connector, like REST or Thrift, could be a future topic of > > discussion, but is out of scope of my proposal, certainly now because it > > doesn't exist yet, and even later when it does. > > > > On Wed, Mar 18, 2015 at 5:15 PM, Nick Dimiduk <ndimi...@gmail.com> > wrote: > > > > > On Wed, Mar 18, 2015 at 4:32 PM, Andrew Purtell <apurt...@apache.org> > > > wrote: > > > > > > > A better analogy would be Pig deciding to ship DataFu UDFs in a > > > > convenience artifact. > > > > > > > > > > A swaying argument. It still seems to me a bit like putting the cart in > > > front of the horse, given the dependency DAG. Does the eventual > inclusion > > > of a Phoenix Query Server (PHOENIX-971) impact your opinion on this? > > > > > > On Wed, Mar 18, 2015 at 2:45 PM, Nick Dimiduk <ndimi...@gmail.com> > > wrote: > > > > > > > > > > > > > > HBase shipping a tgz containing Phoenix sounds backwards to me. It > > > seems > > > > > like Hadoop project shipping a tgz with Hive included as a > > convenience. > > > > > Such convenience packages would be great, but I don't think it's > > HBase > > > > > project's responsibility to do so any more than it would be Hadoop > > > > > project's responsibility in my above example. Upstreamers packaging > > > > > downstreamers sounds backwards to me. > > > > > > > > > > On the other hand, I think it makes a lot of sense for *someone* to > > > ship > > > > a > > > > > "batteries included"Phoenix tgz with Hadoop and HBase. If anyone, > > that > > > > > someone should be Phoenix. I don't think it that unreasonable for > > > Phoenix > > > > > to do this. > > > > > > > > > > Or have a 3rd party packaging, as I mentioned before. I don't know > > what > > > > > BigTop packages, but such a bundle seems reasonable for that > project > > to > > > > > produce, as one of many "batteries included" Apache products. > > > > > > > > > > On Wednesday, March 18, 2015, lars hofhansl <la...@apache.org> > > wrote: > > > > > > > > > > > Again... 'bit late chiming in. > > > > > > > > > > > > I'd be +1 on this. We'd essentially just ship HBase with an extra > > > > shell.I > > > > > > don't think Phoenix can do this, because it does not usually ship > > the > > > > > bits > > > > > > needed to run HBase, but just the bits to add to HBase to have > > > > > > Phoenix.Phoenix should not be in the business to maintain an > HBase > > > > > > distribution. > > > > > > One could argue that we should not be in the business to > maintain a > > > > > > Phoenix distribution. But we wouldn't really. We'd just add the > > > Phoenix > > > > > jar > > > > > > to our binary distro, pre-hook the coprocessors, and add the > > SQLLine, > > > > and > > > > > > psql.On the HBase side I'd suggest we'd add a script to > dev-support > > > to > > > > > > package up the HBase+Phoenix distro, and maybe we add a new > target > > to > > > > the > > > > > > pom to pull a Phoenix version. > > > > > > > > > > > > What exactly we'd ship should be discussed (all the Phoenix > tools, > > or > > > > > just > > > > > > some?) Giving HBase a nice SQL shell seems cool to me, and that's > > > were > > > > > some > > > > > > more of the more hairy issues might be found. > > > > > > > > > > > > -- Lars > > > > > > > > > > > > From: Andrew Purtell <apurt...@apache.org <javascript:;>> > > > > > > To: "dev@hbase.apache.org <javascript:;>" <dev@hbase.apache.org > > > > > > <javascript:;>> > > > > > > Sent: Tuesday, March 17, 2015 12:44 PM > > > > > > Subject: [DISCUSSION] "Convenience binaries" bundling Phoenix > > > > > > > > > > > > Consider if the HBase project starts releasing new "convenience > > > > > binaries", > > > > > > in addition to the existing ones, in which we bundle a > > > > > recent/vetted/stable > > > > > > version of Phoenix, with the site file changes for loading their > > > > > > coprocessors already patched in (to hbase-default.xml) For now > this > > > > would > > > > > > be done for 0.98 only, since that's the only release line > supported > > > by > > > > an > > > > > > actively developed Phoenix version. We could also do this for > 0.94 > > > > > releases > > > > > > with Phoenix 3 if the 0.94 RM wants, but I doubt there would be > any > > > > > demand > > > > > > for this, Phoenix 3 is inactive because that community has all > > moved > > > to > > > > > 4, > > > > > > I'd imagine that carries over here. > > > > > > > > > > > > Advantages: > > > > > > > > > > > > - HBase would ship with a SQL access option. There's the Phoenix > > JDBC > > > > > > driver of course, and we'd also bundle the psql and sqlline exec > > > > wrappers > > > > > > from the Phoenix binary distribution. We'd have both the jruby > > shell > > > > and > > > > > a > > > > > > SQL shell, this is a powerful combination. > > > > > > > > > > > > - HBase ships with a library that assists users in making > efficient > > > > > queries > > > > > > if their data is typed, but this doesn't include the server side > > > > > > optimizations that the Phoenix coprocessors provide, and in that > > case > > > > no > > > > > > hand rolling is necessary. > > > > > > > > > > > > - HBase would ship with secondary indexes. These would not cover > > all > > > > > > possible use cases and requirements, let's stipulate that now and > > > hope > > > > > this > > > > > > doesn't kick off another circular discussion on that front. > > > > > Unquestionably > > > > > > this is a compelling Phoenix feature so some use cases obviously > > can > > > > > > benefit, and if users find the combined distribution useful > enough > > we > > > > > don't > > > > > > have to discuss secondary indexes in HBase core again. > > > > > > > > > > > > - We will have done the necessary integration work for the > combined > > > > > result > > > > > > to be easy to use. Apache software cat herders will appreciate > > this. > > > > > > > > > > > > - It's totally optional, simply ignore the new binary packages if > > you > > > > > don't > > > > > > care. This is not a Grand Unification proposal. > > > > > > > > > > > > Concerns: > > > > > > > > > > > > - More work for the RM. Unquestionably. > > > > > > > > > > > > - Concerns about the quality of the combined convenience > artifact: > > Is > > > > > there > > > > > > an implied warranty? Could we disclaim? Should we disclaim? If > not, > > > how > > > > > > does HBase do QA on this. Related to the above concern about RM > > > > > bandwidth. > > > > > > Maybe Phoenix could help. > > > > > > > > > > > > - Increased coupling between the projects. Frankly, I think this > > > > already > > > > > > there, we just don't see it until we trip over issues that could > > have > > > > > been > > > > > > avoided with more communication between projects. Pushing on > > Phoenix > > > > for > > > > > > bits for a monthly HBase release cadence will surface issues > faster > > > and > > > > > > improve communication between the projects. This benefits Phoenix > > > with > > > > > more > > > > > > QA bandwidth. This benefits HBase because we see Phoenix bringing > > in > > > a > > > > > > significant number of users. > > > > > > > > > > > > - We may want to revisit again normalizing type support in > HBase's > > > > client > > > > > > library and Phoenix, eventually. > > > > > > > > > > > > I could add more items to the advantage or concern lists but > mainly > > > > want > > > > > to > > > > > > float the idea for feedback at this time. > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > -- > > > > > > Best regards, > > > > > > > > > > > > - Andy > > > > > > > > > > > > Problems worthy of attack prove their worth by hitting back. - > Piet > > > > Hein > > > > > > (via Tom White) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Best regards, > > > > > > > > - Andy > > > > > > > > Problems worthy of attack prove their worth by hitting back. - Piet > > Hein > > > > (via Tom White) > > > > > > > > > > > > > > > -- > > Best regards, > > > > - Andy > > > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > > (via Tom White) > > > -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
