RE: [PROPOSAL] Phoenix for Incubation
A bit late to this, but another enthusiastic +1 for Phoenix. Phoenix is an excellent addition to HBase. Data types, loose coupling between your data schemas and applications making schema evolution easier, speedier scans and a good set of aggregation functions - a lot of convenience for a certain class of applications that need to use HBase. And despite the excellent features it already offers, there is much, much more work to be done here and I think making this a part of the Apache umbrella will incentivize more people to contribute. - Original Message - From: general@incubator.apache.org To: general@incubator.apache.org At: Nov 18 2013 04:34:23 +1 (non-binding) for incubation of Phoenix. This is already a very useful project for HBase users, and it will be good to see it driven by a larger community. It may be also be good to get a legal opinion about the name, and/or check for an alternative. There may be some existing software products with this name (e.g. http://www.opwglobal.com/Product.aspx?pid=342). Regards Priyank Rastogi -Original Message- From: James Taylor [mailto:jtay...@salesforce.com] Sent: 14 November 2013 02:14 To: general@incubator.apache.org Subject: [PROPOSAL] Phoenix for Incubation Hi All, We're pleased to share a draft ASF incubation proposal for Phoenix, a SQL layer over HBase, initially developed at Salesforce.com and subsequently open sourced on github (https://github.com/forcedotcom/phoenix). Instead of using Map-reduce to processes queries, it compiles SQL directly into native HBase calls. The complete proposal can be found here: https://wiki.apache.org/incubator/PhoenixProposal, and is also pasted below. Your feedback is greatly appreciated. James == Abstract == Phoenix is an open source SQL query engine for Apache HBase, a NoSQL data store. It is accessed as a JDBC driver and enables querying and managing HBase tables using SQL. == Proposal == Phoenix is an open source SQL skin over HBase delivered as a client-embedded JDBC driver targeting low latency queries over HBase data. Phoenix takes your SQL query, compiles it into a series of HBase scans, and orchestrates the running of those scans to produce regular JDBC result sets. The table metadata is stored in an HBase table and versioned, such that snapshot queries over prior versions will automatically use the correct schema. Direct use of the HBase API, along with coprocessors and custom filters, results in performance on the order of milliseconds for small queries, or seconds for tens of millions of rows. Phoenix interfaces with both Pig and Map-reduce for the input and output of data. == Background == Phoenix initially started as an internal project at Salesforce.com to efficiently analyze big data stored in HBase. It was open sourced on Github about a year ago in Jan 2013. Over time Phoenix, together with HBase as the storage tier, has begun to evolve into a general SQL database with support for metadata management, secondary indexes, joins, query optimization, and multi-tenancy. This is expected to continue as Phoenix implements a cost-based query optimizer and potentially transaction support, and surfaces new HBase security features such as encryption and cell-level security. Phoenix's developer community has also grown to include additional companies such as Intel, who have contributed join support to Phoenix, as well as Hortonworks, who are in the process of porting Phoenix to the 0.96 release of HBase. == Rationale == As usage and the number of contributors to Phoenix has grown, we have sought for a long-term home for the project, and we believe the Apache foundation would be a great fit. Joining Apache would ensure that tried and true processes and procedures are in place for the growing number of organizations interested in contributing to Phoenix. Phoenix is also a good fit for the Apache foundation: Phoenix already interoperates with several existing Apache projects (HBase, Hadoop, Pig). The Phoenix team is familiar with the Apache process and and believes in the Apache mission - the team already includes multiple Apache committers. == Initial Goals == The initial goals will be to move the existing codebase to Apache and integrate with the Apache development process. Once this is accomplished, we plan for incremental development and releases that follow the Apache guidelines. == Current Status == Phoenix has undergone two major and three minor releases (1.0, 1.1, 1.2, 2.0, and 2.1) as well as many patch releases. Phoenix is being used in production by Salesforce.com as well as at other organizations. The Phoenix codebase is currently hosted at github.com, which will form the basis of the Apache git repository. === Meritocracy === The Phoenix project already operates on meritocratic principles. Phoenix has several developers from various organizations outside of Salesforce.com who have contributed major new features. While this pro
RE: [PROPOSAL] Phoenix for Incubation
+1 (non-binding) for incubation of Phoenix. This is already a very useful project for HBase users, and it will be good to see it driven by a larger community. It may be also be good to get a legal opinion about the name, and/or check for an alternative. There may be some existing software products with this name (e.g. http://www.opwglobal.com/Product.aspx?pid=342). Regards Priyank Rastogi -Original Message- From: James Taylor [mailto:jtay...@salesforce.com] Sent: 14 November 2013 02:14 To: general@incubator.apache.org Subject: [PROPOSAL] Phoenix for Incubation Hi All, We're pleased to share a draft ASF incubation proposal for Phoenix, a SQL layer over HBase, initially developed at Salesforce.com and subsequently open sourced on github (https://github.com/forcedotcom/phoenix). Instead of using Map-reduce to processes queries, it compiles SQL directly into native HBase calls. The complete proposal can be found here: https://wiki.apache.org/incubator/PhoenixProposal, and is also pasted below. Your feedback is greatly appreciated. James == Abstract == Phoenix is an open source SQL query engine for Apache HBase, a NoSQL data store. It is accessed as a JDBC driver and enables querying and managing HBase tables using SQL. == Proposal == Phoenix is an open source SQL skin over HBase delivered as a client-embedded JDBC driver targeting low latency queries over HBase data. Phoenix takes your SQL query, compiles it into a series of HBase scans, and orchestrates the running of those scans to produce regular JDBC result sets. The table metadata is stored in an HBase table and versioned, such that snapshot queries over prior versions will automatically use the correct schema. Direct use of the HBase API, along with coprocessors and custom filters, results in performance on the order of milliseconds for small queries, or seconds for tens of millions of rows. Phoenix interfaces with both Pig and Map-reduce for the input and output of data. == Background == Phoenix initially started as an internal project at Salesforce.com to efficiently analyze big data stored in HBase. It was open sourced on Github about a year ago in Jan 2013. Over time Phoenix, together with HBase as the storage tier, has begun to evolve into a general SQL database with support for metadata management, secondary indexes, joins, query optimization, and multi-tenancy. This is expected to continue as Phoenix implements a cost-based query optimizer and potentially transaction support, and surfaces new HBase security features such as encryption and cell-level security. Phoenix's developer community has also grown to include additional companies such as Intel, who have contributed join support to Phoenix, as well as Hortonworks, who are in the process of porting Phoenix to the 0.96 release of HBase. == Rationale == As usage and the number of contributors to Phoenix has grown, we have sought for a long-term home for the project, and we believe the Apache foundation would be a great fit. Joining Apache would ensure that tried and true processes and procedures are in place for the growing number of organizations interested in contributing to Phoenix. Phoenix is also a good fit for the Apache foundation: Phoenix already interoperates with several existing Apache projects (HBase, Hadoop, Pig). The Phoenix team is familiar with the Apache process and and believes in the Apache mission - the team already includes multiple Apache committers. == Initial Goals == The initial goals will be to move the existing codebase to Apache and integrate with the Apache development process. Once this is accomplished, we plan for incremental development and releases that follow the Apache guidelines. == Current Status == Phoenix has undergone two major and three minor releases (1.0, 1.1, 1.2, 2.0, and 2.1) as well as many patch releases. Phoenix is being used in production by Salesforce.com as well as at other organizations. The Phoenix codebase is currently hosted at github.com, which will form the basis of the Apache git repository. === Meritocracy === The Phoenix project already operates on meritocratic principles. Phoenix has several developers from various organizations outside of Salesforce.com who have contributed major new features. While this process has remained mostly informal, as we do not have an official committer list, an implicit organization exists in which individuals who contribute major components act as maintainers for those modules. If accepted, the Phoenix project would include several of these participants as initial committers. We will work to identify all committers and PPMC members for the project and to operate under the ASF meritocratic principles. === Community === Acceptance into the Apache foundation would bolster the already strong user and developer community around Phoenix. That community includes many contributors from various other companies, and an active mailing list c
Re: [PROPOSAL] Phoenix for Incubation
On Sun, Nov 17, 2013 at 5:59 PM, Roman Shaposhnik wrote: >* I am sure with the name like Phoenix there's tons of clashes, > but perhaps Apache Phoenix should be fine We'd be fine with "Apache Phoenix". >* How tightly coupled is it with HBase? IOW, do you anticipate > needing to be constantly in lock-step with HBase implementation > or are you using more of public APIs that shouldn't change that much? We don't anticipate needing to stay in lock-step with an HBase implementation. We try to just surface the excellent lower level features of HBase through the higher level abstraction of SQL. For example, our current release of Phoenix supports 0.94.4 and above (10 releases). However, with the next major release of HBase 0.96, we'll likely need a different version of Phoenix, as it's not backward compatible (though we'll investigate a "shim layer" as 0.94 will likely be around for a while). We have been bitten in the past by changes in the HBase methods, but we've since created Jenkins builds of Phoenix against the head of the HBase 0.94 branch to catch these before they make it into a release. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Phoenix for Incubation
On Mon, Nov 18, 2013 at 3:59 AM, Roman Shaposhnik wrote: > On Wed, Nov 13, 2013 at 12:43 PM, James Taylor > wrote: > > Hi All, > > > > We're pleased to share a draft ASF incubation proposal for Phoenix, a > > SQL layer over HBase, initially developed at Salesforce.com and > > subsequently open sourced on github > > (https://github.com/forcedotcom/phoenix). Instead of using Map-reduce > > to processes queries, it compiles SQL directly into native HBase > > calls. The complete proposal can be found here: > > https://wiki.apache.org/incubator/PhoenixProposal, and is also pasted > > below. > > > > Your feedback is greatly appreciated. > > I think this is a great project and it would make all the sense to have > it join ASF family via the process of incubation. Def! Glad to see you've already got some community forming from Intel, Twitter, and Hortonworks. -- Best Regards, -- Alex
Re: [PROPOSAL] Phoenix for Incubation
On Wed, Nov 13, 2013 at 12:43 PM, James Taylor wrote: > Hi All, > > We're pleased to share a draft ASF incubation proposal for Phoenix, a > SQL layer over HBase, initially developed at Salesforce.com and > subsequently open sourced on github > (https://github.com/forcedotcom/phoenix). Instead of using Map-reduce > to processes queries, it compiles SQL directly into native HBase > calls. The complete proposal can be found here: > https://wiki.apache.org/incubator/PhoenixProposal, and is also pasted > below. > > Your feedback is greatly appreciated. I think this is a great project and it would make all the sense to have it join ASF family via the process of incubation. Two small points are: * I am sure with the name like Phoenix there's tons of clashes, but perhaps Apache Phoenix should be fine * How tightly coupled is it with HBase? IOW, do you anticipate needing to be constantly in lock-step with HBase implementation or are you using more of public APIs that shouldn't change that much? Thanks, Roman. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Phoenix for Incubation
+1. We are currently running phoenix with production traffic, and enjoying the benefits. Joris - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Phoenix for Incubation
+1 for Phoenix as well. SQL access for HBase is a repeated thread in the community and while we probably aren't at the point where there is a single answer for this - and may never be - it would be nice to have a few "preferred options", so to speak, with robust communities around them. Also, per Andy's point about SQL not being a non-goal of HBase proper, I agree, hence another project makes sense. I see both good capabilities and a growing community around Phoenix. Doug Meil On 11/14/13 12:41 PM, "Andrew Purtell" wrote: >On Wed, Nov 13, 2013 at 10:50 PM, Henry Saputra >wrote: > >> It is indeed very specific for HBase use I suppose. Would it be more >> beneficial to make it sub-project of HBase to get full community >> support from HBase? > > >I'm on the HBase PMC and am enthusiastically +1 for incubation of Phoenix >to become a TLP. > >Phoenix can be divided into a front end and a back end. > >The front end is delivered as a JDBC driver and contains, among other >things, the SQL parser and query planner. The front end is currently >written for the HBase client API but could be extended to support other >data stores in the Apache family. > >The back end is, currently, HBase specific components for pushing as much >work to the server as possible. However, if there were sufficient interest >to build them, contributions to Phoenix of new back ends for other data >stores in the Apache family would be feasible. > >Most importantly, as James mentioned in the proposal, much of the Phoenix >project's attention will focus on the query planning part. Supporting SQL >or any declarative query interface is to the best of my knowledge a >non-goal of the HBase community. > >-- >Best regards, > > - Andy > >Problems worthy of attack prove their worth by hitting back. - Piet Hein >(via Tom White) - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Phoenix for Incubation
On Thu, Nov 14, 2013 at 9:41 AM, Andrew Purtell wrote: > > On Wed, Nov 13, 2013 at 10:50 PM, Henry Saputra > wrote: > > > It is indeed very specific for HBase use I suppose. Would it be more > > beneficial to make it sub-project of HBase to get full community > > support from HBase? > > The back end is, currently, HBase specific components for pushing as much > work to the server as possible. However, if there were sufficient interest > to build them, contributions to Phoenix of new back ends for other data > stores in the Apache family would be feasible. Good points, Andrew, and an omission on my part from the proposal. We actually had someone internally do something similar for a POC. I'll update the proposal with this information. Regards, James > > > -- > Best regards, > >- Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via Tom White) - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Phoenix for Incubation
Patrick Reilly php.net> writes: > > Phoenix is a wonderful addition as sub-project of HBase and I use it > everyday in production. > > +1 from me for sure. > > — Patrick > Sorry, I meant a top level project not a sub-project of HBase. I apologize for the confusion. — Patrick - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Phoenix for Incubation
Phoenix is a wonderful addition as sub-project of HBase and I use it everyday in production. +1 from me for sure. — Patrick - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Phoenix for Incubation
On Wed, Nov 13, 2013 at 10:50 PM, Henry Saputra wrote: > It is indeed very specific for HBase use I suppose. Would it be more > beneficial to make it sub-project of HBase to get full community > support from HBase? I'm on the HBase PMC and am enthusiastically +1 for incubation of Phoenix to become a TLP. Phoenix can be divided into a front end and a back end. The front end is delivered as a JDBC driver and contains, among other things, the SQL parser and query planner. The front end is currently written for the HBase client API but could be extended to support other data stores in the Apache family. The back end is, currently, HBase specific components for pushing as much work to the server as possible. However, if there were sufficient interest to build them, contributions to Phoenix of new back ends for other data stores in the Apache family would be feasible. Most importantly, as James mentioned in the proposal, much of the Phoenix project's attention will focus on the query planning part. Supporting SQL or any declarative query interface is to the best of my knowledge a non-goal of the HBase community. -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: [PROPOSAL] Phoenix for Incubation
On Wed, Nov 13, 2013 at 9:43 PM, James Taylor wrote: > Hi All, > > We're pleased to share a draft ASF incubation proposal for Phoenix, a > SQL layer over HBase, initially developed at Salesforce.com and > subsequently open sourced on github > (https://github.com/forcedotcom/phoenix). Instead of using Map-reduce > to processes queries, it compiles SQL directly into native HBase > calls. The complete proposal can be found here: > https://wiki.apache.org/incubator/PhoenixProposal, and is also pasted > below. > > Your feedback is greatly appreciated. > Enthusiastic (and rusty but binding) +1 We're looking forward to at least start using it, as well as hopefully start contributing to Phoenix in the next couple of months. Steven.
Re: [PROPOSAL] Phoenix for Incubation
Hi All, I am sorry for the confusion. I didn't mean to say that Phoenix should be a sub-project of Apache HBase. My +1 for Phoenix as Top Level Project. At Intuit, we have been extensively using Phoenix. Thanks, Anil Gupta Software Engineer, Intuit Inc On Wed, Nov 13, 2013 at 10:31 PM, Anil Gupta wrote: > Phoenix is a great addition as sub-project of HBase. > +1 from me. > > Thanks, > Anil Gupta > > > > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > > -- Thanks & Regards, Anil Gupta
Re: [PROPOSAL] Phoenix for Incubation
It is indeed very specific for HBase use I suppose. Would it be more beneficial to make it sub-project of HBase to get full community support from HBase? On Wed, Nov 13, 2013 at 12:43 PM, James Taylor wrote: > Hi All, > > We're pleased to share a draft ASF incubation proposal for Phoenix, a > SQL layer over HBase, initially developed at Salesforce.com and > subsequently open sourced on github > (https://github.com/forcedotcom/phoenix). Instead of using Map-reduce > to processes queries, it compiles SQL directly into native HBase > calls. The complete proposal can be found here: > https://wiki.apache.org/incubator/PhoenixProposal, and is also pasted > below. > > Your feedback is greatly appreciated. > > James > > == Abstract == > Phoenix is an open source SQL query engine for Apache HBase, a NoSQL > data store. It is accessed as a JDBC driver and enables querying and > managing HBase tables using SQL. > > == Proposal == > Phoenix is an open source SQL skin over HBase delivered as a > client-embedded JDBC driver targeting low latency queries over HBase > data. Phoenix takes your SQL query, compiles it into a series of HBase > scans, and orchestrates the running of those scans to produce regular > JDBC result sets. The table metadata is stored in an HBase table and > versioned, such that snapshot queries over prior versions will > automatically use the correct schema. Direct use of the HBase API, > along with coprocessors and custom filters, results in performance on > the order of milliseconds for small queries, or seconds for tens of > millions of rows. Phoenix interfaces with both Pig and Map-reduce for > the input and output of data. > > == Background == > Phoenix initially started as an internal project at Salesforce.com to > efficiently analyze big data stored in HBase. It was open sourced on > Github about a year ago in Jan 2013. Over time Phoenix, together with > HBase as the storage tier, has begun to evolve into a general SQL > database with support for metadata management, secondary indexes, > joins, query optimization, and multi-tenancy. This is expected to > continue as Phoenix implements a cost-based query optimizer and > potentially transaction support, and surfaces new HBase security > features such as encryption and cell-level security. Phoenix's > developer community has also grown to include additional companies > such as Intel, who have contributed join support to Phoenix, as well > as Hortonworks, who are in the process of porting Phoenix to the 0.96 > release of HBase. > > == Rationale == > As usage and the number of contributors to Phoenix has grown, we have > sought for a long-term home for the project, and we believe the Apache > foundation would be a great fit. Joining Apache would ensure that > tried and true processes and procedures are in place for the growing > number of organizations interested in contributing to Phoenix. Phoenix > is also a good fit for the Apache foundation: Phoenix already > interoperates with several existing Apache projects (HBase, Hadoop, > Pig). The Phoenix team is familiar with the Apache process and and > believes in the Apache mission - the team already includes multiple > Apache committers. > > == Initial Goals == > The initial goals will be to move the existing codebase to Apache and > integrate with the Apache development process. Once this is > accomplished, we plan for incremental development and releases that > follow the Apache guidelines. > > == Current Status == > Phoenix has undergone two major and three minor releases (1.0, 1.1, > 1.2, 2.0, and 2.1) as well as many patch releases. Phoenix is being > used in production by Salesforce.com as well as at other > organizations. The Phoenix codebase is currently hosted at github.com, > which will form the basis of the Apache git repository. > > === Meritocracy === > The Phoenix project already operates on meritocratic principles. > Phoenix has several developers from various organizations outside of > Salesforce.com who have contributed major new features. While this > process has remained mostly informal, as we do not have an official > committer list, an implicit organization exists in which individuals > who contribute major components act as maintainers for those modules. > If accepted, the Phoenix project would include several of these > participants as initial committers. We will work to identify all > committers and PPMC members for the project and to operate under the > ASF meritocratic principles. > > === Community === > Acceptance into the Apache foundation would bolster the already strong > user and developer community around Phoenix. That community includes > many contributors from various other companies, and an active mailing > list composed of hundreds of users. > > === Core Developers === > The core developers of our project are listed in our contributors and > initial PPMC below. Though many are employed at Salesforce.com, there > is a representative cross sampling of other orga
Re: [PROPOSAL] Phoenix for Incubation
Phoenix is a great addition as sub-project of HBase. +1 from me. Thanks, Anil Gupta - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
