+1 > On Aug 31, 2015, at 11:49, Hitesh Shah <hit...@apache.org> wrote: > > +1 > > — Hitesh > >> On Aug 31, 2015, at 11:47 AM, Roman Shaposhnik <r...@apache.org> wrote: >> >> Following the discussion earlier: >> http://s.apache.org/Gaf >> >> I would like to call a VOTE for accepting HAWQ >> as a new incubator project. >> >> The proposal is available at: >> https://wiki.apache.org/incubator/HAWQProposal >> and is also included at the bottom of this email. >> >> Vote is open until at least Thu, 3 September 2015, 23:59:00 PST >> >> [ ] +1 accept HAWQ into the Apache Incubator >> [ ] ±0 >> [ ] -1 because... >> >> Thanks, >> Roman. >> >> == Abstract == >> >> HAWQ is an advanced enterprise SQL on Hadoop analytic engine built >> around a robust and high-performance massively-parallel processing >> (MPP) SQL framework evolved from Pivotal Greenplum DatabaseⓇ. >> >> HAWQ runs natively on Apache HadoopⓇ clusters by tightly integrating >> with HDFS and YARN. HAWQ supports multiple Hadoop file formats such as >> Apache Parquet, native HDFS, and Apache Avro. HAWQ is configured and >> managed as a Hadoop service in Apache Ambari. HAWQ is 100% ANSI SQL >> compliant (supporting ANSI SQL-92, SQL-99, and SQL-2003, plus OLAP >> extensions) and supports open database connectivity (ODBC) and Java >> database connectivity (JDBC), as well. Most business intelligence, >> data analysis and data visualization tools work with HAWQ out of the >> box without the need for specialized drivers. >> >> A unique aspect of HAWQ is its integration of statistical and machine >> learning capabilities that can be natively invoked from SQL or (in the >> context of PL/Python, PL/Java or PL/R) in massively parallel modes and >> applied to large data sets across a Hadoop cluster. These capabilities >> are provided through MADlib – an existing open source, parallel >> machine-learning library. Given the close ties between the two >> development communities, the MADlib community has expressed interest >> in joining HAWQ on its journey into the ASF Incubator and will be >> submitting a separate, concurrent proposal. >> >> HAWQ will provide more robust and higher performing options for Hadoop >> environments that demand best-in-class data analytics for business >> critical purposes. HAWQ is implemented in C and C++. >> >> HAWQ has a few runtime dependencies licensed under the Cat X list: >> * gperf (GPL Version 3) >> * libgsasl (LGPL Version 2.1) >> * libuuid-2.26 (LGPL Version 2) >> However, given the runtime (dynamic linking) nature of these >> dependencies it doesn't represent a problem for HAWQ to be considered >> an ASF project. >> >> == Proposal == >> The goal of this proposal is to bring the core of Pivotal Software, >> Inc.’s (Pivotal) Pivotal HAWQⓇ codebase into the Apache Software >> Foundation (ASF) in order to build a vibrant, diverse and >> self-governed open source community around the technology. Pivotal has >> agreed to transfer the brand name "HAWQ" to Apache Software Foundation >> and will stop using HAWQ to refer to this software if the project gets >> accepted into the ASF Incubator under the name of "Apache HAWQ >> (incubating)". Pivotal will continue to market and sell an analytic >> engine product that includes Apache HAWQ (incubating). While HAWQ is >> our primary choice for a name of the project, in anticipation of any >> potential issues with PODLINGNAMESEARCH we have come up with two >> alternative names: (1) Hornet; or (2) Grove. >> >> Pivotal is submitting this proposal to donate the HAWQ source code and >> associated artifacts (documentation, web site content, wiki, etc.) to >> the Apache Software Foundation Incubator under the Apache License, >> Version 2.0 and is asking Incubator PMC to establish an open source >> community. >> >> == Background == >> While the ecosystem of open source SQL-on-Hadoop solutions is fairly >> developed by now, HAWQ has several unique features that will set it >> apart from existing ASF and non-ASF projects. HAWQ made its debut in >> 2013 as a closed source product leveraging a decade's worth of product >> development effort invested in Greenplum DatabaseⓇ. Since then HAWQ >> has rapidly gained a solid customer base and became available on >> non-Pivotal distributions of Hadoop. >> In 2015 HAWQ still leverages the rock solid foundation of Greenplum >> Database, while at the same time embracing elasticity and resource >> management native to Hadoop applications. This allows HAWQ to provide >> superior SQL on Hadoop performance, scalability and coverage while >> also providing massively-parallel machine learning capabilities and >> support for native Hadoop file formats. In addition, HAWQ's advanced >> features include support for complex joins, rich and compliant SQL >> dialect and industry-differentiating data federation capabilities. >> Dynamic pipelining and pluggable query optimizer architecture enable >> HAWQ to perform queries on Hadoop with the speed and scalability >> required for enterprise data warehouse (EDW) workloads. HAWQ provides >> strong support for low-latency analytic SQL queries, coupled with >> massively parallel machine learning capabilities. This enables >> discovery-based analysis of large data sets and rapid, iterative >> development of data analytics applications that apply deep machine >> learning – significantly shortening data-driven innovation cycles for >> the enterprise. >> >> Hundreds of companies and thousands of servers are running >> mission-critical applications today on HAWQ managing over PBs of data. >> >> == Rationale == >> Hadoop and HDFS-based data management architectures continue their >> expansion into the enterprise. As the amount of data stored on Hadoop >> clusters grows, unlocking the analytics capabilities and democratizing >> access to that treasure trove of data becomes one of the key concerns. >> While Hadoop has no shortage of purposefully designed analytical >> frameworks, the easiest and most cost-effective way to onboard the >> largest amount of data consumers is provided by offering SQL APIs for >> data retrieval at scale. Of course, given the high velocity of >> innovation happening in the underlying Hadoop ecosystem, any >> SQL-on-Hadoop solution has to keep up with the community. We strongly >> believe that in the Big Data space, this can be optimally achieved >> through a vibrant, diverse, self-governed community collectively >> innovating around a single codebase while at the same time >> cross-pollinating with various other data management communities. >> Apache Software Foundation is the ideal place to meet those ambitious >> goals. We also believe that our initial experience of bringing Pivotal >> GemfireⓇ into ASF as Apache Geode (incubating) could be leveraged thus >> improving the chances of HAWQ becoming a vibrant Apache community. >> >> == Initial Goals == >> Our initial goals are to bring HAWQ into the ASF, transition internal >> engineering processes into the open, and foster a collaborative >> development model according to the "Apache Way." Pivotal and its >> partners plan to develop new functionality in an open, >> community-driven way. To get there, the existing internal build, test >> and release processes will be refactored to support open development. >> >> == Current Status == >> Currently, the project code base is commercially licensed and is not >> available to the general public. The documentation and wiki pages are >> available at FIXME. Although Pivotal HAWQ was developed as a >> proprietary, closed-source product, its roots are in the PostgreSQL >> community and the internal engineering practices adopted by the >> development team lend themselves well to an open, collaborative and >> meritocratic environment. >> >> The Pivotal HAWQ team has always focused on building a robust end user >> community of paying and non-paying customers. The existing >> documentation along with StackOverflow and other similar forums are >> expected to facilitate conversions between our existing users so as to >> transform them into an active community of HAWQ members, stakeholders >> and developers. >> >> === Meritocracy === >> Our proposed list of initial committers include the current HAWQ R&D >> team, Pivotal Field Engineers, and several existing partners. This >> group will form a base for the broader community we will invite to >> collaborate on the codebase. We intend to radically expand the initial >> developer and user community by running the project in accordance with >> the "Apache Way". Users and new contributors will be treated with >> respect and welcomed. By participating in the community and providing >> quality patches/support that move the project forward, contributors >> will earn merit. They also will be encouraged to provide non-code >> contributions (documentation, events, community management, etc.) and >> will gain merit for doing so. Those with a proven support and quality >> track record will be encouraged to become committers. >> >> === Community === >> If HAWQ is accepted for incubation, the primary initial goal will be >> transitioning the core community towards embracing the Apache Way of >> project governance. We would solicit major existing contributors to >> become committers on the project from the start. >> >> === Core Developers === >> >> A few of HAWQ's core developers are skilled in working as part of >> openly governed Apache communities (mainly around Hadoop ecosystem). >> That said, most of the core developers are currently NOT affiliated >> with the ASF and would require new ICLAs before committing to the >> project. >> >> === Alignment === >> The following existing ASF projects can be considered when reviewing >> HAWQ proposal: >> >> Apache Hadoop is a distributed storage and processing framework for >> very large datasets, focusing primarily on batch processing for >> analytic purposes. HAWQ builds on top of two key pieces of Hadoop: >> YARN and HDFS. HAWQ's community roadmap includes plans for >> contributing Hadoop around HDFS features and increasing support for C >> and C++ clients. >> >> Apache Spark™ is a fast engine for processing large datasets, >> typically from a Hadoop cluster, and performing batch, streaming, >> interactive, or machine learning workloads. Recently, Apache Spark >> has embraced SQL-like APIs around DataFrames at its core. Because of >> that we would expect a level of collaboration between the two projects >> when it comes to query optimization and exposing HAWQ tables to Spark >> analytical pipelines. >> >> Apache Hive™ is a data warehouse software that facilitates querying >> and managing large datasets residing in distributed storage. Hive >> provides a mechanism to project structure onto this data and query the >> data using a SQL-like language called HiveQL. Hive is also providing >> HCatalog capabilities as table and storage management layer for >> Hadoop, enabling users with different data processing tools to more >> easily define structure for the data on the grid. Currently the core >> Hive and HAWQ are viewed as complimentary solutions, but we expect >> close integration with HCatalog given its dominant position for >> metadata management on the Hadoop clusters. >> >> Apache Phoenix is a high performance relational database layer over >> HBase for low latency applications. Given Phoenix's exclusive focus on >> HBase for its data management backend and its overall architecture >> around HBase's co-processors, it is unlikely that there will be much >> collaboration between the two projects. >> >> == Known Risks == >> Development has been sponsored mostly by a single company (or its >> predecessors) thus far and coordinated mainly by the core Pivotal HAWQ >> team. >> >> For the project to fully transition to the Apache Way governance >> model, development must shift towards the meritocracy-centric model of >> growing a community of contributors balanced with the needs for >> extreme stability and core implementation coherency. >> >> The tools and development practices in place for the Pivotal HAWQ >> product are compatible with the ASF infrastructure and thus we do not >> anticipate any on-boarding pains. >> >> The project currently includes a modified version of PostgreSQL 8.3 >> source code. Given the ASF's position that the PostgreSQL License is >> compatible with the Apache License version 2.0, we do NOT anticipate >> any issues with licensing the code base. However, any new capabilities >> developed by the HAWQ team once part of the ASF would need to be >> consumed by the PostgreSQL community under the Apache License version >> 2.0. >> >> === Orphaned products === >> Pivotal is fully committed to maintaining its position as one of the >> leading providers of SQL-on-Hadoop solutions and the corresponding >> Pivotal commercial product will continue to be based on the HAWQ >> project. Moreover, Pivotal has a vested interest in making HAWQ >> successful by driving its close integration with both existing >> projects contributed by Pivotal including Apache Geode (incubating) >> and MADlib (which is requesting Incubation), and sister ASF projects. >> We expect this to further reduces the risk of orphaning the product. >> >> === Inexperience with Open Source === >> Pivotal has embraced open source software since its formation by >> employing contributors/committers and by shepherding open source >> projects like Cloud Foundry, Spring, RabbitMQ and MADlib. Individuals >> working at Pivotal have experience with the formation of vibrant >> communities around open technologies with the Cloud Foundry >> Foundation, and continuing with the creation of a community around >> Apache Geode (incubating). Although some of the initial committers >> have not had the experience of developing entirely open source, >> community-driven projects, we expect to bring to bear the open >> development practices that have proven successful on longstanding >> Pivotal open source projects to the HAWQ community. Additionally, >> several ASF veterans have agreed to mentor the project and are listed >> in this proposal. The project will rely on their collective guidance >> and wisdom to quickly transition the entire team of initial committers >> towards practicing the Apache Way. >> >> === Homogeneous Developers === >> While most of the initial committers are employed by Pivotal, we have >> already seen a healthy level of interest from existing customers and >> partners. We intend to convert that interest directly into >> participation and will be investing in activities to recruit >> additional committers from other companies. >> >> === Reliance on Salaried Developers === >> Most of the contributors are paid to work in the Big Data space. While >> they might wander from their current employers, they are unlikely to >> venture far from their core expertise and thus will continue to be >> engaged with the project regardless of their current employers. >> >> === Relationships with Other Apache Products === >> As mentioned in the Alignment section, HAWQ may consider various >> degrees of integration and code exchange with Apache Hadoop, Apache >> Spark and Apache Hive projects. We expect integration points to be >> inside and outside the project. We look forward to collaborating with >> these communities as well as other communities under the Apache >> umbrella. >> >> === An Excessive Fascination with the Apache Brand === >> While we intend to leverage the Apache ‘branding’ when talking to >> other projects as testament of our project’s ‘neutrality’, we have no >> plans for making use of Apache brand in press releases nor posting >> billboards advertising acceptance of HAWQ into Apache Incubator. >> >> == Documentation == >> The documentation is currently available at http://hawq.docs.pivotal.io/ >> >> == Initial Source == >> Initial source code will be available immediately after Incubator PMC >> approves HAWQ joining the Incubator and will be licensed under the >> Apache License v2. >> >> == Source and Intellectual Property Submission Plan == >> As soon as HAWQ is approved to join the Incubator, the source code >> will be transitioned via an exhibit to Pivotal's current Software >> Grant Agreement onto ASF infrastructure and in turn made available >> under the Apache License, version 2.0. We know of no legal >> encumberments that would inhibit the transfer of source code to the >> ASF. >> >> == External Dependencies == >> >> Runtime dependencies: >> * gimli (BSD) >> * openldap (The OpenLDAP Public License) >> * openssl (OpenSSL License and the Original SSLeay License, BSD style) >> * proj (MIT) >> * yaml (Creative Commons Attribution 2.0 License) >> * python (Python Software Foundation License Version 2) >> * apr-util (Apache Version 2.0) >> * bzip2 (BSD-style License) >> * curl (MIT/X Derivate License) >> * gperf (GPL Version 3) >> * protobuf (Google) >> * libevent (BSD) >> * json-c (https://github.com/json-c/json-c/blob/master/COPYING) >> * krb5 (MIT) >> * pcre (BSD) >> * libedit (BSD) >> * libxml2 (MIT) >> * zlib (Permissive Free Software License) >> * libgsasl (LGPL Version 2.1) >> * thrift (Apache Version 2.0) >> * snappy (Apache Version 2.0 (up to 1.0.1)/New BSD) >> * libuuid-2.26 (LGPL Version 2) >> * apache hadoop (Apache Version 2.0) >> * apache avro (Apache Version 2.0) >> * glog (BSD) >> * googlemock (BSD) >> >> Build only dependencies: >> * ant (Apache Version 2.0) >> * maven (Apache Version 2.0) >> * cmake (BSD) >> >> Test only dependencies: >> * googletest (BSD) >> >> Cryptography N/A >> >> == Required Resources == >> >> === Mailing lists === >> * priv...@hawq.incubator.apache.org (moderated subscriptions) >> * comm...@hawq.incubator.apache.org >> * d...@hawq.incubator.apache.org >> * iss...@hawq.incubator.apache.org >> * u...@hawq.incubator.apache.org >> >> === Git Repository === >> https://git-wip-us.apache.org/repos/asf/incubator-hawq.git >> >> === Issue Tracking === >> JIRA Project HAWQ (HAWQ) >> >> === Other Resources === >> >> Means of setting up regular builds for HAWQ on builds.apache.org will >> require integration with Docker support. >> >> == Initial Committers == >> * Lirong Jian >> * Hubert Huan Zhang >> * Radar Da Lei >> * Ivan Yanqing Weng >> * Zhanwei Wang >> * Yi Jin >> * Lili Ma >> * Jiali Yao >> * Zhenglin Tao >> * Ruilong Huo >> * Ming Li >> * Wen Lin >> * Lei Chang >> * Alexander V Denissov >> * Newton Alex >> * Oleksandr Diachenko >> * Jun Aoki >> * Bhuvnesh Chaudhary >> * Vineet Goel >> * Shivram Mani >> * Noa Horn >> * Sujeet S Varakhedi >> * Junwei (Jimmy) Da >> * Ting (Goden) Yao >> * Mohammad F (Foyzur) Rahman >> * Entong Shen >> * George C Caragea >> * Amr El-Helw >> * Mohamed F Soliman >> * Venkatesh (Venky) Raghavan >> * Carlos Garcia >> * Zixi (Jesse) Zhang >> * Michael P Schubert >> * C.J. Jameson >> * Jacob Frank >> * Ben Calegari >> * Shoabe Shariff >> * Rob Day-Reynolds >> * Mel S Kiyama >> * Charles Alan Litzell >> * David Yozie >> * Ed Espino >> * Caleb Welton >> * Parham Parvizi >> * Dan Baskette >> * Christian Tzolov >> * Tushar Pednekar >> * Greg Chase >> * Chloe Jackson >> * Michael Nixon >> * Roman Shaposhnik >> * Alan Gates >> * Owen O'Malley >> * Thejas Nair >> * Don Bosco Durai >> * Konstantin Boudnik >> * Sergey Soldatov >> * Atri Sharma >> >> == Affiliations == >> * Barclays: Atri Sharma >> * Bloomberg: Justin Erenkrantz >> * Hortonworks: Alan Gates, Owen O'Malley, Thejas Nair, Don Bosco Durai >> * WANDisco: Konstantin Boudnik, Sergey Soldatov >> * Pivotal: everyone else on this proposal >> >> == Sponsors == >> >> === Champion === >> Roman Shaposhnik >> >> === Nominated Mentors === >> >> The initial mentors are listed below: >> * Alan Gates - Apache Member, Hortonworks >> * Owen O'Malley - Apache Member, Hortonworks >> * Thejas Nair - Apache Member, Hortonworks >> * Konstantin Boudnik - Apache Member, WANDisco >> * Roman Shaposhnik - Apache Member, Pivotal >> * Justin Erenkrantz - Apache Member, Bloomberg >> >> === Sponsoring Entity === >> We would like to propose Apache incubator to sponsor this project. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >> For additional commands, e-mail: general-h...@incubator.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org >
--------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
