On Sun, May 10, 2015 at 7:13 PM, Konstantin Boudnik <c...@apache.org> wrote:
> I think it'd be great to have SQL platform for Hadoop > > +1 > > I am mentoring 4 projects at the moment, but if you need a 1/2 time mentor > - > count me in ;) > > Cos > > We'll take you up on your kind offer if we can't get someone less loaded. Thanks Cos, St.Ack > On Fri, May 08, 2015 at 02:59PM, Stack wrote: > > I would like to start up a discussion on Trafodion joining the ASF as an > > incubating project. > > > > Trafodion is a webscale SQL-on-Hadoop solution that enables transactional > > or operational workloads on Hadoop, . > > > > The proposal is available on the wiki here: > > https://wiki.apache.org/incubator/TrafodionProposal#preview > > > > The proposal text is also attached to the end of this email. > > > > Trafodion is a rich, storied SQL engine that has recently been ported to > > run on HBase and Hadoop. I think it would make for a fine addition to the > > Apache family of projects It would be good to hear what others think. > > > > Thank you in advance for giving the proposal a read. > > > > Yours, > > St.Ack > > > > > > Trafodion Apache Incubator Proposal > > > > Abstract > > > > Trafodion is a webscale SQL-on-Hadoop solution enabling transactional or > > operational workloads on Hadoop. > > > > Proposal > > > > Apache Trafodion builds on the scalability, elasticity, and flexibility > of > > Hadoop. Trafodion extends Hadoop to provide guaranteed transactional > > integrity, enabling new kinds of big data applications to run on Hadoop. > Key > > features of Apache Trafodion include: > > > > * Full-functioned ANSI SQL language support > > * JDBC/ODBC connectivity for Linux/Windows clients > > * Distributed ACID transaction protection across multiple statements, > > tables and rows > > * Performance improvements for OLTP workloads with compile-time and > > run-time optimizations > > * Support for large data sets using a parallel-aware query optimizer > > * ANSI SQL security and data integrity constraints including referential > > integrity > > > > Hewlett-Packard Company submits this proposal to donate its Apache > License, > > Version 2.0 open source project known as Trafodion, its source code, > > documentation, and web site content to the Apache Software Foundation in > > order to build an open source community > > > > Background > > > > Trafodion is an open source project sponsored by HP, incubated at HP Labs > > and HP-IT, to develop an enterprise-class SQL-on-Hadoop solution > targeting > > big data transactional or operational workloads. HP publically announced > > the open source project and uploaded the source code to GitHub in June > 2014. > > > > The SQL compiler, optimizer and executor components of Trafodion have a > > rich heritage. Under development since 1993, they were released as > > commercial closed source software in various flavors such as HP NonStop > > SQL/MX and HP Neoview. NonStop SQL/MX was designed for online transaction > > processing on HP’s NonStop (formerly Tandem) fault-tolerant servers and > is > > known for its high availability, scalability, and performance. Hundreds > of > > companies and thousands of servers are running mission-critical > > applications today on NonStop SQL/MX. In addition, much of these > components > > today are running internal to HP as the core of its Enterprise Data > > Warehouse (EDW), managing over a PB of data. > > > > Starting in 2013, the software was modified to run on HBase and a new > > distributed transaction manager was written to run as an HBase > co-processor. > > > > Unlike most NOSQL and other SQL-on-Hadoop open source projects, Trafodion > > provides comprehensive ANSI SQL language support including > full-functioned > > data definition (DDL), data manipulation (DML), transaction control (TCL) > > and database utility support. > > > > Trafodion provides comprehensive and standard SQL data manipulation > support > > including SELECT, INSERT, UPDATE, DELETE, and UPSERT/MERGE syntax with > > language options including join variants, unions, where predicates, > > aggregations (group by and having), sort ordering, sampling, correlated > and > > nested sub-queries, cursors, and many SQL functions. > > > > Utilities are provided for updating table statistics used by the > optimizer > > for costing (i.e. selectivity/cardinality estimates) plan alternatives, > for > > displaying the chosen SQL execution plan, plan shaping, backup and > > restoring the database, data loading and unloading, and a command line > > utility for interfacing with the database engine. > > > > Explicit control statements are provided to allow applications to define > > transaction boundaries and to abort transactions when warranted, > including > > BEGIN WORK, COMMIT WORK, ROLLBACK WORK and SET TRANSACTION. > > > > Trafodion supports ANSI’s grant/revoke semantics to define user and role > > privileges in terms of managing and accessing the database objects. > > > > Rationale > > > > The name “Trafodion” (the Welsh word for transactions, pronounced > > “Tra-vod-eee-on”) was chosen specifically to emphasize the > differentiation > > that Trafodion provides in closing a critical gap in the Hadoop > ecosystem. > > Trafodion builds on the scalability, elasticity, and flexibility of > Hadoop. > > Trafodion extends Hadoop to provide guaranteed transactional integrity, > > enabling new kinds of big data applications to run on Hadoop. > > > > Current Status > > > > HP released the Trafodion code under the Apache License, Version 2, in > June > > of 2014. Since that time, we have had one major release in January 2015 > and > > one minor release in April 2015. The focus of these releases has been in > > getting our base functionality, including security, working on top of > > Apache HBase, as well as improving performance, availability and > > scalability, and integrating better with HBase. > > > > Meritocracy > > > > We want to build a diverse developer community, based on the Apache Way, > > around Trafodion. To help developers become contributors, we have > > documentation on the wiki about the architecture, the source tree > > structure, and an example enhancement. We plan to publish our project > > backlog to the community, specifically highlighting areas where > developers > > new to Trafodion may best start contributing, such as extending the > > database functionality with User Defined Routines (UDRs) and integrating > > with other Apache projects in the Hadoop ecosystem. > > > > Community > > > > We have already begun building a community but at this time the community > > consists only of Trafodion developers – all HP employees – and > prospective > > users. We have participated in and hosted HBase Meetups and intend to > ramp > > up our community building efforts. > > > > The Trafodion project has seen interest in China, where HP has conducted > > proof-of-concepts with multiple companies and expects to see some of its > > first commercial deployments. To help recruit contributors and users in > > China, members of the team are translating Trafodion wiki content into > > Mandarin. > > > > Core Developers > > > > The core developers are very experienced in database and transaction > > monitor technology, with many having spent more than 20 years working in > > this space. > > > > Alignment > > > > Apache Trafodion relies on Apache HBase as its storage engine. The > > development team has collaborated with and gained valuable advice from > > working with the Apache HBase core developers. Apache Trafodion has > > federation capabilities as well, and can query Trafodion tables stored in > > HBase, native HBase tables, and Apache Hive tables. > > > > Known Risks > > > > Orphaned Products > > > > HP Labs and HP-IT have been incubating Trafodion development for almost > two > > years. This is part of HP’s strategy to leverage its investment in > database > > software and bring software to market as open source and is similar to > HP’s > > efforts with OpenStack. Trafodion builds on HP’s equity investment in the > > Hadoop ecosystem and its efforts to monetize Hadoop through hardware, > > software, and services. HP wants Trafodion to be successful, as HP will > > offer a commercially supported distribution of Trafodion. > > > > Inexperience with Open Source > > > > We have been working with open source software in building closed source > > software for well over two decades. To help transition to doing open > source > > development, the development team received guidance and best practices > from > > HP developers working on OpenStack open source projects, many of whom > have > > experience working on Apache and other open source projects as well. > Since > > releasing Trafodion as an open source project in June of 2014, the > > committers and contributors have moved forward using open source > > development processes and tools for bug tracking and design blueprints > and > > Jenkins for continuous integration. As part of the incubation process, we > > recognize we may need to change some of our development processes/tools > and > > conduct our discussions using Apache email dlists. > > > > Homogenous Developers > > > > Since the initial development of Trafodion has been supported by HP, all > of > > the current developers are HP employees. Through the support of the > Apache > > incubation project, we aim to expand the list of developers and gain > > contributors from related SQL-on-Hadoop projects and the Apache HBase > > project. Trafodion developers are experienced with distributed > development > > processes, being primarily based in Palo Alto, CA; Austin, TX; and > > Shanghai, China. Trafodion is written in C++ and Java. > > > > Reliance on Salaried Developers > > > > Currently all of the developers working on the project are paid by their > > employer to work on the project. These developers will work on the open > > source project as well as work on the commercially supported distribution > > of Trafodion that HP will offer. > > > > Relationship with Other Apache Products > > > > Trafodion is built upon Apache HBase and extends it to support ACID > > transactions with HBase co-processors for distributed transaction > > management and recovery. Trafodion envisions future collaborations with > the > > Apache HBase project on performance optimizations, such as in the areas > of > > mixed workload support, High Availability, etc. It also provides > > transactional support and querying from native HBase tables as well. > > > > Trafodion uses Apache Zookeeper to coordinate and manage the distribution > > of connection services across the cluster for load-balancing and high > > availability reconnection purposes in the event a Trafodion process > should > > fail. > > > > Trafodion also envisions working with the Apache Ambari project on > enabling > > better Trafodion manageability. While Ambari focuses on system and > > component level performance metrics, Trafodion manageability will focus > in > > a complimentary way on database workload monitoring and performance > > analytics with capabilities more geared towards database administrators. > > > > There are alternative open source projects that are providing > SQL-on-Hadoop > > capabilities, such as Apache Hive, Apache Drill, and Apache Phoenix. > These > > are more focused on reporting and analytics across data structures > > supported on HDFS. In comparison to all of these technologies Trafodion > > provides a very complete implementation of ANSI SQL, one of the most > > sophisticated optimizers for such workloads, a completely parallel data > > flow architecture that does not materialize intermediate results unless > > necessary, full ACID transactional support, ANSI GRANT/REVOKE security, > and > > other capabilities that would take decades to build in these products. On > > the other hand currently Trafodion is just focused on HBase and querying > > Hive, whereas Hive and Drill provide access to other data formats in > HDFS. > > > > An Excessive Fascination with the Apache Brand > > > > We understand the reputation and value of the Apache brand, and no doubt > > believe that it will help us attract contributors and users. Our primary > > goal is to follow a proven, open source development and community > building > > model that will make Trafodion successful and enable better collaboration > > with other Apache projects in the Hadoop ecosystem. We also understand > the > > rules and guidelines about the use of the Apache brand and intend to > follow > > them. > > > > Documentation > > > > Documentation and technical details on Trafodion can be found at: > > http://www.trafodion.org/ > > > > Initial Source > > > > The source is available today in a public github repository: > > https://github.com/trafodion/trafodion. > > > > Source and Intellectual Property Submission Plan > > > > The source code has already been released under the Apache License, > Version > > 2. The manuals have been released in Adobe PDF format. As part of the > > submission process, the source for the manuals will be converted from a > > proprietary DocBook XML format to AsciiDoc. > > > > External Dependencies > > > > Two dependencies do not have Apache compatible licenses and will be > > addressed as we enter incubation. One dependency is log4cpp, which is > > licensed under the LGPL. A compatible alternative might be Apache > incubator > > project log4cxx. The other dependency is unixodbc, which is used as the > > ODBC driver manager. We will look into how Apache Hive manages being able > > to use this incompatible software and do similar. All other dependencies > > have Apache compatible licenses, including Apache 2.0, MIT/X11, MIT, and > > BSD. > > > > Cryptography > > > > Trafodion does not contain any cryptographic code. It does call > > cryptographic libraries: OpenSSL for C++ code and Java Cryptography > > Extension (JCE) for Java code. > > > > Required Resources > > > > Mailing Lists > > > > priv...@trafodion.incubator.apache.org > > d...@trafodion.incubator.apache.org > comm...@trafodion.incubator.apache.org > > > > Git Repository > > > > https://git-wip-us.apache.org/repos/afs/incubator-trafodion.git > > > > Issue Tracking > > > > JIRA: JIRA Trafodion (Trafodion) > > > > > > Initial Committers and Affiliation > > > > Dave Birdsall, Hewlett-Packard Company, Dave.Birdsall<AT>hp<DOT>com > > Matt Brown, Hewlett-Packard Company, mattbrown<AT>hp<DOT>com > > Tharak Capirala, Hewlett-Packard Company, Tharak.Capirala<AT>hp<DOT>com > > Alice Chen, Hewlett-Packard Company, Alice.Chen<AT>hp<DOT>com > > John DeRoo, Hewlett-Packard Company, John.Deroo<AT>hp<DOT>com > > Roberta Marton, Hewlett-Packard Company, Roberta.Marton<AT>hp<DOT>com > > Amanda Moran, Hewlett-Packard Company, Amanda.Kay.Moran<AT>hp<DOT>com > > Suresh Subbiah, Hewlett-Packard Company, Suresh.Subbiah<AT>hp<DOT>com > > Sandyha Sundaresan, Hewlett-Packard Company, > > Sandhya.Sundaresan<AT>hp<DOT>com > > > > Sponsors > > > > Champion > > > > Michael Stack, Stack<AT>apache<DOT>org > > > > Nominated Mentors > > > > Michael Stack, Stack<AT>apache<DOT>org > > Roman Shaposhnik, rshaposhnik<AT>pivotal<DOT>io > > > > We are seeking additional mentors. > > > > Sponsoring Entity > > > > Apache Incubator PMC > > --------------------------------------------------------------------- > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > >
