The VOTE has passed with 5 binding +1's and no -1s. I'll start the work to get the podling started.
Thank you. On Sun, Mar 10, 2013 at 3:48 AM, Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov> wrote: > +1 from me (binding). > > Good luck! > > Cheers, > Chris > > > On 3/6/13 9:04 AM, "Leonidas Fegaras" <fega...@cse.uta.edu> wrote: > >>Dear ASF members, >>I would like to call for a VOTE for acceptance of MRQL into the >>Incubator. >>The vote will close on Monday March 11, 2013. >> >>[ ] +1 Accept MRQL into the Apache incubator >>[ ] +0 Don't care. >>[ ] -1 Don't accept MRQL into the incubator because... >> >>Full proposal is pasted below and the corresponding wiki is >> >>http://wiki.apache.org/incubator/MRQLProposal >> >>Only VOTEs from Incubator PMC members are binding, >>but all are welcome to express their thoughts. >>Sincerely, >>Leonidas Fegaras >> >> >>= Abstract = >> >>MRQL is a query processing and optimization system for large-scale, >>distributed data analysis, built on top of Apache Hadoop and Hama. >> >>= Proposal = >> >>MRQL (pronounced ''miracle'') is a query processing and optimization >>system for large-scale, distributed data analysis. MRQL (the MapReduce >>Query Language) is an SQL-like query language for large-scale data >>analysis on a cluster of computers. The MRQL query processing system >>can evaluate MRQL queries in two modes: in MapReduce mode on top of >>Apache Hadoop or in Bulk Synchronous Parallel (BSP) mode on top of >>Apache Hama. The MRQL query language is powerful enough to express >>most common data analysis tasks over many forms of raw ''in-situ'' >>data, such as XML and JSON documents, binary files, and CSV >>documents. MRQL is more powerful than other current high-level >>MapReduce languages, such as Hive and PigLatin, since it can operate >>on more complex data and supports more powerful query constructs, thus >>eliminating the need for using explicit MapReduce code. With MRQL, >>users will be able to express complex data analysis tasks, such as >>PageRank, k-means clustering, matrix factorization, etc, using >>SQL-like queries exclusively, while the MRQL query processing system >>will be able to compile these queries to efficient Java code. >> >>= Background = >> >>The initial code was developed at the University of Texas of Arlington >>(UTA) by a research team, led by Leonidas Fegaras. The software was >>first released in May 2011. The original goal of this project was to >>build a query processing system that translates SQL-like data analysis >>queries to efficient workflows of MapReduce jobs. A design goal was to >>use HDFS as the physical storage layer, without any indexing, data >>partitioning, or data normalization, and to use Hadoop (without >>extensions) as the run-time engine. The motivation behind this work >>was to build a platform to test new ideas on query processing and >>optimization techniques applicable to the MapReduce framework. >> >>A year ago, MRQL was extended to run on Hama. The motivation for this >>extension was that Hadoop MapReduce jobs were required to read their >>input and write their output on HDFS. This simplifies reliability and >>fault tolerance but it imposes a high overhead to complex MapReduce >>workflows and graph algorithms, such as PageRank, which require >>repetitive jobs. In addition, Hadoop does not preserve data in memory >>across consecutive MapReduce jobs. This restriction requires to read >>data at every step, even when the data is constant. BSP, on the other >>hand, does not suffer from this restriction, and, under certain >>circumstances, allows complex repetitive algorithms to run entirely in >>the collective memory of a cluster. Thus, the goal was to be able to >>run the same MRQL queries in both modes, MapReduce and BSP, without >>modifying the queries: If there are enough resources available, and >>low latency and speed are more important than resilience, queries may >>run in BSP mode; otherwise, the same queries may run in MapReduce >>mode. BSP evaluation was found to be a good choice when fault >>tolerance is not critical, data (both input and intermediate) can fit >>in the cluster memory, and data processing requires complex/repetitive >>steps. >> >>The research results of this ongoing work have already been published >>in conferences (WebDB'11, EDBT'12, and DataCloud'12) and the authors >>have already received positive feedback from researchers in academia >>and industry who were attending these conferences. >> >>= Rationale = >> >>* MRQL will be the first general-purpose, SQL-like query language for >>data analysis based on BSP. >>Currently, many programmers prefer to code their MapReduce >>applications in a higher-level query language, rather than an >>algorithmic language. For instance, Pig is used for 60% of Yahoo >>MapReduce jobs, while Hive is used for 90% of Facebook MapReduce >>jobs. This, we believe, will also be the trend for BSP applications, >>because, even though, in principle, the BSP model is very simple to >>understand, it is hard to develop, optimize, and maintain non-trivial >>BSP applications coded in a general-purpose programming >>language. Currently, there is no widely acceptable declarative BSP >>query language, although there are a few special-purpose BSP systems >>for graph analysis, such as Google Pregel and Apache Giraph, for >>machine learning, such as BSML, and for scientific data analysis. >> >>* MRQL can capture many complex data analysis algorithms in >>declarative form. >>Existing MapReduce query languages, such as HiveQL and PigLatin, >>provide a limited syntax for operating on data collections, in the >>form of relational joins and group-bys. Because of these limitations, >>these languages enable users to plug-in custom MapReduce scripts into >>their queries for those jobs that cannot be declaratively coded in >>their query language. This nullifies the benefits of using a >>declarative query language and may result to suboptimal, error-prone, >>and hard-to-maintain code. More importantly, these languages are >>inappropriate for complex scientific applications and graph analysis, >>because they do not directly support iteration or recursion in >>declarative form and are not able to handle complex, nested scientific >>data, which are often semi-structured. Furthermore, current MapReduce >>query processors apply traditional query optimization techniques that >>may be suboptimal in a MapReduce or BSP environment. >> >>* The MRQL design is modular, with pluggable distributed processing >>back-ends, query languages, and data formats. >>MRQL aims to be both powerful and adaptable. Although Hadoop is >>currently the most popular framework for large-scale data analysis, >>there are a few alternatives that are currently shaping form, >>including frameworks based on BSP (eg, Giraph, Pregel, Hama), MPI >>(eg, OpenMPI), etc. MRQL was designed in such a way so that it will >>be easy to support other distributed processing frameworks in the >>future. As an evidence of this claim, the MRQL processor required >>only 2K extra lines of Java code to support BSP evaluation. >> >>= Initial Goals = >> >>Some current goals include: >> >>* apply MRQL to graph analysis problems, such as k-means clustering >>and PageRank >> >>* apply MRQL to large-scale scientific analysis (develop general >>optimization techniques that can apply to matrix multiplication, >>matrix factorization, etc) >> >>* process additional data formats, such as Avro, and column-based >>stores, such as HBase >> >>* map MRQL to additional distributed processing frameworks, such as >>Spark and OpenMPI >> >>* extend the front-end to process more query languages, such as >>standard SQL, SPARQL, XQuery, and PigLatin >> >>= Current Status = >> >>The current MRQL release (version 0.8.10) is a beta release. It is >>built on top of Hadoop and Hama (no extensions are needed). It >>currently works on Hadoop up to 1.0.4 (but not on Yarn yet) and Hama >>0.5.0. It has only been tested on a small cluster of 20 nodes (80 >>cores). >> >>== Meritocracy == >> >>The initial MRQL code base was developed by Leonidas Fegaras in May >>2011, and was continuously improved throughout the years. We will >>reach out other potential contributors through open forums. We plan >>to do everything possible to encourage an environment that supports a >>meritocracy, where contributors will extend their privileges based on >>their contribution. MRQL's modular design will facilitate the >>strategic extensions to various modules, such as adding a standard-SQL >>interface, introducing new optimization techniques, etc. >> >>== Community == >> >>The interest in open-source query processing systems for analyzing >>large datasets has been steadily increased in the last few years. >>Related Apache projects have already attracted a very large community >>from both academia and industry. We expect that MRQL will also >>establish an active community. Several researchers from both academia >>and industry who are interested in using our code have already >>contacted us. >> >>== Core Developers == >> >>The initial core developer was Leonidas Fegaras, who wrote the >>majority of the code. He is an associate professor at UTA, with >>interests in cloud computing, databases, web technologies, and >>functional programming. He has an extensive knowledge and working >>experience in building complex query processing systems for databases, >>and compilers for functional and algorithmic programming languages. >> >>== Alignment == >> >>MRQL is built on top of two Apache projects: Hadoop and Hama. We have >>plans to incorporate other products from the Hadoop ecosystem, such as >>Avro and HBase. MRQL can serve as a testbed for fine-tuning and >>evaluating the performance of the Apache Hama system. Finally, the >>MRQL query language and processor can be used by Apache Drill as a >>pluggable query language. >> >>= Known Risks = >> >>== Orphaned Products == >> >>The initial committer is from academia, which may be a risk, since >>research in academia is publication-driven, rather than >>product-driven. It happens very often in academic research, when a >>project becomes outdated and doesn't produce publishable results, to >>be abandoned in favor of new cutting-edge projects. We do not believe >>that this will be the case for MRQL for the years to come, because it >>can be adapted to support new query languages, new optimization >>techniques, and new distributed back-ends, thus sustaining enough >>research interest. Another risk is that, when graduate students who >>write code graduate, they may leave their work undocumented and >>unfinished. We will strive to gain enough momentum to recruit >>additional committers from industry in order to eliminate these risks. >> >>== Inexperience with Open Source == >> >>The initial developer has been involved with various projects whose >>source code has been released under open source license, but he has no >>prior experience on contributing to open-source projects. With the >>guidance from other more experienced committers and participants, we >>expect that the meritocracy rules will have a positive influence on >>this project. >> >>== Homogeneous Developers == >> >>The initial committer comes from academia. However, given the interest >>we have seen in the project, we expect the diversity to improve in the >>near future. >> >>== Reliance on Salaried Developers == >> >>Currently, the MRQL code was developed on the committer's volunteer >>time. In the future, UTA graduate students who will do some of the >>coding may be supported by UTA and funding agencies, such as NSF. >> >>== Relationships with Other Apache Products == >> >>MRQL has some overlapping functionality with Hive and Tajo, which are >>Data Warehouse systems for Hadoop, and with Drill, which is an >>interactive data analysis system that can process nested data. MRQL >>has a more powerful data model, in which any form of nested data, such >>as XML and JSON, can be defined as a user-defined datatype. More >>importantly, complex data analysis tasks, such as PageRank, k-means >>clustering, and matrix multiplication and factorization, can be >>expressed as short SQL-like queries, while the MRQL system is able to >>evaluate these queries efficiently. Furthermore, the MRQL system can >>run these queries in BSP mode, in addition to MapReduce mode, thus >>achieving low latency and speed, which are also Drill's goals. >>Nevertheless, we will welcome and encourage any help from these >>projects and we will be eager to make contributions to these projects >>too. >> >>== An Excessive Fascination with the Apache Brand == >> >>The Apache brand is likely to help us find contributors and reach out >>to the open-source community. Nevertheless, since MRQL depends on >>Apache projects (Hadoop and Hama), it makes sense to have our software >>available as part of this ecosystem. >> >>= Documentation = >> >>Information about MRQL can be found at http://lambda.uta.edu/mrql/ >> >>= Initial Source = >> >>The initial MRQL code has been released as part of a research project >>developed at the University of Texas at Arlington under the Apache 2.0 >>license for the past two years. The source code is currently hosted >>on GitHub at: https://github.com/fegaras/mrql MRQL’s release artifact >>would consist of a single tarball of packaging and test code. >> >>= External Dependencies = >> >>The MRQL source code is already licensed under the Apache License, >>Version 2.0. MRQL uses JLine which is distributed under the BSD >>license. >> >>= Cryptography = >> >>Not applicable. >> >>= Required Resources = >> >>== Mailing Lists == >> >>* mrql-private >>* mrql-dev >>* mrql-user >> >>== Subversion Directory == >> >>* Git is the preferred source control system: >>git://git.apache.org/mrql >> >>== Issue Tracking == >> >>* A JIRA issue tracker, MRQL >> >>== Wiki == >> >> * Moinmoin wiki, http://wiki.apache.org/mrql >> >>= Initial Committers = >> >>* Leonidas Fegaras <fegaras AT cse DOT uta DOT edu> >>* Upa Gupta <upa.gupta AT mavs DOT uta DOT edu> >>* Edward J. Yoon <edwardyoon AT apache DOT org> >>* Maqsood Alam <maqsoodalam AT hotmail DOT com> >>* John Hope <john.hope AT oracle DOT com> >>* Mark Wall <mark.wall AT oracle DOT com> >>* Kuassi Mensah <kuassi.mensah AT oracle DOT com> >>* Ambreesh Khanna <ambreesh.khanna AT oracle DOT com> >>* Karthik Kambatla <kasha AT cloudera DOT com> >> >>= Affiliations = >> >>* Leonidas Fegaras (University of Texas at Arlington) >>* Upa Gupta (University of Texas at Arlington) >>* Edward J. Yoon (Oracle corp) >>* Maqsood Alam (Oracle corp) >>* John Hope (Oracle corp) >>* Mark Wall (Oracle corp) >>* Kuassi Mensah (Oracle corp) >>* Ambreesh Khanna (Oracle corp) >>* Karthik Kambatla (Cloudera) >> >>= Sponsors = >> >>== Champion == >> >>* Edward J. Yoon <edwardyoon AT apache DOT org> >> >>== Nominated Mentors == >> >>* Alex Karasulu <akarasulu AT apache DOT org> >>* Edward J. Yoon <edwardyoon AT apache DOT org> >> >>== Sponsoring Entity == >> >>Incubator PMC >> > -- Best Regards, Edward J. Yoon @eddieyoon --------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
