Hi, I've worked on the Distributed Transaction Manager that is now part of Trafodion for years. I have a vested interest in the DTM and in the Trafodion project's success, but I'm not opposed to adding additional/competing transaction managers to into incubation. Ultimately, I think it will build a better community and better products.
I also think that having the ability to plug in transaction manager (x) into and application or (no)SQL engine is a good thing. To that end I encourage the other transaction manager projects to read the existing JIRA that attempts to propose a standard API for transactions within HBase [HBASE-11447 Proposal for a generic transaction API for HBase]. Read it, comment, object, suggest alternatives and let's come up with a consensus that allows us to easily swap transaction managers to suit the application or workload. +1 to adding Omid (non binding) Regards, Sean -----Original Message----- From: Pierre Smits [mailto:pierre.sm...@gmail.com] Sent: Sunday, March 20, 2016 1:35 PM To: general@incubator.apache.org Subject: Re: [DISCUSS] [PROPOSAL] Omid for Apache Incubator Hi Henry, It seems you (and several others) are forgetting the Trafodion, which also privides transactions on N*SQL solutions, see http trafodion.apache.org Best regards, Pierre Smits ORRTIZ.COM <http://www.orrtiz.com> OFBiz based solutions & services OFBiz Extensions Marketplace http://oem.ofbizci.net/oci-2/ On Sat, Mar 19, 2016 at 12:19 AM, Henry Saputra <henry.sapu...@gmail.com> wrote: > I know Apache incubator does not play favorite but it is getting > awkward that TWO transaction engine for HBase coming to incubator at the > same time. > > As most people know, the other one is Tephra, that just coming to > incubator few weeks ago. > > As member of IPMC, I would like to see Omid provide some more details > comparisons about the difference that the project bring, in term of > approach and possible integrations with other ASF projects. > > If possible, I would prefer to see Omid team work together with Tephra > to work on working together to make one solid transaction engine for > HBase and later NoSQL databases. > > > - Henry > > On Thu, Mar 17, 2016 at 1:17 PM, Daniel Dai <dai...@gmail.com> wrote: > > > Hi, > > > > I would like to propose Omid as an Apache Incubator project: > > > > https://wiki.apache.org/incubator/OmidProposal > > > > I've posted posted the text of the proposal below: > > > > Thanks, > > Daniel > > > > = Omid Proposal = > > > > === Abstract === > > > > Omid is a flexible, reliable, high performant and scalable ACID > > transactional framework that allows client applications to execute > > transactions on top of MVCC key/value-based NoSQL datastores > > (currently Apache HBase) providing Snapshot Isolation guarantees on > > the accessed data. > > > > > > === Proposal === > > > > Omid is a flexible open-source transactional framework that provides > > ACID transactions with Snapshot Isolation guarantees on top of NoSQL > > datastores. In particular, the current codebase brings the concept > > of transactions to the popular Apache HBase datastore. Omid offers > > great performance, it is highly available, and scalable. Omid's > > current version is able to scale to thousands of clients triggering > > concurrent transactions on application data stored in HBase. Omid > > can scale beyond 100K transactions per second on mid-range hardware > > while incurring in a minimal impact on the speed of data access in > > the datastore. We’re currently experimenting with a prototype > > version that can improve the performance up to ~380K TPS. > > > > > > Omid has been publicly available as an open-source project in Github > > under Apache License Version 2.0 since 2011 [1]. During these years, > > it has generated certain interest in the open source community, > > especially since the public presentation of the first version in > > Hadoop Summit 2013 [2]. Currently the Github project has 241 Stars > > and > > 93 forks. Yahoo Inc. submits this proposal to the Apache Software > > Foundation with the aim to transfer the Omid project -including its > > source code and documentation- to Apache in order to start the build > > of a stable open source community around it. > > > > > > [1] https://github.com/yahoo/omid > > > > [2] Omid presentation at Hadoop Summit 2013: > > > > > https://www.youtube.com/watch?v=Rhdmo9pVGgU&index=68&list=PLSAiKuajRe2 > luyqLU464Nxz4aQe7EPBus > > > > > > === Background === > > > > An Omid prototype was first released as an open-source project back > > in 2011. Inspired by Google Percolator [1], it offered a lock-free > > approach to transactions in NoSQL datastores (See [2]). However, > > during these years, the design of Omid has evolved significantly. > > Whilst the current open-sourced version maintains many aspects of > > the original implementation, it is the result of a major redesign of > > the first prototype released in 2011. > > > > > > Omid has now a more decentralized design that does not sacrifice the > > consistency and performance of the original version. The current > > design also enables Omid to scale to thousands of clients executing > > transactions concurrently on application data stored in HBase. > > Internally, Omid still utilizes a lock-free approach to support > > multiple concurrent clients. Its design also relies on a centralized > > conflict detection component, the TSO, which now resolves in an > > efficient manner writeset collisions among concurrent transactions > > without having to piggyback commit information to the clients. > > Another important benefit of Omid is that it doesn't require any > > modification of the underlying key-value datastore, HBase in this > > case. Moreover, the recently added high availability algorithm > > allows to eliminate the single point of failure represented by the > > TSO in those system deployments requiring a higher degree of > > dependability. Last but not least, the provided user API is very > > simple, mimicking transaction managers in the relational world: begin, > > commit, rollback. > > > > > > Omid is used internally at Yahoo. Sieve, Yahoo’s web-scale content > > management platform powering some of next-generation search and > > personalization products is using Omid as a transaction manager in > > its processing pipeline. Sieve essentially acts as a huge processing > > hub between content feeds and serving systems. It provides an > > environment for highly customizable, real-time, streamed information > > processing, with typical discovery-to-service latencies of just a > > few seconds. In terms of scale and availability, Omid’s new design > > was largely driven by Sieve’s requirements. > > > > > > At Yahoo, we are also making an effort to disseminate the current > > status of the project through blog entries (See [3], [4] and [5]) > > and submissions to technical and academic conferences such as ATC > > 2016, Hadoop Summit 2016, HBaseConf 2016. Last but not least, Omid > > also appeared in a TechCrunch article in the last quarter of 2015 > > (See [6]) > > > > > > [1] D. Peng and F. Dabek, Large-scale Incremental Processing Using > > Distributed Transactions and Notifications. USENIX Symposium on > > Operating Systems Design and Implementation, 2010 > > > > [2] D. Gomez-Ferro, F. Junqueira, I. Kelly, B. Reed, and M. Yabandeh. > > Omid: Lock-free transactional support for distributed data stores. > > In Proc. of ICDE, 2013. > > > > [3] > > > http://yahoohadoop.tumblr.com/post/129089878751/introducing-omid-trans > action-processing-for > > > > [4] > > > http://yahoohadoop.tumblr.com/post/132695603476/omid-architecture-and- > protocol > > > > [5] > > > http://yahoohadoop.tumblr.com/post/138682361161/high-availability-in-o > mid > > > > [6] > > > http://techcrunch.com/2015/10/01/yahoos-open-source-omid-project-bring > s-scalable-transaction-processing-to-hbase/ > > > > > > === Rationale === > > > > Programming with ACID (Atomicity, Consistency, Isolation, > > Durability) transactions is very popular and it is featured in > > relational databases. However, in the Big Data ecosystem, > > applications typically use NoSQL datastores, which do not provide > > ACID transactions. Such NoSQL datastores used to give up > > transactional support for greater agility and scalability. However, > > while early NoSQL data store implementations did not include > > transaction support, the need for transactions soon emerged in Big > > Data applications when accessing shared data; for example, > > transactions are very important for modern, scalable systems that > > process content incrementally. > > > > > > NoSQL datastores -including HBase- don’t provide transactional > > frameworks to coordinate the access to the underlying data for > > preserving consistency. By using Omid, Big Data applications that > > need to bundle multiple read and write operations on HBase into > > logically indivisible units of work can execute transactions with > > ACID properties, just as they would use transactions in the > > relational database world. Omid extends the HBase key-value access > > APl with transaction semantics. It can be exercised either directly, > > or via higher level data management API’s. For example, Apache > > Phoenix > > (SQL-on-top-of-HBase) might use Omid as its transaction management > > component. > > > > > > The following features make Omid an attractive choice for system > > designers and other projects in the Apache community: > > > > > > * Semantics. Omid implements Snapshot Isolation (SI,) supported by > > major SQL and NoSQL technologies (e.g. Google Percolator). > > > > > > * Performance and Scalability. Omid provides a highly scalable, > > lock-free implementation of SI. To the best of our knowledge, it is > > also one of the few open source NoSQL transactional platforms that > > can execute more than 100K transactions per second [1]. A new > > prototype still in development can go even further, up to ~380K TPS. > > > > > > * Reliability. Omid has a high-availability (HA) mode, in which the > > core service performing writeset conflict resolution operates as > > primary-backup process pair with automatic failover. The HA support > > has zero overhead on the mainstream operation. > > > > > > * Adaptability. Omid current version provides transactions on data > > stored in Apache HBase. However, Omid’s components are generic > > enough to be adapted to any other key-value NoSQL datasource that > > supports MVCC. > > > > > > * Development. Omid provides a very simple interface that mimics > > standard HBase APIs, making it developer friendly. Only minimal > > extensions to the standard interfaces have been introduced to enable > > transactions. > > > > > > * Simplicity. Omid leverages the HBase infrastructure for managing > > its own metadata. It entails no additional services apart from those > > provided and used by HBase. > > > > > > * Track Record. As we have mentioned, Omid is already in use by > > very-large-scale production systems at Yahoo. Also, Hortonworks is > > integrating Omid in a metastore implementation for Hive based on > > HBase. > > > > [1] See also Haeinsa: > > https://github.com/vcnc/haeinsa/wiki/Performance > > > > > > === Current Status === > > Current Omid implementation is available in both, Yahoo’s internal > > Github repository for internal use at Yahoo as well as in Yahoo’s > > Github public repository (https://github.com/yahoo/omid.git). Both > > repositories are managed by Omid’s current developers at Yahoo. > > > > As it is mentioned above, Yahoo is currently using Omid for > > providing transactions in Sieve, a web-scale content management > > platform that powers Yahoo’s next-generation search and personalization > > products. > > > > > > ==== Meritocracy ==== > > The first version of Omid was originally created in 2011 by Maysam > > Yabandeh, Daniel Gomez-Ferro, Ivan B. Kelly, Benjamin Reed and > > Flavio Junqueira at the R&D Scalable Computing Group of Yahoo Labs in > > Spain. > > > > > > During the years after its inception, Omid has matured to operate at > > Web scale and has been used internally by strategic projects at > > Yahoo such as Sieve. The current base of committers belong to the > > Yahoo team that took over the initial Omid prototype and rewrote it > > to meet the high availability and scalability requirements of the Sieve > > project. > > This base of committers has recently incorporated Hortonworks > > members that helped in the Omid adaptation to HBase 1.x versions. > > > > > > With this initial committer base, we aim to form a larger community > > that can collaborate with new ideas over the current code base. This > > new community will run the project following the "Apache Way" > > (http://apache.org/foundation/governance/). Users and new > > contributors will be treated with respect and welcomed. To grow the > > community, we will encourage contributors to provide patches, review > > code, propose new features improvements, talk at conferences such as > > Hadoop Summit, HBaseCon, ApacheCon, etc. Committership and PMC > > membership will be offered according to meritocracy. > > > > ==== Community ==== > > > > The public Yahoo Omid repository at Github currently has 241 Stars > > and > > 93 forks, which means that there is an important interest for the > > project in the open-source community, at least compared with other > > similar projects (See https://github.com/yahoo/omid.git). > > > > > > Recently, Hortonworks contributors to the Apache Hive project which > > are working on storing Hive metadata in HBase (Apache Jira > > HIVE-9452) manifested interest in using Omid. We started with them a > > fruitful collaboration that resulted in Omid supporting HBase 1.x > > versions. > > > > > > Salesforce is also interested in collaborating in doing a Proof of > > Concept for integrating Omid as a pluggable transaction manager in > > Apache Phoenix. > > > > > > Yahoo, Hortonworks and Salesforce participants will constitute the > > initial set of committers and mentors for the proposal. > > > > ==== Core Developers ==== > > The core developers of Omid are all skilled software developers and > > research engineers at Yahoo Inc. and Hortonworks with years of > > experiences in their fields. At this moment, developers are > > distributed across U.S. and Israel. The aim is to incorporate more > > committers from different organizations and locations over time. > > > > > > The current set of developers include experienced committers from > > Apache HBase, Hive and Hadoop projects that have been working with > > us in the current codebase found in Github. > > > > Finally, some of the core developers are currently NOT affiliated > > with the ASF and would require new ICLAs to be filed. > > > > > > === Alignment === > > Omid enhances with transactions the already successful Apache HBase > > datastore project. We have collaborated with other developers inside > > and outside Yahoo which are involved in the Apache HBase community, > > so we have had reliable feedback from them. > > > > Although Omid brings value into HBase, the design of the current > > version provides a general transaction scheme that can potentially > > be adapted to other MVCC key-value datastores such as Apache Cassandra. > > > > > > Apache Phoenix is also a potential target. Phoenix is a SQL layer on > > top of HBase that can potentially integrate Omid in order to provide > > the well-know concept of transactions to Phoenix-based applications. > > > > > > === Known Risks === > > ==== Orphaned products ==== > > Yahoo’s Research and Search organizations have been taking care of > > Omid development since the first prototype creation in 2011. Yahoo > > has a long history participating in open-source projects, and has > > been also a long time contributor to the Apache community. For > > example, in Apache, Yahoo is an important contributor in many > > projects in the Hadoop ecosystem such as HBase, Pig, Storm or YARN, > > and has also open-sourced other well-known projects outside Hadoop, > > such as Zookeeper or Bookkeeper. So it is in the best interest of > > Yahoo make Omid also a successful open-source Apache product. If > > this happens, we are sure that a larger community will be formed > > around the project in a relatively short period of time, > > contributing to the diversification and stabilization of the base of > > committers. > > > > > > ==== Inexperience with Open Source ==== This project has long > > standing experienced mentors and interested contributors from Apache > > HBase, Hive and Phoenix to help us moving through the open source > > process. We are actively working with experienced Apache community > > members to improve our project and further testing. > > > > ==== Homogeneous Developers ==== > > Omid has been supported by Yahoo since its inception in 2011. > > However, all current committers are employed by their respective > > companies shown in the Affiliations section. > > > > > > ==== Reliance on Salaried Developers ==== > > > > All the current developers are paid by their employers to contribute > > to this project. Yahoo developers will also continuing maintaining > > the internal Omid repository at their company. > > > > Of course, other developers are welcomed to contribute to this > > project after it is open sourced in Apache. > > > > ==== Relationships with Other Apache Product ==== > > > > Current Omid incarnation serves transactional contexts to > > applications storing their data in HBase. However Omid design > > potentially allows to be adapted to serve transactions on top of > > other MVCC-based key-value datastores in Apache community such as > > Cassandra. > > > > > > As a transactional framework, many other Apache projects such as > > Apache Spark, Apache Phoenix, Apache Storm, Apache Flink could > > potentially benefit from Omid to get transactional contexts. In > > particular, Apache Phoenix -a SQL layer on top of HBase- might use > > Omid as its transaction management component. Once we open source > > Omid as an Apache project, we expect to generate more interest in > > the surrounded communities. > > > > > > Very recently, a new incubator proposal for a similar project called > > Tephra, has been submitted to the ASF. We think this is good for the > > Apache community, and we believe that there’s room for both > > proposals as the design of each of them is based on different principles > > (e.g. > > Omid does not require to maintain the state of ongoing transactions > > on the server-side component) and due to the fact that both -Tephra > > and > > Omid- have also gained certain traction in the open-source community. > > > > > > With regard to the Apache projects that Omid uses, apart from HBase, > > Omid relies on Apache Zookeeper and Curator projects in order to > > coordinate the (re)connection of transaction managers (acting as > > clients) to the conflict resolution component for transactions > > (server > > side.) They’re also used in order to coordinate the master and > > backup replicas in high availability scenarios. > > > > > > ==== An Excessive Fascination with the Apache Brand ==== > > > > We are applying to the Incubator process because we think that it is > > the logical next step for the Omid project after we open-sourced > > the code in Github some years ago. Yahoo has a long-standing history > > of contributing to Apache projects. The developers and contributors > > understand the implications of making it an Apache project, and > > strongly believe that the growing community can benefit from the > > Apache environment, ecosystem, and infrastrastructure. > > > > > > === Documentation === > > Current documentation about the project is available in the wiki of > > Omid’s Github repository: https://github.com/yahoo/omid/wiki . It > > will be moved under https://omid.incubator.apache.org/docs if the > > project is accepted as an Apache Incubator. > > > > === Initial Source === > > Initial source code is currently hosted in Github for general > > viewing and contribution: > > > > https://github.com/yahoo/omid.git > > > > > > Omid source code is written in Java code (99%) mixed with some shell > > script (1%) in order to configure and trigger the execution of main > > components. > > > > > > The code will be moved to Apache http://git.apache.org/ if accepted > > as an Incubator project. > > > > === Source and Intellectual Property Submission Plan === > > > > The current Omid License for the code published in Github is Apache > > 2.0. If Omid fulfills and passes the conditions for being an > > Incubator project in the ASF, the source code will be transitioned > > via the Software Grant Agreement onto the ASF infrastructure and in > > turn made available under the Apache License, version 2.0. > > > > === External Dependencies === > > > > > > The required external dependencies that are not Apache projects are > > all Apache licenses or other compatible Licenses: > > > > Maven & Maven plugins (http://maven.apache.org/) [Apache 2.0] > > > > JDK7 or OpenJDK 7 (http://java.com/) [Oracle or Openjdk JDK License] > > > > Google Guava v11.0.2 (https://github.com/google/guava) [Apache 2.0] > > > > Google Guice v3.0 (https://github.com/google/guice/wiki) [Apache > > 2.0] > > > > Testng v6.8.8 (http://testng.org) [Apache 2.0] > > > > SLF4J (http://www.slf4j.org/) v1.7.7 [MIT License] > > > > Netty (http://netty.io) v3.2.6.Final [Apache 2.0] > > > > Google Protocol Buffers v2.5.0 > > (https://developers.google.com/protocol-buffers/) [BSD License] > > > > Mockito (http://mockito.org/) v1.9.5 [MIT License] > > > > LMAX Disruptor v3.2.0 (https://lmax-exchange.github.io/disruptor/) > > [Apache 2.0] > > > > Coda Hale/Yammer.com Dropwizard Metrics v3.0.1 > > (http://metrics.dropwizard.io/3.1.0/) [Apache 2.0] > > > > C.Beust, JCommander v1.35 (http://jcommander.org/) [Apache 2.0] > > > > Hamcrest v1.3 (http://hamcrest.org/JavaHamcrest/) [BSD License] > > > > > > === Cryptography === > > Omid project does not use cryptography itself. However, Apache HBase > > -the datastore on top of which Omid works in its current version- > > uses standard APIs and tools for SSH and SSL communication where > > necessary. > > > > === Required Resources === > > We request that following resources be created for the project to use: > > > > ==== Mailing lists ==== > > > > omid-private (moderated subscriptions) > > > > omid-commits (commit notification) > > omid-dev (technical discussions) > > > > ==== Git repository ==== > > https://github.com/apache/incubator-omid > > > > ==== Documentation ==== > > https://omid.incubator.apache.org/docs/ > > > > ==== JIRA instance ==== > > https://issues.apache.org/jira/browse/omid > > > > === Initial Committers === > > > > * Daniel Dai, Hortonworks (daijy<AT>hortonworks<DOT>com) > > > > > > * Alan Gates, Hortonworks, (gates<AT>hortonworks<DOT>com) > > > > > > * Lars Hofhansl, Salesforce (larsh<AT>apache<DOT>org) > > > > > > * Flavio P. Junqueira, Confluent (fpj<AT>apache<DOT>org) > > > > > > * Igor Katkov (katkovi<AT>yahoo-inc<DOT>com) > > > > > > * Francis C. Liu (fcliu<AT>yahoo-inc<DOT>com) > > > > * Thejas Nair, Hortonworks (thejas<AT>hortonworks<DOT>com) > > > > > > * Francisco Perez-Sorrosal (fperez<AT>yahoo-inc<DOT>com) > > > > > > * Sameer Paranjpye (sparanjpye<AT>yahoo<DOT>com) > > > > > > * Ohad Shacham (ohads<AT>yahoo-inc<DOT>com) > > > > * James Taylor, Salesforce (jamestaylor<AT>apache<DOT>org>) > > > > > > === Additional Interested Contributors === > > * Ivan Kelly (ivank<AT>apache<DOT>org) > > > > * Maysam Yabandeh (myabandeh<AT>dropbox<DOT>com) > > > > > > === Affiliations === > > > > * Edward Bortnikov, Yahoo Inc. > > > > > > * Daniel Dai, Hortonworks > > > > > > * Flavio P. Junqueira, Confluent > > > > > > * Igor Katkov, Yahoo Inc. > > > > > > * Ivan Kelly, Midokura > > > > > > * Francis C. Liu, Yahoo Inc. > > > > > > * Sameer Paranjpye, Arimo > > > > * Francisco Perez-Sorrosal, Yahoo Inc. > > > > > > * Ohad Shacham, Yahoo Inc. > > > > > > * Maysam Yabandeh, Dropbox Inc. > > > > > > === Sponsors === > > > > ==== Champion ==== > > > > Daniel Dai, Hortonworks (daijy<AT>hortonworks<DOT>com) > > > > ==== Nominated Mentors ==== > > > > Alan Gates, Hortonworks, (gates<AT>hortonworks<DOT>com) > > > > Lars Hofhansl, Salesforce (larsh<AT>apache<DOT>org) > > > > Flavio P. Junqueira, Confluent (fpj<AT>apache<DOT>org) > > > > Thejas Nair, Hortonworks (thejas<AT>hortonworks<DOT>com) > > > > James Taylor, Salesforce (jamestaylor<AT>apache<DOT>org>) > > > > > > ==== Sponsoring Entity ==== > > Apache Incubator PMC > > > > -------------------------------------------------------------------- > > - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > For additional commands, e-mail: general-h...@incubator.apache.org > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
