Re: Advice on binary NOTICE

[Justin Mclean]

Hi,

> *   The PMC is spared from performing analysis of dependency NOTICE content.
> *   Verbatim aggregation can be achieved programmatically, allowing for
>automated solutions.

It simple I like that.

However there's probably no TLP or incubator project (with bundled Apache 
licensed code) that follows that currently. It will also tend to make NOTICE 
files larger and that has a flow on effect to downstream projects. It will have 
a larger impact on connivence binaries NOTICE files.

Would’t it be better for incubating projects to just ask IPMC or legal for help 
when putting together NOTICE files?

Thanks,
Justin
-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: Allowed Champions on podlings

[Ted Dunning]

On Sun, Mar 20, 2016 at 6:39 PM, Marvin Humphrey 
wrote:

> In the meantime, I don't think the present rule offers enough value to
> justify
> its complexity.
>

+1

Re: Allowed Champions on podlings

[Marvin Humphrey]

On Fri, Mar 18, 2016 at 2:53 AM, Tim Williams  wrote:

> Without judging the goodness of it... I'd just point out that
> currently in the non-Member Officer case, they must be a member of the
> Sponsoring PMC.  I thought that was true of Members as well, btw, but
> thought I'd point out that it's not simply Members and Officers.

Thanks for bringing that up, Tim.

In my view, this rule is too complicated, and therefore I support John's
initiative to allow Champions to be any ASF Member or Officer.  Here is a
patch implementing his proposal:

https://paste.apache.org/LwdO

The primary utility of requiring the Champion to have certain qualifications
is to ensure that projects contemplating incubation get good guidance, so that
they either produce a sound proposal or make an informed decision not to
incubate. (I've seen both happen, as have others who have served as a
Champion.)

But by the time a full-blown proposal lands on general@incubator, it's too
late to go back and change the guidance that the candidate project received.
Thus, when this rule is misunderstood, the only effect is that we end up in a
distracting debate about whether to replace a Champion whose most important
work is already done.  This does not help the prospective podling, nor does it
inoculate future prospective podlings against receiving poor guidance from an
unqualified Champion.

These days, the Sponsor is nearly always the Incubator, anyway.  I don't think
the IPMC objects to having Officers who are neither ASF Members nor IPMC
members serve as Champion for proposals where the Incubator is the Sponsor.

In the unlikely event that there is a podling proposal where the Sponsor is
not the Incubator AND the Champion is neither an ASF Member nor a member of
the Sponsoring PMC AND the Sponsoring PMC objects... we can cross that bridge
when we come to it.

In the meantime, I don't think the present rule offers enough value to justify
its complexity.

Marvin Humphrey

-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

RE: [DISCUSS] [PROPOSAL] Omid for Apache Incubator

[Atanu Mishra]

Hi,

Yes, very true, Pierre.

Back in the summer of 2014, a proposal to implement a generic transaction
API was submitted as HBASE-11447 to allow support for multiple
implementations of transaction monitors. Some discussions were held at the
time, but a final decision on the API has not been made.

HBASE-11447 Proposal for a generic transaction API for HBase

Regards,
Atanu

-Original Message-
From: Pierre Smits [mailto:pierre.sm...@gmail.com]
Sent: Sunday, March 20, 2016 1:35 PM
To: general@incubator.apache.org
Subject: Re: [DISCUSS] [PROPOSAL] Omid for Apache Incubator

Hi Henry,

It seems you (and several others) are forgetting the Trafodion, which also
privides transactions on N*SQL solutions, see http trafodion.apache.org

Best regards,

Pierre Smits

ORRTIZ.COM 
OFBiz based solutions & services

OFBiz Extensions Marketplace
http://oem.ofbizci.net/oci-2/

On Sat, Mar 19, 2016 at 12:19 AM, Henry Saputra 
wrote:

> I know Apache incubator does not play favorite but it is getting awkward
> that TWO transaction engine for HBase coming to incubator at the same
> time.
>
> As most people know, the other one is Tephra, that just coming to
> incubator
> few weeks ago.
>
> As member of IPMC, I would like to see Omid provide some more details
> comparisons about the difference that the project bring,  in term of
> approach and possible integrations with other ASF projects.
>
> If possible, I would prefer to see Omid team work together with Tephra to
> work on working together to make one solid transaction engine for HBase
> and
> later NoSQL databases.
>
>
> - Henry
>
> On Thu, Mar 17, 2016 at 1:17 PM, Daniel Dai  wrote:
>
> > Hi,
> >
> > I would like to propose Omid as an Apache Incubator project:
> >
> > https://wiki.apache.org/incubator/OmidProposal
> >
> > I've posted posted the text of the proposal below:
> >
> > Thanks,
> > Daniel
> >
> > = Omid Proposal =
> >
> > === Abstract ===
> >
> > Omid is a flexible, reliable, high performant and scalable ACID
> > transactional framework that allows client applications to execute
> > transactions on top of MVCC key/value-based NoSQL datastores
> > (currently Apache HBase) providing Snapshot Isolation guarantees on
> > the accessed data.
> >
> >
> > === Proposal ===
> >
> > Omid is a flexible open-source transactional framework that provides
> > ACID transactions with Snapshot Isolation guarantees on top of NoSQL
> > datastores. In particular, the current codebase brings the concept of
> > transactions to the popular Apache HBase datastore. Omid offers great
> > performance, it is highly available, and scalable. Omid's current
> > version is able to scale to thousands of clients triggering concurrent
> > transactions on application data stored in HBase. Omid can scale
> > beyond 100K transactions per second on mid-range hardware while
> > incurring in a minimal impact on the speed of data access in the
> > datastore. We’re currently experimenting with a prototype version that
> > can improve the performance up to ~380K TPS.
> >
> >
> > Omid has been publicly available as an open-source project in Github
> > under Apache License Version 2.0 since 2011 [1]. During these years,
> > it has generated certain interest in the open source community,
> > especially since the public presentation of the first version in
> > Hadoop Summit 2013 [2]. Currently the Github project has 241 Stars and
> > 93 forks. Yahoo Inc. submits this proposal to the Apache Software
> > Foundation with the aim to transfer the Omid project -including its
> > source code and documentation- to Apache in order to start the build
> > of a stable open source community around it.
> >
> >
> > [1] https://github.com/yahoo/omid
> >
> > [2] Omid presentation at Hadoop Summit 2013:
> >
> >
> https://www.youtube.com/watch?v=Rhdmo9pVGgU=68=PLSAiKuajRe2luyqLU464Nxz4aQe7EPBus
> >
> >
> > === Background ===
> >
> > An Omid prototype was first released as an open-source project back in
> > 2011. Inspired by Google Percolator [1], it offered a lock-free
> > approach to transactions in NoSQL datastores (See [2]). However,
> > during these years, the design of Omid has evolved significantly.
> > Whilst the current open-sourced version maintains many aspects of the
> > original implementation, it is the result of a major redesign of the
> > first prototype released in 2011.
> >
> >
> > Omid has now a more decentralized design that does not sacrifice the
> > consistency and performance of the original version. The current
> > design also enables Omid to scale to thousands of clients executing
> > transactions concurrently on application data stored in HBase.
> > Internally, Omid still utilizes a lock-free approach to support
> > multiple concurrent clients. Its design also relies on a centralized
> > conflict detection component, the TSO, which now resolves in an
> > efficient manner writeset collisions among concurrent transactions
> > without 

Re: Advice on binary NOTICE

[Henri Yandell]

On Sun, Mar 20, 2016 at 12:28 PM, Marvin Humphrey 
wrote:

> On Sun, Mar 20, 2016 at 11:24 AM, Henri Yandell  wrote:
> > I suspect 'relevant' means those parts of a NOTICE relating to the parts
> of
> > the product you use.
>
> Yes, that was the intent.  That sentence references section 4d of the ALv2,
> specifically this phrase:
>
>excluding those notices that do not pertain to any part of the
>Derivative Works,
>
> > I suspect 'relevant' needs clarification in the docs.
>
> Over time, I have come to think our approach to NOTICE should be more
> formulaic.  The expectation that our PMCs will edit dependency NOTICE
> content
> is too burdensome.
>
> It is quite challenging to analyze which parts of a NOTICE file may be
> omitted
> -- especially when you take into account the vagaries of subsuming the
> notice
> requirements of licenses besides the ALv2.  Only a handful of our PMCs
> possess
> sufficient collective expertise in open source licensing to perform such
> license analysis accurately.
>
> Yet, our goal should be for *every* Apache release to conform to both legal
> requirements and policy.  We can't change our legal obligations -- but we
> can
> and should craft policy and best-practice recommendations so that the
> process
> of assembling LICENSE and NOTICE is not so taxing and yields consistently
> correct results.
>
> The proposal from last month to cease NOTICE aggregation entirely[1]
> failed to
> achieve consensus.  The next-best alternative, it seems to me, is a
> completely
> mechanical approach to aggregation: verbatim copying of NOTICE comment from
> dependency NOTICE files to the top-level NOTICE file.  This has two
> significant advantages:
>
> *   The PMC is spared from performing analysis of dependency NOTICE
> content.
> *   Verbatim aggregation can be achieved programmatically, allowing for
> automated solutions.
>
> In the case of a NOTICE file from a non-ASF ALv2 product, verbatim
> propagation
> is a completely defensible choice.  But I think we should consider
> recommending it for all ALv2 dependencies, including ASF products.
>

+1. License compliance is best when it's simple, and errs on being
over-informative.

I think defining the structure of the files would be valuable. A standard
delimiter ( etc) between different sections and a 'header' to a
section; ie) "Contents of Jackson 1.3 NOTICE file".

Hen

Re: [DISCUSS] [PROPOSAL] Omid for Apache Incubator

[Pierre Smits]

Hi Henry,

It seems you (and several others) are forgetting the Trafodion, which also
privides transactions on N*SQL solutions, see http trafodion.apache.org

Best regards,

Pierre Smits

ORRTIZ.COM 
OFBiz based solutions & services

OFBiz Extensions Marketplace
http://oem.ofbizci.net/oci-2/

On Sat, Mar 19, 2016 at 12:19 AM, Henry Saputra 
wrote:

> I know Apache incubator does not play favorite but it is getting awkward
> that TWO transaction engine for HBase coming to incubator at the same time.
>
> As most people know, the other one is Tephra, that just coming to incubator
> few weeks ago.
>
> As member of IPMC, I would like to see Omid provide some more details
> comparisons about the difference that the project bring,  in term of
> approach and possible integrations with other ASF projects.
>
> If possible, I would prefer to see Omid team work together with Tephra to
> work on working together to make one solid transaction engine for HBase and
> later NoSQL databases.
>
>
> - Henry
>
> On Thu, Mar 17, 2016 at 1:17 PM, Daniel Dai  wrote:
>
> > Hi,
> >
> > I would like to propose Omid as an Apache Incubator project:
> >
> > https://wiki.apache.org/incubator/OmidProposal
> >
> > I've posted posted the text of the proposal below:
> >
> > Thanks,
> > Daniel
> >
> > = Omid Proposal =
> >
> > === Abstract ===
> >
> > Omid is a flexible, reliable, high performant and scalable ACID
> > transactional framework that allows client applications to execute
> > transactions on top of MVCC key/value-based NoSQL datastores
> > (currently Apache HBase) providing Snapshot Isolation guarantees on
> > the accessed data.
> >
> >
> > === Proposal ===
> >
> > Omid is a flexible open-source transactional framework that provides
> > ACID transactions with Snapshot Isolation guarantees on top of NoSQL
> > datastores. In particular, the current codebase brings the concept of
> > transactions to the popular Apache HBase datastore. Omid offers great
> > performance, it is highly available, and scalable. Omid's current
> > version is able to scale to thousands of clients triggering concurrent
> > transactions on application data stored in HBase. Omid can scale
> > beyond 100K transactions per second on mid-range hardware while
> > incurring in a minimal impact on the speed of data access in the
> > datastore. We’re currently experimenting with a prototype version that
> > can improve the performance up to ~380K TPS.
> >
> >
> > Omid has been publicly available as an open-source project in Github
> > under Apache License Version 2.0 since 2011 [1]. During these years,
> > it has generated certain interest in the open source community,
> > especially since the public presentation of the first version in
> > Hadoop Summit 2013 [2]. Currently the Github project has 241 Stars and
> > 93 forks. Yahoo Inc. submits this proposal to the Apache Software
> > Foundation with the aim to transfer the Omid project -including its
> > source code and documentation- to Apache in order to start the build
> > of a stable open source community around it.
> >
> >
> > [1] https://github.com/yahoo/omid
> >
> > [2] Omid presentation at Hadoop Summit 2013:
> >
> >
> https://www.youtube.com/watch?v=Rhdmo9pVGgU=68=PLSAiKuajRe2luyqLU464Nxz4aQe7EPBus
> >
> >
> > === Background ===
> >
> > An Omid prototype was first released as an open-source project back in
> > 2011. Inspired by Google Percolator [1], it offered a lock-free
> > approach to transactions in NoSQL datastores (See [2]). However,
> > during these years, the design of Omid has evolved significantly.
> > Whilst the current open-sourced version maintains many aspects of the
> > original implementation, it is the result of a major redesign of the
> > first prototype released in 2011.
> >
> >
> > Omid has now a more decentralized design that does not sacrifice the
> > consistency and performance of the original version. The current
> > design also enables Omid to scale to thousands of clients executing
> > transactions concurrently on application data stored in HBase.
> > Internally, Omid still utilizes a lock-free approach to support
> > multiple concurrent clients. Its design also relies on a centralized
> > conflict detection component, the TSO, which now resolves in an
> > efficient manner writeset collisions among concurrent transactions
> > without having to piggyback commit information to the clients. Another
> > important benefit of Omid is that it doesn't require any modification
> > of the underlying key-value datastore, HBase in this case. Moreover,
> > the recently added high availability algorithm allows to eliminate the
> > single point of failure represented by the TSO in those system
> > deployments requiring a higher degree of dependability. Last but not
> > least, the provided user API is very simple, mimicking transaction
> > managers in the relational world: begin, commit, rollback.
> >
> >
> > Omid is used 

Re: Advice on binary NOTICE

[Marvin Humphrey]

On Sun, Mar 20, 2016 at 11:24 AM, Henri Yandell  wrote:
> I suspect 'relevant' means those parts of a NOTICE relating to the parts of
> the product you use.

Yes, that was the intent.  That sentence references section 4d of the ALv2,
specifically this phrase:

   excluding those notices that do not pertain to any part of the
   Derivative Works,

> I suspect 'relevant' needs clarification in the docs.

Over time, I have come to think our approach to NOTICE should be more
formulaic.  The expectation that our PMCs will edit dependency NOTICE content
is too burdensome.

It is quite challenging to analyze which parts of a NOTICE file may be omitted
-- especially when you take into account the vagaries of subsuming the notice
requirements of licenses besides the ALv2.  Only a handful of our PMCs possess
sufficient collective expertise in open source licensing to perform such
license analysis accurately.

Yet, our goal should be for *every* Apache release to conform to both legal
requirements and policy.  We can't change our legal obligations -- but we can
and should craft policy and best-practice recommendations so that the process
of assembling LICENSE and NOTICE is not so taxing and yields consistently
correct results.

The proposal from last month to cease NOTICE aggregation entirely[1] failed to
achieve consensus.  The next-best alternative, it seems to me, is a completely
mechanical approach to aggregation: verbatim copying of NOTICE comment from
dependency NOTICE files to the top-level NOTICE file.  This has two
significant advantages:

*   The PMC is spared from performing analysis of dependency NOTICE content.
*   Verbatim aggregation can be achieved programmatically, allowing for
automated solutions.

In the case of a NOTICE file from a non-ASF ALv2 product, verbatim propagation
is a completely defensible choice.  But I think we should consider
recommending it for all ALv2 dependencies, including ASF products.

Marvin Humphrey

[1] https://s.apache.org/mhQi

-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: Update on Apache Toree and LGPL dependency

[Henri Yandell]

Brilliant :)

On Thursday, March 17, 2016, Chip Senkbeil  wrote:

> Just wanted to give a status update with this one. JeroMQ is down to just
> four contributors that have not responded. The current, active committers
> for JeroMQ have reverted the commits for one of the contributors here:
>
> https://github.com/zeromq/jeromq/pull/333
>
> So, progress is still being made on this one!
>
> > +1
> >
> > > On Mar 6, 2016, at 6:58 PM, Gino Bustelo  > wrote:
> > >
> > > @john The 0mq ecosystem is made up of many projects of different sizes
> and maturity.
> > In the case of JeroMQ, the committers are showing an overwhelming
> momentum to transition to
> > MPL. I don't see any reason for us to consider any other alternative at
> this juncture.
> > >
> > > Gino B.
> > >
> > >> On Mar 5, 2016, at 11:42 PM, Henri Yandell  > wrote:
> > >>
> > >> Having chatted around the 0mq community in the past; I've confidence
> in
> > >> their desire to move to MPL; and 26/32 committers is a great step
> forward.
> > >> You raise a good reservation though John - if you remove the blocker
> on the
> > >> usage side, it's easy for the licensing to remain as is.
> > >>
> > >>
> > >> I'm +1 for releasing, with a prominent note of the LGPL dependency
> (along
> > >> with a note of the resolution plan). It might be that the Toree
> committers
> > >> may be motivated to rewrite code over at 0mq if there ends up being
> any
> > >> committers who are unavailable or unwilling to relicense.
> > >>
> > >> Hen
> > >>
> > >>> On Sat, Mar 5, 2016 at 3:45 PM, John D. Ament  >
> wrote:
> > >>>
> > >>> Sorry, misread the revision I was looking at.  The intent to move to
> MPL
> > >>> was done on March 22 2014, 2 years ago this month, not December 2013.
> > >>>
> > >>> John
> > >>>
> > >>> On Sat, Mar 5, 2016 at 6:41 PM John D. Ament  >
> > >>> wrote:
> > >>>
> >  I have some reservations with what you're proposing, and would like
> you
> > >>> to
> >  consult w/ legal-discuss on this first.
> > 
> >  There's a difference between what Mynewt did and what you're
> proposing.
> >  Specifically, this was a transitive dependency that they relied upon
> >  indirectly, so its more of a call out for the library that was
> leveraging
> >  it.  They also intended to replace the library.
> > 
> >  In your case, you're directly tied to a presently LGPL'd library.
> You
> >  have no intentions (from what I can see) of moving off of the
> library.
> > 
> >  I'm also doubting their long term goals of moving to MPL.  If you
> look at
> >  [1], you'll see that the page hasn't been updated since October
> 2014.  In
> >  addition, looking at the pages revision history (the beauty of
> wikis),
> > >>> the
> >  intent to move to MPL was published in December 2013, making the
> > >>> statement
> >  over 2 years old.
> > 
> >  I think while this might be OK for an initial incubator release, the
> >  project needs to weigh very heavily if it wants to continue to
> leverage
> >  ZeroMQ or not going forward.
> > 
> >  [1]: http://zeromq.org/area:licensing
> > 
> > 
> > > On Sat, Mar 5, 2016 at 5:06 PM Gino Bustelo  >
> > wrote:
> > >
> > > Wanted to give folks an update on our progress with dealing with
> JeroMQ,
> > > an
> > > LGPL package that enables us to communicate via 0MQ. The 0MQ
> community
> > >>> is
> > > very aware of the issues with LGPL (LGPLv3 + static link exception)
> and
> > >>> it
> > > is their intention to try to move projects to MPL v2. This is not
> an
> > >>> easy
> > > task depending on the age and size of the projects.
> > >
> > > Apache Toree's API access point is through the 0MQ transport layer
> > >>> (using
> > > JeroMQ) and that is how Apache Toree connects out-of-the-box with
> > >>> Jupyter,
> > > a very common way of consuming Apache Toree that is already in
> > >>> production.
> > >
> > > At this point, the JeroMQ project is still released under LGPL, but
> our
> > > team initiated communications in mid-February with members of the
> JeroMQ
> > > community to begin their transition to MPL v2 (
> > > https://github.com/zeromq/jeromq/issues/326). The JeroMQ community
> > > reacted
> > > very positively and quickly began the process of collecting votes
> from
> > > their committers (https://github.com/zeromq/jeromq/issues/327).
> After
> > >>> 15
> > > days, the current tally stands at 26 out of 32 committers have
> agreed
> > to
> > > switch license.
> > >
> > > Apache Toree has a JIRA (
> > >>> https://issues.apache.org/jira/browse/TOREE-262)
> > > where we keep all the relevant links and update with the latest
> > > information. As that 

Re: Advice on binary NOTICE

[Henri Yandell]

I suspect 'relevant' means those parts of a NOTICE relating to the parts of
the product you use.

In this case you'd include the whole file (ie +1 to Marvin).

I suspect 'relevant' needs clarification in the docs.

Hen



On Friday, March 18, 2016, Stephen Mallette  wrote:

> The Jackson JSON processing lib which is Apache 2.0 licensed carries this
> NOTICE file:
>
> --
> # Jackson JSON processor
>
> Jackson is a high-performance, Free/Open Source JSON processing library.
> It was originally written by Tatu Saloranta (tatu.salora...@iki.fi
> ), and has
> been in development since 2007.
> It is currently developed by a community of developers, as well as
> supported
> commercially by FasterXML.com.
>
> ## Licensing
>
> Jackson core and extension components may be licensed under different
> licenses.
> To find the details that apply to this artifact see the accompanying
> LICENSE file.
> For more information, including possible other licensing options, contact
> FasterXML.com (http://fasterxml.com).
>
> ## Credits
>
> A list of contributors may be found from CREDITS file, which is included
> in some artifacts (usually source distributions); but is always available
> from the source code management (SCM) system project uses.
> --
>
> Does anyone have any advice on what portion of this is relevant for
> inclusion in a binary NOTICE file? Should it all be included perhaps?
>
> Perhaps more generally, given
>
> http://www.apache.org/dev/licensing-howto.html#alv2-dep
>
> where it says,
>
> "If the dependency supplies a NOTICE file, its contents must be analyzed
> and the relevant portions bubbled up into the top-level NOTICE file."
>
> is there any more detailed information on how "relevant portions" get
> determined?
>
> Thanks,
>
> Stephen
>

Re: [DISCUSS] [PROPOSAL] Omid for Apache Incubator

[Henry Saputra]

That seems be a good approach for Apache Phoenix to enable possible
different transaction engine.

- Henry

On Sat, Mar 19, 2016 at 1:59 PM, Andrew Purtell 
wrote:

> Apache Phoenix just released version 4.7.0 with big news: transactions
> support, using Tephra. There's some interest in a successful Tephra
> incubation beyond the podling already. That said, that new code in Phoenix
> can be made pluggable to support more than one transaction oracle. Omid
> might be able to provide workable integration to stand in for Tephra.
> Collaboration between or even a joining of the two communities could be
> good but even if not as a potential downstream consumer it's good to have
> options! (provided the number of alternatives is bounded with reason of
> course). I think it would be good to see Omid get in. I think an Omid
> podling would find interested collaborators in the Phoenix and HBase
> communities right away.
>
>
> > On Mar 19, 2016, at 12:20 PM, Henry Saputra 
> wrote:
> >
> > Thanks for the great explanation, Flavio.
> >
> > As many have mentioned before, it is definitely ok to have similar
> projects
> > in ASF. We have prior acts before and I didn't expect incubator to reject
> > good projects coming in.
> >
> > My intention was to avoid split of resources where both projects have
> > very similar goal and approach. But maybe both projects have different
> > subtle differences that worthy to be done as independent effort.
> >
> > Just being devil advocate a bit to see if potential to collaborate.
> >
> > - Henry
> >
> >> On Saturday, March 19, 2016, Flavio Junqueira  wrote:
> >>
> >> I understand the concern, so let me try to offer some facts and see if
> we
> >> can make progress from there.
> >>
> >> Omid has been around for some time now, and its initial design appeared
> in
> >> a couple of research papers that I actually co-authored. The
> architecture
> >> is based on the idea of having a centralized transaction status oracle
> that
> >> shares transaction status data with clients for scalability. The current
> >> Omid project evolved out of that initial work and it is a much improved
> >> version over that first iteration, with the improvements focusing on
> >> scalability. It currently runs in production at scale at Yahoo! and
> there
> >> is interest from other companies according to the proposal. There is a
> >> series of blog posts about the experience in the project proposal.
> >>
> >> Tephra has a very similar architecture. The description here says that
> it
> >> has a transaction server, which sounds like the TSO in the original Omid
> >> papers. I haven't spent enough time understanding the precise protocol
> they
> >> use, but I must say that the protocol is very important for correctness
> and
> >> scalability. Having two protocols with different properties could
> justify
> >> the presence of two projects, but they both promise snapshot isolation
> so I
> >> suspect they will be doing very similar things.
> >>
> >> Overall, as I see it, it would be very unfair to reject the Omid
> proposal
> >> on the basis that Tephra was incubated a couple of weeks ago. I'd much
> >> rather see how the two communities evolve and have the mentors of the
> >> projects fostering collaboration and possibly a merge of the two
> projects
> >> before graduation. Why not think of a general transaction status oracle
> >> with different protocol implementations assuming it makes sense? I
> wouldn't
> >> like to see any of the two blocked upfront on the basis that they are in
> >> the same space, though. We could postpone this decision until graduation
> >> when we'll have more knowledge about the projects and the growth of the
> two
> >> communities.
> >>
> >> -Flavio
> >>
>  On 18 Mar 2016, at 23:19, Henry Saputra  >>> > wrote:
> >>>
> >>> I know Apache incubator does not play favorite but it is getting
> awkward
> >>> that TWO transaction engine for HBase coming to incubator at the same
> >> time.
> >>>
> >>> As most people know, the other one is Tephra, that just coming to
> >> incubator
> >>> few weeks ago.
> >>>
> >>> As member of IPMC, I would like to see Omid provide some more details
> >>> comparisons about the difference that the project bring,  in term of
> >>> approach and possible integrations with other ASF projects.
> >>>
> >>> If possible, I would prefer to see Omid team work together with Tephra
> to
> >>> work on working together to make one solid transaction engine for HBase
> >> and
> >>> later NoSQL databases.
> >>>
> >>>
> >>> - Henry
> >>>
>  On Thu, Mar 17, 2016 at 1:17 PM, Daniel Dai  >>> > wrote:
> >>>
>  Hi,
> 
>  I would like to propose Omid as an Apache Incubator project:
> 
>  https://wiki.apache.org/incubator/OmidProposal
> 
>  I've posted posted the text of the proposal below:
> 
>  Thanks,
> 

[DISCUSS] [PROPOSAL] Omid for Apache Incubator

[Daniel Dai]

Hi,

I would like to propose Omid as an Apache Incubator project:

https://wiki.apache.org/incubator/OmidProposal

I've posted posted the text of the proposal below:

Thanks,
Daniel

= Omid Proposal =

=== Abstract ===

Omid is a flexible, reliable, high performant and scalable ACID
transactional framework that allows client applications to execute
transactions on top of MVCC key/value-based NoSQL datastores
(currently Apache HBase) providing Snapshot Isolation guarantees on
the accessed data.


=== Proposal ===

Omid is a flexible open-source transactional framework that provides
ACID transactions with Snapshot Isolation guarantees on top of NoSQL
datastores. In particular, the current codebase brings the concept of
transactions to the popular Apache HBase datastore. Omid offers great
performance, it is highly available, and scalable. Omid's current
version is able to scale to thousands of clients triggering concurrent
transactions on application data stored in HBase. Omid can scale
beyond 100K transactions per second on mid-range hardware while
incurring in a minimal impact on the speed of data access in the
datastore. We’re currently experimenting with a prototype version that
can improve the performance up to ~380K TPS.


Omid has been publicly available as an open-source project in Github
under Apache License Version 2.0 since 2011 [1]. During these years,
it has generated certain interest in the open source community,
especially since the public presentation of the first version in
Hadoop Summit 2013 [2]. Currently the Github project has 241 Stars and
93 forks. Yahoo Inc. submits this proposal to the Apache Software
Foundation with the aim to transfer the Omid project -including its
source code and documentation- to Apache in order to start the build
of a stable open source community around it.


[1] https://github.com/yahoo/omid

[2] Omid presentation at Hadoop Summit 2013:
https://www.youtube.com/watch?v=Rhdmo9pVGgU=68=PLSAiKuajRe2luyqLU464Nxz4aQe7EPBus


=== Background ===

An Omid prototype was first released as an open-source project back in
2011. Inspired by Google Percolator [1], it offered a lock-free
approach to transactions in NoSQL datastores (See [2]). However,
during these years, the design of Omid has evolved significantly.
Whilst the current open-sourced version maintains many aspects of the
original implementation, it is the result of a major redesign of the
first prototype released in 2011.


Omid has now a more decentralized design that does not sacrifice the
consistency and performance of the original version. The current
design also enables Omid to scale to thousands of clients executing
transactions concurrently on application data stored in HBase.
Internally, Omid still utilizes a lock-free approach to support
multiple concurrent clients. Its design also relies on a centralized
conflict detection component, the TSO, which now resolves in an
efficient manner writeset collisions among concurrent transactions
without having to piggyback commit information to the clients. Another
important benefit of Omid is that it doesn't require any modification
of the underlying key-value datastore, HBase in this case. Moreover,
the recently added high availability algorithm allows to eliminate the
single point of failure represented by the TSO in those system
deployments requiring a higher degree of dependability. Last but not
least, the provided user API is very simple, mimicking transaction
managers in the relational world: begin, commit, rollback.


Omid is used internally at Yahoo. Sieve, Yahoo’s web-scale content
management platform powering some of next-generation search and
personalization products is using Omid as a transaction manager in its
processing pipeline. Sieve essentially acts as a huge processing hub
between content feeds and serving systems. It provides an environment
for highly customizable, real-time, streamed information processing,
with typical discovery-to-service latencies of just a few seconds. In
terms of scale and availability, Omid’s new design was largely driven
by Sieve’s requirements.


At Yahoo, we are also making an effort to disseminate the current
status of the project through blog entries (See [3], [4] and [5]) and
submissions to technical and academic conferences such as ATC 2016,
Hadoop Summit 2016, HBaseConf 2016. Last but not least, Omid also
appeared in a TechCrunch article in the last quarter of 2015 (See [6])


[1] D. Peng and F. Dabek, Large-scale Incremental Processing Using
Distributed Transactions and Notifications. USENIX Symposium on
Operating Systems Design and Implementation, 2010

[2] D. Gomez-Ferro, F. Junqueira, I. Kelly, B. Reed, and M. Yabandeh.
Omid: Lock-free transactional support for distributed data stores. In
Proc. of ICDE, 2013.

[3] 
http://yahoohadoop.tumblr.com/post/129089878751/introducing-omid-transaction-processing-for

[4] 
http://yahoohadoop.tumblr.com/post/132695603476/omid-architecture-and-protocol

[5] 

Re: Request for Administrator Wiki Access: ChrisNauroth

[Marvin Humphrey]

On Thu, Mar 17, 2016 at 8:07 PM, Chris Nauroth  wrote:
> Can I please be added to the incubator wiki Administrators group?  My wiki
> login is ChrisNauroth.  I would like administrator access so that I can
> field requests from contributors to get edit access on incubator proposals.

Done.

Marvin Humphrey

-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: Request for Wiki Edit Access: SiddharthAnand

[Marvin Humphrey]

On Thu, Mar 17, 2016 at 10:37 PM, Siddharth Anand
 wrote:
> Can I please be added to the incubator wiki Contributors group?  My wiki
> login is SiddharthAnand.  I would like to edit my Airflow Incubator
> Proposal (https://wiki.apache.org/incubator/AirflowProposal) in response to
> feedback.

Done.

Marvin Humphrey

-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

RE: [MARKETING] [Caution: Suspicious URL]: Re: [DISCUSS] [PROPOSAL] Omid for Apache Incubator

[Dor Ben Dov]

Andrew, 

Do you think Cloudera will include this new version in their bundle same as 
Horton ? 

Dor

-Original Message-
From: Andrew Purtell [mailto:andrew.purt...@gmail.com] 
Sent: שבת 19 מרץ 2016 22:59
To: general@incubator.apache.org
Subject: [MARKETING] [Caution: Suspicious URL]: Re: [DISCUSS] [PROPOSAL] Omid 
for Apache Incubator

Apache Phoenix just released version 4.7.0 with big news: transactions support, 
using Tephra. There's some interest in a successful Tephra incubation beyond 
the podling already. That said, that new code in Phoenix can be made pluggable 
to support more than one transaction oracle. Omid might be able to provide 
workable integration to stand in for Tephra. Collaboration between or even a 
joining of the two communities could be good but even if not as a potential 
downstream consumer it's good to have options! (provided the number of 
alternatives is bounded with reason of course). I think it would be good to see 
Omid get in. I think an Omid podling would find interested collaborators in the 
Phoenix and HBase communities right away. 


> On Mar 19, 2016, at 12:20 PM, Henry Saputra  wrote:
> 
> Thanks for the great explanation, Flavio.
> 
> As many have mentioned before, it is definitely ok to have similar 
> projects in ASF. We have prior acts before and I didn't expect 
> incubator to reject good projects coming in.
> 
> My intention was to avoid split of resources where both projects have 
> very similar goal and approach. But maybe both projects have different 
> subtle differences that worthy to be done as independent effort.
> 
> Just being devil advocate a bit to see if potential to collaborate.
> 
> - Henry
> 
>> On Saturday, March 19, 2016, Flavio Junqueira  wrote:
>> 
>> I understand the concern, so let me try to offer some facts and see 
>> if we can make progress from there.
>> 
>> Omid has been around for some time now, and its initial design 
>> appeared in a couple of research papers that I actually co-authored. 
>> The architecture is based on the idea of having a centralized 
>> transaction status oracle that shares transaction status data with 
>> clients for scalability. The current Omid project evolved out of that 
>> initial work and it is a much improved version over that first 
>> iteration, with the improvements focusing on scalability. It 
>> currently runs in production at scale at Yahoo! and there is interest 
>> from other companies according to the proposal. There is a series of blog 
>> posts about the experience in the project proposal.
>> 
>> Tephra has a very similar architecture. The description here says 
>> that it has a transaction server, which sounds like the TSO in the 
>> original Omid papers. I haven't spent enough time understanding the 
>> precise protocol they use, but I must say that the protocol is very 
>> important for correctness and scalability. Having two protocols with 
>> different properties could justify the presence of two projects, but 
>> they both promise snapshot isolation so I suspect they will be doing very 
>> similar things.
>> 
>> Overall, as I see it, it would be very unfair to reject the Omid 
>> proposal on the basis that Tephra was incubated a couple of weeks 
>> ago. I'd much rather see how the two communities evolve and have the 
>> mentors of the projects fostering collaboration and possibly a merge 
>> of the two projects before graduation. Why not think of a general 
>> transaction status oracle with different protocol implementations 
>> assuming it makes sense? I wouldn't like to see any of the two 
>> blocked upfront on the basis that they are in the same space, though. 
>> We could postpone this decision until graduation when we'll have more 
>> knowledge about the projects and the growth of the two communities.
>> 
>> -Flavio
>> 
 On 18 Mar 2016, at 23:19, Henry Saputra >> > wrote:
>>> 
>>> I know Apache incubator does not play favorite but it is getting 
>>> awkward that TWO transaction engine for HBase coming to incubator at 
>>> the same
>> time.
>>> 
>>> As most people know, the other one is Tephra, that just coming to
>> incubator
>>> few weeks ago.
>>> 
>>> As member of IPMC, I would like to see Omid provide some more 
>>> details comparisons about the difference that the project bring,  in 
>>> term of approach and possible integrations with other ASF projects.
>>> 
>>> If possible, I would prefer to see Omid team work together with 
>>> Tephra to work on working together to make one solid transaction 
>>> engine for HBase
>> and
>>> later NoSQL databases.
>>> 
>>> 
>>> - Henry
>>> 
 On Thu, Mar 17, 2016 at 1:17 PM, Daniel Dai >> > wrote:
>>> 
 Hi,
 
 I would like to propose Omid as an Apache Incubator project:
 
 https://wiki.apache.org/incubator/OmidProposal
 
 I've posted posted the text of the proposal below: