Re: [PROPOSAL] Heron

[William Markito Oliveira]

Howdy!

If Heron is looking for some help around incubation process, I'd love to
help while Geode experience is still fresh in my mind and given that it's a
project/space that I do have interest. Since I'm not an ASF member, I don't
think I can offer to be a mentor, but can probably still help and
participate on the process.

Thanks!

On Wed, Jun 14, 2017 at 7:54 PM, P. Taylor Goetz  wrote:

> Hi Bill/Supun,
>
> Sorry for not being a little more clear. I was asking more about how the
> Heron community would seek to engage with Storm community at the
> *community* level as opposed to the technical level (i.e. “Community over
> Code”).
>
> I’ve been asked by many why this has never happened, and have always
> struggled to answer. Maybe you could help answer that question as well as
> if and how that might change if Heron were to incubate.
>
> Another quick question: The proposal mentions Heron being used in
> production at Google, but some Google employees I recently spoke to seemed
> to contradict that. Could you explain? Note that’s nothing that would
> preclude the project from incubating, I’m just curious.
>
> -Taylor
>
> > On Jun 14, 2017, at 7:35 AM, Supun Kamburugamuve 
> wrote:
> >
> > Hi Taylor,
> >
> > For me, one of the interesting differences between Heron and Storm is the
> > execution model. Storm uses a shared memory model while Heron uses a
> > process based model. It will be interesting to see how these two evolve.
> >
> > Thanks,
> > Supun..
> >
> > On Mon, Jun 12, 2017 at 4:15 PM, Bill Graham 
> wrote:
> >
> >> Hi Taylor,
> >>
> >> Thanks for the mentor offer, we'd be glad to have your help.
> >>
> >> I think the best place for collaboration would be around the evolution
> of
> >> the API. In addition we plan to look more into DSL solutions which we
> could
> >> potentially collaborate on. This could be Trident, or Beam or something
> >> else, but there could be synergies for future development here.
> >>
> >> thanks,
> >> Bill
> >>
> >> On Fri, Jun 9, 2017 at 8:53 PM, P. Taylor Goetz 
> wrote:
> >>
> >>> Hi Bill,
> >>>
> >>> Could you comment on how/if the Heron community would be willing to
> work
> >>> with the Storm community? I've seen a number of new features in Storm
> >> being
> >>> ported to Heron, but I have yet to see any attempt by the Heron
> community
> >>> to engage with the Apache Storm community.
> >>>
> >>> I don't think it would be too far off to say that the relationship
> >> between
> >>> Heron and Apache Storm has been somewhat adversarial. The pre- and
> >>> post-open sourcing marketing around Heron seemed, at least to me,
> >> somewhat
> >>> aggressively negative toward Storm.
> >>>
> >>> As a peer to Apache Storm, how would the proposed "Apache Heron"
> >> community
> >>> work to collaborate with the Storm community? If Heron is adopting API
> >>> changes in Storm, then it seems there is an opportunity for
> >> collaboration.
> >>>
> >>> Don't take any of this as an objection to incubating the project. I
> would
> >>> support it. I would also be willing to be a mentor, if you would
> consider
> >>> taking on another.
> >>>
> >>> -Taylor
> >>>
>  On Jun 8, 2017, at 1:23 PM, Bill Graham  wrote:
> 
>  Dear Apache Incubator Community,
> 
>  We are excited to share our proposal for discussion and feedback
>  for entering Apache Incubation. Heron is a real-time, distributed,
>  fault-tolerant stream processing engine.
> 
>  Our proposal can be found at https://wiki.apache.org/
> >>> incubator/HeronProposal
>  and is included below.
> 
> 
>  Thank you,
> 
>  Bill Graham on behalf of the Heron developers
> 
> 
>  # Heron Proposal
> 
>  ## Abstract
>  Heron is a real-time, distributed, fault-tolerant stream processing
> >>> engine
>  initially developed by Twitter.
> 
>  ## Proposal
> 
>  Heron is a real-time stream processing engine built for high
> >> performance,
>  ease of manageability, performance predictability and developer
>  productivity[1]. We wish to develop a community around Heron to
> >> increase
>  contributions and see Heron thrive in an open forum.
> 
>  ## Background
> 
>  Heron provides the ability for developers to compose directed acyclic
>  graphs (DAGs) of real-time query execution logic (i.e. a topology) and
>  submit the topology to execute on a pluggable job scheduling system
> >>> (e.g.,
>  Apache Aurora, YARN, Marathon, etc). Users can employ either the
> native
>  Heron API or the Apache Storm API to develop the topology. Heron
> >> supports
>  the Storm API for ease of migration, but beyond that Heron’s
> >> architecture
>  differs considerably from Storm’s.
> 
>  Users submit a topology to the scheduler using the Heron client, which
> >>> uses
>  the Heron binary libraries to 

Re: [PROPOSAL] Heron

[P. Taylor Goetz]

Hi Bill/Supun,

Sorry for not being a little more clear. I was asking more about how the Heron 
community would seek to engage with Storm community at the *community* level as 
opposed to the technical level (i.e. “Community over Code”).

I’ve been asked by many why this has never happened, and have always struggled 
to answer. Maybe you could help answer that question as well as if and how that 
might change if Heron were to incubate.

Another quick question: The proposal mentions Heron being used in production at 
Google, but some Google employees I recently spoke to seemed to contradict 
that. Could you explain? Note that’s nothing that would preclude the project 
from incubating, I’m just curious.

-Taylor

> On Jun 14, 2017, at 7:35 AM, Supun Kamburugamuve  wrote:
> 
> Hi Taylor,
> 
> For me, one of the interesting differences between Heron and Storm is the
> execution model. Storm uses a shared memory model while Heron uses a
> process based model. It will be interesting to see how these two evolve.
> 
> Thanks,
> Supun..
> 
> On Mon, Jun 12, 2017 at 4:15 PM, Bill Graham  wrote:
> 
>> Hi Taylor,
>> 
>> Thanks for the mentor offer, we'd be glad to have your help.
>> 
>> I think the best place for collaboration would be around the evolution of
>> the API. In addition we plan to look more into DSL solutions which we could
>> potentially collaborate on. This could be Trident, or Beam or something
>> else, but there could be synergies for future development here.
>> 
>> thanks,
>> Bill
>> 
>> On Fri, Jun 9, 2017 at 8:53 PM, P. Taylor Goetz  wrote:
>> 
>>> Hi Bill,
>>> 
>>> Could you comment on how/if the Heron community would be willing to work
>>> with the Storm community? I've seen a number of new features in Storm
>> being
>>> ported to Heron, but I have yet to see any attempt by the Heron community
>>> to engage with the Apache Storm community.
>>> 
>>> I don't think it would be too far off to say that the relationship
>> between
>>> Heron and Apache Storm has been somewhat adversarial. The pre- and
>>> post-open sourcing marketing around Heron seemed, at least to me,
>> somewhat
>>> aggressively negative toward Storm.
>>> 
>>> As a peer to Apache Storm, how would the proposed "Apache Heron"
>> community
>>> work to collaborate with the Storm community? If Heron is adopting API
>>> changes in Storm, then it seems there is an opportunity for
>> collaboration.
>>> 
>>> Don't take any of this as an objection to incubating the project. I would
>>> support it. I would also be willing to be a mentor, if you would consider
>>> taking on another.
>>> 
>>> -Taylor
>>> 
 On Jun 8, 2017, at 1:23 PM, Bill Graham  wrote:
 
 Dear Apache Incubator Community,
 
 We are excited to share our proposal for discussion and feedback
 for entering Apache Incubation. Heron is a real-time, distributed,
 fault-tolerant stream processing engine.
 
 Our proposal can be found at https://wiki.apache.org/
>>> incubator/HeronProposal
 and is included below.
 
 
 Thank you,
 
 Bill Graham on behalf of the Heron developers
 
 
 # Heron Proposal
 
 ## Abstract
 Heron is a real-time, distributed, fault-tolerant stream processing
>>> engine
 initially developed by Twitter.
 
 ## Proposal
 
 Heron is a real-time stream processing engine built for high
>> performance,
 ease of manageability, performance predictability and developer
 productivity[1]. We wish to develop a community around Heron to
>> increase
 contributions and see Heron thrive in an open forum.
 
 ## Background
 
 Heron provides the ability for developers to compose directed acyclic
 graphs (DAGs) of real-time query execution logic (i.e. a topology) and
 submit the topology to execute on a pluggable job scheduling system
>>> (e.g.,
 Apache Aurora, YARN, Marathon, etc). Users can employ either the native
 Heron API or the Apache Storm API to develop the topology. Heron
>> supports
 the Storm API for ease of migration, but beyond that Heron’s
>> architecture
 differs considerably from Storm’s.
 
 Users submit a topology to the scheduler using the Heron client, which
>>> uses
 the Heron binary libraries to deploy all daemons required to run and
>>> manage
 the topology. The topology therefore has no reliance on centrally
>> managed
 Heron services, only on a generic job scheduling system, which lends
>>> itself
 well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN
>> (among
 others).
 
 The scheduler runs each topology as a job consisting of multiple
 containers. One of the containers runs the topology master, responsible
>>> for
 managing the topology. The remaining containers each runs a stream
>>> manager
 responsible for data routing, a metrics manager that collects and
>> 

Re: [DRAFT] What the new Status Pages potentially look like

[Dale LaBossiere]

John, can you confirm appropriate values for Edgent in order to clear up all 
the red under “Licensing”?

From http://incubator.apache.org/projects/edgent.html these seem to be the 
appropriate values (from that copyrights section as well as 1.0.0 release date 
of 2016-12-15):

:sga:2016-03-11
:asfCopyright:2016-12-15
:distributionRights:2016-12-15

Next question: how to update the info?  If it’s “a mentor must do it”, can you 
please handle it :-)

Thanks,
— Dale


-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: [PROPOSAL] Heron

[Supun Kamburugamuve]

Hi Taylor,

For me, one of the interesting differences between Heron and Storm is the
execution model. Storm uses a shared memory model while Heron uses a
process based model. It will be interesting to see how these two evolve.

Thanks,
Supun..

On Mon, Jun 12, 2017 at 4:15 PM, Bill Graham  wrote:

> Hi Taylor,
>
> Thanks for the mentor offer, we'd be glad to have your help.
>
> I think the best place for collaboration would be around the evolution of
> the API. In addition we plan to look more into DSL solutions which we could
> potentially collaborate on. This could be Trident, or Beam or something
> else, but there could be synergies for future development here.
>
> thanks,
> Bill
>
> On Fri, Jun 9, 2017 at 8:53 PM, P. Taylor Goetz  wrote:
>
> > Hi Bill,
> >
> > Could you comment on how/if the Heron community would be willing to work
> > with the Storm community? I've seen a number of new features in Storm
> being
> > ported to Heron, but I have yet to see any attempt by the Heron community
> > to engage with the Apache Storm community.
> >
> > I don't think it would be too far off to say that the relationship
> between
> > Heron and Apache Storm has been somewhat adversarial. The pre- and
> > post-open sourcing marketing around Heron seemed, at least to me,
> somewhat
> > aggressively negative toward Storm.
> >
> > As a peer to Apache Storm, how would the proposed "Apache Heron"
> community
> > work to collaborate with the Storm community? If Heron is adopting API
> > changes in Storm, then it seems there is an opportunity for
> collaboration.
> >
> > Don't take any of this as an objection to incubating the project. I would
> > support it. I would also be willing to be a mentor, if you would consider
> > taking on another.
> >
> > -Taylor
> >
> > > On Jun 8, 2017, at 1:23 PM, Bill Graham  wrote:
> > >
> > > Dear Apache Incubator Community,
> > >
> > > We are excited to share our proposal for discussion and feedback
> > > for entering Apache Incubation. Heron is a real-time, distributed,
> > > fault-tolerant stream processing engine.
> > >
> > > Our proposal can be found at https://wiki.apache.org/
> > incubator/HeronProposal
> > > and is included below.
> > >
> > >
> > > Thank you,
> > >
> > > Bill Graham on behalf of the Heron developers
> > >
> > >
> > > # Heron Proposal
> > >
> > > ## Abstract
> > > Heron is a real-time, distributed, fault-tolerant stream processing
> > engine
> > > initially developed by Twitter.
> > >
> > > ## Proposal
> > >
> > > Heron is a real-time stream processing engine built for high
> performance,
> > > ease of manageability, performance predictability and developer
> > > productivity[1]. We wish to develop a community around Heron to
> increase
> > > contributions and see Heron thrive in an open forum.
> > >
> > > ## Background
> > >
> > > Heron provides the ability for developers to compose directed acyclic
> > > graphs (DAGs) of real-time query execution logic (i.e. a topology) and
> > > submit the topology to execute on a pluggable job scheduling system
> > (e.g.,
> > > Apache Aurora, YARN, Marathon, etc). Users can employ either the native
> > > Heron API or the Apache Storm API to develop the topology. Heron
> supports
> > > the Storm API for ease of migration, but beyond that Heron’s
> architecture
> > > differs considerably from Storm’s.
> > >
> > > Users submit a topology to the scheduler using the Heron client, which
> > uses
> > > the Heron binary libraries to deploy all daemons required to run and
> > manage
> > > the topology. The topology therefore has no reliance on centrally
> managed
> > > Heron services, only on a generic job scheduling system, which lends
> > itself
> > > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN
> (among
> > > others).
> > >
> > > The scheduler runs each topology as a job consisting of multiple
> > > containers. One of the containers runs the topology master, responsible
> > for
> > > managing the topology. The remaining containers each runs a stream
> > manager
> > > responsible for data routing, a metrics manager that collects and
> reports
> > > various metrics and a number of processes called Heron instances which
> > run
> > > the user-defined logic on the stream of tuples. Parallelism is achieved
> > via
> > > process-based isolation of Heron instances, which provides predictable
> > > performance while simplifying debugging. The containers are allocated
> and
> > > managed by the scheduler framework based on resource availability of
> > nodes
> > > in the cluster. The metadata for the topology, such as the physical
> plan
> > > and execution details, are stored in the pluggable Heron State Manager
> > > (e.g. Apache ZooKeeper).
> > >
> > > ## Rationale
> > >
> > > Heron is a general-purpose, modular and extensible platform that can be
> > > leveraged to support common, real-time analytics use cases. There is an
> > > increasing demand for 

Re: [DRAFT] What the new Status Pages potentially look like

[Sam Ruby]

On Tue, Jun 13, 2017 at 9:04 PM, John D. Ament  wrote:
> Jim,
>
> For your specific cases, if all you want is to remove red text:
>
> - for :sga: enter the date in -MM-DD format of when the ASF Secretary
> acknowledged your SGA.  Based on the records I have its 2016-03-15
> - For the :asfCopyright: and :distributionRights: attributes, I would use
> 2017-01-20 as that's the day of the results of 2.8 which was the first
> release to have no issues during review.
>
> But please do continue to provide input, want to make sure we're building
> something useful, not a pain.

Suggestion: let's start talking about how we should make the roster
page prompt for, validate, and apply any changes.

As in, for every field that is editable, there should be a way to
bring up an HTML form.  That form should contain explanatory text, and
perform validations.  Submitting that form should update the
appropriate YAML file in svn, and potentially notify various lists of
the change.

If people can provide the information above for a single field, I can
implement that change for the first field and people can use that as
an example to do the same for other fields.

> HTH,
>
> John

- Sam Ruby

> On Tue, Jun 13, 2017 at 8:52 PM Jim Apple  wrote:
>
>> I'm confused. The message to podlings@ said "If you're not sure what
>> to fill out, feel free to reach out."
>>
>> It sounds like there is no set way yet to fill it out.
>>
>> What can I do to get the bolded, red, inaccurate scare messages off of
>> our whimsy page?
>>
>> On Tue, Jun 13, 2017 at 5:30 PM, John D. Ament 
>> wrote:
>> > Good points Dave.  To be honest, since this is a prototype I don't have
>> > answers any more than "If you think it should do that, we can make it do
>> > that"
>> >
>> > What I did have in mind:
>> >
>> > ---
>> > :issueTracker: jira|github|bugzilla
>> > :wiki: if using confluence, their space key
>> > :jira: if using jira, their issue key (though I suppose this can be
>> > multiple)
>> > :proposal: link to the proposal page.
>> > :asfCopyright: the date that we confirmed ASF copyright headers on the
>> > source code (e.g. the first time a source release passed without any
>> issues)
>> > :distributionRights: the date that we confirmed the resulting binary
>> > satisfied our release policies (e.g. the first time a release included a
>> > binary without any issues)
>> > :ipClearance: a link to the IP Clearance page for this podling.  I guess
>> > this can be multiple?
>> > :sga: date in which a SGA was received
>> > :website: http://impala.incubator.apache.org
>> > :graduationDate: to be filled in when the podling graduates
>> > :resolution: tlp|subproject , if subproject the TLP should be in here
>> > somehwere.  We may want to track as "sponsor."
>> >
>> > I can draft this up on a wiki page to gain more input.  What do you think
>> > about using https://wiki.apache.org/incubator/PodlingStatusBrainstorm to
>> > work through it?  I had a few more questions in line.
>> >
>> > On Tue, Jun 13, 2017 at 7:09 PM Dave Fisher 
>> wrote:
>> >
>> >> Hi John -
>> >>
>> >> These are excellent questions. I see each of these as being tuples of
>> >> various kinds.
>> >>
>> >> For example :sga: could have multiple grants each of which has a date
>> and
>> >> a file name to refer to in the foundation records.
>> >>
>> >>
>> > We don't usually expose the file name to people outside members, so I'm
>> not
>> > sure this would be useful.  Multiple dates are also not common.
>> >
>> >
>> >> The path to the source repository is missing completely.
>> >>
>> >
>> > Which source repository?  The podlings?  I have a prototype for gitbox
>> > podlings and think I can make it work for ASF git as well.  For SVN, we
>> can
>> > assume default or have a list of values.
>> >
>> >
>> >>
>> >> Some of these items could be transitory. For example distributionRights
>> >> may be confirmed but then something is added which negates those rights
>> >> until the issue is cleared up.
>> >>
>> >>
>> > +1
>> >
>> >
>> >> I wonder if it makes sense to provide links to email threads and/or Wiki
>> >> pages with the details?
>> >>
>> >> Regards,
>> >> Dave
>> >>
>> >> > On Jun 13, 2017, at 8:22 AM, Jim Apple  wrote:
>> >> >
>> >> > What format of data is expected in the fields? Impala's new status
>> >> > page SVN config file
>> >> > <
>> >>
>> https://svn.apache.org/repos/asf/incubator/public/trunk/content/podlings/impala.yml
>> >> >
>> >> > is:
>> >> >
>> >> > ---
>> >> > :issueTracker: jira
>> >> > :wiki: IMPALA
>> >> > :jira: IMPALA
>> >> > :proposal: http://wiki.apache.org/incubator/ImpalaProposal
>> >> > :asfCopyright:
>> >> > :distributionRights:
>> >> > :ipClearance:
>> >> > :sga:
>> >> > :website: http://impala.incubator.apache.org
>> >> > :graduationDate:
>> >> > :resolution:
>> >> >
>> >> > I do not know how to fill out asfCopyright, distributionRights,
>> >> > 

Re: [VOTE]: Release Apache RocketMQ 4.1.0(incubating)(RC1)

[John D. Ament]

+1 to release

On Tue, Jun 13, 2017 at 10:02 PM dongeforever <1018815...@qq.com> wrote:

> Hello Incubator PMC,
>
>
> The Apache RocketMQ community has voted and approved the proposal to
> release Apache RocketMQ 4.1.0 (incubating). We now kindly request the IPMC
> review and vote on this incubator release.
>
>
> Apache RocketMQ is a distributed messaging and streaming platform with low
> latency, high performance and reliability, trillion-level capacity and
> flexible scalability.
>
>
>
>
> [VOTE] Thread:
>
> https://lists.apache.org/thread.html/c81e7329d1e47425a50a11e863257a2798bf17c52e9573e0020c42d0@%3Cdev.rocketmq.apache.org%3E
>
>
> [RESULT][VOTE] Thread:
>
> https://lists.apache.org/thread.html/626e93ad0ed4c1df6a5ee5b7466b1ab4f4b10f3e6ec72cac11157368@%3Cdev.rocketmq.apache.org%3E
>
>
> The artifacts:
>
> https://dist.apache.org/repos/dist/dev/incubator/rocketmq/4.1.0-incubating-rc1/
>
>
> The staging repo:
> https://repository.apache.org/content/repositories/orgapacherocketmq-1005
>
>
> Git tag for the release:
>
> https://github.com/apache/incubator-rocketmq/tree/rocketmq-all-4.1.0-incubating
>
>
> Hash for the release tag:
> 308598865addb9a857d432c9607478642541093c
>
>
> Release Notes:
> https://rocketmq.apache.org/release-notes-4.1.0-incubating/
>
>
> The artifacts have been signed with Key : 133C87CA, which can be found in
> the keys file:
> https://dist.apache.org/repos/dist/dev/incubator/rocketmq/KEYS
>
>
>
>
>
>
> The vote will be open for at least 72 hours or until necessary number of
> votes are reached.
>
>
> Please vote accordingly:
>
>
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
>
>
> Thanks,
> The Apache RocketMQ Team