Re: New Committer/PMC Member: Erik Weathers

[Harsha]

Congrats Erik.
-Harsha

On Fri, Feb 23, 2018, at 11:15 AM, Karthick Duraisamy Soundararaj wrote:
> Whoa! Congratulations, Erik. Well deserved!
> 
> On Fri, Feb 23, 2018 at 10:37 AM, Erik Weathers <
> eweath...@groupon.com.invalid> wrote:
> 
> > Thanks everyone!  And yes Alexandre, the relation of my surname to the
> > project name was not lost on me. ;-)(Also, my Grandpa went by "Stormy"
> > too!)
> >
> > Also, I must thank my work teammates Srishty Agrawal and Jessica Hartog who
> > have contributed greatly to any of the work that I've pushed.
> >
> > - Erik
> >
> > On Fri, Feb 23, 2018 at 8:28 AM, Ethan Li <ethanopensou...@gmail.com>
> > wrote:
> >
> > > Congratulations! Erik
> > >
> > > - Ethan
> > >
> > > > On Feb 22, 2018, at 7:42 PM, Xin Wang <data.xinw...@gmail.com> wrote:
> > > >
> > > > Congrats!
> > > >
> > > > 2018-02-23 9:41 GMT+08:00 Hugo Da Cruz Louro <hlo...@hortonworks.com>:
> > > >
> > > >> Congrats & Welcome!
> > > >>
> > > >>> On Feb 22, 2018, at 2:45 PM, Jungtaek Lim <kabh...@gmail.com> wrote:
> > > >>>
> > > >>> Welcome Erik! Congrats!
> > > >>>
> > > >>> -Jungtaek Lim (HeartSaVioR)
> > > >>>
> > > >>> 2018년 2월 23일 (금) 오전 7:05, Stig Rohde Døssing <stigdoess...@gmail.com
> > > >님이
> > > >> 작성:
> > > >>>
> > > >>>> Congratulations Erik. Happy to see you join.
> > > >>>>
> > > >>>> 2018-02-22 20:53 GMT+01:00 Alexandre Vermeerbergen <
> > > >>>> avermeerber...@gmail.com
> > > >>>>> :
> > > >>>>
> > > >>>>> Hi,
> > > >>>>>
> > > >>>>> Welcome to Erik...
> > > >>>>> ... a Stormy Weather(s) sounds like a fantastic match indeed!
> > > >>>>>
> > > >>>>> Alexandre Vermeerbergen (Storm addict)
> > > >>>>>
> > > >>>>> 2018-02-22 20:49 GMT+01:00 P. Taylor Goetz <ptgo...@apache.org>:
> > > >>>>>> The Apache Storm PMC has voted to add Erik Weathers as a Committer
> > > and
> > > >>>>> PMC Member.
> > > >>>>>>
> > > >>>>>> Please join me in congratulating Erik on his new role!
> > > >>>>>>
> > > >>>>>> -Taylor
> > > >>>>>
> > > >>>>
> > > >>
> > > >>
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > > Xin
> > >
> > >
> >

Re: [DISCUSS] Replace storm-kafka-client on 1.1.x-branch / 1.0.x-branch with 1.x-branch

[Harsha]

+1 to replace storm-kafka-client in 1.0.x branch
-Harsha
On Tue, Jan 30, 2018, at 7:04 PM, Jungtaek Lim wrote:
> Bump up this thread so that we could reach consensus earlier. Given that we
> got concern related to this, I think it is ideal to release 1.1.x/1.0.x
> with making decision and applying the change if we want.
> 
> 2018년 1월 30일 (화) 오전 9:25, Jungtaek Lim <kabh...@gmail.com>님이 작성:
> 
> > Erik's concern brought from 1.0.6 RC1, because they can't use Storm 1.1.0
> > or higher (Storm 1.1.0 broke storm-mesos.). While he could take an
> > workaround to use storm-kafka-client 1.2.0 or 1.1.2 (if we decide to
> > replace) with Storm 1.0.6, it would be better if we don't allow leaving
> > storm-kafka-client in 1.0.x in inconsistent state.
> >
> > IMHO, breaking backward compatibility is worse, but leaving broken thing
> > is worst. Hence I'm +1 to replace all, with noticing that it may bring
> > backward incompatibility in release announce.
> >
> > -Jungtaek Lim (HeartSaVioR)
> >
> > 2018년 1월 30일 (화) 오전 4:49, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
> >
> >> As I mentioned else thread I’m open to this but would defer to community
> >> consensus.
> >>
> >> If there’s concern about doing this for 1.0.x, one option would be skip
> >> that version line and only apply it to 1.2.0 and 1.1.x.
> >>
> >> -Taylor
> >>
> >> > On Jan 29, 2018, at 12:12 AM, Jungtaek Lim <kabh...@gmail.com> wrote:
> >> >
> >> > Hi devs,
> >> >
> >> > This is initial post to separate out discussion topic from vote thread,
> >> and
> >> > continue discussing.
> >> >
> >> > Background of the topic:
> >> > 1. Only 1.x-branch of storm-kafka-client got stabilized. (relatively)
> >> > 2. We would avoid to port back patches to 1.1.x and 1.0.x because
> >> they're
> >> > diverged too much.
> >> >
> >> > Downside:
> >> > Backward compatibility might be broken for 1.1.x and 1.0.x. Not sure for
> >> > 1.1.x, but at least 1.0.x, since supported Kafka client version is
> >> > different, and if my memory is right, we already applied backward
> >> > incompatible change into storm-kafka-client 1.1.0.
> >> >
> >> > Please put your opinion regarding topic. You're encouraged to copy your
> >> > previous post in vote thread which helps to centralize opinions in
> >> current
> >> > thread.
> >> >
> >> > Thanks,
> >> > Jungtaek Lim (HeartSaVioR)
> >>
> >>

Re: [DISCUSS] Decouple Storm core and connectors

[Harsha]

Hi,
In general connectors are independent of Storm run-time for most 
parts. I.e if the APIs are not changed (storm-core or trident haven't changed 
in years except the package re-name). You can take the latest connector and run 
in storm 1.0 or higher. So the users doesn't need to upgrade their storm 
cluster just to get a latest connector upgrade. Which they might be doing it 
but by making the release separate and stating the minimum supported storm 
version for the connectors will help the users.
This makes it easier for the connectors to be released  independently of the 
core/run-time and makes it easy for them to be fixed and released more often. 
But moving them to Bahir or other external project will make it detached from 
Storm itself that it might not see any co-ordination as reviewers from storm  
will need to be aware of an external project. 
 My proposal would be 
1. Can we create a sub-project in git under Storm so we can move the connectors 
there and everything else related Storm applies there.
2.  Can we keep maintaining storm connectors within same repo but different 
release module for it .

This is a separate topic but can improve the release timelines if we have 
multiple release managers that are handling the maint release and also main 
release versions. Its good to have rotation of release managers from PMC so 
that everyone will understand the process and can spread the responsibilities. 
There are threads started before but don't think they are addressed or any 
action item is taken. We should start another thread to discuss this process as 
well.

Thanks,
Harsha

On Tue, Jan 30, 2018, at 9:49 AM, Hugo Da Cruz Louro wrote:
> I think that the bahir approach makes sense for connectors that don’t 
> fall into the "first class support” category. I am in favor of moving 
> such lower adoption connectors and have the interested communities 
> support them with the most suitable release cycle. Connectors that are 
> idle, such as some examples that Jungtaek gave, we should consider 
> removing them altogether, especially if they are so outdated that they 
> may not even work.
> 
> Mainstream connectors such as storm-kafka-client should be kept in the 
> Storm repo. For example, Flink keeps flink-connector-kafka-0.x in the 
> Flink repo.
> 
> I am in agreement with Jungtaek when he says: "fixing critical bugs in 
> storm-kafka-client should trigger release, instead of waiting for Storm 
> core to have some fixes to be worth to release”. Storm’s release cadence 
> is currently not very high and one can argue that Storm entirely could 
> benefit from more frequent releases. If it is sto rm-kafka-client 
> triggering those releases, so be it. Moving forward I do not expect the 
> storm-kafka-client connector to be subject to so many changes that it 
> would warrant its own release cycle.
> 
> I also would like to highlight that although storm-kafka-client has been 
> the center of this discussion, as it was mentioned in this 
> thread<https://goo.gl/VY7QTG>, storm-kafka-client has had a much less 
> rocky road to stability compared to for example storm-kafka. Therefore 
> it’s worth evaluating if the challenges that we have faced with storm-
> kafka-client have been out of norm for such an important and complex 
> feature, and if they warrant significant changes in how we do things.
> 
> Thanks,
> Hugo
> 
> On Jan 29, 2018, at 9:18 PM, Jungtaek Lim 
> <kabh...@gmail.com<mailto:kabh...@gmail.com>> wrote:
> 
> Let me add a proof of my opinion: major patch of storm-eventhubs hasn't
> been getting even a comment over 4 months.
> https://github.com/apache/storm/pull/2322
> 
> I'd rather want to discuss regarding discontinue supporting officially if
> we no longer interest of, or we don't have resource to support, or any
> valid reasons. If we agree on discontinue supporting officially, we can
> move out to other repo. and let it self maintained. It may be able to get
> attention and have enough contributors so that we feel better to get to
> Storm core Repository again, or it can be silently forgotten. It shouldn't
> affect Storm core repository at any case.
> 
> 2018년 1월 30일 (화) 오후 2:03, Jungtaek Lim <kabh...@gmail.com>님이 작성:
> 
> If we worry about breaking somethings along with our
> users/consumers/distributors, picking one of less used/updated connector as
> experiment makes more sense to me. It's OK if we want to pick one of most
> active and widely used connector intentionally to accelerate experiment.
> 
> Decoupling connectors and moving to other repo. like Bahir will make it
> clear who are having interest of which connectors. storm-eventhubs for
> example, major code contributions were done from MS developers. Now they
> are gone, and I don't know even storm-eventhubs 

Re: [Discussion]: Storm Improvemement Proposal (SIP) to discuss changes

[Harsha]

Trying to bring attention this again. 
We currently have few big feature PRs going on and there is considerable
discussion about the design and Implementation etcc. My intention of
starting SIP is to add these details before someone goes and writes up a
PR and everyone has to go through reading of design and sometimes those
docs are not clear and we end up having long discussion on the PRs which
should mainly about the code review itself.
We should at least start making this process mandatory for any new big
feature especially to the storm-core. I am less concerned about
connectors and other parts which should have least resistive path and
they are usually easy to review.
If the devs put their thoughts and design and goes through discussion
and get everyone on the same line when the PR shows up it will be less
surprising and everyone involved know how the PR/Code supposed to work. 

-Harsha

On Fri, Jun 9, 2017, at 09:16 PM, Harsha wrote:
> Arun,
>For big features we did follow design doc/review. Making it
>formal makes everyone to follow a process. 
> Again this process is not for bug fixes as we stated its about New
> Features/Config Changes/Public interface changes. I don't think it puts
> any extra effort for anyone who is writing detailed JIRA but by making
> it formal makes everyone to add these details in a centra process. Not
> everyone will look at mailing list but its easier to follow a wiki page.
>  We should atleast give it a try before we vote it out.
> 
> Roshan,
>  Adding connector should require a SIP as well and changing any
>  public interfaces should be a KIP. Intention here is we've
>  central place where everyone can follow in detail whats the
>  public interface/new feature changes went in. We've changed
>  KafkaSpout quite a bit and there is current discussion thats
>  going to change it , having this documented in a central place
>  will make it easy to follow and recording them in release notes
>  as well.
> 
> Taylor,
> We can't call it a too tedious process without even giving it a
> try. This has been followed to a greater success at kafka and
> also Flink started the process as well
> 
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> .
> If it actually proved to more of hindrance than helping the community we
> can move away from it.
> 
> " Kafka has somewhat of a reputation for setting potentially too high a
> bar. I'd rather not see that happen with this community."
> Sure. But it also depends on the community. Just because some community
> enforcing too high bar that doesn't mean we are trying to do it via this
> process. Again we always have option if we ever veer too far in the
> wrong direction to bring up and improve or remove this process.
> 
> We should also as a community strive to have better quality and I am
> hoping this will give us a chance to not only let users know what are
> changes coming in but also keep the dev list to have a chance and join
> the discussion.
> 
> -Harsha
> 
> On Jun 9, 2017, 7:18 PM -0700, Arun Iyer <ai...@hortonworks.com>, wrote:
> I am for documenting and upfront design reviews, but maybe we should
> keep it less formal and make it part of the JIRA to start with.
> 
> Do we have any upcoming features for which we would like to see a
> proposal? May be start with a couple of proposals
> and see it works out before making it formal.
> 
> 
> Thanks,
> Arun
> 
> 
> 
> 6/9/17, 6:49 PM, "P. Taylor Goetz" <ptgo...@gmail.com> wrote:
> 
> -0
> 
> The KIP process feels kind of heavy. I'd rather start with a lighter
> effort like improving JIRA submissions and pull requests (some pull
> requests/JIRAs, even from committers and PMC members, are woefully
> inadequate in terms of detail), and see how that works out.
> 
> I share Bobby's concern that doing so might raise the bar for
> contributions and potentially have a chilling effect. We don't want to
> scare away contributors. Kafka has somewhat of a reputation for setting
> potentially too high a bar. I'd rather not see that happen with this
> community.
> 
> I will say that I like the idea of proposals for big features, ideally
> before any coding even begins -- so that others have a chance to
> collaborate. But I'm hesitant to impose too much process, voting, etc.
> That could scare people off.
> 
> I think we should think carefully before going down this trail.
> 
> -Taylor
> 
> On Jun 9, 2017, at 8:57 PM, Priyank Shah <ps...@hortonworks.com> wrote:
> 
> +1 for SIPs including a new connector. The person writing SIP can
> provide details 

Re: [VOTE] Release Apache Storm 1.1.1 (rc2)

[Harsha]

+1. Deployed on a 3 node cluster and ran sample topologies.

-Harsha

On Sat, Jul 29, 2017, at 08:11 AM, Stig Rohde Døssing wrote:
> Unpacked the binary tar and built ExclamationTopology against Nexus
> staging.
> Started Nimbus, a supervisor and the UI daemon via storm.cmd.
> Submitted ExclamationTopology. There is still an issue with storm.cmd and
> daemonName as described in
> https://issues.apache.org/jira/browse/STORM-2659,
> but I don't think it's a critical issue. Some logging may be missing when
> using commands other than drpc, logviewer, nimbus, supervisor and ui.
> Raised https://issues.apache.org/jira/browse/STORM-2662.
> Verified changing log level, deactivate, reactivate and kill works.
> Built and submitted KafkaSpoutTopologyMainNamedTopics, running against a
> local Kafka instance. Had to submit it via storm.py instead of storm.cmd,
> because storm.cmd doesn't include extlib in the Java classpath. I think
> we
> should put https://github.com/apache/storm/pull/2165 on 1.x-branch as
> well,
> storm.cmd seems to have a lot of inconsistencies with storm.py that make
> it
> annoying to work with.
> 
> The issues I encountered were also present in 1.1.0, are Windows
> specific,
> and can be worked around by users by using storm.py instead. +1.
> 
> 2017-07-28 23:45 GMT+02:00 Bobby Evans <ev...@yahoo-inc.com.invalid>:
> 
> > +1 built from source (v1.1.1 41bfea87b1a002565333bd18a06d766af1ca3816)
> > Ran all of the unit tests and some manual tests.
> >
> >
> > - Bobby
> >
> >
> > On Thursday, July 27, 2017, 1:38:46 PM CDT, P. Taylor Goetz <
> > ptgo...@gmail.com> wrote:
> >
> > This is a call to vote on releasing Apache Storm 1.1.1 (rc2)
> >
> > Full list of changes in this release:
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> > plain;f=CHANGELOG.md;hb=41bfea87b1a002565333bd18a06d766af1ca3816
> >
> > The tag/commit to be voted upon is v1.1.1:
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=
> > 948ce7d63a31fae8c478785985d0ef79808e234e;hb=41bfea87b1a002565333bd18a06d76
> > 6af1ca3816
> >
> > The source archive being voted upon can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> > 1.1-rc2/apache-storm-1.1.1-src.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.1-rc2/
> >
> > The release artifacts are signed with the following key:
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> > plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >
> > The Nexus staging repository for this release is:
> >
> > https://repository.apache.org/content/repositories/orgapachestorm-1050
> >
> > Please vote on releasing this package as Apache Storm 1.1.1.
> >
> > When voting, please list the actions taken to verify the release.
> >
> > This vote will be open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache Storm 1.1.1
> > [ ]  0 No opinion
> > [ ] -1 Do not release this package because...
> >
> > Thanks to everyone who contributed to this release.
> >
> > -Taylor
> >

Re: [VOTE] Release Apache Storm 1.1.1 (rc1)

[Harsha]

Tested source code by running unit tests and installed binaries to run
example topologies. +1.
-Harsha

On Tue, Jul 25, 2017, at 10:08 AM, Bobby Evans wrote:
> I built from source (v1.1.1 88f0b8a45553ea960164fab18c736a5cdbae8e58) ran
> all of the unit tests and ran some manual tests.
> I am +1 on the release.
> 
> 
> - Bobby
> 
> 
> On Tuesday, July 25, 2017, 11:14:14 AM CDT, Kishorkumar Patil
> <kpa...@yahoo-inc.com.INVALID> wrote:
> 
> I built and ran some manual tests. 
> +1 to release this package. 
> 
> -Kishor
> 
> 
> On Tuesday, July 25, 2017, 12:08:32 PM EDT, Stig Rohde Døssing
> <stigdoess...@gmail.com> wrote:
> 
> Is it on purpose that this is cut from 1.x-branch and not 1.1.x-branch?
> 
> 2017-07-25 17:09 GMT+02:00 P. Taylor Goetz <ptgo...@gmail.com>:
> 
> > This is a call to vote on releasing Apache Storm 1.1.1 (rc1)
> >
> > Full list of changes in this release:
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> > plain;f=CHANGELOG.md;hb=88f0b8a45553ea960164fab18c736a5cdbae8e58
> >
> > The tag/commit to be voted upon is v1.1.1:
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=
> > 89bf57855806d84dba8d9b7fe6e76f9074a6aa19;hb=88f0b8a45553ea960164fab18c736a
> > 5cdbae8e58
> >
> > The source archive being voted upon can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> > 1.1-rc1/apache-storm-1.1.1-src.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.1-rc1/
> >
> > The release artifacts are signed with the following key:
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> > plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >
> > The Nexus staging repository for this release is:
> >
> > https://repository.apache.org/content/repositories/orgapachestorm-1049
> >
> > Please vote on releasing this package as Apache Storm 1.1.1.
> >
> > When voting, please list the actions taken to verify the release.
> >
> > This vote will be open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache Storm 1.1.1
> > [ ]  0 No opinion
> > [ ] -1 Do not release this package because...
> >
> > Thanks to everyone who contributed to this release.
> >
> > -Taylor

Re: [DISCUSS] [Storm-Kafka] - Maintenance, Branch Support, and Deprecation Plans for storm-kafka and storm-kaka-client

[Harsha]

+1 on moving away from storm-kafka for Storm 2.0. For existing users we
can provide any critical bug fixes and provide it as part of 1.x
releases. They can still use the existing 1.x storm-kafka against 2.0.
Since kafka itself is moving away from older APIs continuing two
versions of kafka connector doesnt’ make sense and honestly splits the
usage which doesn’t give us any feedback on new storm-kafka-client.
Thanks,
Harsha

On Wed, Jul 19, 2017, at 09:20 AM, Hugo Da Cruz Louro wrote:
> Hi,
> 
> The goal of this email is to summarize and unify the discussion started
> across several email threads (Storm 2.0
> Roadmap<http://search-hadoop.com/?project=Storm=%22%5BDISCUSS%5D+Storm+2.0+Roadmap%22>,
> 1.1.1 Release
> Planning<http://search-hadoop.com/m/Storm/8gnYyGagLDWv1qG?subj=Release+Planning+for+1+1+1+and+others+>,
> Lag
> Issues<http://search-hadoop.com/m/Storm/8gnYyLmjIjYr692?subj=Lag+issues+using+Storm+1+1+1+latest+build+with+StormKafkaClient+1+1+1+vs+old+StormKafka+spouts>)
> concerning the maintenance, branch support, and eventual deprecation of
> storm-kafka and storm-kafka-client.
> 
> It was proposed in an earlier
> discussion<http://search-hadoop.com/?project=Storm=%22%5BDISCUSS%5D+Storm+2.0+Roadmap%22>
> the plan to deprecate storm-kafka in prol of storm-kafka-client. To
> clarify, the idea is not to completely eliminate storm-kafka, but rather
> keep supporting it in the 1.x-branch, while removing it from master (i.e.
> Storm 2.0 onwards). That is, storm-kafka-client will then become the only
> Storm Kafka option available for Storm 2.0 onwards, given that we have
> enough confidence in its stability by the time of the Storm 2.0 release.
> 
> The main reason for this proposal is the fact that the Kafka community
> agreed<https://cwiki.apache.org/confluence/display/KAFKA/KIP-109:+Old+Consumer+Deprecation>
> to deprecate the old consumer APIs starting in version 0.10.2, and will
> remove them in the next major version (0.12). This implies that
> storm-kafka will not work for Kafka 0.12 onwards. Important features
> missing in the old Kafka consumer are: security, new message format, and
> fetching offsets based on time stamp (KIP-79).
> 
> In earlier discussions the Storm community has shown concerns about the
> performance and stability of the storm-kafka-client. Those concerns are
> valid and were mirrored by the Kafka community in their early deprecation
> discussions. I align with what was said in the Kafka
> discussion<http://search-hadoop.com/m/Kafka/uyzND1e4bUP1Rjq721>: the
> storm-kafka-client has bugs, but so does storm-kafka, and all the
> development is currently going into storm-kafka-client, which will be
> even more prevalent in face of Kafka discontinuing the old consumer
> API’s. The only way to stabilize a complex component such as
> storm-kafka-client is to test it extensively in all its variants, which
> inevitably comes from users using it. Furthermore, removing storm-kafka
> from Storm 2.0 does not prevent users from still referring to storm-kafka
> version 1.x in their topologies.
> 
> I did a quick analysis of the JIRA issues for storm-kafka and
> storm-kafka-client [1].  As of July 11 there are 22 open or in-progress
> bugs for storm-kafka (1 blocker) and 15 for storm-kafka-client.
> 
> The recent refactoring around manual partition assignment should solve a
> lot of edge case bugs that occurred during rebalance. There are also a
> few open pull requests for Trident  and fixing some internal state
> details such as maxUncommittedOffsets, topic compaction, etc.
> Nevertheless, there are several areas that need to be addressed to
> stabilize and improve storm-kafka-client. Similarly to what was done for
> Storm SQL I suggest that we create a wiki page where we can centralize
> some points of action such as:
> 
> Features / Stability
>  * Memory Footprint
>  * Retrial Mechanism
>  * Exactly once and at least once guarantees
>  * Kafka Lag
>  * Metrics
>  * Spout Internals (e.g. maxUncommittedOffsets, ack, emitted, failed,
>  ...)
>  * Autocommit mode
> 
> Performance.
>  * Run performance benchmarks
> 
> Integration Testing
> * Test for exactly once in non failure scenarios (e.g.
> activate/deactivate)
> * Test for at least once in failure scenarios
> * Test Trident guarantees
> 
> Unit Testing
>  * Identify unit test coverage and find a modular way to continually add
>  new tests
> 
> Trident
>   * Pull request<https://github.com/apache/storm/pull/2174> for review
> 
> API
>   * Investigate for gaps in API between storm-kafka and
>   storm-kafka-client.
>   * Can we discontinue the old API ?
> 
> Documentation
>   * Check for accuracy and completeness of documentation
>

Re: [DISCUSS] Storm 2.0 Roadmap

[Harsha]

+1 for above stated approach on releasing 1.2.0 with metrics 
-Harsha

On Tue, Jun 27, 2017, at 12:17 PM, P. Taylor Goetz wrote:
> The work on metrics is ready for a pull request to 1.x-branch from the
> feature branch. I’ve held off because we haven’t reached consensus on a
> path forward with the 1.x release lines .
> 
> I’d like to propose the following for the 1.x line:
> 
> 1. Create a branch for 1.2 so we have a branch to review the metrics
> stuff.
> 2. Release 1.1.1
> 3. Review/merge metrics work. Port metrics to master.
> 4. Release 1.2.0
> 5. Put the entire 1.x line into maintenance mode. Drop support for 1.0.x.
> (we would only support 1.2.x and 1.1.x which are very closely aligned).
> 
> Dropping support for 1.0.x line would eliminate the need to maintain one
> of the fairly heavily diverged branches. The 1.2.x and 1.1.x would be
> very closely aligned. I just up merged metrics_v2 against 1.x-branch
> after a while, and there were no conflicts.
> 
> That would give us a little more bandwidth to focus on 2.0 and needed bug
> fixes to the 1.x line like some of the issues raised with
> storm-kafka-client. We could even start releasing alpha/beta versions of
> 2.0 in parallel to the steps above.
> 
> Any thoughts on that approach?
> 
> -Taylor
> 
> 
> > On Jun 24, 2017, at 1:21 AM, Jungtaek Lim <kabh...@gmail.com> wrote:
> > 
> > Yes I prefer option 1, but it might depend on the progress of metrics V2.
> > If it can be done within predictable near future I'm OK to pick option 2,
> > but if not, we may be better to focus releasing 2.0.0 and make it really
> > happen.
> > 
> > Whichever we go, I feel it's time to track remaining work on Storm 2.0.0. I
> > found some bugs on master branch so filed issues, and we've remaining port
> > work (UI and logviewer). We've some other improvements target for 2.0.0:
> > worker redesign, beam integration, and so on, and we don't track its
> > progress at all. I don't think we should wait for features which progress
> > is not transparent (in other words we don't know when it will be finished).
> > 
> > - Jungtaek Lim (HeartSaVioR)
> > 
> > 2017년 6월 24일 (토) 오전 5:19, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
> > 
> >> Bobby/Jungtaek,
> >> 
> >> Are you saying you want to forego the 1.2 “metrics_v2” release and include
> >> it only in 2.0? (I ask because that work is already based on 1.x-branch,
> >> and forward-porting it to master is relatively simple.) I’d kind of like
> >> that work go out soon.
> >> 
> >> If we go with option 1, I would want to see a 2.0 release (even if it’s a
> >> “beta” or “preview) before putting the 1.x line into maintenance mode.
> >> 
> >> -Taylor
> >> 
> >>> On Jun 23, 2017, at 9:51 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID>
> >> wrote:
> >>> 
> >>> I see 2 ways to address this.
> >>> 1) We put the 1.x line into maintenance mode like with 0.10.  We don't
> >> backport anything except bug fixes.2) We backport a lot of the backwards
> >> compatible changes from 2.x to 1.x.
> >>> My personal preference is 1.  It makes it clear the direction we want to
> >> go in.  The biggest issue is that we probably would want to do a 2.x
> >> release sooner rather then later.  Even if we don't get all of the features
> >> that people want, if we just get a release out we can add in new features
> >> if they are backwards compatible, or we can create a 3.x line that would
> >> have the breaking changes in it.
> >>> 
> >>> - Bobby
> >>> 
> >>> 
> >>> On Thursday, June 22, 2017, 7:39:55 PM CDT, Jungtaek Lim <
> >> kabh...@gmail.com> wrote:
> >>> 
> >>> I'd like to bump this again instead of initiating new discussion thread.
> >>> 
> >>> I had having hard time to create and apply pull requests for both master
> >>> and 1.x-branch and that's really painful and sometimes blocker for me to
> >> do
> >>> merge step.
> >>> Two branches are heavily diverged more than between 0.10 and 1.0.0, even
> >>> IDE can't switch between the branch smoothly. We didn't even address
> >>> checkstyle issue yet, but after addressing, it could be "completely"
> >>> diverged. JDK version is another major issue, since the pull requests
> >>> targeted for master branch are not checked against JDK 7, and some of
> >> them
> >>> make some issues regarding JDK version while porting back.
> &

[Discussion]: Storm Improvemement Proposal (SIP) to discuss changes

[Harsha]

Arun,
   For big features we did follow design doc/review. Making it
   formal makes everyone to follow a process. 
Again this process is not for bug fixes as we stated its about New
Features/Config Changes/Public interface changes. I don't think it puts
any extra effort for anyone who is writing detailed JIRA but by making
it formal makes everyone to add these details in a centra process. Not
everyone will look at mailing list but its easier to follow a wiki page.
 We should atleast give it a try before we vote it out.

Roshan,
 Adding connector should require a SIP as well and changing any
 public interfaces should be a KIP. Intention here is we've
 central place where everyone can follow in detail whats the
 public interface/new feature changes went in. We've changed
 KafkaSpout quite a bit and there is current discussion thats
 going to change it , having this documented in a central place
 will make it easy to follow and recording them in release notes
 as well.

Taylor,
We can't call it a too tedious process without even giving it a
try. This has been followed to a greater success at kafka and
also Flink started the process as well

https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
.
If it actually proved to more of hindrance than helping the community we
can move away from it.

" Kafka has somewhat of a reputation for setting potentially too high a
bar. I'd rather not see that happen with this community."
Sure. But it also depends on the community. Just because some community
enforcing too high bar that doesn't mean we are trying to do it via this
process. Again we always have option if we ever veer too far in the
wrong direction to bring up and improve or remove this process.

We should also as a community strive to have better quality and I am
hoping this will give us a chance to not only let users know what are
changes coming in but also keep the dev list to have a chance and join
the discussion.

-Harsha

On Jun 9, 2017, 7:18 PM -0700, Arun Iyer <ai...@hortonworks.com>, wrote:
I am for documenting and upfront design reviews, but maybe we should
keep it less formal and make it part of the JIRA to start with.

Do we have any upcoming features for which we would like to see a
proposal? May be start with a couple of proposals
and see it works out before making it formal.


Thanks,
Arun



6/9/17, 6:49 PM, "P. Taylor Goetz" <ptgo...@gmail.com> wrote:

-0

The KIP process feels kind of heavy. I'd rather start with a lighter
effort like improving JIRA submissions and pull requests (some pull
requests/JIRAs, even from committers and PMC members, are woefully
inadequate in terms of detail), and see how that works out.

I share Bobby's concern that doing so might raise the bar for
contributions and potentially have a chilling effect. We don't want to
scare away contributors. Kafka has somewhat of a reputation for setting
potentially too high a bar. I'd rather not see that happen with this
community.

I will say that I like the idea of proposals for big features, ideally
before any coding even begins -- so that others have a chance to
collaborate. But I'm hesitant to impose too much process, voting, etc.
That could scare people off.

I think we should think carefully before going down this trail.

-Taylor

On Jun 9, 2017, at 8:57 PM, Priyank Shah <ps...@hortonworks.com> wrote:

+1 for SIPs including a new connector. The person writing SIP can
provide details about the external system for which connector is being
written to let others know why a certain design decision was made. This
will make it easy for reviewers.

On 6/9/17, 5:24 PM, "Satish Duggana" <satish.dugg...@gmail.com> wrote:

+1 for SIPs. It is so useful as mentioned by others in earlier mails. It
would be very useful for new contributors and others who are looking out
for a feature design and decisions taken etc.

Whenever a minor feature is added to a connector it may not need a
separate
SIP but the existing README can be updated with details for users. It
can
be discussed and decided apropos whether a SIP is really required for
any
enhancement which is not really big.


On Sat, Jun 10, 2017 at 5:13 AM, Roshan Naik <ros...@hortonworks.com>
wrote:

If I am looking at the Kafka site correctly, I see that Kafka has a
total
of 167 KIPs so far.
So I assume that minor new features would not be parrt of the SIP ?

Unlike Kafka, since Storm has a number of connectors (that keep
growing),
I am speculating the SIP process might get somewhat unwieldy if it were
to
track little changes in each of the connectors.

Also thinking that a SIP may not be needed to justify a new connector,
but
useful if we are replacing an old connector with a new one.

-roshan



On 6/9/17, 3:19 PM, "Harsha" <st...@harsha.io> wrote:

Hi Bobby,
In ge

Re: Use of the KafkaConsumer.subscribe API in the storm-kafka-client spout

[Harsha]

Dynamic assignment is what causing all the issues that we see now. 
1. Duplicates at the start of the KafkaSpout and upon any rebalance
2. Trident Kafka Spout not holding the transactional batches.
Many corner cases can easily produce duplicates.

There is no point in keeping dynamic assignment given all the issues
that are showing up.
Here is the excerpt from Kafka consumer docs
https://www-us.apache.org/dist/kafka/0.10.0.1/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
"If the process itself is highly available and will be restarted if it
fails (perhaps using a cluster management framework like YARN, Mesos, or
AWS facilities, or as part of a stream processing framework). In this
case there is no need for Kafka to detect the failure and reassign the
partition since the consuming process will be restarted on another
machine."

Manual assignment is the right way to go.

-Harsha

On Jun 9, 2017, 4:09 PM -0700, Hugo Da Cruz Louro
<hlo...@hortonworks.com>, wrote:
+1 for simplifying KafkaSpoutConfig. Too many constructors and too many
methods.. I am not sure it’s justifiable to have any methods that simply
set KafkaConsumer properties. All of these properties should just go in
a Map<String, Object>, which is what KafkaConsumer receives, and what
was supported in the initial implementation. The names of the properties
can be retrieved from org.apache.kafka.clients.consumer.ConsumerConfig.
At this point we may have to keep in mind backwards compatibility.

Not sure we should completely discontinue dynamic partition assignment,
as it is one of primary features of the new Storm Kafka Client API. With
this said, manual partition assignment should be supported and would
solve a lot of potential problems arising from dynamic partition
assignment.

Hugo

On Jun 9, 2017, at 3:33 PM, Harsha <st...@harsha.io> wrote:

I think question why we need all those settings when a user can pass it
via Properties with consumer properties defined or via Map conf object.
Having the methods on top of consumer config means every time Kafka
consumer property added or changed one needs add a builder method. We
need to get out of the way and let the user configure it like they do it
for typical Kafka Consumer instead we've 10s of methods that sets
properties for ConsumerConfig.

Examples:
https://github.com/apache/storm/blob/master/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java#L317

https://github.com/apache/storm/blob/master/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java#L309
etc.. all of these are specific to KafkaConsumer config, users should
be able to pass it via Properties all of these.

https://github.com/apache/storm/blob/master/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java#L327

whats the benefit of adding that method? and we are forcing that to set
the protocol to "SSL" in this method
https://github.com/apache/storm/blob/master/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java#L318

Users can set the ssl properties and then can select the
securityProtocol "SASL_SSL" which requires both kerberos and ssl configs
to be set. In above case making a call setSSLTruststore changes the
security.protocol to "SSL". This could easily run into issues if the
users sets securityProtocol first with "SASL_SSL" then later calls
setSSLTruststore which changes it to "SSL".

We are over-taking these settings instead of letting user to figure out
from Kafka consumer config page.

In contrast we've KafkaProducer which does this
https://github.com/apache/storm/blob/master/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/bolt/KafkaBolt.java#L121
. I would add Properties object instead of deriving it from topologyConf
but this is much more easier to understand for the users. The contract
here is put whatever the producer configs that users wants in the conf
object and we create producer out of that config.

Honestly these interfaces needs to be simple and let the user have
control instead of adding our interpretation.



Thanks,
Harsha
On Jun 9, 2017, 2:08 PM -0700, Stig Døssing <generalbas@gmail.com>,
wrote:
I'd be happy with a simpler KafkaSpoutConfig, but I think most of the
configuration parameters have good reasons for being there. Any examples
of
parameters you think we should remove?

2017-06-09 21:34 GMT+02:00 Harsha <st...@harsha.io>:

+1 on using the manual assignment for the reasons specified below. We
will see duplicates even in stable conditions which
is not good. I don’t see any reason not to switch to manual assignment.
While we are at it we should refactor the KafkaConfig part.
It should be as simple as accepting the kafka consumer config or
properties file and forwarding it to KafkaConsumer. We made
it overly complex and unnecessary.

Thanks,
Harsha

Re: Use of the KafkaConsumer.subscribe API in the storm-kafka-client spout

[Harsha]

I think question why we need all those settings when a user can pass it
via Properties with consumer properties defined or via Map conf object.
Having the methods on top of consumer config means every time Kafka
consumer property added or changed one needs add a builder method.  We
need to get out of the way and let the user configure it like they do it
for typical Kafka Consumer instead we've 10s of methods that sets
properties for ConsumerConfig. 

Examples:
https://github.com/apache/storm/blob/master/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java#L317

https://github.com/apache/storm/blob/master/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java#L309
etc.. all of these are specific to KafkaConsumer  config, users should
be able to pass it via Properties all of these.

https://github.com/apache/storm/blob/master/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java#L327

whats the benefit of adding that method? and we are forcing that to set
the protocol to "SSL" in this method
https://github.com/apache/storm/blob/master/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java#L318

Users can set the ssl properties and then can select the
securityProtocol "SASL_SSL" which requires both kerberos and ssl configs
to be set. In above case making a call setSSLTruststore changes the
security.protocol to "SSL". This could easily run into issues if the
users sets securityProtocol first with "SASL_SSL" then later calls
setSSLTruststore which changes it to "SSL".

We are over-taking these settings instead of letting user to figure out
from Kafka consumer config page.

In contrast we've KafkaProducer which does this
https://github.com/apache/storm/blob/master/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/bolt/KafkaBolt.java#L121
. I would add Properties object instead of deriving it from topologyConf
but this is much more easier to understand for the users. The contract
here is put whatever the producer configs that users wants in the conf
object and we create producer out of that config. 

Honestly these interfaces needs to be simple and let the user have
control instead of adding our interpretation. 



Thanks,
Harsha
On Jun 9, 2017, 2:08 PM -0700, Stig Døssing <generalbas@gmail.com>,
wrote:
I'd be happy with a simpler KafkaSpoutConfig, but I think most of the
configuration parameters have good reasons for being there. Any examples
of
parameters you think we should remove?

2017-06-09 21:34 GMT+02:00 Harsha <st...@harsha.io>:

+1 on using the manual assignment for the reasons specified below. We
will see duplicates even in stable conditions which
is not good. I don’t see any reason not to switch to manual assignment.
While we are at it we should refactor the KafkaConfig part.
It should be as simple as accepting the kafka consumer config or
properties file and forwarding it to KafkaConsumer. We made
it overly complex and unnecessary.

Thanks,
Harsha

Re: [Discussion]: Storm Improvemement Proposal (SIP) to discuss changes

[Harsha]

Hi Bobby,
   In general, a KIP is required for adding New features, config
   changes or backward-incompatible changes. Don't require
   adding a KIP for bug-fixes.  Devs who wants to add any
   features will write up a wiki which has JIRA link, mailing
   list discussion link and outline the Motivation, Public
   interface changes and protocol changes etc ..a good example
   here is
   
https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka.
 
They can start the discussion thread once its ready and once everyone
agrees its in a good shape, a Vote thread starts . Once there are
required votes are in one can start the PR process and get it merged in. 
   Each release we can collect what features/fixes especially to
   public interfaces that went in and roll it out in release
   notes. This will give a better idea for the users on what
   changed and added from previous version.
 We can only enforce this to new feature/config/backward
 incompatible change. Having this go through the discussion
 phase will give us the early feedback and potentially caught
 any issues before the implementation.
Thanks,
Harsha

On Fri, Jun 9, 2017 at 2:24 PM Bobby Evans <ev...@yahoo-inc.com.invalid>
wrote:

Can you please explain how KIP currently works and how you would
like to see something similar in storm?
If we make the process more formal we will probably have less people
contributing, but we will probably have better overall patches.  It
is a balancing act and having never used KIP I would like to
understand it better before going all in on it.
- Bobby


On Friday, June 9, 2017, 4:09:38 PM CDT, Stig Døssing
<generalbas@gmail.com> wrote:

This sounds like a good idea. KIPs seem to work well for Kafka. It's
easy
for discussions to get lost or just not seen on the mailing list.

2017-06-09 21:36 GMT+02:00 Harsha <st...@harsha.io>:

> Hi All,
>  We’ve seen good adoption of KIP approach in Kafka community
>  and would like to see we adopt the similar approach for storm
>  as well.
> Its hard to keep track of proposed changes and mailing list threads to
> know what all changes that are coming into  and what design/backward
> incompatible changes being approved.  It will be good to have this
> documented and go through discussion then Vote phase to get them
> approved before we merge the PRs. This will keep everyone informed of
> what changes happened even if they are not following the mailing list
> they can go to wiki to see the list of changes went into a release.
> Community overall will be well informed of the changes as well. Would
> like to hear your thoughts.
>
> Thanks,
> Harsha
>

[Discussion]: Storm Improvemement Proposal (SIP) to discuss changes

[Harsha]

Hi All,
  We’ve seen good adoption of KIP approach in Kafka community
  and would like to see we adopt the similar approach for storm
  as well.
Its hard to keep track of proposed changes and mailing list threads to
know what all changes that are coming into  and what design/backward
incompatible changes being approved.  It will be good to have this
documented and go through discussion then Vote phase to get them
approved before we merge the PRs. This will keep everyone informed of
what changes happened even if they are not following the mailing list
they can go to wiki to see the list of changes went into a release. 
Community overall will be well informed of the changes as well. Would
like to hear your thoughts.

Thanks,
Harsha

Re: Use of the KafkaConsumer.subscribe API in the storm-kafka-client spout

[Harsha]

+1 on using the manual assignment for the reasons specified below.  We
will see duplicates even in stable conditions which 
is not good. I don’t see any reason not to switch to manual assignment.
While we are at it we should refactor the KafkaConfig part.
It should be as simple as accepting the kafka consumer config or
properties file and forwarding it to KafkaConsumer. We made 
it overly complex and unnecessary.

Thanks,
Harsha

Re: Recording - Storm & Kafka Meetup on April 20th 2017

[Harsha Chintalapani]

Hi Aditya,
 Thanks for your interest. We entatively planning one in June
1st week. If you haven't already please register here
https://www.meetup.com/Apache-Storm-Apache-Kafka/ . I'll keep the Storm
lists updated once we finalize the date & location.

Thanks,
Harsha

On Mon, Apr 24, 2017 at 7:02 AM Aditya Desai <adity...@usc.edu> wrote:

> Hello Everyone
>
> Can you please let us know when is the next meet up? It would be great if
> we can have in May.
>
> Regards
> Aditya Desai
>
> On Mon, Apr 24, 2017 at 2:16 AM, Xin Wang <data.xinw...@gmail.com> wrote:
>
>> How about publishing this to Storm site?
>>
>>  - Xin
>>
>> 2017-04-22 19:27 GMT+08:00 steve tueno <stuenofo...@gmail.com>:
>>
>>> great
>>>
>>> Thanks
>>>
>>>
>>>
>>> Cordialement,
>>>
>>> TUENO FOTSO Steve Jeffrey
>>> Ingénieur de conception
>>> Génie Informatique
>>> +237 676 57 17 28 <+237%206%2076%2057%2017%2028>
>>> +237 697 86 36 38 <+237%206%2097%2086%2036%2038>
>>>
>>> +33 6 23 71 91 52 <+33%206%2023%2071%2091%2052>
>>>
>>>
>>> https://jobs.jumia.cm/fr/candidats/CVTF1486563.html
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__jobs.jumia.cm_fr_candidats_CVTF1486563.html=DwMFaQ=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI=aLfk1zsmx4LG4nTElFRiaw=9sHUKQRDjS9DG9IvEZNtjgCtreJjfZbupLo3QX9wBRE=GaQrzR02LWZVUrE0QJsdVvmFqqVLTgaJw8mahNlm9As=>
>>>
>>> __
>>>
>>> https://play.google.com/store/apps/details?id=com.polytech.remotecomputer
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__play.google.com_store_apps_details-3Fid-3Dcom.polytech.remotecomputer=DwMFaQ=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI=aLfk1zsmx4LG4nTElFRiaw=9sHUKQRDjS9DG9IvEZNtjgCtreJjfZbupLo3QX9wBRE=qQphaS7iehggUZXFCjfv7qaAa7b-bv0YPNM3hGRl6WM=>
>>>
>>> https://play.google.com/store/apps/details?id=com.polytech.internetaccesschecker
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__play.google.com_store_apps_details-3Fid-3Dcom.polytech.internetaccesschecker=DwMFaQ=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI=aLfk1zsmx4LG4nTElFRiaw=9sHUKQRDjS9DG9IvEZNtjgCtreJjfZbupLo3QX9wBRE=a48dJmbXEEcCNsVdX-mqdEN_WcfSDYV6VWOVwrO_7Vs=>
>>> *http://www.traveler.cm/
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__remotecomputer.traveler.cm_=DwMFaQ=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI=aLfk1zsmx4LG4nTElFRiaw=9sHUKQRDjS9DG9IvEZNtjgCtreJjfZbupLo3QX9wBRE=fdcEd-WWb2Y9Dc_EENRO1ctKcdCM0Aetcq-VUNkSWWo=>*
>>> http://remotecomputer.traveler.cm/
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__remotecomputer.traveler.cm_=DwMFaQ=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI=aLfk1zsmx4LG4nTElFRiaw=9sHUKQRDjS9DG9IvEZNtjgCtreJjfZbupLo3QX9wBRE=fdcEd-WWb2Y9Dc_EENRO1ctKcdCM0Aetcq-VUNkSWWo=>
>>>
>>> https://play.google.com/store/apps/details?id=com.polytech.androidsmssender
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__play.google.com_store_apps_details-3Fid-3Dcom.polytech.androidsmssender=DwMFaQ=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI=aLfk1zsmx4LG4nTElFRiaw=9sHUKQRDjS9DG9IvEZNtjgCtreJjfZbupLo3QX9wBRE=cbngecoo97iCdOv1b8AFOFW3SXRaRsikcY_xW1LA9RM=>
>>>
>>> https://github.com/stuenofotso/notre-jargon
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_stuenofotso_notre-2Djargon=DwMFaQ=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI=aLfk1zsmx4LG4nTElFRiaw=9sHUKQRDjS9DG9IvEZNtjgCtreJjfZbupLo3QX9wBRE=js0gVdd_GhgUEiUmzI7tbceTxMw-C6a-_j2ZYS-9sCE=>
>>> https://play.google.com/store/apps/details?id=com.polytech.welovecameroon
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__play.google.com_store_apps_details-3Fid-3Dcom.polytech.welovecameroon=DwMFaQ=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI=aLfk1zsmx4LG4nTElFRiaw=9sHUKQRDjS9DG9IvEZNtjgCtreJjfZbupLo3QX9wBRE=Mg-q0FcZEaENh5tpdSndEJNsedpAjzueRxBGAc3Srhs=>
>>> https://play.google.com/store/apps/details?id=com.polytech.welovefrance
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__play.google.com_store_apps_details-3Fid-3Dcom.polytech.welovefrance=DwMFaQ=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI=aLfk1zsmx4LG4nTElFRiaw=9sHUKQRDjS9DG9IvEZNtjgCtreJjfZbupLo3QX9wBRE=8aYw4IRqwbFe6WN7ghOTkDJRT51Keti9f_-JbJ09jNg=>
>>>
>>>
>>>
>>> 2017-04-22 3:08 GMT+02:00 Roshan Naik <ros...@hortonworks.com>:
>>>
>>>> It was a great meetup and for the benefit of those interested but
>>>> unable to attend it, here is a link to the 

Storm Meetup in BayArea on April 20th

[Harsha Chintalapani]

Hi All,
 We are organizing a Storm Meetup at Hortonworks HQ in Santa
Clara,CA. If you are interested in attending please RSVP here
https://www.meetup.com/Apache-Storm-Apache-Kafka/events/238975416/

Thanks,
Harsha

Re: [DISCUSS] Move non-connectors modules to out of external

[Harsha Chintalapani]

We should this in next release of 1.x or 2.0. I am +1 on continue with
current release.
-Harsha
On Fri, Mar 24, 2017 at 8:53 PM P. Taylor Goetz <ptgo...@gmail.com> wrote:

> The question remains if we want to do this in the 1.1.0 release, or later.
>
> If it's the 1.1.0 release we need to make the changes and cut another RC.
> I'm fine with that, but want to make sure we have consensus before going
> down that road.
>
> -Taylor
>
> > On Mar 24, 2017, at 10:57 PM, Harsha Chintalapani <st...@harsha.io>
> wrote:
> >
> > Agree on change like this would be confusing to the users. Lets keep the
> > original plan of moving non-connectors modules of external instead of
> > introducing new changes
> > that are not in scope of this discussion.
> > My +1 still stands on keeping the current external/storm-* in place and
> > move just sql and storm-perf into top-level. We can have discussion for
> > storm 2.0 if we want to do
> > more changes.
> >
> > -Harsha
> >
> >> On Fri, Mar 24, 2017 at 4:31 PM P. Taylor Goetz <ptgo...@gmail.com>
> wrote:
> >>
> >> If we decide to change the structure of the distribution like this, I
> >> think we should do it in masrwe/2.0. If we want this for 1.1.0 we need
> to
> >> cut a new release candidate.
> >>
> >> Changing the structure of the distribution file structure can be
> >> disruptive for users. Even the change to no longer include connector
> >> binaries, as we've learned, will be a headache for some users.
> >>
> >> IMHO, from an ops perspective, changes like this should be handled like
> >> API changes.
> >>
> >> -Taylor
> >>
> >>> On Mar 24, 2017, at 4:07 PM, Hugo Da Cruz Louro <
> hlo...@hortonworks.com>
> >> wrote:
> >>>
> >>> Another possibility is to keep the ‘external' module, and create sub
> >> modules under it. The legacy structure would remain intact, while making
> >> things more modular. An idea would be:
> >>>
> >>> + external
> >>>+ connectors
> >>>+ tools
> >>>+ monitoring
> >>>+ etc
> >>>
> >>> Hugo
> >>>
> >>>> On Mar 24, 2017, at 12:34 PM, P. Taylor Goetz <ptgo...@gmail.com>
> >> wrote:
> >>>>
> >>>> For the background on why “external” was selected, you have to go back
> >> to a lengthy discussion in Feb. 2014.
> >>>>
> >>>> Here’s the start of the thread:
> >>>>
> >>>>
> >>
> http://mail-archives.apache.org/mod_mbox/storm-dev/201402.mbox/%3cee2bd0e2-254c-47af-8a53-257db7f05...@gmail.com%3e
> >> <
> >>
> http://mail-archives.apache.org/mod_mbox/storm-dev/201402.mbox/%3cee2bd0e2-254c-47af-8a53-257db7f05...@gmail.com%3E
> >>>
> >>>>
> >>>> It continues into March:
> >>>>
> >>>>
> >>
> http://mail-archives.apache.org/mod_mbox/storm-dev/201403.mbox/%3ccadimvzum1d3om30zayqq4xxe1vjbn7fumqcsgu+524oqgec...@mail.gmail.com%3e
> >>>>
> >>>> I’m -1 on renaming “external”. That’s the name chosen by the community
> >> and it has been the norm for 3 years. Changing it would likely confuse
> >> users.
> >>>>
> >>>> One of the ideas behind “external” was that it would contain
> components
> >> that were not essential to running storm. That line has recently blurred
> >> with some non-connector code sneaking in, so I’m okay with moving
> >> non-connector code out of external. Another point in that thread was a
> >> desire to avoid cluttering up the root directory, so we need to be
> careful
> >> about what the destination for those components is.
> >>>>
> >>>> -Taylor
> >>>>
> >>>>> On Mar 24, 2017, at 3:11 PM, Hugo Da Cruz Louro <
> >> hlo...@hortonworks.com> wrote:
> >>>>>
> >>>>> +1 non-connectors to top level
> >>>>> +1 to renaming external to connectors
> >>>>>
> >>>>> As for storm-kaka, if we are already touching the external modules,
> >> all the modules should be a submodule of a parent module called
> >> storm-kafka. I don’t think we should have 3 parent modules as we
> currently
> >> have (storm-kafka, storm-kafka-client, storm-kafka-monitor)
> >>>>>
> >>>>> The structure should be something along the lines (I 

Re: [DISCUSS] Move non-connectors modules to out of external

[Harsha Chintalapani]

Agree on change like this would be confusing to the users. Lets keep the
original plan of moving non-connectors modules of external instead of
introducing new changes
that are not in scope of this discussion.
My +1 still stands on keeping the current external/storm-* in place and
move just sql and storm-perf into top-level. We can have discussion for
storm 2.0 if we want to do
more changes.

-Harsha

On Fri, Mar 24, 2017 at 4:31 PM P. Taylor Goetz <ptgo...@gmail.com> wrote:

> If we decide to change the structure of the distribution like this, I
> think we should do it in masrwe/2.0. If we want this for 1.1.0 we need to
> cut a new release candidate.
>
> Changing the structure of the distribution file structure can be
> disruptive for users. Even the change to no longer include connector
> binaries, as we've learned, will be a headache for some users.
>
> IMHO, from an ops perspective, changes like this should be handled like
> API changes.
>
> -Taylor
>
> > On Mar 24, 2017, at 4:07 PM, Hugo Da Cruz Louro <hlo...@hortonworks.com>
> wrote:
> >
> > Another possibility is to keep the ‘external' module, and create sub
> modules under it. The legacy structure would remain intact, while making
> things more modular. An idea would be:
> >
> > + external
> > + connectors
> > + tools
> > + monitoring
> > + etc
> >
> > Hugo
> >
> >> On Mar 24, 2017, at 12:34 PM, P. Taylor Goetz <ptgo...@gmail.com>
> wrote:
> >>
> >> For the background on why “external” was selected, you have to go back
> to a lengthy discussion in Feb. 2014.
> >>
> >> Here’s the start of the thread:
> >>
> >>
> http://mail-archives.apache.org/mod_mbox/storm-dev/201402.mbox/%3cee2bd0e2-254c-47af-8a53-257db7f05...@gmail.com%3e
> <
> http://mail-archives.apache.org/mod_mbox/storm-dev/201402.mbox/%3cee2bd0e2-254c-47af-8a53-257db7f05...@gmail.com%3E
> >
> >>
> >> It continues into March:
> >>
> >>
> http://mail-archives.apache.org/mod_mbox/storm-dev/201403.mbox/%3ccadimvzum1d3om30zayqq4xxe1vjbn7fumqcsgu+524oqgec...@mail.gmail.com%3e
> >>
> >> I’m -1 on renaming “external”. That’s the name chosen by the community
> and it has been the norm for 3 years. Changing it would likely confuse
> users.
> >>
> >> One of the ideas behind “external” was that it would contain components
> that were not essential to running storm. That line has recently blurred
> with some non-connector code sneaking in, so I’m okay with moving
> non-connector code out of external. Another point in that thread was a
> desire to avoid cluttering up the root directory, so we need to be careful
> about what the destination for those components is.
> >>
> >> -Taylor
> >>
> >>> On Mar 24, 2017, at 3:11 PM, Hugo Da Cruz Louro <
> hlo...@hortonworks.com> wrote:
> >>>
> >>> +1 non-connectors to top level
> >>> +1 to renaming external to connectors
> >>>
> >>> As for storm-kaka, if we are already touching the external modules,
> all the modules should be a submodule of a parent module called
> storm-kafka. I don’t think we should have 3 parent modules as we currently
> have (storm-kafka, storm-kafka-client, storm-kafka-monitor)
> >>>
> >>> The structure should be something along the lines (I don’t mean the
> exact names;  we should find better ones. storm-kafka and
> storm-kafka-client are not very self explanatory in my opinion)
> >>>
> >>> + storm-kafka
> >>> + monitoring
> >>> + new-client
> >>> + old-client
> >>>
> >>> If we have to create new modules or submodules (e.g. under utils) so
> be it. The code should be in a module that is named after what its doing.
> >>>
> >>> Hugo
> >>>
> >>>> On Mar 24, 2017, at 11:15 AM, Priyank Shah <ps...@hortonworks.com>
> wrote:
> >>>>
> >>>> +1 to moving non-conncectors to top level. I think we should keep
> stom-kafka-monitor under external or connectors(after renaming).
> >>>>
> >>>> Jungtaek, just to clarify on what you said regarding storm core
> referencing storm-kafka-monitor. Like you said its just calling the script
> from ui jvm. There is no dependency in terms of class files needed to run
> the script from ui. The script itself adds a –cp argument and all it needs
> is storm-kafka-monitor jar in classpath. As far as packaging the script is
> concerned we can do what Satish suggested. i.e. move it to
> storm-kafka-monitor in source and

Re: [DISCUSS] Move non-connectors modules to out of external

[Harsha Chintalapani]

+1 on moving non-connectors to top-level like sql and storm-perf.
Regarding storm-kafka-monitor we can move this into "util" folder or keep
in the external.
-Harsha

On Fri, Mar 24, 2017 at 2:23 AM Satish Duggana <satish.dugg...@gmail.com>
wrote:

> storm-kafka-monitor is not a connector by itself but it is related to kafka
> connectors. So, any utility related to that connector should be part of
> that connector module(can be a submodule) instead of a top level module.
> core/ui uses this utility referring directly in a hacky way, which we may
> want to fix later. storm-kafka-monitor script exists in bin directory which
> can be moved to storm-kafka-monitor module and the same script can be
> packaged as part of storm/bin directory while packaging the distribution.
>
> Thanks,
> ~Satish.
>
> On Fri, Mar 24, 2017 at 1:07 PM, Jungtaek Lim <kabh...@gmail.com> wrote:
>
> > storm-kafka-monitor is referred by storm-core, though it's referenced via
> > executing command. Yes it's a bit odd to place it as top directory, but
> > it's not a connector for that reason too. Neither is ideal for me, so
> > ironically, either is fine.
> >
> > - Jungtaek Lim (HeartSaVioR)
> >
> > 2017년 3월 24일 (금) 오후 4:19, Satish Duggana <satish.dugg...@gmail.com>님이
> 작성:
> >
> > > +1 except for storm-kafka-monitor module as this utility is more about
> > > querying topic/partition offsets of kafka spouts in a topology. Do not
> we
> > > want to push this module into connectors/kafka as a submodule along
> with
> > > other submodules including old/new kafka spout modules?
> > >
> > > Thanks,
> > > Satish.
> > >
> > >
> > > On Fri, Mar 24, 2017 at 12:10 PM, Arun Iyer <ai...@hortonworks.com>
> > wrote:
> > >
> > > > +1
> > > >
> > > > Makes sense to move the non-connectors to top level and keep only the
> > > > connectors under “connectors” folder.
> > > >
> > > >
> > > > On 3/24/17, 12:00 PM, "Jungtaek Lim" <kabh...@gmail.com> wrote:
> > > >
> > > > >(Sent this yesterday but can't find this from storm-dev mbox...
> > sending
> > > it
> > > > >again)
> > > > >
> > > > >Hi dev,
> > > > >
> > > > >I'd like to start discussion regarding moving non-connectors modules
> > out
> > > > of
> > > > >external, maybe top directory.
> > > > >
> > > > >"external" directory has non-connectors (SQL, Flux,
> > storm-kafka-monitor,
> > > > >storm-submit-tools), and except Flux, others should be placed to the
> > > > binary
> > > > >dist. since Storm itself (not from user topology) needs to refer
> them.
> > > > >
> > > > >They're actually tied to the core of Storm, so I feel that it would
> be
> > > > >better to treat them (including Flux) as non-external, maybe same
> > level
> > > as
> > > > >storm-core.
> > > > >(I'm not sure what "external" actually means for Storm project btw.)
> > > > >
> > > > >In addition, after doing that I'd like to change the directory name
> > > > >"external" to "connector" or so, so that the name could be
> > > self-describing
> > > > >and we can only place connectors to that directory.
> > > > >(I know it would be painful for already opened pull requests, so no
> > > strong
> > > > >opinion regarding this.)
> > > > >
> > > > >Looking forward to your opinion!
> > > > >
> > > > >Thanks,
> > > > >Jungtaek Lim (HeartSaVioR)
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Storm 2.0 Roadmap

[Harsha Chintalapani]

Storm 2.0 migration to java in itself is a big win and would attract wider
community and adoption. So my vote would be to resolve the first 3 items to
get a release out.
All the other featured mentioned are great to have but shouldn't be
blockers for 2.0 release.

-Harsha

On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <ptgo...@gmail.com> wrote:

> With the 1.1.0 release nearing completion, I’d like to turn our attention
> to 2.0 and develop a plan for what features, etc. to include.
>
> The following 3 are what I feel are the minimum for a 2.0 release. These
> could likely be resolved relatively quickly:
>
> * Performance — I’ve not benchmarked the master branch vs. 1.0.x or 1.1.x
> in a while, but I feel it will be important to make sure there are no
> performance regressions, and would hope that we actually have a performance
> improvement over previous versions. To that end (e.g. if there is in fact a
> performance regression), the proposals that Roshan Naik put together for
> revising the threading and execution model (STORM-2307) and replacing
> Disruptor with JCTools (STORM-2306) warrant review and consideration. See
> also STORM-2284 which is the parent JIRA.
>
> * Finish porting Storm UI to java (STORM-1311)
>
> * Finish porting log viewer to java (STORM-1280)
>
> The following are items that are nice to have in 2.0, but I don’t feel are
> absolutely necessary for an initial 2.0 release:
>
> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it because it’s
> relevant) — Initially there seemed to be a lot of interest in this, but
> that seems to have trailed off. I spoke with some Beam developers and there
> seems to be interest from that community as well. Do we want to move that
> effort to the Beam community, or keep it here? Moving it to the Beam
> community might lead to better collaboration between projects.
>
> * Bounded Spouts (needed for Beam Runner implementation) — Currently
> spouts are unbounded, there no end to the stream. Beam has the concept of
> bounded sources (roughly analogous to batch processing). To support that,
> we would need to implement a similar concept in Storm. One benefit of such
> a feature would be the ability to handle both bounded and unbounded
> workflows in Storm.
>
> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers behind this
> effort. What improvements do you envision for 2.0?
>
> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been targeting this
> for 1.2.0, but it’s designed to be easily portable to master/2.0.
>
> * JStorm Migration — Original outline can be found here [1]. Note a lot of
> the associated JIRAs below are assigned, but there hasn’t been any recent
> activity or pull requests, we should probably consider them unassigned and
> up for grabs.:
>
> * Worker Classloader Isolation (STORM-1338) — Lack of this has been the
> bane of a lot of Storm users almost since day one. We have largely
> addressed it by shading/relocating dependencies. It would be great to see
> this addressed once and for all.
>
> * JStorm back pressure implementation (STORM-1324) — The current back
> pressure implementation leaves a bit to be desired, and the JStorm approach
> looks promising, though it also depends on the JStorm concept of “topology
> master” (STORM-1323), which may have some implications regarding security.
>
> * Dynamic Topology Updates (STORM-1335) — This would provide a command to
> update topology jars and configuration without stopping the topology, and
> is well suited to leverage the blobstore. The restart command (that can
> also update the topology configuration) also looks compelling (STORM-1334).
>
> * Additional Scheduler Implementations (STORM-1320)
>
> * Additional Grouping Implementations (STORM-1328)
>
>
> As always I’m open to any opinions and suggestions.
>
> -Taylor
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61328109
>
>
>

Re: [VOTE] Release Apache Storm 1.1.0 (RC3)

[Harsha Chintalapani]

I propose that we continue with the release and get this patch in next
minor release (probably 1.1.1?). Users doesn't need to upgrade their
cluster to get this change
so I would say this is not a show-stopper for the release.

-Harsha

On Thu, Mar 23, 2017 at 9:18 AM P. Taylor Goetz <ptgo...@gmail.com> wrote:

> Hugo, now that you are a PMC member your vote is binding.
>
> While releases can’t be vetoed (a release vote needs at least three +1s
> and more +1s than -1s), it’s typical to take negative votes under serious
> consideration.
>
> So considering Hugo’s -1 vote, do we want cancel in order to include the
> proposed fix? It would also give us a chance to update the connector
> documentation regarding location of binary artifacts.
>
> -Taylor
>
>
> > On Mar 23, 2017, at 11:58 AM, Hugo Da Cruz Louro <hlo...@hortonworks.com>
> wrote:
> >
> > -1 (non binding)
> >
> > This blocker bug was just reported -
> https://issues.apache.org/jira/browse/STORM-2432
> > Here is the fix: https://github.com/apache/storm/pull/2027
> >
> > I consider that this is a blocker bug because if the user choses
> UNCOMITTED_LATEST first poll strategy, no data gets polled at all.
> >
> > Thanks,
> > Hugo
> >
> > On Mar 23, 2017, at 8:48 AM, Harsha Chintalapani <st...@harsha.io
> <mailto:st...@harsha.io>> wrote:
> >
> > +1 for documenting and releasing.
> > -Harsha
> >
> > On Thu, Mar 23, 2017 at 7:04 AM Jungtaek Lim <kabh...@gmail.com kabh...@gmail.com>> wrote:
> >
> > +1 to the latter.
> >
> > I'm in favor of documenting the change to release note, and also docs so
> > that website can be reflected. The users who are affected to the change
> > wouldn't be much, since using dependency management tool (Maven, Gradle,
> > and so on) has been recommended for creating topology jar.
> >
> > For me it's not a blocker for release.
> >
> > Arun, I initiated another thread to discuss moving non-connectors to the
> > top directory.
> > Could you cast your vote? If you are still not satisfied with excluding
> > jars you can cast -0 or even -1.
> >
> > - Jungtaek Lim (HeartSaVioR)
> >
> > 2017년 3월 23일 (목) 오후 10:43, P. Taylor Goetz <ptgo...@gmail.com ptgo...@gmail.com>>님이 작성:
> >
> > Do we want to cancel this RC in order to better document the changes, or
> > will documenting it in the release announcement suffice for now (provided
> > documentation is added for subsequent releases)?
> >
> > I’m partial to the latter, but am open to others’ opinions.
> >
> > -Taylor
> >
> >
> > On Mar 22, 2017, at 9:49 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID
> <mailto:ev...@yahoo-inc.com.INVALID>>
> > wrote:
> >
> > +1 I built form the tag and ran using a single node cluster.
> > The examples and external components are excluded because they are
> > huge.  Because of shading they we distribute the same copy of them
> > multiple
> > times.
> > I agree with Alexandre.  We should document this change better, because
> > it is confusing for people to get a release that used to have these in
> > it,
> > but does not any more.
> >
> >
> > - Bobby
> >
> > On Tuesday, March 21, 2017, 10:46:38 PM CDT, Arun Mahadevan <
> > ar...@apache.org<mailto:ar...@apache.org>> wrote:Verified the
> artifacts. Compiled examples and
> > ran
> > some sample topologies. Looks good.
> >
> > BTW, why are the external modules excluded from the binaries (the .zip
> > and .tar.gz). Isn’t it better if the binary distribution includes them?
> > Maybe it was already discussed but I am missing it. The sql directory
> > however seems to include the jars so it looks inconsistent.
> >
> > - Arun
> >
> >
> > On 3/22/17, 12:56 AM, "P. Taylor Goetz" <ptgo...@apache.org ptgo...@apache.org>> wrote:
> >
> > This is a call to vote on releasing Apache Storm 1.1.0 (rc3)
> >
> > Full list of changes in this release:
> >
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;h=68fbab3c4f91359bd397d93a157830542839b002;hb=e40d213de7067f7d3aa4d4992b81890d8ed6ff31
> >
> > The tag/commit to be voted upon is v1.1.0:
> >
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=7fa62404feb6b86b3143c851b46237580720eb6b;hb=e40d213de7067f7d3aa4d4992b81890d8ed6ff31
> >
> > The source archive being voted upon can be found here:
> >
> >
> >
> >
> 

Re: [VOTE] Release Apache Storm 1.1.0 (RC3)

[Harsha Chintalapani]

+1 for documenting and releasing.
-Harsha

On Thu, Mar 23, 2017 at 7:04 AM Jungtaek Lim <kabh...@gmail.com> wrote:

> +1 to the latter.
>
> I'm in favor of documenting the change to release note, and also docs so
> that website can be reflected. The users who are affected to the change
> wouldn't be much, since using dependency management tool (Maven, Gradle,
> and so on) has been recommended for creating topology jar.
>
> For me it's not a blocker for release.
>
> Arun, I initiated another thread to discuss moving non-connectors to the
> top directory.
> Could you cast your vote? If you are still not satisfied with excluding
> jars you can cast -0 or even -1.
>
> - Jungtaek Lim (HeartSaVioR)
>
> 2017년 3월 23일 (목) 오후 10:43, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
>
> > Do we want to cancel this RC in order to better document the changes, or
> > will documenting it in the release announcement suffice for now (provided
> > documentation is added for subsequent releases)?
> >
> > I’m partial to the latter, but am open to others’ opinions.
> >
> > -Taylor
> >
> >
> > > On Mar 22, 2017, at 9:49 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID>
> > wrote:
> > >
> > > +1 I built form the tag and ran using a single node cluster.
> > > The examples and external components are excluded because they are
> > huge.  Because of shading they we distribute the same copy of them
> multiple
> > times.
> > > I agree with Alexandre.  We should document this change better, because
> > it is confusing for people to get a release that used to have these in
> it,
> > but does not any more.
> > >
> > >
> > > - Bobby
> > >
> > > On Tuesday, March 21, 2017, 10:46:38 PM CDT, Arun Mahadevan <
> > ar...@apache.org> wrote:Verified the artifacts. Compiled examples and
> ran
> > some sample topologies. Looks good.
> > >
> > > BTW, why are the external modules excluded from the binaries (the .zip
> > and .tar.gz). Isn’t it better if the binary distribution includes them?
> > Maybe it was already discussed but I am missing it. The sql directory
> > however seems to include the jars so it looks inconsistent.
> > >
> > > - Arun
> > >
> > >
> > > On 3/22/17, 12:56 AM, "P. Taylor Goetz" <ptgo...@apache.org> wrote:
> > >
> > >> This is a call to vote on releasing Apache Storm 1.1.0 (rc3)
> > >>
> > >> Full list of changes in this release:
> > >>
> > >>
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;h=68fbab3c4f91359bd397d93a157830542839b002;hb=e40d213de7067f7d3aa4d4992b81890d8ed6ff31
> > >>
> > >> The tag/commit to be voted upon is v1.1.0:
> > >>
> > >>
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=7fa62404feb6b86b3143c851b46237580720eb6b;hb=e40d213de7067f7d3aa4d4992b81890d8ed6ff31
> > >>
> > >> The source archive being voted upon can be found here:
> > >>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.0-rc3/apache-storm-1.1.0-src.tar.gz
> > >>
> > >> Other release files, signatures and digests can be found here:
> > >>
> > >> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.0-rc3/
> > >>
> > >> The release artifacts are signed with the following key:
> > >>
> > >>
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> > >>
> > >> The Nexus staging repository for this release is:
> > >>
> > >>
> https://repository.apache.org/content/repositories/orgapachestorm-1047
> > >>
> > >> Please vote on releasing this package as Apache Storm 1.1.0.
> > >>
> > >> When voting, please list the actions taken to verify the release.
> > >>
> > >> This vote will be open for at least 72 hours.
> > >>
> > >> [ ] +1 Release this package as Apache Storm 1.1.0
> > >> [ ]  0 No opinion
> > >> [ ] -1 Do not release this package because...
> > >>
> > >> Thanks to everyone who contributed to this release.
> > >>
> > >> -Taylor
> > >
> >
> >
>

Apache Storm Meetup in BayArea

[Harsha Chintalapani]

Hi All,
   We are planning on scheduling a Storm Meetup in April 1st week.
Here is the meetup link https://www.meetup.com/Apache-Storm-Apache-Kafka/.
If you are interested in talking about your use-cases in storm there is 1
more slot available, please reach out to me.

Thanks,
Harsha

Re: [DISCUSS] Release Storm 1.1.0

[Harsha Chintalapani]

STORM-2340 is more of a feature . Auto-commit mode in storm-kafka used
rarely and most users
run the kafka spout with ackers and get at-least once guarantee.  If its
going to longer to address the PR reviews
I am +1 on moving this out of Storm 1.1.0. We already quite a few patches
storm-kafka-client and 1.1.0 release brings in lot of improvements
and bug-fixes.
-Harsha

On Wed, Feb 8, 2017 at 6:15 PM Jungtaek Lim <kabh...@gmail.com> wrote:

> There seems some pull requests for bugfix/improvement on
> storm-kafka-client, and some authors in PRs are not availble for now.
> (waiting 7 days)
>
> If we plan to get 1.1.1 out soon (say 1 month later or even closer) we can
> postpone, but if not, it might be better to coordinate these things ASAP
> and include to 1.1.0.
>
> There seems to be other small PRs, but nothing seems critical so it would
> be OK to not wait for merging.
>
> - Jungtaek Lim
>
> 2017년 2월 9일 (목) 오전 6:48, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
>
> Right now we’re down to 1 open issue on the 1.1.0 release epic: STORM-2250
> which is under active review/discussion.
>
> Assuming that is mergeable in the near future, are there any other open
> issues that should be considered for this release?
>
> -Taylor
>
>
> > On Feb 2, 2017, at 4:48 PM, P. Taylor Goetz <ptgo...@gmail.com> wrote:
> >
> > Thanks for putting this list together Jungtaek. I added a few to the 1.1
> release epic that I think are important. Feel free to do the same.
> >
> > Looks like we have a few to go, but there are pull requests for them.
> It’s mostly just a matter of reviews and review responses, so I think we
> are close.
> >
> > -Taylor
> >
> >> On Feb 2, 2017, at 1:41 AM, Jungtaek Lim <kabh...@gmail.com> wrote:
> >>
> >> Seems like there're not blockers for 1.1.0, but some pull requests are
> >> worth to check.
> >> There're pending pull requests for storm-kafka-client waited on
> STORM-2225.
> >> Given that STORM-2225 is now merged, we might need to take a look at.
> >>
> >> *- reviewing*
> >>
> >> [storm-core]
> >>
> >>> STORM-2324 : Fix deployment failure if resources directory is missing
> in
> >> topology jar
> >> (master) https://github.com/apache/storm/pull/1908
> >> (1.x) https://github.com/apache/storm/pull/1898
> >>
> >>> STORM-2321 Handle blobstore zk key deletion in KeySequenceNumber
> >> (master) https://github.com/apache/storm/pull/1904
> >> (1.x) https://github.com/apache/storm/pull/1905
> >>
> >> [storm-kafka]
> >>
> >>> STORM-2270 Kafka spout should consume from latest when ZK partition
> >> commit offset bigger than the latest offset
> >> (1.x) https://github.com/apache/storm/pull/1851
> >>
> >> [storm-kafka-client]
> >>
> >>> STORM-2281: Running Multiple Kafka Spouts (Trident) Throws Illegal
> State
> >> Exception
> >> (1.x) https://github.com/apache/storm/pull/1902
> >>
> >>> STORM-2315 Storm kafka client does not commit offsets when ack is
> disabled
> >> (1.x) https://github.com/apache/storm/pull/1891
> >>
> >>> fix: KafkaSpout is blocked in AutoCommitMode
> >> (master) https://github.com/apache/storm/pull/1863
> >>
> >>> STORM-2250: Kafka Spout Refactoring to Increase Modularity and
> Testability
> >> (master) https://github.com/apache/storm/pull/1832
> >>
> >>> STORM-2014: Put logic around dropping messages into RetryService,
> remove
> >> maxRetry setting from new KafkaSpout
> >> (master) https://github.com/apache/storm/pull/1605
> >>
> >>> fix NullPointException with acked.get(rtp)
> >> (master) https://github.com/apache/storm/pull/1807
> >>
> >> [storm-sql]
> >>
> >>> STORM-1443 [Storm SQL] Support customizing parallelism in StormSQL
> >> https://github.com/apache/storm/pull/1739
> >>
> >> *- pending*
> >>
> >> [storm-kafka-client]
> >>
> >>> STORM-2296 Kafka spout no dup on leader changes
> >> (1.0.x) https://github.com/apache/storm/pull/1873
> >> (1.x) https://github.com/apache/storm/pull/1888
> >>
> >> [storm-sql]
> >>
> >>> STORM-2148 [Storm SQL] Trident mode: back to code generate and compile
> >> Trident topology
> >> https://github.com/apache/storm/pull/1743
> >>
> >> Thanks,
> >> Jungtaek Lim (HeartSaVioR)
> >>
> >> 2017년 2월 2일 (목) 오전 8:14, Harsha Chintala

Re: [VOTE] Release Apache Storm 1.0.3 (rc1)

[Harsha Chintalapani]

+1
Tested in 3 node vagant cluster and ran few example topologies. Looks good
and verified the signature of artifacts.

On Wed, Feb 1, 2017 at 7:13 AM Bobby Evans 
wrote:

> +1 Ran some simple tests in cluster mode.
>
>
> - Bobby
>
> On Wednesday, February 1, 2017, 2:58:44 AM CST, Jungtaek Lim <
> kabh...@gmail.com> wrote:FYI: both issues are filed to STORM-2335
>  and STORM-2336
> , and patches are
> available for each issue.
>
> Please go ahead verifying RC1. They're not blocker and even we might want
> to include them to 1.0.3, we can verify RC1 first and test only two things
> for RC2 later.
>
> - Jungtaek Lim (HeartSaVioR)
>
> 2017년 2월 1일 (수) 오후 3:45, Jungtaek Lim 님이 작성:
>
> > > 2. running topology with local cluster doesn't terminate after cluster
> > > shutdown: Took stack trace, and found some non-daemon threads are live:
> > > Async Localizer, Localizer Cache Cleanup x 2.
> >
> > Local mode or a single node cluster?
> >
> > In local mode. Cluster mode is OK.
> >
> > Btw, thanks for the kind words. Hope this helps reducing others load for
> > participating RC verification.
> >
> > - Jungtaek Lim (HeartSaVioR)
> >
> > 2017년 2월 1일 (수) 오후 2:43, P. Taylor Goetz 님이 작성:
> >
> > Thanks for thorough review. It's a great example of what a good review
> > looks like.
> >
> > Comments in line below.
> >
> > -Taylor
> >
> > > On Jan 31, 2017, at 11:33 PM, Jungtaek Lim  wrote:
> > >
> > > +1 (binding)
> > >
> > >> source
> > >
> > > - verify file (signature, MD5, SHA)
> > > -- source, tar.gz : OK
> > > -- source, zip : OK
> > >
> > > - extract file
> > > -- source, tar.gz : OK
> > > -- source, zip : OK
> > >
> > > - diff-ing extracted files between tar.gz and zip : OK
> > >
> > > - build source with JDK 7
> > > -- source, tar.gz : OK
> > >
> > > - build source dist
> > > -- source, tar.gz : OK
> > >
> > > - build binary dist
> > > -- source, tar.gz : OK
> > >
> > >> binary
> > >
> > > - verify file (signature, MD5, SHA)
> > > -- binary, tar.gz : OK
> > > -- binary, zip : OK
> > >
> > > - extract file
> > > -- source, tar.gz : OK
> > > -- source, zip : OK
> > >
> > > - diff-ing extracted files between tar.gz and zip : OK
> > >
> > > - launch daemons : OK
> > > - run RollingTopWords (local) : OK
> > > - run RollingTopWords (remote) : OK
> > > - activate / deactivate / rebalance / kill : OK
> > > - logviewer (worker dir, daemon dir) : OK
> > > - change log level : OK
> > >  - thread dump, heap dump, restart worker : OK
> > > - log search : OK
> > > - run WordCountTopology (multilang, remote) : OK
> > > - run StatefulTopology (local) : OK
> > > - run StatefulTopology (remote) : OK
> > > - run StatefulWindowingTopology (remote) : OK
> > >
> > > Found two bugs here which are not blocker:
> > >
> > > 1. visualization: it doesn't show graph correctly while worker is being
> > > initializing, and below error message is printed on console, which I
> > think
> > > is long-lasting bug.
> > >
> >
> > I think this has been around for a while. I agree that it's not a
> blocker.
> >
> > > 2. running topology with local cluster doesn't terminate after cluster
> > > shutdown: Took stack trace, and found some non-daemon threads are live:
> > > Async Localizer, Localizer Cache Cleanup x 2.
> >
> > Local mode or a single node cluster?
> >
> > I saw this when testing the worker shutdown hook bug in local mode. I
> > don't think this is a blocker, but we should definitely follow up with a
> > JIRA. This could affect users who rely on local mode for integration
> tests.
> >
> > >
> > > I'll file two issues afterwards.
> > >
> > > Thanks,
> > > Jungtaek Lim (HeartSaVioR)
> > >
> > > 2017년 2월 1일 (수) 오전 5:55, P. Taylor Goetz 님이 작성:
> > >
> > >> This is a call to vote on releasing Apache Storm 1.0.3 (rc1)
> > >>
> > >> Full list of changes in this release:
> > >>
> > >>
> > >>
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=6cb735d18ec11eccd3e80bab8f879c989a9b3967;hb=a81ec2580fce1f2ee6349a9028dcb75763798bec
> > >>
> > >> The tag/commit to be voted upon is v1.0.3:
> > >>
> > >>
> > >>
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;h=b8adddc6aa20107288e59cba6a2976c0951742fb;hb=a81ec2580fce1f2ee6349a9028dcb75763798bec
> > >>
> > >> The source archive being voted upon can be found here:
> > >>
> > >>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.3-rc1/apache-storm-1.0.3-src.tar.gz
> > >>
> > >> Other release files, signatures and digests can be found here:
> > >>
> > >> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.3-rc1/
> > >>
> > >> The release artifacts are signed with the following key:
> > >>
> > >>
> > >>
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> > >>

Re: [DISCUSS] Release Storm 1.1.0

[Harsha Chintalapani]

Trying to check the status on this release of 1.1.0. Are we going to do
this release anytime soon?


On Fri, Jan 13, 2017 at 7:50 PM S G  wrote:

> Not sure if its a little late to include for the 1.1.0 and 1.0.3 releases
> now, but can we consider using zookeeper 3.4.9 for the future versions as
> 3.4.9 brings in a lot of stability improvements (
> http://zookeeper.apache.org/releases.html) and storm is still using 3.4.6
> (
> https://github.com/apache/storm/blob/master/pom.xml)
>
> On Fri, Jan 6, 2017 at 11:54 AM, P. Taylor Goetz 
> wrote:
>
> > Thanks for the update Jungtaek.
> >
> > I’m verifying the patches now. And they should be mergeable shortly.
> >
> > I think we’ll likely be ready for 1.1.0 and 1.0.3 releases next week.
> >
> > -Taylor
> >
> > > On Jan 6, 2017, at 3:40 AM, Jungtaek Lim  wrote:
> > >
> > > I just submitted a patch for STORM-2176
> > > . Since it's a small
> > fix,
> > > we just need to handle STORM-2228
> > >  to release.
> > >
> > > - Jungtaek Lim (HeartSaVioR)
> > >
> > > 2017년 1월 5일 (목) 오후 1:43, Jungtaek Lim 님이 작성:
> > >
> > >> Recently we receive some requests regarding release Storm 1.0.3, so
> > would
> > >> like to bump this again.
> > >>
> > >> Given that blocker issues for Storm 1.1.0 are also blocker for 1.0.3,
> > I'd
> > >> like to ask a favor of taking care of 'open' / 'in progress' issues on
> > >> 1.1.0 epic.
> > >> https://issues.apache.org/jira/browse/STORM-1856
> > >>
> > >> There're one 'open' issue and three 'in progress' issues. Two of three
> > 'in
> > >> progress' issues are tiny fix so easy to be handled, so actual
> blockers
> > are
> > >> STORM-2176  and
> > >> STORM-2228 .
> > >>
> > >> Thanks,
> > >> Jungtaek Lim (HeartSaVioR)
> > >>
> > >> 2016년 11월 17일 (목) 오후 11:41, Satish Duggana  >님이
> > >> 작성:
> > >>
> > >>STORM-2205: Race condition in getting nimbus summaries while ZK
> > >> connections are reconnected.
> > >>
> > >> This issue seems to occur in our environments and I would like this to
> > be
> > >> part of 1.1.0.
> > >>
> > >> Thanks,
> > >> Satish.
> > >>
> > >> On Thu, Nov 17, 2016 at 9:36 AM, Jungtaek Lim 
> > wrote:
> > >>
> > >>> I have no idea on storm-kafka-client, but some bugfix issues for
> > >>> storm-kafka-client are waiting for reviewing / merging.
> > >>>
> > >>> STORM-2014 
> > >>> STORM-2087 
> > >>> STORM-2104 
> > >>>
> > >>> If someone can review them in several days it would be great.
> > >>>
> > >>> I hope that we include currently opened pull requests for Storm SQL
> so
> > >> that
> > >>> we can release 'usable Storm SQL' more usable, but I'm also OK to
> > >> postpone
> > >>> them to be included to next release if they drag the release.
> > >>>
> > >>> STORM-1446 
> > >>> STORM-1443 
> > >>> STORM-2148 
> > >>> STORM-2170 
> > >>>
> > >>> I can see some pull requests which address Trident implementations
> for
> > >>> storm-kafka-client, storm-mongodb, storm-cassandra.
> > >>>
> > >>> storm-kafka-client: STORM-1694
> > >>>  (patch for 2.0 is
> > >>> merged, patch for 1.x is ready for reviewing)
> > >>> storm-cassandra: STORM-1369
> > >>> 
> > >>> storm-mongodb: STORM-1607  > >>> jira/browse/STORM-1607>
> > >>>
> > >>> If we want to cut the release now, we could include only bugfix
> issues
> > >> and
> > >>> postpone others. Otherwise we could discuss and include some or all
> of
> > >> the
> > >>> above.
> > >>>
> > >>> What do you think? When we want to start the release process for
> 1.1.0?
> > >>>
> > >>> - Jungtaek Lim (HeartSaVioR)
> > >>>
> > >>> 2016년 11월 16일 (수) 오전 4:11, P. Taylor Goetz 님이 작성:
> > >>>
> > >>> Thanks Xin, I added it to the 1.1.0 epic.
> > >>>
> > >>> -Taylor
> > >>>
> >  On Nov 15, 2016, at 9:01 AM, Xin Wang 
> wrote:
> > 
> >  STORM-2198 ( PR: https://github.com/apache/storm/pull/1773 ) fixes
> a
> > >> bug
> > >>> of
> >  storm-hdfs. Do we have a consideration to include this?
> > 
> >  Thanks,
> >  Xin Wang (vesense)
> > 
> >  2016-11-15 10:03 GMT+08:00 Jungtaek Lim :
> > 
> > > Some issues on Storm SQL are resolved but not documented yet. I'll
> > >> file
> > >>> an
> > > issue and 

Re: [DISCUSS] Adopt pull request template

[Harsha Chintalapani]

   I am ok with adopting templates but lets keep this process simpler.
We've new contributors coming in and they probably didn't have chance to go
through process.
We can guide them through the process or have strict template that everyone
needs to adopt to.
   I don't think having guidelines such as having exactly 1 space after
the description is helpful to anyone. Its just a PR as long as the title
reflects the JIRA title and
JIRA is descriptive enough we all understand whats the patch intends do.
   We shouldn't be going through loops for opening a PR with
restrictions such as above.
Hugo,
  "STORM-XXX: Title matching exactly the JIRA Summary (title)" this
looks good to me. But having description details is hard to follow and
unnecessary.
regarding Squashing guidelines, we've gone through this discussion and I
think this should be left to author discretion as they would understand
what commits are more important.
Also as reviewers one can suggest to squash the commits if additional
commits doesn't make sense.

Thanks,
Harsha

On Thu, Jan 19, 2017 at 10:29 AM Hugo Da Cruz Louro <hlo...@hortonworks.com>
wrote:

> I am strongly +1. In addition to the templates suggested by Jungtaek, in
> the past I suggested the template bellow to a different project. It is a
> bit simpler than the ones listed, as I think it may be too demanding to ask
> people to categorize in the git message the type of contribution the patch
> is addressing (BUG, Feature, …), if it has docs, if it has tests, etc. I
> also think that it may make the git message too verbose. That info is
> contained in the JIRA, and if there is a 1 to 1 mapping between Git commit
> and JIRA, it’s fairly easy to track down.
>
> The PR template is important, but more important, in my opinion, is to
> have strong guidelines requiring that people squash commits within
> functional units/features the code is addressing. Furthermore, if the
> commits are for one JIRA/change (when no JIRA), and there are more than one
> commit, all should have the same title (STORM-XXX) in it. We should at all
> costs avoid merging into master extreme cases like 10 commits related to
> one patch, where 5 or 6 of them are commits along the lines, addressed code
> review, added tests, cleaned up tests, addressed more code reviews, etc...
> without even having the STORM-XXX prefix in them.
>
> Squashing commits not only will make cherry-pick much easier, but more
> importantly, if there is a regression, it is very easy to track down which
> change is causing the regression. Not doing this makes it very hard to
> track down which change is causing problems.
>
> If we agree on some guidelines, I volunteer to update the documentation.
>
> === GIT TEMPLATE GUIDELINES ===
>
> STORM-XXX: Title matching exactly the JIRA Summary (title)
>
> - An optional, bulleted (+, -, ., *), description of the contents of
> - the patch. The goal is not to describe the contents of every file,
> - but rather give a quick overview of the main functional areas
> - addressed by the patch.
>
> The text immediately following the JIRA number (STORM-XXX: ) must be an
> exact transcription of the JIRA summary (title), not the a summary of the
> contents of the patch. If the JIRA summary does not accurately describe
> what the patch is addressing, the JIRA summary must be fixed, and then
> copied to the Git commit message.
>
> A description with the contents of the patch is optional but strongly
> encouraged if the patch is large and/or the JIRA title is not expressive
> enough to highlight what the patch is doing. This text must be added
> bellow, with one empty line of space in between, and be bulleted using one
> of the following bullet points (+, -, ., *). There must be at last a 1
> space indent before the bullet char, and exactly one space between bullet
> char and the first letter of the text. Bullets are not optional but
> required.
>
> === GIT TEMPLATE GUIDELINES FOR COMMITS WITHOUT JIRA ===
>
> For the sake of having a pattern one could grep, or exclude from grep, I
> suggest that we prefix every commit that does not have a JIRA associated
> with it with something along the lines:
>
> STORM-MINOR: Title summarizing what the patch is addressing.
>
> If a description of the change is necessary, it should follow the
> guidelines above. The squashing guidelines also apply.
>
> === SQUASHING GUIDELINES ===
>
> - Each commit message should have the first line matching the associated
> JIRA (STORM-XXX, or STORM-MINOR) as described
> - If possible, one JIRA/change should be one commit.
> - If the patch is large, and it makes sense to break it into multiple
> commits to make it easier to understand and cherry-pick (not easy to
> merge), each commit should represent the fu

Re: [DISCUSSION] Disable integration test

[Harsha Chintalapani]

Having integration tests run as part of the travis helps finding any
run-time issues that we might not be able to catch otherwise. Can you
please file a JIRA, Raghav who put this together might help.

Thanks,
Harsha

On Tue, Dec 27, 2016 at 6:15 AM Xin Wang <data.xinw...@gmail.com> wrote:

> Hi Jungtaek,
>
> I agree with you. If we can't fix the issue in short time, we'd better to
> disable integration test.
>
> - Xin
>
> 2016-12-27 16:34 GMT+08:00 Jungtaek Lim <kabh...@gmail.com>:
>
> > Hi devs,
> >
> > Storm build result on Travis CI is failing most of time and it's because
> > integration test. Due to this, we don't see Travis CI result, and miss
> > other things like JDK compatibility (JDK 8 vs JDK 7 on between branches),
> > or RAT failing.
> >
> > Unless there's no volunteer to fix integration test in short time, I
> think
> > we would be better to disable integration test.
> >
> > What do you think?
> >
> > Thanks,
> > Jungtaek Lim (HeartSaVioR)
> >
>

Re: [DISCUSS] breaking changes in 2.x

[Harsha Chintalapani]

Why don't we start introducing protocol versions in thrift API. This way a
old supervisor can still communicate by providing a older version and new
nimbus can handle both protocols. API changes doesn't necessarily mean
we've to break the older API. If the existing users doesn't have good
upgrade path there will less uses on the new version.
Critical API changes can still be possible by versioning the API.
-Harsha

On Thu, Nov 10, 2016 at 7:06 AM Kyle Nusbaum <knusb...@yahoo-inc.com.invalid>
wrote:

> On Wednesday, November 9, 2016, 7:23:09 AM CST, Harsha Chintalapani <
> st...@harsha.io> wrote:> If we want users to upgrade to new version, the
> rolling upgrade is a major
> > decision factor. As a community, we need to look API updates or breaking
> > changes much more diligently.
> Within a major version, I agree. APIs should be as stable as possible
> within a version release.
>
> > I agree to an extent we shouldn't limiting ourselves with rolling
> upgrade.
> > But having announced rolling-upgrade in 0.10 and then not supporting it
> in
> > 1.x and now in 2.x. In User's point of view, Storm is not rolling
> > upgradable although we shipped a release stating that rolling upgrade is
> > supported and in follow-up release we taken that off.
> The user would be correct. Storm would not be rolling-upgradable *between
> major versions.*I don't see how it's possible to develop and improve a
> project if it must remain perpetually backwards compatible, so I think it's
> necessary to reject compatibility as a *primary* goal.
> Eventually (hopefully) we'll arrive at an API that we're happy with that
> we don't feel like we need to change.Then we can claim rolling upgrades
> across major version numbers.
>
> > Does these API changes are critical and worth breaking rolling upgrade?
> My position is that we don't want to limit ourselves to "critical" API
> changes. This will stick us with an inferior API that we can't evolve.It's
> accepting the long-term pain of inconsistent API or old baggage to avoid
> the short-term pain of relaunching or updating topologies when you do a
> major version upgrade.
> Storm is not at the place in its life where it has stopped evolving, and I
> don't want to stifle its development.
>

Re: [DISCUSS] breaking changes in 2.x

[Harsha Chintalapani]

If we want users to upgrade to new version, the rolling upgrade is a major
decision factor. As a community, we need to look API updates or breaking
changes much more diligently.
I agree to an extent we shouldn't limiting ourselves with rolling upgrade.
But having announced rolling-upgrade in 0.10 and then not supporting it in
1.x and now in 2.x. In User's point of view, Storm is not rolling
upgradable although we shipped a release stating that rolling upgrade is
supported and in follow-up release we taken that off.
Does these API changes are critical and worth breaking rolling upgrade?

-Harsha

On Mon, Nov 7, 2016 at 9:16 AM Kyle Nusbaum <knusb...@yahoo-inc.com.invalid>
wrote:

I worry that making it a priority to have rolling upgrades between major
versions significantly restricts the kinds of changes that we can make,
including some kinds of changes that a major version increment is supposed
to mark. I'm not really in support of trying to do that.
If we can't make changes that break compatibility now, when can we make
them? Can changes like that ever be made? I don't know that it's good to
limit ourselves like that.
Trying to accommodate rolling upgrades is a good idea, but I'm not sure
about rejecting code changes across major versions to support them. 2.x
represents a large shift in the project, and I expect once the translation,
etc. is done, things will calm down and APIs will become more stable,
allowing more of our future releases to be rolling even across major
versions. I'd rather get these kinds of changes out of the way now in the
2.x release than cart along the vestiges of 1.x from now on.
What do others think about this?
-- Kyle

On Monday, November 7, 2016, 3:10:08 AM CST, Bobby Evans
<ev...@yahoo-inc.com.INVALID> wrote:Let's distinguish between wire
compatibility changes and API compatibility changes, along with impact to
workers vs impact to clients.
For 3) splitting the classpath up for each daemon wire compatibility is not
impacted, but we are potentially removing a bunch of APIs from the worker
and client classpath.  Most of these where shaded and users should not be
impacted by them being removed, but a few like servlet-api-2.5.jar are
likely to be removed.  So yes the impact here would likely be very small.
However on the client side if a topology wants to include LocalCluster
(like we do in a lot of examples) the topology jar will get a lot bigger.
LocalCluster needs access to nimbus, supervisor, and drpc server.  These
would not be on the worker classpath any more and would then need to be
packaged into the topology jar to make LocalCluster work.  In production I
would expect LocalCluster to be used by tests and not be included like we
do in a lot of examples.  This is more of a shift for how we expect users
to interact with the LocalCluster.
For 1) NodeInfo.port depending on how we do it, it could break wire
compatibility and API compatibility (which is what I would prefer).  We
could play some games to maintain compatibility, but for me it is enough of
a pain that I am not sure it is worth it.  However this is not likely to
impact workers because they don't use these APIs directly.  It might impact
clients but only if they have custom code to profile their topologies.  IF
they use the build in CLI/UI it would not be impacted.
For 2) nocamel would break API compatibility, but not wire compatibility.
This is not likely to impact workers because like with 1 workers don't
really interact with nimbus directly so it would not be a problem.  Old
clients running with older versions of storm would continue to work, but
any custom client code (think what gets run by storm jar) would need to be
recompiled/modified to be able to run on against a storm-2.x client.

- Bobby

Re: Fw: Re: [DISCUSS] breaking changes in 2.x

[Harsha Chintalapani]

My only concern here is the rolling upgrade of storm cluster. We supported
the rolling upgrade going to 0.10 and broke it because of storm 1.x
release. Users are not inclined to upgrade to a new release if it's not
rolling upgradable. In this case, it looks like we are going to break this.
Correct me if I am wrong here.

Thanks,
Harsha

On Mon, Nov 7, 2016 at 7:43 AM Bobby Evans <ev...@yahoo-inc.com.invalid>
wrote:

> Made a mistake and put something on private that never should have been
> there.  Here is the discussion in full so far.
> In response to Jungtake removing the nocamel option will change
> set_bar/get_bar in the generated thrift code to setBar/getBar.  So any
> thrift object that clients interact with will need to make similar changes.
> - Bobby
>
>  - Forwarded Message -From: Bobby Evans <ev...@yahoo-inc.com>To: "
> priv...@storm.apache.org" <priv...@storm.apache.org>Sent: Monday,
> November 7, 2016, 3:37:50 AM CST Subject: Re: [DISCUSS] breaking changes in
> 2.x
>  Sorry you are right, I put in the wrong mailing list.  I feel dumb.  Will
> move it to dev.
> - Bobby
>
> On Sunday, November 6, 2016, 2:18:33 AM CST, P. Taylor Goetz <
> ptgo...@gmail.com> wrote:Can we move this discussion to dev@? There
> doesn't seem to be anything sensitive about the subject.
> -Taylor
> On Nov 6, 2016, at 1:28 AM, Jungtaek Lim <kabh...@gmail.com> wrote:
>
>
> Regarding breaking change, I'm OK to break backward UX compatibility if
> we're sure that it gives better UX. And since we're talking about next
> major, if we don't do it now, we might have no chance to do the near future.
> For 1) this should be corrected, and for 2) I'm not sure how it affects to
> end-users, but it would be better to address if it doesn't hurt user codes
> much. 3) I think expected user impact is similar to what we did it for
> adding dependencies to shade list. Then it doesn't look matter to me.
> Thanks,Jungtaek Lim (HeartSaVioR)
> 2016년 11월 5일 (토) 오후 11:01, Bobby Evans <bo...@apache.org>님이 작성:
>
> I have been working on getting some of clojure code translated to java,
> and I have run into a few issues that I really would like to address, but
> will impact users.  Because 2.x has not been released yet technically these
> should be OK, but I want to get feedback from others before I spend a lot
> of time on them or even file a JIRA.
> 1)WHAT: The thrift NodeInfo.port should be a single long, not a set of
> longs.USER IMPACT: it is exposed to end users through the profiling request
> API. BACK COMPAT: it is possible, but I would prefer not to.
>
> 2)WHAT: Using 'nocamel' when generating thrift makes generated exceptions
> less useful and the API less java like.USER IMPACT: users when upgrading
> would need to modify some API calls when setting/reading thrift
> objects.BACK COMPAT: most older clients should still work for a lot of
> things
> 3) WHAT: split up storm-core so we have class path separation between
> daemons and a true client.USER IMPACT: Users may need to include new
> dependencies to get what they want.  Old topologies may not get access to
> everything that they want.BACK COMPAT:  we could leave some old
> dependencies in place that are not needed, but why?
> If we do all of these it is likely that profiling requests from an old
> client will not work, but for the most part a 1.x client would still work
> with 2.x.
> - Bobby
>
>

Re: [DISCUSS] Accept JW Player SQE Code Donation

[Harsha Chintalapani]

>From the looks of it the benefit accepting this code donation is to attract
more developers on storm-sql . What is stopping SQE developers to come and
contribute to JIRAs that are open on storm-sql?.  I don't see just
accepting code donation and setting aside is not much of a
incentive/motivation for someone to contribute.

-Harsha

On Thu, Aug 25, 2016 at 12:14 PM P. Taylor Goetz <ptgo...@gmail.com> wrote:

> I didn’t mean to imply that the code donation would come with automatic
> committership. I would expect the SQE developers to engage with the Storm
> community and show merit before being invited to become committers. If for
> some reason that didn’t happen, we would always have the option of not
> doing anything with the code donation.
>
> With regard to it recently being open sourced, yes that is the case. JW
> Player open sourced it in order to make the code donation, which explains
> the number of commits/contributors. I’m not sure if it’s due to a github
> auto-watch setting, but the project has a lot of watchers from JW Player
> (which might be an indicator of potential contributor pool):
>
> https://github.com/jwplayer/SQE/watchers
>
> When they approached me about the code contribution, I mentioned the
> existing SQL work that’s been/being done. They are aware of that effort,
> and see this as a collaboration opportunity to improve SQL support in Storm
> (i.e. not a competitor to the existing SQL work).
>
> Personally I would prefer a single SQL API/implementation for Storm. If
> that involves cherry-picking/merging two projects into one, I’m okay with
> that.
>
> -Taylor
>
>
>
> On Aug 25, 2016, at 9:30 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID>
> wrote:
>
> I agree that committership should not come form a code donation, but on
> the other hand a code donation needs to have a community with it to be
> viable.
>
> That is the one thing that makes slightly nervous here.
> https://github.com/jwplayer/SQE/pulse
>
> https://github.com/jwplayer/SQE/graphs/contributors
>
> show that the code was very recently open sourced and only one person has
> made two commits.  Which totally makes since from the perspective of a
> corporation that just released the code.  But by the same right I would
> like to hear from others to see how many people are currently interested in
> helping to maintain this new code base, and also what is each person's long
> term plan/vision for this code.
> HeartSavior has indicated his intention is to try and combine the back-end
> with SQL which seems like a good idea, but at the same time others have
> expressed concern that we have multiple different APIs and execution
> environments for Storm.  Adding another without any plan on where we want
> to go long term gives me pause.
>  - Bobby
>
>On Thursday, August 25, 2016 5:07 AM, Jungtaek Lim <kabh...@gmail.com>
> wrote:
>
>
> I'd postpone to talk about committership a bit later after they show active
> contributions. Personally I don't like associating code donation and
> committership but it's just me.
> Storm SQL (or SQL feature for all projects) has lots of places for
> contributions so they can be eventually invited like other commiters/PMCs,
> similar to John Fang.
>
> Regarding future of storm-sql after SQE, it would be better to discuss
> again when the process of code donation is in place, so that we can discuss
> with comparison between storm-sql and SQE at that time.
>
> Thanks,
> Jungtaek Lim (HeartSaVIoR)
>
> 2016년 8월 25일 (목) 오후 3:44, Abhishek Agarwal <abhishc...@gmail.com>님이 작성:
>
> what happens to current storm-sql once SQE is merged into apache
> repository?
>
> On Thu, Aug 25, 2016 at 10:26 AM, P. Taylor Goetz <ptgo...@gmail.com>
> wrote:
>
>
> On Aug 24, 2016, at 5:37 PM, Jungtaek Lim <kabh...@gmail.com> wrote:
>
> While I feel Storm SQL covers (and will cover in near future) all of
>
> the
>
> features what SQE has, I'd like to consider this as not only code
>
> donation
>
> but also having more contributors on Storm SQL.
>
>
> +1
>
> This is a great opportunity to grow the community interested in streaming
> SQL.
>
>
> As the one working on storm-sql now, I think Storm SQL should have
> more contributors who are experienced or expert on this area, or at
>
> least
>
> have a mind to contribute heavily.
>
>
> +1 again.
>
> In my conversations with JW Player, they indicated that their developers
> are very interested in joining the Storm community.
>
> There're still many features to implement (join and windowing,
>
> parallelism,
>
> and so on) which needs some efforts so we can boost up if more
>
> contribu

Re: Rolling upgrades for a topology using kafka spout

[Harsha Chintalapani]

Abhishek,
Are you looking rolling upgrade kafka cluster or storm?
Harsha

On Fri, Aug 26, 2016 at 6:18 AM Abhishek Agarwal <abhishc...@gmail.com>
wrote:

>
> On Aug 26, 2016 2:50 PM, "Abhishek Agarwal" <abhishc...@gmail.com> wrote:
>
> >
>
> > Here is an interesting use case - To upgrade a topology without any
> downtime. Let's say, the topology has only Kafka as a source and two
> versions of it are running (different topology names of course) in parallel
> and sharing the kafka input load.
> >
> > In old kafka spout, rolling upgrade is not possible, partition
> assignment is derived from the number of tasks in the topology.
> >
> > In new kafka spout, partition assignment is done externally by Kafka
> server. If I deploy two different topologies with same* kafka consumer
> group id*, is it fair to assume that load will be automatically
> distributed across topologies? Are there any corner cases to consider?
> >
> > --
> > Regards,
> > Abhishek Agarwal
> >
>

Re: [VOTE] Release Apache Storm 0.10.2 (RC1)

[Harsha Chintalapani]

We need to get this patch in for 0.10 release
https://github.com/apache/storm/pull/1645

Thanks,
Harsha

On Fri, Aug 26, 2016 at 7:28 AM Bobby Evans <ev...@yahoo-inc.com.invalid>
wrote:

> +1 - Bobby
>
> On Friday, August 26, 2016 9:03 AM, Jungtaek Lim <kabh...@gmail.com>
> wrote:
>
>
>  +1 (binding)
>
> - testing with source distribution : OK
>   - verify signature (zip, tar.gz): OK
>   - uncompress : OK
>   - building from source dist : OK
>   - how to build: running `mvn clean install` on unzipped source dist.
>
> - testing with binary distribution (one machine) : OK
>   - verify signature (zip, tar.gz): OK
>  - launch daemons : OK
>  - run RollingTopWords (local) : OK
>  - run RollingTopWords (remote) : OK
>   - activate / deactivate / rebalance / kill : OK
>   - logviewer (worker) : OK
>   - run WordCountTopology (multilang, local) : OK
>   - run WordCountTopology (multilang, remote) : OK
>
> Btw, would we want to include STORM-2042 to 0.10.2?
> I agree this is definitely not a blocker, but we have been releasing 0.10.x
> line slowly so might be worth to consider.
> As I voted +1 I'm also OK to not include it.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 8월 25일 (목) 오전 5:22, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
>
> > This is a call to vote on releasing Apache Storm 0.10.2 (rc1)
> >
> > Full list of changes in this release:
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=3c24dce8b35527b409cae68a1ebc0e25e5a0b03f
> >
> > The tag/commit to be voted upon is v0.10.2:
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=2c356b9c1085d35cfc6d0869fcca1d570910053e;hb=3c24dce8b35527b409cae68a1ebc0e25e5a0b03f
> >
> > The source archive being voted upon can be found here:
> >
> >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-0.10.2-rc1/apache-storm-0.10.2-src.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-0.10.2-rc1/
> >
> > The release artifacts are signed with the following key:
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >
> > The Nexus staging repository for this release is:
> >
> > https://repository.apache.org/content/repositories/orgapachestorm-1040
> >
> > Please vote on releasing this package as Apache Storm 0.10.2.
> >
> > When voting, please list the actions taken to verify the release.
> >
> > This vote will be open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache Storm 0.10.2
> > [ ]  0 No opinion
> > [ ] -1 Do not release this package because...
> >
> > Thanks to everyone who contributed to this release.
> >
> > -Taylor
> >
>
>

Re: [Vote] Make Java 8 as minimum requirement for Storm 2.0

[Harsha Chintalapani]

I guess everyone has different interpretation of what Bylaws means . More
context

https://github.com/apache/storm/pull/1628

anything wrong with Vote thread?


On Tue, Aug 16, 2016 at 8:04 PM P. Taylor Goetz <ptgo...@gmail.com> wrote:

> Why is this a VOTE?
>
>
> > On Aug 16, 2016, at 9:18 PM, Harsha Chintalapani <st...@harsha.io>
> wrote:
> >
> > Hi All,
> >We had a discussion thread for removing Java 7 support for Storm
> > 2.0.
> > Here is a formal voting thread and the JIRA
> > https://issues.apache.org/jira/browse/STORM-2041.
> >
> > Thanks,
> > Harsha
>
>

[Vote] Make Java 8 as minimum requirement for Storm 2.0

[Harsha Chintalapani]

Hi All,
We had a discussion thread for removing Java 7 support for Storm
2.0.
Here is a formal voting thread and the JIRA
https://issues.apache.org/jira/browse/STORM-2041.

Thanks,
Harsha

Re: [VOTE] Release Apache Storm 0.9.7 (RC1)

[Harsha Chintalapani]

Do we have any user requests on releasing 0.9.x . I rather propose them to
move on to 0.10.x line and retire 0.9.x branches. There are quite few
issues that got fixed in 0.10.x release line and keep maintaining 0.9.x
line wouldn't be beneficial.

Thanks,
Harsha

On Mon, Aug 15, 2016 at 4:48 PM Jungtaek Lim <kabh...@gmail.com> wrote:

> Would it be better to have consensus for this?
>
> In bylaw we have this sentence 'In particular all releases must be approved
> by the PMC.'
> One of PMC can call out the discussion for releasing specific version, but
> general consensus should be made before starting prepare release.
>
> I'd like to see this starting from there. If we just want to also address
> opinions (not verification) of release from VOTE thread, I'll do that.
>
> Btw, personally I'm -1 for this release, since STORM-1928 is even not
> backported to 0.10.x version line.
> I'm even not supportive to maintain 0.x line (because we did the best
> effort for 0.x line users to move on 1.x), but if we really want to ship
> this bugfix to 0.x version line, for me it's more making sense to release
> 0.10.2.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 8월 16일 (화) 오전 5:10, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
>
> > This is a call to vote on releasing Apache Storm 0.9.7 (rc1).
> >
> > This release candidate addresses a critical issue with Storm’s multi-lang
> > component where an improperly formed multi-lang spout message would cause
> > the spout to hang indefinitely.
> >
> > Full list of changes in this release:
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=b75d9b2b6dbfd0fda0b114feafec1384bbfa30aa
> >
> > The tag/commit to be voted upon is v0.9.7:
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=3d7c7be37ecc1dcc25223fde670d8185a6afbf00;hb=b75d9b2b6dbfd0fda0b114feafec1384bbfa30aa
> >
> > The source archive being voted upon can be found here:
> >
> >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-0.9.7-rc1//apache-storm-0.9.7-src.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-0.9.7-rc1/
> >
> > The release artifacts are signed with the following key:
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >
> > The Nexus staging repository for this release is:
> >
> > https://repository.apache.org/content/repositories/orgapachestorm-1039
> >
> > Please vote on releasing this package as Apache Storm 0.9.7.
> >
> > When voting, please list the actions taken to verify the release.
> >
> > This vote will be open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache Storm 0.9.7
> > [ ]  0 No opinion
> > [ ] -1 Do not release this package because...
> >
> > Thanks to everyone who contributed to this release.
> >
> > -Taylor
> >
>

Re: Website update

[Harsha]

Jungtaek,
 Some of the recent slides in hadoop summit san jose

http://www.slideshare.net/HadoopSummit/resource-aware-scheduling-in-apache-spark
http://www.slideshare.net/HadoopSummit/the-future-of-apache-storm-63920895?qid=4a6e1165-6dbf-47b9-a8c0-1648308762aa==_search=3
 
https://www.youtube.com/watch?v=_Q2uzRIkTd8=youtu.be

http://www.slideshare.net/HadoopSummit/managing-hadoop-hbase-and-storm-clusters-at-yahoo-scale
https://www.youtube.com/watch?v=Ib8ch-xVzbw

-Harsha

On Thu, Aug 11, 2016, at 08:42 PM, Jungtaek Lim wrote:
> Hi dev,
> 
> Along with taking care of some documentation issues, I also add some
> missing committer/PMCs into people page.
> 
> I feel that it would be better to update 'Talks and slideshows' to topic
> of
> recent Hadoop summit or others.
> Any ideas or recommendations?
> 
> Thanks,
> Jungtaek Lim (HeartSaVioR)

[Discussion] Dropping Java 7 support on master

[Harsha]

Hi All, 
 Dropping java 7 support on master will allow us to use the new
 api in Java 8 and since the master is being used for java
 migration its good to make the decision now. Let me know your
 thoughts. 
Thanks, 
Harsha

[Discussion] Dropping Java 7 support on master

[Harsha Chintalapani]

Hi All,
  Dropping java 7 support on master will allow us to use the new api in
Java 8 and since the master is being used for java migration its good to
make the decision now. Let me know your thoughts.

Thanks,
Harsha

Re: [VOTE] Release Apache Storm 1.0.2 (rc4)

[Harsha Chintalapani]

+1 (binding)
Ran a single node cluster and tried example topologies. Verified signatures
on the artifcats.
-Harsha

On Fri, Aug 5, 2016 at 8:02 AM P. Taylor Goetz <ptgo...@apache.org> wrote:

> +1 (binding)
>
> Ran on a 3-node cluster and verified fixes.
>
> -Taylor
>
> On Jul 26, 2016, at 4:04 PM, P. Taylor Goetz <ptgo...@apache.org> wrote:
>
> This is a call to vote on releasing Apache Storm 1.0.2 (rc4)
>
> Full list of changes in this release:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=c9768c154166b6217ec7bffc4a9aa73e90f2339d
>
> The tag/commit to be voted upon is v1.0.2:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=54f319fd56c33437364c550a800ac9e6fe058b95
>
> The source archive being voted upon can be found here:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.2-rc4/apache-storm-1.0.2-src.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.2-rc4/
>
> The release artifacts are signed with the following key:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>
> The Nexus staging repository for this release is:
>
> https://repository.apache.org/content/repositories/orgapachestorm-1038
>
> Please vote on releasing this package as Apache Storm 1.0.2.
>
> When voting, please list the actions taken to verify the release.
>
> This vote will be open for at least 72 hours.
>
> [ ] +1 Release this package as Apache Storm 1.0.2
> [ ]  0 No opinion
> [ ] -1 Do not release this package because...
>
> Thanks to everyone who contributed to this release.
>
> -Taylor
>
>
>

Re: New Committers/PMC Members: John Fang and Abhishek Agarwal

[Harsha]

Congrats John & Abhishek !

-Harsha

On Tue, Jun 7, 2016, at 02:14 PM, Aaron.Dossett wrote:
> I second that!
> 
> On 6/7/16, 3:12 PM, "Bobby Evans" <ev...@yahoo-inc.com.INVALID> wrote:
> 
> >Congratulations to both of you.  Well deserved.
> > - Bobby 
> >
> >On Tuesday, June 7, 2016 3:02 PM, P. Taylor Goetz <ptgo...@gmail.com>
> >wrote:
> > 
> >
> > Please join me in welcoming John Fang and Abhishek Agarwal as a new
> >Apache Storm Committers and PMC members.
> >
> >John and Abhishek have demonstrated a strong commitment to the Apache
> >Storm community through active participation and mentoring on the Storm
> >mailing lists. They have also authored many enhancements and bug fixes
> >spanning both Storm¹s core codebase, as well as a numerous integration
> >components.
> >
> >Welcome John and Abhishek!
> >
> >-Taylor
> >
> >  
> 

Re: [DISCUSSION] Squashing commits before merging in.

[Harsha]

Thanks Jungtaek for starting this thread. 
Its on of my pet peeves that our commit log is very noisy. It would be
great if we can squash commits into one commit message before merging
the PR. This would keep the commit log clean and if someone wants to
refer to particular JIRA patch its easy to go back to that patch if
we've squashed the commits. Also cherry-picking will be easier .  
In Kakfa community we use this process and I find it very helpful .
I changed the kafka-merge-pr tool to work with storm  and PR
available here https://github.com/apache/storm/pull/1468 . I
commented out the line that pushes to apache git repo until I get
some feedback but you can safely run it and see how the final commit
looks like.

Thanks,
Harsha 

On Sun, May 15, 2016, at 07:04 PM, Jungtaek Lim wrote:
> Moving opinions from other threads,
> 
> Abhishek
> 
> 1* - Squashing commits is more of a guideline. Committers, specifically,
> should ensure that the patches are squashed or volunteer to do that as
> merging otherwise :) These guidelines along with 'how to git squash' can
> be
> added to http://storm.apache.org/contribute/Contributing-to-Storm.html.
> 
> Aaron
> 
> 1* I like the idea of squashing to a single commit and tagging that
> commit
> and PR with the JIRA ID.
> 
> Thanks,
> Jungtaek Lim (HeartSaVioR)
> 
> 2016년 5월 16일 (월) 오전 11:01, Jungtaek Lim <kabh...@gmail.com>님이 작성:
> 
> > This thread was forgotten, but I would like to discuss this again and
> > reach final result.
> >
> > 1. I believe learning from other projects is the best way to evaluate our
> > process and change if others are better.
> >
> > Let's take a look at commits page on some projects -
> > Hadoop <https://github.com/apache/hadoop/commits/trunk>, HBase
> > <https://github.com/apache/hbase/commits/master>, Kafka
> > <https://github.com/apache/kafka/commits/trunk>, Spark
> > <https://github.com/apache/spark/commits/master>, Flink
> > <https://github.com/apache/flink/commits/master>, Calcite
> > <https://github.com/apache/calcite/commits/master>, Zeppelin (incubator)
> > <https://github.com/apache/incubator-zeppelin/commits/master>.
> >
> > You may notice that most of commit logs are started to JIRA ID (which is
> > what some of us are done implicitly), and there's one commit per one issue,
> > which may be squashed. (there seems to be some exceptions but seems minor)
> > Moreover, recently Github includes this feature to UI and many projects
> > are using it.
> >
> > 2. Since we merge commits, merge commit is placed to last before CHANGELOG
> > commit, but origin commits could be placed to outside of 1 page of commits
> > page. IMO, it is not important when contributors are committing their
> > commits are. Rearrange to be placed to latest are more deterministic. (Btw,
> > just rebasing also do the trick for this.)
> >
> > 3. It's also great for getting rid of maintaining CHANGELOG by hand. I
> > really don't like doing this by hand now, and squashed commits are
> > CHANGELOG. We don't even revert two commits (merge commit / CHANGELOG).
> >
> > 4. If merger squashes commits into one when merging, we can see author and
> > merger of commits for that commit.
> >
> > I'm in favor of squashing commits, but some contributors could have hard
> > time to squash by themselves so I think it would be better to squash in
> > merging step. (committers are in responsible to squash, which should be
> > easier with script tool)
> >
> > 2015년 5월 12일 (화) 오전 2:16, Bobby Evans <ev...@yahoo-inc.com.invalid>님이 작성:
> >
> >> I am fine with doing this, but I also don't see it as that big of a
> >> deal.  Most of the time if I want to cherry pick something I will cherry
> >> pick the merge commit for the JIRA instead of each individual commit.  The
> >> only place for me that this is really nice is with blame, where I can see
> >> the exact JIRA that something is a part of, without having to trace it back
> >> to the merge commit and JIRA it was a part of. The downside is that now I
> >> have to run more commands when committing.  So for me, the questions is how
> >> often do I commit vs cherry pick, and I honestly could go either way.
> >>
> >> - Bobby
> >>
> >>
> >>
> >>  On Friday, May 8, 2015 7:31 PM, Harsha <st...@harsha.io> wrote:
> >>
> >>
> >>  Thanks for the reply.
> >>
> >> 1)  Major point is that commit's changeset size will be completely
> >> relied on
> >> > JIRA issue's size, 

Re: STORM JIRA being spammed - CANNOT comment on JIRA (as a result of the counter spam measures)

[Harsha]

Erik,
If we stop adding comments to JIRA there is no central place to
look at the JIRA and patch history. One has to go to github and
figure out the closed PR and see the history of the patch. Today
if I go to jira I can see all the comments and patch link as
well. At least for us this is helpful to look at what went
though the patch when a user hits a similar issue. 
Thanks,
Harsha

On Mon, May 16, 2016, at 01:22 PM, Erik Weathers wrote:
> :_(   I was happy for a second that the comments weren't getting added to
> JIRA.  I still have not seen a convincing argument for duplicating code
> review comments into the JIRA issues.
> 
> - Erik
> 
> On Mon, May 16, 2016 at 11:33 AM, P. Taylor Goetz <ptgo...@gmail.com>
> wrote:
> 
> > I worked with INFRA and we were able to resolve this. Unfortunately
> > comments made on github before the fix won’t be replicated to JIRA so we’ll
> > have to look at both JIRA and github for recent comments on issues.
> >
> > -Taylor
> >
> > > On May 16, 2016, at 1:48 PM, P. Taylor Goetz <ptgo...@gmail.com> wrote:
> > >
> > > Thanks for the heads up Abhishek. I’ll work with INFRA to see if we can
> > get this resolved as it is kind of disruptive.
> > >
> > > -Taylor
> > >
> > >> On May 16, 2016, at 1:18 PM, Abhishek Agarwal <abhishc...@gmail.com>
> > wrote:
> > >>
> > >> @Taylor - Looks like it didn't work. I don't see github comments getting
> > >> published to the JIRA.  Example
> > >> https://issues.apache.org/jira/browse/STORM-1755
> > >> https://github.com/apache/storm/pull/1386#issuecomment-217697370 (This
> > >> didn't get published to the JIRA)
> > >>
> > >> On Fri, May 13, 2016 at 9:42 PM, P. Taylor Goetz <ptgo...@gmail.com>
> > wrote:
> > >>
> > >>> I added ASF Github Bot to the list of contributors. Hopefully that will
> > >>> fix the issue.
> > >>>
> > >>> On May 13, 2016, at 12:10 PM, P. Taylor Goetz <ptgo...@gmail.com>
> > wrote:
> > >>>
> > >>>
> > >>> On May 13, 2016, at 4:11 AM, Jungtaek Lim <kabh...@gmail.com> wrote:
> > >>>
> > >>> Now github comments are not automatically posted to JIRA, which is less
> > >>> duplicate and more quiet.
> > >>>
> > >>>
> > >>> I think this is due to the spam countermeasures, let me see if I can
> > fix
> > >>> this. This is something we want, since we also want to capture
> > comments,
> > >>> especially +1s from committers.
> > >>>
> > >>> -Taylor
> > >>>
> > >>>
> > >>>
> > >>
> > >>
> > >> --
> > >> Regards,
> > >> Abhishek Agarwal
> > >
> >
> >

Re: Revisiting forgotten discussions regarding more convenience way to maintain project

[Harsha]

Jungtaek,
  If we didn't get any conclusion. Lets continue discussion on
  the same thread. You can just reply to that.
Lets not merge all discussions into major thread where we can't even
track which subject we are discussing.

Thanks,
Harsha

On Fri, May 13, 2016, at 08:56 PM, Harsha wrote:
> 3* - I also get one extra email from notificati...@github.com if I am
> participating in a pull request from notificati...@github.com. It will
> be
> great to avoid that as well. By the way, removing notifications from
> Github
> means that PRs with no JIRA id might go unnoticed for long time.
> 
> -1. notifications important for everyone else. if you are getting
> spammed by this  create a mail rule.
> 
> -Harsha
> 
> On Fri, May 13, 2016, at 07:47 AM, Aaron.Dossett wrote:
> > I like the idea of squashing to a single commit and tagging that commit
> > and PR with the JIRA ID. Agree that limiting duplicate emails would be
> > good.
> > 
> > On 5/13/16, 6:17 AM, "Abhishek Agarwal" <abhishc...@gmail.com> wrote:
> > 
> > >1* - Squashing commits is more of a guideline. Committers, specifically,
> > >should ensure that the patches are squashed or volunteer to do that as
> > >merging otherwise :) These guidelines along with 'how to git squash' can
> > >be
> > >added to http://storm.apache.org/contribute/Contributing-to-Storm.html.
> > >
> > >The above page says that for minor changes, there is no need to create a
> > >JIRA. The definition of what is minor, is of course vague. In my
> > >experience, there were very few changes which were made without a
> > >corresponding JIRA so JIRA creation might as well be made mandatory. If
> > >each commit is tagged with a JIRA id, it becomes easier to track that
> > >change. I have also seen cases where there were multiple commits under
> > >single JIRA but none of them were tagged with JIRA id.
> > >
> > >3* - I also get one extra email from notificati...@github.com if I am
> > >participating in a pull request from notificati...@github.com. It will be
> > >great to avoid that as well. By the way, removing notifications from
> > >Github
> > >means that PRs with no JIRA id might go unnoticed for long time.
> > >
> > >
> > >On Fri, May 13, 2016 at 3:43 PM, Jungtaek Lim <kabh...@gmail.com> wrote:
> > >
> > >> Hi devs,
> > >>
> > >> I just feel we are following a bit hard way to maintain project. Some
> > >>of us
> > >> (including me) pointed out earlier by initiating discussion thread, but
> > >>it
> > >> was forgotten. I found some major items which are worth to discuss.
> > >>
> > >> - [DISCUSSION] Squashing commits before merging in
> > >> <
> > >> 
> > >>https://mail-archives.apache.org/mod_mbox/storm-dev/201505.mbox/%3CetPan.
> > >>554d273a.d58469.139cb@HW10843.local%3E
> > >> >
> > >> - [DISCUSSION] More convenient way to maintain committer / contributor
> > >>list
> > >> and changelogs
> > >> <
> > >> 
> > >>https://mail-archives.apache.org/mod_mbox/storm-dev/201512.mbox/%3CCAF510
> > >>8h9Y-tsHhYukP-A=nai6xmmljzzt_txdzkih22bwxj...@mail.gmail.com%3E
> > >> >
> > >> - does anyone else hate the verbose logging of all PR comments in the
> > >>Storm
> > >> JIRAs?
> > >> <
> > >> 
> > >>https://mail-archives.apache.org/mod_mbox/storm-dev/201509.mbox/%3CCAO5KY
> > >>W-=kzzs12gd8rr+8rwsllg32fbia6fevqaytxee_ty...@mail.gmail.com%3E
> > >> >
> > >>
> > >> I hope that we can come up with final result.
> > >>
> > >> Let me give my opinion about those three items.
> > >>
> > >> *1. Squashing commits before merging in*
> > >>
> > >> I believe learning from other projects is the best way to evaluate our
> > >> process and change if others are better.
> > >>
> > >> Let's take a look at commits page on some projects -
> > >> Hadoop <https://github.com/apache/hadoop/commits/trunk>, HBase
> > >> <https://github.com/apache/hbase/commits/master>, Kafka
> > >> <https://github.com/apache/kafka/commits/trunk>, Spark
> > >> <https://github.com/apache/spark/commits/master>, Flink
> > >> <https://github.com/apache/flink/commits/master>, Calcite
> > >> <https://github.com/apache/cal

Re: Revisiting forgotten discussions regarding more convenience way to maintain project

[Harsha]

3* - I also get one extra email from notificati...@github.com if I am
participating in a pull request from notificati...@github.com. It will
be
great to avoid that as well. By the way, removing notifications from
Github
means that PRs with no JIRA id might go unnoticed for long time.

-1. notifications important for everyone else. if you are getting
spammed by this  create a mail rule.

-Harsha

On Fri, May 13, 2016, at 07:47 AM, Aaron.Dossett wrote:
> I like the idea of squashing to a single commit and tagging that commit
> and PR with the JIRA ID. Agree that limiting duplicate emails would be
> good.
> 
> On 5/13/16, 6:17 AM, "Abhishek Agarwal" <abhishc...@gmail.com> wrote:
> 
> >1* - Squashing commits is more of a guideline. Committers, specifically,
> >should ensure that the patches are squashed or volunteer to do that as
> >merging otherwise :) These guidelines along with 'how to git squash' can
> >be
> >added to http://storm.apache.org/contribute/Contributing-to-Storm.html.
> >
> >The above page says that for minor changes, there is no need to create a
> >JIRA. The definition of what is minor, is of course vague. In my
> >experience, there were very few changes which were made without a
> >corresponding JIRA so JIRA creation might as well be made mandatory. If
> >each commit is tagged with a JIRA id, it becomes easier to track that
> >change. I have also seen cases where there were multiple commits under
> >single JIRA but none of them were tagged with JIRA id.
> >
> >3* - I also get one extra email from notificati...@github.com if I am
> >participating in a pull request from notificati...@github.com. It will be
> >great to avoid that as well. By the way, removing notifications from
> >Github
> >means that PRs with no JIRA id might go unnoticed for long time.
> >
> >
> >On Fri, May 13, 2016 at 3:43 PM, Jungtaek Lim <kabh...@gmail.com> wrote:
> >
> >> Hi devs,
> >>
> >> I just feel we are following a bit hard way to maintain project. Some
> >>of us
> >> (including me) pointed out earlier by initiating discussion thread, but
> >>it
> >> was forgotten. I found some major items which are worth to discuss.
> >>
> >> - [DISCUSSION] Squashing commits before merging in
> >> <
> >> 
> >>https://mail-archives.apache.org/mod_mbox/storm-dev/201505.mbox/%3CetPan.
> >>554d273a.d58469.139cb@HW10843.local%3E
> >> >
> >> - [DISCUSSION] More convenient way to maintain committer / contributor
> >>list
> >> and changelogs
> >> <
> >> 
> >>https://mail-archives.apache.org/mod_mbox/storm-dev/201512.mbox/%3CCAF510
> >>8h9Y-tsHhYukP-A=nai6xmmljzzt_txdzkih22bwxj...@mail.gmail.com%3E
> >> >
> >> - does anyone else hate the verbose logging of all PR comments in the
> >>Storm
> >> JIRAs?
> >> <
> >> 
> >>https://mail-archives.apache.org/mod_mbox/storm-dev/201509.mbox/%3CCAO5KY
> >>W-=kzzs12gd8rr+8rwsllg32fbia6fevqaytxee_ty...@mail.gmail.com%3E
> >> >
> >>
> >> I hope that we can come up with final result.
> >>
> >> Let me give my opinion about those three items.
> >>
> >> *1. Squashing commits before merging in*
> >>
> >> I believe learning from other projects is the best way to evaluate our
> >> process and change if others are better.
> >>
> >> Let's take a look at commits page on some projects -
> >> Hadoop <https://github.com/apache/hadoop/commits/trunk>, HBase
> >> <https://github.com/apache/hbase/commits/master>, Kafka
> >> <https://github.com/apache/kafka/commits/trunk>, Spark
> >> <https://github.com/apache/spark/commits/master>, Flink
> >> <https://github.com/apache/flink/commits/master>, Calcite
> >> <https://github.com/apache/calcite/commits/master>, Zeppelin (incubator)
> >> <https://github.com/apache/incubator-zeppelin/commits/master>.
> >>
> >> You may notice that most of commit logs are started to JIRA ID (which is
> >> what some of us are done implicitly), and there's one commit per one
> >>issue,
> >> which may be squashed. (there seems to be some exceptions but seems
> >>minor)
> >> Moreover, recently Github includes this feature to UI and many projects
> >>are
> >> using it.
> >>
> >> I'm in favor of squashing commits, but some contributors could have hard
> >> time to squash by themselves so I think it would be better to squash in
> &

Re: STORM JIRA being spammed - CANNOT comment on JIRA (as a result of the counter spam measures)

[Harsha]

Bobby,
 Can you check if I am part of commiters list. I don't see admin
 page for STORM project jira.
Thanks,
Harsha

On Thu, May 12, 2016, at 02:23 PM, Hugo Da Cruz Louro wrote:
> Thanks Bobby. It’s working now. 
> 
> > On May 12, 2016, at 1:48 PM, Bobby Evans <ev...@yahoo-inc.com.INVALID> 
> > wrote:
> > 
> > done - Bobby 
> > 
> >On Thursday, May 12, 2016 3:17 PM, Roshan Naik <ros...@hortonworks.com> 
> > wrote:
> > 
> > 
> > Myself too. Thanks.
> > -Roshan 
> > 
> > 
> > On 5/12/16, 1:14 PM, "Bobby Evans" <ev...@yahoo-inc.com.INVALID> wrote:
> > 
> >> Sorry about that you should be added as a contributor now.  Is there
> >> anyone else that I should add as a contributor?
> >> Also we have not been maintaining the commiters ACL very well either.
> >> Are there any commiter/PMC out there that want to be added to the
> >> committer/PMC roles.  There really isn't much different between the
> >> roles, but I am happy to adjust it if needed.
> >> - Bobby 
> >> 
> >> On Thursday, May 12, 2016 2:57 PM, Hugo Da Cruz Louro
> >> <hlo...@hortonworks.com> wrote:
> >> 
> >> 
> >> Hi,
> >> 
> >> I understand that there is a wave of spam going on, but the measure that
> >> was taken, which limits active contributors to comment on JIRA is not
> >> very practical. I have several open JIRA issues that I am working on, or
> >> that I am interested in contributing to, that I cannot comment on right
> >> now.
> >> 
> >> Can I get my access back, please.
> >> 
> >> GitHub: hmcl
> >> Apache JIRA Id: hmclouro
> >> 
> >> Thanks,
> >> Hugo
> >> 
> >>   
> > 
> > 
> 

Re: [VOTE] Release Apache Storm 0.10.1 (rc2)

[Harsha]

Roshan,
This is 0.10 maint release . We are not going to include the
new features that went into 1.0.

+1 (binding)
Deployed on a 3-node cluster
Ran example topologies
verified signatures of binaries.

Thanks,
Harsha

On Tue, May 3, 2016, at 05:16 PM, Roshan Naik wrote:
> This is missing HdfsSpout   (STORM-1199)
>   
> -roshan
> 
> 
> On 5/3/16, 3:39 PM, "Hugo Da Cruz Louro" <hlo...@hortonworks.com> wrote:
> 
> >+1 (non binding)
> >- Found a very minor bug in the storm-solr test topology, for which there
> >is a very simple workaround and by NO means a blocker for the release.
> >- unzipped .zip sources and binaries
> >- mvn clean install in unzipped sources
> >- created uber jar for storm-solr
> >- tested storm-solr in local cluster mode and remode cluster mode
> >- ran test examples
> >
> >On May 3, 2016, at 12:04 AM, Arun Mahadevan
> ><ar...@apache.org<mailto:ar...@apache.org>> wrote:
> >
> >+1 (binding)
> >- Extracted binaries
> >- Ran sample topologies
> >- Browsed storm UI
> >
> >Thanks,
> >Arun
> >
> >
> >
> >
> >On 4/28/16, 12:01 PM, "Jungtaek Lim"
> ><kabh...@gmail.com<mailto:kabh...@gmail.com>> wrote:
> >
> >+1 (binding)
> >
> >- testing with source distribution : OK
> >- unzip : OK
> >- building from source dist : OK
> >  - how to build: running `mvn -P all-tests clean install` on unzipped
> >source dist.
> >
> >- testing with binary distribution (one machine) : OK
> >- launch daemons : OK
> >- run RollingTopWords (local) : OK
> >- run RollingTopWords (remote) : OK
> >  - activate / deactivate / rebalance / kill : OK
> >
> >Thanks,
> >Jungtaek Lim
> >
> >2016년 4월 28일 (목) 오전 3:29, P. Taylor Goetz
> ><ptgo...@apache.org<mailto:ptgo...@apache.org>>님이 작성:
> >
> >This is a call to vote on releasing Apache Storm 0.10.1 (rc2)
> >
> >Full list of changes in this release:
> >
> >
> >https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGEL
> >OG.md;hb=8179921a569b6cf1d97798eed8e7b03b131bc495
> >
> >The tag/commit to be voted upon is v0.10.1:
> >
> >
> >https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=ddf051149f3c3
> >86342937efdabdaf45694602dd1;hb=8179921a569b6cf1d97798eed8e7b03b131bc495;a=
> >tree;h=26291835f22474506f0fe90b0459eab0d00bf4a9;hb=f0d3eae7395b3ee036b94b9
> >22707f74868ba6190
> >
> >The source archive being voted upon can be found here:
> >
> >
> >https://dist.apache.org/repos/dist/dev/storm/apache-storm-0.10.1-rc2/apach
> >e-storm-0.10.1-src.tar.gz
> >
> >Other release files, signatures and digests can be found here:
> >
> >https://dist.apache.org/repos/dist/dev/storm/apache-storm-0.10.1-rc2/
> >
> >The release artifacts are signed with the following key:
> >
> >
> >https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb
> >=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >
> >The Nexus staging repository for this release is:
> >
> >https://repository.apache.org/content/repositories/orgapachestorm-1031/
> >
> >Please vote on releasing this package as Apache Storm 0.10.1.
> >
> >When voting, please list the actions taken to verify the release.
> >
> >This vote will be open for at least 72 hours.
> >
> >[ ] +1 Release this package as Apache Storm 0.10.1
> >[ ]  0 No opinion
> >[ ] -1 Do not release this package because...
> >
> >Thanks to everyone who contributed to this release.
> >
> >-Taylor
> >
> >
> >
> >
> 

Re: [VOTE] Release Apache Storm 1.0.1 (rc3)

[Harsha]

+1 (binding)
  - Deployed 3-node cluster  and example topologies
  - Verified the binaries signature.
Thanks,
Harsha

On Sun, May 1, 2016, at 07:11 PM, Jungtaek Lim wrote:
> +1 (binding)
> 
> - build succeed from source code
> - binary extracted well, and supervisor launches worker fine even though
> JAVA_HOME is unset (STORM-1741) / also checked when JAVA_HOME is set
> 
> Thanks all of efforts you all have been done with this release.
> 
> Thanks,
> Jungtaek Lim (HeartSaVioR)
> 
> 
> 2016년 4월 30일 (토) 오전 10:08, Xin Wang <data.xinw...@gmail.com>님이 작성:
> 
> > +1 (Non binding)
> >
> > 2016-04-30 5:28 GMT+08:00 P. Taylor Goetz <ptgo...@apache.org>:
> >
> > > If at first you don't succeed... ;)
> > >
> > > This is a call to vote on releasing Apache Storm 1.0.1 (rc3)
> > >
> > > Full list of changes in this release:
> > >
> > >
> > >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=b5c16f919ad4099e6fb25f1095c9af8b64ac9f91
> > >
> > > The tag/commit to be voted upon is v1.0.1:
> > >
> > >
> > >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=14b2ac0fe33759a50a2392ebdc7ad3821d43b89f;hb=b5c16f919ad4099e6fb25f1095c9af8b64ac9f91
> > >
> > > The source archive being voted upon can be found here:
> > >
> > >
> > >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.1-rc3/apache-storm-1.0.1-src.tar.gz
> > >
> > > Other release files, signatures and digests can be found here:
> > >
> > > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.1-rc3/
> > >
> > > The release artifacts are signed with the following key:
> > >
> > >
> > >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> > >
> > > The Nexus staging repository for this release is:
> > >
> > > https://repository.apache.org/content/repositories/orgapachestorm-1033/
> > >
> > > Please vote on releasing this package as Apache Storm 1.0.1.
> > >
> > > When voting, please list the actions taken to verify the release.
> > >
> > > This vote will be open for at least 72 hours.
> > >
> > > [ ] +1 Release this package as Apache Storm 1.0.1
> > > [ ]  0 No opinion
> > > [ ] -1 Do not release this package because...
> > >
> > > Thanks to everyone who contributed to this release.
> > >
> > > -Taylor
> > >
> >

Re: [VOTE] Release Apache Storm 1.0.1 (rc2)

[Harsha]

storm-env.sh introduced if users want override any environment
variables. we can either comment out the line as default or what Taylor
proposed looks good too.
-Harsha

On Thu, Apr 28, 2016, at 08:27 AM, P. Taylor Goetz wrote:
> HDP uses storm-env.sh, which is where it came from. That makes sense
> because I’ve noticed HDP builds exhibit the same issue.
> 
> That line in storm-env.sh should probably be:
> 
> if [ -n "$JAVA_HOME" ]; then export JAVA_HOME=${JAVA_HOME}; fi
> 
> The fastest route would be to simply revert STORM-1706, but the correct
> thing to do might be to add the above to storm-env.sh (which would delay
> the release by 24 hrs.).
> 
> Thoughts?
> 
> -Taylor
> 
> 
> 
> > On Apr 28, 2016, at 10:11 AM, Jungtaek Lim <kabh...@gmail.com> wrote:
> > 
> > Yeah, great finding. I agree it's critical so we should take an action.
> > 
> > 1. storm-env.sh should be fixed to not re-set JAVA_HOME (btw, what exactly
> > is the meaning?)
> > 2. rename storm-env.sh to storm-env.sh.example (and also apply 1)
> > 3. just revert STORM-1706.
> > 
> > Anything would be fine for me.
> > 
> > 2016년 4월 28일 (목) 오후 10:58, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
> > 
> >> STORM-1706 introduced it. Before 1.0.1 we weren’t including
> >> `storm-env.sh`. That file does the following:
> >> 
> >> export JAVA_HOME=${JAVA_HOME}
> >> 
> >> Which, if JAVA_HOME is not set, will set it, but leave it empty. So the
> >> clj `if (nil? java-home)` will evaluate to false and we’ll end up with
> >> `/bin/java` as the java command.
> >> 
> >> I think we need to revert STORM-1706.
> >> 
> >> -Taylor
> >> 
> >>> On Apr 28, 2016, at 9:26 AM, P. Taylor Goetz <ptgo...@gmail.com> wrote:
> >>> 
> >>> Yeah, that section hasn’t changed. I think it’s a change further up the
> >> stack.
> >>> 
> >>>> On Apr 28, 2016, at 9:05 AM, Jungtaek Lim <kabh...@gmail.com> wrote:
> >>>> 
> >>>> It's here.
> >>>> 
> >>>> (defn jvm-cmd [cmd]
> >>>> (let [java-home (.get (System/getenv) "JAVA_HOME")]
> >>>>  (if (nil? java-home)
> >>>>cmd
> >>>>(str java-home file-path-separator "bin" file-path-separator cmd
> >>>> 
> >>>> If JAVA_HOME isn't set to system environment, it may works as what we
> >> want.
> >>>> But if it's set to empty string, "/bin/java" will be called.
> >>>> 
> >>>> 
> >>>> 
> >>>> 2016년 4월 28일 (목) 오후 10:01, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
> >>>> 
> >>>>> 
> >>>>> 
> >>>>>> On Apr 28, 2016, at 2:30 AM, Jungtaek Lim <kabh...@gmail.com> wrote:
> >>>>>> 
> >>>>>> - if we don't set JAVA_HOME, supervisor runs "/bin/java" instead of
> >>>>> "java"
> >>>>>> when it launches worker with non-secure mode.
> >>>>> 
> >>>>> That seems like a regression we don't want. I'll look into where that
> >> came
> >>>>> from
> >>>>> 
> >>>>> -Taylor
> >>> 
> >> 
> >> 
> 
> Email had 1 attachment:
> + signature.asc
>   1k (application/pgp-signature)

Re: [VOTE] Release Apache Storm 0.10.1 (rc1)

[Harsha]

Same as 1.0.1 release. I would like to get those two patches in for
0.10.1 release as well.
Thanks,
Harsha

On Tue, Apr 26, 2016, at 05:27 PM, Jungtaek Lim wrote:
> Same on 1.0.1 RC1, but only STORM-1731
> <https://issues.apache.org/jira/browse/STORM-1731> is affected to 0.10.1.
> 
> Below is the repetitive message from 1.0.1 RC1 for tracking archive later
> easier.
> 
> I've observed the points of performance hit (STORM-1729
> <https://issues.apache.org/jira/browse/STORM-1729>), and it seems huge.
> 
> Since patches are small and straightforward, I would love to include this
> as 0.10.1 so that users maintaining high-traffic topologies can enjoy the
> benefit.
> But it's only when performance hit / recovery is observed with stable
> environment. I provided some test reports there, but I want to see the
> test
> reports from other environments.
> 
> Thanks,
> Jungtaek Lim
> 
> 2016년 4월 27일 (수) 오전 4:52, P. Taylor Goetz <ptgo...@apache.org>님이 작성:
> 
> > ***Please note the version! There is another VOTE open that could easily
> > be confused with this one.***
> >
> > This is a call to vote on releasing Apache Storm 0.10.1 (rc1)
> >
> > Full list of changes in this release:
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=f0d3eae7395b3ee036b94b922707f74868ba6190
> >
> > The tag/commit to be voted upon is v0.10.1:
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=26291835f22474506f0fe90b0459eab0d00bf4a9;hb=f0d3eae7395b3ee036b94b922707f74868ba6190
> >
> > The source archive being voted upon can be found here:
> >
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-0.10.1-rc1/apache-storm-0.10.1-src.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-0.10.1-rc1/
> >
> > The release artifacts are signed with the following key:
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >
> > The Nexus staging repository for this release is:
> >
> > https://repository.apache.org/content/repositories/orgapachestorm-1029/
> >
> > Please vote on releasing this package as Apache Storm 0.10.1.
> >
> > When voting, please list the actions taken to verify the release.
> >
> > This vote will be open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache Storm 0.10.1
> > [ ]  0 No opinion
> > [ ] -1 Do not release this package because...
> >
> > Thanks to everyone who contributed to this release.
> >
> > -Taylor
> >

Re: [VOTE] Release Apache Storm 1.0.1

[Harsha]

I am +1 on doing another release candidate with these fixes in. Nice
work Jungtaek.

Thanks,
Harsha

On Tue, Apr 26, 2016, at 05:23 PM, Jungtaek Lim wrote:
> I've observed the points of performance hit (STORM-1729
> <https://issues.apache.org/jira/browse/STORM-1729>, STORM-1731
> <https://issues.apache.org/jira/browse/STORM-1731>), and it seems huge.
> 
> Since patches are small and straightforward, I would love to include this
> as 1.0.1 so that users maintaining high-traffic topologies can enjoy the
> benefit.
> But it's only when performance hit / recovery is observed with stable
> environment. I provided some test reports there, but I want to see the
> test
> reports from other environments.
> 
> Thanks,
> Jungtaek Lim
> 
> 2016년 4월 27일 (수) 오전 4:46, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
> 
> > Two of the links were wrong, please use the following links:
> >
> > The source archive being voted upon can be found here:
> >
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.1-rc1/apache-storm-1.0.1-src.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.1-rc1/
> >
> > Sorry for the inconvenience.
> >
> > -Taylor
> >
> > On Apr 26, 2016, at 12:23 PM, P. Taylor Goetz <ptgo...@gmail.com> wrote:
> >
> > This is a call to vote on releasing Apache Storm 1.0.1 (rc1)
> >
> > Full list of changes in this release:
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=7520b9ae13c4a4fcd823e2d636e9b2cc1cbb7e6f
> >
> > The tag/commit to be voted upon is v1.0.1:
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=9cc5c11896dd83402f6a6836d63d50564a94c932;hb=7520b9ae13c4a4fcd823e2d636e9b2cc1cbb7e6f
> > The source archive being voted upon can be found here:
> >
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.1-rc2/apache-storm-1.0.1-src.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.1-rc2/
> >
> > The release artifacts are signed with the following key:
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >
> > The Nexus staging repository for this release is:
> >
> > https://repository.apache.org/content/repositories/orgapachestorm-1029/
> >
> > Please vote on releasing this package as Apache Storm 1.0.1.
> >
> > When voting, please list the actions taken to verify the release.
> >
> > This vote will be open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache Storm 1.0.1
> > [ ]  0 No opinion
> > [ ] -1 Do not release this package because...
> >
> > Thanks to everyone who contributed to this release.
> >
> > -Taylor
> >
> >
> >

Re: Publishing cluster stats on Apache Storm

[Harsha]

Jungtaek,
  Ideally we should avoid using NimbusClient as much as
  possible for a polling reporter like this. Can we have
  plug gable interface that can be run inside the nimbus at
  configured interval . Since it will be plugged into nimbus
  we would have all cluster details including topologies as
  well.

Thanks,
Harsha


On Mon, Apr 18, 2016, at 10:18 PM, Jungtaek Lim wrote:
> Cody,
> 
> Great idea! Let's try it out.
> I'll send the mail to user@ regarding use cases or concerns of metrics on
> Apache Storm. We may have a luck to find brilliant ideas or use cases we
> didn't know.
> 
> 2016년 4월 19일 (화) 오후 2:07, Cody Innowhere <e.neve...@gmail.com>님이 작성:
> 
> > Jungtaek,
> > Thanks for your explanation, still +1 for your proposal.
> > Moreover, I'd like to hear storm users about this, especially use cases for
> > metrics so that we can see what's best to do this.
> >
> > On Tue, Apr 19, 2016 at 11:19 AM, Jungtaek Lim <kabh...@gmail.com> wrote:
> >
> > > Cody,
> > > Thanks for giving an opinion.
> > >
> > > I guess there was a miscommunication between you and me.
> > > This shouldn't be related to topology. Plugin should be set up via
> > cluster
> > > configuration, and singular daemon like Nimbus (master) or UI should
> > > publish cluster metrics to the plugin.
> > > I agree Nimbus should have this feature, but only master should publish
> > the
> > > cluster metrics so we should handle it.
> > >
> > > Btw, since we don't have built-in metrics storage, we can't provide query
> > > API with time-series fashion.
> > > I guess publishing (pushing) seems the better way what we can address
> > > without huge modification.
> > > (Did you mean Nimbus has query API which queries the external storage?)
> > >
> > > Please note that it's an idea for improvement against 1.x, not 2.0.
> > > In phase 2 we should evaluate and replace metrics feature with JStorm or
> > > another after porting to Java.
> > >
> > > Thanks,
> > > Jungtaek Lim (HeartSaVioR)
> > >
> > > 2016년 4월 19일 (화) 오전 11:59, Cody Innowhere <e.neve...@gmail.com>님이 작성:
> > >
> > > > I'm +1 for publishing cluster stats, but don't quite understand your
> > > > implementation.
> > > > Do you mean that a topology which wants cluster stats can get cluster
> > > stats
> > > > by implementing the pluggable consumer/reporter?
> > > > Personally I would prefer to put this role into nimbus (can be
> > pluggable
> > > > too) since it's the responsibility of nimbus to take care of all
> > > > topologies, naturally all topology stats. Then we expose API's from
> > > nimbus
> > > > for external queries.
> > > >
> > > > On Tue, Apr 19, 2016 at 9:47 AM, Jungtaek Lim <kabh...@gmail.com>
> > wrote:
> > > >
> > > > > Hi devs,
> > > > >
> > > > > While Storm publishes topology metrics by metrics consumers, Storm
> > > > doesn't
> > > > > publish cluster metrics any way.
> > > > >
> > > > > Therefore, I'd like to introduce the feature of publishing cluster
> > > stats
> > > > to
> > > > > pluggable consumers so that users can also push to external storages
> > > and
> > > > > query on it or even configure dashboard.
> > > > > (For topology stats it's supported via MetricsConsumer.)
> > > > >
> > > > > I'm seeing some workarounds like having their own reporter process
> > > > polling
> > > > > cluster information from Nimbus or REST API.
> > > > > (For example, Ambari has cluster metrics reporter for Storm
> > > > > <
> > > > >
> > > >
> > >
> > https://github.com/apache/ambari/blob/trunk/ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsReporter.java
> > > > > >
> > > > > but it relies on custom build of Storm.)
> > > > > Yes it works anyway but if Storm provides the feature naturally, with
> > > > > pluggable way like MetricsConsumer it would be great for users who
> > want
> > > > to
> > > > > configure dashboard regarding this.
> > > > >
> > > > > I also saw that STORM-1158 <
> > > > > http://issues.apache.org/jira/browse/STORM-1158>
> > > > > adds metrics reporter for internal actions, but it has some
> > limitation
> > > on
> > > > > it.
> > > > >
> > > > > - It focuses how many requests the daemon receives, not cluster
> > stats.
> > > > > - It requires reporter to use codahale metrics. Moreover it shades
> > > > > codahale-metrics so other implementations of reporter may not work
> > > > properly
> > > > > if it uses other codahale-metrics plugin like ganglia.
> > > > >
> > > > > So I'm planning to add metrics feature of cluster stat by not relying
> > > on
> > > > > daemon metrics reporter, but introduce new pluggable
> > consumer/reporter.
> > > > >
> > > > > What do you think? Do you have opinion that we need to add cluster
> > > stats
> > > > to
> > > > > daemon metrics reporter?
> > > > >
> > > > > Thanks,
> > > > > Jungtaek Lim (HeartSaVioR)
> > > > >
> > > >
> > >
> >

Re: [Discussion] rolling upgrade of storm topologies with 1.0 release

[Harsha]

Hugo,
Since this is for rolling upgrade , existing topologies still be
using old kafkaSpout with old zookeeper offset storage. 
Jungtaek,
   currently kafkaSpout offsets are stored outside of
   storm.zookeeper.root so we can document saying clear the
   storm.zookeeper.root node and not touch any other nodes in
   the zookeeper. 

-Harsha

On Fri, Apr 15, 2016, at 11:17 AM, Hugo Da Cruz Louro wrote:
> Agree with Jungtaek.
> 
> With the new KafkaSpout we should not have to worry about preserving
> offsets, as it’s done by Kafka. We should encourage our customers to move
> onto the new API. I am going to add more documentation to facilitate the
> adoption of the new Kafka Spout.
> 
> Hugo
> 
> 
> > On Apr 14, 2016, at 6:22 PM, Jungtaek Lim <kabh...@gmail.com> wrote:
> > 
> > Also we need to announce the change of package, and how it will impact to
> > users.
> > (There're some questions and some issues on the mailing list and issue, as
> > you may know.)
> > 
> > I think we'd be better to add the following to announce page, with clear
> > sentences,
> > 
> > - cluster rolling upgrade is not supported, so users need to clear any
> > contents on "storm.local.dir" from all storm nodes and
> > "storm.zookeeper.root" from zookeeper.
> > (If we can guide how to preserve the offset information for KafkaSpout that
> > would be better.)
> > - Storm 1.0.0 rearranged package name to set the prefix to
> > 'org.apache.storm' which is not backward compatible.
> >  - Explanation from Abhishek is sufficient for me. (just paste.)
> >  - IMHO I love backward compatibility but we should add the notice that
> > the hack is "temporary" and users are encouraged to update their topology
> > code.
> > 
> > What do you think?
> > 
> > Thanks,
> > Jungtaek Lim (HeartSaVioR)
> > 
> > 2016년 4월 15일 (금) 오전 3:29, Harsha <st...@harsha.io>님이 작성:
> > 
> >> Erik,
> >>Thanks. Missed that.  I think we should default to that
> >>especially to support the rolling upgrade part.
> >> Thanks,
> >> Harsha
> >> 
> >> On Thu, Apr 14, 2016, at 10:47 AM, Erik Weathers wrote:
> >>> hey Harsha,
> >>> 
> >>> Did you see Abhishek's emailt to the list yesterday?
> >>> 
> >>> In the latest version, the class packages have been changed from
> >>> "backtype.storm"
> >>>> to "org.apache.storm" so the topology code compiled with older version
> >>>> won't run on the Storm 1.0.0 just like that. Backward compatibility is
> >>>> available through following configuration
> >>>> *client.jartransformer.class:
> >>>> "org.apache.storm.hack.StormShadeTransformer"*
> >>>> You need to add the above config in storm installation if you want to
> >> run
> >>>> the code compiled with older versions of storm. The config should be
> >> added
> >>>> in the machine you use to submit your topologies.
> >>>> Refer to https://issues.apache.org/jira/browse/STORM-1202 for more
> >>>> details.
> >>> 
> >>> 
> >>> On Thu, Apr 14, 2016 at 10:35 AM, Harsha <st...@harsha.io> wrote:
> >>> 
> >>>> Hi All,
> >>>> I tried to lookup if there is any previous discussion around
> >>>> this but couldn't find any. Since we moved the package naming
> >>>> to org.apache.storm any one upgrading to storm 1.0 have to go
> >>>> though the manually renaming their imports to
> >> org.apache.storm.
> >>>>  Why not we package backtype.storm and add @deprecated to this
> >>>> package and keep it in lib folder for couple of releases and
> >>>> have a plan around removing it in future versions.
> >>>> 
> >>>> Thanks,
> >>>> Harsha
> >>>> 
> >> 
> 

Re: [Discussion] rolling upgrade of storm topologies with 1.0 release

[Harsha]

Erik,
Thanks. Missed that.  I think we should default to that
especially to support the rolling upgrade part.
Thanks,
Harsha

On Thu, Apr 14, 2016, at 10:47 AM, Erik Weathers wrote:
> hey Harsha,
> 
> Did you see Abhishek's emailt to the list yesterday?
> 
> In the latest version, the class packages have been changed from
> "backtype.storm"
> > to "org.apache.storm" so the topology code compiled with older version
> > won't run on the Storm 1.0.0 just like that. Backward compatibility is
> >  available through following configuration
> > *client.jartransformer.class:
> > "org.apache.storm.hack.StormShadeTransformer"*
> > You need to add the above config in storm installation if you want to run
> > the code compiled with older versions of storm. The config should be added
> > in the machine you use to submit your topologies.
> > Refer to https://issues.apache.org/jira/browse/STORM-1202 for more
> > details.
> 
> 
> On Thu, Apr 14, 2016 at 10:35 AM, Harsha <st...@harsha.io> wrote:
> 
> > Hi All,
> >  I tried to lookup if there is any previous discussion around
> >  this but couldn't find any. Since we moved the package naming
> >  to org.apache.storm any one upgrading to storm 1.0 have to go
> >  though the manually renaming their imports to org.apache.storm.
> >   Why not we package backtype.storm and add @deprecated to this
> >  package and keep it in lib folder for couple of releases and
> >  have a plan around removing it in future versions.
> >
> > Thanks,
> > Harsha
> >

[Discussion] rolling upgrade of storm topologies with 1.0 release

[Harsha]

Hi All,
 I tried to lookup if there is any previous discussion around
 this but couldn't find any. Since we moved the package naming
 to org.apache.storm any one upgrading to storm 1.0 have to go
 though the manually renaming their imports to org.apache.storm.
  Why not we package backtype.storm and add @deprecated to this
 package and keep it in lib folder for couple of releases and
 have a plan around removing it in future versions.

Thanks,
Harsha

Re: 答复: [VOTE] Release Apache Storm 1.0.0 (rc2)

[Harsha]

+1 (binding)
Tested in a 3 node cluster with example topologies.  
Checked the binaries.

Thanks,
Harsha

On Thu, Apr 7, 2016, at 10:02 PM, John Fang wrote:
> +1
> 
> Regards
>John Fang
> 
> 
> 
> -邮件原件-
> 发件人: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID] 
> 发送时间: 2016年4月8日 2:36
> 收件人: dev@storm.apache.org
> 主题: Re: [VOTE] Release Apache Storm 1.0.0 (rc2)
> 
> Forgot to say +1 binding
>  - Bobby 
> 
> On Thursday, April 7, 2016 1:36 PM, Bobby Evans <ev...@yahoo-inc.com>
> wrote:
>  
> 
>  With the default configs everything seems to work fine.  I have run into
>  a few issues with not being able to set storm.workers.artifacts.dir, but
>  I don't think it is a blocker.
>  - Bobby 
> 
> On Wednesday, April 6, 2016 10:40 AM, Jungtaek Lim
> <kabh...@gmail.com> wrote:
>  
> 
>  +1 (binding)
> 
> - testing with source distribution : OK
>   - unzip : OK
>   - building from source dist : OK
> - how to build: running `mvn -P all-tests clean install` on unzipped
> source dist.
> 
> - testing with binary distribution (one machine) : OK
>   - launch daemons : OK
>   - run RollingTopWords (local) : OK
>   - run RollingTopWords (remote) : OK
> - activate / deactivate / rebalance / kill : OK
> - logviewer (worker dir, daemon dir) : OK
> - change log level : OK
> - thread dump, heap dump, restart worker : OK
> - log search : OK
>   - run WordCountTopology (remote) : OK
> - multiple workers and multi-lang bolt
> 
> Amazing works everyone.
> Please note that we should check the site and fix the docs before
> announcing release. I saw some broken links today so I'll open another
> thread to check and fix them.
> 
> Thanks,
> Jungtaek Lim (HeartSaVioR)
> 
> ps. Found minor issues which I think they shouldn’t block releasing. Will
> raise issues when reproducible.
> 
>   - Process is not shutting down clearly in local mode, but ‘Ctrl + C’
>   can
>   terminate the process.
>   - when searching keyword in /logviewer_search.html with daemon log
>   file,
>   is-daemon=yes parameter is gone so search shows no result.
>   - it works with /daemonlog, but since result page is
>   /logviewer_search.html, user’s next search will failed.
>   - When testing, build always fails on co-working space, cause logged
>   host ip in console log was not the ip my dev machine was assigned. I
>   expected private IP but logged ip seems to be public ip. Worker even
>   try to
>   connect to :1024 and :1025 which ports are also
>   not
>   expected.
> 
> 
> 
> 2016년 4월 6일 (수) 오후 7:56, Satish Duggana <sdugg...@hortonworks.com>님이 작성:
> 
> >
> > +1 (Non binding)
> >
> > - Retrieved source archive and built using 'mvn clean install -P all-tests’
> > - Checked Release notes etc.
> > - Built the binary package from the above source archive
> > - Ran different topologies in local cluster
> > - Created a 3 node cluster cluster with worker slots.
> >- Deployed few topologies
> >- Checked various options(like deactivate/kill/activate topology view
> > etc) and monitoring stats in the UI for those topologies.
> >- Ran storm commands on those topologies like
> > deactivate/rebalance/activate/kill with respective options.
> >- Killed some of the workers to check failover etc.
> >- Checked the logs.
> >
> >
> >
> >
> > Thanks,
> > Satish.
> >
> > On 4/6/16, 3:49 PM, "Arun Iyer on behalf of Arun Mahadevan" <
> > ai...@hortonworks.com on behalf of ar...@apache.org> wrote:
> >
> > >+1
> > >
> > >Verified build from the source archive, deployed .tar.gz binary and ran a
> > sample topology.
> > >
> > >- Arun
> > >
> > >
> > >
> > >On 4/6/16, 2:40 AM, "P. Taylor Goetz" <ptgo...@apache.org> wrote:
> > >
> > >>+1 (binding)
> > >>
> > >>- Verified build from source archive with `mvn clean install -P
> > all-tests`
> > >>- Checked LICENSE and NOTICE files
> > >>- Deployed to a small cluster and tested a variety of topologies.
> > >>
> > >>-Taylor
> > >>
> > >>> On Apr 5, 2016, at 2:38 PM, P. Taylor Goetz <ptgo...@apache.org>
> > wrote:
> > >>>
> > >>> This is a call to vote on releasing Apache Storm 1.0.0 (rc2)
> > >>>
> > >>> Full list of changes in this release:
> > >>>
> > >>>
> > https://git-wip-us.apache.org/repos/asf?p=storm.g

Re: [VOTE] Release Apache Storm 1.0.0 (rc1)

[Harsha]

Taylor,
 Lets do another release build since we have other jiras merged
 in .
Thanks,
Harsha

On Mon, Apr 4, 2016, at 02:07 PM, P. Taylor Goetz wrote:
> FYI I’ve merged fixes for following to the 1.x-branch:
>  * STORM-1670: LocalState#get(String) can throw FileNotFoundException
>  which may result supervisor.clj#sync-processes stop assigning new
>  workers/assignments
>  * STORM-1677: Test resource files are excluded from source distribution,
>  which makes logviewer-test failing
> 
> Still waiting to hear opinions on whether we should cancel this vote and
> cut a new RC.
> 
> I’m leaning toward a new RC at this point.
> 
> -Taylor
> 
> > On Apr 4, 2016, at 11:50 AM, P. Taylor Goetz <ptgo...@gmail.com> wrote:
> > 
> > The question at this point is whether we want to cancel the release vote 
> > and cut a new RC, or move forward with this one.
> > 
> > I don’t see the test issue as critical — users will still be able to build 
> > the software with the `-DskipTests=true` flag. I also feel that once the 
> > 1.0 release is out we will go into a phase where we release updates (e.g. 
> > 1.0.x, 1.x) at a relatively rapid rate.
> > 
> > I could go either way. What does everyone else think?
> > 
> > -Taylor
> > 
> > 
> >> On Apr 2, 2016, at 10:51 PM, Harsha <m...@harsha.io> wrote:
> >> 
> >> For rc2  please include STORM-1670 as well. Source distribution is
> >> important as well.
> >> -Harsha
> >> 
> >> On Sat, Apr 2, 2016, at 01:02 AM, Jungtaek Lim wrote:
> >>> Here's my test,
> >>> 
> >>> - testing with source distribution : FAIL
> >>> - unzip : OK
> >>> - building from source dist : FAIL
> >>>   - how to build: running `mvn -P all-tests clean install` on unzipped
> >>> source dist.
> >>>   - Test resource files (.log) are excluded from source distribution,
> >>> which makes logviewer-test failing.
> >>> - filed issue : https://issues.apache.org/jira/browse/STORM-1677
> >>> and
> >>> submit a patch via pull request
> >>>   - after applying PR, tests are all passed.
> >>> 
> >>> - testing with binary distribution (one machine) : OK
> >>> - launch daemons : OK
> >>> - run RollingTopWords (local) : OK (with minor issue)
> >>>   - process doesn't terminate until killing manually or pressing Ctrl +
> >>>   C
> >>> - run RollingTopWords (remote) : OK
> >>>   - activate / deactivate / rebalance / kill : OK
> >>>   - logviewer (worker dir, daemon dir) : OK
> >>>   - change log level : OK
> >>>   - thread dump, heap dump, restart worker : OK
> >>> - run WordCountTopology (remote) : OK
> >>>   - multiple workers and multi-lang bolt
> >>> 
> >>> Minor but just leaving note: I saw /log (logviewer) throwing HTTP STATUS
> >>> 500 sometimes, but I can't reproduce it. At that time, deep-search
> >>> doesn't
> >>> work at that topology, too.
> >>> 
> >>> Total
> >>> - source distribution: -1
> >>> - binary distribution: +1
> >>> 
> >>> I don't know how it is important to build with source distribution, but
> >>> we
> >>> can address STORM-1677 and initiate voting RC2 immediately although we
> >>> think it's a 'blocker' issue.
> >>> 
> >>> Thanks,
> >>> Jungtaek Lim (HeartSaVioR)
> >>> 
> >>> 2016년 4월 2일 (토) 오전 8:50, P. Taylor Goetz <ptgo...@apache.org>님이 작성:
> >>> 
> >>>> This is a call to vote on releasing Apache Storm 1.0.0 (rc1)
> >>>> 
> >>>> Full list of changes in this release:
> >>>> 
> >>>> 
> >>>> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=787e4a6c375d290f724e59b3d8ebe34806ccd0d5
> >>>> 
> >>>> The tag/commit to be voted upon is v1.0.0:
> >>>> 
> >>>> 
> >>>> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=45b1b148401fd05f0f79cc7abdf6b5c7fc43df20;hb=d02f94268dec229d1125a24fdf53fa303cbc2b29
> >>>> 
> >>>> The source archive being voted upon can be found here:
> >>>> 
> >>>> 
> >>>> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.0/apache-storm-1.0.0-src.tar.gz
> >>>> 
> >>>> Other relea

Re: [VOTE] Release Apache Storm 1.0.0 (rc1)

[Harsha]

For rc2  please include STORM-1670 as well. Source distribution is
important as well.
-Harsha

On Sat, Apr 2, 2016, at 01:02 AM, Jungtaek Lim wrote:
> Here's my test,
> 
> - testing with source distribution : FAIL
>   - unzip : OK
>   - building from source dist : FAIL
> - how to build: running `mvn -P all-tests clean install` on unzipped
> source dist.
> - Test resource files (.log) are excluded from source distribution,
> which makes logviewer-test failing.
>   - filed issue : https://issues.apache.org/jira/browse/STORM-1677
>   and
> submit a patch via pull request
> - after applying PR, tests are all passed.
> 
> - testing with binary distribution (one machine) : OK
>   - launch daemons : OK
>   - run RollingTopWords (local) : OK (with minor issue)
> - process doesn't terminate until killing manually or pressing Ctrl +
> C
>   - run RollingTopWords (remote) : OK
> - activate / deactivate / rebalance / kill : OK
> - logviewer (worker dir, daemon dir) : OK
> - change log level : OK
> - thread dump, heap dump, restart worker : OK
>   - run WordCountTopology (remote) : OK
> - multiple workers and multi-lang bolt
> 
> Minor but just leaving note: I saw /log (logviewer) throwing HTTP STATUS
> 500 sometimes, but I can't reproduce it. At that time, deep-search
> doesn't
> work at that topology, too.
> 
> Total
> - source distribution: -1
> - binary distribution: +1
> 
> I don't know how it is important to build with source distribution, but
> we
> can address STORM-1677 and initiate voting RC2 immediately although we
> think it's a 'blocker' issue.
> 
> Thanks,
> Jungtaek Lim (HeartSaVioR)
> 
> 2016년 4월 2일 (토) 오전 8:50, P. Taylor Goetz <ptgo...@apache.org>님이 작성:
> 
> > This is a call to vote on releasing Apache Storm 1.0.0 (rc1)
> >
> > Full list of changes in this release:
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=787e4a6c375d290f724e59b3d8ebe34806ccd0d5
> >
> > The tag/commit to be voted upon is v1.0.0:
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=45b1b148401fd05f0f79cc7abdf6b5c7fc43df20;hb=d02f94268dec229d1125a24fdf53fa303cbc2b29
> >
> > The source archive being voted upon can be found here:
> >
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.0/apache-storm-1.0.0-src.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.0/
> >
> > The release artifacts are signed with the following key:
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >
> > The Nexus staging repository for this release is:
> >
> > https://repository.apache.org/content/repositories/orgapachestorm-1027/
> >
> > Please vote on releasing this package as Apache Storm 1.0.0.
> >
> > When voting, please list the actions taken to verify the release.
> >
> > This vote will be open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache Storm 1.0.0
> > [ ]  0 No opinion
> > [ ] -1 Do not release this package because...
> >
> > Thanks to everyone who contributed to this release. This is a major
> > milestone of which we should all be proud.
> >
> > This is not an April fools joke. ;)
> >
> > -Taylor
> >

Re: 答复: Question on Metrics Server to Alibaba team

[Harsha]

bh...@gmail.com> wrote:
> >
> > > John,
> > >
> > > My concern is H/A of metrics on Storm by default. (I'm not 100% sure
> > > Bobby pointed out same things.)
> > >
> > > Since Apache Storm has been used by various users so that we can't
> > > assume that users have knowledges of external systems (including
> > > Hadoop ecosystem, personal opinion) and operate them smoothly.
> > > It reminds me about the importance to keep in mind about default.
> > >
> > > Therefore, I'm curious that new metrics feature of JStom can work
> > > smoothly without external system (HBase / OTS). And love to see it
> > > supports H/A without other systems, or users have to tolerate lost of
> > > metrics for some scenarios.
> > >
> > > I guess this may be valid questions on H/A (as far as my understanding
> > > of design doc is right): How metrics work when TopologyMaster is down?
> > > And how metrics work when failover of Nimbus occurs?
> > >
> > > Personally I don't mind losing metrics for short durations (just want
> > > to check availability of H/A), but failure shouldn't mess up whole
> > metrics.
> > >
> > > Thanks,
> > > Jungtaek Lim (HeartSaVioR)
> > >
> > > 2016년 3월 23일 (수) 오후 3:39, John Fang <xiaojian@alibaba-inc.com>님이 작성:
> > >
> > > > @ Bobby Evans Jstorm code has experienced a lot of tests over the
> > > > past
> > > few
> > > > years, espatially HA and scalability. We have done a lot of
> > > > optimization about Metrics. The performance is better than Flink in
> > > > my tests. In my personal opinion, the metric in jstorm offers very
> > > > much informations. And the metric can tell us where is the bottleneck
> > when we run a topology.
> > > The
> > > > performance bottleneck maybe serialize/deserialize/netty/executor
> > > > and so on. Of course, I also has some other good monitoring in the
> > > > world. So I hope we can choice the better monitoring before phrase
> > > > 2. And I will
> > > start
> > > > study the Alas. If it is better, I am pleasured to redesign the
> > > > metric by Alas.
> > > > -邮件原件-
> > > > 发件人: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID]
> > > > 发送时间: 2016年3月22日 22:36
> > > > 收件人: dev@storm.apache.org
> > > > 主题: Re: Question on Metrics Server to Alibaba team
> > > >
> > > > My personal opinion is that we should not reinvent the wheel (aka
> > > > distributed fault tolerant metrics) ourselves.  The local file
> > > > blobstore with nimbus HA was a big enough pain to write and it is
> > > > relatively simple in comparison.
> > > > If the JStorm code is simple and offers everything we need in terms
> > > > of HA and scalability then I would be OK with it, but if it doesn't
> > > > I would
> > > lean
> > > > towards a different compatible open source solution.
> > > >
> > > > https://github.com/Netflix/atlas
> > > > looks very promising as a default option.  It is actively maintained
> > > > by a group that I think has some of the best monitoring in the
> > > > world.  And it
> > > is
> > > > both java and apache compatible.  It has no histogram support that I
> > > could
> > > > find, but that I don't see as being super critical.  The biggest
> > > > drawback is there is little documentation on how to use it, to
> > > > really be able to evaluate it for our needs. - Bobby
> > > >
> > > >    On Monday, March 21, 2016 7:29 PM, Jungtaek Lim
> > > > <kabh...@gmail.com>
> > > > wrote:
> > > >
> > > >
> > > >  Harsha,
> > > >
> > > > That's why I think new metric feature of JStorm looks promising.
> > > >
> > > > According to design doc on
> > > > https://issues.apache.org/jira/browse/STORM-1329,
> > > > there's no distinction between topology stat (which Apache Storm
> > > > includes to worker heartbeat) and built-in metrics (which should be
> > > > handled with separate consumer, as you stated).
> > > > All metrics are passed to Nimbus and Nimbus cached metrics, which
> > > > implies we can treat all metrics as same, and we can also provide
> > > > built-in
> > > metrics
> > > > (including 

Re: Question on Metrics Server to Alibaba team

[Harsha]

One of the goals of this work and probably can be addressed in separate
jira is how the topology metrics reporter works. Today its a bolt thats
part of a topology graph that means its another node in the Topology DAG
that needs be tuned for better performance. Some of our users took
performance hits by deploying topology metrics reporter that can send
metrics to Ganglia. Ideally this collection should be asynchronous and
not be a node in topology DAG.

Shipping default metrics server and along with pluggable option for
users who wants to graphite or other timeline servers should be the
goal.

--Harsha


On Mon, Mar 21, 2016, at 08:49 AM, Abhishek Agarwal wrote:
> @Cody - The design looks good. Does the design allow to aggregate metrics
> at the task/executor level? Basically, number of distinct metrics is
> proportional to the number of distinct tasks, did you ever run into such
> a
> use case?
> 
> 
> On Mon, Mar 21, 2016 at 8:46 PM, Cody Innowhere <e.neve...@gmail.com>
> wrote:
> 
> > Also, you can read the code from our latest release JStorm 2.1.1.
> >
> > On Mon, Mar 21, 2016 at 11:10 PM, Cody Innowhere <e.neve...@gmail.com>
> > wrote:
> >
> > > @Jungtaek,
> > > We did some tests on codahale metrics, compared to meters/histograms,
> > > counters are quite fast. So we mainly focused on the optimization of
> > meters
> > > and histograms (they are indeed very slow) including double sampling,
> > > changing the clock from ns (System.nanoTime) to ms, etc.
> > > You can take a look at the
> > > "com.alipay.dw.jstorm.example.sequence.bolt.TotalCount" class of our
> > > sequence-split-merge example code, as the client code entry to metrics.
> > > After that, you may dig to TopologyMaster class, which is still part of a
> > > topology, and then to TopologyMetricsRunnable, which is a part of nimbus
> > > server, finally to MetricUploader plugin, this is where the metrics
> > > interfere with our "metrics server". Still, there're some nits in the
> > code,
> > > but I think that should be no big problem.
> > >
> > > I'd also like to point out that our "metrics server" is not strictly a
> > > real metrics server, since most of the duty lies on nimbus server and
> > > topology master, it's more appropriate to call it metrics storage. The
> > main
> > > reason for this is that we don't want to make a heavy-weight metrics
> > server
> > > out of JStorm, and this makes us very easy to maintain (we have teams
> > that
> > > specifically maintain HBase/OTS in Alibaba since they're so commonly used
> > > in production).
> > >
> > > On Mon, Mar 21, 2016 at 10:54 PM, Jungtaek Lim <kabh...@gmail.com>
> > wrote:
> > >
> > >> Thanks Cody and Bobby for the explanation.
> > >>
> > >> Cody,
> > >> I took a look at design doc and looks promising, especially it doesn't
> > do
> > >> sampling when metric type is 'counter'. As far as I heard (I didn't try
> > >> it)
> > >> it becomes huge performance hit in Apache Storm when we change sample
> > rate
> > >> to 1.0.
> > >> Could you guide the entry point of metric feature in JStorm to dig into?
> > >>
> > >> And just a curiosity, did you consider extracting metric feature (which
> > is
> > >> done with TopologyMasters and Nimbuses) into separate component?
> > >> I understood your mention to 'metrics server' as separate component, but
> > >> after seeing design doc, feature seems to be implemented on Nimbus.
> > >>
> > >> Thanks,
> > >> Jungtaek Lim (HeartSaVioR)
> > >>
> > >> 2016년 3월 19일 (토) 오전 1:25, Cody Innowhere <e.neve...@gmail.com>님이 작성:
> > >>
> > >> > JStorm has provided a MetricUploader interface, which is similar to
> > >> > IMetricsConsumer in storm, and the underlying implementation is
> > >> pluggable,
> > >> > you can use HBase, or any other KV store that supports timeline
> > queries
> > >> or
> > >> > even a database(maybe for it's a small cluster). We provide model
> > >> classes
> > >> > in jstorm-core, as to what kinds of metrics data need to be stored,
> > it's
> > >> > totally up to the detailed implementation. Our internal implementation
> > >> uses
> > >> > OTS, which is a product of aliyun (
> > https://www.aliyun.com/product/ots/
> > >> ),
> > 

Re: [ANNOUNCE] Storm MySql Spout

[Harsha]

Hi Sourav,
 I just took a look. This looks great. It would be great if
 you can contribute to this connector Apache Storm. If so
 can you please open a JIRA and open a PR
 https://github.com/apache/storm.

Thanks,
Harsha

On Sun, Mar 20, 2016, at 08:09 AM, Sourav Mitra wrote:
> Hello,
> 
> We have been developing a storm spout that reads off MySql Bin Logs
> directly and generates the events as a stream.
> 
> Please have a look at the following project on github:
> 
> https://github.com/flipkart-incubator/storm-mysql
> 
> Its currently as a snapshot on Sonatype and will be released shortly.
> 
> Please feel free to review, raise issues.
> 
> PRs most welcome.
> 
> Thanks
> 
> Sourav Mitra

Re: Issues in Storm deployment

[Harsha]

 Vibhay,
 You are using master branch which is under development to
 move to Java. This might be very unstable. I suggest you to
 use 1.x-branch in general as its getting ready for storm
 1.0 release. You don't need to add anything to storm.yaml ,
 defaults should be fine for a single node cluster.

1. git checkout 1.x-branch
2. mvn -DskipTests clean install
3. cd storm-dist/binary
4. mvn clean package
5. cd target
6. find and unzip apache-storm-1.0.0-SNAPSHOT
7. run required services
  bin/storm dev-zookeeper, bin/storm nimbus, bin/storm supervisor,
  bin/storm ui
8. ./bin/storm jar
examples/storm-starter/storm-starter-topologies-1.0.0-SNAPSHOT.jar
org.apache.storm.starter.WordCountTopology wordcount

For bit more details you can refer here
http://blog.harsha.io/setting-up-a-single-node-apache-storm-cluster/
its on a older version of storm but relevant for the latest as well.

-Harsha




On Sun, Mar 13, 2016, at 01:43 PM, vibha goyal wrote:
> Hi,
> 
> I am grad student, and I am working on Storm as a part of my course
> project.
> 
> I have cloned the source code from git, and followed the instructions.
> 
> I am using branch : 0.10.0
> 
> After "mvn package", when I copy the apache-storm-.tar.gz in my
> home,
> untar it and try to run nimbus.
> 
> I get error:
> 
> Exception in thread "main" java.lang.ExceptionInInitializerError
> at
> org.apache.storm.utils.ConfigUtils.readStormConfigImpl(ConfigUtils.java:138)
> at
> org.apache.storm.utils.ConfigUtils.readStormConfig(ConfigUtils.java:134)
> at org.apache.storm.command.ConfigValue.main(ConfigValue.java:27)
> Caused by: java.lang.RuntimeException: java.io.IOException: Found
> multiple
> storm.yaml resources. You're probably bundling the Storm jars with your
> topology jar.
> [jar:file:/home/vgoyal5/apache-storm-2.0.0-SNAPSHOT/lib/storm-core-2.0.0-SNAPSHOT.jar!/storm.yaml,
> file:/home/vgoyal5/apache-storm-2.0.0-SNAPSHOT/conf/storm.yaml]
> at org.apache.storm.utils.Utils.findAndReadConfigFile(Utils.java:372)
> at org.apache.storm.utils.Utils.readStormConfig(Utils.java:456)
> at org.apache.storm.utils.Utils.(Utils.java:173)
> ... 3 more
> 
> 
> I deleted the "/home/vgoyal5/apache-storm-2.0.0-SNAPSHOT/conf/storm.yaml"
> file and again tried.
> 
> This time I was able to connect the nimbus and supervisors and could see
> the connections in UI.
> 
> As soon as I tried to submit the topology with the command:
> 
> storm jar examples/storm-starter/target/storm-starter-2.0.0-SNAPSHOT.jar
> org.apache.storm.starter.ExclamationTopology exclamation-topology
> 
> I got error:
> 
> org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find
> leader
> nimbus from seed hosts [sp16-cs525-g05-01.cs.illinois.edu]. Did you
> specify
> a valid list of nimbus hosts for config nimbus.seeds?
> 
> First ,I am not able to resolve this error,
> 
> and second, I cannot make changes in storm.yaml as I deleted the
> conf/storm.yaml
> 
> I am a beginner in Storm. I have been trying this for a day now. Any help
> would be appreciated!!
> 
> 
> 
> Thanks!
> 
> Vibha Goyal

Re: [DISCUSS] Java REST Framework adoption

[Harsha]

-1 on spring boot or anything related to spring. 
This api is intended to be very simple powering UI and any rest clients
interested in grabbing the metrics from the same api as UI does.

Jersey is good and  dropwizard (http://www.dropwizard.io/0.9.2/docs/)
has been a way to go for java REST api offlate. Underneath it uses
jersey and one can run jetty server as well which is what we've as the
UI and logviewer server.


-Harsha

On Tue, Feb 23, 2016, at 08:23 AM, Ravi Sharma wrote:
> spring boot +
> 
> Ravi
> 
> On Tue, Feb 23, 2016 at 3:16 PM, Ankur Garg <ankurga...@gmail.com> wrote:
> 
> > How about using Spring Boot & Jersey for writing this .  Spring Boot will
> > give us packaged  jar which once executed will bring up its own embedded
> > server (Jetty or Tomcat or some other ) . Although Spring Boot has some
> > disadvantages as well , but worth investigating this option too .
> >
> >
> > Any thoughts??
> >
> > Thanks
> > Ankur
> >
> > On Tue, Feb 23, 2016 at 8:20 PM, Bobby Evans <ev...@yahoo-inc.com.invalid>
> > wrote:
> >
> > > Yes, we need to pick something.  I have used Jersey in the past and I
> > > think it is fairly decent.  I have never used RESTEasy, but it is more or
> > > less the same API, so either one is fine with me, but Jersey is my vote
> > > just because of experience.
> > >
> > > You should keep in mind that we are currently on a very old version of
> > > jetty, and I am not sure if newer libraries will work with it.  But also
> > > the old versions of ring and hiccup that we use don't support newer jetty
> > > versions either.
> > >
> > > I personally think that now would be a good time to separate out the UI
> > > into a separate package + classpath.  This would allow us to package the
> > UI
> > > as both a war with embedded jetty as a default option to run it; start
> > from
> > > scratch with up to date versions of Jetty, Jersey/RESTEasy, and JAXB; and
> > > upgrade the different servers/components one at a time instead of all at
> > > once.  The DRPC server also uses the embedded jetty and exposes a REST
> > > interface, and that is going to be a harder one to tease out so it should
> > > probably be the last one to go.
> > >  - Bobby
> > >
> > > On Tuesday, February 23, 2016 3:40 AM, 伍翀(云邪) <
> > > wuchong...@alibaba-inc.com> wrote:
> > >
> > >
> > >  Hi all, I’m planning to move UI/REST service and logviewer to Java,
> > which
> > > means that we need to pick some alternatives for ring and hiccup.
> > > So the first thing is to pick up a REST framework.
> > > For the REST APIs, I think Jersey is a good choice (RESTEasy is fine
> > too).
> > > It’s easy to develop and good performance.
> > > Now logviewer use hiccup to return HTML we build ourselves, but it’s hard
> > > to debug and maintain. So in my opinion, it’s better to replace it with
> > > static HTML + REST like regular UI.
> > > Please let me know what you think.
> > > – Jark Wu
> > >
> > >
> > >
> >

Re: zookeeper setup for storm cluster

[Harsha]

You can actually use single node cluster for testing this out. The link
I gave you have instructions to start all the required storm services
and once thats up and running you can use storm jar to deploy the
topology.

On Thu, Dec 24, 2015, at 10:58 AM, researcher cs wrote:
> thanks for replying , but what i mean that i found project using storm
> and
> coder wrote this lines
> 
> *To compile and run the project in local mode*, type the following
> commands
> while being on the project root directory
> mvn package
> mvn compile exec:java -Dexec.classpathScope=compile
> -Dexec.mainClass=trident.topology_name
> 
> *To run in production cluster*, first package the code into a jar by
> running
> mvn package.
> This will package your code into a jar at the path
> target/Topology_name-{version}-jar-with-dependencies.jar.
> Then you can submit your jar to the cluster using the storm client:
> storm jar target/Topology_namr-1.0-SNAPSHOT-jar-with-dependencies.jar
> trident.Topology_name
> 
> Please note that in the production cluster mode, you require a DRPCClient
> to feed the topology with tweets and get results.
> 
> 
> i ran it in local mode by the above commands and worked well but now i
> want
> to run it in distrbuted mode as he wote i need to submit the topology to
> storm but what i should need in configuration for zoo.cfg and storm.yaml
> for working ?
> 
> should i need more than single machine or not ?
> 
> how it can be in distributed mode in single machine ?
> 
> 
> Thanks for your help
> 
> 
> 
> On Thu, Dec 24, 2015 at 7:43 PM, Harsha <st...@harsha.io> wrote:
> 
> > In storm, local mode means you run by using LocalCluster class its a
> > simulated cluster for testing topologies and aid development of
> > topologies . Example of LocalCluster
> >
> > https://github.com/apache/storm/blob/master/examples/storm-starter/src/jvm/storm/starter/WordCountTopology.java#L99
> >
> > In production it would be a distributed cluster. One needs to setup a
> > distributed cluster and config you showed seems ok .
> > You can follow steps here setup up a single node cluster
> > http://blog.harsha.io/setting-up-a-single-node-apache-storm-cluster/
> > and same can be extended for multi-node cluster.
> >
> > I am not quite sure about what you mean by importing a topology that run
> > in two modes in local or production. In the first link I gave you ,
> > wordcount topology can run local cluster and on distributed as well.
> >
> > -Harsha
> >
> > On Thu, Dec 24, 2015, at 08:54 AM, researcher cs wrote:
> > > can i find help ?
> > >
> > > On Thu, Dec 24, 2015 at 7:08 AM, researcher cs
> > > <prog.researc...@gmail.com>
> > > wrote:
> > >
> > > > I want to import a project that run in two modes "local and production"
> > > > mode
> > > >
> > > > want to get what is mean by production mode is that mean in cluster or
> > > > distributed mode ?
> > > >
> > > > and if that right . are the configurations in storm.yaml is like that
> > or
> > > > not ?
> > > > storm.zookeeper.servers:
> > > >  - "ipaddress"
> > > > nimbus.host: "ipaddress"
> > > > storm.zookeeper.port: 2181
> > > >
> > > > storm.local.dir: /home/storm
> > > >
> > > > supervisor.slots.ports:
> > > > - 6700
> > > > - 6701
> > > > - 6702
> > > > - 6703
> > > > drpc.port: 3772
> > > > drpc.servers:
> > > >- "localhost"
> > > >
> > > > and zoo.cfg is
> > > > tickTime=2000
> > > > dataDir=/home/storm/zookeeper
> > > > clientPort=2181
> > > > initLimit=5
> > > > syncLimit=2
> > > >
> >

Re: zookeeper setup for storm cluster

[Harsha]

1- is production. mode means distributed mode ?
   Yes. Typically more than one machine. But I am suggesting if you want
   to test this topology you can do so by using single node cluster as
   well.
2. Typical production clusters usually have more than 1 node.
3. Usually there will be one config file that gets deployed across the
nodes and on each node you choose start a nimbus and other nodes will
have supervisors.
4. The config you've is suffice and defaults will takes place for the
configs you don't mention.
5.  drpc.servers:
- "localhost" you should replace this host.name



-Harsha

On Thu, Dec 24, 2015, at 11:12 AM, researcher cs wrote:
> thanks i checked the link it's good for me but the question that i'm
> searching about
> 1- is production mode means distributed mode ?
> 2- if question 1 = yes , is that mean working on more than one machine ?
> 3- if question 2 = yes , is there specific configuration should i wrote
> in
> storm.yaml and zoo.cfg means is that configuration right
> storm.zookeeper.servers:
>  - "ipaddress"
> nimbus.host: "ipaddress"
> storm.zookeeper.port: 2181
> 
> storm.local.dir: /home/storm
> 
> supervisor.slots.ports:
> - 6700
> - 6701
> - 6702
> - 6703
> drpc.port: 3772
> drpc.servers:
>- "localhost"
> 
> and zoo.cfg is
> tickTime=2000
> dataDir=/home/storm/zookeeper
> clientPort=2181
> initLimit=5
> syncLimit=2
> 
> sorry for lot questions i'm searching to find a good solution but i hope
> you can help me , Thanks
> 
> On Thu, Dec 24, 2015 at 9:06 PM, Harsha <m...@harsha.io> wrote:
> 
> > You can actually use single node cluster for testing this out. The link
> > I gave you have instructions to start all the required storm services
> > and once thats up and running you can use storm jar to deploy the
> > topology.
> >
> > On Thu, Dec 24, 2015, at 10:58 AM, researcher cs wrote:
> > > thanks for replying , but what i mean that i found project using storm
> > > and
> > > coder wrote this lines
> > >
> > > *To compile and run the project in local mode*, type the following
> > > commands
> > > while being on the project root directory
> > > mvn package
> > > mvn compile exec:java -Dexec.classpathScope=compile
> > > -Dexec.mainClass=trident.topology_name
> > >
> > > *To run in production cluster*, first package the code into a jar by
> > > running
> > > mvn package.
> > > This will package your code into a jar at the path
> > > target/Topology_name-{version}-jar-with-dependencies.jar.
> > > Then you can submit your jar to the cluster using the storm client:
> > > storm jar target/Topology_namr-1.0-SNAPSHOT-jar-with-dependencies.jar
> > > trident.Topology_name
> > >
> > > Please note that in the production cluster mode, you require a DRPCClient
> > > to feed the topology with tweets and get results.
> > >
> > >
> > > i ran it in local mode by the above commands and worked well but now i
> > > want
> > > to run it in distrbuted mode as he wote i need to submit the topology to
> > > storm but what i should need in configuration for zoo.cfg and storm.yaml
> > > for working ?
> > >
> > > should i need more than single machine or not ?
> > >
> > > how it can be in distributed mode in single machine ?
> > >
> > >
> > > Thanks for your help
> > >
> > >
> > >
> > > On Thu, Dec 24, 2015 at 7:43 PM, Harsha <st...@harsha.io> wrote:
> > >
> > > > In storm, local mode means you run by using LocalCluster class its a
> > > > simulated cluster for testing topologies and aid development of
> > > > topologies . Example of LocalCluster
> > > >
> > > >
> > https://github.com/apache/storm/blob/master/examples/storm-starter/src/jvm/storm/starter/WordCountTopology.java#L99
> > > >
> > > > In production it would be a distributed cluster. One needs to setup a
> > > > distributed cluster and config you showed seems ok .
> > > > You can follow steps here setup up a single node cluster
> > > > http://blog.harsha.io/setting-up-a-single-node-apache-storm-cluster/
> > > > and same can be extended for multi-node cluster.
> > > >
> > > > I am not quite sure about what you mean by importing a topology that
> > run
> > > > in two modes in local or production. In the first link I gave you ,
> > > > wordcount topology can run local cluster

Re: zookeeper setup for storm cluster

[Harsha]

In storm, local mode means you run by using LocalCluster class its a
simulated cluster for testing topologies and aid development of
topologies . Example of LocalCluster
https://github.com/apache/storm/blob/master/examples/storm-starter/src/jvm/storm/starter/WordCountTopology.java#L99

In production it would be a distributed cluster. One needs to setup a
distributed cluster and config you showed seems ok .
You can follow steps here setup up a single node cluster
http://blog.harsha.io/setting-up-a-single-node-apache-storm-cluster/
and same can be extended for multi-node cluster.

I am not quite sure about what you mean by importing a topology that run
in two modes in local or production. In the first link I gave you ,
wordcount topology can run local cluster and on distributed as well.

-Harsha

On Thu, Dec 24, 2015, at 08:54 AM, researcher cs wrote:
> can i find help ?
> 
> On Thu, Dec 24, 2015 at 7:08 AM, researcher cs
> <prog.researc...@gmail.com>
> wrote:
> 
> > I want to import a project that run in two modes "local and production"
> > mode
> >
> > want to get what is mean by production mode is that mean in cluster or
> > distributed mode ?
> >
> > and if that right . are the configurations in storm.yaml is like that or
> > not ?
> > storm.zookeeper.servers:
> >  - "ipaddress"
> > nimbus.host: "ipaddress"
> > storm.zookeeper.port: 2181
> >
> > storm.local.dir: /home/storm
> >
> > supervisor.slots.ports:
> > - 6700
> > - 6701
> > - 6702
> > - 6703
> > drpc.port: 3772
> > drpc.servers:
> >- "localhost"
> >
> > and zoo.cfg is
> > tickTime=2000
> > dataDir=/home/storm/zookeeper
> > clientPort=2181
> > initLimit=5
> > syncLimit=2
> >

Re: Problem with storm since 4 months

[Harsha]

Sam,
  you might be using very old version of storm since its showing
  ZeroMQ. Can you try using newer version storm without zero mq.
-Harsha 

On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote:
> I have this problem since 4months when I submitted topology I got this in
> the worker log file [ERROR] Async loop died! org.zeromq.ZMQException:
> Address already in use(0x62)
> at org.zeromq.ZMQ$Socket.bind(Native Method)
> at zilch.mq$bind.invoke(mq.clj:69)
> at backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57)at
> backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__1629.invoke(loader.clj:26)
> at backtype.storm.util$async_loop$fn__465.invoke(util.clj:375)
> at clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Unknown
> Source)
> 
> when i tried to connect port 6703 and 6702
> 
> And supervisor log file hadn't still start
> 
> 
> I searched everywhere but cannot find any solution I hope you can

Re: Welcome new Apache Storm Committers/PMC Members

[Harsha]

Congrats everyone.
-Harsha

On Tue, Dec 8, 2015, at 03:19 PM, Priyank Shah wrote:
> Congratulation everyone!
> 
> 
> 
> 
> On 12/8/15, 3:17 PM, "Hugo Da Cruz Louro" <hlo...@hortonworks.com> wrote:
> 
> >Congrats everyone. Looking forward to working with you!
> >
> >> On Dec 8, 2015, at 2:54 PM, Aaron.Dossett <aaron.doss...@target.com> wrote:
> >> 
> >> Thanks, everyone, and congratulations to the other new committers as well!
> >> 
> >> On 12/8/15, 4:12 PM, "임정택" <kabh...@gmail.com> wrote:
> >> 
> >>> Congratulation! Looking forward to work with you all.
> >>> 
> >>> Best,
> >>> Jungtaek Lim (HeartSaVioR)
> >>> 
> >>> On 2015년 12월 9일 (수) at 오전 6:55 P. Taylor Goetz <ptgo...@gmail.com> wrote:
> >>> 
> >>>> I’m pleased to announce that the following individuals have joined as
> >>>> Apache Storm Committers/PMC Members:
> >>>> 
> >>>> - Longda Feng
> >>>> - Arun Mahadevan
> >>>> - Boyang Jerry Peng
> >>>> - Matthias J. Sax
> >>>> - Aaron Dossett
> >>>> 
> >>>> Longda, Arun, Jerry, Matthias, and Aaron have all demonstrated technical
> >>>> merit and dedication to Apache Storm and its community, and as PMC
> >>>> members
> >>>> they will help drive innovation and community development.
> >>>> 
> >>>> Please join me in welcoming and congratulating Londa, Arun, Jerry,
> >>>> Matthias, and Aaron. We look forward to your continued dedication to the
> >>>> Storm community.
> >>>> 
> >>>> -Taylor
> >>>> 
> >> 
> >

Re: supervisor的讨论

[Harsha]

Hi John,         Thanks for the doc about supervisor. Could you please
add this to wiki here
https://cwiki.apache.org/confluence/display/STORM/Storm+Home  and start
a discussion thread around it.

Thanks, Harsha On Fri, Nov 27, 2015, at 04:15 AM, John Fang wrote:
>


> Email had 1 attachment:


>  * Discussion about supervisor.docx  25k (application/vnd.openxmlformats-
>officedocument.wordprocessingml.document)

Re: [DISCUSS] Storm 2.0 plan

[Harsha]

Hi All,
  If possible can we have bi-weekly or monthly video hangouts to
  discuss the plan. I think it will make it easier to discuss
  the next steps. We can post the details of the discussion on
  the mailing list so that everyone is involved in whats going .

Thanks,
Harsha
On Fri, Nov 20, 2015, at 01:03 AM, Longda Feng wrote:
> 
> @Sean, Thanks for clarify.
> @Taylor, @Bobby, @Sean, @Jungtaek, @Harsha, @dev,
> Sorry for leading to misunderstanding.
> The biggest point:We would like to merge two community into one
> community, One community is stronger than two single communities. My team
> hopes that Alibaba can directly use the Apache Storm version  in the next
> few years. My team don't need to maintain JStorm any more, this is the
> reason why Alibaba donated JStorm. 
> Second point:Sean's point is right. The migration is not just "copy". It
> should be "merge". I means that the module will not simply as the JStorm
> module. It should be the result of our disccussion. I think the final
> solution after merging can make Storm better. 
> Third point:In fact, I don't scare other streaming process, especially
> for Heron. I have work on Storm for 4 years, I am a deep fans of Storm. I
> know what can Storm do and what storm cannot do . But I want to express
> we need accelerate our evolve speed. This field is so active. We should
> start to learn other framework's advantage as soon as possible.
> Especally, we need more application level programming framework like
> Trident. This wil attract more users to Storm.
> Fourth point:We don't need to do everything from scratch, we can use
> JStorm as much as possible. JStorm is here, why not use.
> Last point:My team is already full time on this merge, we will try our
> best to do contribution, make Storm better. 
> 
> ThanksLongda
> 
> --From:Sean
> Zhong <clock...@gmail.com>Send Time:2015年11月20日(星期五) 11:58To:dev
> <dev@storm.apache.org>Subject:Re: [DISCUSS] Storm 2.0 plan
> Hi All,
> 
> I think there are may be some misproper use of words or misunderstanding.
> 
> Here is what I can see both agrees these goals:
> 1. We want to migrate clojure to java
> 2. We want to merge important features together.
> 3. We want to do this in a step by step, transparent, reviewable way,
> especially with close examination and reviews for code that has
> architecture change.
> 4. We want the final version to remain compatibility.
> 
> The only difference is the process we use to achieve these goals.
> Longda's view:
> 1. do a parallel migration from clojure core to java part by part.
> parallel
> means equivalent, no new features added in this step. He suggest to
> follow
> the order "modules/client/utils/nimbus/supervisor/drpc/worker & task/
> web ui/others"
> He use word "copy", which is mis-proper in my idea. It is more like a
> merging.
> quote on his words.
> 
> >  2.1 Copy modules from JStorm, one module from one module
> 
> 2.2 The sequence is extern modules/client/utils/nimbus/
> > supervisor/drpc/worker & task/web ui/others
> 
> 2. upon the java core code base, incremental add new feature blocks.
> quote on his words.
> 
> > 3.1 Discuss solution for each difference(jira)
> > 3.2 Once the solution is finalized, we can start the
> > merging. (Some issues could be start concurrently. It
> > depends on the discussion.)
> 
> 3.  His goal is to remain compatibility. "this version is stable and
> compatible with API of Storm 1.0." is not accurate statement from my
> point,
> at least not for the security feature.
> 4. He share his concern on other streaming engines.
> 
> 
> Bobby and Jungtaek 's view:
> 1. "Copy" is not acceptable, it will impact the security features. (Copy
> is
> a wrong phase to use, I think Longda means more a merging)
> 2. With JStorm team, we start with clojure -> java translation first,
> 3. By optimistic view, with JStorm team, one month should be enough for
> above stage.
> 3. Adding new features after whole code is migrated to java.
> 4. No need to that worry about other engines.
> 
> If my understanding of both parties are correct. I think we agree on most
> of things about the process.
> first: clojure -> java
> second: merge features.
> 
> With a slight difference about how aggressive we want to do "clojure ->
> java", and how long it takes.
> 
> 
> @Longda, can you clarify whether my understanding of your opinion is
> right?
> 
> 
> Sean
> 
> 
> On Fri, Nov 20, 2015 at 11:40 AM, P. Taylor Goetz <ptgo...@gmail.com>
> wrote:

Re: [DISCUSS] Plan for Merging JStorm Code

[Harsha]

Taylor,
 Can you add me to the wiki as well my id is "harsha".
Thanks,
Harsha

On Fri, Nov 20, 2015, at 01:19 AM, Longda Feng wrote:
> @Taylor, I want to work with Bobby, my wiki name is "longda".
> ThanksLongda--From:Bobby
> Evans <ev...@yahoo-inc.com.INVALID>Send Time:2015年11月20日(星期五) 06:54To:P.
> Taylor Goetz <ptgo...@gmail.com>，dev@storm.apache.org
> <dev@storm.apache.org>Subject:Re: [DISCUSS] Plan for Merging JStorm Code
> Sorry about this taking so long.  I am revans2 on the cwiki. - Bobby 
> 
> 
> On Wednesday, November 18, 2015 4:24 PM, P. Taylor Goetz 
> <ptgo...@gmail.com> wrote:
>  
> 
>  All I have at this point is a placeholder wiki entry [1], and a lot of local 
> notes that likely would only make sense to me.
> 
> Let me know your wiki username and I’ll give you permissions. The same
> goes for anyone else who wants to help.
> 
> -Taylor
> 
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61328109
> 
> > On Nov 18, 2015, at 2:08 PM, Bobby Evans <ev...@yahoo-inc.com.INVALID> 
> >wrote:
> > 
> > Taylor and others I was hoping to get started filing JIRA and planning on 
> >how we are going to do the java migration + JStorm merger.  Is anyone else 
> >starting to do this?  If not would anyone object to me starting on it? - 
> >Bobby
> > 
> > 
> >On Thursday, November 12, 2015 12:04 PM, P. Taylor Goetz 
> ><ptgo...@gmail.com> wrote:
> > 
> > 
> > Thanks for putting this together Basti, that comparison helps a lot.
> > 
> > And thanks Bobby for converting it into markdown. I was going to just 
> >attach the spreadsheet to JIRA, but markdown is a much better solution.
> > 
> > -Taylor
> > 
> >> On Nov 12, 2015, at 12:03 PM, Bobby Evans <ev...@yahoo-inc.com.INVALID> 
> >>wrote:
> >> 
> >> I translated the excel spreadsheet into a markdown file and put up a pull 
> >>request for it.
> >> https://github.com/apache/storm/pull/877
> >> I did a few edits to it to make it work with Markdown, and to add in a few 
> >>of my own comments.  I also put in a field for JIRAs to be able to track 
> >>the migration.
> >> Overall I think your evaluation was very good.  We have a fair amount of 
> >>work ahead of us to decide what version of various features we want to go 
> >>forward with.
> >>  - Bobby
> >> 
> >> 
> >>On Thursday, November 12, 2015 9:37 AM, 刘键(Basti Liu) 
> >><basti...@alibaba-inc.com> wrote:
> >> 
> >> 
> >> Hi Bobby & Jungtaek,
> >> 
> >> Thanks for your replay.
> >> I totally agree that compatibility is the most important thing. Actually, 
> >>JStorm has been compatible with the user API of Storm.
> >> As you mentioned below, we indeed still have some features different 
> >>between Storm and JStorm. I have tried to list them (minor update or 
> >>improvements are not included).
> >> Please refer to attachment for details. If any missing, please help to 
> >>point out. (The current working features are probably missing here.)
> >> Just have a look at these differences. For the missing features in JStorm, 
> >>I did not see any obstacle which will block the merge to JStorm.
> >> For the features which has different solution between Storm and JStorm, we 
> >>can evaluate the solution one by one to decision which one is appropriate.
> >> After the finalization of evaluation, I think JStorm team can take the 
> >>merging job and publish a stable release in 2 months.
> >> But anyway, the detailed implementation for these features with different 
> >>solution is transparent to user. So, from user's point of view, there is 
> >>not any compatibility problem.
> >> 
> >> Besides compatibility, by our experience, stability is also important and 
> >>is not an easy job. 4 people in JStorm team took almost one year to finish 
> >>the porting from "clojure core"
> >> to "java core", and to make it stable. Of course, we have many devs in 
> >>community to make the porting job faster. But it still needs a long time to 
> >>run many online complex topologys to find bugs and fix them. So, that is 
> >>the reason why I proposed to do merging and build on a stable "java core".
> >> 
> >> -Original Message-
> >> From: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID]
> >>

Re: [DISCUSS] Storm 2.0 plan

[Harsha]

Longda,

 "2.1 Copy modules from JStorm, one module from one module
> 2.2 The sequence is extern
> modules/client/utils/nimbus/supervisor/drpc/worker & task/web ui/others"

Are you suggesting we just copy the Jstorm code in place of clojure? If
so thats not going to work. There might be some code that can be easily
replaceable with Jstorm's . But not everything will be that
straightforward especially with feature disparity between Storm &
JStorm. 
We should be moving code to java i.e rewriting parts of the code where
needed and if something that can be picked up from JStorm we should do
that .

Thanks,
Harsha

 

On Wed, Nov 18, 2015, at 10:13 PM, Longda Feng wrote:
> Sorry for changing the Subject.
> 
> I am +1 for releasing Storm 2.0 with java core, which is merged with
> JStorm.
> 
> I think the change of this release will be the biggest one in history. It
> will probably take a long time to develop. At the same time, Heron is
> going to open source, and the latest release of Flink provides the
> compatibility to Storm’s API. These might be the threat to Storm. So I
> suggest we start the development of Storm 2.0 as quickly as possible. In
> order to accelerate the development cycle, I proposed to take JStorm
> 2.1.0 core and UI as the base version since this version is stable and
> compatible with API of Storm 1.0. Please refer to the phases below for
> the detailed merging plan.
> 
> Note: We provide a demo of JStorm’s web UI. Please refer to
> storm.taobao.org . I think JStorm will give a totally different view to
> you.
> 
> I would like to share the experience of initial development of JStorm
> (Migrate from clojure core to java core). 
> Our team(4 developers) have spent almost one year to finish the
> migration. We took 4 months to release the first JStorm version, and 6
> months to make JStorm stable. During this period, we tried to switch more
> than online 100 applications with different scenarios from Storm to
> JStorm, and many bugs were fixed. Then more and more applications were
> switched to JStorm in Alibaba.
> Currently, there are 7000+ nodes of JStorm clusters in Alibaba and 2000+
> applications are running on them. The JStorm Clusters here can handle 1.5
> PB/2 Trillion messages per day. The use cases are not only in BigData
> field but also in many other online scenarios.
> Besides it, we have experienced the November 11th Shopping Festival of
> Alibaba for last three years. At that day, the computation in our cluster
> increased several times than usual. All applications worked well during
> the peak time. I can say the stability of JStorm is no doubt today.
> Actually, besides Alibaba, the most powerful Chinese IT company are also
> using JStorm.
> 
> 
> Phase 1:
>  
> Define the target of Storm 2.0. List the requirement of Storm 2.0
> 1. Open a new Umbrella Jira
> (https://issues.apache.org/jira/browse/STORM-717)
> 2. Create one 2.0 branch, 
> 2.1 Copy modules from JStorm, one module from one module
> 2.2 The sequence is extern
> modules/client/utils/nimbus/supervisor/drpc/worker & task/web ui/others
> 3. Create jira for all differences between JStorm 2.1.0 and Storm 1.0
> 3.1 Discuss solution for each difference(jira)
> 3.2 Once the solution is finalized, we can start the merging. (Some
> issues could be start concurrently. It depends on the discussion.)
> 
> The phase mainly try to define target and finalize the solution.
> Hopefully this phase could be finished in 2 month(before 2016/1/31). . 
> 
> 
> Phase 2:
> Release Storm 2.0 beta
> 1. Based on phrase 1's discussion, finish all features of Storm 2.0
> 2. Integrate all modules, make the simplest storm example can run on the
> system.
> 3. Test with all example and modules in Storm code base.
> 4. All daily test can be passed.
>  
> Hopefully this phase could be finished in 2 month(before 2016/3/31)
> 
> 
> Phase 3:
> Persuade some user to have a try.
> Alibaba will try to run some online applications on the beta version
> 
> Hopefully this phase could be finished in 1 month(before 2016/4/31).
> 
> 
> Any comments are welcome.
> 
> 
> Thanks
> Longda--From:P.
> Taylor Goetz <ptgo...@gmail.com>Send Time:2015年11月19日(星期四) 06:23To:dev
> <dev@storm.apache.org>，Bobby Evans <ev...@yahoo-inc.com>Subject:Re:
> [DISCUSS] Plan for Merging JStorm Code
> All I have at this point is a placeholder wiki entry [1], and a lot of
> local notes that likely would only make sense to me.
> 
> Let me know your wiki username and I’ll give you permissions. The same
> goes for anyone else who wants to help.
> 
> -Taylor
> 
> [1]
> https://cwiki.apache.org/confluence/pa

Re: [DISCUSS] 1.0 Release (was Re: [DISCUSS] Initial 0.11.0 Release)

[Harsha]

+1 on Bobby's suggestion on moving the packages to storm-compat and have
it part of lib folder. 
Moving 1.0 with org.apache.storm will make it easier in the future
rather than wait for 2.0 release we should make 
this change now and in 2.0 we can remove the storm-compat jar.

Thanks,
Harsha


On Thu, Nov 12, 2015, at 07:15 AM, Bobby Evans wrote:
> I agree that having the old package names for a 1.0 release is a little
> odd, but I also am nervous about breaking backwards compatibility for our
> customers in a very significant way.  The upgrade for us from 0.9.x to
> 0.10.x has gone fairly smoothly.  Most customers read the announcement
> and recompiled their code against 0.10, but followed our instructions so
> that their topologies could run on both 0.9.x and 0.10.x.  Having the
> ability for a topology to run on both an old cluster and a new cluster is
> very important for us, because of failover.  If we want to minimize
> downtime we need to have the ability to fail a topology over from one
> cluster to another, usually running on the other side of the
> country/world.  For stability reasons we do not want to upgrade both
> clusters at once, so we need to have confidence that a topology can run
> on both clusters.  Maintaining two versions of a topology is a huge pain
> as well.
> Perhaps what we can do for 1.0 is to move all of the packages to
> org.apache.storm, but provide a storm-compat package that will still have
> the old user facing APIs in it, that are actually (for the most part)
> subclasses/interfaces of the org.apache versions.  I am not sure this
> will work perfectly, but I think I will give it a try.
>  - Bobby 
> 
> 
>  On Thursday, November 12, 2015 9:04 AM, Aaron. Dossett
>  <aaron.doss...@target.com> wrote:
>
> 
>  As a user/developer this sounds great.  The only item that gives me
>  pause
> is #2.  Still seeing backtype.* package names would be unexpected (for
> me)
> for a 1.0 release.  That small reservation aside, I am +1 (non-binding).
> 
> On 11/11/15, 2:45 PM, "임정택" <kabh...@gmail.com> wrote:
> 
> >+1
> >
> >Jungtaek Lim (HeartSaVioR)
> >
> >2015-11-12 7:21 GMT+09:00 P. Taylor Goetz <ptgo...@gmail.com>:
> >
> >> Changing subject in order to consolidate discussion of a 1.0 release in
> >> one thread (there was some additional discussion in the thread regarding
> >> the JStorm code merge).
> >>
> >> I just want to make sure I’m accurately capturing the sentiment of the
> >> community with regard to a 1.0 release. Please correct me if my summary
> >> seems off-base or jump in with an opinion.
> >>
> >> In summary:
> >>
> >> 1. What we have been calling “0.11.0” will become the Storm 1.0 release.
> >> 2. We will NOT be migrating package names for this release (i.e.
> >> “backtype.storm” —> “org.apache.storm”).
> >> 3. Post 1.0 release we will go into feature freeze for core
> >>functionality
> >> to facilitate the JStorm merge.
> >> 4. During the feature freeze only fixes for high priority bugs in core
> >> functionality will be accepted (no new features).
> >> 5. During the feature freeze, enhancements to “external” modules can be
> >> accepted.
> >> 6. We will stop using the “beta” flag in favor of purely numeric version
> >> numbers. Stable vs. non-stable (development) releases can be indicated
> >>on
> >> the download page.
> >>
> >> Do we all agree?
> >>
> >> -Taylor
> >>
> >> > On Nov 11, 2015, at 4:10 PM, P. Taylor Goetz <ptgo...@gmail.com>
> >>wrote:
> >> >
> >> >
> >> >> On Nov 11, 2015, at 9:28 AM, Bobby Evans
> >><ev...@yahoo-inc.com.INVALID>
> >> wrote:
> >> >>
> >> >> I would like to see STORM-1190, STORM-1155, STORM-1198, STORM-1196,
> >> STORM-885, STORM-876, STORM-1145, STORM-831, and STORM-874 added to the
> >> list.
> >> >>
> >> >> Some of them are more important then others but they are all things I
> >> would like to see in a 0.11.0 release.
> >> >
> >> >
> >> > Cool. Thanks for listing them out. I will add them so they get
> >>tracked.
> >> >
> >> >
> >> >> On a side note I don't think the beta releases have been helpful.  I
> >> would much rather just have a 0.11.0 go to 0.11.1 ... instead of
> >> 0.11.0-beta1, 0.11.0-beta2.  To me the beta label is not that helpful,
> >>but
> >> it is not that big of a deal for me.
> &g

Re: [DISCUSS] Plan for Merging JStorm Code

[Harsha]

Thanks Taylor for putting together plan on merging JStorm code.
Few things

1. we should call 0.11.0 as 1.0.0 release since storm has security and
   nimbus ha . Quite a lot of features and improvements added this is
   going to be big release and its about time we call 1.0.0

1.1 "align package names (e.g backtype.storm --> org.apache.storm /
com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin"     I
propose we do this as next release itself instead of pushing it
onto another .

"Phase 3 - Migrate Clojure --> Java"

I would like to propose a Hackathon among storm community along with
jstorm. We need to come up with plan of action on what code needs to be
merged into storm-core. I am thinking it will help better to have
everyone over video chat or something to go over the code and get it
migrated to java.


Thanks, Harsha

On Tue, Nov 10, 2015, at 01:32 PM, P. Taylor Goetz wrote:
> Based on a number of discussions regarding merging the JStorm code,
> I’ve tried to distill the ideas presented and inserted some of my own.
> The result is below.
>
> I’ve divided the plan into three phases, though they are not
> necessarily sequential — obviously some tasks can take place in
> parallel.
>
> None of this is set in stone, just presented for discussion. Any and
> all comments are welcome.
>
> ---
>
> Phase 1 - Plan for 0.11.x Release
> 1. Determine feature set for 0.11.x and publish to wiki [1].
> 2. Announce feature-freeze for 0.11.x
> 3. Create 0.11.x branch from master (Phase 2.4 can begin.)
> 4. Release 0.11.0 (or whatever version # we want to use)
> 5. Bug fixes and subsequent releases from 0.11.x-branch
>
>
> Phase 2 - Prepare for Merge ("master" and "jstorm-import" branches)
> 1. Determine/document unique features in JStorm (e.g. classpath
>isolation, cgroups, etc.) and create JIRA for migrating the
>feature.
> 2. Create JIRA for migrating each clojure component (or logical group
>of components) to Java. Assumes tests will be ported as well.
> 3. Discuss/establish style guide for Java coding conventions. Consider
>using Oracle’s or Google’s Java conventions as a base — they are
>both pretty solid.
> 4. align package names (e.g backtype.storm --> org.apache.storm /
>com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin)
>
>
> Phase 3 - Migrate Clojure --> Java
> 1. Port code/tests to Java, leveraging existing JStorm code wherever
>possible (core functionality only, features distinct to JStorm
>migrated separately).
> 2. Port JStorm-specific features.
> 3. Begin releasing preview/beta versions.
> 4. Code cleanup (across the board) and refactoring using established
>coding conventions, and leveraging PMD/Checkstyle reports for
>reference. (Note: good oportunity for new contributors.)
> 5. Release 0.12.0 (or whatever version # we want to use) and lift
>feature freeze.
>
>
> Notes: We should consider bumping up to version 1.0 sometime soon and
> then switching to semantic versioning [3] from then on.
>
>
> With the exception of package name alignment, the "jstorm-import"
> branch will largely be read-only throughout the process.
>
> During migration, it's probably easiest to operate with two local
> clones of the Apache Storm repo: one for working (i.e. checked out to
> working branch) and one for reference/copying (i.e. checked out to "jstorm-
> import").
>
> Feature-freeze probably only needs to be enforced against core
> functionality. Components under "external" can likely be exempt, but
> we should figure out a process for accepting and releasing new
> features during the migration.
>
> Performance testing should be continuous throughout the process. Since
> we don't really have ASF infrastructure for performance testing, we
> will need a volunteer(s) to host and run the performance tests.
> Performance test results can be posted to the wiki [2]. It would
> probably be a good idea to establish a baseline with the 0.10.0
> release.
>
> I’ve attached an analysis document Sean Zhong put together a while
> back to the JStorm merge JIRA [4]. The analysis was against the 0.9.3
> release but is still relevant and has a lot of good information.
>
>
> [1] 
> https://cwiki.apache.org/confluence/display/STORM/Release+0.11.0+Feature+Set
> [2] https://cwiki.apache.org/confluence/display/STORM/Storm+Home
> [3]http://semver.org
> [4]https://issues.apache.org/jira/browse/STORM-717
>
>
> -Taylor
>
> Email had 1 attachment:


>  * signature.asc  1k (application/pgp-signature)

Re: Builds on Windows

[Harsha]

Bit more details on windows. We do have builds going internally on
windows , but not the latest trunk which we will be doing soon. 
Regarding auth-test yes thats an issue and storm security in general not
going to work in windows. We disable this test for windows. I think we
should put a check in in the code itself to run this test on *nix only.

On Thu, Nov 5, 2015, at 09:06 AM, Bobby Evans wrote:
> Chuck, sorry but windows really does end up being a bit of an after
> though here.  We do have groups that test the official releases on
> windows and go back to fix issues that they find, but it is not
> guaranteed that you will be able to run on windows all the time.  I do
> know that symlinks are supported on windows, but it requires some special
> permissions to make it work.  Hadoop uses symlinks in a very similar way
> to how storm does now, so it might just end up being a documentation
> issue.  Please file a JIRA indicating the issues you are seeing and
> hopefully someone who works with windows will be better able to help you
> out.
> 
> 
> The auth-test failing to impersonate a user might be related to a bug we
> have had in the tests where some of the auth tests failed to stop
> impersonating a user and this leaked into other tests.  Those should have
> been fixed on master, but if you are still seeing them on master please
> file a JIRA.
> 
>  - Bobby 
> 
> 
> 
> On Thursday, November 5, 2015 10:20 AM, Chuck Burgess
>  wrote:
> Hey everyone,
> Trying to build Storm on Windows, I consistently see clojure test
> failures regarding the supervisor-test (seems to fail trying to create
> symlinks) and the auth-test (seems to fail trying to "impersonate user").
>  At first glance, I could see these as not being possible in Windows, but
> that's just a hunch.
> 
> Should these pass on a Windows build?  Or is a Windows build not
> typically considered "in scope" for the project?  I need to understand if
> Windows testing is significant or not, so I'll know how to proceed with
> trying to write up some contributions.
> 
> Thanks.
> CRB 

Re: [Discusson] Storm System Tests

[Harsha]

The reason for suggesting ducktape not just used in apache kafka but
also its getting security services integration like kdc and already has
zookeeper. This framework can work with vagrant vms or amazon ec2 etc..

On Thu, Nov 5, 2015, at 11:49 AM, Hugo Da Cruz Louro wrote:
> Great, will make this a priority. Created a
> JIRA<https://issues.apache.org/jira/browse/STORM-1179> ticket and
> assigned it to me.
> 
> https://issues.apache.org/jira/browse/STORM-1179
> 
> 
> On Nov 5, 2015, at 11:40 AM, Bobby Evans
> <ev...@yahoo-inc.com<mailto:ev...@yahoo-inc.com>> wrote:
> 
> Hugo,
> 
> I would love to see that happen.  It has been on my list for a while, but
> I have never found the time to do it. I personally am +1 on this,
> hopefully we can do this quickly in preparation for a 0.11.0 release.
> 
> - Bobby
> 
> 
> 
> On Thursday, November 5, 2015 1:37 PM, Hugo Da Cruz Louro
> <hlo...@hortonworks.com<mailto:hlo...@hortonworks.com>> wrote:
> 
> 
> I a agree with these three levels of testing, and that at the very least
> we should keep unit tests separated from the system/integration tests.
> One huge advantage would be to quickly run unit tests that hopefully are
> more predictable, and thus avoid intermittent test fails.
> 
> Bobby just mentioned but I had already in mind that it would be useful to
> create different maven profiles for integration and unit tests. I have
> done something similar in a different project and it works really well.
> If we agree that it is something we want to implement here, I will create
> a JIRA ticket for this and get it done. Please let me know.
> 
> Thanks,
> Hugo
> 
> > On Nov 5, 2015, at 11:23 AM, Bobby Evans 
> > <ev...@yahoo-inc.com.INVALID<mailto:ev...@yahoo-inc.com.INVALID>> wrote:
> >
> > I totally agree.  Too many of our "unit" tests are integration tests and 
> > end up spinning up an entire local-mode cluster.
> > I personally would like to see three levels of testing.
> > 1) true unit tests.  They only touch the code under test and very little 
> > else.  The should be what you get when you run mvn test2) integration 
> > tests.  These should spin up local mode clusters and modify configs/etc to 
> > get a decent set of more white box tests.  The should run as a part of 
> > trivis-ci, and probably should run by enabling a special profile.3) Sanity 
> > Integration Tests.  ducktape looks like a great fit here.  I would love to 
> > see us spin up an few different scenarios for testing, with/without 
> > security.  Talking to Kafka, Hadoop(HBase, HDFS, Hive), redis, 
> > elastasearch, etc.
> > These would be run frequently but not necessarily a part of CI initially.
> >  - Bobby
> >
> >
> >On Wednesday, November 4, 2015 7:21 PM, Harsha 
> > <st...@harsha.io<mailto:st...@harsha.io>> wrote:
> >
> >
> > Hi All,
> >  As community we are growing and adding new and exciting
> >  features to Storm and also we've ever growing connector which
> >  only helps in storm adoption. One thing we've severely lacking
> >  is system tests. There are unit tests which acts as
> >  integration tests to storm-core but there are no system wide
> >  integration tests that can spin up kafka nodes and storm ,
> >  hbase and run a topology that can make sure the data is
> >  getting into hbase.
> >We at Hortonworks use our test topologies run some of these
> >tests but this integration code to spin up vms or nodes is hard
> >to share. In apache kafka we are using ducktape to write system
> >tests so far its working out good. If there are any other
> >frameworks you've in mind we can definitely take a look. But as
> >a community we need start looking at
> >https://github.com/confluentinc/ducktape or similar frameworks
> >to start writing systems tests. Ducktape makes it  easy to run
> >in a vm or any other infrastructure.  Appreciate any feedback
> >on this.
> >
> > Thanks,
> > Harsha
> >
> >
> 
> 
> 

Re: [Discusson] Storm System Tests

[Harsha]

Thanks Longda. Can you update this JIRA with list of tests cases that
you  are running in this JIRA
https://issues.apache.org/jira/browse/STORM-1179
-Harsha

On Thu, Nov 5, 2015, at 08:32 PM, 封仲淹(纪君祥LongdaFeng) wrote:
> +1
> By the way, Tong(王桐) can work for this. Tong is responsible for the
> automatically daily JStorm test, the only problem is that the
> automatically daily JStorm test is basing on Alibaba testing framework.
> Some of the components can't run on outside.
> But we can share the test cases firstly. Then we will try to port the old
> test cases to the storm tests.
> 
> regardsLongda
> --From:P.
> Taylor Goetz <ptgo...@gmail.com>Send Time:2015年11月6日(星期五)
> 12:12To:dev@storm.apache.org <dev@storm.apache.org>Subject:Re:
> [Discusson] Storm System Tests
> +1
> I think we should experiment with this. If it can serve us well, and we
> all agree, then we should consider adopting it.
> 
> From what I see:
> 
> * Icense is compatible
> * fairly well documented.
> 
> What potentially worries me:
> 
> * Maintenance. Can we count on updates? Is the ducktape community open to
> accepting patches from us? If so, what would be the turnaround time for
> acceptance?
> 
> -Taylor
> 
> > On Nov 5, 2015, at 8:54 PM, Harsha <st...@harsha.io> wrote:
> > 
> > The reason for suggesting ducktape not just used in apache kafka but
> > also its getting security services integration like kdc and already has
> > zookeeper. This framework can work with vagrant vms or amazon ec2 etc..
> > 
> >> On Thu, Nov 5, 2015, at 11:49 AM, Hugo Da Cruz Louro wrote:
> >> Great, will make this a priority. Created a
> >> JIRA<https://issues.apache.org/jira/browse/STORM-1179> ticket and
> >> assigned it to me.
> >> 
> >> https://issues.apache.org/jira/browse/STORM-1179
> >> 
> >> 
> >> On Nov 5, 2015, at 11:40 AM, Bobby Evans
> >> <ev...@yahoo-inc.com<mailto:ev...@yahoo-inc.com>> wrote:
> >> 
> >> Hugo,
> >> 
> >> I would love to see that happen.  It has been on my list for a while, but
> >> I have never found the time to do it. I personally am +1 on this,
> >> hopefully we can do this quickly in preparation for a 0.11.0 release.
> >> 
> >> - Bobby
> >> 
> >> 
> >> 
> >> On Thursday, November 5, 2015 1:37 PM, Hugo Da Cruz Louro
> >> <hlo...@hortonworks.com<mailto:hlo...@hortonworks.com>> wrote:
> >> 
> >> 
> >> I a agree with these three levels of testing, and that at the very least
> >> we should keep unit tests separated from the system/integration tests.
> >> One huge advantage would be to quickly run unit tests that hopefully are
> >> more predictable, and thus avoid intermittent test fails.
> >> 
> >> Bobby just mentioned but I had already in mind that it would be useful to
> >> create different maven profiles for integration and unit tests. I have
> >> done something similar in a different project and it works really well.
> >> If we agree that it is something we want to implement here, I will create
> >> a JIRA ticket for this and get it done. Please let me know.
> >> 
> >> Thanks,
> >> Hugo
> >> 
> >>> On Nov 5, 2015, at 11:23 AM, Bobby Evans 
> >>><ev...@yahoo-inc.com.INVALID<mailto:ev...@yahoo-inc.com.INVALID>> wrote:
> >>> 
> >>> I totally agree.  Too many of our "unit" tests are integration tests and 
> >>>end up spinning up an entire local-mode cluster.
> >>> I personally would like to see three levels of testing.
> >>> 1) true unit tests.  They only touch the code under test and very little 
> >>>else.  The should be what you get when you run mvn test2) integration 
> >>>tests.  These should spin up local mode clusters and modify configs/etc to 
> >>>get a decent set of more white box tests.  The should run as a part of 
> >>>trivis-ci, and probably should run by enabling a special profile.3) Sanity 
> >>>Integration Tests.  ducktape looks like a great fit here.  I would love to 
> >>>see us spin up an few different scenarios for testing, with/without 
> >>>security.  Talking to Kafka, Hadoop(HBase, HDFS, Hive), redis, 
> >>>elastasearch, etc.
> >>> These would be run frequently but not necessarily a part of CI initially.
> >>> - Bobby
> >>> 
> >>> 
> >>>   On Wednesday, November 4, 2015 

Re: Welcome JStorm Community!

[Harsha]

Welcome JStorm community. Thanks for your contributions to JStorm and
looking forward to working with you to make Storm better.

Thanks, Harsha

On Tue, Nov 3, 2015, at 02:05 PM, P. Taylor Goetz wrote:
> For those who may not be aware, Alibaba has donated JStorm to Apache
> Storm, and we have completed the IP Clearance process. JStorm is a
> fork of Storm with the clojure components reimplemented in Java, and
> includes a number of enhancements in terms of performance and
> features. JStorm is used widely in production at Alibaba across many
> different areas.
>
> Of equal importance is the JStorm community, which has expressed a
> commitment to join forces with the Apache Storm community to bring
> JStorm’s improvements to Apache Storm and collaborate on new features
> and improvements.
>
> Over the coming days and weeks we will have a number of discussions on
> the dev@ list regarding how to best approach merging both the incoming
> code and associated community.
>
> Please join me in welcoming the JStorm contributors to the Apache
> Storm community.
>
>
> JStorm community:
>
> Welcome abord! We’re glad you’re here.
>
> As you’ve probably noticed, all discussions and decision making at
> Apache takes place on the mailing list in English. I understand that
> might be a burden for non-English speakers, but there a number of
> Apache Storm contributors that speak both English and Chinese that
> would likely be willing to help translate. Google translate, while not
> perfect, is also helpful in some cases. We don’t expect perfect
> translation — just enough to get your point across is fine.
>
> When signing your name on an email, etc., please use the western name
> of yourself and others. This makes it easier for those of us who can't
> read Asian characters to identify individuals.
>
> If you’re new to Apache and the way it works, here is a good starting
> point for learning “The Apache Way”:
>
> https://community.apache.org/contributors/
>
> Feel free to introduce yourself to the Storm community by replying to
> this thread and letting us know a little more about yourself.
>
> Again, thank you and welcome!
>
> -Taylor Email had 1 attachment:


>  * signature.asc  1k (application/pgp-signature)

Re: [VOTE] Release Apache Storm 0.9.6 (rc2)

[Harsha]

+1
1. Ran the unit tests
2. Verified artifacts
3. Ran a cluster deployed example topologies.

-Harsha

On Thu, Oct 29, 2015, at 09:45 AM, P. Taylor Goetz wrote:
> Yes, I’ve seen that. It has something to do with clojure tests and the
> maven lifecycle.
> 
> I typically just do `mvn clean install` to run the tests.
> 
> -Taylor
> 
> > On Oct 29, 2015, at 9:38 AM, Julien Nioche <lists.digitalpeb...@gmail.com> 
> > wrote:
> > 
> > Correction. 'mvn clean compile' worked but 'mvn clean test' actually fails
> > on my machine (running linux mint)
> > 
> > [INFO]
> > 
> > [INFO] Reactor Summary:
> > [INFO]
> > [INFO] Storm . SUCCESS [1.250s]
> > [INFO] maven-shade-clojure-transformer ... SUCCESS [2.564s]
> > [INFO] Storm Core  FAILURE
> > [4:36.961s]
> > [INFO] storm-starter . SKIPPED
> > [INFO] storm-kafka ... SKIPPED
> > [INFO] storm-hdfs  SKIPPED
> > [INFO] storm-hbase ... SKIPPED
> > 
> > 
> > [ERROR] Failed to execute goal
> > com.theoryinpractise:clojure-maven-plugin:1.3.18:test-with-junit
> > (test-clojure) on project storm-core: Clojure failed. -> [Help 1]
> > org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> > goal com.theoryinpractise:clojure-maven-plugin:1.3.18:test-with-junit
> > (test-clojure) on project storm-core: Clojure failed.
> > at
> > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:217)
> > at
> > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> > at
> > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> > at
> > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
> > at
> > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
> > at
> > org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
> > at
> > org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
> > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
> > at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
> > at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
> > at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
> > at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> > org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
> > at
> > org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
> > at
> > org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
> > at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> > 
> > 
> > Clojure 1.4.0 is installed.
> > 
> > Can't see anything relevant in surefire-reports. Probably something
> > misconfigured on my machine, has anyone seen this before?
> > 
> > Julien
> > 
> > 
> > 
> > On 29 October 2015 at 11:45, Julien Nioche <lists.digitalpeb...@gmail.com>
> > wrote:
> > 
> >> +1
> >> 
> >>   - 'mvn clean test' successfully on the source
> >>   - storm-crawler compiled and worked fine using the artefacts in Nexus
> >>   - installed the binary release on single node and ran a storm-crawler
> >>   topology successfully
> >> 
> >> Thanks
> >> 
> >> Julien
> >> 
> >> On 28 October 2015 at 21:18, P. Taylor Goetz <ptgo...@gmail.com> wrote:
> >> 
> >>> This is a call to vote on releasing Apache Storm 0.9.6 (rc2).
> >>> 
> >>> Full list of changes in this release:
> >>> 
> >>> 
> >>> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=90a97ad6101de3c01233384e3dd3eeff2dde2ba3
> >>> 
> >>> The tag/commit to be voted upon

Re: [VOTE] Accept Alibaba JStorm Code Donation

[Harsha]

 +1 (binding)
-Harsha
On Tue, Oct 27, 2015, at 12:10 PM, Priyank Shah wrote:
> +1
> 
> From: "P. Taylor Goetz"
> Reply-To: <dev@storm.apache.org<mailto:dev@storm.apache.org>>
> Date: Tuesday, October 27, 2015 at 11:05 AM
> To: <dev@storm.apache.org<mailto:dev@storm.apache.org>>
> Subject: Re: [VOTE] Accept Alibaba JStorm Code Donation
> 
> +1 (binding)
> 
> On Oct 27, 2015, at 1:48 PM, P. Taylor Goetz
> <ptgo...@apache.org<mailto:ptgo...@apache.org>> wrote:
> 
> All,
> 
> The IP Clearance process for the Alibaba JStorm code donation has
> completed.
> 
> The IP Clearance Status document can be found here:
> 
> http://incubator.apache.org/ip-clearance/storm-jstorm.html
> 
> The source code can be found at https://github.com/alibaba/jstorm with
> the following git commit SHA: e935da91a897797dad56e24c4ffa57860ac91878
> 
> This is a VOTE to accept the code donation, and import the donated code
> into the Apache Storm git repository. Discussion regarding how to proceed
> with merging the codebases can take place in separate thread.
> 
> [ ] +1 Accept the Alibaba JStorm code donation.
> [ ] +0 Indifferent
> [ ] -1 Do not accept the code donation because…
> 
> This VOTE will be open for at least 72 hours.
> 
> -Taylor
> 
> 
> 

Re: [VOTE] Release Apache Storm 0.10.0 (rc1)

[Harsha]

+1
1. Ran unit tests
2. Deployed example topologies
3. Verified artifacts

-Harsha

On Mon, Oct 26, 2015, at 07:23 AM, Bobby Evans wrote:
> +1 looks good, built from the git tag and ran some simple test.
>  - Bobby 
> 
> 
>  On Friday, October 23, 2015 9:47 PM, 임정택 <kabh...@gmail.com> wrote:
>
> 
>    -
> 
>   test passed : OK
>   -- extract source tar
>   -- build via "mvn clean install”
> 
>   -
> 
>   deploy to docker cluster using wurstmeister/storm-docker : OK
>   -
> 
>   topology tests via storm-starter : OK
>   -- run WordCounts and RollingTopWords, local mode and remote mode
> 
>   -
> 
>   test REST APIs (retrieve / activate / deactivate / rebalance / kill) : OK
> 
> 
> Looks good overall. +1.
> 
> 2015-10-24 5:26 GMT+09:00 P. Taylor Goetz <ptgo...@gmail.com>:
> 
> > This is a call to vote on releasing Apache Storm 0.10.0 (rc1)
> >
> > Full list of changes in this release:
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=d02f94268dec229d1125a24fdf53fa303cbc2b29
> >
> > The tag/commit to be voted upon is v0.10.0:
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=45b1b148401fd05f0f79cc7abdf6b5c7fc43df20;hb=d02f94268dec229d1125a24fdf53fa303cbc2b29
> >
> > The source archive being voted upon can be found here:
> >
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-0.10.0/apache-storm-0.10.0-src.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-0.10.0/
> >
> > The release artifacts are signed with the following key:
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >
> > The Nexus staging repository for this release is:
> >
> > https://repository.apache.org/content/repositories/orgapachestorm-1025
> >
> > Please vote on releasing this package as Apache Storm 0.10.0.
> >
> > When voting, please list the actions taken to verify the release.
> >
> > This vote will be open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache Storm 0.10.0
> > [ ]  0 No opinion
> > [ ] -1 Do not release this package because...
> >
> > Thanks to everyone who contributed to this release.
> >
> > -Taylor
> >
> 
> 
> 
> -- 
> Name : 임 정택
> Blog : http://www.heartsavior.net / http://dev.heartsavior.net
> Twitter : http://twitter.com/heartsavior
> LinkedIn : http://www.linkedin.com/in/heartsavior
> 
>   

Re: worker-launcher - there is now native code included in Storm?

[Harsha]

Hi Erik,
  yes native code is included as part of distro but this is only
  used if you enable security and set
  supervisor.run.worker.as.users otherwise everything else
  remains the same like before. For more info you can take a
  look at this doc
https://github.com/apache/storm/blob/master/SECURITY.md

-Harsha

On Mon, Sep 14, 2015, at 03:10 PM, Erik Weathers wrote:
> It seems that in storm 0.10 we have native code (C) that is being used
> for
> launching workers.  I haven't found any reference to how this might
> affect
> deployment, testing, etc.  Is there any documentation about this new
> requirement?  I know it's related to the "launch storm under different
> usernames" feature.  But is the native stuff optional?
> 
> - Erik

Re: [VOTE] Release Apache Storm 0.10.0

[Harsha]

Taylor,
   STORM-1039 & STORM-1042 are blockers without these UI won't
   work. We need to include these in the release as well.
Thanks,
Harsha

On Fri, Sep 11, 2015, at 10:41 PM, 임정택 wrote:
> +1
> 
> - test passed : OK
> -- extract source tar
> -- build via "mvn clean install”
> 
> - topology tests via storm-starter : OK
> -- run WordCounts and RollingTopWords, local mode and remote mode
> 
> - test REST APIs (retrieve / activate / deactivate / kill) : OK
> 
> Looks good overall.
> 
> Btw, there're some issues which seems to point to 0.10.0, STORM-912
> <http://issues.apache.org/jira/browse/STORM-912> and STORM-1039
> <https://issues.apache.org/jira/browse/STORM-1039>, STORM-1042
> <https://issues.apache.org/jira/browse/STORM-1042>.
> I'm not sure we should resolve these before releasing.
> No issue is marked as blocker, but STORM-1042 is marked as critical, and
> it
> seems to be regression.
> 
> 2015-09-12 5:37 GMT+09:00 P. Taylor Goetz <ptgo...@gmail.com>:
> 
> > This is a call to vote on releasing Apache Storm 0.10.0.
> >
> > Full list of changes in this release:
> >
> > hhttps://
> > git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=f6c0abb0bd76867b48c7b7c7cfdfdc7e0e6f483a
> >
> > The tag/commit to be voted upon is v0.10.0:
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=66004167d440d418ec6c82c6a134a9c213d632b2;hb=f6c0abb0bd76867b48c7b7c7cfdfdc7e0e6f483a
> >
> > The source archive being voted upon can be found here:
> >
> >
> > http://people.apache.org/~ptgoetz/apache-storm-0.10.0/apache-storm-0.10.0-src.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> > http://people.apache.org/~ptgoetz/apache-storm-0.10.0/
> >
> > The release artifacts are signed with the following key:
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >
> > The Nexus staging repository for this release is:
> >
> > https://repository.apache.org/content/repositories/orgapachestorm-1019
> >
> > Please vote on releasing this package as Apache Storm 0.10.0.
> >
> > When voting, please list the actions taken to verify the release.
> >
> > This vote will be open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache Storm 0.10.0
> > [ ]  0 No opinion
> > [ ] -1 Do not release this package because...
> >
> > Thanks,
> > Taylor
> >
> 
> 
> 
> -- 
> Name : 임 정택
> Blog : http://www.heartsavior.net / http://dev.heartsavior.net
> Twitter : http://twitter.com/heartsavior
> LinkedIn : http://www.linkedin.com/in/heartsavior

Re: Nexusguard is using storm

[Harsha]

Hi Juniman,
  Thanks for the mail. Can you send us your logo I'll get it
  included in the new storm website.
Thanks,
Harsha

On Wed, Sep 9, 2015, at 09:47 PM, Juniman Kasman wrote:
> Hi storm team,
> We are using Storm and would like to be added below
> http://storm.apache.org/documentation/Powered-By.html
> 
> As a longtime leader in DDoS Defense, Nexusguard is at the forefront of
> the
> fight against malicious internet attacks, protecting organization
> worldwide
> from threats to their websites, services and reputations.
> 
> Storm powers Nexusguard's large-scale realtime threat analytics platform,
> processing billions of logs per day and growing. We also use Storm to
> track
> botnet, attack trend globally
> 
> 
> *Juniman Kasman*
> Chief Technology Officer - Service Provider
> T : [852] 3526 0626  ext 6603
> F : [852] 3526 0086
> M : [852] 62974888
> Skype: easternboyz
> 
> *NEXUSGUARD*
> www.nexusguard.com
> LinkedIn <https://www.linkedin.com/company/nexusguard> • Twitter
> <https://www.twitter.com/nexusguard> • Facebook
> <https://www.facebook.com/nxg.pr>
> 
> Disclaimer: This e-mail message contains information intended solely for
> the intended recipient and is confidential or private in nature. If you
> are
> not the intended recipient, you must not read, disseminate, distribute,
> copy or otherwise use this message or any file attached to this message.
> Any such unauthorized use is prohibited and may be unlawful. If you have
> received this message in error, please notify the sender immediately by
> email, facsimile or telephone and then delete the original message from
> your machine.
> 
> San Francisco  |  San Jose  |  Los Angeles  |  Miami  |  London  |  Hong
> Kong  |  Beijing  |  Taipei  |  Manila

Re: Submitting multiple jars to a topology classpath

[Harsha]

Abhishek,
          Can you explain whats your use case and the need for uploading 
multiple jars without packaging together. As others have noted since storm 
expected  to have multiple topologies in ordered to prevent dependency 
conflicts across the topologies its better to submit one jar with all of its 
dependencies included in the jar.

Thanks,
Harsha


On July 29, 2015 at 11:57:19 PM, Abhishek Agarwal (abhishc...@gmail.com) wrote:

Currently, as far as I know one has to package all the dependencies into  
one jar and then submit it along with topology class. StormSubmitter  
interface also allows only one jar. Is there any particular reason for this  
limitation?  

We have a use case where we want to upload more than one jar without  
packaging them together. How could this be achieved?  

--  
Regards,  
Abhishek Agarwal  

Re: Storm Project

[Harsha]

Hi Gaurav,
 Apache Storm development is very active and so is the community
 . If you look at our github repository here 
https://github.com/apache/storm you can see we are all very busy in
building storm to add new features, improvements.
Appreciate that you took timeout to ask us :). As others noted we are
very much active and motivated community to make storm even better.

Thanks,
Harsha


 

On Sat, Jul 18, 2015, at 11:39 AM, Parth Brahmbhatt wrote:
 Have you seen this 
 http://yahoohadoop.tumblr.com/post/122544585051/apache-storm-roadmap-at-yah
 oo ? Or if you still have doubts just look at the last beta release which
 happened less than a month ago.
 https://storm.apache.org/2015/06/15/storm0100-beta-released.html.
 
 
 Such baseless rumors are generally started from marketing/sales people
 from competition but lately I have also seen fellow developers doing this
 which is disheartening to say the least.
 If you can share with us(in a private Email is fine) who said this and
 where we can talk to them and educate them about the hard work, we as a
 community are putting in to make the best stream processing framework
 even
 better. 
 
 Thanks
 Parth
 
 On 7/18/15, 4:06 AM, 임정택 kabh...@gmail.com wrote:
 
 Interesting. :D
 
 I'd like to see slide or video which broadcasts so untrue thing.
 
 I was invited as committer / PMC member at last month. If project is
 inactive, such vote and invitation are not necessary.
 
 Anyway, for your information, you can subscribe Storm dev. mailing list to
 see what Storm community is implementing, discussing, and so on.
 
 Archive page of Storm dev. mailing list is here.
 http://mail-archives.apache.org/mod_mbox/incubator-storm-dev/
 
 You can see a news of new releases from http://storm.apache.org/news.html
 
 Hope this helps.
 
 Best,
 Jungtaek Lim (HeartSaVioR)
 
 2015년 7월 18일 토요일, Gaurav Sehgalgsehg...@gmail.com님이 작성한 메시지:
 
  Hello,
I was recently attending a technical conference of Apache
 Spark,
  where they kept saying that Storm has been deprecated and there is no
 more
  future development in Storm. Is this actually true as we are using this
 in
  our production.
 
  Cheers!
  Gaurav
 
 
 
 -- 
 Name : 임 정택
 Blog : http://www.heartsavior.net / http://dev.heartsavior.net
 Twitter : http://twitter.com/heartsavior
 LinkedIn : http://www.linkedin.com/in/heartsavior
 

Bay area meetup group for Apache Storm Kafka

[Harsha]

If you are interested in attending bay area meetup for Apache Storm  Kafka . 
Please join the group here
http://www.meetup.com/Apache-Storm-Apache-Kafka/

Thanks,
Harsha

Re: Newbie questions about deploying to cluster

[Harsha]

Tim,
 If its production cluster I highly recommend having zk cluster
 with at least 3 nodes. So that your cluster survives in case of
 zk nodes going down.
Thanks,
Harsha

On Thu, Jun 18, 2015, at 06:07 AM, Paul Poulosky wrote:
 The machine that you submit your topology from can be any of the machines
 running daemons, or you can set up a machine that has storm installed but
 no daemons running and submit your topology from there.  All you need to
 do is make sure that all machines with storm installed, whether they are
 running daemons or not, have nimbus.host set to point to the machine
 where the nimbus daemon is running.
  
 
 
  On Thursday, June 18, 2015 6:58 AM, Matthias J. Sax
  mj...@informatik.hu-berlin.de wrote:

 
  Sure.
 
 On 06/18/2015 12:55 PM, Tim Molter wrote:
  Thank you, Matthias.
  
  Can I run nimbus and a supervisor on one machine as well (assuming
  low-load)?
  
  Thanks.
  
  On 2015_06_18 12:45 PM, Matthias J. Sax wrote:
  1) yes. nimbus.
 
  2) running zookeeper and nimbus on the same machine should not be a
  problem in general. if you have a high-load scenario this might become a
  problem (but it should be ok in most cases). If you have an increased
  fault-tolerance requirement, it is also critical to run on different
  machines (but again, it should be not required for most cases)
 
  -Matthias
 
 
  On 06/18/2015 12:42 PM, Tim Molter wrote:
  1) Let's say I have 4 machines. One running zookeeper, one running
  nimbus, and 2 running supervisor. Which host do I deploy the my topology
  via `storm jar`? The nimbus host?
 
  2) Can I run zookeeper, nimbus and manager all on one machine? Or is it
  critical to separate certain ones on separate hosts?
 
 
  
 
 
   

Re: [VOTE] Release Apache Storm 0.10.0 (rc2)

[Harsha]

+1 for the release.

1) verified keys on all release artifacts.
2) verified deploying topologies in secure and non-secure cluster.

-- 
Harsha


On June 8, 2015 at 12:06:31 PM, Parth Brahmbhatt (pbrahmbh...@hortonworks.com) 
wrote:

+1. Agree about not blocking the release.  

‹ unit tests passed  
‹ word count and Exclamation topology executed successfully.  
‹ All UI actions work as expected.  
‹ Rolling upgrade works, Tested by rebuilding the same version and copying  
the new jar under storm/lib and bouncing all daemons.  
‹ Storm secure setup works, WordCount works as expected.  
‹ Could only test storm-jdbc connector with the sample test topology and  
it works.  

Thanks  
Parth  


On 6/8/15, 8:56 AM, P. Taylor Goetz ptgo...@gmail.com wrote:  

In the big picture, the REST upload API is just one of many, many  
features/improvements added in 0.10.0, and we¹ve not leveraged it in any  
way yet (i.e. in Storm UI). I don¹t think it should block the release.  
  

Re: [VOTE] Release Apache Storm 0.9.5

[Harsha]

+1 (binding)

Verified installing a cluster and running example topologies.
Verified release artifacts keys asc, sha and md5.

Thanks,
Harsha


On May 29, 2015 at 4:43:38 PM, 임정택 (kabh...@gmail.com) wrote:

- test passed : OK  
- extract source tar : OK  
- build via mvn clean install” : OK  
- verify STORM-745 : OK  
- install binary version of storm 0.9.5 to Windows 7 : OK  
- use apache-storm-0.9.5.zip  
- run dev-zookeeper, nimbus, supervisor with default setting via  
bin/storm.cmd : OK  
- run RollingTopWords from storm-starter in remote/cluster mode via  
bin/storm.cmd : OK  
- it confirms STORM-745 is fine cause it receives two argument and  
if second parameter is not presented it runs locally  
- some tests that Storm 0.9.5 works with OSX : OK  
- install binary version of storm 0.9.5 to OSX : OK  
- use apache-storm-0.9.5.tar.gz  
- run dev-zookeeper, nimbus, supervisor with default setting via  
bin/storm : OK  
- run RollingTopWords from storm-starter in local cluster mode via  
bin/storm : OK  
- run RollingTopWords from storm-starter in remote/cluster mode via  
bin/storm : OK  

+1 (non-binding)  

Thanks!  

Jungtaek Lim (HeartSaVioR)  

2015-05-30 0:34 GMT+09:00 P. Taylor Goetz ptgo...@apache.org:  

 This is a call to vote on releasing Apache Storm 0.9.5.  
  
 Full list of changes in this release:  
  
  
 https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=c9c47b36071c02488ee1e75eb3851e58191b7534
   
  
 The tag/commit to be voted upon is v0.9.5:  
  
  
 https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=774b229d709af60241c6a239b7c0990ed63ad32a;hb=c9c47b36071c02488ee1e75eb3851e58191b7534
   
  
 The source archive being voted upon can be found here:  
  
  
 http://people.apache.org/~ptgoetz/apache-storm-0.9.5/apache-storm-0.9.5-src.tar.gz
   
  
 Other release files, signatures and digests can be found here:  
  
 http://people.apache.org/~ptgoetz/apache-storm-0.9.5/  
  
 The release artifacts are signed with the following key:  
  
  
 https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
   
  
 The Nexus staging repository for this release is:  
  
 https://repository.apache.org/content/repositories/orgapachestorm-1014/  
  
 Please vote on releasing this package as Apache Storm 0.9.5.  
  
 When voting, please list the actions taken to verify the release.  
  
 This vote will be open for at least 72 hours.  
  
 [ ] +1 Release this package as Apache Storm 0.9.5  
 [ ] 0 No opinion  
 [ ] -1 Do not release this package because...  
  
 Thanks,  
 Taylor  
  



--  
Name : 임 정택  
Blog : http://www.heartsavior.net / http://dev.heartsavior.net  
Twitter : http://twitter.com/heartsavior  
LinkedIn : http://www.linkedin.com/in/heartsavior  

[DISCUSSION] Squashing commits before merging in.

[Harsha]

Hi,
     Today we are merging into storm master branch using PRs. These PRs are 
usually contain multiple commits on different time lines .
There are few issues with this approach , the major pain point is if we want 
cherry-pick a single JIRA current approach makes it harder as we have track 
down all the commits that went in for a particular JIRA.
I am proposing we should squash these multiple commits into one singe commit 
before merging into trunk and associate the STORM JIRA in commit message.
Let me know what you think.

Thanks,
Harsha

Re: [DISCUSS] Release Storm 0.9.4 / 0.10.0

[Harsha]

Hi,
        For storm 0.10.0 please add if you want any JIRAs that should be 
included.
I want to include https://issues.apache.org/jira/browse/STORM-617

Thanks,
Harsha

On March 12, 2015 at 9:30:25 AM, T B (thomas...@arcor.de) wrote:

Hi,
        For storm 0.10.0 please add if you want any JIRAs that should be 
included.
I want to include https://issues.apache.org/jira/browse/STORM-617

Thanks,
Harsha

Re: [DISCUSS] Release Storm 0.9.4 / 0.10.0

[Harsha]

Taylor,
         STORM-617 is on me I’ll send a patch in next couple of days.

Thanks,
Harsha


On March 19, 2015 at 1:05:21 PM, P. Taylor Goetz (ptgo...@gmail.com) wrote:

With the VOTE underway for a 0.9.4 release ( if you have time, please evaluate 
the RC and vote accordingly), I’d like to turn our attention toward the 0.10.0 
release.  

I’d appreciate feedback on the state of the 0.10.0 (master) branch, especially 
from those who have been involved in developing the security-related features 
(which is one of the primary features of this branch).  

Personally, I know of several organizations that are using a build of 0.10.0 
successfully in production with security enabled. The main issue I’ve seen 
people struggle with is not a problem with code itself, but rather difficulties 
in setting up a secure cluster and interacting with other systems that are 
secured with Kerberos. I see that as more of a documentation issue, and not 
something that should block a release.  

One feature that I think is essential for 0.10.0 is STORM-634 [1] (support for 
rolling upgrades), which was recently merged. This should make post-0.10.0 
upgrades a lot less painful for users.  

Based on the feedback received thus far, it looks like all the suggested 
patches have either been merged or are ready to be merged, with the exception 
of STORM-617 [2] (no patch available yet).  

Should we wait for STORM-617? Any other patch suggestions?  

-Taylor  


[1] https://issues.apache.org/jira/browse/STORM-634  
[2] https://issues.apache.org/jira/browse/STORM-617  

On Mar 12, 2015, at 12:29 PM, T B thomas...@arcor.de wrote:  

 +1 for releasing 0.9.4  
  
 Thomas  
  
 On 11 March 2015 at 21:39, Richard Kellogg rmkell...@comcast.net wrote:  
  
 Suggest we pull in STORM-559 as well. It is strictly an update to  
 documentation and has already been merged to trunk.  
  
 -Original Message-  
 From: P. Taylor Goetz [mailto:ptgo...@gmail.com]  
 Sent: Wednesday, March 11, 2015 4:51 PM  
 To: dev@storm.apache.org  
 Subject: Re: [DISCUSS] Release Storm 0.9.4 / 0.10.0  
  
 Thanks for the feedback Richard, Parth, and Jungtaek. I’ve made a note of  
 your suggestions for the releases.  
  
 Does anyone else have any thoughts (esp. committers)? Should we move  
 forward with releasing?  
  
 -Taylor  
  
 On Mar 5, 2015, at 5:00 PM, 임정택 kabh...@gmail.com wrote:  
  
 storm-redis (scheduled to be released at 0.10.0) has one bugfix and  
 one essential feature PRs.  
  
 - bugfix: https://issues.apache.org/jira/browse/STORM-690  
 -- It fixes connection pool issue.  
 - feature: https://issues.apache.org/jira/browse/STORM-691  
 -- It provides basic lookup / persist bolts so I believe it's necessary.  
  
 Furthermore, I'd like to continue to support various data types with  
 storm-redis Trident, after STORM-691 is merged to master.  
  
 Thanks!  
  
 Regards  
 Jungtaek Lim (HeartSaVioR)  
  
  
 2015-03-06 2:37 GMT+09:00 P. Taylor Goetz ptgo...@gmail.com:  
  
 I’d like to start a discussion for releasing 0.9.4 (maintenance  
 release) and 0.10.0 (security release).  
  
 0.9.4 is basically a branch of 0.9.3 with two important bug fixes:  
  
 STORM-329: fix cascading Storm failure by improving  
 reconnection strategy and buffering messages  
 STORM-130: Supervisor getting killed due to  
 java.io.FileNotFoundException: File '../stormconf.ser' does not exist.  
  
 Both are long-standing bugs that have proven problematic for many users.  
 I’d be in favor of releasing 0.9.4 with just those two fixes, but I’m  
 interested in finding out if anyone thinks there are additional  
 patches to master that should be considered for 0.9.4.  
  
 0.10.0 is a much larger release in terms of changes. In addition to  
 the changes above, it includes all the new security features and  
 numerous fixes and enhancements (see the CHANGELOG in the master branch  
 for a full list).  
  
 Do we feel 0.10.0 is ready for release? If not what outstanding  
 bugs/patches should we consider before releasing?  
  
 I’m fine holding off on a 0.10.0 release if we feel there is  
 additional work to be done, but I’d like to at least move forward with  
 0.9.4 release.  
  
 Thoughts?  
  
 -Taylor  
  
  
  
  
 --  
 Name : 임 정택  
 Blog : http://www.heartsavior.net / http://dev.heartsavior.net Twitter  
 : http://twitter.com/heartsavior LinkedIn :  
 http://www.linkedin.com/in/heartsavior  
  
  
  

Re: Contributing to Storm

[Harsha]

Hi Oleg,
       Here is a JIRA https://issues.apache.org/jira/browse/STORM-710 to start 
on.
Thanks,
Harsha
On March 18, 2015 at 9:34:36 AM, Oleg Ostashchuk (ostas...@fel.cvut.cz) wrote:

Hello,  

I would like to contribute to Apache Storm project. Can you please  
advise me some issue or work to do?  
It is my first attempt to contribute in Apache Storm project, so it will  
be fine if I can start with something easier.  

Thanks a lot!  
Oleg Ostashchuk  

Re: [DISCUSS] Release Storm 0.9.4 / 0.10.0

[Harsha]

Sorry for the late reply. I am +1 on releasing 0.9.4 . For 0.10.0 I would like 
to include this https://issues.apache.org/jira/browse/STORM-617

Thanks,
Harsha

On March 11, 2015 at 1:52:02 PM, P. Taylor Goetz (ptgo...@gmail.com) wrote:

Sorry for the late reply. I am +1 on releasing 0.9.4 . For 0.10.0 I would like 
include this https://issues.apache.org/jira/browse/STORM-617

Thanks,
Harsha

Re: New Committer/PMC Member: Thomas Becker

[Harsha]

Congrats Thomas and Welcome :)
-- 
Harsha


On March 10, 2015 at 11:09:05 AM, P. Taylor Goetz (ptgo...@apache.org) wrote:

Please join me in welcoming Thomas Becker (wurstmeister) as a new Apache Storm 
Committer/PMC member.  

Thomas has made significant contributions to the Storm codebase, particularly 
in the area of Kafka integration, and shown dedication to advancing the Storm 
community.  

Congratulations and welcome Thomas!  

-Taylor  

Re: [VOTE] Adopt Apache Storm Project Bylaws

[Harsha]

+1
-Harsha

On Thu, Feb 19, 2015, at 07:48 AM, Melan Nimesh wrote:
 +1
 
 On Thu, Feb 19, 2015 at 9:08 PM, Nathan Marz nat...@nathanmarz.com
 wrote:
 
  +1
 
  On Thu, Feb 19, 2015 at 8:15 AM, Andy Feng andy.f...@gmail.com wrote:
 
   +1
  
   Andy Feng
  
   Sent from my iPhone
  
On Feb 18, 2015, at 1:43 PM, P. Taylor Goetz ptgo...@apache.org
  wrote:
   
As a follow-up to the previous discussion regarding adopting project
   bylaws, I’d like to start an official VOTE to formally adopt the bylaws
  as
   listed below.
   
Please vote on adopting the proposed bylaws.
   
[+1] Adopt the bylaws as listed
[+0] No opinion
[-1] Do not adopt the bylaws because…
   
This vote will be 2/3 Majority as described below, and open for 6 days.
   
-Taylor
   
-
   
# Apache Storm Project Bylaws
   
   
## Roles and Responsibilities
   
Apache projects define a set of roles with associated rights and
   responsibilities. These roles govern what tasks an individual may perform
   within the project. The roles are defined in the following sections:
   
### Users:
   
The most important participants in the project are people who use our
   software. The majority of our developers start out as users and guide
  their
   development efforts from the user's perspective.
   
Users contribute to the Apache projects by providing feedback to
   developers in the form of bug reports and feature suggestions. As well,
   users participate in the Apache community by helping other users on
  mailing
   lists and user support forums.
   
### Contributors:
   
Contributors are all of the volunteers who are contributing time, code,
   documentation, or resources to the Storm Project. A contributor that
  makes
   sustained, welcome contributions to the project may be invited to become
  a
   Committer, though the exact timing of such invitations depends on many
   factors.
   
### Committers:
   
The project's Committers are responsible for the project's technical
   management. Committers have access to all project source repositories.
   Committers may cast binding votes on any technical discussion regarding
   storm.
   
Committer access is by invitation only and must be approved by lazy
   consensus of the active PMC members. A Committer is considered emeritus
  by
   their own declaration or by not contributing in any form to the project
  for
   over six months. An emeritus Committer may request reinstatement of
  commit
   access from the PMC. Such reinstatement is subject to lazy consensus
   approval of active PMC members.
   
All Apache Committers are required to have a signed Contributor License
   Agreement (CLA) on file with the Apache Software Foundation. There is a
   [Committers' FAQ](https://www.apache.org/dev/committers.html) which
   provides more details on the requirements for Committers.
   
A Committer who makes a sustained contribution to the project may be
   invited to become a member of the PMC. The form of contribution is not
   limited to code. It can also include code review, helping out users on
  the
   mailing lists, documentation, testing, etc.
   
### Project Management Committee(PMC):
   
The PMC is responsible to the board and the ASF for the management and
   oversight of the Apache Storm codebase. The responsibilities of the PMC
   include:
   
 * Deciding what is distributed as products of the Apache Storm
  project.
   In particular all releases must be approved by the PMC.
 * Maintaining the project's shared resources, including the codebase
   repository, mailing lists, websites.
 * Speaking on behalf of the project.
 * Resolving license disputes regarding products of the project.
 * Nominating new PMC members and Committers.
 * Maintaining these bylaws and other guidelines of the project.
   
Membership of the PMC is by invitation only and must be approved by a
   consensus approval of active PMC members. A PMC member is considered
   emeritus by their own declaration or by not contributing in any form to
   the project for over six months. An emeritus member may request
   reinstatement to the PMC. Such reinstatement is subject to consensus
   approval of the active PMC members.
   
The chair of the PMC is appointed by the ASF board. The chair is an
   office holder of the Apache Software Foundation (Vice President, Apache
   Storm) and has primary responsibility to the board for the management of
   the projects within the scope of the Storm PMC. The chair reports to the
   board quarterly on developments within the Storm project.
   
The chair of the PMC is rotated annually. When the chair is rotated or
   if the current chair of the PMC resigns, the PMC votes to recommend a new
   chair using Single Transferable Vote (STV) voting. See
   http://wiki.apache.org/general/BoardVoting for specifics. The decision
   must be ratified

Re: HdfsBolt and hdfs in HA mode

[Harsha]

Clay,
 When you are using storm-hdfs connector you need to package
 core-site.xml and hdfs-site.xml form you cluster into your topology
 jar . You can configure the storm-hdfs bolt to pass nameserviceID

HdfsBolt bolt = new HdfsBolt()
   .withFsURL(hdfs://myNameserviceID)
   .withFileNameFormat(fileNameformat)
   .withRecordFormat(format)
   .withRotationPolicy(rotationPolicy)
   .withSynPolicy(syncPolicy);

The above is all that needed to use namenode HA with your storm-hdfs. 

-Harsha

On Thu, Feb 19, 2015, at 08:58 AM, Bobby Evans wrote:
 Hadoop has lots of different configurations in core-site.xml,
 hdfs-site.xml, ... all of which eventually get loaded into the
 Configuration object used to create a FileSystem instance.  There are so
 many different configurations related to security, HA, etc. that it is
 almost impossible for me to guess exactly which ones you need to have set
 correctly to make this work.  Typically what we do for storm to be able
 to talk to HDFS is to package the complete set of configs that appear on
 a Hadoop Gateway with the topology jar when it is shipped.  This
 guarantees that the config is the same as on the gateway and should
 behave the same way.  You can also grab them from the name node or any of
 the hadoop compute nodes. 
  This will work for the HdfsBolt that loads default configurations from the 
 classpath before overriding them with any custom configurations you set for 
 that bolt.
 
 - Bobby
  
 
  On Thursday, February 19, 2015 10:42 AM, clay teahouse
  clayteaho...@gmail.com wrote:

 
  Bobby,What do you mean by client here? In this context, do you consider
  hdfsbolt a client? If yes, then which configuration you are referring
  to? I've seen the following, but I am not sure if I follow.

- dfs.client.failover.proxy.provider.[nameservice ID] - the Java class
that HDFS clients use to contact the Active NameNodeConfigure the name
of the Java class which will be used by the DFS Client to determine
which NameNode is the current Active, and therefore which NameNode is
currently serving client requests. The only implementation which
currently ships with Hadoop is the ConfiguredFailoverProxyProvider, so
use this unless you are using a custom one. For example:   property
   namedfs.client.failover.proxy.provider.mycluster/name
   
 valueorg.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider/value
 /property
 thanks,Clay
 
 On Thu, Feb 19, 2015 at 8:38 AM, Bobby Evans
 ev...@yahoo-inc.com.invalid wrote:
 
 HDFS HA provides fail-over for the name node and the client determines
 which name node is the active one but should be completely transparent to
 you if the client is configured correctly.
  - Bobby
 
 
      On Thursday, February 19, 2015 6:47 AM, clay teahouse 
 clayteaho...@gmail.com wrote:
 
 
  Hi All,
 Has anyone used HdfsBolt with hdfs in HA mode? How would you determine
 which hdfs node is the active node?
 
 thanks
 Clay
 
 
    
 
 
 

Re: [DISCUSS] Adopt Apache Storm Bylaws

[Harsha]

+1 

On Fri, Feb 13, 2015, at 07:57 AM, Bobby Evans wrote:
 +1 - Bobby
  
 
  On Friday, February 13, 2015 1:10 AM, Nathan Marz
  nat...@nathanmarz.com wrote:

 
  +1
 
 On Thu, Feb 12, 2015 at 5:57 PM, P. Taylor Goetz ptgo...@gmail.com
 wrote:
 
  Pull request updated.
 
  Here’s a link to the latest commit:
  https://github.com/ptgoetz/storm/commit/18a68a074570db01fc6377a269feb90ecda898ab
 
  - Taylor
 
  On Feb 12, 2015, at 8:41 PM, P. Taylor Goetz ptgo...@gmail.com wrote:
 
   Great hear. I will update the pull request accordingly.
  
   -Taylor
  
  
   On Feb 12, 2015, at 5:24 PM, Derek Dagit der...@yahoo-inc.com.INVALID
  wrote:
  
   I am OK with codifying the retroactive -1 as proposed by Nathan, and I
   am otherwise OK with the proposed bylaws.
   --
   Derek
  
  
  
   - Original Message -
   From: Bobby Evans ev...@yahoo-inc.com.INVALID
   To: dev@storm.apache.org dev@storm.apache.org
   Cc:
   Sent: Thursday, February 12, 2015 8:12 AM
   Subject: Re: [DISCUSS] Adopt Apache Storm Bylaws
  
   That seems fine to me.  Most other projects I have worked on follow a
  similar procedure, and a retroactive -1 can be applied, without having it
  codified, but making it official seems fine to me.
   I am +1 for those changes.
   - Bobby
  
  
  
      On Thursday, February 12, 2015 2:23 AM, Nathan Marz 
  nat...@nathanmarz.com wrote:
  
  
   Yes, I would like to codify it. It's not about there being a bug with a
   patch – it's about realizing that particular patch does not fit in with
  a
   coherent vision of Storm, or that functionality could be achieved in a
   completely different way. So basically, preventing bloat. With that
  change
   I'm +1 to the bylaws and I believe we would have a consensus.
  
   On Wed, Feb 11, 2015 at 7:34 PM, P. Taylor Goetz ptgo...@gmail.com
  wrote:
  
   I have no problem with your proposal. Actually I never even considered
   setting a timeline for a revert. I've always felt that if there was any
   problem with a patch/modification, it could be reverted at any time --
  no
   deadline. If we find a problem, we fix it. We've reverted changes in
  the
   past, and lived to tell about it :).
  
   So I would think we don't even have to mention any revert timeline. If
  we
   feel the need to codify that, I'm okay with it.
  
   -Taylor
  
   On Feb 11, 2015, at 9:06 PM, Nathan Marz nat...@nathanmarz.com
  wrote:
  
   I'm -1 on these bylaws. This commit process encourages merging as
  fast as
   possible and does not give adequate time for dissenting opinions to
  veto
   a
   patch. I'm concerned about two things:
  
   1. Regressions - Having too lax of a merge process will lead to
   unforeseen
   regressions. We all saw this first hand with ZeroMQ: I had to freeze
  the
   version of ZeroMQ used by Storm because subsequent versions would
  regress
   in numerous ways.
   2. Bloat – All software projects have a tendency to become bloated and
   build complexity because things were added piecemeal without a
  coherent
   vision.
  
   These are very serious issues, and I've seen too many projects become
   messes because of them. The only way to control these problems are
  with
   -1's. Trust isn't even the issue here – one committer may very well
   think a
   new feature looks fine and why not let it in, while another will
   recognize that the feature is unnecessary, adds complexity, and/or
  can be
   addressed via better means. As is, the proposed bylaws are attempting
  to
   make vetoing very difficult.
  
   I have a proposal which I believe gets the best of all worlds:
  allowing
   for
   fast responsiveness on contributions while allowing for regressions
  and
   bloat to be controlled. It is just a slight modification of the
  current
   bylaws:
  
   A minimum of one +1 from a Committer other than the one who authored
  the
   patch, and no -1s. The code can be committed after the first +1. If a
  -1
   is
   received to the patch within 7 days after the patch was posted, it
  may be
   reverted immediately if it was already merged.
  
   To be clear, if a patch was posted on the 7th and merged on the 10th,
  it
   may be -1'd and reverted until the 14th.
  
   With this process patches can be merged just as fast as before, but it
   also
   allows for committers with a more holistic or deeper understanding of
  a
   part of Storm to prevent unnecessary complexity.
  
  
   On Tue, Feb 10, 2015 at 7:48 AM, Bobby Evans
  ev...@yahoo-inc.com.invalid
  
   wrote:
  
   I am fine with this. I mostly want a starting point, and we can
  adjust
   things from there is need be.
   - Bobby
  
  
      On Sunday, February 8, 2015 8:39 PM, Harsha st...@harsha.io
   wrote:
  
  
  
   Thanks for putting this together. Proposed bylaws looks good to
   me. -Harsha
  
  
   On Thu, Feb 5, 2015, at 02:10 PM, P. Taylor Goetz wrote:
   Associated pull request can be found here:
   https://github.com/apache/storm/pull/419
  
  
   This is another