Thanks Bobby for sharing your opinion with great ground data. I agree most of your opinion, but having a bit different view for recent status. Maybe difference of view came from the time period. I don't think project were not healthy for a long time, but from a last half an year is not same as it was.
This is contributors graph for last 6 months: https://github.com/apache/storm/graphs/contributors?from=2016-04-01&to=2016-10-31&type=c and same period in last year: https://github.com/apache/storm/graphs/contributors?from=2015-04-01&to=2015-10-31&type=c Anyway, whether we think this is healthy or not, it's commit-wise aspect. While I strongly agree commit-wise is the huge part of success of project (for competitiveness), other things are also other huge parts of it. (more discussions about project itself, caring users and individual contributors not too late, releasing timely, etc.) Looking at activity of Storm user group and dev group give another view of project's activity. http://mail-archives.apache.org/mod_mbox/storm-user/ http://mail-archives.apache.org/mod_mbox/storm-dev/ http://mail-archives.apache.org/mod_mbox/storm-issues/ I didn't go deep so don't know what kinds of threads are decreased, but we could take it into account. Release period gets longer, and we seem less interested to participate release discussion and vote. (Recent release vote were open more than 1 week due to lack of binding votes.) Personally, any new features or improvements are less meaningful if it's not released to official version with proper documentation or publicity. I see the roadmap now, and I think they're all amazing improvements which users would really love to have. (coincidentally, Heron seems to be working on dynamic scaling.) How about sharing the roadmap? (sure, only if possible) If I were one of users, I'm not able to forecast the near future and far future of Apache Storm. If we have detailed milestone that would be really great. If we break down into small enough sub-tasks, some contributors can help us, and that's what I intend to share Storm SQL milestone. - Jungtaek Lim (HeartSaVioR) 2016년 10월 22일 (토) 오전 12:20, Bobby Evans <[email protected]>님이 작성: Before we get too deeply into your questions I would like to address the overarching question that you seem to be asking. Is the project healthy? I personally think that we are healthy. Could we be doing better, yes of course. I would like to grab some stats from github to try and make a point here. https://github.com/apache/storm/graphs/contributors?from=2014-01-01&to=2016-10-21&type=c Since 2014 we have had a large number of contributions (> 1000 commits) That looks to be about half as much as flink https://github.com/apache/flink/graphs/contributors?from=2014-01-01&to=2016-10-21&type=c but not that far behind subversion (I know subversion on git is a bit odd) https://github.com/apache/subversion/graphs/contributors?from=2014-01-01&to=2016-10-21&type=c I looked at many others. Hadoop core (HDFS + YARN + MR) https://github.com/apache/hadoop/graphs/contributors?from=2014-01-01&to=2016-10-21&type=c Lucene-solrhttps:// github.com/apache/lucene-solr/graphs/contributors?from=2014-01-01&to=2016-10-21&type=c ... even Spark https://github.com/apache/spark/graphs/contributors?from=2014-01-01&to=2016-10-21&type=c In general from a pure commits bases, and even several others metrics I looked at it does not look that bad to me. I had this same concern a year or more ago and have been looking at the metrics ever since. Even the number of PMC members and contributors leaving the project is not that scary to me. I am a PMC member on multiple projects Storm, Spark, Hadoop, and Tez. My job focus has changed over the years so now I concentrate primarily on Storm. Could that change again in the future? Very possibly. This has happened for others too. My biggest concern would be more with do we have customers replacing storm with other streaming systems. If so why are they going? Do we have new customers showing up? What are the pain points that people mention about the project? The more people use the project the more people will want to contribute back is my view of this. Although if people do have suggestions about things we can change, bylaws etc, to make the project more approachable to new devs I would be happy to discuss them and adopt them. I would love to see storm get the attention that spark is seeing, and if we can find a way through marketing/features to make that happen it would be great. To me a lot of this really is marketing. Storm meets the needs of most people at Yahoo for what they want to do. Many of them have tried other solutions, especially spark streaming, and have always come back to storm because it works really well and does what they want it to do. That said I have kind of a closed view of things here. I am not out doing market research, I am looking at what our customers internally want. For us our road map is around better usability and resource utilization. Resource Aware Scheduling, elasticity, better metrics, better UI, and possibly improving the underlying workers to be better efficient. Others would need to talk about their road maps, but mostly it looks like SQL, BEAM, and a better API. - Bobby On Thursday, October 20, 2016 10:58 PM, Jungtaek Lim <[email protected]> wrote: Hi devs, If you also subscribe the user mailing list, you might see the mail 'Is trident dead? < http://mail-archives.apache.org/mod_mbox/storm-user/201610.mbox/%3CCAP3bR3xmOb=pMXa=aj--_=j=bqoakqytdwezfxysxsufdky...@mail.gmail.com%3E > '. I don't remember (and also don't mind) the origin question and following talks, but while talking, he use the word "commit-wise active" which seems to be the thing we may need to think about. (even I replied the opposite to user mailing list.) Apart from contributing features, open source project itself needs some efforts to maintain and publicize. While efforts for former is happening continuously, but I don't see great efforts for latter, for months. We have very limited resource and it could be a way to maximize our efficiency, but it is clearly showing the downsides: community is losing activity, and individual contributors left and don't come. IMHO, we may need to think about below questions clearly. There're multiple subjects so we may want to classify and may discuss. Most of questions may need to think deeply and not be easy to answer. - What's the project's goal? In other words, what's the reason we have been putting great efforts? - Are we documenting or publicizing our efforts, especially new features pretty well? - Why our heavy individual contributors or even PMCs leave the community, and why we haven't seen new heavy contributors? - How we bring back activity of Storm community: users and contributors? - How long is the release cycle of minor version of Storm so that users can take the benefit of our efforts? - Are our contribution / merge / release process pragmatic? Thanks, Jungtaek Lim (HeartSaVioR)
