I love this discussion.
Let's list the facts first:
1. JStorm and Storm have different feature set, but we both target on the
same goals.
2. We have two community and doing things in duplicated fashion.
3. Both realize that It will take huge efforts to absort the common and
good things from the other with different source repo.
4. Storm doesn't preclude java, Clojure and Java works pretty well together
in storm. As Talyor said 66% of code is java.
5. Both want to do thing in a open way in apache that can everyone around
the world is comfortable, and be benefited from this process.
So, IMHO, based on the above facts, we should merge these two together
under the name of storm apache project.
For the difference of these two projects, here is my suggestions:
1. Review the change at the feature level.
Instead of saying whether we should replace the core from clojure to
java as a whole, let's think in the view of single feature. What is the
best decision for this feature. In my opinion, storm are open to the
changes, including changing some part of clojure to java, but only when
necessary.
2. Make the new feature plugable, and general.
If we can make some plug point in storm clojure core, then the
implementation can be either in java or clojure.
For example, like the JStorm scheduler feature, it is more powerful than
storm. Can we stick to the java code JStorm scheduler, and make a plug
point in storm?
They are good examples for this practice in storm already. Like the storm
messaging layer. The netty implementation is in java, the plugin point is
in clojure. By following this process, it can largely reduce the refactory
efforts we need to make.
3. Colloborate to solve the gap of clojure language, across Jstorm and
Storm community.
Though I personally think Clojure is not that hard to learn, but it do
requires a steep learning curve. Insteading of letting JStorm to program
with a new language from zero, I suggest we stick to our most productive
language in the begining, and do things colloborately in community. Java
guys and Clojure guys from two community can decide the interface(plug
point) first, works seperately in their own favorable language.
4. More communication channels beyond mail to negotiate the difference and
make sure we have same understanding.
We also need some online chat, face to face meetup to do a more effiicient
and fast loop communication and negotiation. It is important to establish
common understanding.
Since JStorm guys are mostly in China. There are two guys of storm
commitor in China, Xu Mingming and me(located @Intel Shanghai). I would
love to volunteer to be one additional channel for early stage discussion,
so that we can minimize the differences faster. By then, we will have a
further considerate proposal that are acceptable for both community.
Sean
On Thu, Oct 30, 2014 at 9:39 AM, 임정택 <kabh...@gmail.com> wrote:
> As a small contributor but willing to contribute more, let me give just 2
> cents to Storm community.
>
> I think it's essential to keep Storm's originality, so merging with JStorm
> should be treated seriously. And let JStorm developers learn clojure and
> contribute to Storm may be faster than merging.
>
> But at this time, we can think about why JStorm is born. It's due to
> learning curve of Clojure.
> I agree Taylor's comment, developers need to be polyglot programmer.
> But I don't agree we can/should learn all of languages whenever we need to.
> It should be a big wall.
>
> Think about a newbie Storm contributors that know only Java.
> Not intended to make a flame between languages, but if he/she need to learn
> a language runs with JVM but not Java, he/she will find which language is
> widely used, with regarding trends.
>
> And it's github trending repositories, clojure and scala.
> https://github.com/trending?l=clojure&since=monthly
> https://github.com/trending?l=scala&since=monthly
>
> Since I don't know Scala well either, but I feel sometimes I need to learn
> Scala because many active big projects are relied on. But Clojure seems
> not.
> (Maybe I cannot find right things, so please correct me if I'm wrong.)
>
> And Scala has similar looks with OOP language, especially Java, so Java
> programmer can implement something like Java with small learning, though it
> would not beautiful and someone blames it's not Scala. But Clojure is not.
> He/She cannot read/write if he/she is familiar with Clojure, at least Lisp.
> It's not that simple hurdle to newbies about to learn new language.
>
> Btw, recently I read yahoo's storm contribution doc, 'The Evolution of
> Storm at Yahoo and Apache'. They (including committers of Storm) learned
> unfamiliar language, Clojure.
> So I have a question regarding it.
> What days/hours did you spent to get familiar with Clojure? What materials
> did you use to learn Clojure?
>
> Actually I'm committer of Jedis project, and I feel Jedis can be improved
> with advanced language but I think we can't move Jedis because Java is
> widely-used language so it's better to collect contributors.
> Seems like Storm project has same question about this. So it's interesting
> to look into discussion.
>
> Just my 2 cents.
>
> Regards.
> Jungtaek Lim (HeartSaVioR)
>
>
> 2014-10-30 5:17 GMT+09:00 P. Taylor Goetz <ptgo...@gmail.com>:
>
> > After taking some time to review the JStorm code and history of both
> > projects, here are my thoughts:
> >
> > Apache Storm is largely implemented in Java (~68%) and Clojure (~18%).
> The
> > disparity in LOC is primarily due to the conciseness of Clojure compared
> to
> > Java. Apache Storm's Clojure code consists mainly of implementations of
> > interfaces defined in Java. This fact makes it possible to provide an
> > implementation in Java itself, which is exactly what the JStorm project
> > seems to have done. JStorm is a copy of the Apache Storm codebase, with
> the
> > Clojure code being replaced with a Java implementation, with some
> > additional changes.
> >
> > The justification of the project centers around a distaste for Clojure,
> > and a seeming lack of developers with Clojure skills [1]:
> >
> > "Storm is wonderful product, but it is implemented with Clojure. We don't
> > like clojure. it isn't a populate language, so in Alibaba, few people can
> > fix the bug of storm. We are the first users of Storm since it has been
> > open source. During using, we found several problem such as zeromq,
> > zookeeper, performance.”
> >
> >
> > While I agree that to someone experienced with Java development, the
> first
> > exposure to Clojure can be a bit jarring, I don’t think it is much of a
> > barrier to entry. In my opinion, learning Clojure is a valuable
> investment
> > in making oneself a more well-rounded developer. The technology ecosystem
> > to which Storm belongs is becoming increasingly polyglot with respect to
> > JVM languages (Java, Scala, Clojure, etc.), and developers need to adapt
> to
> > that fact. As an example (perhaps not the best example), lets look at
> > Kafka. If I didn’t like or want to learn Scala, would it make sense to
> > write a clone in Java and attempt to keep feature parity and
> compatibility,
> > or would it be better to bite the bullet and learn Scala? (Yes, I know
> it’s
> > not the best example since Storm’s Clojure code is a subset of the
> > codebase, and the way it’s designed makes it possible for alternative
> > implementations). And in reality, someone from Alibaba had to at least
> > learn enough Clojure to be able to replicate Storm’s Clojure
> implementation
> > in java.
> >
> > Looking at the history of JStorm [2][3], it appears a lot of work has
> gone
> > into keeping feature parity with Apache Storm by copying in new code or
> > reimplementing it. For example a kafka spout was added about two weeks
> ago
> > [4], and Yahoo’s storm-on-yarn project was also recently copied in [5].
> >
> > To me, this kind of work is rather counter-productive. If the Apache
> Storm
> > community feels strongly that Storm should support multiple “core”
> > implementations (i.e. one implemented in Clojure, and one implemented in
> > Java), then it could be accommodated by making them pluggable and
> > rearranging the codebase a bit. That being said, the Apache Storm
> community
> > has invested a significant amount of time and effort maintaining and
> > improving Storm’s Clojure “core.” One place this is most apparent, as
> Bobby
> > mentioned, is the upcoming security release. To maintain feature parity
> > with that release a “JStorm core” would have to be updated to implement
> > that functionality, which is not a trivial undertaking. At some point as
> > developers we need to ask ourselves “what's the point of that work?”
> There
> > are a lot more people using Storm than developing internals. We need to
> > make sure that both groups gain value from whatever we decide, and that
> we
> > don’t sacrifice any value in the process.
> >
> > So to answer the question in the subject line of the email thread: Could
> > the JStorm project collaborate with Apache Storm? My answer would be a
> > resounding “yes.”
> >
> > Though the JStorm community is small in comparison to that of Apache
> > Storm, it’s clear that the JStorm contributors have a solid grasp of
> > Storm’s internals and would be welcomed by the Storm community.
> > Reimplementing Storm’s core in Java no doubt provided insight into areas
> > that might be improved.
> >
> > That being said, replacing Storm’s Clojure-based core is a decision that
> > should not be taken lightly, and would require the support of the Apache
> > Storm PMC and entire community. A “dual-core” setup could be an option,
> but
> > again could become a maintenance nightmare with little value added for
> end
> > users. I’m open to discussing options further, and welcome input from
> other
> > community members.
> >
> > In short, I would rather see us work together than apart.
> >
> > -Taylor
> >
> >
> > [1]
> >
> https://github.com/alibaba/jstorm/commit/903ca941023dd37a54d9574e2a554883111f0b4a
> > [2] https://github.com/alibaba/jstorm/commits/master
> > [3] https://github.com/alibaba/jstorm/blob/master/history.md
> > [4]
> >
> https://github.com/alibaba/jstorm/tree/master/jstorm-utility/jstorm-kafka
> > [5] https://github.com/alibaba/jstorm/tree/master/jstorm-on-yarn
> >
> > On Oct 29, 2014, at 12:14 PM, Bobby Evans <ev...@yahoo-inc.com.INVALID>
> > wrote:
> >
> > Sorry I am coming late to the conversation on this. Looking at the code
> > for jstorm I am willing to entertain the idea of merging the two
> projects.
> > I understand that by using java we can make the project more accessible
> to
> > a larger number of developers, and that does seem to fit with the apache
> > model of community over code. But I don't really want change just for
> > change's sake. Look at kill_topology in both java:https://
> >
> github.com/alibaba/jstorm/blob/master/jstorm-client/src/main/java/backtype/storm/command/kill_topology.java
> >
> > and in clojure:https://
> >
> github.com/apache/storm/blob/master/storm-core/src/clj/backtype/storm/command/kill_topology.clj
> >
> > The java version is 61 lines of code and is not actually compatible with
> > the clojure version because it just counts argument for the wait
> parameter
> > instead of using a -w. The clojure version is 29, but the 15 of which
> are
> > the apache header, so it is closer to 14 lines. Granted, to someone who
> > does not know you have to work a bit to read it, but not that much.
> > My concerns would be two fold.
> > First is the effort involved in maintaining two separate code bases, sub
> > project or otherwise, is a lot. I don't want to do it, and I assume you
> > don't want to either, that is why you are talking about merging. We
> would
> > need to decide which code to keep and which code to drop, and in all
> > likelihood some of it will still be clojure. I also would not want to
> > restrict people from using clojure for new work if they want to, just
> like
> > we don't restrict java usage for new work.
> > Second is true feature parity. Storm is a moving target, I would not
> want
> > to replace any code, even if it is faster, without verifying that it has
> > feature parity, or it is a conscious decision that the missing features
> are
> > not needed. Especially around the security branch, which is scheduled for
> > the 0.10.0 release, but is not a part of jstorm at all.
> > - Bobby
> >
> > From: 封仲淹(纪君祥) <zhongyan.f...@alibaba-inc.com>
> > To: dev@storm.apache.org; tdunn...@maprtech.com
> > Cc: jstorm_...@googlegroups.com
> > Sent: Wednesday, October 29, 2014 4:27 AM
> > Subject: 答复: Could JStorm project collaborate with Storm?
> >
> > Thanks for your suggestion.
> >
> > @dev, Does anyone has proposal? please feel free to raise.
> >
> >
> > Thanks
> > Longda
> >
> > -----邮件原件-----
> > 发件人: Ted Dunning [mailto:tdunn...@maprtech.com <tdunn...@maprtech.com>]
> > 发送时间: 2014年10月25日 7:27
> > 收件人: dev@storm.apache.org
> > 主题: Re: Could JStorm project collaborate with Storm?
> >
> > I just moved all of the non-Storm-dev addresses on this thread to bcc.
> If
> > anybody is not on the Storm dev list and would like to continue to follow
> > this conversation, you should subscribe to the storm dev list. We should
> > not be spamming everybody else.
> >
> >
> > There is a standard Apache process to deal with this. JStorm can become
> > an incubator project. Once JStorm demonstrates the ability to function
> as
> > an Apache project, it can graduate either to a top-level project or to
> > become part of a project like Storm.
> > [Longda]This will be a feasible solution. From the Apache side, it had
> > better Jstorm can collaborate with Storm.
> >
> > A second mechanism is that if the Storm group is interested, they can
> > directly adopt the code as it is, possibly asking one or more of the
> > contributors to become committers to help support this new code. That
> is a
> > much harder thing to do than incubation since it is unusual for a project
> > to be enthusiastic about supporting a large new codebase.
> > [Longda]I agree. It is a hard job.
> >
> >
> >
> >
> >
> >
> > On Thu, Oct 23, 2014 at 8:19 PM, 刘键(夏蒅) <basti...@alibaba-inc.com>
> wrote:
> >
> > Hi Andy,
> >
> >
> >
> > Thanks for your quick response. What you concern is reasonable and
> > understandable.
> >
> >
> >
> > But maybe there is some misunderstanding on our proposal. According to
> > the current development situation in China, java core is indeed to
> > bring more contributors here.
> >
> > That is reason why we begin to think if it is also a way for us to
> > provide our contribution to Storm project. It is true that it is a
> > risk to switch the core in a short period.
> >
> > Actually, if possible, maybe we can consider to take JStorm as a
> > subproject of Storm(A branch or some other ways…). We will be
> > responsible for the maintain of it and try to propagate the features
> > from Storm to JStorm, and continue to develop our features and
> > improvements on this. You guys can try the new core and take a long
> > time to estimate if it is worth doing the switch in the future or
> > taking JStorm as official optional core of Storm for user.
> >
> >
> >
> > Regards
> >
> > Basti
> >
> >
> >
> > *发件人:* Andrew Feng [mailto:af...@yahoo-inc.com <af...@yahoo-inc.com> <
> > af...@yahoo-inc.com>]
> > *发送时间:* 2014年10月24日 0:24
> > *收件人:* 封仲淹(纪君祥); nat...@nathanmarz.com; 徐明明(护城); ja...@cvk.ca;
> > f...@infochimps.com; david...@microsoft.com; ptgo...@gmail.com;
> > cutt...@apache.org; tdunn...@maprtech.com; arv...@apache.org;
> > d...@hortonworks.com; m.ben.frank...@gmail.com;
> > benjamin.hind...@gmail.com
> > *抄送:* aloha-dev; scott.z...@vipshop.com; yannia...@tencent.com;
> > mlc...@iflytek.com; z...@tencent.com
> > *主题:* Re: 答复: Could JStorm project collaborate with Storm?
> >
> >
> >
> > Zhongyan:
> >
> >
> >
> > Yahoo almost took your path 2 years ago. After some discussion with
> > Nathan, we decided to work with community on Storm, and some of us
> > learned Clojure quickly. We are very glad that we did that.
> >
> >
> >
> > Under the assumption that Storm provides all features of JStorm, my
> > personal vote will be NO for your proposal.
> >
> > - Storm has enjoyed its current success before Nathan built a very
> > solid core in Clojure. We should not replace that core until we are
> > 100%
> > sure that alternative implementation is at least as good as our
> Clojure
> > implementation.
> > - Clojure has not prevent Storm attract contributors (currently 108).
> > We have contributors who write code in Clojure and Java. It doesn’t
> > take
> > much time for one to be able to understand Clojure code.
> > - The convergence of streaming processing and batch processing will
> > occur at higher level of abstractions. Trident API, for example, is
> > very
> > much similar to batch API such as Pig or Cascading.
> > - Including Jstorm under Storm will only create confusion to our user
> > community at this stage.
> >
> >
> >
> > Why don’t you have discussion with 徐明明 to figure out an alternative
> > path for JStorm?
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Andy
> >
> >
> >
> > *From: *"封仲淹 (纪君祥)" <zhongyan.f...@alibaba-inc.com>
> > *Reply-To: *"封仲淹(纪君祥)" <zhongyan.f...@alibaba-inc.com>
> > *Date: *Wednesday, October 22, 2014 at 11:37 PM
> > *To: *Andy Feng <af...@yahoo-inc.com>, "nat...@nathanmarz.com" <
> > nat...@nathanmarz.com>, "徐明明(护城)" <mingming.x...@alibaba-inc.com>, "
> > ja...@cvk.ca" <ja...@cvk.ca>, "f...@infochimps.com"
> > <f...@infochimps.com>, "david...@microsoft.com"
> > <david...@microsoft.com>, "ptgo...@gmail.com" < ptgo...@gmail.com>, Doug
> > Cutting <cutt...@apache.org>, "
> > tdunn...@maprtech.com" <tdunn...@maprtech.com>, "arv...@apache.org" <
> > arv...@apache.org>, "d...@hortonworks.com" <d...@hortonworks.com>, "
> > m.ben.frank...@gmail.com" <m.ben.frank...@gmail.com>, "
> > benjamin.hind...@gmail.com" <benjamin.hind...@gmail.com>
> > *Cc: *aloha-dev <aloha-...@list.alibaba-inc.com>, "
> scott.z...@vipshop.com"
> > <scott.z...@vipshop.com>, "yannia...@tencent.com"
> > <yannia...@tencent.com>, "mlc...@iflytek.com" <mlc...@iflytek.com>,
> > "z...@tencent.com" < z...@tencent.com>
> > *Subject: *答复: Could JStorm project collaborate with Storm?
> >
> >
> >
> > Andrew, Thanks for suggest.
> >
> >
> >
> > The problem is that the core of Storm is implemented by Clojure, if it
> > is java, we are glad to merge all our commit into the Strom Trunk.
> >
> > If the core of Storm is implemented by Java, I think the contributor
> > of Storm would be double.
> >
> >
> >
> >
> >
> > May I put forward 2 proposal:
> >
> > 1. Could Jstorm be the son project of Storm?
> >
> > 2. Task a long time to change Storm-core as Jstorm-core.
> >
> > a) If the core of Storm is implemented by Java, I think the
> > contributor of Storm would be double. Once one user found one bug,
> > maybe he is able to fix it by himself, in this method, Storm resolve
> > problem will be double
> >
> > b) In the next one or two years, batch handling and stream handling
> > will merge into one solution, Spark/Flink are doing this job. If we
> > still use clojure, I am a little afraid that we can’t follow the steps
> > of other community and this direction.
> >
> >
> >
> >
> >
> > Thanks
> >
> > Longda
> >
> >
> >
> > *发件人:* Andrew Feng [mailto:af...@yahoo-inc.com <af...@yahoo-inc.com>]
> > *发送时间:* 2014年10月23日 13:31
> > *收件人:* 封仲淹(纪君祥); nat...@nathanmarz.com; 徐明明(护城); ja...@cvk.ca;
> > f...@infochimps.com; david...@microsoft.com; ptgo...@gmail.com;
> > cutt...@apache.org; tdunn...@maprtech.com; arv...@apache.org;
> > d...@hortonworks.com; m.ben.frank...@gmail.com;
> > benjamin.hind...@gmail.com
> > *抄送:* aloha-dev; scott.z...@vipshop.com; yannia...@tencent.com;
> > mlc...@iflytek.com; z...@tencent.com
> > *主题:* Re: Could JStorm project collaborate with Storm?
> >
> >
> >
> > Zhongyan:
> >
> >
> >
> > I would suggest you identify a collection of key features in Jstorm,
> > and then propose pull requests to Storm.
> >
> > Once these pull requests are accepted, you could then simply adopt Storm.
> >
> >
> >
> > Andy
> >
> >
> >
> > *From: *"封仲淹 (纪君祥)" <zhongyan.f...@alibaba-inc.com>
> > *Reply-To: *"封仲淹(纪君祥)" <zhongyan.f...@alibaba-inc.com>
> > *Date: *Wednesday, October 22, 2014 at 8:23 PM
> > *To: *"nat...@nathanmarz.com" <nat...@nathanmarz.com>, "徐明明(护城)" <
> > mingming.x...@alibaba-inc.com>, "ja...@cvk.ca" <ja...@cvk.ca>, Andy
> > Feng < af...@yahoo-inc.com>, "f...@infochimps.com" <f...@infochimps.com
> >,
> > "
> > david...@microsoft.com" <david...@microsoft.com>, "ptgo...@gmail.com"
> > < ptgo...@gmail.com>, Doug Cutting <cutt...@apache.org>, "
> > tdunn...@maprtech.com" <tdunn...@maprtech.com>, "arv...@apache.org" <
> > arv...@apache.org>, "d...@hortonworks.com" <d...@hortonworks.com>, "
> > m.ben.frank...@gmail.com" <m.ben.frank...@gmail.com>, "
> > benjamin.hind...@gmail.com" <benjamin.hind...@gmail.com>
> > *Cc: *aloha-dev <aloha-...@list.alibaba-inc.com>, "
> scott.z...@vipshop.com"
> > <scott.z...@vipshop.com>, "yannia...@tencent.com"
> > <yannia...@tencent.com>, "mlc...@iflytek.com" <mlc...@iflytek.com>,
> > "z...@tencent.com" < z...@tencent.com>
> > *Subject: *Could JStorm project collaborate with Storm?
> >
> >
> >
> > Hi all,
> >
> >
> >
> > Firstly, congratulate Storm becoming the top project of Apache. Since
> > the first day of Storm opened source, it has become one of hottest
> > technologies in the BigData field and brought a new perspective to the
> > world, which has helped many companies resolve hundreds of requirements.
> >
> >
> >
> > Since 2012/4, we have forked a project named "JStorm". At the
> > beginning time, we just wanted to implement Storm with Java. As time
> > goes on, more and more features have been added to JStorm. In 2013/9,
> > JStorm changed to be an open source project; from then on, more and
> > more people and companies join this project.
> >
> >
> >
> > As JStorm is derived from Storm, and born in a great company named
> > Alibaba, which just successfully listed on the New York Stock Exchange
> > on Sept. 19, is a very open company. We also want to share our
> > technology with more people, and favor rapid evolution of streaming
> > technology.
> >
> >
> >
> > *So, could JStorm project collaborate with Storm? *
> >
> >
> >
> > A little introduction for JStorm as follows:
> >
> > JStorm is a distributed and fault-tolerant real-time computation system.
> > Inspired by Apache Storm, JStorm has been completely rewritten in Java
> > and provides more enhanced features. JStorm has been widely used in
> > many enterprise environments and proved robust and stable.
> >
> >
> >
> > Begun from 2012/2, we have released 11 versions till 2014/09/28. More
> > than
> > 10 companies are using it, including several leading companies such as
> > Alibaba, Tencent, Baidu, Huawei, ZTE in China.
> >
> >
> >
> > Please refer to https://github.com/alibaba/jstorm for more details
> >
> >
> >
> >
> >
> > Regards
> >
> > Longda
> >
> >
> >
> >
> >
>
>
> --
> Name : 임 정택
> Blog : http://www.heartsavior.net / http://dev.heartsavior.net
> Twitter : http://twitter.com/heartsavior
> LinkedIn : http://www.linkedin.com/in/heartsavior
>
