Sorry for no clarify why we raise the proposal replacing clojre-core with java-core.
The root cause why we start Jstorm is that the speed of Storm evolution is too slow for us. The Alibaba/Tencent’s application require our team/community moving faster and faster. If you take a deep look of Jstorm history, your will found we will release one version every month. Every version of Jstorm will provide new feature comparing last version, every new feature isn’t a simple job, we can also do it with Clojure(Clojure isn’t a hard thing for us), but it will take triple time to finish it comparing with implement it with java. If we replace Clojure core with java core, I definite think we can move much faster than before. Alibaba and Tencent are one of the most biggest IT companies in the world, thousands of applications run on Jstorm. The fertilizer will drive Jstorm grow more powerful and stronger. In our long term plan, we can do more things than right now, no only just handle online streaming tuples with Java or Clojure, maybe SQL, maybe new Trident, we want to build the ecosystem like Spark. If we plan to do switch core, we will fork a code branch under Storm, Jstorm-dev team will be <app:ds:responsible> responsible <app:ds:for> for merge all storm’s change to jstorm-core in a long term, after enough tests on the new branch, Jstorm-dev team will merge the branch to trunk and do the switch job. The second option is we fork one project, named Jstorm(I am a little confused whether it is a normal incubator project under apache or just one subproject under storm). The Jstorm-dev team will take charge of maintain the Jstorm sub-project, merge all Storm’s changes into Jstorm time to time and evolve by himself. And wait user to select which project is more suitable for them. One year later, we check the result and judge whether we plan to merge or not. Once Storm community plan to do switch, we close the Jstorm subproject and merge into Storm. @dev, what’s your opinion? Here is a simple history of Jstorm, just list new features. 1. 0.9.0, a) Support Nimbus HA b) Redesign the scheduler, assign topology from CPU/Memory/Disk/Networker dimension, support user-define assignment c) Rewrite Netty framework, supporting batch and sync mode. 2. 0.9.1 a) Support Classloader b) Support group quota, different group has different quato 3. 0.9.2 a) Support resource isolation, import CGroups b) Support Local mode c) Redesign Logview 4. 0.9.3 a) Support run in Hadoop yarn/Aliyun Apsara, fix the bug under storm-on-yarn 5. 0.9.4 a) Rewrite netty code, support async mode and batch, improve performance much. b) Add a new transaction programming mode c) Add metrics and statics collection d) Merge Alimama branch into Jstorm, support submit topology with different parameters 6. 0.9.5.x a) Redesign scheduler arithmetic, assign topology basing worker slot, but support CGroups b) Support two Netty mode, sync/async mode 7. 0.9.6.x a) Redesign metrics/statics, support user-defined metrics and warning, web UI display new metrics b) Add management of multiple-cluster to Web UI c) Replace gson with fastjson d) Support assign topology to user-defined supervisors e) Add mechanism to wait tasks finish cleanup job. In fact, there are more than one hundred little improvement, if you take a deep insight of Jstorm and Storm. Regards Longda 发件人: P. Taylor Goetz [mailto:[email protected]] 发送时间: 2014年10月30日 4:17 收件人: [email protected] 抄送: "封仲淹(纪君祥)"; [email protected] 主题: Re: Could JStorm project collaborate with Storm? After taking some time to review the JStorm code and history of both projects, here are my thoughts: Apache Storm is largely implemented in Java (~68%) and Clojure (~18%). The disparity in LOC is primarily due to the conciseness of Clojure compared to Java. Apache Storm's Clojure code consists mainly of implementations of interfaces defined in Java. This fact makes it possible to provide an implementation in Java itself, which is exactly what the JStorm project seems to have done. JStorm is a copy of the Apache Storm codebase, with the Clojure code being replaced with a Java implementation, with some additional changes. The justification of the project centers around a distaste for Clojure, and a seeming lack of developers with Clojure skills [1]: "Storm is wonderful product, but it is implemented with Clojure. We don't like clojure. it isn't a populate language, so in Alibaba, few people can fix the bug of storm. We are the first users of Storm since it has been open source. During using, we found several problem such as zeromq, zookeeper, performance.” While I agree that to someone experienced with Java development, the first exposure to Clojure can be a bit jarring, I don’t think it is much of a barrier to entry. In my opinion, learning Clojure is a valuable investment in making oneself a more well-rounded developer. The technology ecosystem to which Storm belongs is becoming increasingly polyglot with respect to JVM languages (Java, Scala, Clojure, etc.), and developers need to adapt to that fact. As an example (perhaps not the best example), lets look at Kafka. If I didn’t like or want to learn Scala, would it make sense to write a clone in Java and attempt to keep feature parity and compatibility, or would it be better to bite the bullet and learn Scala? (Yes, I know it’s not the best example since Storm’s Clojure code is a subset of the codebase, and the way it’s designed makes it possible for alternative implementations). And in reality, someone from Alibaba had to at least learn enough Clojure to be able to replicate Storm’s Clojure implementation in java. Looking at the history of JStorm [2][3], it appears a lot of work has gone into keeping feature parity with Apache Storm by copying in new code or reimplementing it. For example a kafka spout was added about two weeks ago [4], and Yahoo’s storm-on-yarn project was also recently copied in [5]. To me, this kind of work is rather counter-productive. If the Apache Storm community feels strongly that Storm should support multiple “core” implementations (i.e. one implemented in Clojure, and one implemented in Java), then it could be accommodated by making them pluggable and rearranging the codebase a bit. That being said, the Apache Storm community has invested a significant amount of time and effort maintaining and improving Storm’s Clojure “core.” One place this is most apparent, as Bobby mentioned, is the upcoming security release. To maintain feature parity with that release a “JStorm core” would have to be updated to implement that functionality, which is not a trivial undertaking. At some point as developers we need to ask ourselves “what's the point of that work?” There are a lot more people using Storm than developing internals. We need to make sure that both groups gain value from whatever we decide, and that we don’t sacrifice any value in the process. So to answer the question in the subject line of the email thread: Could the JStorm project collaborate with Apache Storm? My answer would be a resounding “yes.” Though the JStorm community is small in comparison to that of Apache Storm, it’s clear that the JStorm contributors have a solid grasp of Storm’s internals and would be welcomed by the Storm community. Reimplementing Storm’s core in Java no doubt provided insight into areas that might be improved. That being said, replacing Storm’s Clojure-based core is a decision that should not be taken lightly, and would require the support of the Apache Storm PMC and entire community. A “dual-core” setup could be an option, but again could become a maintenance nightmare with little value added for end users. I’m open to discussing options further, and welcome input from other community members. In short, I would rather see us work together than apart. -Taylor [1] https://github.com/alibaba/jstorm/commit/903ca941023dd37a54d9574e2a554883111f0b4a [2] https://github.com/alibaba/jstorm/commits/master [3] https://github.com/alibaba/jstorm/blob/master/history.md [4] https://github.com/alibaba/jstorm/tree/master/jstorm-utility/jstorm-kafka [5] https://github.com/alibaba/jstorm/tree/master/jstorm-on-yarn On Oct 29, 2014, at 12:14 PM, Bobby Evans <[email protected]> wrote: Sorry I am coming late to the conversation on this. Looking at the code for jstorm I am willing to entertain the idea of merging the two projects. I understand that by using java we can make the project more accessible to a larger number of developers, and that does seem to fit with the apache model of community over code. But I don't really want change just for change's sake. Look at kill_topology in both java:https://github.com/alibaba/jstorm/blob/master/jstorm-client/src/main/java/backtype/storm/command/kill_topology.java and in clojure:https://github.com/apache/storm/blob/master/storm-core/src/clj/backtype/storm/command/kill_topology.clj The java version is 61 lines of code and is not actually compatible with the clojure version because it just counts argument for the wait parameter instead of using a -w. The clojure version is 29, but the 15 of which are the apache header, so it is closer to 14 lines. Granted, to someone who does not know you have to work a bit to read it, but not that much. My concerns would be two fold. First is the effort involved in maintaining two separate code bases, sub project or otherwise, is a lot. I don't want to do it, and I assume you don't want to either, that is why you are talking about merging. We would need to decide which code to keep and which code to drop, and in all likelihood some of it will still be clojure. I also would not want to restrict people from using clojure for new work if they want to, just like we don't restrict java usage for new work. Second is true feature parity. Storm is a moving target, I would not want to replace any code, even if it is faster, without verifying that it has feature parity, or it is a conscious decision that the missing features are not needed. Especially around the security branch, which is scheduled for the 0.10.0 release, but is not a part of jstorm at all. - Bobby From: 封仲淹(纪君祥) <[email protected]> To: [email protected]; [email protected] Cc: [email protected] Sent: Wednesday, October 29, 2014 4:27 AM Subject: 答复: Could JStorm project collaborate with Storm? Thanks for your suggestion. @dev, Does anyone has proposal? please feel free to raise. Thanks Longda -----邮件原件----- 发件人: Ted Dunning [mailto:[email protected]] 发送时间: 2014年10月25日 7:27 收件人: [email protected] 主题: Re: Could JStorm project collaborate with Storm? I just moved all of the non-Storm-dev addresses on this thread to bcc. If anybody is not on the Storm dev list and would like to continue to follow this conversation, you should subscribe to the storm dev list. We should not be spamming everybody else. There is a standard Apache process to deal with this. JStorm can become an incubator project. Once JStorm demonstrates the ability to function as an Apache project, it can graduate either to a top-level project or to become part of a project like Storm. [Longda]This will be a feasible solution. From the Apache side, it had better Jstorm can collaborate with Storm. A second mechanism is that if the Storm group is interested, they can directly adopt the code as it is, possibly asking one or more of the contributors to become committers to help support this new code. That is a much harder thing to do than incubation since it is unusual for a project to be enthusiastic about supporting a large new codebase. [Longda]I agree. It is a hard job. On Thu, Oct 23, 2014 at 8:19 PM, 刘键(夏蒅) <[email protected]> wrote: Hi Andy, Thanks for your quick response. What you concern is reasonable and understandable. But maybe there is some misunderstanding on our proposal. According to the current development situation in China, java core is indeed to bring more contributors here. That is reason why we begin to think if it is also a way for us to provide our contribution to Storm project. It is true that it is a risk to switch the core in a short period. Actually, if possible, maybe we can consider to take JStorm as a subproject of Storm(A branch or some other ways…). We will be responsible for the maintain of it and try to propagate the features from Storm to JStorm, and continue to develop our features and improvements on this. You guys can try the new core and take a long time to estimate if it is worth doing the switch in the future or taking JStorm as official optional core of Storm for user. Regards Basti *发件人:* Andrew Feng [mailto:[email protected] <[email protected]>] *发送时间:* 2014年10月24日 0:24 *收件人:* 封仲淹(纪君祥); [email protected]; 徐明明(护城); [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected] *抄送:* aloha-dev; [email protected]; [email protected]; [email protected]; [email protected] *主题:* Re: 答复: Could JStorm project collaborate with Storm? Zhongyan: Yahoo almost took your path 2 years ago. After some discussion with Nathan, we decided to work with community on Storm, and some of us learned Clojure quickly. We are very glad that we did that. Under the assumption that Storm provides all features of JStorm, my personal vote will be NO for your proposal. - Storm has enjoyed its current success before Nathan built a very solid core in Clojure. We should not replace that core until we are 100% sure that alternative implementation is at least as good as our Clojure implementation. - Clojure has not prevent Storm attract contributors (currently 108). We have contributors who write code in Clojure and Java. It doesn’t take much time for one to be able to understand Clojure code. - The convergence of streaming processing and batch processing will occur at higher level of abstractions. Trident API, for example, is very much similar to batch API such as Pig or Cascading. - Including Jstorm under Storm will only create confusion to our user community at this stage. Why don’t you have discussion with 徐明明 to figure out an alternative path for JStorm? Thanks, Andy *From: *"封仲淹 (纪君祥)" <[email protected]> *Reply-To: *"封仲淹(纪君祥)" <[email protected]> *Date: *Wednesday, October 22, 2014 at 11:37 PM *To: *Andy Feng <[email protected]>, "[email protected]" < [email protected]>, "徐明明(护城)" <[email protected]>, " [email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" < [email protected]>, Doug Cutting <[email protected]>, " [email protected]" <[email protected]>, "[email protected]" < [email protected]>, "[email protected]" <[email protected]>, " [email protected]" <[email protected]>, " [email protected]" <[email protected]> *Cc: *aloha-dev <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" < [email protected]> *Subject: *答复: Could JStorm project collaborate with Storm? Andrew, Thanks for suggest. The problem is that the core of Storm is implemented by Clojure, if it is java, we are glad to merge all our commit into the Strom Trunk. If the core of Storm is implemented by Java, I think the contributor of Storm would be double. May I put forward 2 proposal: 1. Could Jstorm be the son project of Storm? 2. Task a long time to change Storm-core as Jstorm-core. a) If the core of Storm is implemented by Java, I think the contributor of Storm would be double. Once one user found one bug, maybe he is able to fix it by himself, in this method, Storm resolve problem will be double b) In the next one or two years, batch handling and stream handling will merge into one solution, Spark/Flink are doing this job. If we still use clojure, I am a little afraid that we can’t follow the steps of other community and this direction. Thanks Longda *发件人:* Andrew Feng [mailto:[email protected] <mailto:[email protected]%20%[email protected]%3e> <[email protected]>] *发送时间:* 2014年10月23日 13:31 *收件人:* 封仲淹(纪君祥); [email protected]; 徐明明(护城); [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected] *抄送:* aloha-dev; [email protected]; [email protected]; [email protected]; [email protected] *主题:* Re: Could JStorm project collaborate with Storm? Zhongyan: I would suggest you identify a collection of key features in Jstorm, and then propose pull requests to Storm. Once these pull requests are accepted, you could then simply adopt Storm. Andy *From: *"封仲淹 (纪君祥)" <[email protected]> *Reply-To: *"封仲淹(纪君祥)" <[email protected]> *Date: *Wednesday, October 22, 2014 at 8:23 PM *To: *"[email protected]" <[email protected]>, "徐明明(护城)" < [email protected]>, "[email protected]" <[email protected]>, Andy Feng < [email protected]>, "[email protected]" <[email protected]>, " [email protected]" <[email protected]>, "[email protected]" < [email protected]>, Doug Cutting <[email protected]>, " [email protected]" <[email protected]>, "[email protected]" < [email protected]>, "[email protected]" <[email protected]>, " [email protected]" <[email protected]>, " [email protected]" <[email protected]> *Cc: *aloha-dev <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" < [email protected]> *Subject: *Could JStorm project collaborate with Storm? Hi all, Firstly, congratulate Storm becoming the top project of Apache. Since the first day of Storm opened source, it has become one of hottest technologies in the BigData field and brought a new perspective to the world, which has helped many companies resolve hundreds of requirements. Since 2012/4, we have forked a project named "JStorm". At the beginning time, we just wanted to implement Storm with Java. As time goes on, more and more features have been added to JStorm. In 2013/9, JStorm changed to be an open source project; from then on, more and more people and companies join this project. As JStorm is derived from Storm, and born in a great company named Alibaba, which just successfully listed on the New York Stock Exchange on Sept. 19, is a very open company. We also want to share our technology with more people, and favor rapid evolution of streaming technology. *So, could JStorm project collaborate with Storm? * A little introduction for JStorm as follows: JStorm is a distributed and fault-tolerant real-time computation system. Inspired by Apache Storm, JStorm has been completely rewritten in Java and provides more enhanced features. JStorm has been widely used in many enterprise environments and proved robust and stable. Begun from 2012/2, we have released 11 versions till 2014/09/28. More than 10 companies are using it, including several leading companies such as Alibaba, Tencent, Baidu, Huawei, ZTE in China. Please refer to https://github.com/alibaba/jstorm for more details Regards Longda
