Taylor and others I was hoping to get started filing JIRA and planning on how we are going to do the java migration + JStorm merger. Is anyone else starting to do this? If not would anyone object to me starting on it? - Bobby
On Thursday, November 12, 2015 12:04 PM, P. Taylor Goetz <ptgo...@gmail.com> wrote: Thanks for putting this together Basti, that comparison helps a lot. And thanks Bobby for converting it into markdown. I was going to just attach the spreadsheet to JIRA, but markdown is a much better solution. -Taylor > On Nov 12, 2015, at 12:03 PM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote: > > I translated the excel spreadsheet into a markdown file and put up a pull > request for it. > https://github.com/apache/storm/pull/877 > I did a few edits to it to make it work with Markdown, and to add in a few of > my own comments. I also put in a field for JIRAs to be able to track the > migration. > Overall I think your evaluation was very good. We have a fair amount of work > ahead of us to decide what version of various features we want to go forward > with. > - Bobby > > > On Thursday, November 12, 2015 9:37 AM, 刘键(Basti Liu) ><basti...@alibaba-inc.com> wrote: > > > Hi Bobby & Jungtaek, > > Thanks for your replay. > I totally agree that compatibility is the most important thing. Actually, > JStorm has been compatible with the user API of Storm. > As you mentioned below, we indeed still have some features different between > Storm and JStorm. I have tried to list them (minor update or improvements are > not included). > Please refer to attachment for details. If any missing, please help to point > out. (The current working features are probably missing here.) > Just have a look at these differences. For the missing features in JStorm, I > did not see any obstacle which will block the merge to JStorm. > For the features which has different solution between Storm and JStorm, we > can evaluate the solution one by one to decision which one is appropriate. > After the finalization of evaluation, I think JStorm team can take the > merging job and publish a stable release in 2 months. > But anyway, the detailed implementation for these features with different > solution is transparent to user. So, from user's point of view, there is not > any compatibility problem. > > Besides compatibility, by our experience, stability is also important and is > not an easy job. 4 people in JStorm team took almost one year to finish the > porting from "clojure core" > to "java core", and to make it stable. Of course, we have many devs in > community to make the porting job faster. But it still needs a long time to > run many online complex topologys to find bugs and fix them. So, that is the > reason why I proposed to do merging and build on a stable "java core". > > -----Original Message----- > From: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID] > Sent: Wednesday, November 11, 2015 10:51 PM > To: dev@storm.apache.org > Subject: Re: [DISCUSS] Plan for Merging JStorm Code > > +1 for doing a 1.0 release based off of the clojure 0.11.x code. Migrating > the APIs to org.apache.storm is a big non-backwards compatible move, and a > major version bump to 2.x seems like a good move there. > +1 for the release plan > > I would like the move for user facing APIs to org.apache to be one of the > last things we do. Translating clojure code into java and moving it to > org.apache I am not too concerned about. > > Basti, > We have two code bases that have diverged significantly from one another in > terms of functionality. The storm code now or soon will have A Heartbeat > Server, Nimbus HA (Different Implementation), Resource Aware Scheduling, a > distributed cache like API, log searching, security, massive performance > improvements, shaded almost all of our dependencies, a REST API for > programtically accessing everything on the UI, and I am sure I am missing a > few other things. JStorm also has many changes including cgroup isolation, > restructured zookeeper layout, classpath isolation, and more too. > No matter what we do it will be a large effort to port changes from one code > base to another, and from clojure to java. I proposed this initially because > it can be broken up into incremental changes. It may take a little longer, > but we will always have a working codebase that is testable and compatible > with the current storm release, at least until we move the user facing APIs > to be under org.apache. This lets the community continue to build and test > the master branch and report problems that they find, which is incredibly > valuable. I personally don't think it will be much easier, especially if we > are intent on always maintaining compatibility with storm. - Bobby > > > On Wednesday, November 11, 2015 5:42 AM, 刘键(Basti Liu) ><basti...@alibaba-inc.com> wrote: > > > Hi Taylor, > > > > Thanks for the merge plan. I have a question about “Phase 2.2”. > > Do you mean community plan to create a fresh new “java core” based on current > “clojure core” firstly, and then migrate the features from JStorm? > > If so, it confused me. It is really a huge job which might require a long > developing time to make it stable, while JStorm is already a stable version. > > The release planned to be release after Nov 11th has already run online > stably several month in Alibaba. > > Besides this, there are many valuable internal requirements in Alibaba, the > fast evolution of JStorm is forseeable in next few months. > > If the “java core” is totally fresh new, it might bring many problems for the > coming merge. > > So, from the point of this view, I think it is much better and easier to > migrate the features of “clojure core” basing on JStorm for the “java core”. > > Please correct me, if any misunderstanding. > > > > Regards > > Basti > > > > 发件人: P. Taylor Goetz [mailto:ptgo...@gmail.com] > 发送时间: 2015年11月11日 5:32 > 收件人: dev@storm.apache.org > 主题: [DISCUSS] Plan for Merging JStorm Code > > > > Based on a number of discussions regarding merging the JStorm code, I’ve > tried to distill the ideas presented and inserted some of my own. The result > is below. > > > > I’ve divided the plan into three phases, though they are not necessarily > sequential — obviously some tasks can take place in parallel. > > > > None of this is set in stone, just presented for discussion. Any and all > comments are welcome. > > > > ------- > > > > Phase 1 - Plan for 0.11.x Release > > 1. Determine feature set for 0.11.x and publish to wiki [1]. > > 2. Announce feature-freeze for 0.11.x > > 3. Create 0.11.x branch from master (Phase 2.4 can begin.) > > 4. Release 0.11.0 (or whatever version # we want to use) > > 5. Bug fixes and subsequent releases from 0.11.x-branch > > > > > > Phase 2 - Prepare for Merge ("master" and "jstorm-import" branches) > > 1. Determine/document unique features in JStorm (e.g. classpath isolation, > cgroups, etc.) and create JIRA for migrating the feature. > > 2. Create JIRA for migrating each clojure component (or logical group of > components) to Java. Assumes tests will be ported as well. > > 3. Discuss/establish style guide for Java coding conventions. Consider using > Oracle’s or Google’s Java conventions as a base — they are both pretty solid. > > 4. align package names (e.g backtype.storm --> org.apache.storm / > com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin) > > > > > > Phase 3 - Migrate Clojure --> Java > > 1. Port code/tests to Java, leveraging existing JStorm code wherever possible > (core functionality only, features distinct to JStorm migrated separately). > > 2. Port JStorm-specific features. > > 3. Begin releasing preview/beta versions. > > 4. Code cleanup (across the board) and refactoring using established coding > conventions, and leveraging PMD/Checkstyle reports for reference. (Note: good > oportunity for new contributors.) > > 5. Release 0.12.0 (or whatever version # we want to use) and lift feature > freeze. > > > > > > Notes: > > We should consider bumping up to version 1.0 sometime soon and then switching > to semantic versioning [3] from then on. > > > > > > With the exception of package name alignment, the "jstorm-import" branch will > largely be read-only throughout the process. > > > > During migration, it's probably easiest to operate with two local clones of > the Apache Storm repo: one for working (i.e. checked out to working branch) > and one for reference/copying (i.e. checked out to "jstorm-import"). > > > > Feature-freeze probably only needs to be enforced against core functionality. > Components under "external" can likely be exempt, but we should figure out a > process for accepting and releasing new features during the migration. > > > > Performance testing should be continuous throughout the process. Since we > don't really have ASF infrastructure for performance testing, we will need a > volunteer(s) to host and run the performance tests. Performance test results > can be posted to the wiki [2]. It would probably be a good idea to establish > a baseline with the 0.10.0 release. > > > > I’ve attached an analysis document Sean Zhong put together a while back to > the JStorm merge JIRA [4]. The analysis was against the 0.9.3 release but is > still relevant and has a lot of good information. > > > > > > [1] > https://cwiki.apache.org/confluence/display/STORM/Release+0.11.0+Feature+Set > > [2] https://cwiki.apache.org/confluence/display/STORM/Storm+Home > > [3] http://semver.org > > [4] https://issues.apache.org/jira/browse/STORM-717 > > > > > > -Taylor > > > > > >