Taylor and others I was hoping to get started filing JIRA and planning on how
we are going to do the java migration + JStorm merger. Is anyone else starting
to do this? If not would anyone object to me starting on it? - Bobby
On Thursday, November 12, 2015 12:04 PM, P. Taylor Goetz
<[email protected]> wrote:
Thanks for putting this together Basti, that comparison helps a lot.
And thanks Bobby for converting it into markdown. I was going to just attach
the spreadsheet to JIRA, but markdown is a much better solution.
-Taylor
> On Nov 12, 2015, at 12:03 PM, Bobby Evans <[email protected]> wrote:
>
> I translated the excel spreadsheet into a markdown file and put up a pull
> request for it.
> https://github.com/apache/storm/pull/877
> I did a few edits to it to make it work with Markdown, and to add in a few of
> my own comments. I also put in a field for JIRAs to be able to track the
> migration.
> Overall I think your evaluation was very good. We have a fair amount of work
> ahead of us to decide what version of various features we want to go forward
> with.
> - Bobby
>
>
> On Thursday, November 12, 2015 9:37 AM, 刘键(Basti Liu)
><[email protected]> wrote:
>
>
> Hi Bobby & Jungtaek,
>
> Thanks for your replay.
> I totally agree that compatibility is the most important thing. Actually,
> JStorm has been compatible with the user API of Storm.
> As you mentioned below, we indeed still have some features different between
> Storm and JStorm. I have tried to list them (minor update or improvements are
> not included).
> Please refer to attachment for details. If any missing, please help to point
> out. (The current working features are probably missing here.)
> Just have a look at these differences. For the missing features in JStorm, I
> did not see any obstacle which will block the merge to JStorm.
> For the features which has different solution between Storm and JStorm, we
> can evaluate the solution one by one to decision which one is appropriate.
> After the finalization of evaluation, I think JStorm team can take the
> merging job and publish a stable release in 2 months.
> But anyway, the detailed implementation for these features with different
> solution is transparent to user. So, from user's point of view, there is not
> any compatibility problem.
>
> Besides compatibility, by our experience, stability is also important and is
> not an easy job. 4 people in JStorm team took almost one year to finish the
> porting from "clojure core"
> to "java core", and to make it stable. Of course, we have many devs in
> community to make the porting job faster. But it still needs a long time to
> run many online complex topologys to find bugs and fix them. So, that is the
> reason why I proposed to do merging and build on a stable "java core".
>
> -----Original Message-----
> From: Bobby Evans [mailto:[email protected]]
> Sent: Wednesday, November 11, 2015 10:51 PM
> To: [email protected]
> Subject: Re: [DISCUSS] Plan for Merging JStorm Code
>
> +1 for doing a 1.0 release based off of the clojure 0.11.x code. Migrating
> the APIs to org.apache.storm is a big non-backwards compatible move, and a
> major version bump to 2.x seems like a good move there.
> +1 for the release plan
>
> I would like the move for user facing APIs to org.apache to be one of the
> last things we do. Translating clojure code into java and moving it to
> org.apache I am not too concerned about.
>
> Basti,
> We have two code bases that have diverged significantly from one another in
> terms of functionality. The storm code now or soon will have A Heartbeat
> Server, Nimbus HA (Different Implementation), Resource Aware Scheduling, a
> distributed cache like API, log searching, security, massive performance
> improvements, shaded almost all of our dependencies, a REST API for
> programtically accessing everything on the UI, and I am sure I am missing a
> few other things. JStorm also has many changes including cgroup isolation,
> restructured zookeeper layout, classpath isolation, and more too.
> No matter what we do it will be a large effort to port changes from one code
> base to another, and from clojure to java. I proposed this initially because
> it can be broken up into incremental changes. It may take a little longer,
> but we will always have a working codebase that is testable and compatible
> with the current storm release, at least until we move the user facing APIs
> to be under org.apache. This lets the community continue to build and test
> the master branch and report problems that they find, which is incredibly
> valuable. I personally don't think it will be much easier, especially if we
> are intent on always maintaining compatibility with storm. - Bobby
>
>
> On Wednesday, November 11, 2015 5:42 AM, 刘键(Basti Liu)
><[email protected]> wrote:
>
>
> Hi Taylor,
>
>
>
> Thanks for the merge plan. I have a question about “Phase 2.2”.
>
> Do you mean community plan to create a fresh new “java core” based on current
> “clojure core” firstly, and then migrate the features from JStorm?
>
> If so, it confused me. It is really a huge job which might require a long
> developing time to make it stable, while JStorm is already a stable version.
>
> The release planned to be release after Nov 11th has already run online
> stably several month in Alibaba.
>
> Besides this, there are many valuable internal requirements in Alibaba, the
> fast evolution of JStorm is forseeable in next few months.
>
> If the “java core” is totally fresh new, it might bring many problems for the
> coming merge.
>
> So, from the point of this view, I think it is much better and easier to
> migrate the features of “clojure core” basing on JStorm for the “java core”.
>
> Please correct me, if any misunderstanding.
>
>
>
> Regards
>
> Basti
>
>
>
> 发件人: P. Taylor Goetz [mailto:[email protected]]
> 发送时间: 2015年11月11日 5:32
> 收件人: [email protected]
> 主题: [DISCUSS] Plan for Merging JStorm Code
>
>
>
> Based on a number of discussions regarding merging the JStorm code, I’ve
> tried to distill the ideas presented and inserted some of my own. The result
> is below.
>
>
>
> I’ve divided the plan into three phases, though they are not necessarily
> sequential — obviously some tasks can take place in parallel.
>
>
>
> None of this is set in stone, just presented for discussion. Any and all
> comments are welcome.
>
>
>
> -------
>
>
>
> Phase 1 - Plan for 0.11.x Release
>
> 1. Determine feature set for 0.11.x and publish to wiki [1].
>
> 2. Announce feature-freeze for 0.11.x
>
> 3. Create 0.11.x branch from master (Phase 2.4 can begin.)
>
> 4. Release 0.11.0 (or whatever version # we want to use)
>
> 5. Bug fixes and subsequent releases from 0.11.x-branch
>
>
>
>
>
> Phase 2 - Prepare for Merge ("master" and "jstorm-import" branches)
>
> 1. Determine/document unique features in JStorm (e.g. classpath isolation,
> cgroups, etc.) and create JIRA for migrating the feature.
>
> 2. Create JIRA for migrating each clojure component (or logical group of
> components) to Java. Assumes tests will be ported as well.
>
> 3. Discuss/establish style guide for Java coding conventions. Consider using
> Oracle’s or Google’s Java conventions as a base — they are both pretty solid.
>
> 4. align package names (e.g backtype.storm --> org.apache.storm /
> com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin)
>
>
>
>
>
> Phase 3 - Migrate Clojure --> Java
>
> 1. Port code/tests to Java, leveraging existing JStorm code wherever possible
> (core functionality only, features distinct to JStorm migrated separately).
>
> 2. Port JStorm-specific features.
>
> 3. Begin releasing preview/beta versions.
>
> 4. Code cleanup (across the board) and refactoring using established coding
> conventions, and leveraging PMD/Checkstyle reports for reference. (Note: good
> oportunity for new contributors.)
>
> 5. Release 0.12.0 (or whatever version # we want to use) and lift feature
> freeze.
>
>
>
>
>
> Notes:
>
> We should consider bumping up to version 1.0 sometime soon and then switching
> to semantic versioning [3] from then on.
>
>
>
>
>
> With the exception of package name alignment, the "jstorm-import" branch will
> largely be read-only throughout the process.
>
>
>
> During migration, it's probably easiest to operate with two local clones of
> the Apache Storm repo: one for working (i.e. checked out to working branch)
> and one for reference/copying (i.e. checked out to "jstorm-import").
>
>
>
> Feature-freeze probably only needs to be enforced against core functionality.
> Components under "external" can likely be exempt, but we should figure out a
> process for accepting and releasing new features during the migration.
>
>
>
> Performance testing should be continuous throughout the process. Since we
> don't really have ASF infrastructure for performance testing, we will need a
> volunteer(s) to host and run the performance tests. Performance test results
> can be posted to the wiki [2]. It would probably be a good idea to establish
> a baseline with the 0.10.0 release.
>
>
>
> I’ve attached an analysis document Sean Zhong put together a while back to
> the JStorm merge JIRA [4]. The analysis was against the 0.9.3 release but is
> still relevant and has a lot of good information.
>
>
>
>
>
> [1]
> https://cwiki.apache.org/confluence/display/STORM/Release+0.11.0+Feature+Set
>
> [2] https://cwiki.apache.org/confluence/display/STORM/Storm+Home
>
> [3] http://semver.org
>
> [4] https://issues.apache.org/jira/browse/STORM-717
>
>
>
>
>
> -Taylor
>
>
>
>
>
>
