I translated the excel spreadsheet into a markdown file and put up a pull
request for it.
https://github.com/apache/storm/pull/877
I did a few edits to it to make it work with Markdown, and to add in a few of
my own comments. I also put in a field for JIRAs to be able to track the
migration.
Overall I think your evaluation was very good. We have a fair amount of work
ahead of us to decide what version of various features we want to go forward
with.
- Bobby
On Thursday, November 12, 2015 9:37 AM, 刘键(Basti Liu)
<[email protected]> wrote:
Hi Bobby & Jungtaek,
Thanks for your replay.
I totally agree that compatibility is the most important thing. Actually,
JStorm has been compatible with the user API of Storm.
As you mentioned below, we indeed still have some features different between
Storm and JStorm. I have tried to list them (minor update or improvements are
not included).
Please refer to attachment for details. If any missing, please help to point
out. (The current working features are probably missing here.)
Just have a look at these differences. For the missing features in JStorm, I
did not see any obstacle which will block the merge to JStorm.
For the features which has different solution between Storm and JStorm, we can
evaluate the solution one by one to decision which one is appropriate.
After the finalization of evaluation, I think JStorm team can take the merging
job and publish a stable release in 2 months.
But anyway, the detailed implementation for these features with different
solution is transparent to user. So, from user's point of view, there is not
any compatibility problem.
Besides compatibility, by our experience, stability is also important and is
not an easy job. 4 people in JStorm team took almost one year to finish the
porting from "clojure core"
to "java core", and to make it stable. Of course, we have many devs in
community to make the porting job faster. But it still needs a long time to run
many online complex topologys to find bugs and fix them. So, that is the reason
why I proposed to do merging and build on a stable "java core".
-----Original Message-----
From: Bobby Evans [mailto:[email protected]]
Sent: Wednesday, November 11, 2015 10:51 PM
To: [email protected]
Subject: Re: [DISCUSS] Plan for Merging JStorm Code
+1 for doing a 1.0 release based off of the clojure 0.11.x code. Migrating the
APIs to org.apache.storm is a big non-backwards compatible move, and a major
version bump to 2.x seems like a good move there.
+1 for the release plan
I would like the move for user facing APIs to org.apache to be one of the last
things we do. Translating clojure code into java and moving it to org.apache I
am not too concerned about.
Basti,
We have two code bases that have diverged significantly from one another in
terms of functionality. The storm code now or soon will have A Heartbeat
Server, Nimbus HA (Different Implementation), Resource Aware Scheduling, a
distributed cache like API, log searching, security, massive performance
improvements, shaded almost all of our dependencies, a REST API for
programtically accessing everything on the UI, and I am sure I am missing a few
other things. JStorm also has many changes including cgroup isolation,
restructured zookeeper layout, classpath isolation, and more too.
No matter what we do it will be a large effort to port changes from one code
base to another, and from clojure to java. I proposed this initially because
it can be broken up into incremental changes. It may take a little longer, but
we will always have a working codebase that is testable and compatible with the
current storm release, at least until we move the user facing APIs to be under
org.apache. This lets the community continue to build and test the master
branch and report problems that they find, which is incredibly valuable. I
personally don't think it will be much easier, especially if we are intent on
always maintaining compatibility with storm. - Bobby
On Wednesday, November 11, 2015 5:42 AM, 刘键(Basti Liu)
<[email protected]> wrote:
Hi Taylor,
Thanks for the merge plan. I have a question about “Phase 2.2”.
Do you mean community plan to create a fresh new “java core” based on current
“clojure core” firstly, and then migrate the features from JStorm?
If so, it confused me. It is really a huge job which might require a long
developing time to make it stable, while JStorm is already a stable version.
The release planned to be release after Nov 11th has already run online stably
several month in Alibaba.
Besides this, there are many valuable internal requirements in Alibaba, the
fast evolution of JStorm is forseeable in next few months.
If the “java core” is totally fresh new, it might bring many problems for the
coming merge.
So, from the point of this view, I think it is much better and easier to
migrate the features of “clojure core” basing on JStorm for the “java core”.
Please correct me, if any misunderstanding.
Regards
Basti
发件人: P. Taylor Goetz [mailto:[email protected]]
发送时间: 2015年11月11日 5:32
收件人: [email protected]
主题: [DISCUSS] Plan for Merging JStorm Code
Based on a number of discussions regarding merging the JStorm code, I’ve tried
to distill the ideas presented and inserted some of my own. The result is below.
I’ve divided the plan into three phases, though they are not necessarily
sequential — obviously some tasks can take place in parallel.
None of this is set in stone, just presented for discussion. Any and all
comments are welcome.
-------
Phase 1 - Plan for 0.11.x Release
1. Determine feature set for 0.11.x and publish to wiki [1].
2. Announce feature-freeze for 0.11.x
3. Create 0.11.x branch from master (Phase 2.4 can begin.)
4. Release 0.11.0 (or whatever version # we want to use)
5. Bug fixes and subsequent releases from 0.11.x-branch
Phase 2 - Prepare for Merge ("master" and "jstorm-import" branches)
1. Determine/document unique features in JStorm (e.g. classpath isolation,
cgroups, etc.) and create JIRA for migrating the feature.
2. Create JIRA for migrating each clojure component (or logical group of
components) to Java. Assumes tests will be ported as well.
3. Discuss/establish style guide for Java coding conventions. Consider using
Oracle’s or Google’s Java conventions as a base — they are both pretty solid.
4. align package names (e.g backtype.storm --> org.apache.storm /
com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin)
Phase 3 - Migrate Clojure --> Java
1. Port code/tests to Java, leveraging existing JStorm code wherever possible
(core functionality only, features distinct to JStorm migrated separately).
2. Port JStorm-specific features.
3. Begin releasing preview/beta versions.
4. Code cleanup (across the board) and refactoring using established coding
conventions, and leveraging PMD/Checkstyle reports for reference. (Note: good
oportunity for new contributors.)
5. Release 0.12.0 (or whatever version # we want to use) and lift feature
freeze.
Notes:
We should consider bumping up to version 1.0 sometime soon and then switching
to semantic versioning [3] from then on.
With the exception of package name alignment, the "jstorm-import" branch will
largely be read-only throughout the process.
During migration, it's probably easiest to operate with two local clones of the
Apache Storm repo: one for working (i.e. checked out to working branch) and one
for reference/copying (i.e. checked out to "jstorm-import").
Feature-freeze probably only needs to be enforced against core functionality.
Components under "external" can likely be exempt, but we should figure out a
process for accepting and releasing new features during the migration.
Performance testing should be continuous throughout the process. Since we don't
really have ASF infrastructure for performance testing, we will need a
volunteer(s) to host and run the performance tests. Performance test results
can be posted to the wiki [2]. It would probably be a good idea to establish a
baseline with the 0.10.0 release.
I’ve attached an analysis document Sean Zhong put together a while back to the
JStorm merge JIRA [4]. The analysis was against the 0.9.3 release but is still
relevant and has a lot of good information.
[1] https://cwiki.apache.org/confluence/display/STORM/Release+0.11.0+Feature+Set
[2] https://cwiki.apache.org/confluence/display/STORM/Storm+Home
[3] http://semver.org
[4] https://issues.apache.org/jira/browse/STORM-717
-Taylor
