Hi Taylor, I'd like to help too, could you add me in? my id is: Cody On Sat, Nov 21, 2015 at 11:51 AM, 刘键(Basti Liu) <basti...@alibaba-inc.com> wrote:
> Hi Taylor， > > Sorry for the late response. > I'd like to help on this. Could you please help to give me the permission? > Thanks. > UserName: basti.lj > > Regards > Basti > > -----Original Message----- > From: P. Taylor Goetz [mailto:ptgo...@gmail.com] > Sent: Thursday, November 19, 2015 6:24 AM > To: dev@storm.apache.org; Bobby Evans > Subject: Re: [DISCUSS] Plan for Merging JStorm Code > > All I have at this point is a placeholder wiki entry [1], and a lot of > local notes that likely would only make sense to me. > > Let me know your wiki username and I’ll give you permissions. The same > goes for anyone else who wants to help. > > -Taylor > > [1] > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61328109 > > > On Nov 18, 2015, at 2:08 PM, Bobby Evans <ev...@yahoo-inc.com.INVALID> > wrote: > > > > Taylor and others I was hoping to get started filing JIRA and planning > > on how we are going to do the java migration + JStorm merger. Is > > anyone else starting to do this? If not would anyone object to me > > starting on it? - Bobby > > > > > > On Thursday, November 12, 2015 12:04 PM, P. Taylor Goetz < > ptgo...@gmail.com> wrote: > > > > > > Thanks for putting this together Basti, that comparison helps a lot. > > > > And thanks Bobby for converting it into markdown. I was going to just > attach the spreadsheet to JIRA, but markdown is a much better solution. > > > > -Taylor > > > >> On Nov 12, 2015, at 12:03 PM, Bobby Evans <ev...@yahoo-inc.com.INVALID> > wrote: > >> > >> I translated the excel spreadsheet into a markdown file and put up a > pull request for it. > >> https://github.com/apache/storm/pull/877 > >> I did a few edits to it to make it work with Markdown, and to add in a > few of my own comments. I also put in a field for JIRAs to be able to > track the migration. > >> Overall I think your evaluation was very good. We have a fair amount > of work ahead of us to decide what version of various features we want to > go forward with. > >> - Bobby > >> > >> > >> On Thursday, November 12, 2015 9:37 AM, 刘键(Basti Liu) < > basti...@alibaba-inc.com> wrote: > >> > >> > >> Hi Bobby & Jungtaek, > >> > >> Thanks for your replay. > >> I totally agree that compatibility is the most important thing. > Actually, JStorm has been compatible with the user API of Storm. > >> As you mentioned below, we indeed still have some features different > between Storm and JStorm. I have tried to list them (minor update or > improvements are not included). > >> Please refer to attachment for details. If any missing, please help > >> to point out. (The current working features are probably missing here.) > Just have a look at these differences. For the missing features in JStorm, > I did not see any obstacle which will block the merge to JStorm. > >> For the features which has different solution between Storm and JStorm, > we can evaluate the solution one by one to decision which one is > appropriate. > >> After the finalization of evaluation, I think JStorm team can take the > merging job and publish a stable release in 2 months. > >> But anyway, the detailed implementation for these features with > different solution is transparent to user. So, from user's point of view, > there is not any compatibility problem. > >> > >> Besides compatibility, by our experience, stability is also important > and is not an easy job. 4 people in JStorm team took almost one year to > finish the porting from "clojure core" > >> to "java core", and to make it stable. Of course, we have many devs in > community to make the porting job faster. But it still needs a long time to > run many online complex topologys to find bugs and fix them. So, that is > the reason why I proposed to do merging and build on a stable "java core". > >> > >> -----Original Message----- > >> From: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID] > >> Sent: Wednesday, November 11, 2015 10:51 PM > >> To: dev@storm.apache.org > >> Subject: Re: [DISCUSS] Plan for Merging JStorm Code > >> > >> +1 for doing a 1.0 release based off of the clojure 0.11.x code. > Migrating the APIs to org.apache.storm is a big non-backwards compatible > move, and a major version bump to 2.x seems like a good move there. > >> +1 for the release plan > >> > >> I would like the move for user facing APIs to org.apache to be one of > the last things we do. Translating clojure code into java and moving it to > org.apache I am not too concerned about. > >> > >> Basti, > >> We have two code bases that have diverged significantly from one > another in terms of functionality. The storm code now or soon will have A > Heartbeat Server, Nimbus HA (Different Implementation), Resource Aware > Scheduling, a distributed cache like API, log searching, security, massive > performance improvements, shaded almost all of our dependencies, a REST API > for programtically accessing everything on the UI, and I am sure I am > missing a few other things. JStorm also has many changes including cgroup > isolation, restructured zookeeper layout, classpath isolation, and more too. > >> No matter what we do it will be a large effort to port changes from > >> one code base to another, and from clojure to java. I proposed this > >> initially because it can be broken up into incremental changes. It > >> may take a little longer, but we will always have a working codebase > >> that is testable and compatible with the current storm release, at > >> least until we move the user facing APIs to be under org.apache. > >> This lets the community continue to build and test the master branch > >> and report problems that they find, which is incredibly valuable. I > >> personally don't think it will be much easier, especially if we are > >> intent on always maintaining compatibility with storm. - Bobby > >> > >> > >> On Wednesday, November 11, 2015 5:42 AM, 刘键(Basti Liu) < > basti...@alibaba-inc.com> wrote: > >> > >> > >> Hi Taylor, > >> > >> > >> > >> Thanks for the merge plan. I have a question about “Phase 2.2”. > >> > >> Do you mean community plan to create a fresh new “java core” based on > current “clojure core” firstly, and then migrate the features from JStorm? > >> > >> If so, it confused me. It is really a huge job which might require a > long developing time to make it stable, while JStorm is already a stable > version. > >> > >> The release planned to be release after Nov 11th has already run online > stably several month in Alibaba. > >> > >> Besides this, there are many valuable internal requirements in Alibaba, > the fast evolution of JStorm is forseeable in next few months. > >> > >> If the “java core” is totally fresh new, it might bring many problems > for the coming merge. > >> > >> So, from the point of this view, I think it is much better and easier > to migrate the features of “clojure core” basing on JStorm for the “java > core”. > >> > >> Please correct me, if any misunderstanding. > >> > >> > >> > >> Regards > >> > >> Basti > >> > >> > >> > >> 发件人: P. Taylor Goetz [mailto:ptgo...@gmail.com] > >> 发送时间: 2015年11月11日 5:32 > >> 收件人: dev@storm.apache.org > >> 主题: [DISCUSS] Plan for Merging JStorm Code > >> > >> > >> > >> Based on a number of discussions regarding merging the JStorm code, > I’ve tried to distill the ideas presented and inserted some of my own. The > result is below. > >> > >> > >> > >> I’ve divided the plan into three phases, though they are not > necessarily sequential — obviously some tasks can take place in parallel. > >> > >> > >> > >> None of this is set in stone, just presented for discussion. Any and > all comments are welcome. > >> > >> > >> > >> ------- > >> > >> > >> > >> Phase 1 - Plan for 0.11.x Release > >> > >> 1. Determine feature set for 0.11.x and publish to wiki [1]. > >> > >> 2. Announce feature-freeze for 0.11.x > >> > >> 3. Create 0.11.x branch from master (Phase 2.4 can begin.) > >> > >> 4. Release 0.11.0 (or whatever version # we want to use) > >> > >> 5. Bug fixes and subsequent releases from 0.11.x-branch > >> > >> > >> > >> > >> > >> Phase 2 - Prepare for Merge ("master" and "jstorm-import" branches) > >> > >> 1. Determine/document unique features in JStorm (e.g. classpath > isolation, cgroups, etc.) and create JIRA for migrating the feature. > >> > >> 2. Create JIRA for migrating each clojure component (or logical group > of components) to Java. Assumes tests will be ported as well. > >> > >> 3. Discuss/establish style guide for Java coding conventions. Consider > using Oracle’s or Google’s Java conventions as a base — they are both > pretty solid. > >> > >> 4. align package names (e.g backtype.storm --> org.apache.storm / > >> com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin) > >> > >> > >> > >> > >> > >> Phase 3 - Migrate Clojure --> Java > >> > >> 1. Port code/tests to Java, leveraging existing JStorm code wherever > possible (core functionality only, features distinct to JStorm migrated > separately). > >> > >> 2. Port JStorm-specific features. > >> > >> 3. Begin releasing preview/beta versions. > >> > >> 4. Code cleanup (across the board) and refactoring using established > >> coding conventions, and leveraging PMD/Checkstyle reports for > >> reference. (Note: good oportunity for new contributors.) > >> > >> 5. Release 0.12.0 (or whatever version # we want to use) and lift > feature freeze. > >> > >> > >> > >> > >> > >> Notes: > >> > >> We should consider bumping up to version 1.0 sometime soon and then > switching to semantic versioning [3] from then on. > >> > >> > >> > >> > >> > >> With the exception of package name alignment, the "jstorm-import" > branch will largely be read-only throughout the process. > >> > >> > >> > >> During migration, it's probably easiest to operate with two local > clones of the Apache Storm repo: one for working (i.e. checked out to > working branch) and one for reference/copying (i.e. checked out to > "jstorm-import"). > >> > >> > >> > >> Feature-freeze probably only needs to be enforced against core > functionality. Components under "external" can likely be exempt, but we > should figure out a process for accepting and releasing new features during > the migration. > >> > >> > >> > >> Performance testing should be continuous throughout the process. Since > we don't really have ASF infrastructure for performance testing, we will > need a volunteer(s) to host and run the performance tests. Performance test > results can be posted to the wiki [2]. It would probably be a good idea to > establish a baseline with the 0.10.0 release. > >> > >> > >> > >> I’ve attached an analysis document Sean Zhong put together a while back > to the JStorm merge JIRA [4]. The analysis was against the 0.9.3 release > but is still relevant and has a lot of good information. > >> > >> > >> > >> > >> > >> [1] > >> https://cwiki.apache.org/confluence/display/STORM/Release+0.11.0+Feat > >> ure+Set > >> > >> [2] https://cwiki.apache.org/confluence/display/STORM/Storm+Home > >> > >> [3] http://semver.org > >> > >> [4] https://issues.apache.org/jira/browse/STORM-717 > >> > >> > >> > >> > >> > >> -Taylor > >> > >> > >> > >> > >> > >> > > > > > > >
