[DISCUSS] Plan for Merging JStorm Code
Based on a number of discussions regarding merging the JStorm code, I’ve tried
to distill the ideas presented and inserted some of my own. The result is below.
I’ve divided the plan into three phases, though they are not necessarily
sequential — obviously some tasks can take place in parallel.
None of this is set in stone, just presented for discussion. Any and all
comments are welcome.
---
Phase 1 - Plan for 0.11.x Release
1. Determine feature set for 0.11.x and publish to wiki [1].
2. Announce feature-freeze for 0.11.x
3. Create 0.11.x branch from master (Phase 2.4 can begin.)
4. Release 0.11.0 (or whatever version # we want to use)
5. Bug fixes and subsequent releases from 0.11.x-branch
Phase 2 - Prepare for Merge ("master" and "jstorm-import" branches)
1. Determine/document unique features in JStorm (e.g. classpath isolation,
cgroups, etc.) and create JIRA for migrating the feature.
2. Create JIRA for migrating each clojure component (or logical group of
components) to Java. Assumes tests will be ported as well.
3. Discuss/establish style guide for Java coding conventions. Consider using
Oracle’s or Google’s Java conventions as a base — they are both pretty solid.
4. align package names (e.g backtype.storm --> org.apache.storm /
com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin)
Phase 3 - Migrate Clojure --> Java
1. Port code/tests to Java, leveraging existing JStorm code wherever possible
(core functionality only, features distinct to JStorm migrated separately).
2. Port JStorm-specific features.
3. Begin releasing preview/beta versions.
4. Code cleanup (across the board) and refactoring using established coding
conventions, and leveraging PMD/Checkstyle reports for reference. (Note: good
oportunity for new contributors.)
5. Release 0.12.0 (or whatever version # we want to use) and lift feature
freeze.
Notes:
We should consider bumping up to version 1.0 sometime soon and then switching
to semantic versioning [3] from then on.
With the exception of package name alignment, the "jstorm-import" branch will
largely be read-only throughout the process.
During migration, it's probably easiest to operate with two local clones of the
Apache Storm repo: one for working (i.e. checked out to working branch) and one
for reference/copying (i.e. checked out to "jstorm-import").
Feature-freeze probably only needs to be enforced against core functionality.
Components under "external" can likely be exempt, but we should figure out a
process for accepting and releasing new features during the migration.
Performance testing should be continuous throughout the process. Since we don't
really have ASF infrastructure for performance testing, we will need a
volunteer(s) to host and run the performance tests. Performance test results
can be posted to the wiki [2]. It would probably be a good idea to establish a
baseline with the 0.10.0 release.
I’ve attached an analysis document Sean Zhong put together a while back to the
JStorm merge JIRA [4]. The analysis was against the 0.9.3 release but is still
relevant and has a lot of good information.
[1] https://cwiki.apache.org/confluence/display/STORM/Release+0.11.0+Feature+Set
[2] https://cwiki.apache.org/confluence/display/STORM/Storm+Home
[3] http://semver.org
[4] https://issues.apache.org/jira/browse/STORM-717
-Taylor
signature.asc
Description: Message signed with OpenPGP using GPGMail
RE: [DISCUSS] Plan for Merging JStorm Code
Hi Bobby & Jungtaek, Thanks for your replay. I totally agree that compatibility is the most important thing. Actually, JStorm has been compatible with the user API of Storm. As you mentioned below, we indeed still have some features different between Storm and JStorm. I have tried to list them (minor update or improvements are not included). Please refer to attachment for details. If any missing, please help to point out. (The current working features are probably missing here.) Just have a look at these differences. For the missing features in JStorm, I did not see any obstacle which will block the merge to JStorm. For the features which has different solution between Storm and JStorm, we can evaluate the solution one by one to decision which one is appropriate. After the finalization of evaluation, I think JStorm team can take the merging job and publish a stable release in 2 months. But anyway, the detailed implementation for these features with different solution is transparent to user. So, from user's point of view, there is not any compatibility problem. Besides compatibility, by our experience, stability is also important and is not an easy job. 4 people in JStorm team took almost one year to finish the porting from "clojure core" to "java core", and to make it stable. Of course, we have many devs in community to make the porting job faster. But it still needs a long time to run many online complex topologys to find bugs and fix them. So, that is the reason why I proposed to do merging and build on a stable "java core". -Original Message- From: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID] Sent: Wednesday, November 11, 2015 10:51 PM To: dev@storm.apache.org Subject: Re: [DISCUSS] Plan for Merging JStorm Code +1 for doing a 1.0 release based off of the clojure 0.11.x code. Migrating the APIs to org.apache.storm is a big non-backwards compatible move, and a major version bump to 2.x seems like a good move there. +1 for the release plan I would like the move for user facing APIs to org.apache to be one of the last things we do. Translating clojure code into java and moving it to org.apache I am not too concerned about. Basti, We have two code bases that have diverged significantly from one another in terms of functionality. The storm code now or soon will have A Heartbeat Server, Nimbus HA (Different Implementation), Resource Aware Scheduling, a distributed cache like API, log searching, security, massive performance improvements, shaded almost all of our dependencies, a REST API for programtically accessing everything on the UI, and I am sure I am missing a few other things. JStorm also has many changes including cgroup isolation, restructured zookeeper layout, classpath isolation, and more too. No matter what we do it will be a large effort to port changes from one code base to another, and from clojure to java. I proposed this initially because it can be broken up into incremental changes. It may take a little longer, but we will always have a working codebase that is testable and compatible with the current storm release, at least until we move the user facing APIs to be under org.apache. This lets the community continue to build and test the master branch and report problems that they find, which is incredibly valuable. I personally don't think it will be much easier, especially if we are intent on always maintaining compatibility with storm. - Bobby On Wednesday, November 11, 2015 5:42 AM, 刘键(Basti Liu) wrote: Hi Taylor, Thanks for the merge plan. I have a question about “Phase 2.2”. Do you mean community plan to create a fresh new “java core” based on current “clojure core” firstly, and then migrate the features from JStorm? If so, it confused me. It is really a huge job which might require a long developing time to make it stable, while JStorm is already a stable version. The release planned to be release after Nov 11th has already run online stably several month in Alibaba. Besides this, there are many valuable internal requirements in Alibaba, the fast evolution of JStorm is forseeable in next few months. If the “java core” is totally fresh new, it might bring many problems for the coming merge. So, from the point of this view, I think it is much better and easier to migrate the features of “clojure core” basing on JStorm for the “java core”. Please correct me, if any misunderstanding. Regards Basti 发件人: P. Taylor Goetz [mailto:ptgo...@gmail.com] 发送时间: 2015年11月11日 5:32 收件人: dev@storm.apache.org 主题: [DISCUSS] Plan for Merging JStorm Code Based on a number of discussions regarding merging the JStorm code, I’ve tried to distill the ideas presented and inserted some of my own. The result is below. I’ve divided the plan into three phases, though they are not necessarily sequential — obviously some tasks can take place in para
Re: [DISCUSS] Plan for Merging JStorm Code
I translated the excel spreadsheet into a markdown file and put up a pull request for it. https://github.com/apache/storm/pull/877 I did a few edits to it to make it work with Markdown, and to add in a few of my own comments. I also put in a field for JIRAs to be able to track the migration. Overall I think your evaluation was very good. We have a fair amount of work ahead of us to decide what version of various features we want to go forward with. - Bobby On Thursday, November 12, 2015 9:37 AM, 刘键(Basti Liu) wrote: Hi Bobby & Jungtaek, Thanks for your replay. I totally agree that compatibility is the most important thing. Actually, JStorm has been compatible with the user API of Storm. As you mentioned below, we indeed still have some features different between Storm and JStorm. I have tried to list them (minor update or improvements are not included). Please refer to attachment for details. If any missing, please help to point out. (The current working features are probably missing here.) Just have a look at these differences. For the missing features in JStorm, I did not see any obstacle which will block the merge to JStorm. For the features which has different solution between Storm and JStorm, we can evaluate the solution one by one to decision which one is appropriate. After the finalization of evaluation, I think JStorm team can take the merging job and publish a stable release in 2 months. But anyway, the detailed implementation for these features with different solution is transparent to user. So, from user's point of view, there is not any compatibility problem. Besides compatibility, by our experience, stability is also important and is not an easy job. 4 people in JStorm team took almost one year to finish the porting from "clojure core" to "java core", and to make it stable. Of course, we have many devs in community to make the porting job faster. But it still needs a long time to run many online complex topologys to find bugs and fix them. So, that is the reason why I proposed to do merging and build on a stable "java core". -Original Message- From: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID] Sent: Wednesday, November 11, 2015 10:51 PM To: dev@storm.apache.org Subject: Re: [DISCUSS] Plan for Merging JStorm Code +1 for doing a 1.0 release based off of the clojure 0.11.x code. Migrating the APIs to org.apache.storm is a big non-backwards compatible move, and a major version bump to 2.x seems like a good move there. +1 for the release plan I would like the move for user facing APIs to org.apache to be one of the last things we do. Translating clojure code into java and moving it to org.apache I am not too concerned about. Basti, We have two code bases that have diverged significantly from one another in terms of functionality. The storm code now or soon will have A Heartbeat Server, Nimbus HA (Different Implementation), Resource Aware Scheduling, a distributed cache like API, log searching, security, massive performance improvements, shaded almost all of our dependencies, a REST API for programtically accessing everything on the UI, and I am sure I am missing a few other things. JStorm also has many changes including cgroup isolation, restructured zookeeper layout, classpath isolation, and more too. No matter what we do it will be a large effort to port changes from one code base to another, and from clojure to java. I proposed this initially because it can be broken up into incremental changes. It may take a little longer, but we will always have a working codebase that is testable and compatible with the current storm release, at least until we move the user facing APIs to be under org.apache. This lets the community continue to build and test the master branch and report problems that they find, which is incredibly valuable. I personally don't think it will be much easier, especially if we are intent on always maintaining compatibility with storm. - Bobby On Wednesday, November 11, 2015 5:42 AM, 刘键(Basti Liu) wrote: Hi Taylor, Thanks for the merge plan. I have a question about “Phase 2.2”. Do you mean community plan to create a fresh new “java core” based on current “clojure core” firstly, and then migrate the features from JStorm? If so, it confused me. It is really a huge job which might require a long developing time to make it stable, while JStorm is already a stable version. The release planned to be release after Nov 11th has already run online stably several month in Alibaba. Besides this, there are many valuable internal requirements in Alibaba, the fast evolution of JStorm is forseeable in next few months. If the “java core” is totally fresh new, it might bring many problems for the coming merge. So, from the point of this view, I think it is much better and easier to migrate the features of “clojure core” basing on
Re: [DISCUSS] Plan for Merging JStorm Code
Thanks for putting this together Basti, that comparison helps a lot. And thanks Bobby for converting it into markdown. I was going to just attach the spreadsheet to JIRA, but markdown is a much better solution. -Taylor > On Nov 12, 2015, at 12:03 PM, Bobby Evans wrote: > > I translated the excel spreadsheet into a markdown file and put up a pull > request for it. > https://github.com/apache/storm/pull/877 > I did a few edits to it to make it work with Markdown, and to add in a few of > my own comments. I also put in a field for JIRAs to be able to track the > migration. > Overall I think your evaluation was very good. We have a fair amount of work > ahead of us to decide what version of various features we want to go forward > with. > - Bobby > > > On Thursday, November 12, 2015 9:37 AM, 刘键(Basti Liu) > wrote: > > > Hi Bobby & Jungtaek, > > Thanks for your replay. > I totally agree that compatibility is the most important thing. Actually, > JStorm has been compatible with the user API of Storm. > As you mentioned below, we indeed still have some features different between > Storm and JStorm. I have tried to list them (minor update or improvements are > not included). > Please refer to attachment for details. If any missing, please help to point > out. (The current working features are probably missing here.) > Just have a look at these differences. For the missing features in JStorm, I > did not see any obstacle which will block the merge to JStorm. > For the features which has different solution between Storm and JStorm, we > can evaluate the solution one by one to decision which one is appropriate. > After the finalization of evaluation, I think JStorm team can take the > merging job and publish a stable release in 2 months. > But anyway, the detailed implementation for these features with different > solution is transparent to user. So, from user's point of view, there is not > any compatibility problem. > > Besides compatibility, by our experience, stability is also important and is > not an easy job. 4 people in JStorm team took almost one year to finish the > porting from "clojure core" > to "java core", and to make it stable. Of course, we have many devs in > community to make the porting job faster. But it still needs a long time to > run many online complex topologys to find bugs and fix them. So, that is the > reason why I proposed to do merging and build on a stable "java core". > > -----Original Message- > From: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID] > Sent: Wednesday, November 11, 2015 10:51 PM > To: dev@storm.apache.org > Subject: Re: [DISCUSS] Plan for Merging JStorm Code > > +1 for doing a 1.0 release based off of the clojure 0.11.x code. Migrating > the APIs to org.apache.storm is a big non-backwards compatible move, and a > major version bump to 2.x seems like a good move there. > +1 for the release plan > > I would like the move for user facing APIs to org.apache to be one of the > last things we do. Translating clojure code into java and moving it to > org.apache I am not too concerned about. > > Basti, > We have two code bases that have diverged significantly from one another in > terms of functionality. The storm code now or soon will have A Heartbeat > Server, Nimbus HA (Different Implementation), Resource Aware Scheduling, a > distributed cache like API, log searching, security, massive performance > improvements, shaded almost all of our dependencies, a REST API for > programtically accessing everything on the UI, and I am sure I am missing a > few other things. JStorm also has many changes including cgroup isolation, > restructured zookeeper layout, classpath isolation, and more too. > No matter what we do it will be a large effort to port changes from one code > base to another, and from clojure to java. I proposed this initially because > it can be broken up into incremental changes. It may take a little longer, > but we will always have a working codebase that is testable and compatible > with the current storm release, at least until we move the user facing APIs > to be under org.apache. This lets the community continue to build and test > the master branch and report problems that they find, which is incredibly > valuable. I personally don't think it will be much easier, especially if we > are intent on always maintaining compatibility with storm. - Bobby > > > On Wednesday, November 11, 2015 5:42 AM, 刘键(Basti Liu) > wrote: > > > Hi Taylor, > > > > Thanks for the merge plan. I have a question about “Phase 2.2”. > > Do you mean community plan to create a fresh
Re: [DISCUSS] Plan for Merging JStorm Code
Taylor and others I was hoping to get started filing JIRA and planning on how we are going to do the java migration + JStorm merger. Is anyone else starting to do this? If not would anyone object to me starting on it? - Bobby On Thursday, November 12, 2015 12:04 PM, P. Taylor Goetz wrote: Thanks for putting this together Basti, that comparison helps a lot. And thanks Bobby for converting it into markdown. I was going to just attach the spreadsheet to JIRA, but markdown is a much better solution. -Taylor > On Nov 12, 2015, at 12:03 PM, Bobby Evans wrote: > > I translated the excel spreadsheet into a markdown file and put up a pull > request for it. > https://github.com/apache/storm/pull/877 > I did a few edits to it to make it work with Markdown, and to add in a few of > my own comments. I also put in a field for JIRAs to be able to track the > migration. > Overall I think your evaluation was very good. We have a fair amount of work > ahead of us to decide what version of various features we want to go forward > with. > - Bobby > > > On Thursday, November 12, 2015 9:37 AM, 刘键(Basti Liu) > wrote: > > > Hi Bobby & Jungtaek, > > Thanks for your replay. > I totally agree that compatibility is the most important thing. Actually, > JStorm has been compatible with the user API of Storm. > As you mentioned below, we indeed still have some features different between > Storm and JStorm. I have tried to list them (minor update or improvements are > not included). > Please refer to attachment for details. If any missing, please help to point > out. (The current working features are probably missing here.) > Just have a look at these differences. For the missing features in JStorm, I > did not see any obstacle which will block the merge to JStorm. > For the features which has different solution between Storm and JStorm, we > can evaluate the solution one by one to decision which one is appropriate. > After the finalization of evaluation, I think JStorm team can take the > merging job and publish a stable release in 2 months. > But anyway, the detailed implementation for these features with different > solution is transparent to user. So, from user's point of view, there is not > any compatibility problem. > > Besides compatibility, by our experience, stability is also important and is > not an easy job. 4 people in JStorm team took almost one year to finish the > porting from "clojure core" > to "java core", and to make it stable. Of course, we have many devs in > community to make the porting job faster. But it still needs a long time to > run many online complex topologys to find bugs and fix them. So, that is the > reason why I proposed to do merging and build on a stable "java core". > > -----Original Message- > From: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID] > Sent: Wednesday, November 11, 2015 10:51 PM > To: dev@storm.apache.org > Subject: Re: [DISCUSS] Plan for Merging JStorm Code > > +1 for doing a 1.0 release based off of the clojure 0.11.x code. Migrating > the APIs to org.apache.storm is a big non-backwards compatible move, and a > major version bump to 2.x seems like a good move there. > +1 for the release plan > > I would like the move for user facing APIs to org.apache to be one of the > last things we do. Translating clojure code into java and moving it to > org.apache I am not too concerned about. > > Basti, > We have two code bases that have diverged significantly from one another in > terms of functionality. The storm code now or soon will have A Heartbeat > Server, Nimbus HA (Different Implementation), Resource Aware Scheduling, a > distributed cache like API, log searching, security, massive performance > improvements, shaded almost all of our dependencies, a REST API for > programtically accessing everything on the UI, and I am sure I am missing a > few other things. JStorm also has many changes including cgroup isolation, > restructured zookeeper layout, classpath isolation, and more too. > No matter what we do it will be a large effort to port changes from one code > base to another, and from clojure to java. I proposed this initially because > it can be broken up into incremental changes. It may take a little longer, > but we will always have a working codebase that is testable and compatible > with the current storm release, at least until we move the user facing APIs > to be under org.apache. This lets the community continue to build and test > the master branch and report problems that they find, which is incredibly > valuable. I personally don't think it will be much easier, especially if we > are intent on always mainta
Re: [DISCUSS] Plan for Merging JStorm Code
All I have at this point is a placeholder wiki entry [1], and a lot of local notes that likely would only make sense to me. Let me know your wiki username and I’ll give you permissions. The same goes for anyone else who wants to help. -Taylor [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61328109 > On Nov 18, 2015, at 2:08 PM, Bobby Evans wrote: > > Taylor and others I was hoping to get started filing JIRA and planning on how > we are going to do the java migration + JStorm merger. Is anyone else > starting to do this? If not would anyone object to me starting on it? - Bobby > > >On Thursday, November 12, 2015 12:04 PM, P. Taylor Goetz > wrote: > > > Thanks for putting this together Basti, that comparison helps a lot. > > And thanks Bobby for converting it into markdown. I was going to just attach > the spreadsheet to JIRA, but markdown is a much better solution. > > -Taylor > >> On Nov 12, 2015, at 12:03 PM, Bobby Evans >> wrote: >> >> I translated the excel spreadsheet into a markdown file and put up a pull >> request for it. >> https://github.com/apache/storm/pull/877 >> I did a few edits to it to make it work with Markdown, and to add in a few >> of my own comments. I also put in a field for JIRAs to be able to track the >> migration. >> Overall I think your evaluation was very good. We have a fair amount of >> work ahead of us to decide what version of various features we want to go >> forward with. >> - Bobby >> >> >> On Thursday, November 12, 2015 9:37 AM, 刘键(Basti Liu) >> wrote: >> >> >> Hi Bobby & Jungtaek, >> >> Thanks for your replay. >> I totally agree that compatibility is the most important thing. Actually, >> JStorm has been compatible with the user API of Storm. >> As you mentioned below, we indeed still have some features different between >> Storm and JStorm. I have tried to list them (minor update or improvements >> are not included). >> Please refer to attachment for details. If any missing, please help to point >> out. (The current working features are probably missing here.) >> Just have a look at these differences. For the missing features in JStorm, I >> did not see any obstacle which will block the merge to JStorm. >> For the features which has different solution between Storm and JStorm, we >> can evaluate the solution one by one to decision which one is appropriate. >> After the finalization of evaluation, I think JStorm team can take the >> merging job and publish a stable release in 2 months. >> But anyway, the detailed implementation for these features with different >> solution is transparent to user. So, from user's point of view, there is not >> any compatibility problem. >> >> Besides compatibility, by our experience, stability is also important and is >> not an easy job. 4 people in JStorm team took almost one year to finish the >> porting from "clojure core" >> to "java core", and to make it stable. Of course, we have many devs in >> community to make the porting job faster. But it still needs a long time to >> run many online complex topologys to find bugs and fix them. So, that is the >> reason why I proposed to do merging and build on a stable "java core". >> >> -Original Message- >> From: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID] >> Sent: Wednesday, November 11, 2015 10:51 PM >> To: dev@storm.apache.org >> Subject: Re: [DISCUSS] Plan for Merging JStorm Code >> >> +1 for doing a 1.0 release based off of the clojure 0.11.x code. Migrating >> the APIs to org.apache.storm is a big non-backwards compatible move, and a >> major version bump to 2.x seems like a good move there. >> +1 for the release plan >> >> I would like the move for user facing APIs to org.apache to be one of the >> last things we do. Translating clojure code into java and moving it to >> org.apache I am not too concerned about. >> >> Basti, >> We have two code bases that have diverged significantly from one another in >> terms of functionality. The storm code now or soon will have A Heartbeat >> Server, Nimbus HA (Different Implementation), Resource Aware Scheduling, a >> distributed cache like API, log searching, security, massive performance >> improvements, shaded almost all of our dependencies, a REST API for >> programtically accessing everything on the UI, and I am sure I am missing a >> few other things. JStorm also has many changes including cgroup isolation, >>
Re: [DISCUSS] Plan for Merging JStorm Code
Sorry about this taking so long. I am revans2 on the cwiki. - Bobby On Wednesday, November 18, 2015 4:24 PM, P. Taylor Goetz wrote: All I have at this point is a placeholder wiki entry [1], and a lot of local notes that likely would only make sense to me. Let me know your wiki username and I’ll give you permissions. The same goes for anyone else who wants to help. -Taylor [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61328109 > On Nov 18, 2015, at 2:08 PM, Bobby Evans wrote: > > Taylor and others I was hoping to get started filing JIRA and planning on how > we are going to do the java migration + JStorm merger. Is anyone else > starting to do this? If not would anyone object to me starting on it? - Bobby > > > On Thursday, November 12, 2015 12:04 PM, P. Taylor Goetz > wrote: > > > Thanks for putting this together Basti, that comparison helps a lot. > > And thanks Bobby for converting it into markdown. I was going to just attach > the spreadsheet to JIRA, but markdown is a much better solution. > > -Taylor > >> On Nov 12, 2015, at 12:03 PM, Bobby Evans >> wrote: >> >> I translated the excel spreadsheet into a markdown file and put up a pull >> request for it. >> https://github.com/apache/storm/pull/877 >> I did a few edits to it to make it work with Markdown, and to add in a few >> of my own comments. I also put in a field for JIRAs to be able to track the >> migration. >> Overall I think your evaluation was very good. We have a fair amount of >> work ahead of us to decide what version of various features we want to go >> forward with. >> - Bobby >> >> >> On Thursday, November 12, 2015 9:37 AM, 刘键(Basti Liu) >> wrote: >> >> >> Hi Bobby & Jungtaek, >> >> Thanks for your replay. >> I totally agree that compatibility is the most important thing. Actually, >> JStorm has been compatible with the user API of Storm. >> As you mentioned below, we indeed still have some features different between >> Storm and JStorm. I have tried to list them (minor update or improvements >> are not included). >> Please refer to attachment for details. If any missing, please help to point >> out. (The current working features are probably missing here.) >> Just have a look at these differences. For the missing features in JStorm, I >> did not see any obstacle which will block the merge to JStorm. >> For the features which has different solution between Storm and JStorm, we >> can evaluate the solution one by one to decision which one is appropriate. >> After the finalization of evaluation, I think JStorm team can take the >> merging job and publish a stable release in 2 months. >> But anyway, the detailed implementation for these features with different >> solution is transparent to user. So, from user's point of view, there is not >> any compatibility problem. >> >> Besides compatibility, by our experience, stability is also important and is >> not an easy job. 4 people in JStorm team took almost one year to finish the >> porting from "clojure core" >> to "java core", and to make it stable. Of course, we have many devs in >> community to make the porting job faster. But it still needs a long time to >> run many online complex topologys to find bugs and fix them. So, that is the >> reason why I proposed to do merging and build on a stable "java core". >> >> -Original Message- >> From: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID] >> Sent: Wednesday, November 11, 2015 10:51 PM >> To: dev@storm.apache.org >> Subject: Re: [DISCUSS] Plan for Merging JStorm Code >> >> +1 for doing a 1.0 release based off of the clojure 0.11.x code. Migrating >> the APIs to org.apache.storm is a big non-backwards compatible move, and a >> major version bump to 2.x seems like a good move there. >> +1 for the release plan >> >> I would like the move for user facing APIs to org.apache to be one of the >> last things we do. Translating clojure code into java and moving it to >> org.apache I am not too concerned about. >> >> Basti, >> We have two code bases that have diverged significantly from one another in >> terms of functionality. The storm code now or soon will have A Heartbeat >> Server, Nimbus HA (Different Implementation), Resource Aware Scheduling, a >> distributed cache like API, log searching, security, massive performance >> improvements, shaded almost all of our dependencies, a REST API for >> programtically accessing eve
Re: [DISCUSS] Plan for Merging JStorm Code
@Taylor, I want to work with Bobby, my wiki name is "longda". ThanksLongda--From:Bobby Evans Send Time:2015年11月20日(星期五) 06:54To:P. Taylor Goetz ，dev@storm.apache.org Subject:Re: [DISCUSS] Plan for Merging JStorm Code Sorry about this taking so long. I am revans2 on the cwiki. - Bobby On Wednesday, November 18, 2015 4:24 PM, P. Taylor Goetz wrote: All I have at this point is a placeholder wiki entry [1], and a lot of local notes that likely would only make sense to me. Let me know your wiki username and I’ll give you permissions. The same goes for anyone else who wants to help. -Taylor [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61328109 > On Nov 18, 2015, at 2:08 PM, Bobby Evans wrote: > > Taylor and others I was hoping to get started filing JIRA and planning on how >we are going to do the java migration + JStorm merger. Is anyone else >starting to do this? If not would anyone object to me starting on it? - Bobby > > >On Thursday, November 12, 2015 12:04 PM, P. Taylor Goetz > wrote: > > > Thanks for putting this together Basti, that comparison helps a lot. > > And thanks Bobby for converting it into markdown. I was going to just attach >the spreadsheet to JIRA, but markdown is a much better solution. > > -Taylor > >> On Nov 12, 2015, at 12:03 PM, Bobby Evans >>wrote: >> >> I translated the excel spreadsheet into a markdown file and put up a pull >>request for it. >> https://github.com/apache/storm/pull/877 >> I did a few edits to it to make it work with Markdown, and to add in a few >>of my own comments. I also put in a field for JIRAs to be able to track the >>migration. >> Overall I think your evaluation was very good. We have a fair amount of >>work ahead of us to decide what version of various features we want to go >>forward with. >> - Bobby >> >> >>On Thursday, November 12, 2015 9:37 AM, 刘键(Basti Liu) >> wrote: >> >> >> Hi Bobby & Jungtaek, >> >> Thanks for your replay. >> I totally agree that compatibility is the most important thing. Actually, >>JStorm has been compatible with the user API of Storm. >> As you mentioned below, we indeed still have some features different between >>Storm and JStorm. I have tried to list them (minor update or improvements are >>not included). >> Please refer to attachment for details. If any missing, please help to point >>out. (The current working features are probably missing here.) >> Just have a look at these differences. For the missing features in JStorm, I >>did not see any obstacle which will block the merge to JStorm. >> For the features which has different solution between Storm and JStorm, we >>can evaluate the solution one by one to decision which one is appropriate. >> After the finalization of evaluation, I think JStorm team can take the >>merging job and publish a stable release in 2 months. >> But anyway, the detailed implementation for these features with different >>solution is transparent to user. So, from user's point of view, there is not >>any compatibility problem. >> >> Besides compatibility, by our experience, stability is also important and is >>not an easy job. 4 people in JStorm team took almost one year to finish the >>porting from "clojure core" >> to "java core", and to make it stable. Of course, we have many devs in >>community to make the porting job faster. But it still needs a long time to >>run many online complex topologys to find bugs and fix them. So, that is the >>reason why I proposed to do merging and build on a stable "java core". >> >> -Original Message- >> From: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID] >> Sent: Wednesday, November 11, 2015 10:51 PM >> To: dev@storm.apache.org >> Subject: Re: [DISCUSS] Plan for Merging JStorm Code >> >> +1 for doing a 1.0 release based off of the clojure 0.11.x code. Migrating >>the APIs to org.apache.storm is a big non-backwards compatible move, and a >>major version bump to 2.x seems like a good move there. >> +1 for the release plan >> >> I would like the move for user facing APIs to org.apache to be one of the >>last things we do. Translating clojure code into java and moving it to >>org.apache I am not too concerned about. >> >> Basti, >> We have two code bases that have diverged significantly from one another in >>terms of functionality. The storm code now or soon will have A Heartbeat >>Server, Nimbus
Re: [DISCUSS] Plan for Merging JStorm Code
Taylor, Can you add me to the wiki as well my id is "harsha". Thanks, Harsha On Fri, Nov 20, 2015, at 01:19 AM, Longda Feng wrote: > @Taylor, I want to work with Bobby, my wiki name is "longda". > ThanksLongda--From:Bobby > Evans Send Time:2015年11月20日(星期五) 06:54To:P. > Taylor Goetz ，dev@storm.apache.org > Subject:Re: [DISCUSS] Plan for Merging JStorm Code > Sorry about this taking so long. I am revans2 on the cwiki. - Bobby > > > On Wednesday, November 18, 2015 4:24 PM, P. Taylor Goetz > wrote: > > > All I have at this point is a placeholder wiki entry [1], and a lot of local > notes that likely would only make sense to me. > > Let me know your wiki username and I’ll give you permissions. The same > goes for anyone else who wants to help. > > -Taylor > > [1] > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61328109 > > > On Nov 18, 2015, at 2:08 PM, Bobby Evans > >wrote: > > > > Taylor and others I was hoping to get started filing JIRA and planning on > >how we are going to do the java migration + JStorm merger. Is anyone else > >starting to do this? If not would anyone object to me starting on it? - > >Bobby > > > > > >On Thursday, November 12, 2015 12:04 PM, P. Taylor Goetz > > wrote: > > > > > > Thanks for putting this together Basti, that comparison helps a lot. > > > > And thanks Bobby for converting it into markdown. I was going to just > >attach the spreadsheet to JIRA, but markdown is a much better solution. > > > > -Taylor > > > >> On Nov 12, 2015, at 12:03 PM, Bobby Evans > >>wrote: > >> > >> I translated the excel spreadsheet into a markdown file and put up a pull > >>request for it. > >> https://github.com/apache/storm/pull/877 > >> I did a few edits to it to make it work with Markdown, and to add in a few > >>of my own comments. I also put in a field for JIRAs to be able to track > >>the migration. > >> Overall I think your evaluation was very good. We have a fair amount of > >>work ahead of us to decide what version of various features we want to go > >>forward with. > >> - Bobby > >> > >> > >>On Thursday, November 12, 2015 9:37 AM, 刘键(Basti Liu) > >> wrote: > >> > >> > >> Hi Bobby & Jungtaek, > >> > >> Thanks for your replay. > >> I totally agree that compatibility is the most important thing. Actually, > >>JStorm has been compatible with the user API of Storm. > >> As you mentioned below, we indeed still have some features different > >>between Storm and JStorm. I have tried to list them (minor update or > >>improvements are not included). > >> Please refer to attachment for details. If any missing, please help to > >>point out. (The current working features are probably missing here.) > >> Just have a look at these differences. For the missing features in JStorm, > >>I did not see any obstacle which will block the merge to JStorm. > >> For the features which has different solution between Storm and JStorm, we > >>can evaluate the solution one by one to decision which one is appropriate. > >> After the finalization of evaluation, I think JStorm team can take the > >>merging job and publish a stable release in 2 months. > >> But anyway, the detailed implementation for these features with different > >>solution is transparent to user. So, from user's point of view, there is > >>not any compatibility problem. > >> > >> Besides compatibility, by our experience, stability is also important and > >>is not an easy job. 4 people in JStorm team took almost one year to finish > >>the porting from "clojure core" > >> to "java core", and to make it stable. Of course, we have many devs in > >>community to make the porting job faster. But it still needs a long time to > >>run many online complex topologys to find bugs and fix them. So, that is > >>the reason why I proposed to do merging and build on a stable "java core". > >> > >> -Original Message- > >> From: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID] > >> Sent: Wednesday, November 11, 2015 10:51 PM > >> To: dev@storm.apache.org > >> Subject: Re: [DISCUSS] Plan for Merging JStorm Code > >> > >> +1 for doing a 1.0 release based off of the
RE: [DISCUSS] Plan for Merging JStorm Code
Hi Taylor， Sorry for the late response. I'd like to help on this. Could you please help to give me the permission? Thanks. UserName: basti.lj Regards Basti -Original Message- From: P. Taylor Goetz [mailto:ptgo...@gmail.com] Sent: Thursday, November 19, 2015 6:24 AM To: dev@storm.apache.org; Bobby Evans Subject: Re: [DISCUSS] Plan for Merging JStorm Code All I have at this point is a placeholder wiki entry [1], and a lot of local notes that likely would only make sense to me. Let me know your wiki username and I’ll give you permissions. The same goes for anyone else who wants to help. -Taylor [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61328109 > On Nov 18, 2015, at 2:08 PM, Bobby Evans wrote: > > Taylor and others I was hoping to get started filing JIRA and planning > on how we are going to do the java migration + JStorm merger. Is > anyone else starting to do this? If not would anyone object to me > starting on it? - Bobby > > >On Thursday, November 12, 2015 12:04 PM, P. Taylor Goetz > wrote: > > > Thanks for putting this together Basti, that comparison helps a lot. > > And thanks Bobby for converting it into markdown. I was going to just attach > the spreadsheet to JIRA, but markdown is a much better solution. > > -Taylor > >> On Nov 12, 2015, at 12:03 PM, Bobby Evans >> wrote: >> >> I translated the excel spreadsheet into a markdown file and put up a pull >> request for it. >> https://github.com/apache/storm/pull/877 >> I did a few edits to it to make it work with Markdown, and to add in a few >> of my own comments. I also put in a field for JIRAs to be able to track the >> migration. >> Overall I think your evaluation was very good. We have a fair amount of >> work ahead of us to decide what version of various features we want to go >> forward with. >> - Bobby >> >> >> On Thursday, November 12, 2015 9:37 AM, 刘键(Basti Liu) >> wrote: >> >> >> Hi Bobby & Jungtaek, >> >> Thanks for your replay. >> I totally agree that compatibility is the most important thing. Actually, >> JStorm has been compatible with the user API of Storm. >> As you mentioned below, we indeed still have some features different between >> Storm and JStorm. I have tried to list them (minor update or improvements >> are not included). >> Please refer to attachment for details. If any missing, please help >> to point out. (The current working features are probably missing here.) Just >> have a look at these differences. For the missing features in JStorm, I did >> not see any obstacle which will block the merge to JStorm. >> For the features which has different solution between Storm and JStorm, we >> can evaluate the solution one by one to decision which one is appropriate. >> After the finalization of evaluation, I think JStorm team can take the >> merging job and publish a stable release in 2 months. >> But anyway, the detailed implementation for these features with different >> solution is transparent to user. So, from user's point of view, there is not >> any compatibility problem. >> >> Besides compatibility, by our experience, stability is also important and is >> not an easy job. 4 people in JStorm team took almost one year to finish the >> porting from "clojure core" >> to "java core", and to make it stable. Of course, we have many devs in >> community to make the porting job faster. But it still needs a long time to >> run many online complex topologys to find bugs and fix them. So, that is the >> reason why I proposed to do merging and build on a stable "java core". >> >> -Original Message- >> From: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID] >> Sent: Wednesday, November 11, 2015 10:51 PM >> To: dev@storm.apache.org >> Subject: Re: [DISCUSS] Plan for Merging JStorm Code >> >> +1 for doing a 1.0 release based off of the clojure 0.11.x code. Migrating >> the APIs to org.apache.storm is a big non-backwards compatible move, and a >> major version bump to 2.x seems like a good move there. >> +1 for the release plan >> >> I would like the move for user facing APIs to org.apache to be one of the >> last things we do. Translating clojure code into java and moving it to >> org.apache I am not too concerned about. >> >> Basti, >> We have two code bases that have diverged significantly from one another in >> terms of functionality. The storm code now or soon will have A Heartbeat >> Server, Nimbus HA (Different Implementat
Re: [DISCUSS] Plan for Merging JStorm Code
Hi Taylor, I'd like to help too, could you add me in? my id is: Cody On Sat, Nov 21, 2015 at 11:51 AM, 刘键(Basti Liu) wrote: > Hi Taylor， > > Sorry for the late response. > I'd like to help on this. Could you please help to give me the permission? > Thanks. > UserName: basti.lj > > Regards > Basti > > -Original Message- > From: P. Taylor Goetz [mailto:ptgo...@gmail.com] > Sent: Thursday, November 19, 2015 6:24 AM > To: dev@storm.apache.org; Bobby Evans > Subject: Re: [DISCUSS] Plan for Merging JStorm Code > > All I have at this point is a placeholder wiki entry [1], and a lot of > local notes that likely would only make sense to me. > > Let me know your wiki username and I’ll give you permissions. The same > goes for anyone else who wants to help. > > -Taylor > > [1] > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61328109 > > > On Nov 18, 2015, at 2:08 PM, Bobby Evans > wrote: > > > > Taylor and others I was hoping to get started filing JIRA and planning > > on how we are going to do the java migration + JStorm merger. Is > > anyone else starting to do this? If not would anyone object to me > > starting on it? - Bobby > > > > > >On Thursday, November 12, 2015 12:04 PM, P. Taylor Goetz < > ptgo...@gmail.com> wrote: > > > > > > Thanks for putting this together Basti, that comparison helps a lot. > > > > And thanks Bobby for converting it into markdown. I was going to just > attach the spreadsheet to JIRA, but markdown is a much better solution. > > > > -Taylor > > > >> On Nov 12, 2015, at 12:03 PM, Bobby Evans > wrote: > >> > >> I translated the excel spreadsheet into a markdown file and put up a > pull request for it. > >> https://github.com/apache/storm/pull/877 > >> I did a few edits to it to make it work with Markdown, and to add in a > few of my own comments. I also put in a field for JIRAs to be able to > track the migration. > >> Overall I think your evaluation was very good. We have a fair amount > of work ahead of us to decide what version of various features we want to > go forward with. > >> - Bobby > >> > >> > >> On Thursday, November 12, 2015 9:37 AM, 刘键(Basti Liu) < > basti...@alibaba-inc.com> wrote: > >> > >> > >> Hi Bobby & Jungtaek, > >> > >> Thanks for your replay. > >> I totally agree that compatibility is the most important thing. > Actually, JStorm has been compatible with the user API of Storm. > >> As you mentioned below, we indeed still have some features different > between Storm and JStorm. I have tried to list them (minor update or > improvements are not included). > >> Please refer to attachment for details. If any missing, please help > >> to point out. (The current working features are probably missing here.) > Just have a look at these differences. For the missing features in JStorm, > I did not see any obstacle which will block the merge to JStorm. > >> For the features which has different solution between Storm and JStorm, > we can evaluate the solution one by one to decision which one is > appropriate. > >> After the finalization of evaluation, I think JStorm team can take the > merging job and publish a stable release in 2 months. > >> But anyway, the detailed implementation for these features with > different solution is transparent to user. So, from user's point of view, > there is not any compatibility problem. > >> > >> Besides compatibility, by our experience, stability is also important > and is not an easy job. 4 people in JStorm team took almost one year to > finish the porting from "clojure core" > >> to "java core", and to make it stable. Of course, we have many devs in > community to make the porting job faster. But it still needs a long time to > run many online complex topologys to find bugs and fix them. So, that is > the reason why I proposed to do merging and build on a stable "java core". > >> > >> -Original Message- > >> From: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID] > >> Sent: Wednesday, November 11, 2015 10:51 PM > >> To: dev@storm.apache.org > >> Subject: Re: [DISCUSS] Plan for Merging JStorm Code > >> > >> +1 for doing a 1.0 release based off of the clojure 0.11.x code. > Migrating the APIs to org.apache.storm is a big non-backwards compatible > move, and a major version bump to 2.x seems like a good move there. > >> +1 for the release plan > >> > >> I
Re: [DISCUSS] Plan for Merging JStorm Code
Hi Cody, I wasn’t able to find your username. Are you sure you have an account on cwiki.apache.org? -Taylor > On Nov 22, 2015, at 8:46 AM, Cody Innowhere wrote: > > Hi Taylor, > I'd like to help too, could you add me in? my id is: Cody > > On Sat, Nov 21, 2015 at 11:51 AM, 刘键(Basti Liu) > wrote: > >> Hi Taylor， >> >> Sorry for the late response. >> I'd like to help on this. Could you please help to give me the permission? >> Thanks. >> UserName: basti.lj >> >> Regards >> Basti >> >> -Original Message- >> From: P. Taylor Goetz [mailto:ptgo...@gmail.com] >> Sent: Thursday, November 19, 2015 6:24 AM >> To: dev@storm.apache.org; Bobby Evans >> Subject: Re: [DISCUSS] Plan for Merging JStorm Code >> >> All I have at this point is a placeholder wiki entry [1], and a lot of >> local notes that likely would only make sense to me. >> >> Let me know your wiki username and I’ll give you permissions. The same >> goes for anyone else who wants to help. >> >> -Taylor >> >> [1] >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61328109 >> >>> On Nov 18, 2015, at 2:08 PM, Bobby Evans >> wrote: >>> >>> Taylor and others I was hoping to get started filing JIRA and planning >>> on how we are going to do the java migration + JStorm merger. Is >>> anyone else starting to do this? If not would anyone object to me >>> starting on it? - Bobby >>> >>> >>> On Thursday, November 12, 2015 12:04 PM, P. Taylor Goetz < >> ptgo...@gmail.com> wrote: >>> >>> >>> Thanks for putting this together Basti, that comparison helps a lot. >>> >>> And thanks Bobby for converting it into markdown. I was going to just >> attach the spreadsheet to JIRA, but markdown is a much better solution. >>> >>> -Taylor >>> >>>> On Nov 12, 2015, at 12:03 PM, Bobby Evans >> wrote: >>>> >>>> I translated the excel spreadsheet into a markdown file and put up a >> pull request for it. >>>> https://github.com/apache/storm/pull/877 >>>> I did a few edits to it to make it work with Markdown, and to add in a >> few of my own comments. I also put in a field for JIRAs to be able to >> track the migration. >>>> Overall I think your evaluation was very good. We have a fair amount >> of work ahead of us to decide what version of various features we want to >> go forward with. >>>> - Bobby >>>> >>>> >>>>On Thursday, November 12, 2015 9:37 AM, 刘键(Basti Liu) < >> basti...@alibaba-inc.com> wrote: >>>> >>>> >>>> Hi Bobby & Jungtaek, >>>> >>>> Thanks for your replay. >>>> I totally agree that compatibility is the most important thing. >> Actually, JStorm has been compatible with the user API of Storm. >>>> As you mentioned below, we indeed still have some features different >> between Storm and JStorm. I have tried to list them (minor update or >> improvements are not included). >>>> Please refer to attachment for details. If any missing, please help >>>> to point out. (The current working features are probably missing here.) >> Just have a look at these differences. For the missing features in JStorm, >> I did not see any obstacle which will block the merge to JStorm. >>>> For the features which has different solution between Storm and JStorm, >> we can evaluate the solution one by one to decision which one is >> appropriate. >>>> After the finalization of evaluation, I think JStorm team can take the >> merging job and publish a stable release in 2 months. >>>> But anyway, the detailed implementation for these features with >> different solution is transparent to user. So, from user's point of view, >> there is not any compatibility problem. >>>> >>>> Besides compatibility, by our experience, stability is also important >> and is not an easy job. 4 people in JStorm team took almost one year to >> finish the porting from "clojure core" >>>> to "java core", and to make it stable. Of course, we have many devs in >> community to make the porting job faster. But it still needs a long time to >> run many online complex topologys to find bugs and fix them. So, that is >> the reason why I proposed to do merging and build on a stable "java core". >>>> >>>> -
Re: [DISCUSS] Plan for Merging JStorm Code
Sorry I've mistaken it with the apache issues account, it's ok now, my id is: cody. On Tue, Nov 24, 2015 at 11:54 PM, P. Taylor Goetz wrote: > Hi Cody, > > I wasn’t able to find your username. Are you sure you have an account on > cwiki.apache.org? > > -Taylor > > > On Nov 22, 2015, at 8:46 AM, Cody Innowhere wrote: > > > > Hi Taylor, > > I'd like to help too, could you add me in? my id is: Cody > > > > On Sat, Nov 21, 2015 at 11:51 AM, 刘键(Basti Liu) < > basti...@alibaba-inc.com> > > wrote: > > > >> Hi Taylor， > >> > >> Sorry for the late response. > >> I'd like to help on this. Could you please help to give me the > permission? > >> Thanks. > >> UserName: basti.lj > >> > >> Regards > >> Basti > >> > >> -----Original Message----- > >> From: P. Taylor Goetz [mailto:ptgo...@gmail.com] > >> Sent: Thursday, November 19, 2015 6:24 AM > >> To: dev@storm.apache.org; Bobby Evans > >> Subject: Re: [DISCUSS] Plan for Merging JStorm Code > >> > >> All I have at this point is a placeholder wiki entry [1], and a lot of > >> local notes that likely would only make sense to me. > >> > >> Let me know your wiki username and I’ll give you permissions. The same > >> goes for anyone else who wants to help. > >> > >> -Taylor > >> > >> [1] > >> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61328109 > >> > >>> On Nov 18, 2015, at 2:08 PM, Bobby Evans > >> wrote: > >>> > >>> Taylor and others I was hoping to get started filing JIRA and planning > >>> on how we are going to do the java migration + JStorm merger. Is > >>> anyone else starting to do this? If not would anyone object to me > >>> starting on it? - Bobby > >>> > >>> > >>> On Thursday, November 12, 2015 12:04 PM, P. Taylor Goetz < > >> ptgo...@gmail.com> wrote: > >>> > >>> > >>> Thanks for putting this together Basti, that comparison helps a lot. > >>> > >>> And thanks Bobby for converting it into markdown. I was going to just > >> attach the spreadsheet to JIRA, but markdown is a much better solution. > >>> > >>> -Taylor > >>> > >>>> On Nov 12, 2015, at 12:03 PM, Bobby Evans > > >> wrote: > >>>> > >>>> I translated the excel spreadsheet into a markdown file and put up a > >> pull request for it. > >>>> https://github.com/apache/storm/pull/877 > >>>> I did a few edits to it to make it work with Markdown, and to add in a > >> few of my own comments. I also put in a field for JIRAs to be able to > >> track the migration. > >>>> Overall I think your evaluation was very good. We have a fair amount > >> of work ahead of us to decide what version of various features we want > to > >> go forward with. > >>>> - Bobby > >>>> > >>>> > >>>>On Thursday, November 12, 2015 9:37 AM, 刘键(Basti Liu) < > >> basti...@alibaba-inc.com> wrote: > >>>> > >>>> > >>>> Hi Bobby & Jungtaek, > >>>> > >>>> Thanks for your replay. > >>>> I totally agree that compatibility is the most important thing. > >> Actually, JStorm has been compatible with the user API of Storm. > >>>> As you mentioned below, we indeed still have some features different > >> between Storm and JStorm. I have tried to list them (minor update or > >> improvements are not included). > >>>> Please refer to attachment for details. If any missing, please help > >>>> to point out. (The current working features are probably missing > here.) > >> Just have a look at these differences. For the missing features in > JStorm, > >> I did not see any obstacle which will block the merge to JStorm. > >>>> For the features which has different solution between Storm and > JStorm, > >> we can evaluate the solution one by one to decision which one is > >> appropriate. > >>>> After the finalization of evaluation, I think JStorm team can take the > >> merging job and publish a stable release in 2 months. > >>>> But anyway, the detailed implementation for these features with > >> different solution is transparent to user. So, from user's point
Re: [DISCUSS] Plan for Merging JStorm Code
Thanks Taylor for putting together plan on merging JStorm code.
Few things
1. we should call 0.11.0 as 1.0.0 release since storm has security and
nimbus ha . Quite a lot of features and improvements added this is
going to be big release and its about time we call 1.0.0
1.1 "align package names (e.g backtype.storm --> org.apache.storm /
com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin" I
propose we do this as next release itself instead of pushing it
onto another .
"Phase 3 - Migrate Clojure --> Java"
I would like to propose a Hackathon among storm community along with
jstorm. We need to come up with plan of action on what code needs to be
merged into storm-core. I am thinking it will help better to have
everyone over video chat or something to go over the code and get it
migrated to java.
Thanks, Harsha
On Tue, Nov 10, 2015, at 01:32 PM, P. Taylor Goetz wrote:
> Based on a number of discussions regarding merging the JStorm code,
> I’ve tried to distill the ideas presented and inserted some of my own.
> The result is below.
>
> I’ve divided the plan into three phases, though they are not
> necessarily sequential — obviously some tasks can take place in
> parallel.
>
> None of this is set in stone, just presented for discussion. Any and
> all comments are welcome.
>
> ---
>
> Phase 1 - Plan for 0.11.x Release
> 1. Determine feature set for 0.11.x and publish to wiki [1].
> 2. Announce feature-freeze for 0.11.x
> 3. Create 0.11.x branch from master (Phase 2.4 can begin.)
> 4. Release 0.11.0 (or whatever version # we want to use)
> 5. Bug fixes and subsequent releases from 0.11.x-branch
>
>
> Phase 2 - Prepare for Merge ("master" and "jstorm-import" branches)
> 1. Determine/document unique features in JStorm (e.g. classpath
>isolation, cgroups, etc.) and create JIRA for migrating the
>feature.
> 2. Create JIRA for migrating each clojure component (or logical group
>of components) to Java. Assumes tests will be ported as well.
> 3. Discuss/establish style guide for Java coding conventions. Consider
>using Oracle’s or Google’s Java conventions as a base — they are
>both pretty solid.
> 4. align package names (e.g backtype.storm --> org.apache.storm /
>com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin)
>
>
> Phase 3 - Migrate Clojure --> Java
> 1. Port code/tests to Java, leveraging existing JStorm code wherever
>possible (core functionality only, features distinct to JStorm
>migrated separately).
> 2. Port JStorm-specific features.
> 3. Begin releasing preview/beta versions.
> 4. Code cleanup (across the board) and refactoring using established
>coding conventions, and leveraging PMD/Checkstyle reports for
>reference. (Note: good oportunity for new contributors.)
> 5. Release 0.12.0 (or whatever version # we want to use) and lift
>feature freeze.
>
>
> Notes: We should consider bumping up to version 1.0 sometime soon and
> then switching to semantic versioning [3] from then on.
>
>
> With the exception of package name alignment, the "jstorm-import"
> branch will largely be read-only throughout the process.
>
> During migration, it's probably easiest to operate with two local
> clones of the Apache Storm repo: one for working (i.e. checked out to
> working branch) and one for reference/copying (i.e. checked out to "jstorm-
> import").
>
> Feature-freeze probably only needs to be enforced against core
> functionality. Components under "external" can likely be exempt, but
> we should figure out a process for accepting and releasing new
> features during the migration.
>
> Performance testing should be continuous throughout the process. Since
> we don't really have ASF infrastructure for performance testing, we
> will need a volunteer(s) to host and run the performance tests.
> Performance test results can be posted to the wiki [2]. It would
> probably be a good idea to establish a baseline with the 0.10.0
> release.
>
> I’ve attached an analysis document Sean Zhong put together a while
> back to the JStorm merge JIRA [4]. The analysis was against the 0.9.3
> release but is still relevant and has a lot of good information.
>
>
> [1]
> https://cwiki.apache.org/confluence/display/STORM/Release+0.11.0+Feature+Set
> [2] https://cwiki.apache.org/confluence/display/STORM/Storm+Home
> [3]http://semver.org
> [4]https://issues.apache.org/jira/browse/STORM-717
>
>
> -Taylor
>
> Email had 1 attachment:
> * signature.asc 1k (application/pgp-signature)
Re: [DISCUSS] Plan for Merging JStorm Code
+1 for 1.0.0 release.
I also like Hackathon to encourage bigger participation. It will be fun.
On 11/10/15, 3:38 PM, "Harsha" wrote:
>Thanks Taylor for putting together plan on merging JStorm code.
>Few things
>
>1. we should call 0.11.0 as 1.0.0 release since storm has security and
> nimbus ha . Quite a lot of features and improvements added this is
> going to be big release and its about time we call 1.0.0
>
>1.1 "align package names (e.g backtype.storm --> org.apache.storm /
>com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin" I
>propose we do this as next release itself instead of pushing it
>onto another .
>
>"Phase 3 - Migrate Clojure --> Java"
>
>I would like to propose a Hackathon among storm community along with
>jstorm. We need to come up with plan of action on what code needs to be
>merged into storm-core. I am thinking it will help better to have
>everyone over video chat or something to go over the code and get it
>migrated to java.
>
>
>Thanks, Harsha
>
>On Tue, Nov 10, 2015, at 01:32 PM, P. Taylor Goetz wrote:
>> Based on a number of discussions regarding merging the JStorm code,
>> I¹ve tried to distill the ideas presented and inserted some of my own.
>> The result is below.
>>
>> I¹ve divided the plan into three phases, though they are not
>> necessarily sequential ‹ obviously some tasks can take place in
>> parallel.
>>
>> None of this is set in stone, just presented for discussion. Any and
>> all comments are welcome.
>>
>> ---
>>
>> Phase 1 - Plan for 0.11.x Release
>> 1. Determine feature set for 0.11.x and publish to wiki [1].
>> 2. Announce feature-freeze for 0.11.x
>> 3. Create 0.11.x branch from master (Phase 2.4 can begin.)
>> 4. Release 0.11.0 (or whatever version # we want to use)
>> 5. Bug fixes and subsequent releases from 0.11.x-branch
>>
>>
>> Phase 2 - Prepare for Merge ("master" and "jstorm-import" branches)
>> 1. Determine/document unique features in JStorm (e.g. classpath
>>isolation, cgroups, etc.) and create JIRA for migrating the
>>feature.
>> 2. Create JIRA for migrating each clojure component (or logical group
>>of components) to Java. Assumes tests will be ported as well.
>> 3. Discuss/establish style guide for Java coding conventions. Consider
>>using Oracle¹s or Google¹s Java conventions as a base ‹ they are
>>both pretty solid.
>> 4. align package names (e.g backtype.storm --> org.apache.storm /
>>com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin)
>>
>>
>> Phase 3 - Migrate Clojure --> Java
>> 1. Port code/tests to Java, leveraging existing JStorm code wherever
>>possible (core functionality only, features distinct to JStorm
>>migrated separately).
>> 2. Port JStorm-specific features.
>> 3. Begin releasing preview/beta versions.
>> 4. Code cleanup (across the board) and refactoring using established
>>coding conventions, and leveraging PMD/Checkstyle reports for
>>reference. (Note: good oportunity for new contributors.)
>> 5. Release 0.12.0 (or whatever version # we want to use) and lift
>>feature freeze.
>>
>>
>> Notes: We should consider bumping up to version 1.0 sometime soon and
>> then switching to semantic versioning [3] from then on.
>>
>>
>> With the exception of package name alignment, the "jstorm-import"
>> branch will largely be read-only throughout the process.
>>
>> During migration, it's probably easiest to operate with two local
>> clones of the Apache Storm repo: one for working (i.e. checked out to
>> working branch) and one for reference/copying (i.e. checked out to
>>"jstorm-
>> import").
>>
>> Feature-freeze probably only needs to be enforced against core
>> functionality. Components under "external" can likely be exempt, but
>> we should figure out a process for accepting and releasing new
>> features during the migration.
>>
>> Performance testing should be continuous throughout the process. Since
>> we don't really have ASF infrastructure for performance testing, we
>> will need a volunteer(s) to host and run the performance tests.
>> Performance test results can be posted to the wiki [2]. It would
>> probably be a good idea to establish a baseline with the 0.10.0
>> release.
>>
>> I¹ve attached an analysis document Sean Zhong put together a while
>> back to the JStorm merge JIRA [4]. The analysis was against the 0.9.3
>> release but is still relevant and has a lot of good information.
>>
>>
>> [1]
>>https://cwiki.apache.org/confluence/display/STORM/Release+0.11.0+Feature+
>>Set
>> [2] https://cwiki.apache.org/confluence/display/STORM/Storm+Home
>> [3]http://semver.org
>> [4]https://issues.apache.org/jira/browse/STORM-717
>>
>>
>> -Taylor
>>
>> Email had 1 attachment:
>
>
>> * signature.asc 1k (application/pgp-signature)
Re: [DISCUSS] Plan for Merging JStorm Code
Thanks Taylor.
The plan looks good.
> During migration, it's probably easiest to operate with two local clones
of the Apache Storm repo: one for working (i.e. checked out to working
branch) > and one for reference/copying (i.e. checked out to
"jstorm-import").
Do you mean two storm branches(both clojure core) or one storm branch with
one JStorm-imported branch?
Tong.Wang is responsible for integration tests of JStorm, @Tong.Wang, could
you see to it if you can do the performace test?
On Wed, Nov 11, 2015 at 8:00 AM, Suresh Srinivas
wrote:
> +1 for 1.0.0 release.
>
> I also like Hackathon to encourage bigger participation. It will be fun.
>
> On 11/10/15, 3:38 PM, "Harsha" wrote:
>
> >Thanks Taylor for putting together plan on merging JStorm code.
> >Few things
> >
> >1. we should call 0.11.0 as 1.0.0 release since storm has security and
> > nimbus ha . Quite a lot of features and improvements added this is
> > going to be big release and its about time we call 1.0.0
> >
> >1.1 "align package names (e.g backtype.storm --> org.apache.storm /
> >com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin" I
> >propose we do this as next release itself instead of pushing it
> >onto another .
> >
> >"Phase 3 - Migrate Clojure --> Java"
> >
> >I would like to propose a Hackathon among storm community along with
> >jstorm. We need to come up with plan of action on what code needs to be
> >merged into storm-core. I am thinking it will help better to have
> >everyone over video chat or something to go over the code and get it
> >migrated to java.
> >
> >
> >Thanks, Harsha
> >
> >On Tue, Nov 10, 2015, at 01:32 PM, P. Taylor Goetz wrote:
> >> Based on a number of discussions regarding merging the JStorm code,
> >> I¹ve tried to distill the ideas presented and inserted some of my own.
> >> The result is below.
> >>
> >> I¹ve divided the plan into three phases, though they are not
> >> necessarily sequential ‹ obviously some tasks can take place in
> >> parallel.
> >>
> >> None of this is set in stone, just presented for discussion. Any and
> >> all comments are welcome.
> >>
> >> ---
> >>
> >> Phase 1 - Plan for 0.11.x Release
> >> 1. Determine feature set for 0.11.x and publish to wiki [1].
> >> 2. Announce feature-freeze for 0.11.x
> >> 3. Create 0.11.x branch from master (Phase 2.4 can begin.)
> >> 4. Release 0.11.0 (or whatever version # we want to use)
> >> 5. Bug fixes and subsequent releases from 0.11.x-branch
> >>
> >>
> >> Phase 2 - Prepare for Merge ("master" and "jstorm-import" branches)
> >> 1. Determine/document unique features in JStorm (e.g. classpath
> >>isolation, cgroups, etc.) and create JIRA for migrating the
> >>feature.
> >> 2. Create JIRA for migrating each clojure component (or logical group
> >>of components) to Java. Assumes tests will be ported as well.
> >> 3. Discuss/establish style guide for Java coding conventions. Consider
> >>using Oracle¹s or Google¹s Java conventions as a base ‹ they are
> >>both pretty solid.
> >> 4. align package names (e.g backtype.storm --> org.apache.storm /
> >>com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin)
> >>
> >>
> >> Phase 3 - Migrate Clojure --> Java
> >> 1. Port code/tests to Java, leveraging existing JStorm code wherever
> >>possible (core functionality only, features distinct to JStorm
> >>migrated separately).
> >> 2. Port JStorm-specific features.
> >> 3. Begin releasing preview/beta versions.
> >> 4. Code cleanup (across the board) and refactoring using established
> >>coding conventions, and leveraging PMD/Checkstyle reports for
> >>reference. (Note: good oportunity for new contributors.)
> >> 5. Release 0.12.0 (or whatever version # we want to use) and lift
> >>feature freeze.
> >>
> >>
> >> Notes: We should consider bumping up to version 1.0 sometime soon and
> >> then switching to semantic versioning [3] from then on.
> >>
> >>
> >> With the exception of package name alignment, the "jstorm-import"
> >> branch will largely be read-only throughout the process.
> >>
> >> During migration, it's probably easiest to operate with two local
> >> clones of the Apache Storm repo: one for working (i.e. checked out to
> >> working branch) and one for reference/copying (i.e. checked out to
> >>"jstorm-
> >> import").
> >>
> >> Feature-freeze probably only needs to be enforced against core
> >> functionality. Components under "external" can likely be exempt, but
> >> we should figure out a process for accepting and releasing new
> >> features during the migration.
> >>
> >> Performance testing should be continuous throughout the process. Since
> >> we don't really have ASF infrastructure for performance testing, we
> >> will need a volunteer(s) to host and run the performance tests.
> >> Performance test results can be posted to the wiki [2]. It would
> >> probably be a good idea to establish a baseline with the 0.10.0
> >> release.
> >>
> >> I¹ve attached an analysis document Sean
Re: [DISCUSS] Plan for Merging JStorm Code
Hi Taylor,
Thanks for the merge plan. I have a question about “Phase 2.2”.
Do you mean community plan to create a fresh new “java core” based on current
“clojure core” firstly, and then migrate the features from JStorm?
If so, it confused me. It is really a huge job which might require a long
developing time to make it stable, while JStorm is already a stable version.
The release planned to be release after Nov 11th has already run online stably
several month in Alibaba.
Besides this, there are many valuable internal requirements in Alibaba, the
fast evolution of JStorm is forseeable in next few months.
If the “java core” is totally fresh new, it might bring many problems for the
coming merge.
So, from the point of this view, I think it is much better and easier to
migrate the features of “clojure core” basing on JStorm for the “java core”.
Please correct me, if any misunderstanding.
Regards
Basti
发件人: P. Taylor Goetz [mailto:ptgo...@gmail.com]
发送时间: 2015年11月11日 5:32
收件人: dev@storm.apache.org
主题: [DISCUSS] Plan for Merging JStorm Code
Based on a number of discussions regarding merging the JStorm code, I’ve tried
to distill the ideas presented and inserted some of my own. The result is below.
I’ve divided the plan into three phases, though they are not necessarily
sequential — obviously some tasks can take place in parallel.
None of this is set in stone, just presented for discussion. Any and all
comments are welcome.
---
Phase 1 - Plan for 0.11.x Release
1. Determine feature set for 0.11.x and publish to wiki [1].
2. Announce feature-freeze for 0.11.x
3. Create 0.11.x branch from master (Phase 2.4 can begin.)
4. Release 0.11.0 (or whatever version # we want to use)
5. Bug fixes and subsequent releases from 0.11.x-branch
Phase 2 - Prepare for Merge ("master" and "jstorm-import" branches)
1. Determine/document unique features in JStorm (e.g. classpath isolation,
cgroups, etc.) and create JIRA for migrating the feature.
2. Create JIRA for migrating each clojure component (or logical group of
components) to Java. Assumes tests will be ported as well.
3. Discuss/establish style guide for Java coding conventions. Consider using
Oracle’s or Google’s Java conventions as a base — they are both pretty solid.
4. align package names (e.g backtype.storm --> org.apache.storm /
com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin)
Phase 3 - Migrate Clojure --> Java
1. Port code/tests to Java, leveraging existing JStorm code wherever possible
(core functionality only, features distinct to JStorm migrated separately).
2. Port JStorm-specific features.
3. Begin releasing preview/beta versions.
4. Code cleanup (across the board) and refactoring using established coding
conventions, and leveraging PMD/Checkstyle reports for reference. (Note: good
oportunity for new contributors.)
5. Release 0.12.0 (or whatever version # we want to use) and lift feature
freeze.
Notes:
We should consider bumping up to version 1.0 sometime soon and then switching
to semantic versioning [3] from then on.
With the exception of package name alignment, the "jstorm-import" branch will
largely be read-only throughout the process.
During migration, it's probably easiest to operate with two local clones of the
Apache Storm repo: one for working (i.e. checked out to working branch) and one
for reference/copying (i.e. checked out to "jstorm-import").
Feature-freeze probably only needs to be enforced against core functionality.
Components under "external" can likely be exempt, but we should figure out a
process for accepting and releasing new features during the migration.
Performance testing should be continuous throughout the process. Since we don't
really have ASF infrastructure for performance testing, we will need a
volunteer(s) to host and run the performance tests. Performance test results
can be posted to the wiki [2]. It would probably be a good idea to establish a
baseline with the 0.10.0 release.
I’ve attached an analysis document Sean Zhong put together a while back to the
JStorm merge JIRA [4]. The analysis was against the 0.9.3 release but is still
relevant and has a lot of good information.
[1] https://cwiki.apache.org/confluence/display/STORM/Release+0.11.0+Feature+Set
[2] https://cwiki.apache.org/confluence/display/STORM/Storm+Home
[3] http://semver.org
[4] https://issues.apache.org/jira/browse/STORM-717
-Taylor
Re: [DISCUSS] Plan for Merging JStorm Code
+1 for doing a 1.0 release based off of the clojure 0.11.x code. Migrating the
APIs to org.apache.storm is a big non-backwards compatible move, and a major
version bump to 2.x seems like a good move there.
+1 for the release plan
I would like the move for user facing APIs to org.apache to be one of the last
things we do. Translating clojure code into java and moving it to org.apache I
am not too concerned about.
Basti,
We have two code bases that have diverged significantly from one another in
terms of functionality. The storm code now or soon will have A Heartbeat
Server, Nimbus HA (Different Implementation), Resource Aware Scheduling, a
distributed cache like API, log searching, security, massive performance
improvements, shaded almost all of our dependencies, a REST API for
programtically accessing everything on the UI, and I am sure I am missing a few
other things. JStorm also has many changes including cgroup isolation,
restructured zookeeper layout, classpath isolation, and more too.
No matter what we do it will be a large effort to port changes from one code
base to another, and from clojure to java. I proposed this initially because
it can be broken up into incremental changes. It may take a little longer, but
we will always have a working codebase that is testable and compatible with the
current storm release, at least until we move the user facing APIs to be under
org.apache. This lets the community continue to build and test the master
branch and report problems that they find, which is incredibly valuable. I
personally don't think it will be much easier, especially if we are intent on
always maintaining compatibility with storm. - Bobby
On Wednesday, November 11, 2015 5:42 AM, 刘键(Basti Liu)
wrote:
Hi Taylor,
Thanks for the merge plan. I have a question about “Phase 2.2”.
Do you mean community plan to create a fresh new “java core” based on current
“clojure core” firstly, and then migrate the features from JStorm?
If so, it confused me. It is really a huge job which might require a long
developing time to make it stable, while JStorm is already a stable version.
The release planned to be release after Nov 11th has already run online stably
several month in Alibaba.
Besides this, there are many valuable internal requirements in Alibaba, the
fast evolution of JStorm is forseeable in next few months.
If the “java core” is totally fresh new, it might bring many problems for the
coming merge.
So, from the point of this view, I think it is much better and easier to
migrate the features of “clojure core” basing on JStorm for the “java core”.
Please correct me, if any misunderstanding.
Regards
Basti
发件人: P. Taylor Goetz [mailto:ptgo...@gmail.com]
发送时间: 2015年11月11日 5:32
收件人: dev@storm.apache.org
主题: [DISCUSS] Plan for Merging JStorm Code
Based on a number of discussions regarding merging the JStorm code, I’ve tried
to distill the ideas presented and inserted some of my own. The result is below.
I’ve divided the plan into three phases, though they are not necessarily
sequential — obviously some tasks can take place in parallel.
None of this is set in stone, just presented for discussion. Any and all
comments are welcome.
---
Phase 1 - Plan for 0.11.x Release
1. Determine feature set for 0.11.x and publish to wiki [1].
2. Announce feature-freeze for 0.11.x
3. Create 0.11.x branch from master (Phase 2.4 can begin.)
4. Release 0.11.0 (or whatever version # we want to use)
5. Bug fixes and subsequent releases from 0.11.x-branch
Phase 2 - Prepare for Merge ("master" and "jstorm-import" branches)
1. Determine/document unique features in JStorm (e.g. classpath isolation,
cgroups, etc.) and create JIRA for migrating the feature.
2. Create JIRA for migrating each clojure component (or logical group of
components) to Java. Assumes tests will be ported as well.
3. Discuss/establish style guide for Java coding conventions. Consider using
Oracle’s or Google’s Java conventions as a base — they are both pretty solid.
4. align package names (e.g backtype.storm --> org.apache.storm /
com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin)
Phase 3 - Migrate Clojure --> Java
1. Port code/tests to Java, leveraging existing JStorm code wherever possible
(core functionality only, features distinct to JStorm migrated separately).
2. Port JStorm-specific features.
3. Begin releasing preview/beta versions.
4. Code cleanup (across the board) and refactoring using established coding
conventions, and leveraging PMD/Checkstyle reports for reference. (Note: good
oportunity for new contributors.)
5. Release 0.12.0 (or whatever version # we want to use) and lift feature
freeze.
Notes:
We should consider bumping up to version 1.0 sometime soon and then switching
to semantic versioning [3] from then on.
With the exception of package name ali
Re: [DISCUSS] Plan for Merging JStorm Code
Hi Basti,
Seems like Phase 3.1 covers Phase 2.2 since Phase 3.1 is actual work for
Phase 2.2.
So in Phase 3.1, if JStorm code is fully compatible, we can just pick it.
If JStorm code is not compatible, we can port Clojure code, or implement it
based on JStorm code.
If you were talking about baseline, please read my opinion on phase 3.
All,
For phase 1, I'm also +1 for releasing 0.11.0 as 1.0.0.
Apache Storm already has various use cases, which feels me no longer beta.
For phase 3, I'm +1 for what Taylor suggests.
Precondition of my opinion is that we agree that our merged product will be
named as 'Apache Storm'.
Based on this decision, we may want to set baseline to Apache Storm, cause
users expect features based backward compatibility to newer releases.
(Brand and community is most important thing for open source product, and
our moves should avoid hurting them. Losing users lose all.)
Many features are included via 0.10.x and 0.11.0 (master branch), which
means Apache Storm and JStorm is diverged. 'Core' is no except.
Not fast but safest is, keeping Apache Storm as it is (or port Clojure code
to Java, as we start to discuss), and applying JStorm's amazing features
via pull requests, like normal contributions. I love this approach since it
is more natural.
Thanks,
Jungtaek Lim (HeartSaVioR)
2015-11-11 20:41 GMT+09:00 刘键(Basti Liu) :
> Hi Taylor,
>
>
>
> Thanks for the merge plan. I have a question about “Phase 2.2”.
>
> Do you mean community plan to create a fresh new “java core” based on
> current “clojure core” firstly, and then migrate the features from JStorm?
>
> If so, it confused me. It is really a huge job which might require a long
> developing time to make it stable, while JStorm is already a stable version.
>
> The release planned to be release after Nov 11th has already run online
> stably several month in Alibaba.
>
> Besides this, there are many valuable internal requirements in Alibaba,
> the fast evolution of JStorm is forseeable in next few months.
>
> If the “java core” is totally fresh new, it might bring many problems for
> the coming merge.
>
> So, from the point of this view, I think it is much better and easier to
> migrate the features of “clojure core” basing on JStorm for the “java core”.
>
> Please correct me, if any misunderstanding.
>
>
>
> Regards
>
> Basti
>
>
>
> 发件人: P. Taylor Goetz [mailto:ptgo...@gmail.com]
> 发送时间: 2015年11月11日 5:32
> 收件人: dev@storm.apache.org
> 主题: [DISCUSS] Plan for Merging JStorm Code
>
>
>
> Based on a number of discussions regarding merging the JStorm code, I’ve
> tried to distill the ideas presented and inserted some of my own. The
> result is below.
>
>
>
> I’ve divided the plan into three phases, though they are not necessarily
> sequential — obviously some tasks can take place in parallel.
>
>
>
> None of this is set in stone, just presented for discussion. Any and all
> comments are welcome.
>
>
>
> ---
>
>
>
> Phase 1 - Plan for 0.11.x Release
>
> 1. Determine feature set for 0.11.x and publish to wiki [1].
>
> 2. Announce feature-freeze for 0.11.x
>
> 3. Create 0.11.x branch from master (Phase 2.4 can begin.)
>
> 4. Release 0.11.0 (or whatever version # we want to use)
>
> 5. Bug fixes and subsequent releases from 0.11.x-branch
>
>
>
>
>
> Phase 2 - Prepare for Merge ("master" and "jstorm-import" branches)
>
> 1. Determine/document unique features in JStorm (e.g. classpath isolation,
> cgroups, etc.) and create JIRA for migrating the feature.
>
> 2. Create JIRA for migrating each clojure component (or logical group of
> components) to Java. Assumes tests will be ported as well.
>
> 3. Discuss/establish style guide for Java coding conventions. Consider
> using Oracle’s or Google’s Java conventions as a base — they are both
> pretty solid.
>
> 4. align package names (e.g backtype.storm --> org.apache.storm /
> com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin)
>
>
>
>
>
> Phase 3 - Migrate Clojure --> Java
>
> 1. Port code/tests to Java, leveraging existing JStorm code wherever
> possible (core functionality only, features distinct to JStorm migrated
> separately).
>
> 2. Port JStorm-specific features.
>
> 3. Begin releasing preview/beta versions.
>
> 4. Code cleanup (across the board) and refactoring using established
> coding conventions, and leveraging PMD/Checkstyle reports for reference.
> (Note: good oportunity for new contributors.)
>
> 5. Release 0.12.0 (or whatever version # we want to use) and lift feature
> freeze.
>
>
>
>
>
> Notes:
>
> We should consider bumping up to version 1.0 som
Re: [DISCUSS] Plan for Merging JStorm Code
>>> We should consider bumping up to version 1.0 sometime soon and then
>>> switching to semantic versioning [3] from then on.
> 1.0 release based off of the clojure 0.11.x code.
Will 0.11.x -> 1.0 also communicate a move from "beta" to "stable" or anything
like this, or will it be simply a version scheme change?
> It may take a little longer, but we will always have a working codebase
Agree 100% this is important.
--
Derek
- Original Message -
From: Bobby Evans
To: "dev@storm.apache.org"
Cc:
Sent: Wednesday, November 11, 2015 8:51 AM
Subject: Re: [DISCUSS] Plan for Merging JStorm Code
+1 for doing a 1.0 release based off of the clojure 0.11.x code. Migrating the
APIs to org.apache.storm is a big non-backwards compatible move, and a major
version bump to 2.x seems like a good move there.
+1 for the release plan
I would like the move for user facing APIs to org.apache to be one of the last
things we do. Translating clojure code into java and moving it to org.apache I
am not too concerned about.
Basti,
We have two code bases that have diverged significantly from one another in
terms of functionality. The storm code now or soon will have A Heartbeat
Server, Nimbus HA (Different Implementation), Resource Aware Scheduling, a
distributed cache like API, log searching, security, massive performance
improvements, shaded almost all of our dependencies, a REST API for
programtically accessing everything on the UI, and I am sure I am missing a few
other things. JStorm also has many changes including cgroup isolation,
restructured zookeeper layout, classpath isolation, and more too.
No matter what we do it will be a large effort to port changes from one code
base to another, and from clojure to java. I proposed this initially because
it can be broken up into incremental changes. It may take a little longer, but
we will always have a working codebase that is testable and compatible with the
current storm release, at least until we move the user facing APIs to be under
org.apache. This lets the community continue to build and test the master
branch and report problems that they find, which is incredibly valuable. I
personally don't think it will be much easier, especially if we are intent on
always maintaining compatibility with storm. - Bobby
On Wednesday, November 11, 2015 5:42 AM, 刘键(Basti Liu)
wrote:
Hi Taylor,
Thanks for the merge plan. I have a question about “Phase 2.2”.
Do you mean community plan to create a fresh new “java core” based on current
“clojure core” firstly, and then migrate the features from JStorm?
If so, it confused me. It is really a huge job which might require a long
developing time to make it stable, while JStorm is already a stable version.
The release planned to be release after Nov 11th has already run online stably
several month in Alibaba.
Besides this, there are many valuable internal requirements in Alibaba, the
fast evolution of JStorm is forseeable in next few months.
If the “java core” is totally fresh new, it might bring many problems for the
coming merge.
So, from the point of this view, I think it is much better and easier to
migrate the features of “clojure core” basing on JStorm for the “java core”.
Please correct me, if any misunderstanding.
Regards
Basti
发件人: P. Taylor Goetz [mailto:ptgo...@gmail.com]
发送时间: 2015年11月11日 5:32
收件人: dev@storm.apache.org
主题: [DISCUSS] Plan for Merging JStorm Code
Based on a number of discussions regarding merging the JStorm code, I’ve tried
to distill the ideas presented and inserted some of my own. The result is below.
I’ve divided the plan into three phases, though they are not necessarily
sequential — obviously some tasks can take place in parallel.
None of this is set in stone, just presented for discussion. Any and all
comments are welcome.
---
Phase 1 - Plan for 0.11.x Release
1. Determine feature set for 0.11.x and publish to wiki [1].
2. Announce feature-freeze for 0.11.x
3. Create 0.11.x branch from master (Phase 2.4 can begin.)
4. Release 0.11.0 (or whatever version # we want to use)
5. Bug fixes and subsequent releases from 0.11.x-branch
Phase 2 - Prepare for Merge ("master" and "jstorm-import" branches)
1. Determine/document unique features in JStorm (e.g. classpath isolation,
cgroups, etc.) and create JIRA for migrating the feature.
2. Create JIRA for migrating each clojure component (or logical group of
components) to Java. Assumes tests will be ported as well.
3. Discuss/establish style guide for Java coding conventions. Consider using
Oracle’s or Google’s Java conventions as a base — they are both pretty solid.
4. align package names (e.g backtype.storm --> org.apache.storm /
com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin)
Phase 3 - Migrate Clojure --> Java
1. Port code/tests to Java, lev
Re: [DISCUSS] Plan for Merging JStorm Code
I personally would like to see the move to 1.0 be when we are stable.
- Bobby
On Wednesday, November 11, 2015 9:42 AM, Derek Dagit
wrote:
>>> We should consider bumping up to version 1.0 sometime soon and then
>>> switching to semantic versioning [3] from then on.
> 1.0 release based off of the clojure 0.11.x code.
Will 0.11.x -> 1.0 also communicate a move from "beta" to "stable" or anything
like this, or will it be simply a version scheme change?
> It may take a little longer, but we will always have a working codebase
Agree 100% this is important.
--
Derek
- Original Message -
From: Bobby Evans
To: "dev@storm.apache.org"
Cc:
Sent: Wednesday, November 11, 2015 8:51 AM
Subject: Re: [DISCUSS] Plan for Merging JStorm Code
+1 for doing a 1.0 release based off of the clojure 0.11.x code. Migrating the
APIs to org.apache.storm is a big non-backwards compatible move, and a major
version bump to 2.x seems like a good move there.
+1 for the release plan
I would like the move for user facing APIs to org.apache to be one of the last
things we do. Translating clojure code into java and moving it to org.apache I
am not too concerned about.
Basti,
We have two code bases that have diverged significantly from one another in
terms of functionality. The storm code now or soon will have A Heartbeat
Server, Nimbus HA (Different Implementation), Resource Aware Scheduling, a
distributed cache like API, log searching, security, massive performance
improvements, shaded almost all of our dependencies, a REST API for
programtically accessing everything on the UI, and I am sure I am missing a few
other things. JStorm also has many changes including cgroup isolation,
restructured zookeeper layout, classpath isolation, and more too.
No matter what we do it will be a large effort to port changes from one code
base to another, and from clojure to java. I proposed this initially because
it can be broken up into incremental changes. It may take a little longer, but
we will always have a working codebase that is testable and compatible with the
current storm release, at least until we move the user facing APIs to be under
org.apache. This lets the community continue to build and test the master
branch and report problems that they find, which is incredibly valuable. I
personally don't think it will be much easier, especially if we are intent on
always maintaining compatibility with storm. - Bobby
On Wednesday, November 11, 2015 5:42 AM, 刘键(Basti Liu)
wrote:
Hi Taylor,
Thanks for the merge plan. I have a question about “Phase 2.2”.
Do you mean community plan to create a fresh new “java core” based on current
“clojure core” firstly, and then migrate the features from JStorm?
If so, it confused me. It is really a huge job which might require a long
developing time to make it stable, while JStorm is already a stable version.
The release planned to be release after Nov 11th has already run online stably
several month in Alibaba.
Besides this, there are many valuable internal requirements in Alibaba, the
fast evolution of JStorm is forseeable in next few months.
If the “java core” is totally fresh new, it might bring many problems for the
coming merge.
So, from the point of this view, I think it is much better and easier to
migrate the features of “clojure core” basing on JStorm for the “java core”.
Please correct me, if any misunderstanding.
Regards
Basti
发件人: P. Taylor Goetz [mailto:ptgo...@gmail.com]
发送时间: 2015年11月11日 5:32
收件人: dev@storm.apache.org
主题: [DISCUSS] Plan for Merging JStorm Code
Based on a number of discussions regarding merging the JStorm code, I’ve tried
to distill the ideas presented and inserted some of my own. The result is below.
I’ve divided the plan into three phases, though they are not necessarily
sequential — obviously some tasks can take place in parallel.
None of this is set in stone, just presented for discussion. Any and all
comments are welcome.
---
Phase 1 - Plan for 0.11.x Release
1. Determine feature set for 0.11.x and publish to wiki [1].
2. Announce feature-freeze for 0.11.x
3. Create 0.11.x branch from master (Phase 2.4 can begin.)
4. Release 0.11.0 (or whatever version # we want to use)
5. Bug fixes and subsequent releases from 0.11.x-branch
Phase 2 - Prepare for Merge ("master" and "jstorm-import" branches)
1. Determine/document unique features in JStorm (e.g. classpath isolation,
cgroups, etc.) and create JIRA for migrating the feature.
2. Create JIRA for migrating each clojure component (or logical group of
components) to Java. Assumes tests will be ported as well.
3. Discuss/establish style guide for Java coding conventions. Consider using
Oracle’s or Google’s Java conventions as a base — they are both pretty solid.
4. align package names (e.g backtype.storm -
Re: [DISCUSS] Plan for Merging JStorm Code
> On Nov 11, 2015, at 2:04 AM, Cody Innowhere wrote: > >> During migration, it's probably easiest to operate with two local clones > of the Apache Storm repo: one for working (i.e. checked out to working > branch) > and one for reference/copying (i.e. checked out to > "jstorm-import"). > Do you mean two storm branches(both clojure core) or one storm branch with > one JStorm-imported branch? That paragraph was mainly me thinking out loud, and probably could have been left out. I meant that as a developer working on merging two divergent branches branches, it might be easiest to have two local copies of the repo to do comparisons, diffs, etc., rather than working in one repo and having to constantly switch between branches. That’s just the way I would likely do it. Ultimately it’s up to each individual developer’s personal preference. -Taylor signature.asc Description: Message signed with OpenPGP using GPGMail
答复: [DISCUSS] Plan for Merging JStorm Code
Hi Taylor, I'd like to help too, could you add me in? my id is: hustfxj -邮件原件- 发件人: P. Taylor Goetz [mailto:ptgo...@gmail.com] 发送时间: 2015年11月24日 23:54 收件人: dev@storm.apache.org 主题: Re: [DISCUSS] Plan for Merging JStorm Code Hi Cody, I wasn’t able to find your username. Are you sure you have an account on cwiki.apache.org? -Taylor > On Nov 22, 2015, at 8:46 AM, Cody Innowhere wrote: > > Hi Taylor, > I'd like to help too, could you add me in? my id is: Cody > > On Sat, Nov 21, 2015 at 11:51 AM, 刘键(Basti Liu) > > wrote: > >> Hi Taylor， >> >> Sorry for the late response. >> I'd like to help on this. Could you please help to give me the permission? >> Thanks. >> UserName: basti.lj >> >> Regards >> Basti >> >> -Original Message- >> From: P. Taylor Goetz [mailto:ptgo...@gmail.com] >> Sent: Thursday, November 19, 2015 6:24 AM >> To: dev@storm.apache.org; Bobby Evans >> Subject: Re: [DISCUSS] Plan for Merging JStorm Code >> >> All I have at this point is a placeholder wiki entry [1], and a lot >> of local notes that likely would only make sense to me. >> >> Let me know your wiki username and I’ll give you permissions. The >> same goes for anyone else who wants to help. >> >> -Taylor >> >> [1] >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=6132 >> 8109 >> >>> On Nov 18, 2015, at 2:08 PM, Bobby Evans >>> >> wrote: >>> >>> Taylor and others I was hoping to get started filing JIRA and >>> planning on how we are going to do the java migration + JStorm >>> merger. Is anyone else starting to do this? If not would anyone >>> object to me starting on it? - Bobby >>> >>> >>> On Thursday, November 12, 2015 12:04 PM, P. Taylor Goetz < >> ptgo...@gmail.com> wrote: >>> >>> >>> Thanks for putting this together Basti, that comparison helps a lot. >>> >>> And thanks Bobby for converting it into markdown. I was going to >>> just >> attach the spreadsheet to JIRA, but markdown is a much better solution. >>> >>> -Taylor >>> >>>> On Nov 12, 2015, at 12:03 PM, Bobby Evans >>>> >> wrote: >>>> >>>> I translated the excel spreadsheet into a markdown file and put up >>>> a >> pull request for it. >>>> https://github.com/apache/storm/pull/877 >>>> I did a few edits to it to make it work with Markdown, and to add >>>> in a >> few of my own comments. I also put in a field for JIRAs to be able >> to track the migration. >>>> Overall I think your evaluation was very good. We have a fair >>>> amount >> of work ahead of us to decide what version of various features we >> want to go forward with. >>>> - Bobby >>>> >>>> >>>>On Thursday, November 12, 2015 9:37 AM, 刘键(Basti Liu) < >> basti...@alibaba-inc.com> wrote: >>>> >>>> >>>> Hi Bobby & Jungtaek, >>>> >>>> Thanks for your replay. >>>> I totally agree that compatibility is the most important thing. >> Actually, JStorm has been compatible with the user API of Storm. >>>> As you mentioned below, we indeed still have some features >>>> different >> between Storm and JStorm. I have tried to list them (minor update or >> improvements are not included). >>>> Please refer to attachment for details. If any missing, please help >>>> to point out. (The current working features are probably missing >>>> here.) >> Just have a look at these differences. For the missing features in >> JStorm, I did not see any obstacle which will block the merge to JStorm. >>>> For the features which has different solution between Storm and >>>> JStorm, >> we can evaluate the solution one by one to decision which one is >> appropriate. >>>> After the finalization of evaluation, I think JStorm team can take >>>> the >> merging job and publish a stable release in 2 months. >>>> But anyway, the detailed implementation for these features with >> different solution is transparent to user. So, from user's point of >> view, there is not any compatibility problem. >>>> >>>> Besides compatibility, by our experience, stability is also >>>> important >> and is not an easy job. 4 people in JStorm team took almost one year
答复：[DISCUSS] Plan for Merging JStorm Code
sure, I can do the
performance test, do we have any
performance test cases
?--发件人：Cody
Innowhere 发送时间：2015年11月11日(星期三) 15:04收件人：dev
主 题：Re: [DISCUSS] Plan for Merging JStorm Code
Thanks Taylor.
The plan looks good.
> During migration, it's probably easiest to operate with two local clones
of the Apache Storm repo: one for working (i.e. checked out to working
branch) > and one for reference/copying (i.e. checked out to
"jstorm-import").
Do you mean two storm branches(both clojure core) or one storm branch with
one JStorm-imported branch?
Tong.Wang is responsible for integration tests of JStorm, @Tong.Wang, could
you see to it if you can do the performace test?
On Wed, Nov 11, 2015 at 8:00 AM, Suresh Srinivas
wrote:
> +1 for 1.0.0 release.
>
> I also like Hackathon to encourage bigger participation. It will be fun.
>
> On 11/10/15, 3:38 PM, "Harsha" wrote:
>
> >Thanks Taylor for putting together plan on merging JStorm code.
> >Few things
> >
> >1. we should call 0.11.0 as 1.0.0 release since storm has security and
> > nimbus ha . Quite a lot of features and improvements added this is
> > going to be big release and its about time we call 1.0.0
> >
> >1.1 "align package names (e.g backtype.storm --> org.apache.storm /
> >com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin" I
> >propose we do this as next release itself instead of pushing it
> >onto another .
> >
> >"Phase 3 - Migrate Clojure --> Java"
> >
> >I would like to propose a Hackathon among storm community along with
> >jstorm. We need to come up with plan of action on what code needs to be
> >merged into storm-core. I am thinking it will help better to have
> >everyone over video chat or something to go over the code and get it
> >migrated to java.
> >
> >
> >Thanks, Harsha
> >
> >On Tue, Nov 10, 2015, at 01:32 PM, P. Taylor Goetz wrote:
> >> Based on a number of discussions regarding merging the JStorm code,
> >> I¹ve tried to distill the ideas presented and inserted some of my own.
> >> The result is below.
> >>
> >> I¹ve divided the plan into three phases, though they are not
> >> necessarily sequential ‹ obviously some tasks can take place in
> >> parallel.
> >>
> >> None of this is set in stone, just presented for discussion. Any and
> >> all comments are welcome.
> >>
> >> ---
> >>
> >> Phase 1 - Plan for 0.11.x Release
> >> 1. Determine feature set for 0.11.x and publish to wiki [1].
> >> 2. Announce feature-freeze for 0.11.x
> >> 3. Create 0.11.x branch from master (Phase 2.4 can begin.)
> >> 4. Release 0.11.0 (or whatever version # we want to use)
> >> 5. Bug fixes and subsequent releases from 0.11.x-branch
> >>
> >>
> >> Phase 2 - Prepare for Merge ("master" and "jstorm-import" branches)
> >> 1. Determine/document unique features in JStorm (e.g. classpath
> >>isolation, cgroups, etc.) and create JIRA for migrating the
> >>feature.
> >> 2. Create JIRA for migrating each clojure component (or logical group
> >>of components) to Java. Assumes tests will be ported as well.
> >> 3. Discuss/establish style guide for Java coding conventions. Consider
> >>using Oracle¹s or Google¹s Java conventions as a base ‹ they are
> >>both pretty solid.
> >> 4. align package names (e.g backtype.storm --> org.apache.storm /
> >>com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin)
> >>
> >>
> >> Phase 3 - Migrate Clojure --> Java
> >> 1. Port code/tests to Java, leveraging existing JStorm code wherever
> >>possible (core functionality only, features distinct to JStorm
> >>migrated separately).
> >> 2. Port JStorm-specific features.
> >> 3. Begin releasing preview/beta versions.
> >> 4. Code cleanup (across the board) and refactoring using established
> >>coding conventions, and leveraging PMD/Checkstyle reports for
> >>reference. (Note: good oportunity for new contributors.)
> >> 5. Release 0.12.0 (or whatever version # we want to use) and lift
> >>feature freeze.
> >>
> >>
> >> Notes: We should consider bumping up to version 1.0 sometime soon and
> >> then switching to semantic versioning [3] from then on.
> >>
> >>
> >> With the exception of package name alignment, the "jstorm-import"
>
