Re: Road Forward
On Thu, Apr 17, 2008 at 2:23 PM, Hontvari Jozsef <[EMAIL PROTECTED]> wrote: > > > Robert Burrell Donkin írta: > > > > 1. i know that quite a lot of the IMAP/mailbox stuff is junk (both > > mine code and the old stuff). IMAP/mailbox is most of trunk. so yes, a > > substantial quantity of code in trunk is junk. a more important issue > > is that it's difficult to gain consensus on the production quality of > > the remaining code on trunk or make rational judgments about what's > > basically sound but needs testing in production. i've only reviewed a > > small proportion and some looks ok, some looks questionable but i'm > > not enough of an expert to judge. > > > > 2. i want people to innovate on trunk rather than on their own > > branches. innovating on branches is harmful to the community since > > it's much harder to collaborate and entropy makes it difficult to > > merge changes into trunk. we need to allow new ideas and we need to > > allow it on trunk. IMHO modularisation is an inevitable consequence of > > this approach. noel prefers to see this process as allowing anyone to > > dump junk in trunk and i have no probably about that language but > > that's not the way i see things. > > > > - robert > > > > > First of all, I don't want to take your time, so this will be my last post > in this subject. post as much as you want: feedback's good > I feel the workflow above would require development capacity which is > rarely available in the James project. > Somebody must have the authority and long term commitment to decide what > can be used from the playing ground trunk and extract those and create a > production quality version himself. Unfortunately, because it is a playing > ground, nobody trusts in trunk. By consequence it is not used on many > production like servers. Therefore nobody has reliable information about > what is useable in trunk. Moreover, even committers don't develop against > trunk, because they cannot use the result on their production machines. The > function of this developer is more like a paying job and not like the ad-hoc > volunteer work, which is avalible for James. one of the community issues facing JAMES is that almost every developer has different interests. production means fit to run live on the internet. some developers run production servers, others are more interested in intranet ready solutions. JAMES serves on my intranet. trunk is fine for me. > Regarding modularized code, I had a glance at the trunk, it was nice and > easy to understand. But if you have 20 smaller playing grounds instead of > one larger playing ground that doesn't really help on the trust problem > above. Actually, if the modules indeed start to be developed independently - > but they will never be completely independent - then you get a new, > difficult task: coordinate the release of 20 modules at the same time. i intentionally propose not to coordinate component releases: let components be released when they are ready. release early, release often. > It seems that while the goal was to avoid innovation in long-term branches, > the current approach lead to the exactly opposite result: everybody has his > own long-term custom build. I haven't seen any change which make the result > different next time. long terms branches are a community issue for me. sharing our code and ideas more intimately will help to bring the community together. i'm not bothered about code people choose to use for their main mail server. - robert - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Road Forward
Robert Burrell Donkin írta: 1. i know that quite a lot of the IMAP/mailbox stuff is junk (both mine code and the old stuff). IMAP/mailbox is most of trunk. so yes, a substantial quantity of code in trunk is junk. a more important issue is that it's difficult to gain consensus on the production quality of the remaining code on trunk or make rational judgments about what's basically sound but needs testing in production. i've only reviewed a small proportion and some looks ok, some looks questionable but i'm not enough of an expert to judge. 2. i want people to innovate on trunk rather than on their own branches. innovating on branches is harmful to the community since it's much harder to collaborate and entropy makes it difficult to merge changes into trunk. we need to allow new ideas and we need to allow it on trunk. IMHO modularisation is an inevitable consequence of this approach. noel prefers to see this process as allowing anyone to dump junk in trunk and i have no probably about that language but that's not the way i see things. - robert First of all, I don't want to take your time, so this will be my last post in this subject. I feel the workflow above would require development capacity which is rarely available in the James project. Somebody must have the authority and long term commitment to decide what can be used from the playing ground trunk and extract those and create a production quality version himself. Unfortunately, because it is a playing ground, nobody trusts in trunk. By consequence it is not used on many production like servers. Therefore nobody has reliable information about what is useable in trunk. Moreover, even committers don't develop against trunk, because they cannot use the result on their production machines. The function of this developer is more like a paying job and not like the ad-hoc volunteer work, which is avalible for James. Regarding modularized code, I had a glance at the trunk, it was nice and easy to understand. But if you have 20 smaller playing grounds instead of one larger playing ground that doesn't really help on the trust problem above. Actually, if the modules indeed start to be developed independently - but they will never be completely independent - then you get a new, difficult task: coordinate the release of 20 modules at the same time. It seems that while the goal was to avoid innovation in long-term branches, the current approach lead to the exactly opposite result: everybody has his own long-term custom build. I haven't seen any change which make the result different next time. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Road Forward
On Wed, Apr 16, 2008 at 9:50 PM, Robert Burrell Donkin <[EMAIL PROTECTED]> wrote: > the major issue is that we don't collectively understand the quality > of the code in trunk nor how it differs from 2.4.x +1 I think, and hope, that that is what Noel, in his inimitable style, is pushing us towards doing. > > modularisation has some significant side benefits (which i'll outline > in another mail) but it also allows us to release code a chunk at a > time: we don't need to review all of trunk, just each bit in turn that > we want to release. +1 ... > manage the progression of code from unstable (trunk) to stable > (production) should be the management point +1 Agreed, if we look at sucessful projects with small numbers of commiters the role of the comitter is one of QA, patches are applied freely (CTR) but its the role of the committers to manage their progress thereafter as gatekeepers, not as coders or innovaters. > > for example, the loosely coupled mailets (whether cryto or standard) > should be extracted into separate products within a week or two. it > would not be unreasonable to diff the mailets with production and then > look to release these products soon. once we're happy with the > quality, we delete the duplicates from production and trunk and depend > on the library releases. +1 > i advocate managing QA not in the transition from branches to trunk > but from trunk to production +1 Exactly what I was trying to say :-) d. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Road Forward
On Wed, Apr 16, 2008 at 8:23 PM, Danny Angus <[EMAIL PROTECTED]> wrote: > Glad you guys seem to have had some productive chat @apachecon. Sorry > I had to miss it this year, its the easter holidays and I went to > Aviemore with the family. Loads of snow on the ski slopes, which is > unusual for the time of year. > http://www.cairngormmountain.org.uk/web-cam i was snowed on on the way to apachecon :-) > > >some looks ok, some looks questionable but i'm > > > not enough of an expert to judge. > > Oh I'm sure you're probably being modest. not really: the stuff that noel was worried about is something which requires in-depth knowledge of the specifications > I think that much of the > diff between 2.x branch and trunk suffers from not being tested > enough, and not having been subjected to the level of scrutiny which > it might have received in more active projects. > Which means my opinion is that the trunk isn't fit to release, but it > may contain code worth "working up" into production quality. the major issue is that we don't collectively understand the quality of the code in trunk nor how it differs from 2.4.x modularisation has some significant side benefits (which i'll outline in another mail) but it also allows us to release code a chunk at a time: we don't need to review all of trunk, just each bit in turn that we want to release. > > > 2. i want people to innovate on trunk rather than on their own > > > branches. > > This is a worthwhile goal, but has resulted in the lack of quality > assurance which blights the trunk. > We need to make a clear statement about the purpose and nature of the trunk. > We also need to work out how we avoid a situation where the delta > between branch and trunk continues to grow until they are too > different to manage. Perhaps we are already at that point. whether we've reached that point already or are just very close, i advocate abandoning the attempt to manage the delta: just accept that trunk is the classic unstable branch where committers are free to try things out there on a modular basis manage the progression of code from unstable (trunk) to stable (production) should be the management point for example, the loosely coupled mailets (whether cryto or standard) should be extracted into separate products within a week or two. it would not be unreasonable to diff the mailets with production and then look to release these products soon. once we're happy with the quality, we delete the duplicates from production and trunk and depend on the library releases. > > > IMHO modularisation is an inevitable consequence of > > > this approach. > > I've long said that modularisation is absolutely, fundamentally, > totally and utterly essential to the survival of the project. James > server is too complex to get your head round all of it in one go, > which dissuades people from casual participation, or put another way > James requires a significant commitment from contributors just to > understand what to change. The shame is that I believe that we loose > valuable contributions because of this overhead. +1 > > > noel prefers to see this process as allowing anyone to > > > dump junk in trunk and i have no probably about that language but > > > that's not the way i see things. > > Nor I, but Noel, as you say, is prone to talking in soundbites and > likes to make a drama out of a crisis. > I have no problem with people throwing stones into the pond to see > what floats to the surface, but I think the truth here is that we need > to establish an environment where there is a low cost for people to > innovate James, and the higher cost is associated with taking those > ideas into production. +1 > We lack any notion of degrees of quality assurance, and we lack a > lifecycle which allows changes to be promoted up the quality scale. > I think we need somewhere to dump junk, a way to pan the junk for > nuggets, and a clear path to "release" for chosen changes. +1 i advocate managing QA not in the transition from branches to trunk but from trunk to production - robert - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Road Forward
Glad you guys seem to have had some productive chat @apachecon. Sorry I had to miss it this year, its the easter holidays and I went to Aviemore with the family. Loads of snow on the ski slopes, which is unusual for the time of year. http://www.cairngormmountain.org.uk/web-cam However to get to the point... my comments are below: > >some looks ok, some looks questionable but i'm > > not enough of an expert to judge. Oh I'm sure you're probably being modest. I think that much of the diff between 2.x branch and trunk suffers from not being tested enough, and not having been subjected to the level of scrutiny which it might have received in more active projects. Which means my opinion is that the trunk isn't fit to release, but it may contain code worth "working up" into production quality. > > 2. i want people to innovate on trunk rather than on their own > > branches. This is a worthwhile goal, but has resulted in the lack of quality assurance which blights the trunk. We need to make a clear statement about the purpose and nature of the trunk. We also need to work out how we avoid a situation where the delta between branch and trunk continues to grow until they are too different to manage. Perhaps we are already at that point. > > IMHO modularisation is an inevitable consequence of > > this approach. I've long said that modularisation is absolutely, fundamentally, totally and utterly essential to the survival of the project. James server is too complex to get your head round all of it in one go, which dissuades people from casual participation, or put another way James requires a significant commitment from contributors just to understand what to change. The shame is that I believe that we loose valuable contributions because of this overhead. > > noel prefers to see this process as allowing anyone to > > dump junk in trunk and i have no probably about that language but > > that's not the way i see things. Nor I, but Noel, as you say, is prone to talking in soundbites and likes to make a drama out of a crisis. I have no problem with people throwing stones into the pond to see what floats to the surface, but I think the truth here is that we need to establish an environment where there is a low cost for people to innovate James, and the higher cost is associated with taking those ideas into production. We lack any notion of degrees of quality assurance, and we lack a lifecycle which allows changes to be promoted up the quality scale. I think we need somewhere to dump junk, a way to pan the junk for nuggets, and a clear path to "release" for chosen changes. d. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Road Forward
Am Mittwoch, den 16.04.2008, 14:40 +0100 schrieb Robert Burrell Donkin: > On Wed, Apr 16, 2008 at 2:26 PM, Stefano Bagnara <[EMAIL PROTECTED]> wrote: > > Noel J. Bergman ha scritto: > > > > > > > Hontvari Jozsef wrote: > > > > > > > > > > The trunk receives all changes. It must always be in a useable state. > > > > > > > > > > Trying convincing people of that. :-) Robert, for example, explicitly > > > considers that to be the wrong approach, and indicates that he considers a > > > substantial amount of trunk to be junk. His thought is to let everyone > > keep > > > dumping code into trunk until it is so bad that we are forced to make more > > > and more modular packaging. > > > > > > > Just to better understand PMC member opinions: Robert, can you confirm > > that: > > 1) You consider a substatial amount of trunk is junk > > 2) Your thought is to let everyone keep dumping code into trunk until it is > > so bad that we are forced to make more and more modular packaging? > > noel prefers to paraphrase complex arguments into simple soundbites :-) > > 1. i know that quite a lot of the IMAP/mailbox stuff is junk (both > mine code and the old stuff). IMAP/mailbox is most of trunk. so yes, a > substantial quantity of code in trunk is junk. a more important issue > is that it's difficult to gain consensus on the production quality of > the remaining code on trunk or make rational judgments about what's > basically sound but needs testing in production. i've only reviewed a > small proportion and some looks ok, some looks questionable but i'm > not enough of an expert to judge. > > 2. i want people to innovate on trunk rather than on their own > branches. innovating on branches is harmful to the community since > it's much harder to collaborate and entropy makes it difficult to > merge changes into trunk. we need to allow new ideas and we need to > allow it on trunk. IMHO modularisation is an inevitable consequence of > this approach. noel prefers to see this process as allowing anyone to > dump junk in trunk and i have no probably about that language but > that's not the way i see things. > > - robert Makes all sense for me.. Cheers, Norman - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Road Forward
On Wed, Apr 16, 2008 at 2:26 PM, Stefano Bagnara <[EMAIL PROTECTED]> wrote: > Noel J. Bergman ha scritto: > > > > Hontvari Jozsef wrote: > > > > > > > The trunk receives all changes. It must always be in a useable state. > > > > > > > Trying convincing people of that. :-) Robert, for example, explicitly > > considers that to be the wrong approach, and indicates that he considers a > > substantial amount of trunk to be junk. His thought is to let everyone > keep > > dumping code into trunk until it is so bad that we are forced to make more > > and more modular packaging. > > > > Just to better understand PMC member opinions: Robert, can you confirm > that: > 1) You consider a substatial amount of trunk is junk > 2) Your thought is to let everyone keep dumping code into trunk until it is > so bad that we are forced to make more and more modular packaging? noel prefers to paraphrase complex arguments into simple soundbites :-) 1. i know that quite a lot of the IMAP/mailbox stuff is junk (both mine code and the old stuff). IMAP/mailbox is most of trunk. so yes, a substantial quantity of code in trunk is junk. a more important issue is that it's difficult to gain consensus on the production quality of the remaining code on trunk or make rational judgments about what's basically sound but needs testing in production. i've only reviewed a small proportion and some looks ok, some looks questionable but i'm not enough of an expert to judge. 2. i want people to innovate on trunk rather than on their own branches. innovating on branches is harmful to the community since it's much harder to collaborate and entropy makes it difficult to merge changes into trunk. we need to allow new ideas and we need to allow it on trunk. IMHO modularisation is an inevitable consequence of this approach. noel prefers to see this process as allowing anyone to dump junk in trunk and i have no probably about that language but that's not the way i see things. - robert - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Road Forward
Stefano Bagnara ha scritto: Noel J. Bergman ha scritto: Hontvari Jozsef wrote: The trunk receives all changes. It must always be in a useable state. Trying convincing people of that. :-) Robert, for example, explicitly considers that to be the wrong approach, and indicates that he considers a substantial amount of trunk to be junk. His thought is to let everyone keep dumping code into trunk until it is so bad that we are forced to make more and more modular packaging. Just to better understand PMC member opinions: Robert, can you confirm that: 1) You consider a substatial amount of trunk is junk 2) Your thought is to let everyone keep dumping code into trunk until it is so bad that we are forced to make more and more modular packaging? Sorry, I received your answers after posting this one. I'm fine with your explanation. Stefano - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Road Forward
Noel J. Bergman ha scritto: Hontvari Jozsef wrote: The trunk receives all changes. It must always be in a useable state. Trying convincing people of that. :-) Robert, for example, explicitly considers that to be the wrong approach, and indicates that he considers a substantial amount of trunk to be junk. His thought is to let everyone keep dumping code into trunk until it is so bad that we are forced to make more and more modular packaging. Just to better understand PMC member opinions: Robert, can you confirm that: 1) You consider a substatial amount of trunk is junk 2) Your thought is to let everyone keep dumping code into trunk until it is so bad that we are forced to make more and more modular packaging? Thank you, Stefano - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Road Forward
On Tue, Apr 15, 2008 at 11:17 PM, Hontvari Jozsef <[EMAIL PROTECTED]> wrote: > Moreover James is a mature product, nobody is forced to upgrade. > I am using the same custom version for about 3 years if not more. > > If there are enough features or a few months elapsed, release a beta > version from the trunk. The beta can be put into a new branch if a serious > bug is found. Leave it in beta for a few months, and if nobody reports > serious bugs declare it stable. The version number should reflect the actual > backward compatibility of the beta, there is no need to plan it in advance. one of the problems now is that *everyone* runs their own custom version of JAMES. there are quite a number of new features in trunk but IMO it's going to be easier to delivery them as pluggable components rather than a big-bang. for example, there is really no reason why IMAP couldn't be added to 2.4.x provided that it was a plugin and that users were asked to accept that the handling might not be absolutely compliant (there are some subtle differences in the socket transport code). same goes for the mailets: there are quite a lot of cool new features which could be delivered just as library upgrades rather than full blow server releases. - robert - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Road Forward
On Wed, Apr 16, 2008 at 12:36 PM, Noel J. Bergman <[EMAIL PROTECTED]> wrote: > Hontvari Jozsef wrote: > > > The trunk receives all changes. It must always be in a useable state. > > Trying convincing people of that. :-) Robert, for example, explicitly > considers that to be the wrong approach, and indicates that he considers a > substantial amount of trunk to be junk. His thought is to let everyone keep > dumping code into trunk until it is so bad that we are forced to make more > and more modular packaging. that's overstating my position (other than code i wrote or reviewed myself) i don't consider that trunk has a substantial amount of junk: just that trunk has a substantial quantity of interesting code whose production qualities are unknown. i want people to be able to try interesting and experimental ideas out on trunk without arguing. there needs to be space where we can all collaborate on innovative ideas without endangering the mature JAMES server. > I agree with the goal of modular releases, but want something that we can > work with today. So he and I agree that I'll start with something that we > know works. And as each modular piece comes out of trunk, we'll see if we > can fit it into the production branch. Slowly but surely we'll get to where > we want to be, but always try to maintain a production-ready branch, which > is what you are calling trunk. it's much easier to extract code from trunk, review then release in a modular fashion than to freeze and review trunk. anyone who really needs the feature can download and install. everyone else can wait until it's stable. > > Compatibility doesn't matter at all, you don't release more then once a > > year anyway now. > > That's a bug, not a feature. And if I don't do this, it will be another two > years before anything happens. +1 this is consensus (it's just about the only thing everyone agrees on) and i expect at least one component release a month for the next year - robert - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Road Forward
On Tue, Apr 15, 2008 at 11:17 PM, Hontvari Jozsef <[EMAIL PROTECTED]> wrote: > Considering the low frequency of releases and the small number of new > features, there are plenty of new features on trunk - robert - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Road Forward
Hontvari Jozsef wrote: > The trunk receives all changes. It must always be in a useable state. Trying convincing people of that. :-) Robert, for example, explicitly considers that to be the wrong approach, and indicates that he considers a substantial amount of trunk to be junk. His thought is to let everyone keep dumping code into trunk until it is so bad that we are forced to make more and more modular packaging. I agree with the goal of modular releases, but want something that we can work with today. So he and I agree that I'll start with something that we know works. And as each modular piece comes out of trunk, we'll see if we can fit it into the production branch. Slowly but surely we'll get to where we want to be, but always try to maintain a production-ready branch, which is what you are calling trunk. > Compatibility doesn't matter at all, you don't release more then once a > year anyway now. That's a bug, not a feature. And if I don't do this, it will be another two years before anything happens. > Moreover James is a mature product, nobody is forced to upgrade. If you are running a public facing mail server using JAMES, that's just not the case. The spammers have raised the noise level so high that, for example, fast-fail is really mandatory, not optional. Improvements in fast-fail are important. My private build gets by; the public JAMES build would be seriously hindered. I want that to change. --- Noel - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Road Forward
Considering the low frequency of releases and the small number of new features, the version management structure of James seems to be an overkill. (Or maybe I just don't understand the system.) Although I don't think this is new for anybody here is the simplest procedure: The trunk receives all changes. It must always be in a useable state. Compatibility doesn't matter at all, you don't release more then once a year anyway now. Moreover James is a mature product, nobody is forced to upgrade. I am using the same custom version for about 3 years if not more. If there are enough features or a few months elapsed, release a beta version from the trunk. The beta can be put into a new branch if a serious bug is found. Leave it in beta for a few months, and if nobody reports serious bugs declare it stable. The version number should reflect the actual backward compatibility of the beta, there is no need to plan it in advance. Noel J. Bergman írta: After discussion of various technical and non-technical things at ApacheCon by Robert, Bernd, Norman and myself, here's a road forward. When I get a moment (right now, I am in the planning meeting for ApacheCon US New Orleans), I am going to branch from v2.3.1. I am going to re-review the diff between v2.3.0 and v2.3.1, and welcome others to do the same. It is not that I don't want some of the code from trunk, but this is just part of the process. The production branch is not necessarily intended to be config.xml and DB-format compatible with v2.x. The purpose is to start from a production stable codebase, and maintain a production stable code base, while incrementally incorporating stable parts pulled from and shared with trunk, or developed in a temporary sandbox. As another part of the process, Robert is going to work on refactoring and splitting from trunk. To illustrate the overall process, one of the things that Robert is going to do is pull out all of the portable Matchers and Mailets into a separate releasable. Once that's clean, we compare them against the same set in the production branch, and after approving, we delete them from the production branch, and reference the new package. Trunk will also share that same releasable, so in that manner, we will effectively "merge" production and trunk incrementally over time. --- Noel - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Road Forward
After discussion of various technical and non-technical things at ApacheCon by Robert, Bernd, Norman and myself, here's a road forward. When I get a moment (right now, I am in the planning meeting for ApacheCon US New Orleans), I am going to branch from v2.3.1. I am going to re-review the diff between v2.3.0 and v2.3.1, and welcome others to do the same. It is not that I don't want some of the code from trunk, but this is just part of the process. The production branch is not necessarily intended to be config.xml and DB-format compatible with v2.x. The purpose is to start from a production stable codebase, and maintain a production stable code base, while incrementally incorporating stable parts pulled from and shared with trunk, or developed in a temporary sandbox. As another part of the process, Robert is going to work on refactoring and splitting from trunk. To illustrate the overall process, one of the things that Robert is going to do is pull out all of the portable Matchers and Mailets into a separate releasable. Once that's clean, we compare them against the same set in the production branch, and after approving, we delete them from the production branch, and reference the new package. Trunk will also share that same releasable, so in that manner, we will effectively "merge" production and trunk incrementally over time. --- Noel - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]