(Long post, please bear with me and please read!) Now that we have the release done (I'm working through the publication process now), I want to start the process of thinking about how we can improve the release process. As I see it, building the artifacts and checking the legal items are now almost completely automated and testable at earlier stages in the game.
We have kept saying we want to release more often, but we have never defined actionable steps with which we can get there. Goals without actionable steps are useless. So, with that in mind, I'd like to brainstorm on how we can improve things a bit more. Several us acted as RM this time around, so I think we have some common, shared knowledge to take advantage of this time as opposed to in the past where one person mostly just did the release in the background and then we all voted. So, let's start with what we have right: 1. The Ant process for building a release candidate for both Lucene and Solr is almost identical now and fairly straightforward. 2. I think the feature freeze is a good thing, although it is a bit too long perhaps. 3. Pretty good documentation on the steps involved to branch, etc. 4. The new license validation stuff is a start for enforcing licensing up front more effectively. What else can we validate up front in terms of packaging? 5. We have an awesome test infrastructure now. I think it is safe to say that this version of Lucene is easily the most tested version we have ever shipped. Things I see that can be improved, and these are only suggestions: 1. We need to define the Minimum Effective Dose (MED - http://gizmodo.com/#!5709902/4+hour-body-the-principle-of-the-minimum-effective-dose) for producing a quality release. Nothing more, nothing less. I think one of our biggest problems is we don't know when we are done. It's this loosey-goosey "we all agree" notion, but that's silly. It's software, we should be able to test almost all of the artifacts for certain attributes and then release when they pass. If we get something wrong, put in a test for it in the next release. The old saying about perfect being the enemy of great applies here. In other words, we don't have well defined things that we all are looking for when vetting a release candidate, other than what the ASF requires. Look at the last few vote threads or for any of the previous threads. It's obvious that we have a large variety of people doing a large variety of things when it comes to testing the candidates. For instance, I do the following: a. check sigs., md5 hashes, etc. b. run the demos, c. run the Solr example and index some content, d. check over the LICENSE, NOTICE, CHANGES files e. Check the overall packaging, etc. is reasonable f. I run them through my training code Others clearly do many other things. Many of you have your own benchmark tests you run, others read over every last bit of documentation others still put the RC into their own application and test it. All of this is good, but the problem is it is not _shared_ until the actual RC is up and it is not repeatable (not that all of it can be). If you have benchmark code/tests that your run on an RC that doesn't involve proprietary code, why isn't it donated to the project so that we can all use it? That way we don't have to wait until your -1 at the 11th hour to realize the RC is not good. I personally don't care whether it's python or perl or whatever. Something that works is better than nothing. For instance, right now some of the committers have an Apache Extras project going for benchmarking. Can we get this running on ASF resources on a regular basis? If it's a computing resource issue, let's go to Infrastructure and ask for resources. Infrastructure has repeatedly said that if a project needs resources to put together a proposal of what you want. I bet we could get budget to spin up an EC2 instance once a week, run those long running tests (Test2B and other benchmarks) and then report back. All of that can be automated. Also, please think hard about whether the things you test can be automated and built into our test suite or at least run nightly or something on Jenkins and then donating them. I know reading documentation can't, but what else? For instance, could we auto-generate the file formats documentation? 2. We should be running and testing the release packaging process more regularly. 3. I had an epiphany this release and it came via Hoss on a non release related issue where, likely unbeknownst to him, he called me out for not being focused on the release (https://issues.apache.org/jira/browse/SOLR-2366?focusedCommentId=12996154&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12996154) I have often been a passive observer when it comes to releasing other than testing the RC (either that or I am the RM, it seems) It's easy to sit and complain about there not being more frequent releases, but what are you doing about it? Why are we working on trunk when we are trying to get a release out? Obviously you are all free to do as you wish, but then don't complain when there aren't more frequent releases. Yes, I am calling you out just like Hoss called me out. If you aren't sure how you can help, ask. Assuming we can better document #1 above, helping should be a lot easier in the future. Now, obviously, not everyone can be involved in the actual building of the RC, but that doesn't mean you can't go build it yourself and start testing on it before the Vote comes out. And it also doesn't mean you can't take a hard look at the process and see if there aren't better ways to do it. I also think if we all focus more and coordinate our testing by defining test plans up front, we could get releases out in far less time. 4. Are those last minute changes really blockers? We are all perfectionists, I know. Our guilt gets to us if we happen to know there is something "broken" and we don't fix it. I think we should add a "Known Issues" section to CHANGES.txt and we should be a bit tougher about keeping out last minute changes. I was also Guilty as Charged in this case on the highlighter issue that Mark and I were working on. Thankfully, Robert set me right. I think that if it is only likely to effect a very small percentage of users AND there is a workaround, then we should simply document it and fix it on trunk and in the branch but not in the release. If there are too many of these, then we are not ready to release. 5. Most of the steps in http://wiki.apache.org/lucene-java/ReleaseTodo under the Branching & Feature Freeze section can be automated. Same with the Publication section. So, what are some possible actionable steps we can take? 1. To address #1, we can publish a test plan just like you would do in your day job. See what OpenNLP does: https://cwiki.apache.org/confluence/display/OPENNLP/TestPlan1.5.1. It starts by everyone addressing #1 above on our Wiki. See http://wiki.apache.org/lucene-java/TestPlans and http://wiki.apache.org/solr/TestPlans for a starting spot. 2. To address #2, we can setup a Jenkins job that runs the packaging process regularly and somehow figure out how to reliably test it. 3. Shrugs on #3. No one can make you participate, but we can make you feel guilty just like Hoss made me feel guilty. For PMC members specifically, however, _perhaps_ if you don't vote for two or three releases in a row, then you shouldn't be on the PMC. Your job on the PMC primarily consists of two things: vote on committers, vote on releases. I can see giving a pass once given life happens. Two or three in a row means you likely consider yourself emeritus or you just like having the title but not having any of the responsibility. At a minimum send in a +0 and let us know you have a pulse. 4. For #4, this is a collective commitment to make the release happen as quickly as possible while still being correct for most people in most situations. Focus on the goal. 5. For #5, we just need some people with some Ant skills to make it happen. 6. If we want to release every 6 months, then let's setup some sort of reminder system that nags us as we get closer or something. Let's also be honest with ourselves about what features are in the release and what are not. We don't have to have jam everything in all the time. OK, enough of my ideas. And I do stress that they are only ideas/observations. What do others think? Several of you have lived the RM pain now, let's take a good hard look at what can be done better so that we can release more often. -Grant --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
