Hi Piotr, Thanks for bringing the issue up. I agree that this is something we should look into improving.
Strictly speaking, I don’t think it is an issue only limited to after a RC vote has started. Taking 1.5.0 as an example, release-1.5 was branched out at the announcement for feature freeze for nearly a month before the first RC for 1.5.0 was created. That period of time was meant for release testing before the actual votes took place, but since commits still went through to release-1.5 during this time without much restrictions, there never really was a point-in-time where individual testing efforts could have a commonly agreed scope. We usually create RC0s for the initial testing efforts, but by the time RC1 is out, it is hard to say that test efforts on RC0 were still valid because there was little restraint on what was merged in between. As for the issue under the context of during RC voting: Up to now, the default way we have mostly used to deal with non-blocking bug fixes was to “not block RC votes on them, merge them to the release branch and see if it happens to get in if the previous RC was cancelled.” Sometimes, some critical bug fixes were made non-blocking due to the fact that it was introduced already in a previous released version. The fact that these critical bug fixes sneak into the release in the end, as you mentioned, arguably changes the scope of the release across RCs. Previous RC voters usually expect that previous test efforts can be carried over to the new RC because additional commits to the new RC *should* only incrementally affect the scope. As the case for the two bug fixes you pointed out (FLINK-9349 and FLINK-9295), it is arguable that isn’t the case, since a lot of our previous test efforts involved Kafka substantially. Minor thing is that Jira issues should be updated accordingly. If you merge something to release branch, please pay attention to cancelled RCs and from which commit new RC is being created. I don’t think this is as minor as it seems. The fact that this correction is a very manual process that occurs at a non-anticipated event can easily lead to incorrect release notes (which I think has happened a few times already in the past, partially because of this issue). I would propose that any merges to release branches during RC votes (after release testing kicked off) should go through release manager, so that he could decide whether to include such fixes or postpone them to next release. Maybe this would require to create separate release branches for 1.5.0 and 1.5.1. Would +1 to this. Just to rephrase this concretely by taking 1.5.0 as an example: A specific release-1.5.0 release branch is created from release-1.5 as soon as feature freeze is announced. All JIRAs resolved from this point on should have ‘fix version’ set as ‘1.5.1’, and commits only go to the release-1.5 branch. They will only be included in release-1.5.1 once that branches out to prepare for 1.5.1 release. Any commits that wish to be included in release-1.5.0 strictly needs to go through the release manager; a commit that does go through to release-1.5.0 should mean that a new RC should be created and current ongoing votes are cancelled. Managing changing such JIRAs’ fix version from ‘1.5.1’ to ‘1.5.0’ should be much more easier, since that should happen along with the action of bringing over commits to release-1.5.0, and not at the event of an RC being cancelled and the commits just happen to sneak in. For minor bug fix releases, it should be quite easy to execute this proposal. For major releases, we might need to consider the overhead of cherry-picking commits over. Again, looking at 1.5.0 - there were hundreds of commits that were merged to both master (1.6-SNAPSHOT) and release-1.5. Assuming we had this process in place, how many of those would have also needed to be merged to release-1.5.0 (as decided by the release manager)? OTOH, if such a process allows more control over scope creep / more valid and meaningful test efforts early on right after feature freeze, than the overhead might be worth it. Cheers, Gordon On 24 May 2018 at 8:24:29 PM, Piotr Nowojski (pi...@data-artisans.com) wrote: Hey, I would like to raise an issue that happensa at couple of times during 1.5.0 RCx votings. There were some bug fixes like: https://issues.apache.org/jira/browse/FLINK-9349 <https://issues.apache.org/jira/browse/FLINK-9349> https://issues.apache.org/jira/browse/FLINK-9295 <https://issues.apache.org/jira/browse/FLINK-9295> That were merged to release branch after creating some RC, they were marked with fix versions set to 1.5.1 and new when previous RC vote was cancelled and new RC was created, they sneaked into 1.5.0. Minor thing is that Jira issues should be updated accordingly. If you merge something to release branch, please pay attention to cancelled RCs and from which commit new RC is being created. Much bigger issue is that when such fixes are sneaking so late to the release, they do not go through normal release testing procedure. I don’t know how should we handle such situations, but if something like that happened to your commits during this 1.5.0, please double check them when new RC comes out. They might require some manual testing or running some end to end tests. I would propose that any merges to release branches during RC votes (after release testing kicked off) should go through release manager, so that he could decide whether to include such fixes or postpone them to next release. Maybe this would require to create separate release branches for 1.5.0 and 1.5.1. Piotrek
