With the rollback of 2.10.1 done and 2.10.2 building, I thought it would be good to go over exactly what went happened. Hopefully this info will help to get us all on the same page as we come up with ways to prevent this from happening again. I'll be pasting links inline instead of using markdown footnotes to make things a little easier if you want to follow along step-by-step. Without further ado...
A user reported a failed migration in 2.10.1 shortly after its GA release: https://pulp.plan.io/issues/2377 While this is certainly troubling, what was particularly worrisome about this was that the failing migration was related to a new feature introduced in 2.11, which had no business popping up in a 2.10 release: https://pulp.plan.io/issues/1983 & https://github.com/pulp/pulp/pull/2637 It became immediately apparent that at some point master had been merged to 2.10-dev, introducing 2.11's features and bug fixes to 2.10's next release. Shortly thereafter, I pulled the 2.10.1 release out of our stable repo, leaving 2.10.0 in its place, so that more users would not be affected. 2.10.1 is now unavailable. mhrivnak tracked down the merge commit that brought the 2.11 changes back to 2.10: https://github.com/pulp/pulp/commit/1d4088385ae973e2b14c7e591468ac9e51dbdb64 While he was finding that, I went through the issues open against the 2.11 platform release and first checked to see which commits related to those issues existed on 2.10-dev in platform. Second, I went through the plugins to see if any plugins were affected. None were. I filed a bug to track the effort of fixing 2.10-dev in the platform repository at this point: https://pulp.plan.io/issues/2378 Once the merge commit was identified, I did some diffing to find and fix other commits to 2.10-dev that did not belong. All commits are documented in #2378, and have since been reverted. The merge commit is related to PR 2770: https://github.com/pulp/pulp/pull/2770 This PR was merged to master, but should have been merged to 2.10-dev. This is where it happened. The commit from this PR was merged back to 2.10 from master. Since master was version 2.11 at this point in time, so when it was merged all 2.11-related commits on master appeared on 2.10-dev. These are the commits that were reverted to fix #2378. While I have avoided editorializing in this post-mortem, I think it's important to point out that the problem here, in my opinion, isn't really that this merge happened. I'm more interested in how 2.10.1 got released with these commits included so that we can improve our processes and prevent this from happening again. Given that the merge of master back to 2.10-dev wasn't detected until a user reported a failed migration, I'm also interested in improving our processes to catch upgrade failures like this before our users do. Had this migration not failed, I think the "extra" commits on 2.10-dev may have continued to go unnoticed for an indefinite amount of time. Finally, I think it's also worth mentioning that, as seen on pulp-list, 2.10-dev has been fixed, and we have a workaround to give to folks affected by this. 2.10-dev has been merged forward through 2.11-dev to master, and I'm currently in the release process for a 2.10.2 hotfix. _______________________________________________ Pulp-dev mailing list [email protected] https://www.redhat.com/mailman/listinfo/pulp-dev
