> > Taking all of this into account, would this be a reasonable idea? > > 1. At Go/No-Go voting time, all updates which block F-N release but > > belong to F-M (M<N) release, must be already pushed stable. If this > > is not the case and it's the last blocking issue, selected tasks > > (like copying compose trees into appropriate places) can be > > performed, and Go/No-Go will be rescheduled to the day and time when > > it is expected that those updates will have been pushed. > > I think thats not a great idea. It gets back to why we only slip in one > week increments. If say we have a go/no-go on a thursday and the only > thing blocking it is some update thats not pushed stable all the way > yet, we reschedule for friday and if it's not done then we schedule for > saturday? This means everyone has to work extra hours without even > being sure when the release will be.
If the update is pending stable and just not pushed, it might sense to move it one day, yes (most probably skipping weekends, though). If it needs more testing, we might decide to postpone it a several days. If it's not available at all yet, waiting an extra week might be the right choice. So it would depend on the situation and best guess of folks at Go/No-Go. > Leaves less time to sync mirrors, > update common bugs, etc etc. I would say the opposite - all of that can start happening right away, it's not blocked on waiting for the FN-x push. So in case the announcement gets out on Tuesday as usual, it's the same time, but if it gets pushed back to Wednesday or later day, it's more time for these tasks to happen. The only exception is that FN-x updates repo, which will get shorter sync time because we want to make sure people download the fixed packages, not old ones. Currently that behavior is undefined. > > So, the alternative there would be to slip a week to get it pushed, but > some people may find that excessive. That's why I wanted to propose something more flexible, but hey, it's just an idea. > > > 2. We will > > create a new mirrormanager script which will go through the specified > > metalink(s) and remove all metadata hashes which are older than > > provided timestamp/hash. > > Something like that should be pretty easy to do I would think. > (Although I am not a mm developer) Looking into existing MM scripts, I have the same opinion, but I can contact Adrian to confirm. If we want to make it even simpler, we can drop all alternative metadata and leave just the current hash (that script would be run once the push containing that critical update is performed). > > > 3. If there are such updates as mentioned in > > point 1., RelEng will use this script to remove old metadata > > alternatives from the metalink, which means only metadata from the > > day this update was pushed or newer will be kept. In order to not > > increase mirror strain too much, this doesn't need to be used > > immediately, but just shortly before the release announcement (so > > that mirrors have time to sync latest packages, and the user load is > > distributed among more mirrors including those with current-1 or > > current-2 trees as long as possible). 4. Once the script is run in > > point 3., we can post the release announcement in 6 hours. > > > > I know there still one manual step involved (figuring out in which > > push the blocker update was included), but I don't know how to better > > solve it, especially if we don't want to wait for too long. > > > > I would be interested in Infra/RelEng feedback for the technical part > > of this (CCing Kevin and Dennis). Do you think this is reasonable > > solution, or am I completely off the track here? Do you see any > > better options? > > So, looking back, we had the case of that dnf-system-upgrade. Are there > any others in the past, or are we making a bigger than life deal out of > one case? I don't want to exaggerate the topic, but I'd also like to find and describe a process how we can avoid it next time. It will be needed twice a year at maximum. I believe there were a few similar issues in the past, but I can't really point to any other examples. In majority of cases, this is likely to be related to system upgrade (system-upgrade, dnf, plymouth, systemd, gpg keys). > > Also, that case could have been solved by dropping the alternates in > metalink as you suggest above at 2 right? Yes. > > One thing that perhaps we could improve is to somehow note these sorts > of things to releng. I just checked irc logs and I didn't see any > mention of that dnf-system-upgrade plugin update being important until > nov 3rd. Would a tracker ticket help this? In the future, these issues should be tracked by blocker bugs app using bugzilla tracker and a specific keyword, so we should not lose track of this. But as mentioned, pushing to stable is not enough, we also need to make sure old content is not served to users. That's why the "dropping alternative metadata from metalink" idea. We can file a releng ticket for this, and either include a description of what needs to be done, or link to some wiki SOP. QA can take care of all of that. The only thing that we need to ensure is that it really is handled before the announcement goes live, so it needs to be listed somewhere in RelEng/Infra "new release" SOP. -- test mailing list test@lists.fedoraproject.org To unsubscribe: http://lists.fedoraproject.org/admin/lists/test@lists.fedoraproject.org
