My main worry or first check I wanted to put to rest was that it was an actual issue. Given that everything is sane, I think warning users that upgrades will scale with the amount of content and giving them some rough estimates based on other users data is the best option. That way they can at least prepare themselves with knowledge for planning their upgrade outages.
Thanks for looking into this so quickly, Eric On Fri, Jul 1, 2016 at 11:50 AM, Brian Bouterse <bbout...@redhat.com> wrote: > With 2.8 specifically there is a *lot* of work done by the migrations and > the runtime should scale with the content size. In other words I don't > think it's doing useless work, there is just a lot of it. > > I just did a quick audit of the two major offending migrations and they > are implemented sanely so I don't think there is a quick fix to improve the > runtime. > > I believe to get a shorter runtime we would have to parallelize the > migrations. > > Regarding things going wrong, we do assume that migrations should be > re-entrant meaning that if for some reason something strange occurs it > should always be safe to run pulp-manage-db and it will pick up where it > left off. Just a behavioral FYI. > > These migration with 2.8 should not be norm, so I don't expect this to be > very common in the future until maybe a major release (3.0). > > Given all ^, what do you think we should do? > > -Brian > > On 07/01/2016 11:23 AM, Eric Helms wrote: > >> I think there are a couple of considerations. >> >> 1) The first issue is that a 6-18 hour upgrade window is not something >> users expect and we've not been warning them of such so they can plan an >> outage accordingly. Lengthy upgrades also have that tendency to make >> users feel something is wrong or increase the risk that something can go >> wrong in between. >> 2) The fundamental question of - is this a bug or does this make >> perfect sense and how it has to work? >> 3) Applying the upgrade on an existing 2.6 if it changed nothing of the >> environment could work, the tough part is having to distribute that >> backwards. Pulp would have to distribute it back to 2.6, and Katello >> would have to push out patches to our 2.4 release channel. >> >> Eric >> >> On Fri, Jul 1, 2016 at 11:14 AM, Brian Bouterse <bbout...@redhat.com >> <mailto:bbout...@redhat.com>> wrote: >> >> I'm trying to understand if the pain point is related to downtime or >> total runtime. >> >> For instance, what if these migration could be run as a >> pre-migration step, ahead of time while Pulp was still online? The >> upgrade would still take just as long but you could use your (in >> this case) 2.6 install normally while the migrations are applying. >> Once they are done then the actual upgrade of the codebase could be >> very short. >> >> -Brian >> >> On 07/01/2016 09:20 AM, Eric Helms wrote: >> >> >> >> On Fri, Jul 1, 2016 at 8:52 AM, Ashby, Jason (IMS) >> <ash...@imsweb.com <mailto:ash...@imsweb.com> >> <mailto:ash...@imsweb.com <mailto:ash...@imsweb.com>>> wrote: >> >> FWIW I just upgraded from 2.7 -> 2.8 and it was approx. 1-2 hr >> upgrade to get through the migrations in pulp-manage-db.____ >> >> __ __ >> >> 290 GB /var/lib/pulp____ >> >> 16 GB MongoDB____ >> >> __ __ >> >> *From:*pulp-list-boun...@redhat.com >> <mailto:pulp-list-boun...@redhat.com> >> <mailto:pulp-list-boun...@redhat.com >> <mailto:pulp-list-boun...@redhat.com>> >> [mailto:pulp-list-boun...@redhat.com >> <mailto:pulp-list-boun...@redhat.com> >> <mailto:pulp-list-boun...@redhat.com >> <mailto:pulp-list-boun...@redhat.com>>] *On Behalf Of *Michael >> Hrivnak >> *Sent:* Friday, July 01, 2016 8:31 AM >> *To:* Eric Helms <ehe...@redhat.com >> <mailto:ehe...@redhat.com> <mailto:ehe...@redhat.com >> <mailto:ehe...@redhat.com>>> >> *Cc:* pulp-list <pulp-list@redhat.com >> <mailto:pulp-list@redhat.com> <mailto:pulp-list@redhat.com >> <mailto:pulp-list@redhat.com>>> >> *Subject:* Re: [Pulp-list] Long upgrade times from 2.6 -> >> 2.8____ >> >> __ __ >> >> Did you get any feedback on whether one particular migration >> seemed >> to be running for a lot of that time? >> >> >> For the 1.5TB/100GB MongoDB scenario here is what I am able to >> glean >> from user logs (which I can share privately with anyone >> debugging): >> >> ~5 hours: Applying pulp_puppet.plugins.migrations version 4 >> ~10 hours: Applying pulp_rpm.plugins.migrations version 28 >> >> Use reports "lots of stating, unlinking, and linking of all the >> symlinks >> in /var/lib/pulb" if that helps. >> >> Another user reports ~6 hours on 176G of data. >> >> Eric >> >> >> ____ >> >> __ __ >> >> Michael____ >> >> __ __ >> >> On Fri, Jul 1, 2016 at 7:23 AM, Eric Helms >> <ehe...@redhat.com <mailto:ehe...@redhat.com> >> <mailto:ehe...@redhat.com <mailto:ehe...@redhat.com>>> >> wrote:____ >> >> Howdy,____ >> >> __ __ >> >> We (Katello) have had users reporting incredibly long >> upgrade >> times when upgrading from 2.6 to 2.8. This occurs during >> the >> pulp-manage-db step that is run as the beginning of our >> installers upgrade process. Based on the numbers below, >> does >> this make sense at all?____ >> >> __ __ >> >> Some numbers:____ >> >> __ __ >> >> 18 hour upgrade____ >> >> 1.5 TB /var/lib/pulp____ >> >> 100GB MongoDB____ >> >> __ __ >> >> __ __ >> >> Thanks,____ >> >> Eric____ >> >> >> _______________________________________________ >> Pulp-list mailing list >> Pulp-list@redhat.com <mailto:Pulp-list@redhat.com> >> <mailto:Pulp-list@redhat.com <mailto:Pulp-list@redhat.com>> >> https://www.redhat.com/mailman/listinfo/pulp-list____ >> >> __ __ >> >> >> >> >> ------------------------------------------------------------------------ >> >> Information in this e-mail may be confidential. It is >> intended only >> for the addressee(s) identified above. If you are not the >> addressee(s), or an employee or agent of the addressee(s), >> please >> note that any dissemination, distribution, or copying of this >> communication is strictly prohibited. If you have received >> this >> e-mail in error, please notify the sender of the error. >> >> >> >> >> _______________________________________________ >> Pulp-list mailing list >> Pulp-list@redhat.com <mailto:Pulp-list@redhat.com> >> https://www.redhat.com/mailman/listinfo/pulp-list >> >> >> _______________________________________________ >> Pulp-list mailing list >> Pulp-list@redhat.com <mailto:Pulp-list@redhat.com> >> https://www.redhat.com/mailman/listinfo/pulp-list >> >> >>
_______________________________________________ Pulp-list mailing list Pulp-list@redhat.com https://www.redhat.com/mailman/listinfo/pulp-list
