With 2.8 specifically there is a *lot* of work done by the migrations
and the runtime should scale with the content size. In other words I
don't think it's doing useless work, there is just a lot of it.
I just did a quick audit of the two major offending migrations and they
are implemented sanely so I don't think there is a quick fix to improve
the runtime.
I believe to get a shorter runtime we would have to parallelize the
migrations.
Regarding things going wrong, we do assume that migrations should be
re-entrant meaning that if for some reason something strange occurs it
should always be safe to run pulp-manage-db and it will pick up where it
left off. Just a behavioral FYI.
These migration with 2.8 should not be norm, so I don't expect this to
be very common in the future until maybe a major release (3.0).
Given all ^, what do you think we should do?
-Brian
On 07/01/2016 11:23 AM, Eric Helms wrote:
I think there are a couple of considerations.
1) The first issue is that a 6-18 hour upgrade window is not something
users expect and we've not been warning them of such so they can plan an
outage accordingly. Lengthy upgrades also have that tendency to make
users feel something is wrong or increase the risk that something can go
wrong in between.
2) The fundamental question of - is this a bug or does this make
perfect sense and how it has to work?
3) Applying the upgrade on an existing 2.6 if it changed nothing of the
environment could work, the tough part is having to distribute that
backwards. Pulp would have to distribute it back to 2.6, and Katello
would have to push out patches to our 2.4 release channel.
Eric
On Fri, Jul 1, 2016 at 11:14 AM, Brian Bouterse <bbout...@redhat.com
<mailto:bbout...@redhat.com>> wrote:
I'm trying to understand if the pain point is related to downtime or
total runtime.
For instance, what if these migration could be run as a
pre-migration step, ahead of time while Pulp was still online? The
upgrade would still take just as long but you could use your (in
this case) 2.6 install normally while the migrations are applying.
Once they are done then the actual upgrade of the codebase could be
very short.
-Brian
On 07/01/2016 09:20 AM, Eric Helms wrote:
On Fri, Jul 1, 2016 at 8:52 AM, Ashby, Jason (IMS)
<ash...@imsweb.com <mailto:ash...@imsweb.com>
<mailto:ash...@imsweb.com <mailto:ash...@imsweb.com>>> wrote:
FWIW I just upgraded from 2.7 -> 2.8 and it was approx. 1-2 hr
upgrade to get through the migrations in pulp-manage-db.____
__ __
290 GB /var/lib/pulp____
16 GB MongoDB____
__ __
*From:*pulp-list-boun...@redhat.com
<mailto:pulp-list-boun...@redhat.com>
<mailto:pulp-list-boun...@redhat.com
<mailto:pulp-list-boun...@redhat.com>>
[mailto:pulp-list-boun...@redhat.com
<mailto:pulp-list-boun...@redhat.com>
<mailto:pulp-list-boun...@redhat.com
<mailto:pulp-list-boun...@redhat.com>>] *On Behalf Of *Michael
Hrivnak
*Sent:* Friday, July 01, 2016 8:31 AM
*To:* Eric Helms <ehe...@redhat.com
<mailto:ehe...@redhat.com> <mailto:ehe...@redhat.com
<mailto:ehe...@redhat.com>>>
*Cc:* pulp-list <pulp-list@redhat.com
<mailto:pulp-list@redhat.com> <mailto:pulp-list@redhat.com
<mailto:pulp-list@redhat.com>>>
*Subject:* Re: [Pulp-list] Long upgrade times from 2.6 ->
2.8____
__ __
Did you get any feedback on whether one particular migration
seemed
to be running for a lot of that time?
For the 1.5TB/100GB MongoDB scenario here is what I am able to glean
from user logs (which I can share privately with anyone debugging):
~5 hours: Applying pulp_puppet.plugins.migrations version 4
~10 hours: Applying pulp_rpm.plugins.migrations version 28
Use reports "lots of stating, unlinking, and linking of all the
symlinks
in /var/lib/pulb" if that helps.
Another user reports ~6 hours on 176G of data.
Eric
____
__ __
Michael____
__ __
On Fri, Jul 1, 2016 at 7:23 AM, Eric Helms
<ehe...@redhat.com <mailto:ehe...@redhat.com>
<mailto:ehe...@redhat.com <mailto:ehe...@redhat.com>>>
wrote:____
Howdy,____
__ __
We (Katello) have had users reporting incredibly long
upgrade
times when upgrading from 2.6 to 2.8. This occurs during the
pulp-manage-db step that is run as the beginning of our
installers upgrade process. Based on the numbers below, does
this make sense at all?____
__ __
Some numbers:____
__ __
18 hour upgrade____
1.5 TB /var/lib/pulp____
100GB MongoDB____
__ __
__ __
Thanks,____
Eric____
_______________________________________________
Pulp-list mailing list
Pulp-list@redhat.com <mailto:Pulp-list@redhat.com>
<mailto:Pulp-list@redhat.com <mailto:Pulp-list@redhat.com>>
https://www.redhat.com/mailman/listinfo/pulp-list____
__ __
------------------------------------------------------------------------
Information in this e-mail may be confidential. It is
intended only
for the addressee(s) identified above. If you are not the
addressee(s), or an employee or agent of the addressee(s),
please
note that any dissemination, distribution, or copying of this
communication is strictly prohibited. If you have received this
e-mail in error, please notify the sender of the error.
_______________________________________________
Pulp-list mailing list
Pulp-list@redhat.com <mailto:Pulp-list@redhat.com>
https://www.redhat.com/mailman/listinfo/pulp-list
_______________________________________________
Pulp-list mailing list
Pulp-list@redhat.com <mailto:Pulp-list@redhat.com>
https://www.redhat.com/mailman/listinfo/pulp-list
_______________________________________________
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list
