On Fri, Jun 10, 2011 at 4:19 PM, Robert Collins <[email protected]> wrote: > On Thu, Jun 9, 2011 at 10:44 PM, Robert Collins > <[email protected]> wrote: >> See bug 794802 for some of the gory details; in short bugsummary >> managed to spring a nasty surprise on us. > ... >> At this point I've handed over the incident to Stuart, and am halting() >> myself. >> >> I hope to awaken to great news :) > > So this is basically behind us - there is a cowboyed schema change > live that preserves performance - the branch for it to exist > officially will be landing on devel soon. Until that lands the tags > portlets for projects won't adjust (specifically closing all the bugs > for a tag won't remove that tag). > > Some observations: > - we had a 900 timeout spike over the period of degraded service. > Thats not too bad - its just over one every 2 minutes.
I think this will include some timeout spikes on unrelated pages. Applying DB patches to live and busy systems is a learning experience. Yesterday I learned that for some reason I don't yet understand, creating an empty table with foreign key references requires locks on the targeted table that conflicts with normal read/write operations, and that lock can never be fully gained when the table is always being read by at least one transaction. -- Stuart Bishop <[email protected]> http://www.stuartbishop.net/ _______________________________________________ Mailing list: https://launchpad.net/~launchpad-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
