On Wed, Aug 11, 2010 at 6:40 AM, Danilo Šegan <[email protected]> wrote: > Hi Rob, > > У сре, 11. 08 2010. у 10:55 +1200, Robert Collins пише: > >> What we can do now is: >> - we receive the request >> - we commit the *intent* and return a 'pending' message to the >> browser, which then renders a spinner locally and polls an API for >> completion of the queued item >> - the queue processes the work >> - the polling js sees the work is complete and renders completion >> >> Now, queue latency and when the user gets their benefits. The current >> job system has on average 30 second latency + processing time. We can >> address that for specific queues with a small amount of work to poll >> the DB every second in a new transaction. >> >> This should be pretty transparent to the user, will make the page time >> come -way- down, and handle very large team rearrangements gracefully. > > We want this for POFile exports as well. At the moment, we have an > export queue and we notify the user by email. In most cases, we can > figure out that the generation of the POFile will be quick (even doable > in 2s[1]), but because there are many (0.5%?) POFiles where it wouldn't, > we have this old queue mechanism. Users hate it.
I think this shows that if we run everything through a queue, there needs to be a way to run really quick tasks in parallel with really big tasks. You wouldn't want a single POFile export for a project to be stuck behind the processing of 75k files for all of ubuntu. There could be a slow lane and a fast lane. If the jobs have a method for estimating the length of time reasonably well, we could also have a super-fast lane, where the page decides not to queue the task at all. Though _fillTeamParticipation() should run sufficiently quick for most teams after turning it into a single query, I looked at the worst case scenario, which is the team with the most members (3,800) being added to the team with the most super teams (60). That could require 228,000 (60 * 3,800) new rows in TeamParticipation, assuming no overlap with existing members of the new super teams. In that case, queued jobs would probably need to be split apart into smaller queued jobs that could be processed reasonably quickly in a single db transaction. Of course, processing the same type of job in parallel slow/fast lanes could wreak havoc on the TeamParticipation table if adding and removal of team members is done out of order. -Edwin > If this can be made easier to implement and support, I'd be very > happy :) > > Cheers, > Danilo > > [1] For a full language pack export for Ubuntu, containing ~75k files, > we produce it in 9h or roughly 0.5s per POFile. > > PS. To be honest, in the past, entire PO file exports were not doable in > 5s for anything but the most trivial cases. We've since improved so can > think about better solutions for users. _______________________________________________ Mailing list: https://launchpad.net/~launchpad-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
