On Sun, Oct 11, 2009 at 2:38 AM, William Stein <wst...@gmail.com> wrote:
> On Wed, Aug 19, 2009 at 1:19 AM, Robert Bradshaw > <rober...@math.washington.edu> wrote: > > > > <snip> > > The Sage Notebook is a Twisted application, and Twisted's "deferreds" > might seem like a good idea for solving the above problem. However, > they are actually *not* at all meant to solve the above sort of > problem, which is made I think very clear by the Twisted > documentation, which lists two types of async problems -- cpu bound > and "waiting for a resource" bound. The problem, at its simplest > level, is that no matter you do with Twisted deferreds -- making the > zip file little by little -- everything happens in a single thread, > and a total of at least 30 seconds of CPU time has to be spent by the > Sage notebook server making that zip archive. And that's 30 seconds > that the notebook server isn't responding to users, so overall the > notebook is going to feel sluggish to users. Also, it just seems > dumb to slow the notebook server down like this, given that, e.g., > sagenb.org is running on an 8-core multicore virtual machine. > > Fortunately, > > http://twistedmatrix.com/projects/core/documentation/howto/gendefer.html > > gives an example similar to this problem as an example, and explains > how to easily solve it in two lines using *threads*. So I took the > big chunk of scary blocking code that Robert Bradshaw wrote, put it in > a closure (a little next function f), and added the following two > lines to the server: > > from twisted.internet import threads > return threads.deferToThread(f) > > That's it. It worked first try, and solves the problem. What > happens behind the scenes is that Twisted uses a separate thread to > run the one function f, then when f completes it returns the output of > f. So it wraps the idea of "do something in a thread" with a > deferred. > > Twisted experts -- please explain the drawbacks of this approach... > This sounds fine to me. Threads are nasty when they can clobber each other. This function "appears" to be off in its own little corner doing its own thing. Doing a bunch of work to segment the zipping seems a bit overkill unless you fall into GIL madness and the entire process bogs down from the lock (all depends on how zip is implemented in python) I don't know how the zipping is done, but one alternative is to spawn a child process using Twisted which also uses a deferred which fires when the child dies. The advantage here would be other cores doing the work... This probably only makes sense if a command line zip mechanism is easy. > > By the way, the Twisted documentation has got way way better than I > remember it being in 2006. > Horrors!!! This must be stopped! What happens if this sort of thing catches on and, say, somebody documents Linux. All the benefits of being "gurus" would disappear... think of it... personal hygiene, appearance... even.... I hesitate to say... girls (or guys... you get the picture)... We'd just be geeks all over again... 40 years of liberation and dominance wiped out. > -- William > -- Glenn H. Tarbox, PhD || 206-274-6919 || gl...@tarbox.org - xmpp || ghtdak - aim,jabber,IRC,yahoo --~--~---------~--~----~------------~-------~--~----~ To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org -~----------~----~----~----~------~----~------~--~---
