On Sun, 19 Jul 2009 17:20:46 +0400 (MSD), Ilya Etingof <[email protected]> wrote: > >>> But why main loop does not fire timed events when it has a chance to do >>> that? Note that my data processor takes 0.2 sec on each run, while my >>> timed event period is 1 sec. >>> >>> In other words, I'd understand this behavior if my data processor >>> would block main loop for a few periods of timed event, but this >>> is not the case. >> >> Because Twisted receives as many datagrams as possible before going back >> round the select loop (I think). So, 10 calls to datagramReceived are >> done (taking 10*0.2 seconds) before twisted gets a chance to schedule >> any other things. > >Perhaps it works that way. And what makes things worse is that if datagram >input rate is steady and higher than datagram processing time, the >LoopingCall calls will never be invoked.
It will eventually stop reading datagrams and go do something else. The exact way it decides when to stop is completely arbitrary and I don't think anyone has ever demonstrated that it's clever or appropriate in the general case (it stops after reading 256k of datagrams). > >> As JP has said, you're not allowed to block the main loop like that, but >> I've seen similar problems with even relatively quick datagramReceived >> handlers (~0.1msec) under very high load (1000pps) resulting in a form >> of starvation. This isn't generally a problem with TCP calls due to flow >> control. > >I do not think I'm blocking the main loop so much. I'd say I create a >condition when data processing time is a little bit longer than the period >of incoming datagrams arrival. > >By way of feedback, it looks to me that if reactor API would give >user some degree of control on timed versus receiption calls sequencing, >that would become a more-or-less elegant solution for issues like mine. One possibility is that we could document the limit I described above and tell people to adjust it upwards or downwards as they so desire. > >> The solution is: >> >> def datagramReceived(...): >> reactor.callLater(0, dorealwork, ...) >> >> def dorealwork(self, ...): >> ... >> >> This will schedule the "real" work as soon as possible in the next >> reactor loop, but also "fairly" in line with other calls. > >Thanks for the hint! > >With this callLater approach, timer function works better, though, still >not absolutely fair: > > [snip] > >As we can see, timer call period seems to grow over high load. > reactor.callLater(0, f) isn't magic. I probably wouldn't have suggested it as a solution here. All it means (more or less) is "do this work after all current i/o events have been dispatched." You still might end up with more than one second of processing bunched up together, and so you'll still miss a LoopingCall iteration. Jean-Paul _______________________________________________ Twisted-Python mailing list [email protected] http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
