Re: [Twisted-Python] deferred graph?
Dan Stromberg writes:
> Strangely, this doesn't give the report until after the sleep finishes...
That is because the code you included doesn't actually print the
returned traceback. The reason that it gets printed at the end is
because it gets garbage collected then.
The following code prints out the traceback twice, with the first before
the first sleep and the second (prefixed with "Unhandled error in
Deferred:") before the second sleep.
Tom
#!/usr/bin/python
import time
from twisted.internet import defer
defer.setDebugging(True)
def functionReturningDeferred():
return defer.succeed('Some value')
d = functionReturningDeferred()
def printValue(value):
print 'Yay, I got %r' % value
return value
def second_callback(value):
print 'still %r' % value
return gen_error()
def third_callback(value):
print 'and still %r' % value
return value
def gen_error():
return defer.fail(AssertionError)
def got_error(value):
print 'bad thing: %r' % value
d.addCallback(printValue)
d.addCallback(second_callback)
d.addCallback(third_callback)
print d._debugInfo._getDebugTracebacks()
time.sleep(2)
del d
time.sleep(2)
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Effects of backlog parameter on listenTCP
On Wed, Jul 18, 2012 at 04:43:01PM -0700, Tobias Oberstein wrote: > Could someone shortly elaborate on the semantics / effect of the parameter > "backlog" with listenTCP? It exactly corresponds to the "backlog" parameter of the BSD socket API function "listen(int sockfd, int backlog)". On my Debian Testing machine, the listen(2) manpage says: The backlog argument defines the maximum length to which the queue of pending connections for sockfd may grow. If a connection request arrives when the queue is full, the client may receive an error with an indication of ECONNREFUSED or, if the underlying protocol supports retransmission, the request may be ignored so that a later reattempt at connection succeeds. ..and later: The behavior of the backlog argument on TCP sockets changed with Linux 2.2. Now it specifies the queue length for completely established sockets waiting to be accepted, instead of the number of incomplete connection requests. The maximum length of the queue for incomplete sockets can be set using /proc/sys/net/ipv4/tcp_max_syn_backlog. When syncookies are enabled there is no logical maximum length and this setting is ignored. See tcp(7) for more information. If the backlog argument is greater than the value in /proc/sys/net/core/somaxconn, then it is silently truncated to that value; the default value in this file is 128. In kernels before 2.4.25, this limit was a hard coded value, SOMAXCONN, with the value 128. Obviously the Twisted documentation can't go into all this detail because Twisted runs on more operating systems than just Linux. Every OS that supports the BSD socket API should have documentation explaining the precise definition and meaning of the backlog parameter on that platform. For example, here's the MSDN docs for the listen() function in WinSock: http://msdn.microsoft.com/en-us/library/windows/desktop/ms739168%28v=vs.85%29.aspx ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Effects of backlog parameter on listenTCP
On 07/18/2012 07:43 PM, Tobias Oberstein wrote: > Could someone shortly elaborate on the semantics / effect of the parameter > "backlog" with listenTCP? > > The docs say: "size of the listen queue". > > Is that a tunable within Twisted? > > I mean, there are kernel parameters for queue sizes of not yet completely TCP > handshaked connections and so on. But those are kernel tunables. > > What does the "backlog" parameter? > > The reason I am asking: I am stress testing a Twisted server and I see client > connections denied upon high connection rates .. does the parameter help? > It's the parameter passed to the listen(2) system call - from the man page: The backlog argument defines the maximum length to which the queue of pending connections for sockfd may grow. If a connection request arrives when the queue is full, the client may receive an error with an indication of ECONNREFUSED or, if the underlying protocol supports retransmission, the request may be ignored so that a later reattempt at connection succeeds. So, yes, higher values will prevent connection refused. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] deferred graph?
Dan Stromberg wrote: > On Wed, Jul 18, 2012 at 3:32 AM, wrote: > > > > I don't think anything in the thread suggested that this approach will > > circumvent a time.sleep(10) call. > > I'm not sure where you're getting this circumvention issue from. At this point I'm not sure what the problem you're trying to solve is. To look at this from another direction… here's the built-in facilities in Twisted for debugging what's happening with Deferreds: - Deferred's __str__ and __repr__ summarises the key state (i.e. has this been fired, is this waiting on another Deferred). - unhandled errors of GC'd Deferreds triggers an “Unhandled error” log message, to let you know that your code is probably missing something it should act on. Due to the nature of GC this is a best effort feature, there's no guarantee it will fire reliably or at all. - you can setDebugging(True) to have Twisted capture more information to include in the above log message: the traceback for what created the Deferred, and the traceback that initially called it. This is very helpful for helping you pinpoint which parts of your code are failing to add an errback. - as a bonus, because this is Python, you can abuse the private variables used to implement the previous points for your own ad hoc debugging, if you are ok with the fact that they are undocumented and unsupported APIs. And that's basically it. (If I've forgotten something I'm sure someone will reply to remind me.) It sounds like you're looking for something else, but I'm not sure quite what. Something roughly along the lines of “what are all the live Deferreds in my process?” maybe? -Andrew. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Effects of backlog parameter on listenTCP
Could someone shortly elaborate on the semantics / effect of the parameter "backlog" with listenTCP? The docs say: "size of the listen queue". Is that a tunable within Twisted? I mean, there are kernel parameters for queue sizes of not yet completely TCP handshaked connections and so on. But those are kernel tunables. What does the "backlog" parameter? The reason I am asking: I am stress testing a Twisted server and I see client connections denied upon high connection rates .. does the parameter help? Thanks! Tobias ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] deferred graph?
On Wed, Jul 18, 2012 at 3:32 AM, wrote: > > > I don't think anything in the thread suggested that this approach will > circumvent a time.sleep(10) call. I'm not sure where you're getting this circumvention issue from. > In any case, it won't. Why is there > a time.sleep(10) call there at all? Does it do anything except cause a > problem? > The sleep is not a problem. The sleep allows me to see whether the debugging info I want is output immediately, or whether I need to wait for the program to terminate to see it. I put in the sleep simply because I have another program with an infinite loop, and I'd like to do something similar to it to get debugging info periodically (no sleep, but lots of other stuff going on), without having to wait for a program termination that is unlikely to come anytime soon. I'm starting to think I need to temporarily make the infinite loop finite. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] unit testing question: twisted with tk
In article <5006fc68.8050...@itamarst.org>, Itamar Turner-Trauring wrote: > On 07/18/2012 01:01 PM, Russell E. Owen wrote: > > I have a bit of code that combines twisted with Tkinter > > and now I'd like to write some unittests for it. > > > > I can't seem to figure out how to get TwistedTrial to handle this case. > > I've written a unit test like this: > > > > from twisted.trial import unittest > > from twisted.internet.defer import Deferred > > import twisted.internet.tksupport > > root = Tkinter.Tk() > > twisted.internet.tksupport.install(root) > > from twisted.internet import reactor > > > > class TestTkSocket(unittest.TestCase): > > def test...(...): > > > > > > the one test so far returns a deferred and calls errback on that > > deferred if the test fails, and callback if it succeeds. When I run the > > unit test with trial I get this error error: > > twisted.trial.util.DirtyReactorAggregateError > > The problem is that setting up tk support involves a repeatedly > scheduled event, which means it's in the reactor causing that warning. > Make sure you call tksupport.uninstall() at the end of each test, by > either putting in tearDown or doing self.addCleanup(tksupport.uninstall). I put tksupport.install(root) in the setUp and tksupport.uninstall() in the tearDown and it worked perfectly. Thank you very much. I'm thrilled to be able to run these unit tests. -- Russell ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] A few simple questions
On 07/18/2012 01:31 PM, Russell E. Owen wrote: > If there is a "best practices for error handling" document I'd love to > read it. I found an overview of deferreds that was helpful. It pointed > out that addCallbacks is not the same as addCallback followed by > addErrback and I'm not sure I'm handling the difference correctly. Right > now I use addCallbacks(callback, errback). This does not call the > errback if the callback fails. I do this intentionally because I want > the errback to focus on problems with the connection, not my reaction to > it, and the default error handler seems to report errors in the > callback, which is fine. > One thing to realize is that errors that end up in a Deferred are only logged if it gets garbage collected... which won't happen if you keep a reference to it. So: 1. You should always delete a reference to a Deferred when you do callback() or errback(). 2. In case it never gets GC'ed, always do .addErrback(log.err) as the last callback on the Deferred, if there's no other final error handler. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] unit testing question: twisted with tk
On 07/18/2012 01:01 PM, Russell E. Owen wrote: > I have a bit of code that combines twisted with Tkinter > and now I'd like to write some unittests for it. > > I can't seem to figure out how to get TwistedTrial to handle this case. > I've written a unit test like this: > > from twisted.trial import unittest > from twisted.internet.defer import Deferred > import twisted.internet.tksupport > root = Tkinter.Tk() > twisted.internet.tksupport.install(root) > from twisted.internet import reactor > > class TestTkSocket(unittest.TestCase): > def test...(...): > > > the one test so far returns a deferred and calls errback on that > deferred if the test fails, and callback if it succeeds. When I run the > unit test with trial I get this error error: > twisted.trial.util.DirtyReactorAggregateError The problem is that setting up tk support involves a repeatedly scheduled event, which means it's in the reactor causing that warning. Make sure you call tksupport.uninstall() at the end of each test, by either putting in tearDown or doing self.addCleanup(tksupport.uninstall). ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] A few simple questions
If you haven't seen it already, Dave Peticolas has a great introduction to twisted (it's a bit long, but well worth it) at http://krondo.com/?page_id=1327. In particular, until I saw this graphic on callback/errback chaining ( http://krondo.com/blog/wp-content/uploads/2009/10/deferred-2.png), I never properly understood what twisted was doing (linked from this http://krondo.com/?p=1825). Naveen On Wed, Jul 18, 2012 at 1:31 PM, Russell E. Owen wrote: > In article , > Glyph wrote: > > > On Jul 17, 2012, at 9:25 AM, Russell E. Owen wrote: > > > > >> Can you elaborate on the case? > > > > > > I've found that Twisted sometimes swallows errors unless I am extremely > > > careful. I would like to be able to check a protocol to make sure it is > > > operational (connected and happy) as a means of assuring that I've not > > > missed an error. > > > > This isn't really elaborating. > > > > What error does Twisted swallow? In what manner do you have to be > "careful"? > > What characterizes a protocol's "happiness" beyond its momentary > > connected-ness? How would you "miss" an error? > > > > Twisted is event-driven, so pretty much all the state changes you're > > interested in are delivered as events (method calls on your object). If > you > > are "missing" them because Twisted isn't calling them, that sounds like a > > really serious bug we should investigate. If you're missing them for > some > > other reason then you should just fix your code so it doesn't miss them > any > > more :). > > Regarding swallowing errors: > > I'm not claiming Twisted has bugs in this area (though I found and > reported one obscure case that may be a bug: errors in tk command > ::tk::Mac::quit are silently ignored). > > However, I found it rather easy to make coding errors that cause Twisted > to not report errors. I have heard the same complaint from colleagues. > > At this point I think my code is robust, though I will be more confident > once I finish my unit tests. > > If there is a "best practices for error handling" document I'd love to > read it. I found an overview of deferreds that was helpful. It pointed > out that addCallbacks is not the same as addCallback followed by > addErrback and I'm not sure I'm handling the difference correctly. Right > now I use addCallbacks(callback, errback). This does not call the > errback if the callback fails. I do this intentionally because I want > the errback to focus on problems with the connection, not my reaction to > it, and the default error handler seems to report errors in the > callback, which is fine. > > Regarding state: > > I'm used to event-driven frameworks that both offer callbacks and the > ability to query state. I find it helpful for reporting errors as soon > as possible (e.g. before writing check if the socket is connected; raise > an exception if not). For a TCP/IP socket, I view a socket as having > these states: > - connecting > - connected > - disconnecting (with associated reason) > - disconnected (with associated reason) > TCP/IP Servers have a similar set of states, with connected -> listening > > However, it's certainly true that being able to query state is not > essential; callbacks suffice. > > -- Russell > > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] unit testing question: twisted with tk
On 07/18/2012 06:01 PM, Russell E. Owen wrote: > root = Tkinter.Tk() > twisted.internet.tksupport.install(root) Does the TK support work like other reactors i.e. it must be the very first thing in the python file, before all other Twisted imports? ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] A few simple questions
In article , Glyph wrote: > On Jul 17, 2012, at 9:25 AM, Russell E. Owen wrote: > > >> Can you elaborate on the case? > > > > I've found that Twisted sometimes swallows errors unless I am extremely > > careful. I would like to be able to check a protocol to make sure it is > > operational (connected and happy) as a means of assuring that I've not > > missed an error. > > This isn't really elaborating. > > What error does Twisted swallow? In what manner do you have to be "careful"? > What characterizes a protocol's "happiness" beyond its momentary > connected-ness? How would you "miss" an error? > > Twisted is event-driven, so pretty much all the state changes you're > interested in are delivered as events (method calls on your object). If you > are "missing" them because Twisted isn't calling them, that sounds like a > really serious bug we should investigate. If you're missing them for some > other reason then you should just fix your code so it doesn't miss them any > more :). Regarding swallowing errors: I'm not claiming Twisted has bugs in this area (though I found and reported one obscure case that may be a bug: errors in tk command ::tk::Mac::quit are silently ignored). However, I found it rather easy to make coding errors that cause Twisted to not report errors. I have heard the same complaint from colleagues. At this point I think my code is robust, though I will be more confident once I finish my unit tests. If there is a "best practices for error handling" document I'd love to read it. I found an overview of deferreds that was helpful. It pointed out that addCallbacks is not the same as addCallback followed by addErrback and I'm not sure I'm handling the difference correctly. Right now I use addCallbacks(callback, errback). This does not call the errback if the callback fails. I do this intentionally because I want the errback to focus on problems with the connection, not my reaction to it, and the default error handler seems to report errors in the callback, which is fine. Regarding state: I'm used to event-driven frameworks that both offer callbacks and the ability to query state. I find it helpful for reporting errors as soon as possible (e.g. before writing check if the socket is connected; raise an exception if not). For a TCP/IP socket, I view a socket as having these states: - connecting - connected - disconnecting (with associated reason) - disconnected (with associated reason) TCP/IP Servers have a similar set of states, with connected -> listening However, it's certainly true that being able to query state is not essential; callbacks suffice. -- Russell ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] unit testing question: twisted with tk
I have a bit of code that combines twisted with Tkinter and now I'd like to write some unittests for it. I can't seem to figure out how to get TwistedTrial to handle this case. I've written a unit test like this: from twisted.trial import unittest from twisted.internet.defer import Deferred import twisted.internet.tksupport root = Tkinter.Tk() twisted.internet.tksupport.install(root) from twisted.internet import reactor class TestTkSocket(unittest.TestCase): def test...(...): the one test so far returns a deferred and calls errback on that deferred if the test fails, and callback if it succeeds. When I run the unit test with trial I get this error error: twisted.trial.util.DirtyReactorAggregateError I have an identical unit test for a pure-twisted version of my communication code (no Tk) and it works fine. I am guessing the problem is cleanly getting rid of Tkinter's root at the end of the test and disentangling it from the reactor, but I have no idea how to do that. Any suggestions on how to proceed (e.g. a way to get TwistedTrial to work, or a good alternative)? -- Russell ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] deferred graph?
On 02:24 am, drsali...@gmail.com wrote: >On Tue, Jul 17, 2012 at 12:53 PM, wrote: >>On 03:40 pm, drsali...@gmail.com wrote: >> >> >> >>Strangely, this doesn't give the report until after the sleep >> >>finishes... ? >> >>What's strange about that? "time.sleep(10)" doesn't mean "immediately >>print out debug information". >Well, if you read the whole thread, I believe you'll see that this was >intended to print some debugging information without needing to wait >for >the process to terminate. I don't think anything in the thread suggested that this approach will circumvent a time.sleep(10) call. In any case, it won't. Why is there a time.sleep(10) call there at all? Does it do anything except cause a problem? Jean-Paul > >> >> >However, it always appears to print the debug tracebacks for the >> >deferred >> >named - in this case, d. It doesn't appear to be selecting the >>correct >> >deferred(s) to print. >> >> >>All it does is print the debug information for the Deferred you got >>the >>_debugInfo attribute from. > >Um, yes, I was kind of saying that. Sadly though, this renders it >ineffective for the purpose it was suggested for. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
