Hi all with periods of seemingly no activity the gc comes to mind.
Is the working set size stable or ever increasing, swap operations might block activity of the process I had problems in the beginning with twisted, because I was holding on to a lot of objects and thus creating circular non collectible object references leading to ever increasing work sets over time, with the gc hitting me hard once in a while. First check I do always is keep an eye on the size of the process at constant network load, size shrinking when network load is dwindling and increasing when network load is growing. If it is possible to manipulate network load precisely then returning to the same load should give you the same process size after some time repeatedly. As a reference I have twisted processes running with up times measured in years on FreeBSD. Werner On 5/14/19 05:39, Scott, Barry wrote: > On Monday, 13 May 2019 15:26:01 BST Chris Withers wrote: >> Hi All, >> >> On some busy instances of the app I'm working with/on, we see websocket >> connections being dropped due to not ping/ponging in time, I'm fairly >> sure this is as a result of some user-written callbacks blocking the >> reactor for long periods of time (seconds...). >> >> What's the best way to debug a twisted application to find out what's >> blocking the reactor? > Is the process CPU bound or IO bound? > > If CPU bound then cprofile or the tool Glyph mentioned would help. > > This advice is from a Linux perspective. > > If its IO bound the you may be waiting on a blocking operation. > You could use strace on the process to see what its doing. > > As its hung for seconds you could connect gdb to the process and see a python > backtrace using the py-bt command. You are likely to catch the offending code. > > The normal bt command will show you what the last python command it calling. > > You could adding logging to your code to show the time of callback entry and > exit. > > If all else fails patch the twisted code to time callbacks and report the > slow > ones. > > Barry > >> cheers, >> >> Chris >> >> _______________________________________________ >> Twisted-Python mailing list >> Twisted-Python@twistedmatrix.com >> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python