Re: [Twisted-Python] Twisted receiving buffers swamped?
I am aware of http://twistedmatrix.com/documents/14.0.0/core/howto/producers.html, but that seems to cover the sending side only. It covers the receiving side as well. If you pauseProducing() on a transport, it stops calling dataReceived on its transport. Not sure I understand that. But you say, this will stop Twisted reading incoming data from a socket into userspace? And hence TCP backpressure results? What's the cause? What can I do? My initial hypothesis is that netperf is sending traffic but not bothering to receive it. I haven't looked through the netperf sources .. but I guess netperf will send as fast as the receiving side can digest .. only throttle down because of TCP backpressure, not app-level flow-control. If this hypothesis is correct, then self.transport.registerProducer(self.transport) should solve the problem. Presuming that there is no problem with crossing the streams - I don't think i've ever done that particular incantation, and I'm almost shocked it's taken this long to come up :). Unfortunately, it doesn't seem to work (the problem persists): https://github.com/oberstet/scratchbox/blob/master/python/asyncio/tcp_echo_server_tx.py http://picpaste.com/pics/Clipboard07-HlqQmTW0.1420188656.png Btw: the problem also arises when running over real network .. at least fast networks. I tested on fully switched 10GbE. And: asyncio (Trollius that is), has the same issue. The target server just gets swamped .. and then killed. Cheers, /Tobias ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted receiving buffers swamped?
Unfortunately, it doesn't seem to work (the problem persists): Your streaming flag is wrong. A TCP transport is an IPushProducer (it will produce data without being asked). Try setting it to True and see if that helps? With streaming == True and cProfile added https://github.com/oberstet/scratchbox/blob/master/python/asyncio/tcp_echo_server_tx.py I get strange results. Sluggish performance: [oberstet@brummer1 ~]$ netperf -N -H 127.0.0.1 -t TCP_STREAM -l 10 -- -P 9000 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 9000 AF_INET to 127.0.0.1 () port 9000 AF_INET : no control : histogram : interval : dirty data : demo Recv SendSend Socket Socket Message Elapsed Size SizeSize Time Throughput bytes bytes bytessecs.10^6bits/sec 0 32768 3276810.03 0.31 === That is 310kb/s. Which is totally slow. The native C based netserver that comes with netperf does 46Gb/s on this test. With streaming == False, I get something like 11Gb/s with that Twisted server (until it collapsed due to OOM) The CPU load is very low (near 0%). The memory stays flat. Here is the profile: https://github.com/oberstet/scratchbox/blob/master/python/asyncio/profile.log /Tobias ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted receiving buffers swamped?
On Jan 2, 2015, at 4:06 AM, Tobias Oberstein tobias.oberst...@tavendo.de wrote: Fact is: somehow memory runs away. How do I track down _where_ exactly the mem is spent? Probably that leads to the why then .. The first place to look - since sometimes looking in a specific place makes memory profilers easier to use - would be twisted.internet.tcp.Server._tempDataBuffer. The fact that it's extremely slow when you turn on consumer/producer logic in this way makes sense to me. twisted.internet.abstract.FileDescriptor.bufferSize is hard-coded to 65,536 bytes; every time the write side outpaces the read side by that buffer size, it will result in a call to (in your case) twisted.internet.kqreactor.KQueueReactor._updateRegistration, which makes the kcontrol syscall. One thing you might try is to run with the environment variable PYPYLOG=jit-summary:- set. This will give you a bunch of statistics about what the JIT did at the end of the run, on standard out; compare the fast (and run out of memory) to the slow (and work right) run to see what the differences are. -g___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted receiving buffers swamped?
Now, my suspicion is that Twisted is reading off the TCP stack from the kernel and buffering in userspace faster than the echo server is pushing out stuff to the TCP stack into the kernel. Hence, no TCP backpressure results, netperf happily sends more and more, and the memory of the Twisted process runs away. What you said here about buffering in userspace is ambiguous. It's not clear if you meant data is being buffered in userspace on the read side before your protocol gets a chance to handle it Yes. That's what I meant. Buffering in userspace inside Twisted, and before data is handled by the app in dataReceived. .. or if you meant that data being written to the transport by the protocol is being buffered in userspace. Nope, I didn't meant that. That's the sending side. The former doesn't happen. There are no no userspace read buffers in Twisted between the transport and the protocol. Bytes are read from the Ok. socket and then passed to dataReceived. The latter would be addressed by using producer/consumer APIs as Glyph suggested. Mmh. Fact is: somehow memory runs away. How do I track down _where_ exactly the mem is spent? Probably that leads to the why then .. /Tobias ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted receiving buffers swamped?
On 1 Jan, 10:21 am, tobias.oberst...@tavendo.de wrote: Hi, I am doing network performance tests using netperf on a trivial Twisted TCP echo server (code at the end). One of the tests that netperf offers is throughput, and I am running into an issue with this. ? [snip] Now, my suspicion is that Twisted is reading off the TCP stack from the kernel and buffering in userspace faster than the echo server is pushing out stuff to the TCP stack into the kernel. Hence, no TCP backpressure results, netperf happily sends more and more, and the memory of the Twisted process runs away. What you said here about buffering in userspace is ambiguous. It's not clear if you meant data is being buffered in userspace on the read side before your protocol gets a chance to handle it or if you meant that data being written to the transport by the protocol is being buffered in userspace. The former doesn't happen. There are no no userspace read buffers in Twisted between the transport and the protocol. Bytes are read from the socket and then passed to dataReceived. The latter would be addressed by using producer/consumer APIs as Glyph suggested. Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted receiving buffers swamped?
On 12:06 pm, tobias.oberst...@tavendo.de wrote: Fact is: somehow memory runs away. How do I track down _where_ exactly the mem is spent? Probably that leads to the why then .. There are memory profiling tools for Python. For example, memory_profiler: https://pypi.python.org/pypi/memory_profiler As far as I know, there are none that are specific to Twisted. Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted receiving buffers swamped?
On Jan 2, 2015, at 2:25 AM, Tobias Oberstein tobias.oberst...@tavendo.de wrote: Unfortunately, it doesn't seem to work (the problem persists): Your streaming flag is wrong. A TCP transport is an IPushProducer (it will produce data without being asked). Try setting it to True and see if that helps? -g___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
