Anyone know how do you determine the buffer size of a transport, to know how much data is waiting to be transmitted from using transport.write?
Or how you would go about adding that ability to a reactor/transport? On Wed, Aug 17, 2016 at 3:43 PM, Steve Morin <steve.mo...@gmail.com> wrote: > Twisted Community > > Problem: How do you determine the buffer size of a transport, to know how > much data is waiting to be transmitted from using transport.write? > > Wait! You're going to say: use the Producer Consumer API ( > http://twistedmatrix.com/documents/current/core/howto/producers.html ) > > To do what: So that instead of using back pressure I can check the buffer > and when it's "too big/full" can decide to do something to the transport I > am writing to: > > Slow transport handling options: > > - Buffer to disk instead of memory > - Kill the transport > - Decide to skip sending some data > - Send an error or message to the transport I am writing to > - Reduce the resolution, increase the compression (things like video or > audio) > > Why not use back pressure?: Some use-cases and some protocols this doesn't > make sense. > > Examples: > - You're sending video and if the receiver can't keep up you want to > downgrade or upgrade the quality of the video, but if you don't know if you > can't tell how much it buffering. > - You're receiving from one connection and then broadcasting what you > received to multiple clients and you want to handle it by sending an error > to a client that doesn't keep up > - You're sending a real-time protocol and want to skip sending some data > that's no longer relevant if the buffer is too slow. > > On a server what are the consequences: > Too much buffering in many transport write buffer can cause a server to > fail. I don't know how to keep track of this to proactively without access > to the buffer sizes. Resolutions can then be to, not accept new > connections when memory pressure is too high, kill connections with the > weakest/slowest clients or have a protocol that can have client switch > connections to new servers when instructed to spread out the load. > 1) I would like to hear on how people would solve this sort of problem in > Twisted for a server? > 2) I would like to hear people opinions on this in general. > > Tobias Oberstein - BCC'ed you on this email because you seems to have > tackled similar problems (based on the mailing list) and would really love > to get your take on this too. > > Glyph and Jean-Paul, you're also big on those threads so any opinions you > have would be appreciated as well. > > Some of my background research > > * http://twistedmatrix.com/pipermail/twisted-python/2015- > January/029064.html > * Later but good in the chain: http://twistedmatrix.com/ > pipermail/twisted-python/2015-January/029071.html > * Twisted receiving buffers swamped? > * Summary: Great thread but runs into a tangent for a while and picks up > good at the end again discussing producer again and the need for > backpressure > * Other: > * http://crossbar.io/ > * http://tavendo.com/ > * Scenario: "Twisted is reading off the TCP stack from the kernel and > buffering in userspace faster than the echo server is pushing out stuff to > the TCP stack into the kernel. Hence, no TCP backpressure results, netperf > happily sends more and more, and the memory of the Twisted process runs > away." > * Confirmed: Data isn't Buffered in "userspace inside Twisted, and > before data is handled by the app in dataReceived." > * https://twistedmatrix.com/pipermail/twisted-python/2010- > September/022900.html > * How to cap the buffering size of data to be sent in Protocol class > * Summary: Same issue, very short no good info > * https://twistedmatrix.com/pipermail/twisted-python/2012- > March/025416.html > * Limit on transport.write > * Summary: Similar issue, very short no good info, glyph confirms that > transport.write buffers everything sent to it. > * https://twistedmatrix.com/pipermail/twisted-python/2012- > February/025215.html > * pushing out same message on 100k TCPs > * Summary: Interesting discussion but different issue - interesting > aside about irc spanning trees for a large broadcasts > * http://twistedmatrix.com/pipermail/twisted-python/2008-April/017503.html > * Question on push/pull producers inter-workings, was : "Is there a > simple Producer/Consumer example or tutorial?" > * Summary: Related and goes into what a producer is - an explanation of > it > * http://twistedmatrix.com/pipermail/twisted-python/2004-May/007732.html > * When do calls to transport.write() block ? > * Summary: Discusses the right issue, talks about buffer call back if > full which would be great (if configurable) > * https://pythonhosted.org/qbuf/qbuf.twisted_support. > MultiBufferer-class.html > * https://launchpad.net/qbuf > * Summary: c buffer implementation that's incomplete thought might be > promising > * http://stackoverflow.com/questions/21821119/twisted- > producer-to-many-clients-with-buffer > * Summary: Was indicating a similar use-case but source doesn't seem to > exist on the internet > * http://twistedmatrix.com/documents/current/core/howto/producers.html > * Summary: Documentation on Producer and Consumers but only help with a > backpressure scenario > * http://twistedmatrix.com/pipermail/twisted-python/2007-July/015690.html > * Summary: Discussion on how producer-consumer API's work and future > enhancement no help > * MANY MORE THAT 100% NOT RELEVANT > > > -- > Steve Morin | Hacker, Entrepreneur, Startup Advisor > twitter.com/SteveMorin | stevemorin.com > *Live the dream start a startup. Make the world ... a better place.* > -- *Steve Morin | Managing Partner - CTO* *Nvent* O 800-407-1156 ext 803 <800-407-1156;803> | M 347-453-5579 smo...@nventdata.com <smo...@nventdata.com> *Enabling the Data Driven Enterprise* *(Ask us how we can setup scalable open source realtime billion+ event/data collection/analytics infrastructure in weeks)* Service Areas: Management & Strategy Consulting | Data Engineering | Data Science & Visualization
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python