Thank you again! The data IS in a file, but it requires an application to extract the requested salient pieces. I will look at the file wrapper extension.
After more testing, I now think that I can fix my problem just by using your second suggestion of increasing the Timeout configuration variable in Apache. That is an easy fix and so far seems to be working well. On Thursday, January 11, 2024 at 4:22:38 PM UTC-6 Graham Dumpleton wrote: > If using the file_wrapper feature, make sure you also add: > > WSGIEnableSendfile On > > to mod_wsgi configuration as not on by default: > > > https://modwsgi.readthedocs.io/en/master/release-notes/version-4.1.0.html#features-changed > > The file_wrapper mechanism would still have worked, but to use kernel > sendfile feature have to also have the directive enabled. > > Can't remember if also need to add: > > EnableSendfile On > > to enable it in Apache itself. I don't think so. > > Graham > > On 12 Jan 2024, at 9:14 am, Graham Dumpleton <graham.d...@gmail.com> > wrote: > > Also not sure whether it will help or not, but if the data you are sending > is stored in a file and not generated on demand, then you might consider > using the WSGI file_wrapper extension instead. > > > https://modwsgi.readthedocs.io/en/master/user-guides/file-wrapper-extension.html > > I don't know how this will behave when buffer fills up since when working > properly it is all handled in the OS kernel and not in Apache. > > Along similar lines, if is stored as a file, you might try mod_sendfile. > It also would use kernel sendfile mechanism, but way it interacts may also > see different behaviour in your situation. > > Graham > > On 12 Jan 2024, at 1:38 am, Greg Popp <pop...@gmail.com> wrote: > > Thank you very much! This is most helpful, though I don't think any of > them will actually solve my issues, for many of the reasons you mentioned. > > I was thinking that perhaps the mod_wsgi interface had access to the file > descriptor for the network socket used by Apache and could call "select" to > see if it had enough buffer space for the requested write. If it didn't, it > could (optionally) sleep some configurable duration and try again some > configurable number of times. I understand though, that for most > applications this would not be necessary. > > Yesterday, I tried implementing that same behavior in my wsgi app. I don't > set the SendBufferSize and so use the system default. I grab the system TCP > send buff value by running the 'sysctl' command. Then I keep track of the > total bytes sent. If that value exceeds the system tcp send queue value, I > run the 'ss' command from within my wsgi app to grab the Send-Q value for > this connection (fortunately wsgi gives us the source-ip and source-port > and I can filter the 'ss' output using that). If the Send-Q value is too > high to accommodate another write, I sleep a second and try again until I > get enough space. It's kind of a Rube Goldberg solution, but so far it > seems to be working! > > Thank you for taking the time to answer my questions! I very much > appreciate the assistance! > > On Wednesday, January 10, 2024 at 3:38:51 PM UTC-6 Graham Dumpleton wrote: > >> So what you are encountering is limitations in the socket buffer size >> enforced by the operating system, in combination with Apache httpd applying >> a socket timeout. >> >> In other words what happens is that the HTTP client isn't reading data >> and so the operating system level socket buffer fills up. At that point the >> Apache httpd write of the response data blocks with it eventually timing >> out, causing the initial error you see. In that situation Apache httpd will >> close down the connection, which results in you seeing the second error >> when still trying to write out more data anyway. >> >> You may be able to adjust some Apache configuration settings to try and >> solve this, but it would affect all requests in the context which you apply >> the configuration (dependent on whether done in server, VirtualHost, >> Directory or Location contexts). So not something you could selectively do >> on a per client basis. >> >> The first Apache directive to look at is SendBufferSize. >> >> https://httpd.apache.org/docs/2.4/mod/mpm_common.html#sendbuffersize >> >> If this is not set it should default to 0, which means it uses the >> operating system default. >> >> So you might be able to fiddle with this by setting it larger than the >> operating system default (although there is still some upper bound set by >> operating system you can go to). >> >> The next Apache directive to look at is Timeout. >> >> https://httpd.apache.org/docs/2.4/mod/core.html#timeout >> >> This would usually default to 60 seconds but some Linux distributions may >> override this in the Apache configuration they ship. >> >> In very old Apache versions this actually defaulted to 300 seconds, but >> it was made lower at some point. >> >> If playing with these, do be careful since they cause increased memory >> usage or cause other undesirable effects depending on traffic profile your >> server gets. >> >> One other thing you may be able to use is mod_ratelimit. >> >> https://httpd.apache.org/docs/2.4/mod/mod_ratelimit.html >> >> I have never actually used this and not exactly sure how it works, so is >> a bit of a guess, but you may be able to use this to slow down how quickly >> your application outputs the data. >> >> I am assuming here that this module will introduce waits into your >> application, by blocking your writes for a bit, to keep the flow of data >> being written by it under the rate limit. This would have the effect of not >> stuffing so much data into the response pipeline such that things work >> better with slower clients. Obviously using it would though penalise faster >> clients but you might find an acceptable balance by setting a higher rate >> limit for the initial burst of data and then using a lower rate after that. >> >> Graham >> >> >> On 11 Jan 2024, at 7:09 am, Greg Popp <pop...@gmail.com> wrote: >> >> embedded >> >> On Wednesday, January 10, 2024 at 1:32:52 PM UTC-6 Graham Dumpleton wrote: >> >>> Are you using mod_wsgi embedded mode or daemon mode? >>> >>> Graham >>> >>> On 11 Jan 2024, at 2:44 am, Greg Popp <pop...@gmail.com> wrote: >>> >>> Hello! >>> >>> My version of mod_wsgi is running on a Centos-7 system and is at version >>> 3.4, (I know - very old) with python 2.7 >>> >>> I have been using mod_wsgi for a python application that runs a >>> command-line program and marshals the output of the command line program >>> back to an http client. The data being sent is binary and can be tens of >>> gigs in size. >>> >>> This app is "unconventional", in that it calls 'write' directly, instead >>> of returning an iterable. The problem I have had recently, is that some >>> clients are slow to read the data and the TCP buffer gets filled up. When >>> this happens, the next call to write on a full buffer causes a "failed to >>> write data" exception (which I trap) but if I try again to send the data I >>> get "client connection closed". >>> >>> Is there some config setting or methodology I can use to alleviate this >>> issue? In other words, some way to back off and wait for the buffer to >>> drain sufficiently to resume sending the data? OR - is there some way to >>> get the current size (fullness) of the TCP write buffer on the connected >>> socket? (Something like what you see from the 'ss' command line utility >>> "Send-Q" column). If I could tell how full it is and what the max size is, >>> I could implement a sleep/retry cycle of some kind. >>> >>> I have looked - even in the source code - but haven't been able to >>> figure it out if there is a way to achieve this. Thanks in advance, for >>> your attention. >>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to modwsgi+u...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/modwsgi/3d97c06f-38ff-4345-af2f-eb86c2ef204cn%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/modwsgi/3d97c06f-38ff-4345-af2f-eb86c2ef204cn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to modwsgi+u...@googlegroups.com. >> >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/modwsgi/9fc6ab3e-b791-4503-a3c0-20ba273b92bdn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/modwsgi/9fc6ab3e-b791-4503-a3c0-20ba273b92bdn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> >> > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to modwsgi+u...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/2cdbd013-0f3f-4e38-bb05-dddbb5b0deaan%40googlegroups.com > > <https://groups.google.com/d/msgid/modwsgi/2cdbd013-0f3f-4e38-bb05-dddbb5b0deaan%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/f0ffd806-f527-4344-80bb-280d7348b1d9n%40googlegroups.com.
