Here is code demonstrating what I am talking about. Wrap the Throttle WSGI
middleware wrapper around your existing WSGI application object, specifying
the number of tokens you want to be able to dole out in that process.
application = Thottle(application, tokens=1, timeout=0.1)
That is, the maximum number of concurrent requests which should be allowed
to execute within the wrapped WSGI application. The number of threads the
process had, should be at least 1 more than the number of tokens
specified. The timeout should then be the maximum amount of time that a new
request is permitted to wait to get the token and then execute the request.
The length of the timeout may dictate if you need to have more than just
the 1 extra thread in the process, as you want queued requests to be
waiting on the queue object managed by this wrapper, not simply backlogged
in the listener socket queue where you have no idea how long they have been
there. As I said mentioned before, you could start monitoring queue time
for requests in daemon mode to combat the latter if number of threads
specified for waiting is made sufficient, but left that out for now.
Although people are welcome to use this in your own code, I ask that people
refrain from blogging or writing about it to allow me to blog about it at
at some point myself. Any version I write about later will be a lot more
comprehensive anyway, as I will add generation of metrics so one can track
wait times for requests, how many timeout etc etc. I will also detail
various other use cases for such a concept if I blog about it. Should be
worth the wait. :-)
from Queue import Queue, Empty
class Generator:
def __init__(self, iterable, callback):
self._iterable = iterable
self._callback = callback
def __iter__(self):
for item in self._iterable:
yield item
def close(self):
try:
if hasattr(self._iterable, 'close'):
self._iterable.close()
finally:
self._callback()
class Throttle:
def __init__(self, application, tokens, timeout=0.0):
self._application = application
self._timeout = timeout
self._tokens = Queue()
for i in range(tokens):
self._tokens.put(i)
def __call__(self, environ, start_response):
try:
token = self._tokens.get(timeout=self._timeout)
except Empty:
status = '503 Service Unavailable'
output = 'Service has reached capacity.'
response_headers = [('Content-type', 'text/plain'),
('Content-Length', str(len(output)))]
start_response(status, response_headers)
return [output]
try:
result = self._application(environ, start_response)
except:
_release()
raise
return Generator(result, lambda: self._tokens.put(token))
On 2 March 2013 18:53, Graham Dumpleton <[email protected]> wrote:
> You can do it with mod_wsgi daemon mode, but it doesn't entail needing to
> do anything special with Apache or mod_wsgi. You just need to implement a
> smart gating system as a WSGI middleware wrapper around your application.
>
> If your SOAP code is not known to be thread safe, or you still want to
> stay with only one SOAP handler running in a process at a time, then you
> allocate one or more threads on top of the number of threads to handle the
> SOAP request (one in this case). In other words, you might use:
>
> WSGIDaemonProcess processes=3 threads=3
>
> Then you could write a WSGI middleware around the SOAP application entry
> point which uses a threading.Queue object into which you initially add one
> token object (or as many as you want really as far as how many concurrent
> SOAP requests you want to handle in that process).
>
> When a new request arrives, it does a queue get to get one of the tokens,
> supplying a timeout value for how long it is prepared to wait. If it times
> out, you return a 503 HTTP response from the WSGI application. If instead
> it got the token, then it would continue on to execute the SOAP request.
> When done with the SOAP request, it would return the token back onto the
> queue object so another request can get the token.
>
> If you want to be even smarter and deal with potential for backlog and
> clearing it as quickly as possible, you would ensure you are using mod_wsgi
> 3.4 and in the WSGI middleware look at the timestamp (microseconds since
> epoch) in mod_wsgi.queue_start of WSGI environ, comparing that to the
> current time. If the difference is over a certain amount, you might decide
> that the request has been queued too long and no point returning a response
> because the client may have timed out or canceled request anyway, and
> return a 503 without even trying to get the token from the queue.
>
> So it is possible to control, but how you set the capacity (number of
> tokens), timeout and number of processes and threads still really depends
> on traffic and response times, which you need monitoring for.
>
> If I had the time I would probably blog about this specific approach for
> handling backlog as it ties in with issues I have explained before about
> how backlog can occur when using daemon mode.
>
> Graham
>
>
> On 1 March 2013 23:48, Thomas Guettler <[email protected]> wrote:
>
>> Sorry for the delay, I was not connected the last two days.
>>
>> Thank you for you hints how to optimize my current setup. You are right,
>> the worker MPM is better in my environment.
>> I know newrelic could help me to optimize my setup even more.
>>
>> But I still see a problem: The soap client is a powerful machine in the
>> same LAN, but outside my control and it can send a lot of requests to my
>> server. It could be compared to a denial of service attack. The soap client
>> must not
>> get all slots of the apache connection pool, since there are other (more
>> important) wsgi applications.
>>
>> Is there really no possibility to tell apache or mod_wsgi this: "don't
>> put more than N requests in to the connection pool for this wsgi-app"?
>>
>> If not, I need to implement the answer given on stackoverflow (mod_proxy
>> and gunicorn).
>>
>> Thomas
>>
>>
>> On 27.02.2013 23:56, Graham Dumpleton wrote:
>>
>>> So I could suggest you use worker MPM with a greater number for
>>> MaxClients. Because it is a multithread MPM, would be
>>> less processes and so less memory and thus could handle queueing more
>>> requests, but even if you do that, you are still
>>> going to restrict yourself because you are allowing so few SOAP requests
>>> to run so you really need to address that as well.
>>>
>>> What I cant tell is what the traffic is you are getting for SOAP
>>> requests and how CPU intensive they really are. They
>>> may well take 5 seconds to handle a request, but is it CPU bound or is
>>> it waiting on I/O with a backend service or
>>> database. If it isn't CPU bound then trying to limit the number of
>>> processes it runs on wouldn't serve any purpose.
>>>
>>> If it isn't CPU bound (and even if it is to a degree) and as long as
>>> your SOAP code is thread safe, you can probably do
>>> things better by by using multithread in daemon process for SOAP code as
>>> well.
>>>
>>> Either way, you really need to get a better idea on amount of traffic
>>> for SOAP requests, plus get an idea of how much
>>> SOAP requests are backing up by watching queue time, which is time from
>>> when Apache accepted the request and when SOAP
>>> code started handling it.
>>>
>>> As biased and self serving as it may seem since I work there and wrote
>>> the Python agent for it, this is a case where New
>>> Relic would really tell you what is going on so you can better tune the
>>> server.
>>>
>>> So, suggest you have a look at newrelic.com <http://newrelic.com> and
>>> if it is something you are able to install and get
>>>
>>> running, can then walk you through some possible extra setup for it and
>>> can then show you visually what your problems
>>> are and what you need to do.
>>>
>>> Graham
>>>
>>>
>>>
>>>
>>> On 26 February 2013 23:25, Thomas Guettler <[email protected] <mailto:
>>> [email protected]>> wrote:
>>>
>>> Hi Graham, thank you for your patience.
>>>
>>>
>>>
>>> On 26.02.2013 12 <tel:26.02.2013%2012>:47, Graham Dumpleton wrote:
>>>
>>> Sorry, two more questions.
>>>
>>> Is you SOAP code thread safe?
>>>
>>>
>>> I don't know. Up to now I avoided threads. We use threads=1 in the
>>> WSGIDaemonProcess config.
>>>
>>>
>>> Is there a particular reason you are using Apache prefork MPM
>>> given that all main Apache processes appear to be
>>> doing is
>>> serving static files and proxying to mod_wsgi daemons?
>>>
>>>
>>> No, there is no reason except "it was always done like this in the
>>> past".
>>> You are right, we waste a bit of performance. But I don't think it
>>> is much.
>>>
>>>
>>> Sleep time for me. Will see what you say tomorrow.
>>>
>>>
>>> Thank you for your fast replies. Good night .... good morning.
>>>
>>>
>>> Thomas Güttler
>>>
>>>
>>> --
>>> Thomas Guettler, http://www.thomas-guettler.de/
>>> E-Mail: guettli (*) thomas-guettler + de
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "modwsgi" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>>> send an email to
>>> modwsgi+unsubscribe@__googlegr**oups.com
>>> <http://googlegroups.com><mailto:
>>> modwsgi%2Bunsubscribe@**googlegroups.com<modwsgi%[email protected]>
>>> >.
>>> To post to this group, send email to [email protected]<mailto:
>>> modwsgi@googlegroups.**com <[email protected]>>.
>>> Visit this group at
>>> http://groups.google.com/__**group/modwsgi?hl=en<http://groups.google.com/__group/modwsgi?hl=en><
>>> http://groups.google.com/**group/modwsgi?hl=en<http://groups.google.com/group/modwsgi?hl=en>
>>> >.
>>> For more options, visit
>>> https://groups.google.com/__**groups/opt_out<https://groups.google.com/__groups/opt_out><
>>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
>>> >.
>>>
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "modwsgi" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to
>>> modwsgi+unsubscribe@**googlegroups.com<modwsgi%[email protected]>
>>> .
>>> To post to this group, send email to [email protected].
>>> Visit this group at
>>> http://groups.google.com/**group/modwsgi?hl=en<http://groups.google.com/group/modwsgi?hl=en>
>>> .
>>> For more options, visit
>>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
>>> .
>>>
>>>
>>>
>> --
>> Thomas Guettler, http://www.thomas-guettler.de/
>> E-Mail: guettli (*) thomas-guettler + de
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "modwsgi" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to
>> modwsgi+unsubscribe@**googlegroups.com<modwsgi%[email protected]>
>> .
>> To post to this group, send email to [email protected].
>> Visit this group at
>> http://groups.google.com/**group/modwsgi?hl=en<http://groups.google.com/group/modwsgi?hl=en>
>> .
>> For more options, visit
>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
>> .
>>
>>
>>
>
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/modwsgi?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
