Hi Specifically addressing your questions:
- I would look at using shared memory to store data that you can share across processes, plenty of resources on the web that talk about this. - You might want to use a small file to store data and fcntl.flock() to control access so you are not reading/writing at the same time. I run Django via wsgi which spawns multiple processes so there is nothing within Django per-se that you can use, you have to go outside, i.e. shared memory, database or file. François > On Aug 10, 2015, at 1:24 PM, Peter Coles <peter.m.co...@gmail.com> wrote: > > Thank you for your input. I appreciate you trying to steer my away from > potentially terrible design, however, I am still interested in someone > addressing my main questions. Some level of service calls is inevitable in > any interesting web application—even a call to an in-memory cache, like Redis > or Memcached, counts as a service call that could fail (a truly resilient > web-site would not fall over if the cache disappeared from DNS for 10 > minutes). In fact one of the examples I linked to was a circuit breaker for > memcached. > > I could leverage something like the local memory cache to store this very > light-weight information for a service client (which maybe would off-set that > disclaimer in the docs about not using it in production) or as seen in the > other examples, try something like storing values on a “global” class object > or instance. I was curious if anyone who has worked with stuff like this in > Django might have some pointers. Maybe Django Developers would be a better > forum? > > My original questions: > • What’s the most global, persisting, shared-across-requests type of > data I can read & write to memory? > • However I can do this, does this require using `threading.Lock()`? > (looking around `dispatch.Signal` I see some use of it) > • If this data is shared per-process, what’s a good way to see how many > processes it would be (for uwsgi is that just the # of worker processes)? > > > > On Monday, August 10, 2015 at 12:59:19 PM UTC-4, ke1g wrote: > In Django, requests should not wait, since the threads are relatively > heavyweight in python, and need to be a limited resource. The alternative is > a large number of processes, which is also expensive. > > So you must design a scheme in which the client polls for a result, meaning > that you cannot expect it to be handled by the same process, let alone > thread. So the success of a response (and its data) must be stored in a > resource shared across processes and threads. Most obviously, this is you > database, whether by and ORM Model, or (if small enough) in session data. > But it is also possible to use a secondary, in RAM store, such as redis, or > maybe memcached (or an ad hoc separate process, by why reinvent the wheel?). > > An alternative is to have these requests handled by something that does well > with Websockets, such as tornado. (Django may have work toward Websocket > support, but I'm not aware of anything satisfying.) > > Or, in the non-browser case of the client being able to provide a response > endpoint, a Celery task could retry, and wen successful could POST to that > endpoint. > > Doing Websockets or long poll type interactions isn't what Django is designed > to do well. Some machinations are appropriate if most of what you have to do > fits Django, but if all you need to do is drive a screw, don't waste time > filing a blade onto the face of a hammer. > > On Fri, Aug 7, 2015 at 5:54 PM, Peter Coles <peter....@gmail.com> wrote: > I’d like to write a circuit breaker wrapper to use with services that I call > from inside a Django app, and I’d like to get some pointers on the following > questions: > • What’s the most global, persisting, shared-across-requests type of > data I can read & write to memory? > • However I can do this, does this require using `threading.Lock()`? > (looking around `dispatch.Signal` I see some use of it) > • If this data is shared per-process, what’s a good way to see how many > processes it would be (for uwsgi is that just the # of worker processes)? > The term “circuit breaker” is referring to service clients that cut > themselves off from making future requests to a service after the service has > failed to connect or read/write after a certain error threshold. They can > employ retries to eventually right themselves. If a service continues to > fail, even despite strict timeouts, it can significantly slow down a site or > even cause the webserver to run out of threads and it will effectively reach > a DoS scenario. The clients are per-webserver (or maybe in this case, > per-process) and all independently can open or close their “circuits” based > on what they’re experiencing. > > To effectively track the effectiveness of service calls without having a > performance impact on the webserver, it really needs to store this info in > memory—which is the source of my original questions—and I assume this means > I’m stuck with only info per-process? I’m OK with the info disappearing after > server restarts, but might there be some gotchas around other configurations, > like the `--harakiri` option for uwsgi? As stated above, I’m also a little > unclear on when using threading locks would be required vs not, given that > python uses the GIL—but maybe certain configurations do allow for > non-thread-safe things to happen? > > I found 2 barebones approaches on github, but the correctness/effectiveness > of each was quite unclear: > • > https://github.com/cuker/django-patchboard/blob/master/patchboard/circuitbreaker.py > • > https://github.com/globocom/memcached_memoize/blob/master/memcached_memoize/decorators/circuit_breaker.py > Any help would be appreciated—whatever I build for this would definitely be > open-sourced to the community! > > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-users...@googlegroups.com. > To post to this group, send email to django...@googlegroups.com. > Visit this group at http://groups.google.com/group/django-users. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-users/35f791f2-d857-44de-a9e7-e52e346728b9%40googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-users+unsubscr...@googlegroups.com. > To post to this group, send email to django-users@googlegroups.com. > Visit this group at http://groups.google.com/group/django-users. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-users/1935fd88-1c8d-4cb3-aa67-ef3aa4b36e75%40googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/E6038848-35B4-46E6-877F-C02A94A23B8A%40gmail.com. For more options, visit https://groups.google.com/d/optout.
