On Oct 8, 12:36 am, Russell Keith-Magee <[email protected]>
wrote:
> > * Pylibmc 1.1 doesn't play that nice with mod_wsgi due to it's use of
> > the Simplified GIL API
> > (see:http://www.dctrwatson.com/2010/09/beware-of-using-pylibmc-1-1-and-mod...).
> > Probably just need to make note of it in the documentation, but it's
> > worth noting.
> ...
> > * Because it isn't bound by the GIL, pylibmc by default isn't thread
> > safe. There are a few different solutions, the most apparent being to
> > use pylibmc's ThreadMappedPool. More info here:
> >http://lericson.blogg.se/code/2009/september/draft-sept-20-2009.html
> > andhttp://blog.sendapatch.se/2009/september/pooling-with-pylibmc-pt-2.html
>
> Am I missing something, or aren't these variants of the same problem?
These are two different problems. 1.1 Specifically doesn't play nice
with mod_wsgi's lack of Simplified GIL API, all versions of pylibmc
are not multi-thread, multiprocessing safe out of the box
>
> > Originally Myself and Noah Silas were working on a solution that would
> > have two separate backends, one using of the ThreadMappedPool and one
> > using the basic client. At this point, I think it's probably not even
> > worth using one that uses the basic client, and having us keep one
> > memached backend that can use multiple libraries (defaulting to python-
> > memcached, and having initially pylibmc support as well.)
> > ing.
>
> I'm a little confused on the state of play here. If I'm understanding
> this correctly:
>
> * In the simple/naive usage of the API ('basic client') pylibmc isn't
> thread-safe, and therefore can't be used under mod_wsgi without
> special configuration.
>
> * It is possible to use a ThreadMappedPool to access the pylibmc
> client in a thread-safe way.
>
> If this is the case, a ThreadMappedPool-based implementation sounds
> like the only viable option; it doesn't strike me as a good idea for
> Django to ship a cache backend that doesn't play well with mod_wsgi
> without special configuration.
>
> I'm also slightly unclear as to exactly what solution you're proposing. Is it:
>
> 1) A completely new "pylibmc" backend implementing a ThreadMappedPool
> interface (possibly inheriting from the memcached backend or a
> factored base class)
>
> 2) A revised import order inside the existing memcached backend that
> would import pylibmc before or after attempted imports of cmemcache
> and memcache
>
> 3) A configuration option for the existing memcached backend that
> controls which client library is used
>
> The older mailing list thread seems to suggest (3) as the right
> approach, but it isn't clear to me that this is still what you're
> proposing.
>
3) Is where I suggest we go, but it also depends on how we decide to
handle the thread safety issue--and I wanted to pose it to the
community to make sure there's no drawback to forcing users of pylibmc
to use ThreadMappedPool if they want to use pylibmc. I can't think of
any, but I may be missing a use case. As Jeremy noted, there's a few
different ways to solve the thread safety issues (tying the client to
a thread local is another).
> Also, are there any API-level discrepancies remaining that need to be
> considered? The earlier django-dev thread suggests that there are some
> problems with get_multi(), but it also says that they've been fixed.
> Is this the case?
>
There are (how they handle atomic operations to non-existant keys
comes to mind), although these are fairly minor/don't require a design
decision on how to handle them, and will be addressed in the first
patch candidate we put up.
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
