#35757: memcached.PyMemcacheCache reentrancy problem with ASGI-based runserver
-------------------------------------+-------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Bug | Status: closed
Component: Core (Cache system) | Version: 5.1
Severity: Normal | Resolution: duplicate
Keywords: asgi, async | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Harm Verhagen):
I don't think its a duplicate of #33625 (although it seems related)
I reopened for reconsideration.
=== Why this is different?
#33625 is an optimization problem. All works fine, but could be optimized
for performance. The connection is opened/closed which adds (a bit of)
latency to each request. (which might be acceptable, or not depending on
your usecase)
This problem (#35757) however is about that the system actually breaks.
When running the test above. runserver stops working completely, until
restarted.
If you are developing a javascript app, that issues multiple requests
simultaniously, then runserver (in this configuration) is unusabe.
Another reason why its no duplicate. I can reproduce the problem also with
`django.core.cache.backends.memcached.PyLibMCCache` instead of
`'....memcached.PyMemcacheCache`.
PyLibMCCache already does provide connection pooling, so the optimization
requested in #33625 is not applicable for PyLibMCCache.
=== Reproducibility
Where you able to reproduce the problem with the example attached? (the
problem that runserver actually gives 500's and eventually stops working
completely)
=== Results with PyLibMCCache
NB: this might be a different problem, triggered by the same testcase
==== Result (PyLibMCCache)
When running requests in parallel, some of the requests get a 500.
{{{
Internal Server Error: /polls/
Traceback (most recent call last):
File "venv/lib/python3.12/site-packages/asgiref/sync.py", line 518, in
thread_handler
raise exc_info[1]
File "venv/lib/python3.12/site-
packages/django/core/handlers/exception.py", line 42, in inner
response = await get_response(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "venv/lib/python3.12/site-packages/asgiref/sync.py", line 518, in
thread_handler
raise exc_info[1]
File "venv/lib/python3.12/site-packages/django/core/handlers/base.py",
line 253, in _get_response_async
response = await wrapped_callback(
^^^^^^^^^^^^^^^^^^^^^^^
File "venv/lib/python3.12/site-packages/asgiref/sync.py", line 468, in
__call__
ret = await asyncio.shield(exec_coro)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/homebrew/Cellar/python@3.12/3.12.6/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/thread.py",
line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "venv/lib/python3.12/site-packages/asgiref/sync.py", line 522, in
thread_handler
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "django-35757/mysite/polls/views.py", line 22, in index
v = incr()
^^^^^^
File "/tmp/dbg/django-35757/mysite/polls/views.py", line 11, in incr
val = cache.incr(key, 1)
^^^^^^^^^^^^^^^^^^
File
"/Users/harmverhagen/Documents/symeon/intemo/sensoren/cityznz/cityzenz/api/venv/lib/python3.12
/site-packages/django/core/cache/backends/memcached.py", line 110, in incr
val = self._cache.incr(key, delta)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pylibmc.Error: 1 keys failed
HTTP GET /polls/ 500 [5.13, 127.0.0.1:63568]
}}}
NB: Sometimes I even noticed a python crash !
{{{
HTTP GET /polls/ 200 [0.11, 127.0.0.1:63819]
Python(51613,0x17d037000) malloc: double free for ptr 0x13980a400
Python(51613,0x17d037000) malloc: *** set a breakpoint in
malloc_error_break to debug
}}}
So, maybe the `PyLibMCCache` issue is something different
--
Ticket URL: <https://code.djangoproject.com/ticket/35757#comment:15>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/010701924d32efa7-338c52b7-7454-4952-8410-5d4603be57fb-000000%40eu-central-1.amazonses.com.
