#32815: Failed to reset ContextVars in sync/async middlewares
-------------------------------------+-------------------------------------
Reporter: Michael Manganiello | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.2
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Description changed by Michael Manganiello:
Old description:
> When using a middleware that can process both sync and async requests,
> and trying to set and reset a ContextVar (in different methods of its
> request lifecycle), Python fails with error:
> `ValueError: <Token var=<ContextVar name='current_context' at
> 0x7f9a8b9ad900> at 0x7f9a68575180> was created in a different Context`
>
> This is a simple middleware example to reproduce the mentioned issue:
>
> {{{
> @sync_and_async_middleware
> class TemplateResponseMiddleware(BaseMiddleware):
> def process_view(self, request, view_func, view_args, view_kwargs):
> request.META['_CONTEXT_RESET_TOKEN'] =
> current_context.set(id(request))
>
> def process_template_response(self, request, response):
> current_context.reset(request.META['_CONTEXT_RESET_TOKEN'])
> response.context_data['mw'].append(self.__class__.__name__)
> return response
> }}}
>
> This use case is what the OpenTelemetry integration uses for spans to be
> traced in Django: https://github.com/open-telemetry/opentelemetry-python-
> contrib/blob/main/instrumentation/opentelemetry-instrumentation-
> django/src/opentelemetry/instrumentation/django/middleware.py
>
> * In `process_request`, a `ContextVar` is set, and the generated token is
> persisted in the `request.META` object.
> * In `process_response`, the `ContextVar` is reset, by using the
> persisted token.
>
> This approach works correctly for synchronous requests. However, as part
> of adding ASGI support to the Django integration for OpenTelemetry (in
> https://github.com/open-telemetry/opentelemetry-python-contrib/pull/391),
> we found that the `ContextVar` triggers the mentioned error when we want
> to reset it to its previous value. OpenTelemetry inherits from
> `MiddlewareMixin`, but I'm attaching a diff for a simple test scenario
> that reproduces the issue, using the new Middleware format.
>
> The main suspects here are the calls to `sync_to_async`, which adapt the
> middleware methods to the async flow. However, both those calls
> explicitly set `thread_sensitive=True`.
>
> Traceback for the attached test scenario:
>
> {{{
> $ ./runtests.py -k
> MiddlewareSyncAsyncTests.test_async_process_template_response
> # ...
> ERROR: test_async_process_template_response
> (middleware_exceptions.tests.MiddlewareSyncAsyncTests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/home/mike/.virtualenvs/django/lib/python3.9/site-
> packages/asgiref/sync.py", line 222, in __call__
> return call_result.result()
> File "/usr/lib/python3.9/concurrent/futures/_base.py", line 438, in
> result
> return self.__get_result()
> File "/usr/lib/python3.9/concurrent/futures/_base.py", line 390, in
> __get_result
> raise self._exception
> File "/home/mike/.virtualenvs/django/lib/python3.9/site-
> packages/asgiref/sync.py", line 287, in main_wrap
> result = await self.awaitable(*args, **kwargs)
> File "/mnt/data/Proyectos/third_party/django/django/test/utils.py",
> line 423, in inner
> return await func(*args, **kwargs)
> File
> "/mnt/data/Proyectos/third_party/django/tests/middleware_exceptions/tests.py",
> line 319, in test_async_process_template_response
> response = await self.async_client.get(
> File "/mnt/data/Proyectos/third_party/django/django/test/client.py",
> line 911, in request
> self.check_exception(response)
> File "/mnt/data/Proyectos/third_party/django/django/test/client.py",
> line 580, in check_exception
> raise exc_value
> File "/home/mike/.virtualenvs/django/lib/python3.9/site-
> packages/asgiref/sync.py", line 458, in thread_handler
> raise exc_info[1]
> File
> "/mnt/data/Proyectos/third_party/django/django/core/handlers/exception.py",
> line 38, in inner
> response = await get_response(request)
> File
> "/mnt/data/Proyectos/third_party/django/django/core/handlers/base.py",
> line 249, in _get_response_async
> response = await middleware_method(request, response)
> File "/home/mike/.virtualenvs/django/lib/python3.9/site-
> packages/asgiref/sync.py", line 423, in __call__
> ret = await asyncio.wait_for(future, timeout=None)
> File "/usr/lib/python3.9/asyncio/tasks.py", line 442, in wait_for
> return await fut
> File "/home/mike/.virtualenvs/django/lib/python3.9/site-
> packages/asgiref/current_thread_executor.py", line 22, in run
> result = self.fn(*self.args, **self.kwargs)
> File "/home/mike/.virtualenvs/django/lib/python3.9/site-
> packages/asgiref/sync.py", line 462, in thread_handler
> return func(*args, **kwargs)
> File
> "/mnt/data/Proyectos/third_party/django/tests/middleware_exceptions/middleware.py",
> line 135, in process_template_response
> current_context.reset(request.META['_CONTEXT_RESET_TOKEN'])
> ValueError: <Token var=<ContextVar name='current_context' at
> 0x7f1dd4ec2720> at 0x7f1db129a880> was created in a different Context
> }}}
New description:
When using a middleware that can process both sync and async requests, and
trying to set and reset a ContextVar (in different methods of its request
lifecycle), Python fails with error:
`ValueError: <Token var=<ContextVar name='current_context' at
0x7f9a8b9ad900> at 0x7f9a68575180> was created in a different Context`
This is a simple middleware example to reproduce the mentioned issue:
{{{
import contextvars
current_context = contextvars.ContextVar('current_context')
@sync_and_async_middleware
class TemplateResponseMiddleware(BaseMiddleware):
def process_view(self, request, view_func, view_args, view_kwargs):
request.META['_CONTEXT_RESET_TOKEN'] =
current_context.set(id(request))
def process_template_response(self, request, response):
current_context.reset(request.META['_CONTEXT_RESET_TOKEN'])
response.context_data['mw'].append(self.__class__.__name__)
return response
}}}
This use case is what the OpenTelemetry integration uses for spans to be
traced in Django: https://github.com/open-telemetry/opentelemetry-python-
contrib/blob/main/instrumentation/opentelemetry-instrumentation-
django/src/opentelemetry/instrumentation/django/middleware.py
* In `process_request`, a `ContextVar` is set, and the generated token is
persisted in the `request.META` object.
* In `process_response`, the `ContextVar` is reset, by using the persisted
token.
This approach works correctly for synchronous requests. However, as part
of adding ASGI support to the Django integration for OpenTelemetry (in
https://github.com/open-telemetry/opentelemetry-python-contrib/pull/391),
we found that the `ContextVar` triggers the mentioned error when we want
to reset it to its previous value. OpenTelemetry inherits from
`MiddlewareMixin`, but I'm attaching a diff for a simple test scenario
that reproduces the issue, using the new Middleware format.
The main suspects here are the calls to `sync_to_async`, which adapt the
middleware methods to the async flow. However, both those calls explicitly
set `thread_sensitive=True`.
Traceback for the attached test scenario:
{{{
$ ./runtests.py -k
MiddlewareSyncAsyncTests.test_async_process_template_response
# ...
ERROR: test_async_process_template_response
(middleware_exceptions.tests.MiddlewareSyncAsyncTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/mike/.virtualenvs/django/lib/python3.9/site-
packages/asgiref/sync.py", line 222, in __call__
return call_result.result()
File "/usr/lib/python3.9/concurrent/futures/_base.py", line 438, in
result
return self.__get_result()
File "/usr/lib/python3.9/concurrent/futures/_base.py", line 390, in
__get_result
raise self._exception
File "/home/mike/.virtualenvs/django/lib/python3.9/site-
packages/asgiref/sync.py", line 287, in main_wrap
result = await self.awaitable(*args, **kwargs)
File "/mnt/data/Proyectos/third_party/django/django/test/utils.py", line
423, in inner
return await func(*args, **kwargs)
File
"/mnt/data/Proyectos/third_party/django/tests/middleware_exceptions/tests.py",
line 319, in test_async_process_template_response
response = await self.async_client.get(
File "/mnt/data/Proyectos/third_party/django/django/test/client.py",
line 911, in request
self.check_exception(response)
File "/mnt/data/Proyectos/third_party/django/django/test/client.py",
line 580, in check_exception
raise exc_value
File "/home/mike/.virtualenvs/django/lib/python3.9/site-
packages/asgiref/sync.py", line 458, in thread_handler
raise exc_info[1]
File
"/mnt/data/Proyectos/third_party/django/django/core/handlers/exception.py",
line 38, in inner
response = await get_response(request)
File
"/mnt/data/Proyectos/third_party/django/django/core/handlers/base.py",
line 249, in _get_response_async
response = await middleware_method(request, response)
File "/home/mike/.virtualenvs/django/lib/python3.9/site-
packages/asgiref/sync.py", line 423, in __call__
ret = await asyncio.wait_for(future, timeout=None)
File "/usr/lib/python3.9/asyncio/tasks.py", line 442, in wait_for
return await fut
File "/home/mike/.virtualenvs/django/lib/python3.9/site-
packages/asgiref/current_thread_executor.py", line 22, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/mike/.virtualenvs/django/lib/python3.9/site-
packages/asgiref/sync.py", line 462, in thread_handler
return func(*args, **kwargs)
File
"/mnt/data/Proyectos/third_party/django/tests/middleware_exceptions/middleware.py",
line 135, in process_template_response
current_context.reset(request.META['_CONTEXT_RESET_TOKEN'])
ValueError: <Token var=<ContextVar name='current_context' at
0x7f1dd4ec2720> at 0x7f1db129a880> was created in a different Context
}}}
--
--
Ticket URL: <https://code.djangoproject.com/ticket/32815#comment:1>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/068.c7af18c690d3a0a91d20f0e30ec54bb5%40djangoproject.com.
