#32815: Failed to reset ContextVars in sync/async middlewares -------------------------------------+------------------------------------- Reporter: Michael Manganiello | Owner: nobody Type: Uncategorized | Status: new Component: Uncategorized | Version: 3.2 Severity: Normal | Resolution: Keywords: | Triage Stage: | Unreviewed Has patch: 0 | Needs documentation: 0 Needs tests: 0 | Patch needs improvement: 0 Easy pickings: 0 | UI/UX: 0 -------------------------------------+------------------------------------- Description changed by Michael Manganiello:
Old description: > When using a middleware that can process both sync and async requests, > and trying to set and reset a ContextVar (in different methods of its > request lifecycle), Python fails with error: > `ValueError: <Token var=<ContextVar name='current_context' at > 0x7f9a8b9ad900> at 0x7f9a68575180> was created in a different Context` > > This is a simple middleware example to reproduce the mentioned issue: > > {{{ > import contextvars > > current_context = contextvars.ContextVar('current_context') > > @sync_and_async_middleware > class TemplateResponseMiddleware(BaseMiddleware): > def process_view(self, request, view_func, view_args, view_kwargs): > request.META['_CONTEXT_RESET_TOKEN'] = > current_context.set(id(request)) > > def process_template_response(self, request, response): > current_context.reset(request.META['_CONTEXT_RESET_TOKEN']) > response.context_data['mw'].append(self.__class__.__name__) > return response > }}} > > This use case is what the OpenTelemetry integration uses for spans to be > traced in Django: https://github.com/open-telemetry/opentelemetry-python- > contrib/blob/main/instrumentation/opentelemetry-instrumentation- > django/src/opentelemetry/instrumentation/django/middleware.py > > * In `process_request`, a `ContextVar` is set, and the generated token is > persisted in the `request.META` object. > * In `process_response`, the `ContextVar` is reset, by using the > persisted token. > > This approach works correctly for synchronous requests. However, as part > of adding ASGI support to the Django integration for OpenTelemetry (in > https://github.com/open-telemetry/opentelemetry-python-contrib/pull/391), > we found that the `ContextVar` triggers the mentioned error when we want > to reset it to its previous value. OpenTelemetry inherits from > `MiddlewareMixin`, but I'm attaching a diff for a simple test scenario > that reproduces the issue, using the new Middleware format. > > The main suspects here are the calls to `sync_to_async`, which adapt the > middleware methods to the async flow. However, both those calls > explicitly set `thread_sensitive=True`. > > Traceback for the attached test scenario: > > {{{ > $ ./runtests.py -k > MiddlewareSyncAsyncTests.test_async_process_template_response > # ... > ERROR: test_async_process_template_response > (middleware_exceptions.tests.MiddlewareSyncAsyncTests) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/mike/.virtualenvs/django/lib/python3.9/site- > packages/asgiref/sync.py", line 222, in __call__ > return call_result.result() > File "/usr/lib/python3.9/concurrent/futures/_base.py", line 438, in > result > return self.__get_result() > File "/usr/lib/python3.9/concurrent/futures/_base.py", line 390, in > __get_result > raise self._exception > File "/home/mike/.virtualenvs/django/lib/python3.9/site- > packages/asgiref/sync.py", line 287, in main_wrap > result = await self.awaitable(*args, **kwargs) > File "/mnt/data/Proyectos/third_party/django/django/test/utils.py", > line 423, in inner > return await func(*args, **kwargs) > File > "/mnt/data/Proyectos/third_party/django/tests/middleware_exceptions/tests.py", > line 319, in test_async_process_template_response > response = await self.async_client.get( > File "/mnt/data/Proyectos/third_party/django/django/test/client.py", > line 911, in request > self.check_exception(response) > File "/mnt/data/Proyectos/third_party/django/django/test/client.py", > line 580, in check_exception > raise exc_value > File "/home/mike/.virtualenvs/django/lib/python3.9/site- > packages/asgiref/sync.py", line 458, in thread_handler > raise exc_info[1] > File > "/mnt/data/Proyectos/third_party/django/django/core/handlers/exception.py", > line 38, in inner > response = await get_response(request) > File > "/mnt/data/Proyectos/third_party/django/django/core/handlers/base.py", > line 249, in _get_response_async > response = await middleware_method(request, response) > File "/home/mike/.virtualenvs/django/lib/python3.9/site- > packages/asgiref/sync.py", line 423, in __call__ > ret = await asyncio.wait_for(future, timeout=None) > File "/usr/lib/python3.9/asyncio/tasks.py", line 442, in wait_for > return await fut > File "/home/mike/.virtualenvs/django/lib/python3.9/site- > packages/asgiref/current_thread_executor.py", line 22, in run > result = self.fn(*self.args, **self.kwargs) > File "/home/mike/.virtualenvs/django/lib/python3.9/site- > packages/asgiref/sync.py", line 462, in thread_handler > return func(*args, **kwargs) > File > "/mnt/data/Proyectos/third_party/django/tests/middleware_exceptions/middleware.py", > line 135, in process_template_response > current_context.reset(request.META['_CONTEXT_RESET_TOKEN']) > ValueError: <Token var=<ContextVar name='current_context' at > 0x7f1dd4ec2720> at 0x7f1db129a880> was created in a different Context > }}} New description: When using a middleware that can process both sync and async requests, and trying to set and reset a ContextVar (in different methods of its request lifecycle), Python fails with error: `ValueError: <Token var=<ContextVar name='current_context' at 0x7f9a8b9ad900> at 0x7f9a68575180> was created in a different Context` This is a simple middleware example to reproduce the mentioned issue: {{{ import contextvars current_context = contextvars.ContextVar('current_context') @sync_and_async_middleware class TemplateResponseMiddleware(BaseMiddleware): def process_view(self, request, view_func, view_args, view_kwargs): request.META['_CONTEXT_RESET_TOKEN'] = current_context.set(id(request)) def process_template_response(self, request, response): current_context.reset(request.META['_CONTEXT_RESET_TOKEN']) return response }}} This use case is what the OpenTelemetry integration uses for spans to be traced in Django: https://github.com/open-telemetry/opentelemetry-python- contrib/blob/main/instrumentation/opentelemetry-instrumentation- django/src/opentelemetry/instrumentation/django/middleware.py * In `process_request`, a `ContextVar` is set, and the generated token is persisted in the `request.META` object. * In `process_response`, the `ContextVar` is reset, by using the persisted token. This approach works correctly for synchronous requests. However, as part of adding ASGI support to the Django integration for OpenTelemetry (in https://github.com/open-telemetry/opentelemetry-python-contrib/pull/391), we found that the `ContextVar` triggers the mentioned error when we want to reset it to its previous value. OpenTelemetry inherits from `MiddlewareMixin`, but I'm attaching a diff for a simple test scenario that reproduces the issue, using the new Middleware format. The main suspects here are the calls to `sync_to_async`, which adapt the middleware methods to the async flow. However, both those calls explicitly set `thread_sensitive=True`. Traceback for the attached test scenario: {{{ $ ./runtests.py -k MiddlewareSyncAsyncTests.test_async_process_template_response # ... ERROR: test_async_process_template_response (middleware_exceptions.tests.MiddlewareSyncAsyncTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/mike/.virtualenvs/django/lib/python3.9/site- packages/asgiref/sync.py", line 222, in __call__ return call_result.result() File "/usr/lib/python3.9/concurrent/futures/_base.py", line 438, in result return self.__get_result() File "/usr/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result raise self._exception File "/home/mike/.virtualenvs/django/lib/python3.9/site- packages/asgiref/sync.py", line 287, in main_wrap result = await self.awaitable(*args, **kwargs) File "/mnt/data/Proyectos/third_party/django/django/test/utils.py", line 423, in inner return await func(*args, **kwargs) File "/mnt/data/Proyectos/third_party/django/tests/middleware_exceptions/tests.py", line 319, in test_async_process_template_response response = await self.async_client.get( File "/mnt/data/Proyectos/third_party/django/django/test/client.py", line 911, in request self.check_exception(response) File "/mnt/data/Proyectos/third_party/django/django/test/client.py", line 580, in check_exception raise exc_value File "/home/mike/.virtualenvs/django/lib/python3.9/site- packages/asgiref/sync.py", line 458, in thread_handler raise exc_info[1] File "/mnt/data/Proyectos/third_party/django/django/core/handlers/exception.py", line 38, in inner response = await get_response(request) File "/mnt/data/Proyectos/third_party/django/django/core/handlers/base.py", line 249, in _get_response_async response = await middleware_method(request, response) File "/home/mike/.virtualenvs/django/lib/python3.9/site- packages/asgiref/sync.py", line 423, in __call__ ret = await asyncio.wait_for(future, timeout=None) File "/usr/lib/python3.9/asyncio/tasks.py", line 442, in wait_for return await fut File "/home/mike/.virtualenvs/django/lib/python3.9/site- packages/asgiref/current_thread_executor.py", line 22, in run result = self.fn(*self.args, **self.kwargs) File "/home/mike/.virtualenvs/django/lib/python3.9/site- packages/asgiref/sync.py", line 462, in thread_handler return func(*args, **kwargs) File "/mnt/data/Proyectos/third_party/django/tests/middleware_exceptions/middleware.py", line 135, in process_template_response current_context.reset(request.META['_CONTEXT_RESET_TOKEN']) ValueError: <Token var=<ContextVar name='current_context' at 0x7f1dd4ec2720> at 0x7f1db129a880> was created in a different Context }}} -- -- Ticket URL: <https://code.djangoproject.com/ticket/32815#comment:2> Django <https://code.djangoproject.com/> The Web framework for perfectionists with deadlines. -- You received this message because you are subscribed to the Google Groups "Django updates" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-updates+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/068.ac03ac5b6f7a72bf2e87ce5cf4eb3fb6%40djangoproject.com.