Re: [Django] #32815: Failed to reset ContextVars in sync/async middlewares

[Django]

#32815: Failed to reset ContextVars in sync/async middlewares
-+-
 Reporter:  Michael Manganiello  |Owner:  nobody
 Type:  Uncategorized|   Status:  closed
Component:  Uncategorized|  Version:  3.2
 Severity:  Normal   |   Resolution:  invalid
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Michael Manganiello):

 Thanks for the feedback. I've filed
 https://github.com/django/asgiref/issues/267 to find the root cause.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/068.7c5568c8b74e5a5d542cf38b7836152f%40djangoproject.com.

Re: [Django] #32815: Failed to reset ContextVars in sync/async middlewares

[Django]

#32815: Failed to reset ContextVars in sync/async middlewares
-+-
 Reporter:  Michael Manganiello  |Owner:  nobody
 Type:  Uncategorized|   Status:  closed
Component:  Uncategorized|  Version:  3.2
 Severity:  Normal   |   Resolution:  invalid
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * status:  new => closed
 * resolution:   => invalid


Comment:

 Thanks for this report, however it looks like a support question and Trac
 is not a support channel. Moreover you're discussing `asgiref` behavior
 not Django itself. I would recommend to open
 [https://github.com/django/asgiref/issues an issue] in `asgiref`.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/068.60cdea499688b544af523383733f7fc2%40djangoproject.com.

Re: [Django] #32815: Failed to reset ContextVars in sync/async middlewares

[Django]

#32815: Failed to reset ContextVars in sync/async middlewares
-+-
 Reporter:  Michael Manganiello  |Owner:  nobody
 Type:  Uncategorized|   Status:  new
Component:  Uncategorized|  Version:  3.2
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Michael Manganiello):

 It seems this is not an issue only related to middlewares, but to how
 `sync_to_async` works in general? I am able to reproduce the same issue
 with this simple endpoint (when running Django using Gunicorn with Uvicorn
 workers):

 {{{
 import contextvars

 from asgiref.sync import sync_to_async
 from django.http import HttpResponse
 from django.urls import path

 current_context = contextvars.ContextVar('current_context')


 async def healthcheck(request):
 token = await sync_to_async(current_context.set,
 thread_sensitive=True)(id(request))
 await sync_to_async(current_context.reset,
 thread_sensitive=True)(token)
 return HttpResponse('OK')

 urlpatterns = [path('', healthcheck)]
 }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/068.d7c6b97e727831b434ec8bd3c2ff5800%40djangoproject.com.

Re: [Django] #32815: Failed to reset ContextVars in sync/async middlewares

[Django]

#32815: Failed to reset ContextVars in sync/async middlewares
-+-
 Reporter:  Michael Manganiello  |Owner:  nobody
 Type:  Uncategorized|   Status:  new
Component:  Uncategorized|  Version:  3.2
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Description changed by Michael Manganiello:

Old description:

> When using a middleware that can process both sync and async requests,
> and trying to set and reset a ContextVar (in different methods of its
> request lifecycle), Python fails with error:
> `ValueError:  0x7f9a8b9ad900> at 0x7f9a68575180> was created in a different Context`
>
> This is a simple middleware example to reproduce the mentioned issue:
>
> {{{
> import contextvars
>
> current_context = contextvars.ContextVar('current_context')
>
> @sync_and_async_middleware
> class TemplateResponseMiddleware(BaseMiddleware):
> def process_view(self, request, view_func, view_args, view_kwargs):
> request.META['_CONTEXT_RESET_TOKEN'] =
> current_context.set(id(request))
>
> def process_template_response(self, request, response):
> current_context.reset(request.META['_CONTEXT_RESET_TOKEN'])
> response.context_data['mw'].append(self.__class__.__name__)
> return response
> }}}
>
> This use case is what the OpenTelemetry integration uses for spans to be
> traced in Django: https://github.com/open-telemetry/opentelemetry-python-
> contrib/blob/main/instrumentation/opentelemetry-instrumentation-
> django/src/opentelemetry/instrumentation/django/middleware.py
>
> * In `process_request`, a `ContextVar` is set, and the generated token is
> persisted in the `request.META` object.
> * In `process_response`, the `ContextVar` is reset, by using the
> persisted token.
>
> This approach works correctly for synchronous requests. However, as part
> of adding ASGI support to the Django integration for OpenTelemetry (in
> https://github.com/open-telemetry/opentelemetry-python-contrib/pull/391),
> we found that the `ContextVar` triggers the mentioned error when we want
> to reset it to its previous value. OpenTelemetry inherits from
> `MiddlewareMixin`, but I'm attaching a diff for a simple test scenario
> that reproduces the issue, using the new Middleware format.
>
> The main suspects here are the calls to `sync_to_async`, which adapt the
> middleware methods to the async flow. However, both those calls
> explicitly set `thread_sensitive=True`.
>

> Traceback for the attached test scenario:
>
> {{{
> $ ./runtests.py -k
> MiddlewareSyncAsyncTests.test_async_process_template_response
> # ...
> ERROR: test_async_process_template_response
> (middleware_exceptions.tests.MiddlewareSyncAsyncTests)
> --
> Traceback (most recent call last):
>   File "/home/mike/.virtualenvs/django/lib/python3.9/site-
> packages/asgiref/sync.py", line 222, in __call__
> return call_result.result()
>   File "/usr/lib/python3.9/concurrent/futures/_base.py", line 438, in
> result
> return self.__get_result()
>   File "/usr/lib/python3.9/concurrent/futures/_base.py", line 390, in
> __get_result
> raise self._exception
>   File "/home/mike/.virtualenvs/django/lib/python3.9/site-
> packages/asgiref/sync.py", line 287, in main_wrap
> result = await self.awaitable(*args, **kwargs)
>   File "/mnt/data/Proyectos/third_party/django/django/test/utils.py",
> line 423, in inner
> return await func(*args, **kwargs)
>   File
> "/mnt/data/Proyectos/third_party/django/tests/middleware_exceptions/tests.py",
> line 319, in test_async_process_template_response
> response = await self.async_client.get(
>   File "/mnt/data/Proyectos/third_party/django/django/test/client.py",
> line 911, in request
> self.check_exception(response)
>   File "/mnt/data/Proyectos/third_party/django/django/test/client.py",
> line 580, in check_exception
> raise exc_value
>   File "/home/mike/.virtualenvs/django/lib/python3.9/site-
> packages/asgiref/sync.py", line 458, in thread_handler
> raise exc_info[1]
>   File
> "/mnt/data/Proyectos/third_party/django/django/core/handlers/exception.py",
> line 38, in inner
> response = await get_response(request)
>   File
> "/mnt/data/Proyectos/third_party/django/django/core/handlers/base.py",
> line 249, in _get_response_async
> response = await middleware_method(request, response)
>   File "/home/mike/.virtualenvs/django/lib/python3.9/site-
> packages

Re: [Django] #32815: Failed to reset ContextVars in sync/async middlewares

[Django]

#32815: Failed to reset ContextVars in sync/async middlewares
-+-
 Reporter:  Michael Manganiello  |Owner:  nobody
 Type:  Uncategorized|   Status:  new
Component:  Uncategorized|  Version:  3.2
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Description changed by Michael Manganiello:

Old description:

> When using a middleware that can process both sync and async requests,
> and trying to set and reset a ContextVar (in different methods of its
> request lifecycle), Python fails with error:
> `ValueError:  0x7f9a8b9ad900> at 0x7f9a68575180> was created in a different Context`
>
> This is a simple middleware example to reproduce the mentioned issue:
>
> {{{
> @sync_and_async_middleware
> class TemplateResponseMiddleware(BaseMiddleware):
> def process_view(self, request, view_func, view_args, view_kwargs):
> request.META['_CONTEXT_RESET_TOKEN'] =
> current_context.set(id(request))
>
> def process_template_response(self, request, response):
> current_context.reset(request.META['_CONTEXT_RESET_TOKEN'])
> response.context_data['mw'].append(self.__class__.__name__)
> return response
> }}}
>
> This use case is what the OpenTelemetry integration uses for spans to be
> traced in Django: https://github.com/open-telemetry/opentelemetry-python-
> contrib/blob/main/instrumentation/opentelemetry-instrumentation-
> django/src/opentelemetry/instrumentation/django/middleware.py
>
> * In `process_request`, a `ContextVar` is set, and the generated token is
> persisted in the `request.META` object.
> * In `process_response`, the `ContextVar` is reset, by using the
> persisted token.
>
> This approach works correctly for synchronous requests. However, as part
> of adding ASGI support to the Django integration for OpenTelemetry (in
> https://github.com/open-telemetry/opentelemetry-python-contrib/pull/391),
> we found that the `ContextVar` triggers the mentioned error when we want
> to reset it to its previous value. OpenTelemetry inherits from
> `MiddlewareMixin`, but I'm attaching a diff for a simple test scenario
> that reproduces the issue, using the new Middleware format.
>
> The main suspects here are the calls to `sync_to_async`, which adapt the
> middleware methods to the async flow. However, both those calls
> explicitly set `thread_sensitive=True`.
>

> Traceback for the attached test scenario:
>
> {{{
> $ ./runtests.py -k
> MiddlewareSyncAsyncTests.test_async_process_template_response
> # ...
> ERROR: test_async_process_template_response
> (middleware_exceptions.tests.MiddlewareSyncAsyncTests)
> --
> Traceback (most recent call last):
>   File "/home/mike/.virtualenvs/django/lib/python3.9/site-
> packages/asgiref/sync.py", line 222, in __call__
> return call_result.result()
>   File "/usr/lib/python3.9/concurrent/futures/_base.py", line 438, in
> result
> return self.__get_result()
>   File "/usr/lib/python3.9/concurrent/futures/_base.py", line 390, in
> __get_result
> raise self._exception
>   File "/home/mike/.virtualenvs/django/lib/python3.9/site-
> packages/asgiref/sync.py", line 287, in main_wrap
> result = await self.awaitable(*args, **kwargs)
>   File "/mnt/data/Proyectos/third_party/django/django/test/utils.py",
> line 423, in inner
> return await func(*args, **kwargs)
>   File
> "/mnt/data/Proyectos/third_party/django/tests/middleware_exceptions/tests.py",
> line 319, in test_async_process_template_response
> response = await self.async_client.get(
>   File "/mnt/data/Proyectos/third_party/django/django/test/client.py",
> line 911, in request
> self.check_exception(response)
>   File "/mnt/data/Proyectos/third_party/django/django/test/client.py",
> line 580, in check_exception
> raise exc_value
>   File "/home/mike/.virtualenvs/django/lib/python3.9/site-
> packages/asgiref/sync.py", line 458, in thread_handler
> raise exc_info[1]
>   File
> "/mnt/data/Proyectos/third_party/django/django/core/handlers/exception.py",
> line 38, in inner
> response = await get_response(request)
>   File
> "/mnt/data/Proyectos/third_party/django/django/core/handlers/base.py",
> line 249, in _get_response_async
> response = await middleware_method(request, response)
>   File "/home/mike/.virtualenvs/django/lib/python3.9/site-
> packages/asgiref/sync.py", line 423, in __call__
> ret = await asyncio.wait_for(future, tim

Re: [Django] #32815: Failed to reset ContextVars in sync/async middlewares

[Django]

#32815: Failed to reset ContextVars in sync/async middlewares
-+-
 Reporter:  Michael Manganiello  |Owner:  nobody
 Type:  Uncategorized|   Status:  new
Component:  Uncategorized|  Version:  3.2
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Michael Manganiello):

 * Attachment "django-issue-32815.diff" added.

 Test scenario (diff over main commit f10c52afab)

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/068.5cb363133eaa83e9633e7cbd4b0fbd03%40djangoproject.com.

[Django] #32815: Failed to reset ContextVars in sync/async middlewares

[Django]

#32815: Failed to reset ContextVars in sync/async middlewares
---+
   Reporter:  Michael Manganiello  |  Owner:  nobody
   Type:  Uncategorized| Status:  new
  Component:  Uncategorized|Version:  3.2
   Severity:  Normal   |   Keywords:
   Triage Stage:  Unreviewed   |  Has patch:  0
Needs documentation:  0|Needs tests:  0
Patch needs improvement:  0|  Easy pickings:  0
  UI/UX:  0|
---+
 When using a middleware that can process both sync and async requests, and
 trying to set and reset a ContextVar (in different methods of its request
 lifecycle), Python fails with error:
 `ValueError:  at 0x7f9a68575180> was created in a different Context`

 This is a simple middleware example to reproduce the mentioned issue:

 {{{
 @sync_and_async_middleware
 class TemplateResponseMiddleware(BaseMiddleware):
 def process_view(self, request, view_func, view_args, view_kwargs):
 request.META['_CONTEXT_RESET_TOKEN'] =
 current_context.set(id(request))

 def process_template_response(self, request, response):
 current_context.reset(request.META['_CONTEXT_RESET_TOKEN'])
 response.context_data['mw'].append(self.__class__.__name__)
 return response
 }}}

 This use case is what the OpenTelemetry integration uses for spans to be
 traced in Django: https://github.com/open-telemetry/opentelemetry-python-
 contrib/blob/main/instrumentation/opentelemetry-instrumentation-
 django/src/opentelemetry/instrumentation/django/middleware.py

 * In `process_request`, a `ContextVar` is set, and the generated token is
 persisted in the `request.META` object.
 * In `process_response`, the `ContextVar` is reset, by using the persisted
 token.

 This approach works correctly for synchronous requests. However, as part
 of adding ASGI support to the Django integration for OpenTelemetry (in
 https://github.com/open-telemetry/opentelemetry-python-contrib/pull/391),
 we found that the `ContextVar` triggers the mentioned error when we want
 to reset it to its previous value. OpenTelemetry inherits from
 `MiddlewareMixin`, but I'm attaching a diff for a simple test scenario
 that reproduces the issue, using the new Middleware format.

 The main suspects here are the calls to `sync_to_async`, which adapt the
 middleware methods to the async flow. However, both those calls explicitly
 set `thread_sensitive=True`.


 Traceback for the attached test scenario:

 {{{
 $ ./runtests.py -k
 MiddlewareSyncAsyncTests.test_async_process_template_response
 # ...
 ERROR: test_async_process_template_response
 (middleware_exceptions.tests.MiddlewareSyncAsyncTests)
 --
 Traceback (most recent call last):
   File "/home/mike/.virtualenvs/django/lib/python3.9/site-
 packages/asgiref/sync.py", line 222, in __call__
 return call_result.result()
   File "/usr/lib/python3.9/concurrent/futures/_base.py", line 438, in
 result
 return self.__get_result()
   File "/usr/lib/python3.9/concurrent/futures/_base.py", line 390, in
 __get_result
 raise self._exception
   File "/home/mike/.virtualenvs/django/lib/python3.9/site-
 packages/asgiref/sync.py", line 287, in main_wrap
 result = await self.awaitable(*args, **kwargs)
   File "/mnt/data/Proyectos/third_party/django/django/test/utils.py", line
 423, in inner
 return await func(*args, **kwargs)
   File
 "/mnt/data/Proyectos/third_party/django/tests/middleware_exceptions/tests.py",
 line 319, in test_async_process_template_response
 response = await self.async_client.get(
   File "/mnt/data/Proyectos/third_party/django/django/test/client.py",
 line 911, in request
 self.check_exception(response)
   File "/mnt/data/Proyectos/third_party/django/django/test/client.py",
 line 580, in check_exception
 raise exc_value
   File "/home/mike/.virtualenvs/django/lib/python3.9/site-
 packages/asgiref/sync.py", line 458, in thread_handler
 raise exc_info[1]
   File
 "/mnt/data/Proyectos/third_party/django/django/core/handlers/exception.py",
 line 38, in inner
 response = await get_response(request)
   File
 "/mnt/data/Proyectos/third_party/django/django/core/handlers/base.py",
 line 249, in _get_response_async
 response = await middleware_method(request, response)
   File "/home/mike/.virtualenvs/django/lib/python3.9/site-
 packages/asgiref/sync.py", line 423, in __call__
 ret = await asyncio.wait_for(future, timeout=None)
   File "/usr/lib/python3.9/asyncio/tasks.py", line 442, in wait_for
 return await fut
   File "/home/mike/.virtualenvs/django/lib/python3.9/site-
 packages/asgiref/current_thread_executor.py", line 22, in run
 result = self