Re: [Django] #35021: Debug query capturing on psycopg3 disregards execute wrappers.

[Django]

#35021: Debug query capturing on psycopg3 disregards execute wrappers.
-+-
 Reporter:  Ran Benita   |Owner:  Michail
 Type:   |  Chatzis
  Cleanup/optimization   |   Status:  closed
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Normal   |   Resolution:  fixed
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak ):

 * resolution:   => fixed
 * status:  assigned => closed

Comment:

 In [changeset:"4426b1a72dc289643e2ae8c190b8dc4b3a39daf7" 4426b1a]:
 {{{#!CommitTicketReference repository=""
 revision="4426b1a72dc289643e2ae8c190b8dc4b3a39daf7"
 Fixed #35021 -- Fixed capturing queries when using client-side parameters
 binding with psycopg 3+.
 }}}
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018e13789a5e-adbd255b-434b-4be5-bdf9-1c0b31e2889a-00%40eu-central-1.amazonses.com.

Re: [Django] #35021: Debug query capturing on psycopg3 disregards execute wrappers.

[Django]

#35021: Debug query capturing on psycopg3 disregards execute wrappers.
-+-
 Reporter:  Ran Benita   |Owner:  Michail
 Type:   |  Chatzis
  Cleanup/optimization   |   Status:  assigned
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * stage:  Accepted => Ready for checkin

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018e134fd9bf-0e370ae4-ab3b-45b1-b958-722450e4702e-00%40eu-central-1.amazonses.com.

Re: [Django] #35021: Debug query capturing on psycopg3 disregards execute wrappers.

[Django]

#35021: Debug query capturing on psycopg3 disregards execute wrappers.
-+-
 Reporter:  Ran Benita   |Owner:  Michail
 Type:   |  Chatzis
  Cleanup/optimization   |   Status:  assigned
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Michail Chatzis):

 * needs_better_patch:  1 => 0

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018e130df0bd-76c329e0-71f3-4ac9-89f4-99ea1d69d15a-00%40eu-central-1.amazonses.com.

Re: [Django] #35021: Debug query capturing on psycopg3 disregards execute wrappers.

[Django]

#35021: Debug query capturing on psycopg3 disregards execute wrappers.
-+-
 Reporter:  Ran Benita   |Owner:  Michail
 Type:   |  Chatzis
  Cleanup/optimization   |   Status:  assigned
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Comment (by Michail Chatzis):

 Replying to [comment:11 Mariusz Felisiak]:
 > Replying to [comment:10 Michail Chatzis]:
 >
 > Why you unset "needs improvement" flag without fixing my comments?


 My bad, I thought you wanted me to unset the 'needs' flag when I wanted a
 review on Gitlab. I left a new comment on Gitlab, hence I wanted a review
 of it.

 I am guessing I misunderstood that.
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018e05e6a22c-4d1ad6d8-91bb-4afb-a5ed-b9dd98e4dbb2-00%40eu-central-1.amazonses.com.

Re: [Django] #35021: Debug query capturing on psycopg3 disregards execute wrappers.

[Django]

#35021: Debug query capturing on psycopg3 disregards execute wrappers.
-+-
 Reporter:  Ran Benita   |Owner:  Michail
 Type:   |  Chatzis
  Cleanup/optimization   |   Status:  assigned
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * needs_better_patch:  0 => 1

Comment:

 Replying to [comment:10 Michail Chatzis]:

 Why you unset "needs improvement" flag without fixing my comments?
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018e05be71b5-144a66b3-121f-4bb2-88db-0c1e0a34fb76-00%40eu-central-1.amazonses.com.

Re: [Django] #35021: Debug query capturing on psycopg3 disregards execute wrappers.

[Django]

#35021: Debug query capturing on psycopg3 disregards execute wrappers.
-+-
 Reporter:  Ran Benita   |Owner:  Michail
 Type:   |  Chatzis
  Cleanup/optimization   |   Status:  assigned
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Michail Chatzis):

 * needs_better_patch:  1 => 0

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018e031576c1-934513ab-37e4-4b89-aabc-aed80174db3b-00%40eu-central-1.amazonses.com.

Re: [Django] #35021: Debug query capturing on psycopg3 disregards execute wrappers.

[Django]

#35021: Debug query capturing on psycopg3 disregards execute wrappers.
-+-
 Reporter:  Ran Benita   |Owner:  Michail
 Type:   |  Chatzis
  Cleanup/optimization   |   Status:  assigned
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * needs_better_patch:  0 => 1

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018d62e9b303-6f103d8c-9cd3-44f1-a21a-9e80d30e5c16-00%40eu-central-1.amazonses.com.

Re: [Django] #35021: Debug query capturing on psycopg3 disregards execute wrappers.

[Django]

#35021: Debug query capturing on psycopg3 disregards execute wrappers.
-+-
 Reporter:  Ran Benita   |Owner:  Michail
 Type:   |  Chatzis
  Cleanup/optimization   |   Status:  assigned
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Natalia Bidart):

 * has_patch:  0 => 1

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018d610d3a3c-1942ea61-3eb3-4643-8093-5cc98e9f28d8-00%40eu-central-1.amazonses.com.

Re: [Django] #35021: Debug query capturing on psycopg3 disregards execute wrappers.

[Django]

#35021: Debug query capturing on psycopg3 disregards execute wrappers.
-+-
 Reporter:  Ran Benita   |Owner:  Michail
 Type:   |  Chatzis
  Cleanup/optimization   |   Status:  assigned
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Michail Chatzis):

 * owner:  nobody => Michail Chatzis
 * status:  new => assigned

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018d5a5dd135-a3a0c993-b148-4784-8af0-4d4c3d3d7c77-00%40eu-central-1.amazonses.com.

Re: [Django] #35021: Debug query capturing on psycopg3 disregards execute wrappers. (was: Debug query capturing on psycopg3 disregards execute wrappers)

[Django]

#35021: Debug query capturing on psycopg3 disregards execute wrappers.
-+-
 Reporter:  Ran Benita   |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * stage:  Unreviewed => Accepted


Comment:

 Tentatively accepted. I think it's worth fixing even just for
 `ClientCursor`. Server-side binding cursors have a few other minor
 limitations in Django, so one more should not be a big surprise.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018c5a959a19-9bb1c2ce-fdf2-43ea-ae0e-3e8a04127993-00%40eu-central-1.amazonses.com.

Re: [Django] #35021: Debug query capturing on psycopg3 disregards execute wrappers

[Django]

#35021: Debug query capturing on psycopg3 disregards execute wrappers
-+-
 Reporter:  Ran Benita   |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Florian Apolloner):

 Replying to [comment:4 Daniele Varrazzo]:
 > I understand that Django use `ClientCursor` only, right? I think that
 the first iterations of the backend used to use server-side-binding
 cursors and maybe the current implementation of `last_executed_query()`
 was catered to that.

 Django defaults to `ClientCursor` but it can be changed to server-side-
 binding cursors via settings.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018c599ab197-e8b62d9e-dc8f-4046-84fb-17fcee73beb8-00%40eu-central-1.amazonses.com.

Re: [Django] #35021: Debug query capturing on psycopg3 disregards execute wrappers

[Django]

#35021: Debug query capturing on psycopg3 disregards execute wrappers
-+-
 Reporter:  Ran Benita   |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Daniele Varrazzo):

 I understand that Django use `ClientCursor` only, right? I think that the
 first iterations of the backend used to use server-side-binding cursors
 and maybe the current implementation of `last_executed_query()` was
 catered to that.

 I have added a proposal to [https://github.com/psycopg/psycopg/issues/696
 the upstream ticket].

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018c592d8a95-6d3461db-6730-4453-9175-376cf49f3854-00%40eu-central-1.amazonses.com.

Re: [Django] #35021: Debug query capturing on psycopg3 disregards execute wrappers

[Django]

#35021: Debug query capturing on psycopg3 disregards execute wrappers
-+-
 Reporter:  Ran Benita   |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Florian Apolloner):

 I think the new code was written like this because we cannot ever
 determine the actual query for server side bindings.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018c58b018b8-e12fab03-560b-476f-af2c-d7ed9efbffab-00%40eu-central-1.amazonses.com.

Re: [Django] #35021: Debug query capturing on psycopg3 disregards execute wrappers

[Django]

#35021: Debug query capturing on psycopg3 disregards execute wrappers
-+-
 Reporter:  Ran Benita   |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Mariusz Felisiak):

 > Daniele, What do you think about extending psycopg's API?

 I opened an [https://github.com/psycopg/psycopg/issues/696 issue] to
 continue a discussion with the `psycopg` maintainers.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018c58abed45-e9aae73d-22ab-467c-964b-ac715826ef8d-00%40eu-central-1.amazonses.com.

Re: [Django] #35021: Debug query capturing on psycopg3 disregards execute wrappers

[Django]

#35021: Debug query capturing on psycopg3 disregards execute wrappers
-+-
 Reporter:  Ran Benita   |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * cc: Florian Apolloner, Daniele Varrazzo (added)
 * type:  Bug => Cleanup/optimization


Comment:

 As far as I'm aware, there is no way to support it with psycopg's public
 API, so I'd not treat this as a bug.

 Daniele, What do you think about extending psycopg's API?

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018c3f3961b0-41706609-bb77-4a04-b194-fee5f2edf8bf-00%40eu-central-1.amazonses.com.

[Django] #35021: Debug query capturing on psycopg3 disregards execute wrappers

[Django]

#35021: Debug query capturing on psycopg3 disregards execute wrappers
-+-
   Reporter:  Ran|  Owner:  nobody
  Benita |
   Type:  Bug| Status:  new
  Component:  Database   |Version:  5.0
  layer (models, ORM)|
   Severity:  Normal |   Keywords:
   Triage Stage: |  Has patch:  0
  Unreviewed |
Needs documentation:  0  |Needs tests:  0
Patch needs improvement:  0  |  Easy pickings:  0
  UI/UX:  0  |
-+-
 == Problem

 I use [https://docs.djangoproject.com/en/4.2/topics/db/instrumentation
 /#connection-execute-wrapper connection.execute_wrapper] as a way to add
 comments to certain queries (see code at the end for reference). And I
 test the wrapper using `assertNumQueries` (see code at end for reference).
 Once I switched from psycopg2 to psycopg3, this test started failing: the
 unmodified query is captured, instead of the query modified by the execute
 wrapper. Note however that the modified query is what is actually sent to
 the DB, so this is only a debug issue.

 I expect it's the same for `django.db.backends` debug logs but I haven't
 verified this.

 == Analysis

 I've traced the issue to the following code in
 `postgresql.DatabaseOperations.last_executed_query`
 
https://github.com/django/django/blob/66d58e77de3196404e0820a6fef0a6144186015a/django/db/backends/postgresql/operations.py#L296-L311:

 {{{
 #!python

 if is_psycopg3:

 def last_executed_query(self, cursor, sql, params):
 try:
 return self.compose_sql(sql, params)
 except errors.DataError:
 return None

 else:

 def last_executed_query(self, cursor, sql, params):
 # https://www.psycopg.org/docs/cursor.html#cursor.query
 # The query attribute is a Psycopg extension to the DB API
 2.0.
 if cursor.query is not None:
 return cursor.query.decode()
 return None
 }}}

 psycopg2 uses `cursor.query` which ends up being the modified query.
 psycopg3 uses whatever's passed in which ends up being the unmodified
 query.

 It seems like psycopg3 has an equivalent in
 [https://www.psycopg.org/psycopg3/docs/api/cursors.html#psycopg.Cursor._query
 cursor._query]. It is documented in the API reference but with a warning
 "You shouldn’t consider it part of the public interface of the object: it
 might change without warnings. [...] If you would like to build reliable
 features using this object, please get in touch so we can try and design
 an useful interface for it.". So if this is the desired solution, will
 need to work with psycopg to expose a stable interface.

 == Reproduction code

 Example execute wrapper:

 {{{
 #!python
 def db_comment_wrapper(comment: str) -> AbstractContextManager[None]:
 def handler(execute, sql, params, many, context):
 clean_comment = escape(comment)
 return execute(f'/* {clean_comment} */ {sql}', params, many,
 context)
 return db_connection.execute_wrapper(handler)
 }}}

 Test:

 {{{
 #!python
 with self.assertNumQueries(1) as captured:
 with db_comment_wrapper('This is a comment'):
 list(ContentType.objects.all())
 sql = captured[0]['sql']
 assert sql.startswith('/* This is a comment */')
 }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018c3f1fbd3d-93443bed-909f-4211-9e23-7e2fe44cfc35-00%40eu-central-1.amazonses.com.