Re: Subquery join support

[Andre Terra]

Thanks, Alexandr, for your effort in addressing what I personally consider 
a much needed feature in the ORM. I gave your PR a try and noticed that 
evaluating the same Product.objects.annotate line below twice in a row 
returns two different SQL queries, the second of which results in an error. 

Product.objects.annotate(taxes=Subquery(taxes, join=True, join_type=LOUTER))

I'm using different sample models in my project since I didn't have your 
models handy to test, but here's a gist with the terminal session and 
traceback: 
https://gist.github.com/airstrike/e6ac4feb96136b71dda5e3fcfa32418e You 
should be able to recreate the error easily with your own models by just 
running the line above twice. Seems like the Subquery() instantiation is 
having some unintended side-effects...

Now, more broadly, it seems like we currently have various initial attempts 
at expanding the ability to handle subqueries in the ORM but with a few 
different purposes – I count at least 5:

   - Improving support for subqueries in UPDATE statements 
   (https://code.djangoproject.com/ticket/29214)
   - Adding support for CTEs 
   (https://github.com/ashleywaite/django/tree/cte-dev)
   - Adding support for *recursive* CTEs for hierarchical / tree data 
   (https://code.djangoproject.com/ticket/28919 and 
   https://github.com/matthiask/django-cte-forest)
   - Alexandr's suggested approach of joining tables with Subqueries (this 
   thread)
   - My personal desire / need for derived tables for advanced annotation / 
   aggregation, i.e. subqueries in the FROM clause 
   
(https://www.essentialsql.com/get-ready-to-learn-sql-server-22-using-subqueries-in-the-from-clause/)
 
   which to me feels essentially equivalent to non-recursive CTE
   
Rather than letting each of these efforts operate (and for the most part, 
die) independently, I believe Django would benefit from a concerted 
approach for a few reasons:

   1. leveraging each other's expertise
   2. leveraging code between those different efforts and minimizing 
   complexity / churn
   3. creating a consistent API
   4. actually getting buy-in from the core developers to have this merged 
   at some point
   
I'm not close enough to Django or sufficiently well-versed in the ORM to 
know exactly what the next steps are, but it feels to me that we need a DEP 
that starts by reviewing all of the past efforts, existing needs and 
outlines a consistent API along with the equivalent SQL for clarity.

So I invite others to provide feedback on whether they believe a DEP is the 
right path forward – though I'll note that I don't have the time or, most 
importantly, the expertise to be its Author 



Best,
Andy


On Wednesday, April 15, 2020 at 3:52:03 AM UTC-4, Alexandr Tatarinov wrote:

> This has not received a lot of feedback so far, so I think some code can 
> help. I am not sure what is the next steps; as I've understood, we need 
> some kind of consensus to open a ticket.
>
> Please, don't hesitate to express your opinion on the feature and the API, 
> as well as the implementation to move this forward.
> https://github.com/tatarinov1997/django/pull/1/
>
> Cheers,
> Alexandr.
>
>
>
> On Sunday, 12 April 2020 19:35:55 UTC+3, Alexandr Tatarinov wrote:
>>
>> Hello everyone.
>> So far, I've managed to implement joins via Subquery and OuterRef with 
>> some caveats.
>>
>> To begin with, the API looks like this
>>
>> Product.objects.create(name='Pizza', country='USA', type='FOOD')
>> Tax.objects.create(name='USA Food Tax', amount=10, product_country='USA', 
>> product_type='FOOD')
>> 
>> taxes = Tax.objects.filter(
>> product_type=OuterRef('type'),
>> product_country=OuterRef('country')
>> )
>> products = Product.objects.annotate(taxes=Subquery(taxes, join=True, 
>> join_type=LOUTER))
>> product = products.annotate(tax=F('taxes__amount'), tax_name=F(
>> 'taxes__name'))
>>
>> The generated SQL is
>> SELECT "core_product"."id", "core_product"."name", "core_product".
>> "country", "core_product"."type", "taxes"."__col3" AS "tax", "taxes".
>> "__col4" AS "tax_name" 
>> FROM "core_product" 
>> LEFT OUTER JOIN 
>> (
>> SELECT 
>> "core_tax"."id", "core_tax"."product_country", "core_tax".
>> "product_type", "core_tax"."name", "core_tax"."amount", 
>> "core_tax"."product_country" AS "__col1", "core_tax".
>> "product_type" AS "__col2", "core_tax"."amount" AS "__col3", "core_tax".
>> "name" AS "__col4"
>> FROM "core_tax"
>> ) "taxes" ON ("taxes"."__col1" = ("core_product"."country") AND "taxes".
>> "__col2" = ("core_product"."type"))
>>
>> We use *.annotate* for joins to avoid adding new methods, also 
>> *.annotate* has some useful checks to avoid name collisions.
>> To distinguish plain subquery from the joined subquery, a *join *flag is 
>> added. To specify join type, there is an optional *join_type* param to 
>> Subquery.
>>
>> Then, one can refer to joined 

Re: Swappable sessions

[Tim Graham]

I don't see a strong argument for adding it. As far as I can tell, this 
would require adding 'request' to the signature of SessionStore.__init___() 
(so it could be stored as an attribute on SessionStore and thereby 
available in create_model_instance()). Backwards compatibility would be 
non-trivial and coupling the session store to the request seems unnecessary.

On Wednesday, April 15, 2020 at 5:38:18 PM UTC-4, Lorenzo Peña wrote:
>
> Yes, of course. I am already working things around in order to get what I 
> need.
>
> Now, the database backed SessionStore defines a method 
> "create_model_instance(self, data)" which is called from 
> "django.contrib.sessions.middleware.SessionMiddleware". Django docs invite 
> to redefine this method in order to populate the custom session with the 
> required data.
>
> My question is: would it be okay to formally ask that the request object, 
> which is already available there, be passed to that method when called, 
> like " create_model_instance(self, data, request=None)"?
>
> El miércoles, 15 de abril de 2020, 17:01:30 (UTC-4), Tim Graham escribió:
>>
>> Can't you copy whatever data you want from the request into the session 
>> data (request.session. = request.) in a view, 
>> middleware, or wherever? That seems like the proper separation of concerns.
>>
>> On Wednesday, April 15, 2020 at 11:50:13 AM UTC-4, Lorenzo Peña wrote:
>>>
>>> Now, in order to follow the path suggested in the Django documentation 
>>> for overriding database backed sessions, some of the data one might need to 
>>> store in the session model is coming from the request, and Django is not 
>>> passing the request when instantiating the session store.
>>>
>>> Do you think there is a point in requesting that the request object (if 
>>> available) is passed as an optional parameter when creating the session?
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6a00f99e-ce00-4003-b64e-5a2502d8ab86%40googlegroups.com.

Re: Swappable sessions

[Lorenzo Peña]

Yes, of course. I am already working things around in order to get what I 
need.

Now, the database backed SessionStore defines a method 
"create_model_instance(self, data)" which is called from 
"django.contrib.sessions.middleware.SessionMiddleware". Django docs invite 
to redefine this method in order to populate the custom session with the 
required data.

My question is: would it be okay to formally ask that the request object, 
which is already available there, be passed to that method when called, 
like " create_model_instance(self, data, request=None)"?

El miércoles, 15 de abril de 2020, 17:01:30 (UTC-4), Tim Graham escribió:
>
> Can't you copy whatever data you want from the request into the session 
> data (request.session. = request.) in a view, 
> middleware, or wherever? That seems like the proper separation of concerns.
>
> On Wednesday, April 15, 2020 at 11:50:13 AM UTC-4, Lorenzo Peña wrote:
>>
>> Now, in order to follow the path suggested in the Django documentation 
>> for overriding database backed sessions, some of the data one might need to 
>> store in the session model is coming from the request, and Django is not 
>> passing the request when instantiating the session store.
>>
>> Do you think there is a point in requesting that the request object (if 
>> available) is passed as an optional parameter when creating the session?
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9047b481-9127-4c50-b5da-34b7fd6f7b47%40googlegroups.com.

Re: Swappable sessions

[Lorenzo Peña]

Yes, of course. I am already working things around in order to get what I
need.

Now, the database backed SessionStore defines a method
"create_model_instance(self, data)" which is called from
"django.contrib.sessions.middleware.SessionMiddleware". Django docs invite
to redefine this method in order to populate the custom session with the
required data.

My question is: would it be okay to formally ask that the request object,
which is already available when the method is called, be passed to that
method when called, like " create_model_instance(self, data,
request=None)"?

El mié., 15 abr. 2020 a las 17:01, Tim Graham ()
escribió:

> Can't you copy whatever data you want from the request into the session
> data (request.session. = request.) in a view,
> middleware, or wherever? That seems like the proper separation of concerns.
>
> On Wednesday, April 15, 2020 at 11:50:13 AM UTC-4, Lorenzo Peña wrote:
>>
>> Now, in order to follow the path suggested in the Django documentation
>> for overriding database backed sessions, some of the data one might need to
>> store in the session model is coming from the request, and Django is not
>> passing the request when instantiating the session store.
>>
>> Do you think there is a point in requesting that the request object (if
>> available) is passed as an optional parameter when creating the session?
>>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/django-developers/DOUfhMv4T1c/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/d9eef373-28cf-4ccb-9801-90d7c38b3678%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAA7tyWt7o_Mdk_sjRN2CssYKujO_Ru9eZotpyvn%3Dxi7ZgCEjWA%40mail.gmail.com.

Re: Swappable sessions

[Tim Graham]

Can't you copy whatever data you want from the request into the session 
data (request.session. = request.) in a view, 
middleware, or wherever? That seems like the proper separation of concerns.

On Wednesday, April 15, 2020 at 11:50:13 AM UTC-4, Lorenzo Peña wrote:
>
> Now, in order to follow the path suggested in the Django documentation for 
> overriding database backed sessions, some of the data one might need to 
> store in the session model is coming from the request, and Django is not 
> passing the request when instantiating the session store.
>
> Do you think there is a point in requesting that the request object (if 
> available) is passed as an optional parameter when creating the session?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d9eef373-28cf-4ccb-9801-90d7c38b3678%40googlegroups.com.

Re: Generate JWTs with Django

[Claude Paroz]

Thanks Abhijeet for the pointer, I know there are some rather complete
JWT libs around, but my proposal is not about a complete package to
manage JWT in general.
It's rather some low level ability for Django to produce and decode
simple HS256 JWT. Then other third-party libs could build on that
ability to write more elaborate packages.

The main doubt I have about my proposal is whether HS256 JWTs are too
limited for most usages or in the contrary if they are appropriate for a
fair amount of use cases.

Claude

Le 15.04.20 à 21:13, Abhijeet Viswa a écrit :
> Hi,
> 
> You might want check out django-restframework-simplejwt. It requires the
> Django Rest Framework. But, then again, if you are making an API, you'd
> already be using it.
> 
> Regards,
> Abhijeet
> 
> On Thu, 16 Apr, 2020, 00:39 Claude Paroz,  > wrote:
> 
> Hi all,
> 
> With the recent addition of the algorithm parameter to the
> signing.Signer class, it's now rather straightforward for Django to
> generate HS256 (non-encrypted) JSON Web Tokens.
> With a growing popularity of JS-client/Django server communications
> (DRF and al.), I think there might be some interest for Django to be
> able to generate and decode such tokens. For any other types of JWTs
> which generally require access to a cryptography module, we can
> point users to third-party libs like PyJWT (the docs should be clear
> about that).
> 
> I made a proof-of-concept PR (docs missing) here:
>  - https://github.com/django/django/pull/12728
> 
> What people here think about that proposal?
> 
> Claude

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9f5f4df0-64d3-68aa-0ae7-302e037d406a%402xlibre.net.

Re: Generate JWTs with Django

[Abhijeet Viswa]

Hi,

You might want check out django-restframework-simplejwt. It requires the
Django Rest Framework. But, then again, if you are making an API, you'd
already be using it.

Regards,
Abhijeet

On Thu, 16 Apr, 2020, 00:39 Claude Paroz,  wrote:

> Hi all,
>
> With the recent addition of the algorithm parameter to the signing.Signer
> class, it's now rather straightforward for Django to generate HS256
> (non-encrypted) JSON Web Tokens.
> With a growing popularity of JS-client/Django server communications (DRF
> and al.), I think there might be some interest for Django to be able to
> generate and decode such tokens. For any other types of JWTs which
> generally require access to a cryptography module, we can point users to
> third-party libs like PyJWT (the docs should be clear about that).
>
> I made a proof-of-concept PR (docs missing) here:
>  - https://github.com/django/django/pull/12728
>
> What people here think about that proposal?
>
> Claude
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/5f67fefb-d158-4722-b704-6c34d72692a8%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAP1-Yrqiryo%2BuapQHkLjebwP7aaOQ8igSx3_cr6Q1Q16bKgaRw%40mail.gmail.com.

Generate JWTs with Django

[Claude Paroz]

Hi all,

With the recent addition of the algorithm parameter to the signing.Signer 
class, it's now rather straightforward for Django to generate HS256 
(non-encrypted) JSON Web Tokens.
With a growing popularity of JS-client/Django server communications (DRF 
and al.), I think there might be some interest for Django to be able to 
generate and decode such tokens. For any other types of JWTs which 
generally require access to a cryptography module, we can point users to 
third-party libs like PyJWT (the docs should be clear about that).

I made a proof-of-concept PR (docs missing) here:
 - https://github.com/django/django/pull/12728

What people here think about that proposal?

Claude

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5f67fefb-d158-4722-b704-6c34d72692a8%40googlegroups.com.

Re: Swappable sessions

[Lorenzo Peña]

Now, in order to follow the path suggested in the Django documentation for 
overriding database backed sessions, some of the data one might need to 
store in the session model is coming from the request, and Django is not 
passing the request when instantiating the session store.

Do you think there is a point in requesting that the request object (if 
available) is passed as an optional parameter when creating the session?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/78cc1f20-556f-44e6-ae40-2a930cfa0a58%40googlegroups.com.

Re: Subquery join support

[Alexandr Tatarinov]

This has not received a lot of feedback so far, so I think some code can 
help. I am not sure what is the next steps; as I've understood, we need 
some kind of consensus to open a ticket.

Please, don't hesitate to express your opinion on the feature and the API, 
as well as the implementation to move this forward.
https://github.com/tatarinov1997/django/pull/1/

Cheers,
Alexandr.



On Sunday, 12 April 2020 19:35:55 UTC+3, Alexandr Tatarinov wrote:
>
> Hello everyone.
> So far, I've managed to implement joins via Subquery and OuterRef with 
> some caveats.
>
> To begin with, the API looks like this
>
> Product.objects.create(name='Pizza', country='USA', type='FOOD')
> Tax.objects.create(name='USA Food Tax', amount=10, product_country='USA', 
> product_type='FOOD')
> 
> taxes = Tax.objects.filter(
> product_type=OuterRef('type'),
> product_country=OuterRef('country')
> )
> products = Product.objects.annotate(taxes=Subquery(taxes, join=True, 
> join_type=LOUTER))
> product = products.annotate(tax=F('taxes__amount'), tax_name=F(
> 'taxes__name'))
>
> The generated SQL is
> SELECT "core_product"."id", "core_product"."name", "core_product".
> "country", "core_product"."type", "taxes"."__col3" AS "tax", "taxes".
> "__col4" AS "tax_name" 
> FROM "core_product" 
> LEFT OUTER JOIN 
> (
> SELECT 
> "core_tax"."id", "core_tax"."product_country", "core_tax".
> "product_type", "core_tax"."name", "core_tax"."amount", 
> "core_tax"."product_country" AS "__col1", "core_tax".
> "product_type" AS "__col2", "core_tax"."amount" AS "__col3", "core_tax".
> "name" AS "__col4"
> FROM "core_tax"
> ) "taxes" ON ("taxes"."__col1" = ("core_product"."country") AND "taxes".
> "__col2" = ("core_product"."type"))
>
> We use *.annotate* for joins to avoid adding new methods, also *.annotate* 
> has 
> some useful checks to avoid name collisions.
> To distinguish plain subquery from the joined subquery, a *join *flag is 
> added. To specify join type, there is an optional *join_type* param to 
> Subquery.
>
> Then, one can refer to joined subquery columns by the given alias, but 
> only via *F()* expression.
> This means it is not possible to write *.values('taxes__name') *directly*, 
> *but can be achieved via *.annotate(**tax_name=F('taxes__name'))* 
> *.values('tax_name').*
> Other examples are *.order_by(F('taxes__name')) *and* 
> .values_list(F('taxes__name')), *they work without *.annotate() *but 
> require *F() *wrapper. 
>
> The reasons why the string fields and *F* expressions are treated 
> separately and differently in various QuerySet methods are still unclear 
> for me.
> The *F *expression works because the implementation relies on 
> *Query.resolve_ref 
> *to be called to short-circuit columns from the joined subqueries.
> Maybe, we will be able to fix this later or leave as-is with the 
> documentation in place to always use *.annotate*.
>
> The implementation is ~170 lines long, but I think some corner cases are 
> yet to be discovered.
> For now, it would be great to hear your feedback on the API and the 
> feature itself, so we can move on with the ticket and discuss the 
> implementation.
>
> Thanks, Alexandr.
>
>
> On Monday, 6 April 2020 16:34:55 UTC+3, Alexandr Tatarinov wrote:
>>
>> Hello folks,
>>
>> Following the discussion 
>> https://groups.google.com/forum/#!topic/django-developers/b370mxfKCHg, I 
>> would like to suggest adding the ability to join QuerySet with a subquery 
>> clause to Django.
>> The benefits are pretty much described and discussed in the linked topic, 
>> the only one I would like to add (my use case) is the ability to select 
>> multiple columns from a Subquery without the need to repeat the subquery 
>> clause, which hurts performance.
>> As Anssi Kääriäinen mentioned, generic subquery supports seems to be a 
>> better idea than CTEs, and simple implementation as a starting point will 
>> allow moving gradually and add RETURNING support later on (for nested 
>> creates and updates).
>>
>> I have created a working prototype and successfully using it in one of my 
>> projects in production. After reading Daniel Miller's 
>> https://github.com/dimagi/django-cte, I've rewritten the API inspired by 
>> his ideas, plus tried to keep it as Django-style as possible.
>> Here is the usage example.
>>
>> taxes = JoinedSubquery(Tax.objects.all())
>> products = Product.objects.all()
>>
>> # .join is probably better be placed in QuerySet class, so it will be 
>> products.join(taxes, ...)
>> products_with_taxes = taxes.join(
>> products,
>> join_type=LOUTER,  # Optional argument, LOUTER by default
>> country=taxes.F('product_country'),  # Also Q objects are supported 
>> type=taxes.F('product_type')
>> ).annotate(
>> tax_amount=taxes.F('amount')  # .join() and .annotate() calls 
>> needn't to be chained
>> )
>>
>>
>> I am not sure about named joins discussed in the CTE topic
>> qs.attach(q1=some_qs).filter(a=F('q1__b'))
>>
>> It seems to be 

Re: Making the admin more navigable

[Carlton Gibson]

Do we know anyone who's a *Design/CSS/UX specialist* that can offer some 
input here?  

Hi Tom. Thanks for this PR.

https://github.com/django/django/pull/12159

I think it's a good addition. It's perhaps not perfect, but we can take it 
as an incremental change and then improve it over time. 

There's one comment re the styling of the sidebar headers and the 
breadcrumbs bar, that I think we should do something about. 

*But then*, I think a small dose of *Actual Designer™* would go a long way 
here. 
I have insecurities about a bunch of backend devs discussing CSS & design 
as if we really know anything about it... 
(Maybe you do. I don't.) 

We're a community of web professionals. We must know some people with *bone 
fide* chops here. 
*To all*: Can anybody input or ask somebody to input? 
(Posting here to ask that.) 

Otherwise, I think we should probably accept, with a last round of 
tweaking. 

Kind Regards,

Carlton


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/16212931-af24-4cb3-a0d3-7d82c3857e9b%40googlegroups.com.