Fellow Report - March 18, 2017

[Tim Graham]

Triaged

---

https://code.djangoproject.com/ticket/27931 - Clarify the meaning of 
"django catch-all logger" (accepted)

https://code.djangoproject.com/ticket/27937 - Potential issue with 
field.queryset._result_cache persisting (reopened ticket that caused the 
regression)

https://code.djangoproject.com/ticket/27940 - Migrations fail when when a 
model's parent class is changed to abstract --  Wrong SQL being generated 
(duplicate)

https://code.djangoproject.com/ticket/27947 - Document that model field 
error_messages don't carry over to forms (accepted)

https://code.djangoproject.com/ticket/27946 - Improve "ValueError: Found 
wrong number of constraints" error message (accepted)

https://code.djangoproject.com/ticket/27954 - Prevent PostgreSQL dbshell 
from exiting on CTRL-C (accepted)

https://code.djangoproject.com/ticket/27955 - JSONField.default not applied 
in admin (invalid)

https://code.djangoproject.com/ticket/27956 - In DEBUG interface, the 
section "error during template rendering" doesn't go inside an included 
template (duplicate)

Authored

--

https://github.com/python/cpython/pull/661 - bpo-27657: Fix urlparse() with 
numeric paths

https://github.com/django/django/pull/8178 - Refs #27563 -- Fixed 
ModelChoiceField.__deepcopy__() so forms don't share a queryset cache.

https://github.com/django/django/pull/8184 - Fixed #27948 -- Removed 
incorrect unquote() in static serving views.

https://github.com/django/django/pull/8199 - Fixed #27939 -- Updated 
OpenLayersWidget.map_srid for OpenLayers 3.

https://github.com/django/django/pull/8210 - Fixed #27890 -- Fixed 
FileNotFound cleanup exception in runtests.py on Python 3.6+.

Reviewed/committed

--

https://github.com/django/django/pull/8166 - Fixed #27918 -- Documented 
ChoiceWidget.option_template_name

https://github.com/django/django/pull/8120 - Fixed #27882 -- Allowed {% 
cache %} to cache indefinitely.

https://github.com/django/django/pull/8148 - Fixed #27897 -- Fixed crash 
with 'pk' in ModelAdmin.search_filters.

https://github.com/django/django/pull/8171 - Fixed #27935 -- Fixed crash 
with BrinIndex name > 30 characters.
https://github.com/django/django/pull/8209 - Fixed #27724 -- Fixed 
SelectDateWidget redisplay if a year isn't chosen.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/bc466125-9fa9-4cf3-9397-17642888bd80%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: status of 1.11 release blockers

[Tim Graham]

A handful of blockers (mostly in the final stages) remain as of right now 
[0]. We're on track to issue the release candidate on Monday.

[0] 
https://code.djangoproject.com/query?status=!closed=Release%20blocker=1=changetime

On Saturday, February 11, 2017 at 5:14:34 PM UTC-5, Tim Graham wrote:
>
> Thank you to everyone who's tested their apps with Django 1.11. The beta 
> release is planned for Monday, February 20. After that, only release 
> blocking bug fixes will be accepted on the stable/1.11.x branch. There are 
> currently no reported, unfixed regressions or other release blockers for 
> 1.11.
>
> On Tuesday, January 17, 2017 at 11:20:36 AM UTC-5, Tim Graham wrote:
>>
>> I've reviewed and merged as much as I can (thank you to everyone who 
>> helped!) and plan to create the stable/1.11.x branch and issue the alpha 
>> release in approximately 7 hours.
>>
>> On Monday, December 26, 2016 at 1:17:01 PM UTC-5, Tom Christie wrote:
>>>
>>> > 
>>> this feature https://github.com/django/django/pull/7482 will also be a 
>>> very good inclusion.
>>>
>>> Can't see simplified URL routing being complete before the feature 
>>> freeze. What might be feasible is getting the URL kwarg type conversion 
>>> basics in, so that it can fully exist as a third party package, with full 
>>> support slated for 2.0.
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/cda655f8-b7cd-4214-939c-71971b61b312%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Add ability to choose a different secret for PasswordResetToken

[Collin Anderson]

"the self-service site is basically a small subset of our internal site. So
if somebody would gain access to our interal site, he/she would already
have access to a superset of data of the other site. So there is really no
point to also take over to the other site."

Just curious: why not just use the same SECRET_KEY for both sites?

On Sat, Mar 18, 2017 at 6:11 AM, jann.haber via Django developers
(Contributions to Django itself)  wrote:

> Thank you for your input. Yes I meant the PasswordResetTokenGenerator,
> sorry for this.
>
> It agree, it would be a fairly small addition to Django, however there
> doesn't seem to be an easy (non-hackish) way to get around. Since the
> impact on Django would be very small, I wanted to share my thoughts here.
> In our scenario, the self-service site is basically a small subset of our
> internal site. So if somebody would gain access to our interal site, he/she
> would already have access to a superset of data of the other site. So there
> is really no point to also take over to the other site.
>
> I'll give the override_settings a closer look, but this seems like
> something one wouldn't want in a production environment.
>
> Am Samstag, 18. März 2017 10:32:06 UTC+1 schrieb Adam Johnson:
>>
>> Presumably you mean PasswordResetTokenGenerator when you write
>> PasswordResetToken.
>>
>> Seems like a fairly small feature, but my security sense is tingling when
>> you say you're putting the secret key of one application in a variable for
>> another.
>>
>> Normally in a situation where multiple applications need to share login
>> data, I would consider having a centralized application for identity
>> management.
>>
>> Also in your current structure, you don't necessarily need to replace
>> _make_token_with_timestamp (which is prone to break if Django changes
>> that code) - you could (somewhat hackily) wrap with override_settings(SE
>> CRET_KEY=other_key) around your call to PasswordResetTokenGenerator
>> .make_token.
>>
>> On 17 March 2017 at 20:54, jann.haber via Django developers
>> (Contributions to Django itself)  wrote:
>>
>>> Hi everybody,
>>>
>>> I have two sites in my project. One is for internal use to add members
>>> of our club to our database, one is a self-service page for our members.
>>> On registration of a member using our internal site, I want to send a
>>> password reset link to the member for easy registration to our self-service
>>> page.
>>> Since the two sites run on different servers and different SECRET_KEYs,
>>> I need to specify a different secret for the token generated with our
>>> internal site to work on the self-service page.
>>> I managed to do so by overwriting the method _make_token_with_timestamp
>>> of PasswordResetToken, however, I think this is quite an ugly solution.
>>>
>>> How would you think about the secret being an attribute of the
>>> PasswordResetToken class, which is then passed to salted_hmac? The default
>>> could be settings.SECRET_KEY for backwards compatibility.
>>>
>>> I have so far not contributed to django, but I would like to start and I
>>> believe, this small change might be a good start. If you agree, I could
>>> open a ticket in trac.
>>>
>>> Best Regards,
>>> Jann
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-develop...@googlegroups.com.
>>> To post to this group, send email to django-d...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit https://groups.google.com/d/ms
>>> gid/django-developers/ca9e1cde-002d-491b-899a-02bfd50fbed6%
>>> 40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>> Adam
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/4e1c275f-1ea5-482d-affa-
> e70b95ffac24%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 

Re: Add ability to choose a different secret for PasswordResetToken

[jann.haber via Django developers (Contributions to Django itself)]

Thank you for your input. Yes I meant the PasswordResetTokenGenerator, 
sorry for this.

It agree, it would be a fairly small addition to Django, however there 
doesn't seem to be an easy (non-hackish) way to get around. Since the 
impact on Django would be very small, I wanted to share my thoughts here.
In our scenario, the self-service site is basically a small subset of our 
internal site. So if somebody would gain access to our interal site, he/she 
would already have access to a superset of data of the other site. So there 
is really no point to also take over to the other site.

I'll give the override_settings a closer look, but this seems like 
something one wouldn't want in a production environment.

Am Samstag, 18. März 2017 10:32:06 UTC+1 schrieb Adam Johnson:
>
> Presumably you mean PasswordResetTokenGenerator when you write 
> PasswordResetToken.
>
> Seems like a fairly small feature, but my security sense is tingling when 
> you say you're putting the secret key of one application in a variable for 
> another.
>
> Normally in a situation where multiple applications need to share login 
> data, I would consider having a centralized application for identity 
> management.
>
> Also in your current structure, you don't necessarily need to replace 
> _make_token_with_timestamp (which is prone to break if Django changes 
> that code) - you could (somewhat hackily) wrap 
> with override_settings(SECRET_KEY=other_key) around your call to 
> PasswordResetTokenGenerator.make_token.
>
> On 17 March 2017 at 20:54, jann.haber via Django developers (Contributions 
> to Django itself)  wrote:
>
>> Hi everybody,
>>
>> I have two sites in my project. One is for internal use to add members of 
>> our club to our database, one is a self-service page for our members.
>> On registration of a member using our internal site, I want to send a 
>> password reset link to the member for easy registration to our self-service 
>> page.
>> Since the two sites run on different servers and different SECRET_KEYs, I 
>> need to specify a different secret for the token generated with our 
>> internal site to work on the self-service page.
>> I managed to do so by overwriting the method _make_token_with_timestamp 
>> of PasswordResetToken, however, I think this is quite an ugly solution.
>>
>> How would you think about the secret being an attribute of the 
>> PasswordResetToken class, which is then passed to salted_hmac? The default 
>> could be settings.SECRET_KEY for backwards compatibility.
>>
>> I have so far not contributed to django, but I would like to start and I 
>> believe, this small change might be a good start. If you agree, I could 
>> open a ticket in trac.
>>
>> Best Regards,
>> Jann
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to django-d...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/ca9e1cde-002d-491b-899a-02bfd50fbed6%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> -- 
> Adam
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4e1c275f-1ea5-482d-affa-e70b95ffac24%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Add ability to choose a different secret for PasswordResetToken

[Adam Johnson]

Presumably you mean PasswordResetTokenGenerator when you write
PasswordResetToken.

Seems like a fairly small feature, but my security sense is tingling when
you say you're putting the secret key of one application in a variable for
another.

Normally in a situation where multiple applications need to share login
data, I would consider having a centralized application for identity
management.

Also in your current structure, you don't necessarily need to replace
_make_token_with_timestamp (which is prone to break if Django changes that
code) - you could (somewhat hackily) wrap
with override_settings(SECRET_KEY=other_key) around your call to
PasswordResetTokenGenerator.make_token.

On 17 March 2017 at 20:54, jann.haber via Django developers (Contributions
to Django itself)  wrote:

> Hi everybody,
>
> I have two sites in my project. One is for internal use to add members of
> our club to our database, one is a self-service page for our members.
> On registration of a member using our internal site, I want to send a
> password reset link to the member for easy registration to our self-service
> page.
> Since the two sites run on different servers and different SECRET_KEYs, I
> need to specify a different secret for the token generated with our
> internal site to work on the self-service page.
> I managed to do so by overwriting the method _make_token_with_timestamp
> of PasswordResetToken, however, I think this is quite an ugly solution.
>
> How would you think about the secret being an attribute of the
> PasswordResetToken class, which is then passed to salted_hmac? The default
> could be settings.SECRET_KEY for backwards compatibility.
>
> I have so far not contributed to django, but I would like to start and I
> believe, this small change might be a good start. If you agree, I could
> open a ticket in trac.
>
> Best Regards,
> Jann
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/ca9e1cde-002d-491b-899a-
> 02bfd50fbed6%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM1HSZhwVzU9pNuUHv28cWeRqc6n22PzsPk9Zgo_wtedQw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Adding generated common table expressions

[Josh Smeaton]

Thanks for bringing this up Ashley, and for all of the detail you provided. 
I'd certainly like to see CTEs make their way into Django, provided we 
could come up with a nice enough API. From the look of it, you've already 
got something that works with an okay API so I'm hopeful.

I'd be very interested in seeing your POC too if you're able to share.

>From looking very briefly at django-cte-trees it doesn't aim to support 
user defined CTEs for anything other than recursive queries. I'd be 
interested in seeing, as part of a DEP, how CTE inclusion in django core 
could support the cte-trees project from an API perspective.

On Friday, 17 March 2017 22:28:17 UTC+11, Ashley Waite wrote:
>
> Hey all,
>
>
> I'd like to suggest adding Common Table Expression (CTE) query generation 
> as a feature to Django.
>
> I've been working on a project that required manipulation of many records 
> at once, and as with many ORMs found that this wasn't an ideal use-case in 
> Django. As the rest of our code base and related projects are in Django, 
> there was a strong preference to find a way to do it and keep to the same 
> model-is-the-truth design.
>
> I first did this by writing some hackish functions using raw querysets and 
> generating my own CTE based queries, but it lacked ideal flexibility and 
> maintainability. So I've now written some modifications into my Django to 
> do this in a more Django-esque way and think that this functionality would 
> be beneficial within the project itself, but am unsure exactly where to 
> start the conversation about that.
>
>
> *Why generate CTE based queries from querysets?*
>
> By allowing querysets to be attached to each other, and setting 
> appropriate WHERE clauses, arbitrary and nested SQL queries can be 
> generated. Where the results of the queries are only necessary for the 
> execution of following queries this saves a very substantial amount of time 
> and database work. Once these features exist, other functionality can also 
> transparently use these to generate more efficient queries (such as large 
> IN clauses).
>
> This allows several powerful use cases I think Django would benefit from:
>
>
> *Large 'IN' clauses*, can be implemented as CTEs reducing expensive 
> lookups to a single CTE INNER JOIN. For sets of thousands to match from 
> tables of millions of records this can be a very substantial gain.
>
>
> *Composite 'IN' conditions,* where multiple fields must match and you're 
> matching against a large set of condition rows. In my usage this was "where 
> the md5/sha hashes match one of the million md5/sha tuples in my match 
> set". This is simply a CTE JOIN with two clauses in the WHERE.
>
>
> *Nested data creation*, where the parent doesn't yet exist. Django 
> doesn't currently do this as the primary keys are needed, and this makes 
> normalised data structures unappealing. Using INSERTs as CTEs that supply 
> those keys to following statements means that entire nested data structures 
> of new information can be recreated in the database at once, efficiently 
> and atomically.
>
>
> *Non-uniform UPDATE*s, such that a modified set of objects can all be 
> updated with different data at the same time by utilising a CTE values 
> statement JOINed to the UPDATE statement. As there's currently no way to do 
> this kind of bulk update the alternative is to update each instance 
> individually, and this doesn't scale well.
>
> These could also be used with aggregations and other calculated fields to 
> create complex queries that aren't possible at the moment.
>
>
> *What my PoC looks like*
>
> With another mildly hackish PoC that creates a VALUEs set from a 
> dict/namedtuple which can be used to provide large input data, my present 
> modified version syntax looks a bit like this (not perfect queries):
>
> class Hashes(models.Model):
> md5 = models.UUIDField(verbose_name="MD5 hash (base16)", db_index=True)
> sha2 = models.CharField(max_length=44, null=True, verbose_name="SHA256 
> hash (base64)")
>
> # Mock QuerySet of values
> q_mo = Hashes.as_literal(input_hashes).values("md5", "sha2")
> # A big IN query
> q_in = Hashes.objects.attach(q_mo).filter(md5=q_mo.ref("md5"))
>
> # Matched existing values with composite 'IN' (where md5 and sha2 match, or 
> md5 matches and existing record lacks sha2)
> q_ex = 
> Hashes.objects.attach(q_mo).filter(md5=q_mo.ref("md5")).filter(Q(sha160=q_mo.ref("sha160"))
>  | Q(sha160=None))
>
> # Create new records that don't exist
> q_cr = Hashes.objects.attach(q_mo, 
> q_ex).filter(md5=q_mo.ref("md5")).exclude(md5=q_ex.ref("md5")).values("md5", 
> "sha2").as_insert()
>
> Returning the newly created records.
>
> SQL can be generated that looks something like this:
>
> WITH cte_1_0 (md5, sha2) AS (
>   VALUES ('2d30243bfe9d06673765c432c2bd'::uuid, 
> 'fsA8okuCuq9KybxqcAzNdjlIyAx1QJjTPdf1ZFK/hDI='::varchar(44)),
>   ('f20a46e4e60338697948a0917423', 
> '6bVZgpYZtit1E32BlANWXoKnFFFDNierDSIi0SraND4=')),
> 

Re: Adding generated common table expressions

[Adam Johnson]

>
>  supported is added in MySQL 8 [0]
>

Additionally MariaDB 10.2 supports them, and it's nearing release.

On 18 March 2017 at 07:59, Matthias Kestenholz  wrote:

> Hi,
>
> On Sat, Mar 18, 2017 at 12:49 AM, Tim Graham  wrote:
> > Hi, I don't know anything about CTE, but did you see this third-party
> > package? https://github.com/petrounias/django-cte-trees -- It seems to
> be
> > PostgreSQL only.
>
> Just chiming in to point out a maintained and up-to-date friendly fork
> of the project above:
> https://github.com/matthiask/django-cte-forest
>
> Ashley, is your PoC available somewhere? Is your implementation in any
> way similar to the implementation in
> django-cte-trees/django-cte-forest?
>
> Thanks,
> Matthias
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/CANvPqgDWUPuUA6rw8uawa7_
> AsUkXFhuegFM1cJ-gjVUCxvBUwg%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM1pXBMBEOJNmg8w%3D1PWxnFCUCZSpONgYu%3Dk7WXG55yLew%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Adding generated common table expressions

[Matthias Kestenholz]

Hi,

On Sat, Mar 18, 2017 at 12:49 AM, Tim Graham  wrote:
> Hi, I don't know anything about CTE, but did you see this third-party
> package? https://github.com/petrounias/django-cte-trees -- It seems to be
> PostgreSQL only.

Just chiming in to point out a maintained and up-to-date friendly fork
of the project above:
https://github.com/matthiask/django-cte-forest

Ashley, is your PoC available somewhere? Is your implementation in any
way similar to the implementation in
django-cte-trees/django-cte-forest?

Thanks,
Matthias

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANvPqgDWUPuUA6rw8uawa7_AsUkXFhuegFM1cJ-gjVUCxvBUwg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.