Re: Can we move the activity on this list to the Forum now?

[Andrew Godwin]

We should at least update those Trac and Triage Workflow docs to point to both, 
maybe with the Forum first?

Andrew

On Thu, Jan 19, 2023, at 12:30 AM, Carlton Gibson wrote:
> I'm trying to begin new conversations there where I can. 
> 
> The main issue is that we're still pointing people here from Trac and the 
> Triage Workflow docs — so if there's a rough "Yeah, let's do it" we can 
> adjust practice there. 
> 
> I expect there will always be the "How do I start?" posts. The Forum at least 
> has a pinned post for that. ... 
> 
> On Thu, 19 Jan 2023 at 01:04, 'Kye Russell' via Django developers 
> (Contributions to Django itself)  wrote:
>> Hi all,
>> 
>> I find that the signal-to-noise ratio on this mailing list is (by my 
>> determination) quite bad around this time of year.
>> 
>> Is a move to the forum still on the cards? 
>> 
>> Kye
>> On 6 Dec 2022 at 7:16 AM +0800, Andrew Godwin , wrote:
>> 
>>> I did some investigation of moving django-users and django-developers to 
>>> the Forum right after DjangoCon; I wanted to see if we could import all the 
>>> old posts too, which we probably could, but I'm not entirely sure of the 
>>> utility of that. 
>>> 
>>> I will say that the forum is a lot easier to moderate - the ability to 
>>> moderate after a post has gone out, rather than gating all posts behind 
>>> approval if they're untrusted, is a big step in itself, not to mention the 
>>> ability to remove sensitive or offensive content once it's posted.
>>> 
>>> Andrew
>>> 
>>> On Monday, November 28, 2022 at 10:01:17 PM UTC-7 m...@kye.id.au wrote:
>>>> IMO django-announce and django-updates serve a very different purpose and 
>>>> I would be against moving them if it were suggested.
>>>> 
>>>> I am incredibly strongly in favour of moving django-developers and 
>>>> django-users to the forums. IMO being able to more easily trap people 
>>>> misusing this list as a tech support channel is itself reason enough to 
>>>> move. Beyond that, I’d argue that the plentiful UX issues with Google 
>>>> Groups, and mailing lists in general, certainly don’t do the community any 
>>>> favours in terms of getting more people on board. 
>>>> 
>>>> Kye
>>>> 
>>>> On 28 Nov 2022 at 11:40 PM +0800, 'Tobias McNulty' via Django developers 
>>>> (Contributions to Django itself) , wrote:
>>>>> As someone who only just joined the forum -- I'm +1: 
>>>>>  * The forum has seen great adoption from what I can tell (nearly half 
>>>>> the number of posts as django-developers during the same time period, not 
>>>>> bad given the mailing list's head start in subscribers).
>>>>>  * It seems beneficial to house future conversations in a single place, 
>>>>> e.g., so members don't need to subscribe to both the mailing list and 
>>>>> forum to get the full picture of current active development, set up two 
>>>>> different sets of mail filters to tag things appropriately, etc...
>>>>> Would the plan be to switch django-users as well? I think similar 
>>>>> arguments could be made for consolidating those...
>>>>> 
>>>>> (On the other hand, I see little value in switching django-announce and 
>>>>> django-updates, but I'm not necessarily opposed to it either, especially 
>>>>> if there's a way to import the subscribers to those lists...)
>>>>> 
>>>>> Cheers,
>>>>> 
>>>>> *Tobias McNulty
*Chief Executive Officer
>>>>> 
>>>>> 
>>>>> tob...@caktusgroup.com
>>>>> www.caktusgroup.com
>>>>> 
>>>>> 
>>>>> 
>>>>> On Mon, Nov 28, 2022 at 9:05 AM Carlton Gibson  
>>>>> wrote:
>>>>>> Hey Roger,  
>>>>>> 
>>>>>> Indeed it does. You can set up Email Mode (that may not be the actual 
>>>>>> name) and it’ll work just like a mailing list. 
>>>>>> 
>>>>>> You can also subscribe to just a particular category, so the Internals 
>>>>>> one would map to the discussion on this list. 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Monday, 28 November 2022, Roger Gammans  
>>>>>> wrote:
>>>>>> 
>>>>>>> Hi
>>>>>>> 

DEP 12 (Steering Council) Fully Adopted

[Andrew Godwin]

Hi everyone,

I am pleased to report that we've completed the extended approval process for 
DEP 12, which was needed since it was a governance change. Both the Technical 
Board and the DSF Board voted to not require a vote from the membership on the 
change, so it is now officially adopted and I will begin implementation 
immediately (which is mostly going to be renaming things).

Thanks everyone for your comments and help refining this DEP! You'll be hearing 
more from myself or one of the other Steering Council members soon about our 
first proper "call for feature proposals", as it'll be timed right after the 
4.2 feature freeze in January.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/89537c96-377d-4e86-8a8e-5ef738576ea2%40app.fastmail.com.

Re: Can we move the activity on this list to the Forum now?

[Andrew Godwin]

I did some investigation of moving django-users and django-developers to 
the Forum right after DjangoCon; I wanted to see if we could import all the 
old posts too, which we probably could, but I'm not entirely sure of the 
utility of that.

I will say that the forum is a lot easier to moderate - the ability to 
moderate after a post has gone out, rather than gating all posts behind 
approval if they're untrusted, is a big step in itself, not to mention the 
ability to remove sensitive or offensive content once it's posted.

Andrew

On Monday, November 28, 2022 at 10:01:17 PM UTC-7 m...@kye.id.au wrote:

> IMO django-announce and django-updates serve a very different purpose and 
> I would be against moving them if it were suggested.
>
> I am incredibly strongly in favour of moving django-developers and 
> django-users to the forums. IMO being able to more easily trap people 
> misusing this list as a tech support channel is itself reason enough to 
> move. Beyond that, I’d argue that the plentiful UX issues with Google 
> Groups, and mailing lists in general, certainly don’t do the community any 
> favours in terms of getting more people on board. 
>
> Kye
> On 28 Nov 2022 at 11:40 PM +0800, 'Tobias McNulty' via Django developers 
> (Contributions to Django itself) , wrote:
>
> As someone who only just joined the forum -- I'm +1: 
>
>- The forum has seen great adoption from what I can tell (nearly half 
>the number of posts as django-developers during the same time period, not 
>bad given the mailing list's head start in subscribers).
>- It seems beneficial to house future conversations in a single place, 
>e.g., so members don't need to subscribe to both the mailing list and 
> forum 
>to get the full picture of current active development, set up two 
> different 
>sets of mail filters to tag things appropriately, etc... 
>
> Would the plan be to switch django-users as well? I think similar 
> arguments could be made for consolidating those...
>
> (On the other hand, I see little value in switching django-announce and 
> django-updates, but I'm not necessarily opposed to it either, especially if 
> there's a way to import the subscribers to those lists...)
>
> Cheers,
>
>
> *Tobias McNulty*Chief Executive Officer
>
> tob...@caktusgroup.com
> www.caktusgroup.com
>
>
> On Mon, Nov 28, 2022 at 9:05 AM Carlton Gibson  
> wrote:
>
>> Hey Roger,  
>>
>> Indeed it does. You can set up Email Mode (that may not be the actual 
>> name) and it’ll work just like a mailing list. 
>>
>> You can also subscribe to just a particular category, so the Internals 
>> one would map to the discussion on this list. 
>>
>>
>>
>> On Monday, 28 November 2022, Roger Gammans  
>> wrote:
>>
>>> Hi
>>>
>>> I can't speak for others, but I personally STRONGLY value the fact that 
>>> this discussion happens in my inbox, not on yet another website. 
>>>
>>> But perhaps the forum still supports this reading mode?
>>>
>>> On Mon, 2022-11-28 at 05:38 -0800, Carlton Gibson wrote:
>>>
>>> Hi all.  
>>>
>>> Given the issues with Tom's access to the mailing list here, and the 
>>> fact that the Forum has been active for a few years now, and is a great 
>>> success, I'd like to revisit whether we can move on-mass (all few of us :) 
>>> over there? 
>>>
>>> We'd enjoy the benefits of a much nicer system. We'd not have issues 
>>> such as the current, and we'd be one less item in the pocket of a 
>>> mega-corp. (You can rank those as you will :) 
>>>
>>> Initially when this can up (a long time ago) Andrew and Tom discussed 
>>> whether we could import the history here into the forum. I think that's 
>>> unnecessary. We can still access the history here (until such a day as 
>>> Google takes it away) at worst -- but, rather, if we can get an archive we 
>>> could import it into read-only Datasette instance[0] — and that would 
>>> likely be good enough. 
>>>
>>> Can we move now? 
>>>
>>> Thanks. 
>>>
>>> Kind Regards,
>>>
>>> Carlton
>>>
>>>
>>> [0]: I'd happily do this. 
>>>
>>> --
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-developers/101f4e6d-9b83-47ab-bb1b-b571402e037dn%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>>> --
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> 

Re: Technical Board vote on DEP 0012: The Steering Council

[Andrew Godwin]

Hi all,

I am pleased to announce that this vote has passed with 5 Board members voting 
+1 and no other votes.

I will now proceed to notify the DSF Board that they must hold a vote, as is 
required by DEP 10. Either they or I will update the list with information once 
that vote has been concluded and a result reached.

Andrew

On Wed, Nov 30, 2022, at 10:44 AM, Andrew Godwin wrote:
> Yes, I agree we can use the forum in future since it's less tied to Google.
> 
> Provided the current +5 vote carries through to the end of the voting period, 
> I will be suggesting that the Technical Board triggers the DEP 10 mechanism 
> where we move this to the membership for a vote once the DSF Board has 
> performed their vote, as it only seems appropriate.
> 
> Andrew
> 
> On Mon, Nov 28, 2022, at 9:30 PM, 'Adam Johnson' via Django developers  
> (Contributions to Django itself) wrote:
>> +1 from me
>> 
>> And +1 to using the forum in future
>> 
>> On Tue, 29 Nov 2022 at 00:23, charettes  wrote:
>>> +1 from me as well.
>>> 
>>> Le lundi 28 novembre 2022 à 08:32:01 UTC-5, carlton...@gmail.com a écrit :
>>>> Hi All. 
>>>> 
>>>> Tom Forbes is currently unable to post to the Google Group here, due to 
>>>> Currently Undiagnosed Google Account Weirdness™. 
>>>> 
>>>> For the record he's asked me to say that his vote is +1 in support of the 
>>>> change, but I feel we should probably move voting to the forum because of 
>>>> this. 
>>>> 
>>>> 
>>>> Kind Regards,
>>>> 
>>>> Carlton
>>>> 
>>>> 
>>>> On Monday, 28 November 2022 at 09:52:35 UTC+1 f.apo...@gmail.com wrote:
>>>>> Hi,
>>>>> 
>>>>> +1 from me, but I'd like to ask the wider community (ie DSF members) 
>>>>> whether they support this change.
>>>>> 
>>>>> While there has been some opposition on whether a change likes this will 
>>>>> actually change things, I think that given the overall good reception of 
>>>>> the proposal it is at least worth to try it.
>>>>> 
>>>>> Cheers,
>>>>> Florian
>>>>> On Friday, November 25, 2022 at 10:04:36 AM UTC+1 Mariusz Felisiak wrote:
>>>>>> Thanks!
>>>>>> 
>>>>>> Reminder: according to DEP 10 voting will end on *December 2nd, 2022* 
>>>>>> *AoE* (members of the Technical Board may change their votes at any time 
>>>>>> prior to closing of the final voting period).
>>> 
>>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to django-developers+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-developers/378f606b-5d70-4013-a8d7-e8abb605f178n%40googlegroups.com
>>>  
>>> <https://groups.google.com/d/msgid/django-developers/378f606b-5d70-4013-a8d7-e8abb605f178n%40googlegroups.com?utm_medium=email_source=footer>.
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/CAMyDDM1dGUTiKTezaYDbqF0PYv1%3DEf%2Ba_VqT1J%2ByP%2BbgbdePZQ%40mail.gmail.com
>>  
>> <https://groups.google.com/d/msgid/django-developers/CAMyDDM1dGUTiKTezaYDbqF0PYv1%3DEf%2Ba_VqT1J%2ByP%2BbgbdePZQ%40mail.gmail.com?utm_medium=email_source=footer>.
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/12cfb130-17d0-4f42-9d58-631c927f473f%40app.fastmail.com
>  
> <https://groups.google.com/d/msgid/django-developers/12cfb130-17d0-4f42-9d58-631c927f473f%40app.fastmail.com?utm_medium=email_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1d72d359-eff3-4b7d-90e7-2d971445b618%40app.fastmail.com.

Re: Technical Board vote on DEP 0012: The Steering Council

[Andrew Godwin]

Yes, I agree we can use the forum in future since it's less tied to Google.

Provided the current +5 vote carries through to the end of the voting period, I 
will be suggesting that the Technical Board triggers the DEP 10 mechanism where 
we move this to the membership for a vote once the DSF Board has performed 
their vote, as it only seems appropriate.

Andrew

On Mon, Nov 28, 2022, at 9:30 PM, 'Adam Johnson' via Django developers  
(Contributions to Django itself) wrote:
> +1 from me
> 
> And +1 to using the forum in future
> 
> On Tue, 29 Nov 2022 at 00:23, charettes  wrote:
>> +1 from me as well.
>> 
>> Le lundi 28 novembre 2022 à 08:32:01 UTC-5, carlton...@gmail.com a écrit :
>>> Hi All. 
>>> 
>>> Tom Forbes is currently unable to post to the Google Group here, due to 
>>> Currently Undiagnosed Google Account Weirdness™. 
>>> 
>>> For the record he's asked me to say that his vote is +1 in support of the 
>>> change, but I feel we should probably move voting to the forum because of 
>>> this. 
>>> 
>>> 
>>> Kind Regards,
>>> 
>>> Carlton
>>> 
>>> 
>>> On Monday, 28 November 2022 at 09:52:35 UTC+1 f.apo...@gmail.com wrote:
 Hi,
 
 +1 from me, but I'd like to ask the wider community (ie DSF members) 
 whether they support this change.
 
 While there has been some opposition on whether a change likes this will 
 actually change things, I think that given the overall good reception of 
 the proposal it is at least worth to try it.
 
 Cheers,
 Florian
 On Friday, November 25, 2022 at 10:04:36 AM UTC+1 Mariusz Felisiak wrote:
> Thanks!
> 
> Reminder: according to DEP 10 voting will end on *December 2nd, 2022* 
> *AoE* (members of the Technical Board may change their votes at any time 
> prior to closing of the final voting period).
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/378f606b-5d70-4013-a8d7-e8abb605f178n%40googlegroups.com
>>  
>> .
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/CAMyDDM1dGUTiKTezaYDbqF0PYv1%3DEf%2Ba_VqT1J%2ByP%2BbgbdePZQ%40mail.gmail.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/12cfb130-17d0-4f42-9d58-631c927f473f%40app.fastmail.com.

Technical Board vote on DEP 0012: The Steering Council

[Andrew Godwin]

Hi all,

As it seems discussion on DEP 12 has reached its end, and we received generally 
positive feedback, I am requesting a vote from the technical board on the 
following:

"Shall we accept DEP 12 and send it to the DSF Board for further approval?"

Note that as this is a governance change, it requires a 4/5 majority from the 
Technical Board, after which it will go to the DSF Board for a vote, and after 
which it *may* go to the membership for a vote if the two Boards believe that 
it should (you can read more about these requirements in DEP 10).

You can see the text of the DEP here: 
https://github.com/django/deps/blob/main/draft/0012-steering-council.rst

My vote is +1, as I am the author of the DEP and believe it is in the best 
interests of the longevity of the Django project and sustainable governance.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5d1ba4e9-6eca-41a8-aa4b-64d9ee2310fb%40app.fastmail.com.

Re: Proposal: Clarify individual members page

[Andrew Godwin]

Just want to pop in and say these are great ideas - feel free to copy me in 
on any PR if you want extra opinions!

On Tuesday, November 8, 2022 at 8:26:28 AM UTC-7 Carlton Gibson wrote:

> Great, Thanks Andrew. No urgency 
>
> On Tue, 8 Nov 2022 at 16:16, Andrew Mshar  wrote:
>
>> Will do, Carlton.
>>
>> Tim and Cory, thanks for the suggestions. I'll incorporate those in the 
>> PR and post here when it's ready. Probably not today, but I should be able 
>> to open it before the end of the week.
>>
>> Thanks,
>> Andrew
>>
>> On Tuesday, November 8, 2022 at 10:10:51 AM UTC-5 carlton...@gmail.com 
>> wrote:
>>
>>> Hey Andrew. 
>>>
>>> I had thought this was a Flatpage (stored in the database) but it's not. 
>>> The source is here: 
>>> https://github.com/django/djangoproject.com/blob/main/djangoproject/templates/members/individualmember_list.html
>>> If you wanted to open a PR suggesting your changes, that would be 
>>> amazing 朗
>>>
>>> Thanks. 
>>>
>>> Kind Regards,
>>>
>>> Carlton
>>>
>>> On Mon, 7 Nov 2022 at 19:51, Tim Allen  
>>> wrote:
>>>
 I'm of the opinion that if you care enough about Django to investigate 
 becoming a member of the DSF, that's enough of a qualification - it is 
 just 
 challenging to formalize that into proper text for the website. Maybe two 
 changes to encourage people to join:

- We could tweak *"Running Django-related events or user groups"  *
to *"Attending or organizing Django-related events or user groups"*.
- Add a sentence to the end of the first stanza: "The following are 
Individual Members of the Django Software Foundation. The DSF appoints 
individual Members in recognition of their service to the Django 
 community. 
If you would like to join the DSF, we welcome you. Please feel free to 
self-nominate for membership."

 Regards,

 Tim

 On Monday, November 7, 2022 at 11:12:41 AM UTC-5 cory...@gmail.com 
 wrote:

> Hey Andrew,
>
> Thanks for drafting this language and I think it looks great. As 
> someone who only recently applied after hearing it discussed on an 
> episode 
> of Django Chat[1], I'm all for the goals of making it more encouraging 
> and 
> accessible and think this is a great step in that direction.
>
> Here are a few minor thoughts to specific bits:
>
> Service to the Django community takes many forms. Here are some 
>> examples (non-exhaustive) of categories of work performed by members:
>>
>
> "performed by members" is a little ambiguous as to whether it means 
> "this is how we evaluate applicants" vs "this is what you'll do if part 
> of 
> the DSF". Since I think the intention is the former it might make sense 
> to 
> change to something like:
>
> *Service to the Django community takes many forms. Here are some 
> (non-exhaustive) examples of the categories of work that might qualify as 
> "service":*
>
> Borrowed the list of categories from Andrew Godwin's DEP for the 
>> update to the technical board. Per Tim's recommendation, do we want to 
>> include anything about the review process?
>>
>
> When I applied I didn't (and still don't, really) have any visibility 
> into the process, so it wasn't a deterrent for me, personally, but I 
> think 
> having information certainly wouldn't hurt. My two cents would be good to 
> put something in, but not necessarily if it slows down/stalls this change 
> if for whatever reason that isn't super easy, since I think this 
> represents 
> an improvement on its own.
>
> Also, I'm a little unsure about that last bit about applying, but I 
>> wanted to put something encouraging to folks to apply. Happy to reword 
>> that 
>> if someone has a better suggestion. I'd prefer that to having a full 
>> rubric 
>> for membership on this page, primarily because I think it would be very 
>> difficult to nail that down because the work that folks perform can be 
>> so 
>> disparate (must have run X django meetups, or triaged Y tickets). 
>>
>
> Definitely agree a rubric would cause more problems than it would help 
> at this stage. The goals of rubrics in terms of increasing objectivity 
> and 
> reducing bias are great, but as applied to the already-squishy definition 
> of "service to the community" it doesn't seem like a good fit here.
>
> Finally, this is wildly out of scope, but it may make sense to (either 
> here or separately) attempt to create a bit more content about what it 
> means to be an individual member of the DSF. That information is also 
> somewhat lacking, and having it somewhere may encourage more people to 
> apply. One possibility could be to link to one of the recent conference 
> talks[2][3] on the DSF. But wouldn't want that 

Re: Draft Steering Council DEP

[Andrew Godwin]

On Tue, Nov 1, 2022, at 6:54 AM, C. Kirby wrote:
> Having run the elections for the current technical board I agree with 
> Andrew's assessment that a more open requirement to run is a good idea. It 
> may create a bit more work on candidate verification for the DSF Board and 
> Fellows, but anything that can work to encourage new blood in the 
> "Leadership" of Django and the DSF has a +1 from me.
> 
> I don't have a strong feeling on renaming of the board, but as such don't 
> really see it as necessary.

It is not an absolute requirement from me, but I think it's a better 
description of what it should do and also helps us distinguish it from the 
"Technical Team" a bit more, so I will keep it in the DEP unless there's strong 
objection.

> 1. There are several of timelines and triggers listed in DEP 10. A section 
> that lays them out explicitly with references back to the details could be 
> immensely useful. A flow chart perhaps - what are all the things that happen 
> when the final release of a major version occurs. A list of trigger actions 
> would be useful as well - Fewer Than 3 remaining members of the technical 
> board - elections, Fewer than 3 mergers - select new Merger, etc. A TL;DR; 
> for the Technical Board on the "day to day" working of the board.

Agreed, and I think this is something we should do either way - if this DEP 
goes in, doubly so, as we want to have a single place to reference things 
rather than multiple DEPs.

> 
> 2. Probably controversial - an enforcement mechanism. The DSF Board has a 
> regulatory requirement to meet and follow the bylaws as a registered 
> non-profit. We can be subject to lawsuit if we don't. The Technical Board has 
> no such liability, except for the, perhaps stronger, moral liability to the 
> community. To be clear I am not suggesting a legal enforcement mechanism, but 
> perhaps a community one. I dearly hope that it would never be needed, but not 
> having one at all seems an oversight. Something along the lines of:
> "DEP 10 enforcement: Any DFS individual member may make a public statement of 
> no-confidence in the technical board by identifying a material breach of DEP 
> 10. Upon seconding by another individual member of the DSF the DSF Board 
> SHALL no later than the next scheduled board meeting evaluate the merits of 
> the statement of no-confidence. If the statement is found to be accurate and 
> correct the Board shall inform the Technical Board of the breach and provide 
> 2 weeks to rectify said breach. If the Technical Board fails to rectify the 
> breach in the time allotted Technical Board elections SHALL be triggered and 
> current members of the Technical Board shall be barred from running in the 
> no-confidence election"

Interesting - having the DSF board moderate that makes it more agreeable to me, 
though if we are going to introduce this we should _also_ introduce wording for 
what happens if we fail to elect a Board, as this makes it much more likely 
(barring the entire previous board from running).

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/46c7e2a6-78be-48f9-95f9-66b891d1d5d3%40app.fastmail.com.

Re: Draft Steering Council DEP

[Andrew Godwin]

On Sun, Oct 30, 2022, at 10:42 PM, James Bennett wrote:
> On Wed, Oct 26, 2022 at 4:34 PM Andrew Godwin  wrote:
>> __
>> 
>> I have copied in the DSF Members mailing list as it is a governance-related 
>> DEP, but if we could keep all discussion on the thread in the Django 
>> Developers mailing list, as per DEP 0001, that would be great.
> 
> My main concern remains the thing I've been saying repeatedly in the other 
> thread: how does this actually solve a problem that Django is facing right 
> now?

By widening the set of people who can run for the Board/Council.

> 
> We've established that the Technical Board was not carrying out its duties. 
> There's also been a claim advanced that probably there is not a replacement 
> slate of board members who would step up and have the capacity to do what DEP 
> 10 asks.
> 
> So, either this new DEP is intended to be a slight clarification of the 
> Technical Board's role, in which case I don't see how saying "same as before, 
> but this time we actually expect you to do the thing" solves the issue. Or 
> it's intended to be a first step toward a more active group, in which case I 
> don't see how it can succeed given that a lower required level of activity 
> has already failed.

It is intended to allow for a more active group by giving us a wider set of 
candidates who can run and thus a higher chance of people being on the 
Board/Council who want to be that active.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/347c6859-5315-45ff-80e6-5480b99e95a7%40app.fastmail.com.

Re: Proposal for Django Core Sprints

[Andrew Godwin]

Hi Paolo,

I do like the overall idea - a few thoughts below.

My first concern for this, which somewhat echoes James, is that trying to 
organise an additional in-person event that a large number of contributors are 
expected to go to is difficult. Funding considerations are one concern - we 
would need to make sure everyone whose companies did not cover them to go had 
their trip paid for - but time away from home is also something that we should 
be considerate of, be it for family or for anything else.

That, too, brings an interesting challenge - who do you invite (and presumably 
help pay for)? We no longer have a core team, and I don't think it's worth 
singling the Technical Board out in this regard - the Fellows are an obvious 
requirement if they can make it, but past that, it's a hard metric to gauge.

I do like your point about disconnecting it from needing to give a talk, though 
- conferences and sprints are certainly Different Moods for me, and it can be 
draining to be sociable for three days and *then* have to go into sprints and 
be even more so. I don't know if there's a useful resolution to this specific 
problem.

My second concern is almost the opposite of that, ironically, which is that I 
do believe there are some things that benefit from synchronous, in-person 
communication. A single week at the most recent DjangoCon has done wonders for 
me in that regard, and while I do believe it should be a hybrid event, I also 
think it should *not* be a remote-only event. While it is a personal opinion, I 
find in-person sprinting significantly easier (due to the nature of the work as 
opposed to purely coding) and would probably find it hard to attend a 
remote-only sprint.

The final question I would raise is that of location. There are two elements 
here - where in the world do you hold it, and what the actual venue is like.

There is no single location in the world you will ever find where everyone can 
attend - even with very patient Australians who are far too willing to sit on 
planes for ages. It's likely you'd have to geolocate in either the Americas or 
in Europe, and if you additionally want to host it in a location that is safe 
for all attendees, that cuts down the number of US states and European 
countries even more.

And then, the venue. In the modern world, I ideally want an *outdoors* sprint 
venue, so that we're not all stuck inside a room hoping the masks work well 
enough. That places limitations on climate and season and also availability of 
power - so many that it starts to look like an impossible task to satisfy this 
along with all the above.

I don't write this as a means of discouragement - I think there's a lot to be 
said for your idea, and even though I've outlined a lot of potential flaws, I 
think a flawed event in this fashion would be better than nothing happening at 
all! I just want to highlight all the talking points we're going to need to 
really think about and have answers to if we take this forward.

(I didn't even discuss how we might fund this, which is also a conversation to 
have, but waving our hands in the air and going "sponsorship" is enough for me 
to start with)

Andrew

On Wed, Oct 26, 2022, at 4:01 PM, Paolo Melchiorre wrote:
> Hi everyone,
> 
> Following Andrew Godwin's example, I too share with you a proposal
> that I made during DjangoCon US 2022 and that I have already shared
> with some of you.
> 
> Inspired by an article by Will Vincent, I wrote my proposal for Django
> Core Sprints, and its genesis, in an article and would also like to
> have the opinions of the Django community members interested in the
> idea:
> https://www.paulox.net/2022/10/26/about-my-proposal-for-the-django-core-sprints/
> 
> Ciao,
> Paolo
> -- 
> Paolo Melchiorre
> 
> https://www.paulox.net
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/CAKFO%2Bx6FAf-M14fKx5QS1K8rru2L3mOiR1R%3Dn6FzDk%3DrnaUq3A%40mail.gmail.com.
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ebd86080-6735-45f7-8dd7-3ddc23c4a44d%40app.fastmail.com.

Draft Steering Council DEP

[Andrew Godwin]

Hi all,

As a followup to my previous post about potential changes to the Technical 
Board - for which I thank you all for the feedback - I have taken the process 
to the next step and written a draft DEP:

https://github.com/django/deps/pull/75/files

(If you wish to see the DEP with styling, it can be viewed at 
https://github.com/andrewgodwin/deps/blob/steering/draft/0012-steering-council.rst)

I could just merge this into the DEPs repo as a draft given our process, but I 
would like to invite both review of the DEP styling and layout (to ensure it's 
ready to merge in that regard) as well as initial feedback on its content as 
well.

I have copied in the DSF Members mailing list as it is a governance-related 
DEP, but if we could keep all discussion on the thread in the Django Developers 
mailing list, as per DEP 0001, that would be great.

Thanks,
Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4e6893a8-8e22-45a3-96f6-d56d69db7b43%40app.fastmail.com.

Re: Changing the role of the Technical Board

[Andrew Godwin]

My intention is indeed for us to run a new Technical Board election come the 
end of 4.2, with much better and more explicit communication about what will be 
expected of the new members, and a larger candidate pool to pull from to 
hopefully make that work.

I will be posting my actual proposed DEP shortly so it's more clear exactly 
what I want to change at a written-rules level - I suspect feedback on a more 
concrete proposal will help us talk about it more clearly.

Andrew

On Wed, Oct 26, 2022, at 4:55 PM, James Bennett wrote:
> On Wed, Oct 26, 2022 at 12:02 PM Andrew Godwin  wrote:
>> At this point, it is my view that it is our job to govern with the people we 
>> have, and the time and energy they can provide, and that's my intention with 
>> these suggested changes.
> 
> If the problem in front of us is that the Technical Board isn't up to the 
> level of activity DEP 10 asks of them, and you believe no substitute group of 
> members exists that would be up to it either, I'm not sure I see how your 
> proposals are going to fix that, especially since you seem to eventually want 
> to create *more* responsibility for active intervention and leadership.
>> 
> 
>> At the end of the day, my feeling is that inaction is not the right path - 
>> we need to enact some sort of change. I'm more than willing to hear 
>> alternative suggestions for what that change should be (though as outlined 
>> previously, I really don't think that change should be "remove the entire 
>> current Board for underperformance and have another election").
> 
> A Technical Board election will automatically trigger at the release of 
> Django 4.2, which is not *that* far off. But the Technical Board could always 
> trigger an election any time they want to.
> 
> I personally would still like to understand how the current Technical Board 
> came to what seems to be such a misunderstanding of how the governance was 
> supposed to work. And while there have been explanations presented for why 
> the Technical Board didn't communicate the problems it was having, they also 
> come across as worrying -- we're all adults here, and we need to be able to 
> trust each other to speak up when there's a problem. How did we go, 
> apparently, multiple years with the Technical Board not carrying out their 
> responsibilities and also nobody saying anything about it?
> 
> Until we understand that I don't think we should be trying to change the 
> governance again.
> 
> And I still would like to see DEP 10 actually tried out. Maybe it involves 
> electing a Technical Board with much more explicit up-front communication of 
> expectations, since it seems a lot of the current members were unaware of the 
> Technical Board's actual responsibilities. Maybe it involves someone just 
> constantly poking them with reminders. Maybe something else. But it feels 
> wrong on many levels to start moving on from DEP 10 when it's becoming 
> increasingly clear that DEP 10 was never really *tried*.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/CAL13Cg_%3DbczbtMP5B6avbNTLi%3DLf1SU_Z%3Dchz6j0u5L1MPMbzw%40mail.gmail.com
>  
> <https://groups.google.com/d/msgid/django-developers/CAL13Cg_%3DbczbtMP5B6avbNTLi%3DLf1SU_Z%3Dchz6j0u5L1MPMbzw%40mail.gmail.com?utm_medium=email_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/88084dd9-a2e9-4271-9160-79bb463a8adc%40app.fastmail.com.

Re: Changing the role of the Technical Board

[Andrew Godwin]

I agree the Technical Board has not followed the letter of DEP 10, and I think 
the things you have highlighted are all valid failings, but I want to focus on 
- what should we do to remedy them?

Given the lack of candidates we already have, if we ditch the current Board and 
try to elect a new one that has to do all these things, it's likely we will 
fail to have a valid election, and from what I recall, there is no provision 
for that in DEP 10.

We could consider changing to a board size of three to overcome this, but that 
would increase the workload even further.  We could hope that five incredibly 
willing people are waiting in the wings, but I doubt it.

At this point, it is my view that it is our job to govern with the people we 
have, and the time and energy they can provide, and that's my intention with 
these suggested changes.

I honestly think you and I both want the Board to do the same rough role - my 
current draft "extension DEP" on top of DEP 10 is literally just "change the 
name and tweak the eligibility clauses", it's not like we're going to throw it 
out. I just want to come at this a bit more incrementally with our current set 
of people, and throw the net a bit wider so we do have more of a chance of 
finding five people with the energy to run the tasks assigned to the Board with 
the vision we initially had.

At the end of the day, my feeling is that inaction is not the right path - we 
need to enact some sort of change. I'm more than willing to hear alternative 
suggestions for what that change should be (though as outlined previously, I 
really don't think that change should be "remove the entire current Board for 
underperformance and have another election").

Andrew

On Wed, Oct 26, 2022, at 12:23 PM, James Bennett wrote:
> I'm going to avoid trying to get too much into point-by-point back-and-forth 
> argument here, and try to present the larger picture.
> 
> The Technical Board has multiple active responsibilities under DEP 10. Let's 
> look at some of them:
> 
> 1. Canvas for feature proposals once per feature release of Django. This also 
> comes with a responsibility to ensure the proposals are archived in a useful 
> way -- we used to have wiki pages in Trac for this, but DEP 10 doesn't 
> require any specific mechanism. This is supposed to happen within one week 
> after feature freeze of each feature release.
> 
> 2. Set the release schedule for each feature release of Django. This is 
> supposed to happen within one week after the prior feature release coming out.
> 
> 3. Maintain the teams of Mergers and Releasers, including by nominating and 
> confirming new members of those teams when the Technical Board considers it 
> necessary (or when the current roster falls below the DEP 10 quorum).
> 
> 4. Vote, as requested, on DEPs.
> 
> 5. Vote, as requested, on any technical decision that fails to resolve 
> through normal public discussion.
> 
> (1) has not been happening. It's possible that (2) has been happening and 
> just hasn't been that formally published, but I suspect it's been the Fellows 
> who've been doing that one. For 3-5, things have gone... not so great.
> 
> It took multiple attempts to get the Technical Board to vote on the first 
> Merger nomination (Claude), and if we're being pedantic it still wasn't quite 
> done correctly because his appointment as a Merger should have been temporary 
> and auto-expired after the first Technical Board election unless the new 
> Board voted to confirm him permanently.
> 
> The first time the Technical Board voted on a DEP was the proposed DEP 484 
> for type hints. The Technical Board's discussion and voting on it apparently 
> took place entirely in private and the result was communicated publicly via 
> Carlton copy/pasting the reply he got from them. And we lucked out that the 
> Board didn't accept the DEP, because it was at a time when an election was 
> pending and thus they had no power to accept a DEP (similar to the Merger 
> nomination -- once an election triggers, the Board cannot accept DEPs and can 
> only make temporary appointments of Mergers/Releasers).
> 
> On (5) the only explicit request I can find in this mailing list's archive is 
> one from Carlton to make a decision on ticket 31747. Although several 
> Technical Board members posted opinions in response, I'm not finding any 
> statements of their votes on the matter.
> 
> If grant a half point for the Merger nomination (though the Board really 
> ought to hold a public vote to properly permanently confirm Claude), and 
> maybe another half point for the fact that release schedules have been 
> getting set (though, I suspect, not explicitly by the Board), that gets us to 
> 1 out of a possible 5. This is not a great track record.
> 
> Before we propose a different setup, I think we need to figure out what went 
> wrong here. It already seems like some members of the Technical Board may not 
> have really been familiar with the 

Re: Changing the role of the Technical Board

[Andrew Godwin]

On Tue, Oct 25, 2022, at 12:12 AM, James Bennett wrote:
> 
> My first reaction to this is: if having a DEP that says the Technical Board 
> is supposed to take the lead in gathering feature proposals didn't get them 
> to do it, it doesn't feel like another DEP saying they're responsible for 
> that is going to fix it.

I agree. Me proposing the DEP changes is mostly merely formalising some other 
changes I want to catalyse here - the DEP is an outcome, not the start, here.

> Getting burned out or overcommitted is a thing that happens, and a thing that 
> was anticipated in drafting the governance -- DEP 10 has a procedure for it!
> 
> Why did no member of the Technical Board do that?

Again, speaking for myself - because we were doing almost all the functions of 
the Board except for the feature canvassing. I also saw the relative lack of 
candidates for Board elections and essentially thought "better burnt-out me 
than literally nobody".

> And from the sound of what you're saying in this thread, the Technical Board 
> is mostly communicating with itself about this, in private, when the 
> direction of Django is supposed to be worked out publicly and transparently. 
> That's why we shut down the old private communication spaces for the former 
> "core team" after DEP 10 was adopted.

That is not the case - this thread is the majority of the discussion. I opened 
a thread on the TB mailing list in case people there wanted to give specific, 
blunt feedback about how they would feel about being affected by this as 
current Board members, but it has only a couple of replies, and nothing that 
hasn't been discussed here.

> So forgive me for being blunt, but: if the Technical Board is not following 
> the governance we have, I think replacing the Technical Board's current 
> membership should be higher on the list of remedies than replacing the 
> governance.

Forgive me for being blunt but... that seems like a really bad idea? The 
current board ran uncontested, so if all five of us step down I suspect there 
may not *be* a Board at the end of the day. The DSF Board's current 
difficulties finding candidates I think reinforces that fact - one of the 
changes I was considering bringing in here was "what if we have to reduce the 
TB to 4 or 3 people in the near future".

Again, I'm not saying "we should write a new DEP and *that'll* fix it", I'm 
trying to come from a position of working out what we can and should be 
*doing*, and then ensuring our rules match that.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e04f4e94-7466-4d58-b0d8-21a5e4ba3faa%40app.fastmail.com.

Re: Changing the role of the Technical Board

[Andrew Godwin]

> Has that not been occurring? Because if it hasn't, then we have a major 
> problem, and I don't see how the current proposal would resolve it.

It has not. While I cannot speak for the other members of the Board, I got 
burnt out in 2019, and then the pandemic began, and so it has not really been 
something I've pushed for in the past three years (and I believe I was one of 
the drivers of getting that in DEP 10 in the first place). We never really got 
into the rhythm of doing it, and I think a lot of us got busy or burnt out.

My personal belief is that if something is not working, it is time for a 
re-analysis of the situation and a change, even if what's written down in the 
rules is fine on its face.

This proposal is part of a larger set of community changes I would like to 
consider - I am, for obvious reasons, not blasting the community with them all 
at once, but I do believe that a strong, visible set of leaders, with a clearly 
outlined set of principles and shepherding a community-guided vision, is the 
most important thing to establish - both in terms of the Technical Board and 
the DSF Board.

I have a much longer blog post in me about my evolving belief in the need for 
visible, servant leaders in OSS communities rather than trying to embrace a 
flat hierarchy with mechanical checks and balances - but that is for another 
day.

Andrew

On Mon, Oct 24, 2022, at 4:26 PM, James Bennett wrote:
> On Mon, Oct 24, 2022 at 2:24 PM Andrew Godwin  wrote:
>> __
>> Proposing features - this is already in DEP 10, so I more just want to get 
>> that aspect of the Board actually going (and, as a side-effect, have 
>> something to aid fundraising). I am talking with the current Board 
>> separately on an internal thread, where the current stance (not everyone has 
>> responded) is that I am personally happy to take on all the work here for 
>> now - but I want to make sure it's not just me in the long run, be that 
>> merely proving that the idea works or attracting board members who want to 
>> specifically mediate such discussions and interaction with the wider 
>> community.
> 
> I admit I haven't been following Django development all that closely since I 
> mic-dropped after DEP 10 passed, but this is worrying, because canvassing for 
> feature proposals is not an optional thing -- DEP 10 *requires* the Technical 
> Board to do this at least once per feature release of Django. Has that not 
> been occurring? Because if it hasn't, then we have a major problem, and I 
> don't see how the current proposal would resolve it.
> 
>> Overall, if all we do is change the name and start actually doing calls for 
>> features as outlined in DEP 10, I'll honestly be happy - but I think, given 
>> the most recent TB election was uncontested and several long-time Django 
>> contributors have told me they'd be more willing to join a TB that was less 
>> strictly technical-all-the-time, that it makes sense for us to also look at 
>> those requirements.
>>> 
> I have concerns about this because what it really feels like is a first step 
> toward merging the Technical Board and the DSF Board into a single unified 
> governance board of everything to do with Django, and I think the split 
> between governance of Django-the-codebase and 
> Django-the-everything-the-DSF-does is a useful and important one (not least 
> for legal reasons). 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/CAL13Cg93CvLgVU567MsvqKZAiWVBxxm1-L_YuEUhjr_JRySBwg%40mail.gmail.com
>  
> <https://groups.google.com/d/msgid/django-developers/CAL13Cg93CvLgVU567MsvqKZAiWVBxxm1-L_YuEUhjr_JRySBwg%40mail.gmail.com?utm_medium=email_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6196974e-a1a5-4585-87ab-02f79509%40app.fastmail.com.

Re: Changing the role of the Technical Board

[Andrew Godwin]

These are some great points, James - let me try to tackle them roughly in order.

Proposing features - this is already in DEP 10, so I more just want to get that 
aspect of the Board actually going (and, as a side-effect, have something to 
aid fundraising). I am talking with the current Board separately on an internal 
thread, where the current stance (not everyone has responded) is that I am 
personally happy to take on all the work here for now - but I want to make sure 
it's not just me in the long run, be that merely proving that the idea works or 
attracting board members who want to specifically mediate such discussions and 
interaction with the wider community.

Engagement - It's not about "lack of engagement", and I think any issues there 
are deeper problems with OSS communities and the fact that we have to learn to 
sustainably work with the people we have rather than throwing everything at 
trying to recruit fresh, new people. I have ideas around this topic 
specifically, but they will not be solved in terms of the Board alone.

Loosening eligibility - If you're up for it I would very much value your help 
here in terms of refining wording once I have a first draft. My initial 
direction was to still require the 18 month history of contributions, but widen 
it from "technical" to more kinds of work (obviously the discussion part is in 
there too, but I think in general we can do a bit more of an OR rather than an 
AND on the current requirements, keeping a minimum time of contribution to 
prevent bad actors)

Serving on the DSF Board - you are of course right, I misread the DEP last week.

Overall, if all we do is change the name and start actually doing calls for 
features as outlined in DEP 10, I'll honestly be happy - but I think, given the 
most recent TB election was uncontested and several long-time Django 
contributors have told me they'd be more willing to join a TB that was less 
strictly technical-all-the-time, that it makes sense for us to also look at 
those requirements.

Andrew

On Mon, Oct 24, 2022, at 2:54 PM, James Bennett wrote:
> Something I note here is that it's presenting a solution, but not clearly (at 
> least, from my reading) presenting the problem to be solved.
> 
> Is it a lack of people proposing features? If so, I'm not sure this helps -- 
> it would, to me, suggest that only members of the Technical Board/Steering 
> Council/whatever-it's called are supposed to do that, since it's in their job 
> description. Would people then expect to, or be expected to, run for a seat 
> in order to contribute something?
> 
> Is it a more general lack of engagement? If so, I'm still not sure how this 
> helps -- the idea of DEP 10 was to make it *easier* for people to step up and 
> get involved, since it got rid of the idea of the "core team" with their 
> special privileges, but I don't think any form of technical governance 
> actually solves engagement issues. At best it can make engagement-specific 
> efforts easier, but I don't see how re-centralizing authority (or creating 
> the impression of it) would achieve that.
> 
> Is it to make fundraising easier? That sounds again like something that 
> technical governance really can't do on its own -- it needs to involve the 
> DSF Board, and there are reasons why the DSF was historically wary about 
> doing targeted fundraising for specific features in Django.
> 
> Loosening eligibility is fine, though I agree it's going to be very difficult 
> to write down in an enforceable way -- the DEP 10 language and process was 
> intended primarily to prevent trolls and other bad-faith actors from being 
> able to run effectively for the Technical Board, and there's a balance where 
> the more you loosen it up, the more you also open the door for those kinds of 
> people.
> 
> Also, regarding the multiple roles restriction: it currently is allowed for a 
> single person to simultaneously be on both the Technical Board and the DSF 
> Board, and there are even procedures in DEP 10 for things like mandatory 
> recusal for DSF Board votes and actions that affect the Technical Board. 
> What's not allowed is simultaneously being a Merger and on the Technical 
> Board.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/CAL13Cg-mWWK0%2Bzkvi%3DCWu0e%3DbX-rOsLq4gHHdBuKQ8UL_8pRSg%40mail.gmail.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe 

Changing the role of the Technical Board

[Andrew Godwin]

Hi everyone,

I want to start a conversation about the Technical Board and its role in 
Django, and how I'd like to change it, including its name.

Since its inception, the Technical Board has effectively only functioned as a 
backstop vote for large features that require DEPs, of which there have been 
only two in the last five years. I believe it can be much more useful than 
this, while still not dominating Django's direction.

I would like to initially suggest the following high-level changes, that I will 
start formalising into a process DEP if we reach broad agreement:

*Eligibility requirements are loosened and made less code-oriented*

Anyone who demonstrates a "decent" history of contributing to Django in any 
fashion, and who does not have any conflicts of interest, and clearly wants to 
work in the best interests of the framework, would be eligible. (Yes, this is 
going to be fun to define in a DEP).

The current eligibility requirements include the need to have participated in 
discussions about the direction or development of Django in the last two years, 
which given the relative lack of big discussions in the last two years makes it 
rather problematic.

*A more active role is taken in suggesting features*

A regular "call for big ideas" is taken by the Board, and a clear set of "these 
are the technical/design/process/etc. ideas that we'd like to do" is published. 
This list would be, among other things, useful to raise money for grants to 
work directly on those features.

DEP 10 already mostly provides for this - it says that the Board should "put 
out calls for proposals and ideas for the future technical direction of 
Django", so this would more be us just Doing The Thing rather than changing too 
many rules.

*The name is changed to "Steering Council"* (like Python)

I believe this is a more accurate reflection of what the group *should* be 
doing and a reflection that you should not need to be directly coding Django 
every day to participate - and names have meaning, and thus power.

*The overall election process, other powers, and current members remain the 
same*

I think this part works relatively well. I did consider if we should allow 
simultaneous Technical Board/Steering Council membership *and* DSF Board 
membership, as this is currently banned, but I think the reasons behind this 
still hold for now.

This of course would be a "major change" and trigger a lot of procedure, 
notably a supermajority vote of the current Board, and a DSF Board vote, but I 
do believe it is in the best interests of the framework.

I am very interested in any feedback on these suggested ideas, including any 
additional changes you think might be appropriate that I have not covered here.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4d7a9a73-f231-4f01-93c9-59ba32b67a99%40app.fastmail.com.

Re: deconstruct returns 3-tuple?

[Andrew Godwin]

As the note in that section explains, that is for custom *classes*, not for 
custom *fields*. Your observation is correct when applied to field 
deconstruction methods, but not for the system talked about there which lets 
you do it for arbitrary classes (as part of the values in field 
desconstructions, usually).

Andrew

On Sat, Sep 18, 2021, at 9:06 AM, Christian González wrote:
> Hi,
> 
> before I issue a bugreport, I'll ask here first.
> 
> On 
> https://docs.djangoproject.com/en/3.2/topics/migrations/#adding-a-deconstruct-method
>  I can read that deconstruct() returns a 3-tuple.
> 
> When I see the code in django.db.models.fields.__init__.py, 
> Field.deconstruct() does a
> 
> *return *(self.name, path, [], keywords)
>
> 
> This is a 4-tuple, and even the clone() method written after it calls it with
> 
> name, path, args, kwargs = self.deconstruct()
> So, is 3.2 documentation here on a <1.9 status? Am I understanding something 
> not correctly?
> 
> Thanks,
> 
> Christian
> 
> -- 
> Dr. Christian González
> https://nerdocs.at
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/84db36b4-dc6a-fcc7-28e4-4810d152d2fb%40nerdocs.at
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e4c9b368-3fa9-47f4-80c2-22998551cf28%40www.fastmail.com.

Re: Do people actually squash migrations?

[Andrew Godwin]

Migration squashing was always meant to be something that was useful in a rapid 
development environment where you can't control all the installs (since it was 
a feature developed alongside a CMS run by many clients at the time).

If you have control of all the places your project is installed and can have 
everyone developing on it do something at once, you can just do it via a 
complete migration reset and be happy. Squashing is for a very specific 
backwards-compatibility scenario that, I suspect, many Django projects 
developed internally are not in.

Andrew

On Tue, May 11, 2021, at 7:58 PM, Kye Russell wrote:
> I’ve never successfully squashed my migrations to any material degree, but 
> I’ve chalked that up to lack of doing it with any regularity. I suspect that 
> squashing works a lot better if you aren’t trying to clean up a mess of 
> hundreds of migrations files over 5 years, which is where I find myself! 
> 
> Kye
> On 12 May 2021, 9:31 AM +0800, Matthew Pava , wrote:
> 
>> I've had similar issues. I just avoid squashing anymore. It's just not with 
>> the pain, and having so many little files that get looked at a minimal 
>> amount of time isn't worth fretting over. Saying that, I'd love to get it 
>> fixed...
>> 
>> 
>> Sent from my Verizon, Samsung Galaxy smartphone
>> 
>> Get Outlook for Android 
>> 
>> 
>> *From:* 'Mike Lissner' via Django developers (Contributions to Django 
>> itself) 
>> *Sent:* Tuesday, May 11, 2021 7:50:31 PM
>> *To:* Django developers (Contributions to Django itself) 
>> 
>> *Subject:* Do people actually squash migrations? 
>>  
>> I have a pretty big django project, and since I created the 100th migration 
>> within one of its apps today, I thought I'd finally do some squashing. It 
>> hasn't gone well, but I eventually got the data migrations cleaned up.
>> 
>> Finally, I run it, and it runs smack into a CircularDependencyError, as 
>> described here:
>> 
>> https://code.djangoproject.com/ticket/23337
>> 
>> Basically, from what I understand, after the squash you have one migration 
>> that depends on various others from your other apps. Naturally, that totally 
>> falls over, because can't go from this series of migrations:
>> 
>> app1: migration 1
>> app2: migration 1
>> app2: migration 2
>> app1: migration 2
>> 
>> To, well...any series of migrations in which migration 1&2 from app1 or app2 
>> have been squashed. The docs have something to say about this*, but it feels 
>> like this must affect practically any biggish project.
>> 
>> Stackoverflow also has a variety of dubious (and very complex) advice (read 
>> it and weep):
>> 
>> https://stackoverflow.com/questions/37711402/circular-dependency-when-squashing-django-migrations
>> 
>> So, my question is: Do people actually use squashmigrations with success? 
>> And if not, is it reasonable to consider deprecating it or fixing the bug, 
>> or updating the docs to loudly say it largely doesn't work? I'm surprised 
>> the issue above has so little movement since it was created seven years ago.
>> 
>> Maybe it's just me? If not, it'd be nice to do something to help future 
>> people with ambitions of a simple squash.
>> 
>> Thanks,
>> 
>> 
>> Mike
>>  
>> * Note that model interdependencies in Django can get very complex, and 
>> squashing may result in migrations that do not run; either mis-optimized (in 
>> which case you can try again with --no-optimize, though you should also 
>> report an issue), or with a CircularDependencyError, in which case you can 
>> manually resolve it.
>> To manually resolve a CircularDependencyError, break out one of the 
>> ForeignKeys in the circular dependency loop into a separate migration, and 
>> move the dependency on the other app with it. If you’re unsure, see how 
>> makemigrations 
>> 
>>  deals with the problem when asked to create brand new migrations from your 
>> models. In a future release of Django, squashmigrations 
>> 
>>  will be updated to attempt to resolve these errors itself. [Author's note: 
>> These sentences really leave me blowing in the wind...maybe I can figure out 
>> what they mean, I guess? I thought squashing was supposed to be easy.]

>> 
>> 
>> 

>> --
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group 

Re: Revisiting Python support for after Django 3.2 LTS

[Andrew Godwin]

I agree we should not be quite so beholden to our existing Python version 
policy - that was mostly to get us out of the early 3.x era. Now things are 
more stable, I'd support a policy that is much more like "any stable version of 
Python currently out there and supported".

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e1f40b9d-a017-4507-b855-2055713d960e%40www.fastmail.com.

Re: Welcome email

[Andrew Godwin]

Just to keep everyone updated - it looks like the import instructions/script 
are for self-hosted instances rather than Discourse-hosted instances, so I've 
pinged their support to see how best to proceed. Hopefully I can get a trial 
import done this weekend.

(worst case, I'll write my own script to mash the APIs over HTTP)

Andrew

On Sun, Nov 8, 2020, at 2:36 PM, Andrew Godwin wrote:
> I have been moving house this week (plus, yknow, the election) so I haven't 
> got anything done, but hope to poke at it early next week!
> 
> Andrew
> 
> On Sun, Nov 8, 2020, at 4:34 AM, Carlton Gibson wrote:
>> Hi Tom and Andrew. 
>> 
>> This week has been *busy *(shall we say) I know — can I assume you two are 
>> on this between you, or is there anything I can do to facilitate progress 
>> here? 
>> 
>> Thanks. 
>> Carlton.
>> 
>>> On 30 Oct 2020, at 13:25, Tom Forbes  wrote:
>>> 
>>> The export finally finished after a whole day running.
>>> 
>>> The gzip compressed mbox: 
>>> https://www.icloud.com/iclouddrive/09XBHCr5L5BNcOBLob_HA2x6g 
>>> <https://www.icloud.com/iclouddrive/09XBHCr5L5BNcOBLob_HA2x6g#django-messages>
>>> The entire raw output from the tool: 
>>> https://www.icloud.com/iclouddrive/0LY_ccOgG6VjfV3INexGMXwZQ 
>>> <https://www.icloud.com/iclouddrive/0LY_ccOgG6VjfV3INexGMXwZQ#raw_django_messages>
>>> 
>>> We’ve got 56,282 total messages. I think the script from Discourse I linked 
>>> above would be the perfect way to import these as it creates the users on 
>>> demand and isn’t limited by the external Discourse API. I’ll leave that to 
>>> anyone who has access and is willing (Andrew?).
>>> 
>>> Tom
>>> 
>>> On 29 Oct 2020 at 16:43:07, Tom Forbes  wrote:
>>>> It actually might be simpler than I suggested. Discourse has a script to 
>>>> import an mbox: 
>>>> https://github.com/discourse/discourse/blob/master/script/import_scripts/mbox.rb
>>>> 
>>>> I’m running the box importer script now and it appears to work fine. While 
>>>> some people might have historical mbox files, it might be better to get 
>>>> the source of truth from Google Groups?
>>>> 
>>>> Tom
>>>> 
>>>> 
>>>> On 29 Oct 2020 at 16:11:04, Andrew Godwin  wrote:
>>>>> I'd be more than happy to assist a trial of moving things to the forum; 
>>>>> we've had it running for over a year now, and I feel it's a much easier 
>>>>> way to run a community.
>>>>> 
>>>>> Among other things, we can:
>>>>> - Move posts to the right forum when they post in the wrong one (rather 
>>>>> than emailing back and saying "please post again over here")
>>>>> - Delete people's personal information when they accidentally post it
>>>>> - Remove CoC-violating posts and not leave any history
>>>>> 
>>>>> Either I can try the import out if someone pops the right permissions 
>>>>> over to me, or I'm happy to supervise enough forum API access for you to 
>>>>> try it, Tom.
>>>>> 
>>>>> Andrew
>>>>> On Thursday, October 29, 2020 at 9:23:45 AM UTC-6 carlton...@gmail.com 
>>>>> wrote:
>>>>>> I don’t have any controls here. I’m pretty sure Florian would be the 
>>>>>> most likely candidate. (No doubt it’s Jacob.) 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> -- 
>>>>> You received this message because you are subscribed to the Google Groups 
>>>>> "Django developers (Contributions to Django itself)" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>>>> email to django-developers+unsubscr...@googlegroups.com.
>>>>> To view this discussion on the web visit 
>>>>> https://groups.google.com/d/msgid/django-developers/44e041d6-9397-4a73-87a0-e4643c034db0n%40googlegroups.com
>>>>>  
>>>>> <https://groups.google.com/d/msgid/django-developers/44e041d6-9397-4a73-87a0-e4643c034db0n%40googlegroups.com?utm_medium=email_source=footer>.
>>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to django-developers+unsubscr...@goo

Re: Welcome email

[Andrew Godwin]

I have been moving house this week (plus, yknow, the election) so I haven't got 
anything done, but hope to poke at it early next week!

Andrew

On Sun, Nov 8, 2020, at 4:34 AM, Carlton Gibson wrote:
> Hi Tom and Andrew. 
> 
> This week has been *busy *(shall we say) I know — can I assume you two are on 
> this between you, or is there anything I can do to facilitate progress here? 
> 
> Thanks. 
> Carlton.
> 
>> On 30 Oct 2020, at 13:25, Tom Forbes  wrote:
>> 
>> The export finally finished after a whole day running.
>> 
>> The gzip compressed mbox: 
>> https://www.icloud.com/iclouddrive/09XBHCr5L5BNcOBLob_HA2x6g 
>> <https://www.icloud.com/iclouddrive/09XBHCr5L5BNcOBLob_HA2x6g#django-messages>
>> The entire raw output from the tool: 
>> https://www.icloud.com/iclouddrive/0LY_ccOgG6VjfV3INexGMXwZQ 
>> <https://www.icloud.com/iclouddrive/0LY_ccOgG6VjfV3INexGMXwZQ#raw_django_messages>
>> 
>> We’ve got 56,282 total messages. I think the script from Discourse I linked 
>> above would be the perfect way to import these as it creates the users on 
>> demand and isn’t limited by the external Discourse API. I’ll leave that to 
>> anyone who has access and is willing (Andrew?).
>> 
>> Tom
>> 
>> On 29 Oct 2020 at 16:43:07, Tom Forbes  wrote:
>>> It actually might be simpler than I suggested. Discourse has a script to 
>>> import an mbox: 
>>> https://github.com/discourse/discourse/blob/master/script/import_scripts/mbox.rb
>>> 
>>> I’m running the box importer script now and it appears to work fine. While 
>>> some people might have historical mbox files, it might be better to get the 
>>> source of truth from Google Groups?
>>> 
>>> Tom
>>> 
>>> 
>>> On 29 Oct 2020 at 16:11:04, Andrew Godwin  wrote:
>>>> I'd be more than happy to assist a trial of moving things to the forum; 
>>>> we've had it running for over a year now, and I feel it's a much easier 
>>>> way to run a community.
>>>> 
>>>> Among other things, we can:
>>>> - Move posts to the right forum when they post in the wrong one (rather 
>>>> than emailing back and saying "please post again over here")
>>>> - Delete people's personal information when they accidentally post it
>>>> - Remove CoC-violating posts and not leave any history
>>>> 
>>>> Either I can try the import out if someone pops the right permissions over 
>>>> to me, or I'm happy to supervise enough forum API access for you to try 
>>>> it, Tom.
>>>> 
>>>> Andrew
>>>> On Thursday, October 29, 2020 at 9:23:45 AM UTC-6 carlton...@gmail.com 
>>>> wrote:
>>>>> I don’t have any controls here. I’m pretty sure Florian would be the most 
>>>>> likely candidate. (No doubt it’s Jacob.) 
>>>>> 
>>>>> 
>>>> 
>>>> -- 
>>>> You received this message because you are subscribed to the Google Groups 
>>>> "Django developers (Contributions to Django itself)" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>>> email to django-developers+unsubscr...@googlegroups.com.
>>>> To view this discussion on the web visit 
>>>> https://groups.google.com/d/msgid/django-developers/44e041d6-9397-4a73-87a0-e4643c034db0n%40googlegroups.com
>>>>  
>>>> <https://groups.google.com/d/msgid/django-developers/44e041d6-9397-4a73-87a0-e4643c034db0n%40googlegroups.com?utm_medium=email_source=footer>.
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/CAFNZOJOAVFTNe%2BdJBjoJ8MWChswmLAZiGzqRPmPKYuVwuMqqiw%40mail.gmail.com
>>  
>> <https://groups.google.com/d/msgid/django-developers/CAFNZOJOAVFTNe%2BdJBjoJ8MWChswmLAZiGzqRPmPKYuVwuMqqiw%40mail.gmail.com?utm_medium=email_source=footer>.
> 
> 

> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/CF8FD1E3-9E28-4BBF-9C77-880817A7A576%40gmail.com
>  
> <https://groups.google.com/d/msgid/django-developers/CF8FD1E3-9E28-4BBF-9C77-880817A7A576%40gmail.com?utm_medium=email_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6c289804-7c7e-49e3-ace0-6e087586f1a6%40www.fastmail.com.

Re: Welcome email

[Andrew Godwin]

I'd be more than happy to assist a trial of moving things to the forum; 
we've had it running for over a year now, and I feel it's a much easier way 
to run a community.

Among other things, we can:
- Move posts to the right forum when they post in the wrong one (rather 
than emailing back and saying "please post again over here")
- Delete people's personal information when they accidentally post it
- Remove CoC-violating posts and not leave any history

Either I can try the import out if someone pops the right permissions over 
to me, or I'm happy to supervise enough forum API access for you to try it, 
Tom.

Andrew

On Thursday, October 29, 2020 at 9:23:45 AM UTC-6 carlton...@gmail.com 
wrote:

> I don’t have any controls here. I’m pretty sure Florian would be the most 
> likely candidate. (No doubt it’s Jacob.) 
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/44e041d6-9397-4a73-87a0-e4643c034db0n%40googlegroups.com.

Re: async_unsafe's get_event_loop usage leads to "Too many open files" in tests

[Andrew Godwin]

Yeah, I'm not quite sure what this is, but I just ran get_event_loop() a 
thousand times and it gave me the same thing every time and didn't even budge 
the number of file descriptors.

Can you replicate this behaviour in a brand new Django project? That's what I'd 
need to help debug it further.

Andrew

On Tue, Oct 13, 2020, at 10:37 AM, Adam Johnson wrote:
> I'd like to see what Andrew thinks on this.
> 
> I had a thought that the issue you're seeing may be related to a dependency 
> of your project creating/destroying event loops in the background. If you 
> could try and reproduce it that'd be neat.
> 
> On Mon, 12 Oct 2020 at 21:54, Patrick Arminio  
> wrote:
>> On Fri, 9 Oct 2020 at 10:40, Adam Johnson  wrote:
 That's why we need the RuntimeError check, no?
>>> 
>>> I don't see how RuntimeError can be raised. The docs I spoke about say 
>>> get_*running*_loop() cannot be used synchronously, not get_event_loop() .
>> We still check for RuntimeErrors though: 
>> https://github.com/django/django/blob/855fc06236630464055b4f9ea422c68a07c6d02a/django/utils/asyncio.py#L19
>> 
>> So I think using (if we could) `get_running_loop` would be safe, since we 
>> want to check if we are running under asyncio.
>> 
>> Maybe I can get back to this "issue" in future, when support for python 3.6 
>> is dropped. Doesn't look like it is something that's bothering many people 
>> anyway :)
>> For now we had this issue only during tests, so we disabled the check. We 
>> might get back to it when we start using async.
>> 
>> I'll report any useful findings :)
>>  
>>> 
>>> 
>>> On Thu, 8 Oct 2020 at 22:44, Patrick Arminio  
>>> wrote:
 
 
 On Thu, 8 Oct 2020 at 22:28, Adam Johnson  wrote:
>>  I was going to write this in django developers
> 
> This is django-developers?
 I meant django-users, sorry :D
   
>> which creates a new loop every time, which seems to be the cause of the 
>> too many files open.
> 
> Can you give a bit more explanation how it creates a new loop every time?
 Uhm, I'll try to investigate a bit more tomorrow, we have a CI failure 
 where ~152 fail due to the same issue,
 I wonder if I can set up a small test suite that can reproduce this. But I 
 guess every test runs the same
 
> Two successive calls to get_event_loop() in ipython give me the same 
> object:
> 
> In [4]: l = asyncio.get_event_loop()
> 
> In [5]: l2 = asyncio.get_event_loop()
> 
> In [6]: id(l)
> Out[6]: 4561753280
> 
> In [7]: id(l2)
> Out[7]: 4561753280
> 
> Additionally, the docs for get_running_event_loop() say:
>> This function can only be called from a coroutine or a callback.

 That's why we need the RuntimeError check, no?
  
> Plus it was added in Python 3.7, whilst Django 3.1 supports 3.6+. So it's 
> not clear we can use it.
 Oh, this might be a problem considering the implementation of 
 `get_running_event_loop` is written in C, 
 not sure it can be backported easily.
  
> 
> That said, it would be nice to avoid the overhead of creating an event 
> loop for sync-only use cases.
 Yup :) 
  
> 
> On Thu, 8 Oct 2020 at 21:28, patrick91  wrote:
>> Hi folks, I was going to write this in django developers, but looks like 
>> I might have found a small area for improvement for the async_unsafe 
>> decorator, but first let me give you a bit of background:
>> 
>> We're upgrading to Django 3.0 and we've been encountering this error on 
>> ci when running all the tests: "OSError: [Errno 24] Too many open files".
>> 
>> Here's the full stack trace:  https://dpaste.com/ABYJFP6CR
>> 
>> By inspecting it and the source code it seems to be related to the 
>> async_unsafe decorator: 
>> https://github.com/django/django/blob/855fc06236630464055b4f9ea422c68a07c6d02a/django/utils/asyncio.py#L19
>> 
>> This decorator calls `get_event_loop`: 
>> https://github.com/python/cpython/blob/3.7/Lib/asyncio/events.py#L632
>> 
>> which creates a new loop every time, which seems to be the cause of the 
>> too many files open. Wouldn't be best to use `get_running_loop` 
>> https://github.com/python/cpython/blob/3.7/Lib/asyncio/events.py#L681 ?
>> 
>> We are already checking for RuntimeError so it should be an easy fix, 
>> but I wonder if the usage of `get_event_loop` is intentional and I'm 
>> missing something here.
>> 
>> Happy to provide a patch for this :)
>> 
>> Patrick
>> 

>> --
>> You received this message because you are subscribed to the Google 
>> Groups "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send 
>> an email to django-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> 

Re: Async Caching

[Andrew Godwin]

So the debate between async_foo and foo_async has not really happened yet; I'm 
really split on the choice. As you say, async_foo allows you to get to the 
async versions quicker, but foo_async allows you to start your typing with the 
operation you're doing and then select the variant.

I think right now I would prefer get_async all things considered, since I don't 
think autocomplete is necessarily easier either way, but I'm open to 
suggestions.

Andrew

On Sun, Sep 27, 2020, at 6:03 PM, Andrew Wang wrote:
> Cool, thanks for the clarification. One thing I noticed while programming the 
> test cases (so WIP is here: 
> https://github.com/Andrew-Chen-Wang/django-async-redis) is that it's not 
> really fun having to type out get_async since autocomplete. If a dev knows 
> that they'll be using an await, then I think it's best if everything is 
> prefixed with async_* rather than suffixed. Thoughts?
> 
> As a side note, the only other change that would need to happen is in 
> django.core.cache close_caches AFAIK.
> 
> Andrew 
> On Sunday, September 27, 2020 at 3:30:56 AM UTC-4 Adam Johnson wrote:
>>> I'm slightly confused by what you mean (and what the dep means).
>> 
>> I understand the DEP as saying that we'll implement the async methods in the 
>> BaseCache class as backing onto the sync versions:
>> 
>> class BaseCache
>> 
>> async def get_async(...):
>> return await asgiref.sync_to_async(self.get)(*args, **kwargs)
>> 
>> This means that existing cache backends don't all need to be converted in 
>> one go.
>> 
>>> Also just saw https://forum.djangoproject.com/. Any preference for where 
>>> future talks should be held: Google or that forum?
>> 
>> Essentially both are equivalent. I think there remain more readers on the 
>> mailing list though, which can help if you're discussing bigger picture 
>> things.
>> 
>> 
>> On Sun, 27 Sep 2020 at 04:30, Andrew Wang  wrote:
>>> Hey Adam and Andrew,
>>> 
>>> I can definitely make the naming scheme something like get_async() rather 
>>> than just get().
>>> 
>>> > This section specifically says that the default implementations will back 
>>> > onto the sync methods by default, so built-in cache backends won't need 
>>> > to all be converted (at once?).
>>> 
>>> I'm slightly confused by what you mean (and what the dep means). Are you 
>>> saying in case someone is using a built-in cache backend like LocMem for 
>>> local development and then an async third-party cache backend for 
>>> production, then the code is like:
>>> 
>>> class BaseCache:
>>> def get(...):
>>> NotImplementedError
>>> async def get_async(...):
>>> 
>>> NotImplementedError
>>> 
>>> class LocMemCache:
>>> def get(...):
>>> ...
>>> async def get_async(...):
>>> return await asgiref.sync_to_async(self.get)(*args, **kwargs)
>>> 
>>> >  It could be informative to have a working backend using this interface, 
>>> > and if you encounter any edge cases whilst creating it.
>>> 
>>> You can view my progress (once midterms are over :P) over here. 
>>> <https://github.com/Andrew-Chen-Wang/django-async-redis> I'm planning on 
>>> making the structure very similar to django-redis to make any migration 
>>> process to an async Django easier. When I compared redis-py to aioredis, 
>>> there weren't that many differences besides file structure (and passing the 
>>> event loop around), so hopefully nothing weird pops up. But I will 
>>> definitely post in this thread or in a ticket (if I do make a PR for 
>>> BaseCache) if something odd does pop up.
>>> 
>>> > The main work to do here is to work how quite how possible it is to offer 
>>> > all of them and if everything can be made to work regardless of what mode 
>>> > (sync or async) it's in
>>> 
>>> In terms of the package or the other built-in backends or just everything 
>>> in general like the ORM? If in terms of the package, most if not all 
>>> methods would have to be awaited... I could also just be confusing all 
>>> these words, so sorry about uhh me I guess!
>>> 
>>> Also just saw https://forum.djangoproject.com/. Any preference for where 
>>> future talks should be held: Google or that forum?
>>> 
>>> Thanks for the suggestions! Have fun with the ORM!
>>> Andrew
>>> On Saturday, September 26, 2020 at 8:38:07

Re: Async Caching

[Andrew Godwin]

Agreed - there's no work on caching inside Django yet, since the ORM is my next 
focus, but I would definitely suggest writing a new pluggable third-party 
backend that somehow provides async versions of the methods.

The main work to do here is to work how quite how possible it is to offer all 
of them and if everything can be made to work regardless of what mode (sync or 
async) it's in; hopefully there's a lot less long-lived-connection issues than 
in the ORM, say.

Andrew

On Sat, Sep 26, 2020, at 3:56 PM, Adam Johnson wrote:
> Hi Andrew
> 
> I don't believe any work has started on async caching in core. However there 
> is a plan in DEP 9, the document that outlines how asynchronous caching will 
> work, using suffixed methods like get_async() etc. See 
> https://github.com/django/deps/blob/master/accepted/0009-async.rst#caching . 
> This section specifically says that the default implementations will back 
> onto the sync methods by default, so built-in cache backends won't need to 
> all be converted (at once?).
> 
> I think a good start would be making your aioredis-based backend in a third 
> party package, using the future naming scheme `*_async`. It could be 
> informative to have a working backend using this interface, and if you 
> encounter any edge cases whilst creating it.
> 
> Thanks,
> 
> Adam
> 
> On Sat, 26 Sep 2020 at 05:55, Andrew Wang  wrote:
>> Hey guys, I'd like to contribute to the effort to make Django more async 
>> capable. I've started to write an aioredis based cache backend based on 
>> django-redis, but I noticed the BaseCache in Django is still all synchronous 
>> basically.
>> 
>> I was wondering which backends I should make async capable and how would I 
>> go about it?
>> 
>> I was thinking instead of creating a new class, we could just add the async 
>> methods (e.g. add, get) to BaseClass. And for the FileBased, Dummy, and 
>> LocMem backends, the plan would be the same? Problem with those are Python's 
>> local file IO is synchronous...
>> 

>> --
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/68307bf9-e6f4-4dc3-9e0f-d6e660075a85o%40googlegroups.com
>>  
>> .
> 
> 
> -- 
> Adam
> 

> --
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/CAMyDDM00ourgfJs53v7NmXOGEV6_t8vV3%3D%2BP%2Bf61DHEhOcFvDw%40mail.gmail.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/2efe271c-2626-45a9-a6f0-bd808562b2a2%40www.fastmail.com.

Re: Redundant migration code

[Andrew Godwin]

I suspect the reason for this might be to undo circular references of 
ForeignKeys between apps - in this situation, you have to first have a 
migration that removes one FK, then in the other app remove the model, then in 
the first app remove the model.

I can't quite remember though - but that would be my suspicion as to why this 
would be structured this way in my original code. As you say, the optimiser has 
improved in the years since, but I don't think you can optimise away the 
circular reference problem?

Andrew

On Sat, Sep 19, 2020, at 3:20 PM, Silvio J. Gutierrez wrote:
> 3 tests fail when I comment out the code, however, I suspect it's because of 
> this:
> 
> test_autodetector.py#586
> 
> It's directly calling _detect_changes before piping it to the optimizer. So 
> rightly so, it expects more operations than post optimization.
> 
> I could be totally wrong though.
> 
> Basically, with code (and what the unit test asserts):
> 
> ['RemoveField', 'RemoveField', 'DeleteModel', 'DeleteModel']
> 
> Without code:
> 
> ['DeleteModel', 'DeleteModel']
> 
> With code but after optimizer:
> 
> ['DeleteModel', 'DeleteModel']
> 
> But the unit test is looking at pre-optimizer output. I think.
> 
> On Sat, Sep 19, 2020 at 2:23 PM Adam Johnson  wrote:
>> Did you try deleting this code and seeing if any test failed?
>> 
>> On Sat, 19 Sep 2020 at 17:54, Silvio  wrote:
>>> I've been digging around the 3.1 migration code just out of curiosity, and 
>>> I stumbled upon this in autodetector.py#772:
>>> 
>>> for name in sorted(related_fields):
>>> self.add_operation(
>>> app_label,
>>> operations.RemoveField(
>>> model_name=model_name,
>>> name=name,
>>> )
>>> )
>>> 
>>> It appears that all related fields for a model are removed before removing 
>>> a model. This is great, *except*, optimizer.py will always strip these 
>>> migrations out.
>>> 
>>> That is: the RemoveField operations for related fields are always adjacent 
>>> to the DeleteModel operation, so I cannot think of any case where 
>>> optimizer.py will *not* strip these out.
>>> 
>>> Am I missing something, or is this code that no longer contributes value? 
>>> Perhaps the optimizer has gotten better since?
>>> 
>>> Best,
>>> 
>>> Silvio
>>> 
>>> 

>>> --
>>> You received this message because you are subscribed to the Google Groups 
>>> "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to django-developers+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-developers/46e53585-64f3-4e88-87a7-2cab7722aefan%40googlegroups.com
>>>  
>>> .
>> 
>> 
>> -- 
>> Adam
>> 

>> --
>> You received this message because you are subscribed to a topic in the 
>> Google Groups "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this topic, visit 
>> https://groups.google.com/d/topic/django-developers/fwwSmD3zBpQ/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to 
>> django-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/CAMyDDM3Y9WD-8VR2hC93tu7BtgWrEr0j4nfOx9HSLbi0mm%3D50A%40mail.gmail.com
>>  
>> .
> 

> --
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/CANfEt_abmcbUp_QdUJSebRm9od5dKjtz7DkFgOXVBL5O8yisfg%40mail.gmail.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ff59c1a1-f6b7-49d2-b588-26edf2b8293f%40www.fastmail.com.

Re: Critical hints about Django migrations

[Andrew Godwin]

While I agree with some of the author's points, I think a critical piece of 
context is that Django migrations are designed for the 90% case - i.e., people 
who just want something to work on a small scale and don't need to worry about 
many aspects of the database yet.

Like all parts of Django, it's designed to be progressively ignorable if you 
need to, and even deliberately includes a way to run SQL migrations (as the 
author suggests) complete with state tracking and no need to write a separate 
script, and is the intended approach for larger teams/codebases like the ones I 
work on. Migrations isn't meant to only be "makemigrations" and the model-based 
approach; there's also an underlying SQL application and dependency ordering 
engine that can be used standalone.

Andrew

On Thu, Aug 6, 2020, at 1:27 PM, Paolo Melchiorre wrote:
> HI all,
> 
> I would suggest reading this interesting article by Daniele Varrazzo
> (the maintainer of psycopg2 and creator of psycopg3) on Django
> migrations.
> 
> It contains some criticisms but I also think some interesting hints
> for improving Django ORM :
> https://www.varrazzo.com/blog/2020/07/25/surviving-django/
> 
> See you,
> Paolo
> 
> -- 
> Paolo Melchiorre
> 
> https://www.paulox.net
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/CAKFO%2Bx6_1qffQt56u58fNuAv6P6_kTkV258t-3ShCQNAkRXvKQ%40mail.gmail.com.
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/b275d5cf-1405-43c5-8062-d15e5ff57507%40www.fastmail.com.

Re: The blacklist / master issue

[Andrew Godwin]

I've definitely in favour of fixing all of the problematic word usage - after 
all, we eliminated master/slave from the database documentation years ago, 
we've just been a bit negligent at fixing the others.

Agreed with Adam, though, about seeing what GitHub builds - they announced 
they're working on something, and if it allows seamless migration, that'd be 
great. That said, if they take more than a month or two, we should just change 
it and get it over with.

Andrew

On Tue, Jun 16, 2020, at 12:59 PM, Adam Johnson wrote:
> On the branch rename, right now I'd rather wait to see what GitHub builds 
> that could help with this. It might allow aliasing master->main so that 
> existing PR's don't need targeting, local clones don't need updating, etc. It 
> also seems Git will make a change and it might be worth waiting to see what 
> that looks like.
> 
> On the blacklist/whitelist rename, I've reopened the ticket and Tim Graham 
> already reopened the PR.
> 
> On Tue, 16 Jun 2020 at 19:46, Tom Carrick  wrote:
>> On moving away from the master branch, it would be a lot of effort (not just 
>> in the repo, but docs, Trac, etc.), but I think it's worth doing regardless. 
>> I think there is often some reluctance to do something time-consuming 
>> because it takes someone's time away from technical issues. I don't think 
>> this is necessarily true. Although we could always use more contributors, 
>> this is something that's fairly easy for a newcomer (talking specifically 
>> about the changes to docs). There's no requirement to know about ORM 
>> internals, async implementation, or anything else. Just some time and 
>> thoroughness. I'd be happy to put the time in, and I'm sure there are many 
>> other (potential) contributors willing to do so, and It doesn't stop the 
>> more technical people working on the issues they want to. I think all that 
>> is really required - if/when a decision is made, would be to review the PR, 
>> change the branch in Github, migrate Trac. Of course I don't know what else 
>> is affected, so maybe I'm being optimistic here.
>> 
>> This does have some precedent also, there was a move from trunk to master 
>> when moving from SVN to git. That was part of a bigger change to a new VCS 
>> system, admittedly, but I think this is also important. With that said, I do 
>> think we should wait and see what naming convention git / GitHub / GitLab 
>> etc. decide on. It seems like "main" has the most traction, which seems fine 
>> to me, and, as has been mentioned, is less of a misnomer anyway.
>> 
>> One argument I've seen against both of these that I don't think has been 
>> addressed in this thread is that this will set off a trend of renaming more 
>> things, mastery, masters degrees, and so on. In case it needs to be 
>> addressed, this strikes me as a slippery slope fallacy. Just because we do 
>> one thing, doesn't mean we must necessarily do another vaguely related 
>> thing. I don't see (or foresee) any interest in this, and I think as is 
>> always the case in these things, we go where the consensus is. For example, 
>> trolls aside, I don't see any criticism of e.g. Frontend Masters.
>> 
>> Just some further thoughts.
>> Tom
>> 
>> 
>> On Tue, 16 Jun 2020 at 15:44, Roger Gammans  
>> wrote:
>>> Funny you should say that but the git developers mailing list in is awash 
>>> with patches and shouting about just this at the moment.
>>> 
>>> It looks likely the patches will go in too - so that's not much of an 
>>> arguement against.
>>> 
>>> 
>>> On Tue, 2020-06-16 at 16:35 +0300, אורי wrote:
 I think *master* is the default branch name in any Git repository. It's 
 not about Django and even not GitHub. Do you want to change the default 
 branch name in Git?
 אורי
 u...@speedy.net
 
 
 On Mon, Jun 15, 2020 at 7:28 PM Tom Carrick  wrote:
> This ticket was closed wontfix 
>  as requiring a 
> discussion here.
> 
> David Smith mentioned this Tox issue 
>  stating it had been closed, 
> but to me it seems like it hasn't been closed (maybe there's something I 
> can't see) and apparently a PR would be accepted to add aliases at the 
> least (this is more recent than the comment on the Django ticket).
> 
> My impetus to bring this up mostly comes from reading this ZDNet article 
> 
>  - it seems like Google have already made moves in this direction and 
> GitHub is also planning to. Usually Django is somewhere near the front 
> for these types of changes.
> 
> I'm leaning towards renaming the master branch and wherever else we use 
> that terminology, but I'm less sure about black/whitelist, though right 
> now it seems more positive than negative. Most arguments against use some 

Re: What's wrong with the jenkins testrun failing on ASGI (I think?)

[Andrew Godwin]

Yup, I'm seeing if we can get asgiref fixed today, otherwise I'll revert the 
change that broke Django and issue 3.2.9.

Andrew

On Tue, Jun 16, 2020, at 2:48 AM, Florian Apolloner wrote:
> Ok, so rebasing PRs to current master will fix this (leaving this here as 
> note for others who run into this).
> 
> On Tuesday, June 16, 2020 at 10:39:13 AM UTC+2, Mariusz Felisiak wrote:
>> It's an issue with the asgiref==3.2.8, see 
>> https://github.com/django/asgiref/issues/170. We temporarily pinned 
>> asgiref==3.2.7 [1].
>> 
>> Best,
>> Mariusz
>> 
>> [1] 
>> https://github.com/django/django/commit/dcb4d79ef719d824431a8b3ff8ada879bbab21cc
> 

> --
>  You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
>  To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
>  To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/c775e1a0-9485-46da-b4d9-015e8ee019b9o%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/49b08050-05ac-418b-a5a0-e4d57ace87e2%40www.fastmail.com.

Re: [FEATURE] Allow squashmigrations to squash the whole project

[Andrew Godwin]

I definitely think this is worth pursuing - the reason I didn't do it back
in the day was that squashing the entire project involved solving some
rather nasty dependency graph issues if there were any cycles of apps
depending on models from other apps.

If we can overcome that, though, and squash down to just a couple of
migrations each app, it would be nice to have that be a shortcut.

Andrew

On Wed, Apr 22, 2020 at 10:35 AM Javier Buzzi 
wrote:

> This comes from a place of headaches: we're a large team, and our sprint
> are a month long, in that month we generate a lot of migrations, after a
> couple of sprints we see our tests take a real toll when running
> migrations, this is due to constantly adding columns/deleting columns,
> moving data around -- what normally happens with an ever changing app.
>
> The way quashing migrations currently works is by an app by app, I'm
> proposing that if no arguments are supplied (no app_label,
> start_migration_name, migration_name) to squash the entire project, who's
> apps belong to the project of course.
>
> I'm hopping that I can get all the modifications already done to this
> project: https://github.com/kingbuzzman/django-squash and simply
> integrate it directly into the django source. I don't want to take such an
> undertaking and just have the PR be rejected.
>
> Is there any interest for this?
>
> Thanks,
> Javier Buzzi
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/ad3e7507-01af-49a6-ab15-e1b6d7d32bb7%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uoC9_QJKpR9-49EABHKukDHgu8FvP3m_k6YVkijcmc2iQ%40mail.gmail.com.

Re: Remove automatic date-naming of migrations (00XX_auto_YYYMMDD)

[Andrew Godwin]

I am a little mixed on this change - the behaviour during the initial
development was indeed to concatenate names like this, albeit only for
adding fields or models; I thought it looked unwieldy, which is why I added
the "auto" name.

That said, the number is the only part that actually matters, and while the
date is sometimes useful for people to resolve merge conflicts, I don't
think it's better than more informative autogenerated names, so I'm happy
to go with the change.

(60 is a bit long though, maybe we can bump it down to something a bit
shorter?)

Andrew

On Wed, Apr 22, 2020 at 7:06 AM Adam Johnson  wrote:

> Hi djangonauts,
>
> In a blog post earlier this year I outlined my technique to prevent Django
> creating migration files with automatic date names. I had a lot of response
> with other techniques and ended up adding two more techniques to the post.
> It's at
> https://adamj.eu/tech/2020/02/24/how-to-disallow-auto-named-django-migrations/
> .
>
> The problem with such migration names:
>
> When you run Django’s manage.py makemigrations, it will try to generate a
>> name for the migration based upon its contents. For example, if you are
>> adding a single field, it will call the migration 0002_mymodel_myfield.py.
>> However when your migration contains more than one step, it instead uses a
>> simple ‘auto’ name with the current date + time, e.g.
>> 0002_auto_20200113_1837.py. You can provide the -n/--name argument to
>> makemigrations, but developers often forget this.
>>
>> Naming things is a known hard problem in programming. Having migrations
>> with these automatic names makes managing them harder: You can’t tell which
>> is which without opening them, and you could still mix them up if they have
>> similar names due to being generated on the same day.
>>
> Django *currently* sets the migration name to something other than the
> automatic date name in two cases:
>
>- If the migration contains a single operation, it uses a name based
>on that operation, e.g. 00023_remove_model_field, or 0024_add_model_field
>(but not for all operation types)
>- If the migration consists *only* of model creation operations, it
>combines their operation names names, which come as just the lower-cased
>model names e.g. 0025_book_author_sale
>
> I opened a PR to expand the operation naming for the single case to cover
> all built-in operations: https://github.com/django/django/pull/12131
>
> I'd like to propose using this new full coverage of operation naming to
> remove the "auto_MMDD" behaviour, and instead always combine
> operations' "suggested migration names" up until a limit of say 60
> characters. I made a commit for that here:
> https://github.com/adamchainz/django/commit/c2bc6893a05c2c8099e1fb68e688618446641ed6
>
> This would lead to migration names such as:
>
>- 0026_remove_book_title_add_book_description
>
> Whilst perhaps long, they're explict and imo easier to work with than the
> auto_MMDD behaviour.
>
> Mariusz wrote on the PR:
>
> Personally, I'm against removing auto named migrations. IMO chaining
>> operation names is even more confusing.
>>
> --
> Adam
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAMyDDM10ACO6wEMZXukN6_xy766Bxa8PQOc7teFEQhRmzWxqwA%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1urecp9nGUqwrsxYox%2BtWzv5FcBBWbANg7E9PGAtUG4m_A%40mail.gmail.com.

Re: New Merger nomination.

[Andrew Godwin]

I also vote in favour of Claude becoming a Merger!

Andrew

On Tuesday, April 21, 2020 at 4:28:41 AM UTC-6, Markus Holtermann wrote:
>
> I vote in favor of Claude becoming a MERGER. 
>
> Cheers, 
>
> Markus 
>
> On Thu, Apr 16, 2020, at 10:31 PM, charettes wrote: 
> > I cast my vote in favor of Claude's nomination as well. 
> > 
> > Le jeudi 16 avril 2020 16:16:31 UTC-4, Adam Johnson a écrit : 
> > > This has fallen by the wayside, let's try restarting. 
> > > 
> > > As Carlton points out, Claude hasn't been merging in code without 
> others reviewing it. But as I understand it is useful to keep translations 
> moving that he can merge in his or others' (accepted) PR's. It gets us to 
> the minimum of three mergers, and Claude has stated he's willing to do his 
> best in the role of merger. And I hope Claude can help inform us what we 
> can do to make i18n more smooth. 
> > > 
> > > I nominate Claude as a merger. 
> > > 
> > > Technical board, please post your votes. 
> > > 
> > > On Thu, 19 Mar 2020 at 07:59, Carlton Gibson  
> wrote: 
> > >> I don't know if this got blocked in the now-obsolete use of cognates 
> of "commit"? 
> > >> 
> > >> > The Merger role is not just a new name for "committer". If Claude 
> is 
> > >> > appointed a Merger, he will be forbidden to do what you are saying 
> -- 
> > >> > a Merger cannot push their own contributions directly into Django! 
> > >> 
> > >> Claude hasn't, and wouldn't, "commit" directly to Django here. He's 
> always opened a 
> > >> PR with the translations updates, which has then been reviewed, and 
> which then, yes, he has 
> > >> merged, and forward/backported as necessary — which IS the Merger 
> role. (IIUC) 
> > >> 
> > >> In the Merger role, Claude would also be able to approve and Merge 
> other PRs. 
> > >> 
> > >> Beyond this important translations case, we do need a third Merger, 
> and I cannot think of a better candidate. 
> > >> (The two most active contributors are Claude and Simon, and Simon 
> cannot serve as a Merger because of his 
> > >> role on the Technical Board.) 
> > >> 
> > >> I would ask the Technical Board to proceed with the nomination, as, 
> bar the loose language, I think it is in line with the DEP. 
> > >> 
> > >> Kind Regards, 
> > >> 
> > >> Carlton 
> > >> 
> > 
> > >>  -- 
> > >>  You received this message because you are subscribed to the Google 
> Groups "Django developers (Contributions to Django itself)" group. 
> > >>  To unsubscribe from this group and stop receiving emails from it, 
> send an email to django-d...@googlegroups.com. 
> > >>  To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/14af7b19-bcd3-4462-9e25-606f30ff6eda%40googlegroups.com
>  
> <
> https://groups.google.com/d/msgid/django-developers/14af7b19-bcd3-4462-9e25-606f30ff6eda%40googlegroups.com?utm_medium=email_source=footer>.
>  
>
> > > 
> > > 
> > > -- 
> > > Adam 
> > 
> >  -- 
> >  You received this message because you are subscribed to the Google 
> > Groups "Django developers (Contributions to Django itself)" group. 
> >  To unsubscribe from this group and stop receiving emails from it, send 
> > an email to django-d...@googlegroups.com . 
> >  To view this discussion on the web visit 
> > 
> https://groups.google.com/d/msgid/django-developers/c13e6529-0e75-4e04-8ea6-501982420504%40googlegroups.com
>  
> <
> https://groups.google.com/d/msgid/django-developers/c13e6529-0e75-4e04-8ea6-501982420504%40googlegroups.com?utm_medium=email_source=footer>.
>  
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/15a11b73-d6dd-4ea4-af42-0919e75a5147%40googlegroups.com.

Re: Psycopg3 Redesign

[Andrew Godwin]

I am particularly excited for native async support arriving - that's
something I'd love to have and ship support for from our side when it's
done. The rest looks quite sensible too.

Andrew

On Wed, Mar 11, 2020 at 3:39 AM Jure Erznožnik 
wrote:

> His proposed changes look awesome to me!
>
> IMHO, the biggest issue going async with current implementation was
> connection pooling for async code. I feared that we might need huge numbers
> of connections in certain scenarios, especially if the code required
> altering the connections themselves. The proposed changes look like they
> eliminate the issue altogether and I'm not referring to direct async
> support alone.
>
> LP,
> Jure
>
>
> On 11/03/2020 09:28, Asif Saif Uddin wrote:
>
> Hi,
>
> As this library is used by django, I thought it would be good to share
> with django internals deveopers?
>
> https://www.varrazzo.com/blog/2020/03/06/thinking-psycopg3/
>
> Hope it is useful.
>
> ./auvipy
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/faea6068-ec43-4da6-9780-1beaedaf0b16%40googlegroups.com
> 
> .
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/110c8328-c0ea-e32e-081a-ca42657193b7%40gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1urMR9G5vFUjOKSE3TtyA%3DoHcJtUY9r_fifSSOPXuwXLpw%40mail.gmail.com.

Re: Sounding out for GSoC 2020.

[Andrew Godwin]

Here's my take on each of these, leaving out async as that'd be more
dependent on where we were:

Secrets Manager
Many environments Django is deployed in use a separate secrets manager to
store and provide secrets per environment - either as environment
variables, files, or via a direct HTTP API. The project would be to design
and add an abstraction interface over secrets managers that allows users to
easily map to an external secret in a settings file. There's also room for
giving Django a better, built-in per-environments settings option too,
pulling from popular third-party patterns.

Django Service Hooks
Django has a lot tied to the HTTP lifecycle, and if you run it outside of
this, you miss out on some key things - like database connections being
closed properly, or logging, or some middleware. The project would be to
design and implement a separate way of calling Django from a service
framework - even if that framework was itself HTTP based (basically, don't
rely on Django having to run through WSGI/ASGI). Initial idea is to provide
a context manager that replaces the wrapping of a request, but the project
would have to include requirements-gathering and interviewing people who
use Django to power microservices to work out what they want.

Evented Datastores
Evented, or event-sourcing, database schemas are growing in popularity -
essentially, a way to implement an append-only pattern on top of a
relational database, where every single event that has happened is stored,
but you can still efficiently retrieve the current state of an item. Given
the way the pattern works, this would be possible to implement on top of
Django models - likely as a whole new Model subclass - and implement with a
matching QuerySet/Model API. It would require someone very skilled in
database design, and it's also arguable if it should be in Django itself,
or comprise of developing a third-party app alongside adding hooks/fixes to
the Django ORM where needed to make it work well.

GraphQL
Basically, work on an "official" graphql-to-ORM mapper for Django, probably
re-using a considerable amount of third-party work. This one could really
vary in its application, based on how much other "API-ish" work the project
includes (given that Django REST Framework is still separate, we should
follow that rough pattern).

Plus, some bonus ones that aren't Django code itself:

Django Case Studies
Our long-talked-about website which brings together case studies and white
papers about why people choose Django, aimed at companies working out what
they want to choose. This is likely only some coding and a lot of
writing/legwork, so may not be a good fit for GSOC, but should be on any
future plan we have.

Django Benchmarking
Django has had a benchmarking suite for a while, but we've generally got
more slack at tracking the effects each commit and release has. We should
institute a set of top-level benchmarks aimed at various areas (the current
set in djangobench are weak in, say, the HTTP-serving-speed area) and then
run them against pull requests and each release, so we can make sure Django
doesn't get slower while nobody is looking.

I'm sure there's a lot more, but these are top of mind at the moment!

Andrew


On Tue, Dec 10, 2019 at 11:52 AM Carlton Gibson 
wrote:

> Hey Andrew.
>
> Yes, Async, of course. It's not until *next July* so I'd still think we
> should list this. Worst case is there's exactly zero progress between now
> and decision day, in which case you could say that we could only accept a
> clearly capable candidate. (Jannis was once our GSoC person, so... :)
>
> The other ideas seem good: the secrets manager abstraction is a nice one
> (!)
>
> No rush, but when you have a moment, perhaps you could draft a couple of
> paragraphs for each? (Even if they don't go to GSoC directly, it would come
> in handy for the TB's direction guidance...)
>
> Kind Regards,
>
> Carlton
>
>
> On Tuesday, 10 December 2019 19:14:51 UTC+1, Andrew Godwin wrote:
>>
>> I agree that 2FA could be a good choice - some of the async support work
>> would also have been good, had I made more progress in the latter half of
>> this year.
>>
>> A couple of other ideas for big projects:
>>
>> * A secrets manager abstraction and built-in support for Vault, KMS, and
>> other common ones
>> * A proper story and hooks for running Django as a service (i.e. outside
>> of a traditional HTTP request/response cycle)
>>
>> And some ludicrous projects I'd still consider:
>>
>> * Evented/event-sourcing database work on top of the ORM
>> * A GraphQL-to-ORM mapper
>>
>> Andrew
>>
>>
>> On Tue, Dec 10, 2019 at 8:25 AM Carlton Gibson 
>> wrote:
>>
>>> Hi all.
>>>
>>> It's time to start thinking about Google Summer of Code (GSoC). I

Re: Sounding out for GSoC 2020.

[Andrew Godwin]

I agree that 2FA could be a good choice - some of the async support work
would also have been good, had I made more progress in the latter half of
this year.

A couple of other ideas for big projects:

* A secrets manager abstraction and built-in support for Vault, KMS, and
other common ones
* A proper story and hooks for running Django as a service (i.e. outside of
a traditional HTTP request/response cycle)

And some ludicrous projects I'd still consider:

* Evented/event-sourcing database work on top of the ORM
* A GraphQL-to-ORM mapper

Andrew


On Tue, Dec 10, 2019 at 8:25 AM Carlton Gibson 
wrote:

> Hi all.
>
> It's time to start thinking about Google Summer of Code (GSoC). If we're
> going to participate and projects we might propose.
>
> This year was interesting. Sage in particular did well putting together a
> cross-db JSONField, but he was probably under-mentored,
> since Mariusz has spent quite a bit of time reworking the PR, and still
> has a bit to go, before we can pull it in, hopefully for 3.1
>
> So, one consideration we need to think about seriously is our capacity for
> mentoring. (This isn't just about the candidate's ability — Sage was able
> to implement all suggestions — we just didn't have as much capacity as we
> might have liked to think about the requirement implementation — and there
> were four of us actively giving some time each... — Anyhow, to think
> about.)
>
> Then it's projects. There are three that I have on my list that would
> require a "competent candidate":
>
> 1. Work on the migrations. Markus mentioned a particular ticket here
> but...
> 2. Make the parallel test runner work on Windows. ("fork" vs "spawn")
>
> And 3, and this is the big one:
>
> 3. Add 2FA to Django.
>
> This has been raised a few times:
>
> *
> https://groups.google.com/d/topic/django-developers/T-kBSvry6z0/discussion
> *
> https://groups.google.com/d/topic/django-developers/d92P2V0YrbI/discussion
>
> * ... others...
>
> If I'm honest, in 2020, it's the one "battery" I feel a little bit
> embarrassed we haven't got a story for. Maybe it's not possible in a GSoC
> type scope but...
> — What would it look like? What can we leverage? Is it worth a go?
>
> I'm looking at James, Florian, Joe, ... — who else has been keen here?
> I'm also looking at the Technical Board, which I'm thinking has (will
> have) a new guiding role to come up with suggestions for the direction of
> Django.
>
>
> Other Projects: Are there other ideas? (Do you have one?) Are there easier
> things that we could take "weaker" candidates for? But with that is there a
> commitment for the mentoring help they'd need?
>
> Anyhow, we have until January, so I'm just starting the discussion here.
>
> Kind Regards,
>
> Carlton
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/af075d59-ece3-4044-9204-b690c746b9e0%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uqEoEMxv%3D6pZS7oGNWN6%2BGdrQJBNFp%3DdETwy9DTm-hfSg%40mail.gmail.com.

Re: Django 3.0 Release Notes - ASGI

[Andrew Godwin]

I agree - we need to communicate that ASGI support does *not *mean you can
start writing async def views. I think we should put a big disclaimer to
that effect next to it in the release notes and say it should be coming
next release.

Andrew

On Mon, Oct 14, 2019 at 5:45 PM Josh Smeaton  wrote:

> A co-worker just linked me to
> https://docs.djangoproject.com/en/dev/releases/3.0/#asgi-support and
> asked me (basically) if we can start doing all kinds of async work in one
> of our projects. Unfortunately, I didn't really know how to answer.
>
> Preface: I haven't followed the ASGI plan very closely (I read the DEP and
> have a vague understanding of the vision).
>
> It's my understanding that there's only very limited support for ASGI, and
> most features of Django won't work properly when running under ASGI. But
> that's not clear from reading the release notes or the deploy on ASGI
> section of the docs. Should we have a section in the docs that show what is
> and is not supported, along with examples?
>
> It'd be good to have a spot in the docs to point to that shows what is in
> and out of scope for each milestone.
>
> Thoughts?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/9b11ecd8-997f-4edc-a627-5523da611a55%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1ur9uqHjSn1-Ynj1CVT%2B5AF4X2J3-nRCZ0dF4ZUWeJG1MQ%40mail.gmail.com.

Announcing the Django Forum

[Andrew Godwin]

Hi all,

If you recall, last month I brought up the idea of a Django Forum (
https://groups.google.com/forum/#!topic/django-developers/HGAHQqKp7rs) -
we're now ready to launch this forum in a test phase!

You can find it at https://forum.djangoproject.com

Like our Trac, it's tied into GitHub authentication, so there's no need to
register another account or anything.

While this is still technically a test period, we'll likely make it fully
official within the next month or two.

There are sub-forums/categories for both Django Internals (the things this
list talks about) and also Using Django (django-users content) - and if you
feel like helping people out, you can watch an entire category inside of
Discourse (click the circle on the top-right of a category page) and it'll
email you about everything that gets posted. You can even reply to those
emails, or sign up for a daily digest, if you want to keep the mailing list
feel.

We're not deprecating the mailing lists or anything - the forum is meant to
supplement them, and especially to provide a more accessible place for new
Django users and developers to ask questions without the perceived
overhead/global broadcast of a mailing list. It'll also provide us with a
nice way to have a lot of async discussions (
https://forum.djangoproject.com/c/internals/async) without bombarding the
rest of you!

Feedback about the forum is welcome either in this thread, or in the
feedback thread over in the forum (
https://forum.djangoproject.com/t/forum-feedback/17/).

See you there!

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1upqT%2B7wiNgdYHHueSqaB_Noc6LWUB-JnGKQM6f1mPMqVQ%40mail.gmail.com.

Re: Proposing development discussion forums

[Andrew Godwin]

Another point has been raised to me by someone off-list - adding a forum
would significantly increase the surface area that the Code of Conduct team
have to cover (and potentially become one of the biggest time sinks
required), so we need to consider them in any decision.

Andrew

On Mon, Aug 12, 2019 at 10:10 PM Andrew Godwin  wrote:

> I agree James - forums tend to age slightly worse than mailing lists for
> archival content, but I'm hoping the improved experience in the moment
> makes up for it.
>
> Plus, our current mailing list archive depends on a service from Google,
> and I trust those less these days (though I hope Google Groups is probably
> going to be around for a long time).
>
> Andrew
>
> On Monday, August 12, 2019 at 4:04:23 AM UTC-5, James Bennett wrote:
>>
>> I'm not necessarily opposed to this, but I am a bit skeptical of the
>> long-term archival utility of forums, in large part due to my experience as
>> a moderator of some decent-sized ones. I think making them useful for that
>> purpose is going to require about the same level of manual curation as,
>> say, the wiki on code.djangoproject.com does now.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/f548bdc5-89f9-48e4-8609-ce8e9f718f32%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/f548bdc5-89f9-48e4-8609-ce8e9f718f32%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uoJ%2BuUe%3DZQnNVosxW1vuq7gBuUduqHVjq82OHzwi9Z4JA%40mail.gmail.com.

Re: Proposing development discussion forums

[Andrew Godwin]

I agree James - forums tend to age slightly worse than mailing lists for 
archival content, but I'm hoping the improved experience in the moment 
makes up for it.

Plus, our current mailing list archive depends on a service from Google, 
and I trust those less these days (though I hope Google Groups is probably 
going to be around for a long time).

Andrew

On Monday, August 12, 2019 at 4:04:23 AM UTC-5, James Bennett wrote:
>
> I'm not necessarily opposed to this, but I am a bit skeptical of the 
> long-term archival utility of forums, in large part due to my experience as 
> a moderator of some decent-sized ones. I think making them useful for that 
> purpose is going to require about the same level of manual curation as, 
> say, the wiki on code.djangoproject.com does now.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f548bdc5-89f9-48e4-8609-ce8e9f718f32%40googlegroups.com.

Re: Proposing development discussion forums

[Andrew Godwin]

On Sat, Aug 10, 2019 at 3:30 AM Carlton Gibson 
wrote:

> What does it need from Ops? (Is there a `docker run-my-service`? Could we
> leverage djangoproject.com (and GitHub) logins, or are they always going
> to be separate?)
>

Markus has done more investigation on how to run it, though my impression
is that it's relatively easy since he had it up before I'd got off the
plane back to the US. Not sure about the djangoproject.com login situation
- we need to investigate how flexible the plugin system is, and it might
need OpenID on the Django end.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uoY%3DaVuCJB0bHB7uBnT%2BFEVTktb%2BsaEz5J9bDhzWGTZ0A%40mail.gmail.com.

Re: Proposing development discussion forums

[Andrew Godwin]

‪On Fri, Aug 9, 2019 at 10:16 PM ‫אורי‬‎  wrote:‬

> Every Google Group also has an online forum:
> https://groups.google.com/forum/#!forum/django-developers
>

Indeed, I am aware of this, I usually use it when I link threads on Twitter.

However, it is not really a forum in the sense I am describing - one that
has categories, editable posts, and the ability to selectively get email
for certain categories or threads rather than all-or-nothing. The Groups
forum interface is more just an online mailing list interface, with all the
problems of the underlying list model.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1urt8pfhdyoGe%3D2u31YSg8sf%2BanDNEO6ogX3y4xnfa4WPQ%40mail.gmail.com.

Proposing development discussion forums

[Andrew Godwin]

Hi everyone,

This might be slightly controversial, but I would like to propose that we
have a forum for discussing Django development (and potentially user
support), alongside the mailing list and maybe, eventually replacing it.

My full reasoning is below, but in short, it would be more accessible for
new users, have a better UI, give us the ability to moderate away
problematic posts, be better for privacy, and still allow email-based
interaction.

At DjangoCon AU, the opening keynote was an invited speaker from the Rust
community (E. Dunham, https://twitter.com/QEDunham). I invite you to watch
the full talk if you are at all interested in how another language handles
their community (https://www.youtube.com/watch?v=TW7PxyrCBR0), but the
takeaway for me was their use of forums rather than mailing lists.

The Django mailing lists were an excellent choice when Django began, but I
feel they have aged out of the modern Web somewhat. The user interface for
accessing them is particularly poor, which makes it particularly bad for
new contributors.

In addition, when looking at how to organise the effort to help bring async
into Django, something more dynamic, and more segmented, than mailing lists
would be incredibly useful. I don't want to drown out the list in specific
discussions of how to port certain features, but we need to have those
discussions somewhere permanent and asynchronous (so not IRC).

The mutability of a forum is also not to be overlooked - as well as
allowing things like pinned posts and post edits for small issues (or a
living header on a long discussion topic), it also allows for permanent
removal of things that break the Code of Conduct. On the people front, it
also allows people to post without their email being public, allows for
name changes, and provides for someone's right to be forgotten via
anonymisation of prior content.

Now, I'm not suggesting we kill the mailing list and switch over or
anything like that; instead, I suggest we run an instance of Discourse as a
test, and use it as the primary discussion area for async work, as well as
anything else that people want to discuss - with the expectation that
anything important still goes out to this mailing list.

Why Discourse? Apart from being a mature, open source forum project, it's
also very fully featured, and even supports subscribing and interacting
with the forum over email, so it can still fit into an email-based
workflow. There are also plenty of small niceties, like the option to have
it hosted for us via a paid service, or the ability to use GitHub for login
rather than requiring a separate username and password. It also helps that
Rust seems quite happy with it.

I'm mostly asking for the "temperature of the room" on this one - if we get
some small objections, I think a trial period is still worthwhile. If there
are major objections, then I'd like to ask people what their alternative
suggestions are for solving this sort of communication.

Do I think this would replace the mailing list? Not in the short term, but
maybe if it takes off and we all like it better. I personally would
interact with django-developers a whole lot more if I could just subscribe
to certain topics (rather than trying to emulate that with an email filter
as I do now!), and honestly the same thing for django-users. That said, I
also recognise that diluting the support/discussion pool is not exactly an
attractive idea, which is why I'm asking for input!

Thanks,
Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1urJmpwTYqXSF_L57hDLL5-9T4Bkvj9Sb9p8na_-_YWEgA%40mail.gmail.com.

Re: Make Development More Accessible

[Andrew Godwin]

We actually discussed this a little at the PyCon AU sprints and the
consensus was that GitHub issues would be great *if only they were a bit
more featureful*.

The problems I feel are specifically an issue:

- Ticket states; this is not easily replicated with labels, while
components etc. are mapped slightly more easily

- Assignment of tickets to people outside the organisation - while bots
allow triage, I believe taking ticket ownership requires that you are
either a member of the organization or have commented on the ticket, which
makes the process there a little longer.

- Ticket history, as mentioned (we'd need to preserve ticket numbers and
redirect links from the current domain).

Exporting from GitHub seems very possible - exporting pull requests and
issues can both be done via the API, though this does, like the bot, need
extra work.

This all seems eminently doable, but the question I'd really want everyone
to answer is if it's worth the porting effort. I suspect the answer is yes,
but this does need a process DEP and some discussion, and maybe also
looking at what cpython are doing and comparing and contrasting.

Andrew

On Wed, Aug 7, 2019 at 4:12 AM Carlton Gibson 
wrote:

> The more I use Trac, the more I appreciate its power. I'm normally all for
> Progress™ but I'm not sure GitHub's UI is up to it.
> (Being able to find the old discussion is super handy: it's not that often
> that an idea has not come up before at this stage.)
>
> *I'd be interested to see what a prototype export looks like in a test
> GitHub repo. Maybe it's possible... (Note this is in bold.)*
>
> Maybe more people would participate, but I'm not sure... Do we just
> suspect that? I worry we go to a load of effort for no real gain.
>
> Current input is quite good I'd say. Claude and Simon are regulars.
> There's a good number more who make frequent appearances.
> I think there's more people commenting than we suspect. (Anyone trying the
> export would be able to do numbers I'd guess...)
>
> If the new Triaging role on GH would allow "Request a review..." I think
> it would be super handy. (But currently that's restricted to the more
> powerful roles.) (I'm nagging GH about as best I can but if anyone knows
> anyone...)
>
> Happy to comment more if people want, but those are the highlights.
>
> C.
>
> On Wednesday, 7 August 2019 12:56:18 UTC+2, Josh Smeaton wrote:
>>
>> Mariatta has put together a some PEPs for migrating CPython issues over
>> to GitHub.
>>
>> https://www.python.org/dev/peps/pep-0581/ proposing the migration.
>> https://www.python.org/dev/peps/pep-0588/ migration plan.
>>
>> Django and Cpython are not the same, so there'll be substantial
>> differences. But it's worth familiarising oneself with prior art.
>>
>> For what it's worth I'd strongly support such a move just for the
>> increase in engagement.
>>
>> Carlton, Mariusz, how would you gauge the level of triage activity in
>> Trac from non-core members? High/Medium/Low?
>>
>> https://github.blog/changelog/2019-05-23-triage-and-maintain-roles-beta/ 
>> describes
>> the new triage and maintain roles, but they're still to be granted to
>> trusted individuals (which would be an excellent gateway into full core
>> membership if that is the direction Django is going to continue in).
>>
>> On Wednesday, 7 August 2019 17:46:18 UTC+10, John Gooding wrote:
>>>
>>> Hi Aymeric,
>>>
>>> You bring up a lot of good points. There will undoubtedly be challenges
>>> and huge amount of work in moving to a new system, or implementing any big
>>> sweeping changes, however, I truly honestly believe that it would be worth
>>> it in the long run, and the payoff would far outweigh the cost.
>>>
>>> As far as Microsoft owning github, etc I think it is almost moot. Any
>>> process will have some amount of vendor lock in, whether github, atlassian
>>> (jira & bitbucket parent company), or even gitlab. I think what is
>>> important is to pick one system as a community that we are happy with. Any
>>> one of those three could do what is ultimately needed, which is a
>>> centralized and consistent development platform.
>>>
>>> On Wednesday, August 7, 2019 at 12:33:59 AM UTC-7, Aymeric Augustin
>>> wrote:

 Hello John,

 This was discussed before, when we moved from self-hosted svn to
 GitHub-hosted git, but I'm not sure there are public archives of all
 discussions.

 As far as I remember, the main points to tackle are:

 1. Does GitHub allow "anonymous triage" i.e. labelling, closing, and
 reopening issues by non-committers? I think there was a recent announcement
 in this area. I didn't check the details. Previously, bot-powered
 workarounds were suggested, but they wouldn't provide a good user
 experience. You want discoverable buttons, not a cheat sheet of magic
 comments.

 2. Does the GitHub UI scale to thousands of issues? In theory, any
 classification system can be reproduced with namespaced labels e.g.

Re: Django Websocket Implementation Request

[Andrew Godwin]

Unfortunately, from my perspective, websocket support is tricky enough that
it's at least not on the short-term plan to bake into Django; it's used by
only a small fraction of our users, and the Channels project is an official
Django project, so it's as close as we can get so far. Future Django
versions with async support will shrink Channels down significantly, and
remove the need for it to be hosted differently, but the plan is still to
have it as a separate package - we may write about it in the official docs,
though.

socket.io is even harder as it's fundamentally hard to scale - it needs
stateful storage to aid in transferring connections, and this means it's
really hard to have a single method that will work for everyone. Having
"one right way to do it" is sort of a necessary prerequisite to having
something in Django, and I just don't feel like it's there.

Andrew

On Fri, Aug 2, 2019 at 6:00 AM Confi Yobo  wrote:

> And socket.io kind of implementation is one thing developers from js like
> us are missing in django.
>
> On Thu, Aug 1, 2019 at 8:58 PM Confi Yobo  wrote:
>
>> Please consider it, cause the procedures of using third party websocket
>> library (mostly for production) is cumbersome
>>
>> On Thu, Aug 1, 2019 at 2:22 PM Carlton Gibson 
>> wrote:
>>
>>> Hmmm. I guess Channel is doing more than just websockets, but I've
>>> always thought it had good docs... Anyhow...
>>>
>>> We'll see how the Async work goes, but a lot of channel will end up
>>> being absorbed into Django itself. What, at this point, I guess that will
>>> leave is "just" the websocket bit. (Maybe the Channel Layers bit... too but
>>> maybe that would/could be separate.) Hopefully this'll make it more focused
>>> and easier to follow
>>>
>>> But... I think websockets haven't turned out to be the One True Hammer
>>> people thought. (They're not that popular actually, it seems...) So I'm not
>>> sure we'd include support in core (vs keeping it in the third-party
>>> package.)
>>>
>>> I guess something to revisit after 3.0.
>>>
>>> Kind Regards,
>>>
>>> Carlton
>>>
>>>
>>> On Thursday, 1 August 2019 15:09:21 UTC+2, Confi Yobo wrote:

 This might be a little redundant as you would say but i see importance
 in it.

 Django should come chipped in with websocket implementation rather
 developers using third party libraries like django-channels which are
 difficult and most at times have poor documentation. I have seen many cases
 where developers want to build an application that requires websocket they
 use another web framework.

 So in my opinion and that of few other developers i'v  seen i think
 websocket feature should be added to django out of the box(something like
 socket.io would make a lot of sense).

>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-developers+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/django-developers/c561814d-941c-446f-8707-c2ef1ce8170e%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAD9uRzdwK6Luj9cKZN70FgRRRO%2BtrybZ16XwB-FjWiPXZu9Eqw%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1upE%2BikHmiserS%3DKKQsT_xu5wZO14jUaDUCqVXZ_43WZOQ%40mail.gmail.com.

Re: Translation templatetag aliases

[Andrew Godwin]

I agree too. Let's change it.

Andrew

On Sat, Jul 27, 2019 at 4:03 AM Markus Holtermann 
wrote:

> Easy: +1 from me as well for reasons state before.
>
> /Markus
>
> On Sat, Jul 27, 2019, at 6:15 PM, Adam Johnson wrote:
> > +1 from me too for the reasons that Aymeric states.
> >
> > Another small pro: "translate" is a few more characters to type, but it
> > should make it easier to understand the purpose of the tags to
> > newcomers. "trans" is a prefix used for many words - Wiktionary lists
> > 609:
> >
> https://en.wiktionary.org/wiki/Category:English_words_prefixed_with_trans-
> . I guess quite a few Django apps out there have *some* domain term on this
> list, so using the full word "translate" can help differentiate the tags
> from those other concepts.
> >
> > On Sat, 27 Jul 2019 at 09:51, Aymeric Augustin
> >  wrote:
> > > Hello,
> > >
> > > I'm in favor of adding support for {% translate %} and {%
> blocktranslate %} and switching the Django documentation to use these for
> two reasons:
> > >
> > > - As stated by Mike, the mental associations that "block trans"
> creates for those who identify as trans are just bad — I can't believe it
> took us 12 years to notice :-/
> > > - Even if we leave that aside, I'm not sure saving 4 characters (8
> with the closing tag) is really worth the uncommon abbreviation of
> "translate".
> > >
> > > Since this change brings mostly a social benefit rather than a
> technical benefit, we could keep the {% trans %} and {% blocktrans %}
> aliases forever. Also, this could minimize arguments between those who
> recognize the benefits of such changes and those who don't, as we've seen
> when we changed master / slave to primary / replica.
> > >
> > > In my opinion, such debates mostly reflect the political rift that
> appeared in many Western countries in the recent years. It's completely
> predictable that they spill into our community. Since they aren't our main
> focus, we're trying to avoid spending too much time there. But we can't
> escape the fact that we're making Django first for people writing software,
> then for computers running that software.
> > >
> > > As a community, we chose to state in our Code of Conduct that "we
> strive to be a community that welcomes and supports people of all
> backgrounds and identities [including] sexual orientation, gender identity
> and expression". I believe that the change proposed here is in line with
> this statement. I know that some community members will feel that we're
> doing too much here — and that others feel that we aren't doing enough in
> general — which is why I'm referencing something we already agreed upon.
> > >
> > > Best regards,
> > >
> > > --
> > > Aymeric.
> > >
> > >
> > >
> > >> On 26 Jul 2019, at 13:17, 'Mike Hansen' via Django developers
> (Contributions to Django itself) 
> wrote:
> > >>
> > >> Hello all,
> > >>
> > >> Recently I had a member of my team bring up that it was uncomfortable
> > >> for them to work in parts of our codebase where they regularly had to
> > >> see "blocktrans" in the template files. To make our work environment
> > >> more inclusive, I wrote a Django package <
> https://pypi.org/project/django-translation-aliases/> which adds
> "translate" and
> > >> "blocktranslate" templatetag aliases so that we could update our own
> > >> internal templates to use these.
> > >>
> > >> We felt that this change would be in line with the Django community
> > >> so I made a ticket and pull request to Django at
> > >> https://code.djangoproject.com/ticket/30585 . The ticket was closed
> as
> > >> "wontfix", and it was mentioned that I should bring it to
> django-developers
> > >> if I wanted to make further progress on the ticket.
> > >>
> > >> Thanks,
> > >> --Mike
> > >>
> > >>
> > >>  --
> > >>  You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> > >>  To unsubscribe from this group and stop receiving emails from it,
> send an email to django-developers+unsubscr...@googlegroups.com.
> > >>  To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/a43ec0ab-f73a-4c11-8fc1-b05d081b24c3%40googlegroups.com
> <
> https://groups.google.com/d/msgid/django-developers/a43ec0ab-f73a-4c11-8fc1-b05d081b24c3%40googlegroups.com?utm_medium=email_source=footer
> >.
> > >
> >
> > >  --
> > >  You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> > >  To unsubscribe from this group and stop receiving emails from it,
> send an email to django-developers+unsubscr...@googlegroups.com.
> > >  To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/EF4EE680-BBA1-40EB-979E-2671349186D6%40polytechnique.org
> <
> https://groups.google.com/d/msgid/django-developers/EF4EE680-BBA1-40EB-979E-2671349186D6%40polytechnique.org?utm_medium=email_source=footer
> >.
> >
> >
> > --

Re: Django Async DEP

[Andrew Godwin]

OK, here is some of the feedback from the Technical Board, aggregated
together:

* There were questions around contextvars and if they might supplant the
need for a threading.local override - I clarified why this doesn't work in
the DEP.

* Several board members queried around how we might distinguish async
versions of functions/method from sync ones, and if we could get core
Python to implement a better, language-level way of having both under the
same name. For now, the conclusion is to pursue just having them as
separate names, likely under a sub-object (so, for example, cache.get and
cache.async.get), and ensure the documentation is very clear when we must
present both

* Templating was queried by multiple people, with one board member
referencing the streaming template work that happened in the past (
https://github.com/django/django/pull/11157), and another wondering if we
should just say that Django templates are sync forever and potentially look
at Jinja2 for async support if users desire. My personal takeaway from this
is that we should address async templates on its own, with a team who can
do some experimentation and come back with a recommendation.

* There was a concern over the fact that asgiref is now a core dependency
of Django, joining tzdata, and that there had already been some negative
user reactions about this in the wild. As part of helping out this problem,
I removed asgiref's own dependency on async_timeout, so it's now just a
single flat package.

* There was concern about users being able to opt out of async mode should
it cause a performance hit (like we allow with localisation), and another
member mentioned the djangobench benchmark project. I stated that if adding
async to the core flow impacted synchronous use by 10% we would find a way
to make it strictly opt-in, as well as my personal belief that this may
actually make things faster even for synchronous code and projects. I also
said that I'd like to revive regular benchmarking and add an async one in
to both help stop potential slowdown caused by adding async, as well as
illustrating the advantage of async code for certain workloads.

* A board member mentioned that the behaviour of transactions is already
tricky, and adding this in might make it worse. I agree with them, but I've
already started looking at ways to keep transactions and threads tied
strictly together and put some explicit safeguards to stop them leaking
across threads. That said, transactions and the ORM are my own personal
biggest worry about this first phase of the project.

* It was pointed out that while Django does not ship a threadlocal for
request, this is a very common pattern and we need to make sure it works. I
said that asgiref.local.Local is intended to be a drop-in,
backwards-compatible replacement for threadlocal for this very reason; most
projects will merely need to change a single import to get the correct
behaviour.

* There was some questioning about how the debug server should run and if
it should enable the asyncio debug mode. I think it should, but I'm not
sure at what stage we will make an async runserver the default, considering
we will likely have to bring in Daphne or Uvicorn as a dependency to do so.

* The final query I'm going to pull out here was non-technical - that the
Django project has lost many contributors over the years, is essentially in
a maintenance mode, and that we likely do not have the people to staff a
project like this. I agree with the observation that things have
substantially slowed down, but I personally believe that a project like
async is exactly what Django needs to get going again. There's now a large
amount of fertile ground to change and update things that isn't just fixing
five year old bugs.

Hopefully this gives you some idea of the conversation we had. In my years
on the Board, this is by far the most detailed a vote has ever gotten, and
I can only apologise to the incoming board for springing this on them right
after an election!

If anyone on this list would like to continue to talk about the above, or
if a Board member wants to bring their conversation out here, you are all
more than welcome.

Andrew

On Sun, Jul 21, 2019 at 1:02 PM Andrew Godwin  wrote:

> I'll ask permission and then summarise the points raised back out here!
>
> Andrew
>
> On Sun, Jul 21, 2019 at 1:01 PM Jacob Kaplan-Moss 
> wrote:
>
>> Congratulations, and great news!
>>
>> I hope the TB will consider sharing details and/or a summary of the "long
>> and involved vote"; I'll bet there's a bunch the broader community could
>> learn from the specifics.
>>
>> Jacob
>>
>> On Sun, Jul 21, 2019 at 3:54 PM Andrew Godwin 
>> wrote:
>>
>>> Hi everyone,
>>>
>>> After a long and involved vote, I can announce that the Technical Board
>>> has voted in favour of DEP 0009 (Async Django), and so the 

Re: Django Async DEP

[Andrew Godwin]

On Sun, Jul 21, 2019 at 1:11 PM Ehigie Aito  wrote:

> Django 3.0?
>

Django follows time-based releases; what's in Django 3.0 will depend on
when we can get it landed. At the moment I am optimistic something will
make it in, but I make no promises!

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1ur%2BgWcjavb6%2B-_qH0bE-P2qVOuWULVq%3DuNRq5j-TWkpTA%40mail.gmail.com.

Re: Django Async DEP

[Andrew Godwin]

I'll ask permission and then summarise the points raised back out here!

Andrew

On Sun, Jul 21, 2019 at 1:01 PM Jacob Kaplan-Moss 
wrote:

> Congratulations, and great news!
>
> I hope the TB will consider sharing details and/or a summary of the "long
> and involved vote"; I'll bet there's a bunch the broader community could
> learn from the specifics.
>
> Jacob
>
> On Sun, Jul 21, 2019 at 3:54 PM Andrew Godwin  wrote:
>
>> Hi everyone,
>>
>> After a long and involved vote, I can announce that the Technical Board
>> has voted in favour of DEP 0009 (Async Django), and so the DEP has been
>> moved to the "accepted" state.
>>
>> As some may have seen, I've started work on adding async support to views
>> (
>> https://github.com/django/django/compare/master...andrewgodwin:async_views)
>> - this is, as the DEP states, the last "blocking" change before we can open
>> up lots of parallel work on making other parts of Django async, and so now
>> the DEP is approved the next step is to work out funding and organisation
>> for future async work.
>>
>> If you are interested in helping with fundraising, then please get in
>> touch with me directly; I have some ideas about how to structure it, but I
>> could do with some people to help out. Otherwise, stay tuned for more
>> information about how to get involved contributing and what to work on!
>>
>> Andrew
>>
>> On Sat, Jun 8, 2019 at 9:14 AM Andrew Godwin  wrote:
>>
>>>
>>>
>>> On Sat, Jun 8, 2019 at 3:14 AM Pascal Chambon 
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> There is something a little scary for me, in changing all the core of
>>>> Django to async, when this really helps only, imho, a tiny fraction of
>>>> users : websocket/long polling services, and reddit-like sites with
>>>> thousands+ hits per second. For most webpages and webservices, async
>>>> artillery sounds quite overkill.
>>>>
>>>> Are cpython threads inefficient ? As far as I know they are only kernel
>>>> threads constrained by the Gil, so they shouldnt wake up when they are
>>>> blocked on io syscalls/mutexes (or do they?), and context switches remain
>>>> acceptable compared to the slowness of python itself.
>>>>
>>>
>>> It's fine when you only at 5/10 threads - which, notably, is what most
>>> WSGI servers run at. When you get to the hundreds, though, you start losing
>>> a large proportion of your execution time (tens of percent, in some cases).
>>>
>>>
>>>>
>>>> We used to provide provisioning and automatic authentication for 20
>>>> million users, with partner webservices tar-pitting us for sometimes 1mn.
>>>> The nightmare scenario. But with 2 machines, 1 process by core, and 800
>>>> threads by process, it did the job, enough for us to answer millions of
>>>> hits a day. Without even relying on other no-recoding optimizations like
>>>> pypy or gevent.
>>>>
>>>> Async would certainly have been a relevant techno if we had known in
>>>> advance that our partners would be so slow, but avoiding the extra
>>>> complexity burden of this style (where a single buggy dependency can block
>>>> all requests in a process, where all modules have to be recoded for it) was
>>>> also a huge benefit. And the limited thread pool also protected our DB from
>>>> unbearable loads.
>>>>
>>>
>>> Please remember that even after this change, Django will still expect
>>> you to write synchronously by default, and not impose any of that extra
>>> complexity on you. We will only swap out the "native" implementation of
>>> things if the performance matches (within ~10%) or exceeds that of the
>>> synchronous version when there's a couple of threads going; it's expected
>>> this will largely be the case due to the direct benefits of idling less.
>>>
>>> But - the plan is not to make it more complex by default (you only have
>>> to interact with the async if you want to) or slower.
>>>
>>>
>>>>
>>>> It's very nice if a proper async ecosystem emerges in python, but I
>>>> fear lots of people are currently jumping into it without a need for such
>>>> performance, and at the expense of lots of much more important matters like
>>>> robust ess, correctness, compatibility... like it happened for docker and
>>>> mic

Re: Django Async DEP

[Andrew Godwin]

Hi everyone,

After a long and involved vote, I can announce that the Technical Board has
voted in favour of DEP 0009 (Async Django), and so the DEP has been moved
to the "accepted" state.

As some may have seen, I've started work on adding async support to views (
https://github.com/django/django/compare/master...andrewgodwin:async_views)
- this is, as the DEP states, the last "blocking" change before we can open
up lots of parallel work on making other parts of Django async, and so now
the DEP is approved the next step is to work out funding and organisation
for future async work.

If you are interested in helping with fundraising, then please get in touch
with me directly; I have some ideas about how to structure it, but I could
do with some people to help out. Otherwise, stay tuned for more information
about how to get involved contributing and what to work on!

Andrew

On Sat, Jun 8, 2019 at 9:14 AM Andrew Godwin  wrote:

>
>
> On Sat, Jun 8, 2019 at 3:14 AM Pascal Chambon 
> wrote:
>
>> Hello,
>>
>> There is something a little scary for me, in changing all the core of
>> Django to async, when this really helps only, imho, a tiny fraction of
>> users : websocket/long polling services, and reddit-like sites with
>> thousands+ hits per second. For most webpages and webservices, async
>> artillery sounds quite overkill.
>>
>> Are cpython threads inefficient ? As far as I know they are only kernel
>> threads constrained by the Gil, so they shouldnt wake up when they are
>> blocked on io syscalls/mutexes (or do they?), and context switches remain
>> acceptable compared to the slowness of python itself.
>>
>
> It's fine when you only at 5/10 threads - which, notably, is what most
> WSGI servers run at. When you get to the hundreds, though, you start losing
> a large proportion of your execution time (tens of percent, in some cases).
>
>
>>
>> We used to provide provisioning and automatic authentication for 20
>> million users, with partner webservices tar-pitting us for sometimes 1mn.
>> The nightmare scenario. But with 2 machines, 1 process by core, and 800
>> threads by process, it did the job, enough for us to answer millions of
>> hits a day. Without even relying on other no-recoding optimizations like
>> pypy or gevent.
>>
>> Async would certainly have been a relevant techno if we had known in
>> advance that our partners would be so slow, but avoiding the extra
>> complexity burden of this style (where a single buggy dependency can block
>> all requests in a process, where all modules have to be recoded for it) was
>> also a huge benefit. And the limited thread pool also protected our DB from
>> unbearable loads.
>>
>
> Please remember that even after this change, Django will still expect you
> to write synchronously by default, and not impose any of that extra
> complexity on you. We will only swap out the "native" implementation of
> things if the performance matches (within ~10%) or exceeds that of the
> synchronous version when there's a couple of threads going; it's expected
> this will largely be the case due to the direct benefits of idling less.
>
> But - the plan is not to make it more complex by default (you only have to
> interact with the async if you want to) or slower.
>
>
>>
>> It's very nice if a proper async ecosystem emerges in python, but I fear
>> lots of people are currently jumping into it without a need for such
>> performance, and at the expense of lots of much more important matters like
>> robust ess, correctness, compatibility... like it happened for docker and
>> microservices, transforming into fragile bloatwares simple intranets, which
>> just needed a single django codebase deployed in a single container.
>>
>> A few days ago I audited a well used django module, the current user was
>> stored in a global variable (!!!). People might eventually fix that ticket,
>> use threadlocals, and then switch to a future django-async without
>> realizing that the security issue has come back due to the way async works.
>>
>> Still I hope I'm wrong, that the performance gains will prove worth the
>> software fragmentation and complexity brought by asyncio, but I still dont
>> understand them for 99% users... Especially as long as key-in-hand
>> solutions like greenlets exist for power users.
>>
>>
> I agree with you that there's a chance this is all useless and doesn't
> bear fruit, in which case I will be the first person to pull the plug and
> say that Python async isn't ready. However, I've been working with it for
> the last four years, including on several very large deployments

Re: Resource loading (Django without a filesystem)

[Andrew Godwin]

Well, remember that in Python 3 you don't need an __init__.py file to have
something be a module (because of  https://www.python.org/dev/peps/pep-0420/),
but I wonder if there still needs to be a proper discovery mechanism that
flags that they should be considered as implicit packages/modules.

Andrew

On Thu, Jun 27, 2019 at 2:01 PM Peter Baumgartner 
wrote:

> The big issue I see is that a resource must reside directly in a
> Python module. You can not load a resource from a child directory,
> e.g. I can load "index.html" from the Python module
> "myproject.templates", but I can't load "app1/index.html" from the
> same module.
>
> This would require developers to scatter __init__.py files in every
> template directory. I think migrations would be easy (you can list
> Python files in a module without the filesystem). I need to look at
> how staticfiles works a little closer. I think we could still use
> __file__ for collectstatic (you would run that before converting your
> project), but I'm not sure what Django needs access to at runtime
> there.
>
> Pytz is also an issue. It looks like an easier fix there because the
> project controls the "resource" directories and could sprinkle in the
> necessary __init__.py files. I filed an issue to start the discussion
> there as well, https://bugs.launchpad.net/pytz/+bug/1834363
>
> On Thu, Jun 27, 2019 at 2:40 PM Andrew Godwin  wrote:
> >
> > My impression reading over the problem a little yesterday was that we
> could work to provide a common "get me a resource" abstraction in Django
> that papers over the couple of different ways it has to work, though I
> haven't looked super far into things that require directory listing (e.g.
> migrations) rather than direct file access (like templates).
> >
> > It would be nice to investigate this a bit more, though. If we can get
> Django compatible, or work with PyOxidiser if we find a reasonable
> workaround they could implement, it would be great.
> >
> > Andrew
> >
> > On Thu, Jun 27, 2019 at 12:39 PM Markus Holtermann <
> i...@markusholtermann.eu> wrote:
> >>
> >> Hi Peter,
> >>
> >> PyOxidizer looks indeed super interesting. Talking about templates and
> specifically Jinja2 templates, they are internally converted to the Python
> AST if I'm not mistaking. Turning them into Python modules that a new
> Jinja2ModuleTemplateLoader could load doesn't seem like that far fetched.
> >>
> >> Similarly for Django's migration files. Swapping out the
> MigrationLoader would already be sufficient.
> >>
> >> I'd definitely be interested to see what's needed to change in core to
> have a 3rd party package provide the necessary support.
> >>
> >> Cheers,
> >>
> >> Markus
> >>
> >> On Thu, Jun 27, 2019, at 9:09 PM, Peter Baumgartner wrote:
> >> > I'm interested in using PyOxidizer [1] to create single-file
> executable
> >> > Django projects. This currently isn't possible because of all the
> >> > places Django assumes it is operating on a proper filesystem. I
> haven't
> >> > done a full audit, but templates, migrations, and static files come to
> >> > mind. Python has full support this scenario (aka resource loading),
> but
> >> > it would require some changes to Django internals to use it. One of
> the
> >> > biggest hurdles I see on the surface is that you can only load a
> >> > resource from a Python module (not from a sub-directory). That being
> >> > said, I have a feeling resource loading could be added such that users
> >> > could opt-in or backwards compatibility is maintained with the current
> >> > implementation.
> >> >
> >> > Some additional reading/watching on the subject:
> >> >
> >> > * Topic overview:
> >> >
> https://pyoxidizer.readthedocs.io/en/latest/packaging_pitfalls.html#reliance-on-file
> >> > * Barry Warsaw talk on resource loading:
> >> > https://www.youtube.com/watch?v=ZsGFU2qh73E
> >> >
> >> > I'm posting here to see if there is general support for this and to
> >> > discuss what sort of changes would be required. Thanks!
> >> >
> >> > [1] https://pyoxidizer.readthedocs.io/
> >> >
> >> >  --
> >> >  You received this message because you are subscribed to the Google
> >> > Groups "Django developers (Contributions to Django itself)" group.
> >> >  To unsubscribe from this group and stop receiving emails from it,
> send

Re: Resource loading (Django without a filesystem)

[Andrew Godwin]

My impression reading over the problem a little yesterday was that we could
work to provide a common "get me a resource" abstraction in Django that
papers over the couple of different ways it has to work, though I haven't
looked super far into things that require directory listing (e.g.
migrations) rather than direct file access (like templates).

It would be nice to investigate this a bit more, though. If we can get
Django compatible, or work with PyOxidiser if we find a reasonable
workaround they could implement, it would be great.

Andrew

On Thu, Jun 27, 2019 at 12:39 PM Markus Holtermann 
wrote:

> Hi Peter,
>
> PyOxidizer looks indeed super interesting. Talking about templates and
> specifically Jinja2 templates, they are internally converted to the Python
> AST if I'm not mistaking. Turning them into Python modules that a new
> Jinja2ModuleTemplateLoader could load doesn't seem like that far fetched.
>
> Similarly for Django's migration files. Swapping out the MigrationLoader
> would already be sufficient.
>
> I'd definitely be interested to see what's needed to change in core to
> have a 3rd party package provide the necessary support.
>
> Cheers,
>
> Markus
>
> On Thu, Jun 27, 2019, at 9:09 PM, Peter Baumgartner wrote:
> > I'm interested in using PyOxidizer [1] to create single-file executable
> > Django projects. This currently isn't possible because of all the
> > places Django assumes it is operating on a proper filesystem. I haven't
> > done a full audit, but templates, migrations, and static files come to
> > mind. Python has full support this scenario (aka resource loading), but
> > it would require some changes to Django internals to use it. One of the
> > biggest hurdles I see on the surface is that you can only load a
> > resource from a Python module (not from a sub-directory). That being
> > said, I have a feeling resource loading could be added such that users
> > could opt-in or backwards compatibility is maintained with the current
> > implementation.
> >
> > Some additional reading/watching on the subject:
> >
> > * Topic overview:
> >
> https://pyoxidizer.readthedocs.io/en/latest/packaging_pitfalls.html#reliance-on-file
> > * Barry Warsaw talk on resource loading:
> > https://www.youtube.com/watch?v=ZsGFU2qh73E
> >
> > I'm posting here to see if there is general support for this and to
> > discuss what sort of changes would be required. Thanks!
> >
> > [1] https://pyoxidizer.readthedocs.io/
> >
> >  --
> >  You received this message because you are subscribed to the Google
> > Groups "Django developers (Contributions to Django itself)" group.
> >  To unsubscribe from this group and stop receiving emails from it, send
> > an email to django-developers+unsubscr...@googlegroups.com.
> >  To post to this group, send email to
> > django-developers@googlegroups.com.
> >  Visit this group at https://groups.google.com/group/django-developers.
> >  To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/django-developers/b6229b20-0647-4e0a-b9e6-31136a774e13%40googlegroups.com
> <
> https://groups.google.com/d/msgid/django-developers/b6229b20-0647-4e0a-b9e6-31136a774e13%40googlegroups.com?utm_medium=email_source=footer
> >.
> >  For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/2faec64a-da9f-4d55-9378-a3cb6a308d53%40www.fastmail.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1ur36dYm7xohN0PQRkY3Y8H5m8-Ws307Wo-u%2B3xMtjLBKg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: A Django Async Roadmap

[Andrew Godwin]

The DEP is drafted and in the DEPs repo, and awaiting approval by the
freshly-elected Technical Board once I submit it. In the meantime, we
landed the ASGI patch, as well.

Andrew

On Tue, Jun 25, 2019 at 3:30 PM Chris Barry 
wrote:

> Hey all,
>
> Just wondering what the future of this is looking like?
>
> CB
>
> On Friday, 12 April 2019 07:33:35 UTC+1, Shaggy wrote:
>>
>> and how it is going ?
>> is there some interest from django devs?
>>
>> On Monday, 4 June 2018 15:18:23 UTC+2, Andrew Godwin wrote:
>>>
>>> Hello everyone,
>>>
>>> For a while now I have been working on potential plans for making Django
>>> async-capable, and I finally have a plan I am reasonably happy with and
>>> which I think we can actually do.
>>>
>>> This proposed roadmap, in its great length, is here:
>>>
>>> https://www.aeracode.org/2018/06/04/django-async-roadmap/
>>>
>>> I'd like to invite discussion on this potential plan - including:
>>>
>>>  - Do we think async is worth going after? Note that this is just async
>>> HTTP capability, not WebSockets (that would remain in Channels)
>>>
>>>  - Can we do this in a reasonable timeframe? If not, is there a way
>>> around that?
>>>
>>>  - Are the proposed modifications to how Django runs sensible?
>>>
>>>  - How should we fund this?
>>>
>>> There's many more potential questions, and I really would love feedback
>>> on this. I'm personally pretty convinced that we can and should do this,
>>> but this is a decision we cannot take lightly, and I would love to hear
>>> what you have to say.
>>>
>>> Andrew
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/6ea76507-4041-4850-ac6c-bb13a09af941%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/6ea76507-4041-4850-ac6c-bb13a09af941%40googlegroups.com?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uoPkM6bZUnAvkK6Yq7P0%2BvtRm4yFHW-ZxzuQKOvr%3DPv%3Dw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Andrew Godwin]

On Sat, Jun 8, 2019 at 3:14 AM Pascal Chambon 
wrote:

> Hello,
>
> There is something a little scary for me, in changing all the core of
> Django to async, when this really helps only, imho, a tiny fraction of
> users : websocket/long polling services, and reddit-like sites with
> thousands+ hits per second. For most webpages and webservices, async
> artillery sounds quite overkill.
>
> Are cpython threads inefficient ? As far as I know they are only kernel
> threads constrained by the Gil, so they shouldnt wake up when they are
> blocked on io syscalls/mutexes (or do they?), and context switches remain
> acceptable compared to the slowness of python itself.
>

It's fine when you only at 5/10 threads - which, notably, is what most WSGI
servers run at. When you get to the hundreds, though, you start losing a
large proportion of your execution time (tens of percent, in some cases).


>
> We used to provide provisioning and automatic authentication for 20
> million users, with partner webservices tar-pitting us for sometimes 1mn.
> The nightmare scenario. But with 2 machines, 1 process by core, and 800
> threads by process, it did the job, enough for us to answer millions of
> hits a day. Without even relying on other no-recoding optimizations like
> pypy or gevent.
>
> Async would certainly have been a relevant techno if we had known in
> advance that our partners would be so slow, but avoiding the extra
> complexity burden of this style (where a single buggy dependency can block
> all requests in a process, where all modules have to be recoded for it) was
> also a huge benefit. And the limited thread pool also protected our DB from
> unbearable loads.
>

Please remember that even after this change, Django will still expect you
to write synchronously by default, and not impose any of that extra
complexity on you. We will only swap out the "native" implementation of
things if the performance matches (within ~10%) or exceeds that of the
synchronous version when there's a couple of threads going; it's expected
this will largely be the case due to the direct benefits of idling less.

But - the plan is not to make it more complex by default (you only have to
interact with the async if you want to) or slower.


>
> It's very nice if a proper async ecosystem emerges in python, but I fear
> lots of people are currently jumping into it without a need for such
> performance, and at the expense of lots of much more important matters like
> robust ess, correctness, compatibility... like it happened for docker and
> microservices, transforming into fragile bloatwares simple intranets, which
> just needed a single django codebase deployed in a single container.
>
> A few days ago I audited a well used django module, the current user was
> stored in a global variable (!!!). People might eventually fix that ticket,
> use threadlocals, and then switch to a future django-async without
> realizing that the security issue has come back due to the way async works.
>
> Still I hope I'm wrong, that the performance gains will prove worth the
> software fragmentation and complexity brought by asyncio, but I still dont
> understand them for 99% users... Especially as long as key-in-hand
> solutions like greenlets exist for power users.
>
>
I agree with you that there's a chance this is all useless and doesn't bear
fruit, in which case I will be the first person to pull the plug and say
that Python async isn't ready. However, I've been working with it for the
last four years, including on several very large deployments, and there are
some direct benefits that I believe we can get without making things a lot
more complex, even inside Django.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1upp%3D0L9diMZr5qTxLeri329PuWWdMeN8uxyfmTt2jZM1w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Andrew Godwin]

On Fri, Jun 7, 2019 at 9:19 AM John Obelenus  wrote:

> I wonder about the end-result payoff of this approach. In general,
> Django/Python code is not going to be I/O bound, which is where
> asynchronous approaches are going to get the bang for your buck. Even when
> it comes to DB access—the DB is a lot faster than the python and django
> code running against the result set. And too much context-switching (as you
> noted) has painful ramifications for performance.
>

To the contrary, I have found that as you scale up, a large amount of your
time becomes I/O (either HTTP calls to other components/hosted serviecs or
database calls). Our APM at work shows me that it's around 80% of request
time.

Obviously we don't design Django just for large use cases, which is why
it's not going to be the default, but with the massive growth of hosted
services, I suspect this trend will continue to trickle down to smaller
deploys too. And ultimately, for smaller deploys performance is rarely a
concern anyway.


>
> I can absolutely see why creating a layer that handles asgi, websockets,
> and http requests asynchronously is going to pay off. Bit time. But I'm
> less certain that the ORM access will benefit from an asyncio approach. Do
> we have anything that approaches a hard number that would tell us re-doing
> the ORM layer in asyncio would get us X% performance benefit?
>
> I'm basing my thoughts off this well-reasoned look at performance:
> https://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/
>
>
I do not personally have hard numbers that I am allowed to share,
unfortunately, but I would encourage you to look at results on benchmarks
that include database access - like this one (
https://twitter.com/_tomchristie/status/1005001902092967936) using Python
asyncio/ASGI - and see that it does make a difference. Obviously it doesn't
matter for all deploys, but I believe it matters for the majority of site
architectures as they scale up.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1urJVeamhLWLLDpkz_-e1fAM59x6b6rzyAV%3DZAUgg9qrvQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Andrew Godwin]

On Thu, Jun 6, 2019 at 3:20 PM Pkl  wrote:

> Hello,
>
> I'm a little late to the party, thanks for the big overview on this
> complex matter.
>
> There are lots of thinks I still don't understand though, for example
> regarding "what it unlocks". What asyncio structures would allow to run
> several DB queries concurrently safely and easily, that can't be achieved
> with some threadpool-like mechanism in nowadays' cod ?
>

Python threads perform badly, as they force context-switches even when no
work is pending; you can't sensibly run more than 20/30 threads before the
overheads seriously start eating into your performance.


>
> Most importantly, I'm not getting why gevent/eventlet-style solutions are
> systematically being dismissed in favor of asyncio. And the more I
> read/talk about it, the less I understand what all that hype around asyncio
> is. This is a whole new language, which forces people to trashbin half of
> the Python ecosystem, and remake it with similar code but filled with
> incompatible async/await statements.
>
> ...
>
> Granted, I have little experience with Geven and Asyncio, but all
> experience feedbacks I've read so far mainly insist on minor limitations of
> greenlets, and on the fact that "people are mainly going with asyncio" (a
> self-fulfilling prophecy?). Considered the dramatic difference in workload
> between the two, I'd really love to understand what killer-features justify
> to go for the "recode everything" solution (or what greenlet limitations
> would be show-stoppers on the long term).
>

I can't give you a full answer about why they are being dismissed, but the
move is clearly systematic. My own personal experience is that writing with
gevent in particular ends up being very difficult, as it is not very
explicit about what is async, what causes a context switch, and so you end
up wrapping a lot of your code in locks to even try and get code that isn't
susceptible to nasty race conditions.

I suspect asyncio fits much more with the Zen of Python - you know exactly
when a context switch might occur (when you see an await), and modules
explicitly add support for it rather than having it monkey-patched in. In
addition, most people I know in the Python community who are actively
working on async libraries are doing so against asyncio.

You could make the same argument with trio - it's arguably a better,
cleaner async implementation. But, again, it's not where Python is at. This
change is already big enough that it's very important we keep consistency
with Python core to lower the workload. If Python core turned around and
blessed greenlets and gevent as the chosen async solution, I'd change my
mind, but I haven't seen any evidence of that over many years.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uqgUUwL_%3DXS%2Be6tzieycn%3DjDZK6wFCKzHKxRtfukacfQw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Andrew Godwin]

Just a heads up that feedback on this draft DEP has slowed down, and so I
have merged it into the deps repository.

This is an invitation for any additional feedback before I take the DEP to
the Technical Board to get their opinion in a week or two. In the meantime,
I am going to start work on writing up a funding plan for this, including
various options for how we can pay people for their work.

Andrew

On Tue, May 14, 2019 at 2:55 AM Andrew Godwin  wrote:

>
>
> On Mon, May 13, 2019 at 4:31 AM Tobias Kunze  wrote:
>
>> Hi Andrew (and everybody following the discussion, of course),
>>
>> First off, thank you for your work here. DEP9 is an excellent technical
>> document, and it was as easy and pleasant to read as a document of this
>> scope and depth can be.
>>
>> Especially the Motivation section was very insightful – it might be
>> worth moving it up a bit, as I found myself dropping about a third of my
>> notes and questions from the first half of the DEP after reaching the
>> motivation section.
>> Also, discussing the Why first and the How later is a bit better
>> argumentatively – if I just spent half an hour reading about the How,
>> I'll start to see the Why as a given, so it's harder to reason about
>> alternatives upon reaching the motivation section. Maybe that's just me,
>> though.
>>
>
> The ordering that's there is defined by the DEP template, alas - though
> maybe that's just because of historical reasons. I agree it might be better
> to swap them.
>
>
>>
>> The Sequencing section is equally helpful to get a feeling for the
>> implementation work. It might be worth including a note on additional
>> future DEPs there, as those are only mentioned in the high level summary
>> (and in the templating section, I think).
>>
>
> I've added another paragraph in there to flesh it out a bit.
>
>
>>
>> I have a couple of questions and comments – none of which are meant to
>> criticise the direction of the DEP or arguments. It's just niggles,
>> really. I also left some comments on the PR that concern only some
>> phrasing.
>>
>> The DEP doesn't really include a discussion of potential downsides. Even
>> if that's not in scope of a DEP, I'd like to ask this here:
>>
>> - Is there a potential negative impact for Django users who just
>>   continue to use Django? The only mention of this says "Running code in
>>   threads is likely not going to increase performance - the overhead
>>   added will probably decrease it very slightly in the case where you're
>>   just running normal, linear code". Do you have any details on that? I
>>   would have expected some (even just a sentence or two) discussion of
>>   potential downsides in the Rationale section next to the alternatives.
>> - Is there a potential negative impact on Django if work on this
>>   proposal takes a long time/is abandoned?
>>
>
> These are both good questions - I have added in explainers about them to
> the "Motivation" section. (In short: Yes, but we'll keep it to a small
> impact; No, as long as we're good about code committing and being
> sustainable).
>
>
>>
>>
>> Less general remarks:
>>
>> > Every single feature that is converted to be async internally will also
>> present
>> > a synchronous interface that is backwards-compatible with the API as it
>> stands
>> > today (in 2.2), at least for the normal deprecation period.
>>
>> As it stands, this will sound a lot like "we'll deprecate lots of
>> synchronous interfaces soon" to people who are afraid of exactly that.
>> It's probably worth to clarify this here, or to choose different
>> phrasing, unless that's what you're planning to do.
>>
>
> Good point, I've made this a lot clearer.
>
>
>>
>> > This general overview works on nearly all features on Django that need
>> to be
>> > async, with the exceptions mostly being places where the Python
>> language itself
>> > does not provide async equivalents to features we already use.
>>
>> Can you give examples for this? They don't need to be in the draft, I
>> think, I'd just like to understand this part better.
>>
>
> The classic one is attribute access - Django uses "
> model_instance.related_field.name" to do work in the background.
> Attribute access can't be async, though, so we can no longer call the
> database.
>
> It also exists for length - doing "len(queryset)" does not pass down an
> async context, so you can't do blocking work down there in a subthread. The
> pattern conti

Re: Django Async DEP

[Andrew Godwin]

On Mon, May 13, 2019 at 4:31 AM Tobias Kunze  wrote:

> Hi Andrew (and everybody following the discussion, of course),
>
> First off, thank you for your work here. DEP9 is an excellent technical
> document, and it was as easy and pleasant to read as a document of this
> scope and depth can be.
>
> Especially the Motivation section was very insightful – it might be
> worth moving it up a bit, as I found myself dropping about a third of my
> notes and questions from the first half of the DEP after reaching the
> motivation section.
> Also, discussing the Why first and the How later is a bit better
> argumentatively – if I just spent half an hour reading about the How,
> I'll start to see the Why as a given, so it's harder to reason about
> alternatives upon reaching the motivation section. Maybe that's just me,
> though.
>

The ordering that's there is defined by the DEP template, alas - though
maybe that's just because of historical reasons. I agree it might be better
to swap them.


>
> The Sequencing section is equally helpful to get a feeling for the
> implementation work. It might be worth including a note on additional
> future DEPs there, as those are only mentioned in the high level summary
> (and in the templating section, I think).
>

I've added another paragraph in there to flesh it out a bit.


>
> I have a couple of questions and comments – none of which are meant to
> criticise the direction of the DEP or arguments. It's just niggles,
> really. I also left some comments on the PR that concern only some
> phrasing.
>
> The DEP doesn't really include a discussion of potential downsides. Even
> if that's not in scope of a DEP, I'd like to ask this here:
>
> - Is there a potential negative impact for Django users who just
>   continue to use Django? The only mention of this says "Running code in
>   threads is likely not going to increase performance - the overhead
>   added will probably decrease it very slightly in the case where you're
>   just running normal, linear code". Do you have any details on that? I
>   would have expected some (even just a sentence or two) discussion of
>   potential downsides in the Rationale section next to the alternatives.
> - Is there a potential negative impact on Django if work on this
>   proposal takes a long time/is abandoned?
>

These are both good questions - I have added in explainers about them to
the "Motivation" section. (In short: Yes, but we'll keep it to a small
impact; No, as long as we're good about code committing and being
sustainable).


>
>
> Less general remarks:
>
> > Every single feature that is converted to be async internally will also
> present
> > a synchronous interface that is backwards-compatible with the API as it
> stands
> > today (in 2.2), at least for the normal deprecation period.
>
> As it stands, this will sound a lot like "we'll deprecate lots of
> synchronous interfaces soon" to people who are afraid of exactly that.
> It's probably worth to clarify this here, or to choose different
> phrasing, unless that's what you're planning to do.
>

Good point, I've made this a lot clearer.


>
> > This general overview works on nearly all features on Django that need
> to be
> > async, with the exceptions mostly being places where the Python language
> itself
> > does not provide async equivalents to features we already use.
>
> Can you give examples for this? They don't need to be in the draft, I
> think, I'd just like to understand this part better.
>

The classic one is attribute access - Django uses "
model_instance.related_field.name" to do work in the background. Attribute
access can't be async, though, so we can no longer call the database.

It also exists for length - doing "len(queryset)" does not pass down an
async context, so you can't do blocking work down there in a subthread. The
pattern continues for most things that Python makes objects supply
__special__ methods for; unless serious work is done to Python as an async
language, these will likely stay the same indefinitely.


>
> > Asynchronous views will continue to be wrapped in an ``atomic()`` block
> by
> > default - while this reduces immediate performance gains, as it will
> lock all
> > ORM queries to a single subthread (see "The ORM" below), it is what our
> users
> > will expect and much safer. If they want to run ORM queries
> concurrently, they
> > will have to explicitly opt out of having the transaction around the
> view using
> > the existing ``non_atomic_requests`` mechanism, though we will need to
> improve
> > the documentation around it.
>
> By default, Django's views are not wrapped in ``atomic()`` blocks. This
> is only the case if ``ATOMIC_REQUESTS`` is ``True``, which it isn't by
> default. Not sure if an off-by-default feature is worth an entire
> paragraph here, but in any case, please make it clear that not every
> async view will be wrapped in an ``atomic()`` block (unless I'm mistaken
> and they will be?).
>

I have updated this. I have kept 

Re: DEP 0008 (Formatting using Black) is accepted

[Andrew Godwin]

One quick clarification - when I said "stable (1.0)" release, I in fact 
meant the first release that the Black project officially marks as stable.

Black doesn't use versioning that would result in a stable release being 
called 1.0, as far as I know, given they are on 19.3b0 right now!

Andrew

On Friday, May 10, 2019 at 6:46:37 PM UTC-7, Andrew Godwin wrote:
>
> Hi everyone,
>
> The Technical Board has voted on DEP 0008 (
> https://github.com/django/deps/blob/master/accepted/0008-black.rst), 
> after the extensive discussion here and the updates done as a result.
>
> The board voted in favour, and so I've moved the DEP into the accepted 
> directory.
>
> One of the key changes to the proposal was to wait until Black got to a 
> stable (1.0) release, so it will sit in "accepted" until that happens, at 
> which point we will do the implementation (including the big reformat 
> itself) and then move the DEP to "final".
>
> There's no hard timeline on when Black will hit a stable release, but it's 
> very close; there are a few changes we'd like to make sure are included 
> when we reformat Django, such as keeping lists with trailing commas on 
> multiple lines (https://github.com/python/black/pull/826). That's why 
> we're waiting for the stable release rather than reformatting now.
>
> Thanks to everyone who chimed in on the discussion - this was not an easy 
> decision, and the feedback helped a lot to work out what was best for us.
>
> I also want to personally thank Herman Schistad, for prompting this 
> discussion, and Aymeric Augustin, for his work on writing and updating the 
> DEP!
>
> Yours in auto-formatting,
> Andrew
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c7cd08c4-6d16-4141-b21c-438306a8d458%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

DEP 0008 (Formatting using Black) is accepted

[Andrew Godwin]

Hi everyone,

The Technical Board has voted on DEP 0008 (
https://github.com/django/deps/blob/master/accepted/0008-black.rst), after
the extensive discussion here and the updates done as a result.

The board voted in favour, and so I've moved the DEP into the accepted
directory.

One of the key changes to the proposal was to wait until Black got to a
stable (1.0) release, so it will sit in "accepted" until that happens, at
which point we will do the implementation (including the big reformat
itself) and then move the DEP to "final".

There's no hard timeline on when Black will hit a stable release, but it's
very close; there are a few changes we'd like to make sure are included
when we reformat Django, such as keeping lists with trailing commas on
multiple lines (https://github.com/python/black/pull/826). That's why we're
waiting for the stable release rather than reformatting now.

Thanks to everyone who chimed in on the discussion - this was not an easy
decision, and the feedback helped a lot to work out what was best for us.

I also want to personally thank Herman Schistad, for prompting this
discussion, and Aymeric Augustin, for his work on writing and updating the
DEP!

Yours in auto-formatting,
Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uox9b7i895zS0pymzJ-kUQtd_USkQi-%3DbY-q58noo3Mmg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Andrew Godwin]

On Fri, May 10, 2019 at 3:51 AM Asif Saif Uddin  wrote:

> I have a separate question. Is it possible to get the django 3.0 asgi
> things into a different package to use with django 2.2?
>
> Thanks for the great work.
>
>
Unfortunately not - this is covered a little in the section in the DEP that
says why we can't do it in a separate package, but basically, the changes
required to Django are too deep to do separately (or even as a long-running
fork).

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uppnH8r_%3Dgk_o%2BuuP%2BpxR7Da6HQzR8%2B%3DmyM3yHkEZcyDw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Andrew Godwin]

On Thu, May 9, 2019 at 2:44 PM KimSia Sim  wrote:

> I have a question that strays slightly away from the main topic. I have
> looked at Tom's repos. Is it encode/databases
> https://github.com/encode/databases that you're referring to?
>
> Or do you mean Tom's working on an async ORM that works within Django?
>

It is indeed encode/databases!

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uoRf2Ug5b870zspfudncKPK%3DaWjff-z%2BGCFbW53pGsxxQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Andrew Godwin]

On Thu, May 9, 2019 at 2:15 PM Patryk Zawadzki  wrote:

> I don't think it makes a significant difference from a readability
>> perspective in this example. It does have some advantages, though:
>>
>> - It could be easier to implement a sync version based on the async one,
>> or vice-versa, if each one has its own class. It will probably be more DRY.
>> - It could also be a bit more usable by developers, especially in IDEs
>>
>
> Speaking about convenience, it would be great if we could make the sync
> versions raise exceptions when used with an event loop attached and the
> async ones fail when there isn't a loop running. I assume synchronous views
> would be executed within a worker thread with the event loop explicitly set
> to None.
>

My ASGI patch already does this with the @async_unsafe decorator all over
the ORM so people can't screw up by accident. It would be quite easy to
extend this to enforcement on both the sync and async versions - there's
maybe an edge case that you can call an async function from a thread you
have not started an event loop in _yet_, but I'd rather see if and when
that happens and provide a workaround, maybe.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uq4HX%3DHQa9Gmymckv-T_qXbGgA%3DtUrHn2pG8ejVSML3GA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Andrew Godwin]

>
> I don't think it makes a significant difference from a readability
> perspective in this example. It does have some advantages, though:
>
> - It could be easier to implement a sync version based on the async one,
> or vice-versa, if each one has its own class. It will probably be more DRY.
> - It could also be a bit more usable by developers, especially in IDEs
>

Right. It could also be a sub-class on the class in question to make the
implementation easier - e.g. "core.cache.cache.sync" is a literal
passthrough, and "core.cache.cache.async" is the async wrapper
implementation.


>
> Also it could provide an alternative solution for the async attribute
> access problem: `model_instance.async.related_field` could be a Future
> that, when awaited, resolves to the related field. So you could `await
> model_instance.async.related_field` instead of having to `select_related`.
> To be honest I didn't investigate all the consequences of this idea. It
> seemed worth mentioning, though, even if it turns out to be impractical :-)
>

I think this might be feasible? I'd want to probably push it to have its
own research and proposal, probably, but it might be a nice way out of the
initial thing of requiring select_related. I just don't know enough about
how that might cascade down the ORM internals to judge it at this point!

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1upEsCpnL-81guSw_vKLn%3DWmfM3pamuFY%2BSZWLMerNC0yA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Andrew Godwin]

On Thu, May 9, 2019 at 1:04 PM Aymeric Augustin <
aymeric.augus...@polytechnique.org> wrote:

> Hello Andrew,
>
> Thanks for your work putting together this plan. Within our constraints,
> it's a good plan.
>
> Regarding templating, I would say it isn't a priority because a developer
> who knows how to parallelize I/O bound operations will prefer (or at least
> accept) to perform these operations in the view, not in the template.
>
> I'm on the fence about the convention for async APIs. I'm not super
> excited by spraying async code with _async prefixes. The namespacing
> approach would allow for cleaner async code. Most Python modules should be
> either sync or async, not a mix of both styles. But I might be
> underestimating the importance of explicitness...
>
>
I am also not a fan of the approach, but I did err towards being explicit.
Jannis suggested what I think might be a nicer approach on Twitter, which
is to add an async "proxy object" to access methods with, e.g.:

cache.get("foo")
cache.async.get("foo")

This is still explicit but looks less ugly, and gives us a way in future to
make some modules transition to async-by-default (by e.g. supplying
cache.sync.get). What do you think?

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uowvL3FH9BR80hpOoBdRw-qUwpu5bA%3DBDLXmhY0WzBM%2Bg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Andrew Godwin]

> Slightly off-topic but once we have an async ORM, making it synchronous
is not impossible (I believe either Channels or Daphne already have shims
that use a worker thread to spin the event loop until a future is
fulfilled).

This is in fact the proposed way to make the ORM work with both sync and
async worlds once the interior is fully-async.

> In one project I really enjoyed using channels just as a background
worker instead of celery, and the DEP does not talk much in the Email
section: use the async variant, can be tackled separately, low priority.
>
> For now the ORM works, but if sending an email fails (ie. SMTP down) it
will raise an exception that can end up in a 500 error, it would be great
if Django could retry so that emails leave when its SMTP is up again, would
that maybe weight in priority ?

If you want guaranteed email delivery, that's a task for something like
Celery or a third-party API; any method of sending emails in the background
in the same process, be it threads or async coroutines, is going to be
unreliable as the server may die any time. I don't think it's sensible for
Django to try and solve this problem internally, at least not as part of
this async push.

Andrew

On Thu, May 9, 2019 at 12:56 PM Patryk Zawadzki  wrote:

> That said, I also think it's important to allow the ORM to support both
>> modes in the long term. I truly believe the best way to be able to write
>> async code is to _have the choice to write it_, rather than being made to
>> all the time; if we make people use a separate, async ORM, then we force
>> them to write every view asynchronously, with all the extra danger and
>> thinking that requires. It's much better for Django to do the hard work,
>> and say "hey, if you want to write asynchronously or synchronously, that's
>> fine - it takes literally zero extra effort to go either way".
>>
>
> Slightly off-topic but once we have an async ORM, making it synchronous is
> not impossible (I believe either Channels or Daphne already have shims that
> use a worker thread to spin the event loop until a future is fulfilled).
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/402dbe77-44fc-485b-b058-e619a1576ddb%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1urOnhEJDb6Sm45U59iF5hCZmN%3Dw-J%2B0er4UCnqh3qsmCw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Andrew Godwin]

I would agree with both of you - I think the most important thing is to get
the view layer async-capable, as that then lets sites use any manner of
asynchronous libraries that already exist to get experiments and unique
things going. Tom Christie, for example, has already started work on an
asynchronous ORM. Some of Django's biggest sites don't use the ORM - for
example, Instagram.

That said, I also think it's important to allow the ORM to support both
modes in the long term. I truly believe the best way to be able to write
async code is to _have the choice to write it_, rather than being made to
all the time; if we make people use a separate, async ORM, then we force
them to write every view asynchronously, with all the extra danger and
thinking that requires. It's much better for Django to do the hard work,
and say "hey, if you want to write asynchronously or synchronously, that's
fine - it takes literally zero extra effort to go either way".

This is why I propose in the DEP that we do the view layer first, and then
move onto the ORM as a second wave.

Andrew

On Thu, May 9, 2019 at 12:29 PM Patryk Zawadzki  wrote:

> I'm not sure but for me the "What is Django" section answers the question.
>> For me Django is full of philosophy that seeds a great ecosystem of apps of
>> all sorts with a growing user base nonetheless, and a bunch of brilliant
>> hackers to look up to and inspire for more. Of course if you're into fixing
>> a particular technical issue such as "I want to make an efficient GraphQL
>> server", then even Django works even if it's not necessarily the most fit
>> for X reason. But for "I want to get a web page going with some forms and a
>> database and see where that goes" then Django is definitely enjoyable, and
>> at this point in time where tons of more sophisticated frameworks are born
>> it's clear that Django is still in the game and will be for the next 10
>> years. So, why not also get more features out of it while we've not been
>> abandoned by all the talented contributors ? It's not because we like
>> Django that we cant haz nice things ;)
>>
>
> I'm arguing the opposite, that by limiting the scope of the MVP to
> asynchronous views (even if middleware stays synchronous for now) and using
> an existing ORM we can make Django a viable solution for many new
> applications. And some of them don't need the full stack.
>
> I'd go as far as arguing that fewer and fewer applications benefit from
> Django providing all of the pieces (as many new applications only implement
> an API and render in the client) so treating them all as blockers here may
> not be beneficial.
>
> Best regards
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/314d7b71-1bf1-441b-ae49-e4fbeb5e1891%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1urQVr%3DZ55Xqn7E%2B5-AeZU-egJAZ_%2BooAp_3nSSYnbNX7w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Django Async DEP

[Andrew Godwin]

Hello everyone,

While the ASGI patch (https://github.com/django/django/pull/11209) is maybe
the first small step in a long road to async, it's the only real one we
could do without a DEP as it purely pulls on existing specs and bugs.

To that end, I have drafted a DEP (provisionally #0009), "Async-capable
Django". It is a summary - and I use that word very lightly - of the rest
of the work to be done to make a version of Django that one could call
"async capable".

You can view it here:
https://github.com/andrewgodwin/deps/blob/async/draft/0009-async.rst

It is approximately 7,000 words long - while it could probably do with some
editing, the subject matter means it has to be quite in-depth. Even then,
it does not propose an exact plan to follow; instead, it proposes the
overall strategy and the high-level ideas about what needs to be done.

If you are short on time, please read the Foreword and the High-Level
Summary, and then jump around using the table of contents to the sections
that interest you.

Feedback on this is very much appreciated; you can either reply here or, if
you have comments that would benefit from precise location on the diff, use
the pull request here: https://github.com/django/deps/pull/56

This is a very complex topic, and I appreciate feedback might take a while;
at least a couple of weeks to get everyone's thoughts. If you want a more
private channel to discuss concerns or design questions you're not happy to
talk about publicly, feel free to email me directly.

This proposal does not quite fit the DEP template - we never really planned
for something of this scale - but I have done my best to make it work; I
think we still benefit from discussing and voting on it in the DEP format,
but suggestions on how to make it work better are welcome, provided they
fit within how Django operates as a community now.

Finally, if there is positive feedback on this, we have the separate
question of funding it that I will discuss separately, if and when we get
that far. The DEP covers some of the topics, but it's notable that the
implementation speed of this varies considerably; if we could get, say, me
and someone else on it full-time (which is not a cheap or likely prospect),
we can probably be done inside of a single release. If we rely just on
volunteer time, it may take years. There is hopefully somewhere between
those two that works.

Thanks for taking the time to read through!

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uqav3JXQA3m%2Bajf0Bd15QQX3JTZTfmH3Hc3ECWyRR3CVg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: First ASGI pull request is ready for review

[Andrew Godwin]

This is quite unrelated to frontend - I'll explain more of the impact and
potential impact in the DEP when I write it up.

Andrew

On Wed, 1 May 2019, 08:30 Elad Yaniv,  wrote:

> Exciting stuff!
> does this mean that django 3.0 COULD  compete with frontend js frameworks
> ? (angular react vue)
> if the answer is yes  i will be a very happy developer
>
> Great work!
> Elad.
>
>
> On Wed, May 1, 2019 at 9:55 AM Carlton Gibson 
> wrote:
>
>> Yes, I’ll review properly first half of next week.
>>
>> For the DEP, can you break out how and where people might input. There’s
>> massive interest. 
>>
>> Great work as ever Andrew. Thank you so much!
>>
>> C.
>>
>> On Wed, 1 May 2019 at 08:46, Andrew Godwin  wrote:
>>
>>>
>>>
>>> On Tue, Apr 30, 2019 at 11:34 PM Mariusz Felisiak <
>>> felisiak.mari...@gmail.com> wrote:
>>>
>>>> Thanks for this patch. Can you add a trac ticket? also Can you give me
>>>> & Carlton few days for review? I should be able to do this somewhere in the
>>>> next week.
>>>>
>>>
>>> I can indeed. I wasn't sure if you wanted to get around to reviewing it
>>> or not, but take all the time you need to review. I'm sure there's some
>>> more in there that could be tightened up.
>>>
>>> (Forgive me if I seem like I am being pushy - just trying to make sure
>>> we keep forward progress!)
>>>
>>>
>>>>
>>>> Are we going to create "Async" DEP?
>>>>
>>>>
>>> We probably should do now there is a more solid plan - the discussion on
>>> the mailing list from before was only really about this first step, not
>>> about what it would look like to adopt this more fully. I don't think the
>>> solid agreement from before means we get to skip writing this up, and from
>>> here on out it becomes much more of a new feature than merely adding safety
>>> and a new handler.
>>>
>>> Andrew
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-developers+unsubscr...@googlegroups.com.
>>> To post to this group, send email to django-developers@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/django-developers/CAFwN1urbURE-v2wtts49or54OD-ui%3DWEKVaZY%2BFUk9xQbOsu1Q%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/django-developers/CAFwN1urbURE-v2wtts49or54OD-ui%3DWEKVaZY%2BFUk9xQbOsu1Q%40mail.gmail.com?utm_medium=email_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/CAJwKpyR4yME8VyMKrSaFZGkvZdOuh4eQiX6u74Vm-4UFEWs%2Bvw%40mail.gmail.com
>> <https://groups.google.com/d/msgid/django-developers/CAJwKpyR4yME8VyMKrSaFZGkvZdOuh4eQiX6u74Vm-4UFEWs%2Bvw%40mail.gmail.com?utm_medium=email_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAO3nhSsZ1ep_VO425kUb8_rMabe9RwWf1Tk%3D2CmhriZqqQRtDw%40mail.gmail.com
> <https://groups.google.com/d/msgid/django-developers/CAO3nhSsZ1ep_VO425kUb8_rMabe9RwWf1Tk%3D2CmhriZqqQRtDw%40mail.gmail.com?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1urwjtOJrZRJRHF2TpXB1eP2ccjg-GAXchGA%3DbuZ5U288g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: First ASGI pull request is ready for review

[Andrew Godwin]

On Tue, Apr 30, 2019 at 11:34 PM Mariusz Felisiak <
felisiak.mari...@gmail.com> wrote:

> Thanks for this patch. Can you add a trac ticket? also Can you give me &
> Carlton few days for review? I should be able to do this somewhere in the
> next week.
>

I can indeed. I wasn't sure if you wanted to get around to reviewing it or
not, but take all the time you need to review. I'm sure there's some more
in there that could be tightened up.

(Forgive me if I seem like I am being pushy - just trying to make sure we
keep forward progress!)


>
> Are we going to create "Async" DEP?
>
>
We probably should do now there is a more solid plan - the discussion on
the mailing list from before was only really about this first step, not
about what it would look like to adopt this more fully. I don't think the
solid agreement from before means we get to skip writing this up, and from
here on out it becomes much more of a new feature than merely adding safety
and a new handler.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1urbURE-v2wtts49or54OD-ui%3DWEKVaZY%2BFUk9xQbOsu1Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: First ASGI pull request is ready for review

[Andrew Godwin]

So, it looks like most of the comments on this PR have happened and been
resolved - unless anyone has any objections, I will merge it in after a
couple more days (just in time for PyCon US).

Andrew

On Wed, Apr 24, 2019 at 1:50 PM Andrew Godwin  wrote:

> Hi everyone,
>
> Just wanted to drop a note and say that the first pull request in the
> series needed to enable async in Django is now ready for review:
> https://github.com/django/django/pull/11209
>
> This is a very minimal amount of work to get Django async-safe and
> understanding ASGI as an application interface, but the idea is to land
> this so it's definitely in 3.0, and then work on getting async abilities
> down to views before the 3.0 feature freeze if we can.
>
> Once those two things are down, we can then expand the effort out a lot
> more and have some work for new contributors to get their teeth into, as we
> can then start making all the other parts of Django async-capable as a
> parallel effort.
>
> Reviews and comments on the PR are encouraged; I want to make sure this is
> not going to hurt existing sync Django when it lands, and that it's a
> useful stepping stone towards async in views.
>
> Andrew
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uoyQb_f2fbEaYfDHDgJL-L_oazV74Pa-5gaF8GXiQRoTQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: A different approach for the auto-reloader

[Andrew Godwin]

>From my read this also looks like it would make the auto-reloader able to
work a lot better with an async-capable server, so I would be in favour
given that is likely in the future as well.

Andrew

On Tue, Apr 23, 2019 at 9:33 PM Ramiro Morales  wrote:

> Hi all,
>
> I had a stab at a somewhat simpler development server automatic reloading
> strategy
> https://github.com/django/django/compare/master...ramiro:synch-reloader
>
> Intention is to test how an implementation of a design by Gary Bernhardt
> would look. The best written description I could find is this:
>
> https://github.com/devlocker/tychus/issues/3
>
> Gary also had posted some tweets (this is how I got interested in the
> topic) which seems to have been deleted since then.
>
> Main idea is: Actual checking of changes on the filesystem for modules
> under monitoring isn't performed in a loop or by depending on a OS kernel
> feature but per-HTTP request by a front-end proxy process which is in
> charge of restarting the 'upstream' web server process (in our case a
> dumbed-down runserver dev server) only when it detects there have been
> changes.
>
> Been meaning to try this for some time. It would have been much harder
> before Tom Forbes' work on refactoring and cleaning up the reloading code
> for Django 2.2. IMHO Tom's code is so very well thought that for example I
> just had to lightly subclass StatReload to implement this totally different
> strategy.
>
> Current form of the code is a new experimental 'serverrun' (for lack of a
> better name) added to the Django code base whose command line UI mimics
> 100% the runserver one.
>
> It copies code from a few places of our code base: The runserver command,
> the WSGI app hosting code, etc.
>
> I decided to implement this as a new built-in command for now a) to ease
> experimentation and b) because it needs some minor changes to the
> 'runserver' command to handle cosmetic details (logging). If the idea is
> accepted (read further below for reasons in favor of this) then maybe we
> can switch runserver to this code. Or if the idea isn't deemed appropate
> for Django core them I might implement it as an standalone django
> app/project.
>
> If the idea of a smarter stat()-based FS status monitor like this gets
> actually tested and validated in the field (i.e. by users with big source
> code trees) it could allow us to possibly stop needing to depend on all of:
>
> * watchman
> * pyinotify
> * watchdog
> (and removing our support code for them from the Django code base).
>
> Also, this would mean:
>
> * Setup simplification for final users (no third party Python libraries or
> system daemon to install)
> * Better cross-platform portability for Django (we go back to
> piggy-backing stat() from the stdlib as our only way yo trigger code
> reloading).
>
> Additionally, as the reloading is performed fully (by restarting the whole
> HTTP server) and is triggered from another process (the transparent http
> proxy one) we can drop some contortions we currently need to make:
>
> - Having to wait for the app registry stabilization
> - Avoiding race conditions with the url resolver
>
> I suspect there could be power efficiency advantages too as:
>
> * The scanning for changes is triggered by HTTP requests which should be
> less frequent than periodically every N seconds.
> * If the developer modifies more than one file before switching to the
> browser there is need of only one FS scan to cater for all these changes,
> which is performed just in time for the first HTTP request so the code
> executed to render/serve it is 100% accurate in regard to actually
> reflecting the state of the code on disk.
>
> Similar projects include:
> - serveit: https://github.com/garybernhardt/serveit
> - tychus: https://github.com/devlocker/tychus
> - wsgiwatch: https://github.com/dpk/wsgiwatch
>
> Feedback is welcome!
>
> Regards,
>
> --
> Ramiro Morales
> @ramiromorales
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAO7PdF99hUobeXs8JQyb%3DywDJ6bkkKWyhUYC%3DEa9JzwQM%2BH_5Q%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 

Re: Proposal to format Django using black

[Andrew Godwin]

Hi all,

As per the DEP process, I have merged DEP 8, Formatting Code With Black,
into the DEP repo as a draft. This doesn't mean it's decided yet - any DEP
that meets quality requirements gets merged as a draft - but it means that
we're one step closer to doing so.

What follows is further discussion until it is either revised, withdrawn,
or the author thinks they have enough feedback to put it to the technical
board.

I will draw attention to the following part of DEP 1:

"However, wherever possible, long open-ended discussions on public mailing
lists should be avoided. Strategies to keep the discussions efficient
include: setting up a separate mailing list for the topic, having the DEP
author accept private comments in the early design phases, setting up a
wiki page, etc. DEP authors should use their discretion here"

Given this thread is now over 100 replies long, we might want to consider a
better avenue for constructive feedback.

Andrew

On Tue, Apr 30, 2019 at 12:31 PM Christian González <
christian.gonza...@nerdocs.at> wrote:

>
> Am 30.04.19 um 14:28 schrieb 'Laurens A. Bosscher' via Django developers
> (Contributions to Django itself):
> > The Chrome team fixed this issue with
> > git-hyper-blame:
> https://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/git-hyper-blame.html
> >
> >
> > That could be a solution that would work for Django. IDE support for
> > git-hyper-blame is lacking (at the moment) but might improve in the
> > future.
>
> I made a feature request for git itself, and they told me this is
> already in the works *within* git dev:
>
> https://public-inbox.org/git/20190410162409.117264-1-b...@google.com/
>
> So on the long run, no need for git-hyper-blame.
>
> Christian
>
> --
> Dr. Christian González
> https://nerdocs.at
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/74e4842c-24fa-6055-2b1e-d70b42153b69%40nerdocs.at
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uq0AEP4wL0%3D1xQOF%2BWBVQfGgU5Zfz0V5BurVCbyMeOA3Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

First ASGI pull request is ready for review

[Andrew Godwin]

Hi everyone,

Just wanted to drop a note and say that the first pull request in the
series needed to enable async in Django is now ready for review:
https://github.com/django/django/pull/11209

This is a very minimal amount of work to get Django async-safe and
understanding ASGI as an application interface, but the idea is to land
this so it's definitely in 3.0, and then work on getting async abilities
down to views before the 3.0 feature freeze if we can.

Once those two things are down, we can then expand the effort out a lot
more and have some work for new contributors to get their teeth into, as we
can then start making all the other parts of Django async-capable as a
parallel effort.

Reviews and comments on the PR are encouraged; I want to make sure this is
not going to hurt existing sync Django when it lands, and that it's a
useful stepping stone towards async in views.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uqxeLhs1bCpZLVuTFbC0S1bEKrbvjWWVvxGQCCtm0ddkg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Proposal to format Django using black

[Andrew Godwin]

We've used Black for the Channels/ASGI projects for about the last year,
and I have nothing but good opinions on it from that perspective. It's made
pull requests much easier to get formatted, because fixing it is just a
case of running a single command (and if you do the right pre-commit hook,
you never even get that far).

As for Django, the single biggest problem I can see is the single reformat
commit, but I think this is worth it. We've had a few of those in our
history - and indeed at work - and while it is a bit of a pain, GitHub has
a "see the blame before this commit" button to enable much easier browsing
of files prior to it.

For that reason, I am +1 to using the Black code formatter.

Andrew


On Sun, Apr 14, 2019 at 10:22 AM Curtis Maloney  wrote:

> So to summarise the discussion so far:
>
> 1. automated code formatting will be a great boon - reduce work, lower
> barrier for new committers, reduce work for the fellows, etc.
>
> 2. there are issues with git history in doing "the great commit".
>
> 3. there are issues with black's formatting choices.
>
>
> So to address 1:
>
> I am entirely +1 for automated code formatting to take the work out of
> our hands.
>
> Can such a tool be automated into, say, github in a way that doesn't
> create extra commit noise?
>
> To address 2:
>
> I side with those who favour a progressive solution, whereby new code
> only has the new tool applied.
>
> That said, there might be cause to suggest a deadline [perhaps a N.0
> release?] where all remaining code is "cleaned".
>
> And finally 3:
>
> My perspective on the goal of any code formatting tool is this:
>
> When we as developers approach a piece of code, our goal is the
> understand its intent and its implementation of that intent.
>
> In the process of reaching that, we pass through the stages of (a)
> identifying the relevant code, (b) understanding the action of the code,
> and (c) understanding the intent of the code.
>
> Good code formatting choices will remove or reduce the cognitive load in
> (b) and (c).
>
> In my experience with using black [we use it at work], there are
> numerous choices (including those demonstrated in this list already)
> where it can significantly _increase_ the cognitive load in simply
> parsing the code.
>
> As simple as black can make the job of code formatting, I feel I'd
> rather see a different tool that retained the benefits of "trivial code
> reformatting", but still allowed us to retain some of Django's existing
> code formatting rules.
>
> (An interesting [and defensible] choice, we had a module with a lot of
> strings wrapped across lines. black opted to push them onto the same
> line, but NOT to merge them.  This is because in Python prior to 3.7, it
> would have altered the generated AST - one of the guides black uses)
>
> --
> Curtis
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/56392596-0246-b62b-4725-628a2b0801ae%40tinbrain.net
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1up9ZKUOLQH_N_M3wMB4ffhxcMWoFMtwemwuTUEXKKJucA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Post mortem on today's packaging error.

[Andrew Godwin]

I also ran into this several times while releasing Channels - several
releases had the wrong files ship out in them due to Git weirdness.

In the end, my solution was to build the release artifacts over on Travis
to guarantee a fresh build environment each time, but I doubt that would
work for Django. Adding "git clean" to the script is probably good enough,
but I might be tempted to make a script where you pass it a commit hash and
it checks it out to a fresh temporary directory and packages from there?

Andrew

On Mon, Feb 11, 2019 at 8:28 AM Carlton Gibson 
wrote:

> Hi all.
>
> This morning I released four versions of Django. Three of which, for 2.1,
> 2.0 and 1.11. (i.e. all the actually supported versions) were broken.
> In the package were additional files from `master`/2.2 which shouldn't
> have been there.
>
> This afternoon I have released follow-ups to correct this issue.
>
> First of all, sorry about that, and for any inconvenience caused.
>
>
> Then, these are process issues so, how can we do better next time?
>
> I'm not 100% sure what occurred.
>
> * The history in Git is correct.
> * I must have had the right commits checked out, because the package
> metadata is correct. (Filenames, version numbers and so on.)
>
> My best guess is that I've failed to `git clean` correctly before building
> each release.
> I'm not certain here because switching between branches doesn't leave the
> repos in an unclean state, and I'm pretty sure it was clean, but this seems
> the most likely error.
>
>
> Q: is there a nice git command to "assert I'm at exactly this tag"?
>
>
> Steps I've taken:
>
> * Moved the `git clean` step into the helper script used to build the
> packages. No chance of then missing it.
> * Added a `diff`-step after building just to make sure what's in the
> `django` module matches the checkout.
>
> The second of these, whilst just a visual check, would have worked with
> what ended up in the package vs what was in my working tree when I checked
> later, but I'm not sure it would have caught the issue when the package was
> created (because presumably my working tree was wrong at that point).
>
> A similar issue affects the checksums, which check that nothing changed
> since it was packaged, but not that the right things were packaged...
> (Similarly, pip install worked without error.)
>
> On the new 1.11 package, I created a virtualenv with Python 2 and ran the
> test suite. This worked so it should be good. But it would be good to
> automate this.
> (I'll look into it.) We don't ship the tests in the wheel, so we have to
> use the src-dist for this. (We could then diff the two django modules to
> make sure they were the same.)
>
> On the other packages I visually looked for incorrect files, but, beyond
> specific migrations (etc) that were known bad from the previous release,
> this doesn't generalise.
>
> "zaytsev" on IRC suggested running `makemigrations --check`, so having a
> test project to install against might also be worth it.
> (I'd guess running the test suite would be sufficient...)
>
>
> Any other thoughts very welcome.
>
>
> Kind Regards,
>
> Carlton
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/90c531fc-3d5b-4d9e-a55f-b88f2e04bb54%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uqvLhhEoB2WFy%3DC4KWA2gLns%3DadhCGwfnYidJsHOjVpUg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Potential suspension of Channels development

[Andrew Godwin]

OK, another update on the update (sorry, I am using the last of my energy
to arrange this handover, so I missed some things yesterday)

I'm going to hand over to a team of Carlton, Jamesie and Asif for now -
I'll get the docs on the repo sorted out tonight and make write up a few
more docs on how to run the release side of things.

I'll give Carlton admin access to the repo if he doesn't have it already,
and the two of you I'll grant commit access. Hopefully that should be
enough to give this a solid go. I'll leave notifications for @andrewgodwin
on, on github, for now, so if you really need me try that, but I may turn
that off if it turns out to be too much.

Thanks all who came forward. I'm hopeful that things can be kept going!

Andrew

On Wed, Jan 30, 2019 at 2:18 PM Andrew Godwin  wrote:

> Just to update on this - nobody has individually come forward to help
> full-time, though I have seen Carlton help out on a few issues (thanks for
> that).
>
> I've added the PySlackers group into the support docs, as well - thanks
> for that offer.
>
> I'm still planning to remove myself from watching all the repos come Feb
> 1st, and barring positive confirmation someone else is going to actively
> take over I'll put up notices on all the projects that they are actively
> unmaintained apart from security issues.
>
> Andrew
>
>
> On Thu, Jan 17, 2019 at 10:06 AM Andrew Godwin 
> wrote:
>
>> Hi all,
>>
>> I'm writing to you all to update you on the current situation of Channels
>> and related libraries (channels-redis and Daphne) and potentially ask for
>> help.
>>
>> I've been the sole maintainer of these projects for quite a while and it
>> has become unsustainable - all of my energy is taken up fielding issues and
>> support requests and I haven't been able to even get myself to start
>> looking at Django async stuff because of it.
>>
>> Given that, if nobody else can step forward to take over, I'll have to
>> put those three projects (Channels, channels-redis, and Daphne) into an
>> explicit maintenance mode where they only accept security requests via the
>> normal security@ route, and start the process of retiring them as active
>> Django projects, as I don't want to give the impression they're still
>> maintained if they're not.
>>
>> (note: the asgiref project is still fine and should probably move out of
>> Django to its own effort at some point giving the growing set of ASGI tools)
>>
>> If people are willing to take over maintenance, I'm happy to help explain
>> some things but I don't have the bandwidth to bring someone completely up
>> from scratch, so I can't help mentor someone who is totally new to
>> maintaining open-source Python (sorry!).
>>
>> Once I recover a bit from the burnout I'll be able to come back and help
>> with the really complex bugs; the main thing I need out of is the seemingly
>> endless support requests and weird WebSocket client bugs.
>>
>> My personal deadline for this is two weeks, on February 1st. If you want
>> to help out, please feel free to reply either here or get in touch with me
>> personally to chat about what's involved.
>>
>> Andrew
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uq3tsuWiW6AkBegMV8O2XiYxCvwimBFOg8KLj6MVDZweQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Potential suspension of Channels development

[Andrew Godwin]

Just to update on this - nobody has individually come forward to help
full-time, though I have seen Carlton help out on a few issues (thanks for
that).

I've added the PySlackers group into the support docs, as well - thanks for
that offer.

I'm still planning to remove myself from watching all the repos come Feb
1st, and barring positive confirmation someone else is going to actively
take over I'll put up notices on all the projects that they are actively
unmaintained apart from security issues.

Andrew


On Thu, Jan 17, 2019 at 10:06 AM Andrew Godwin  wrote:

> Hi all,
>
> I'm writing to you all to update you on the current situation of Channels
> and related libraries (channels-redis and Daphne) and potentially ask for
> help.
>
> I've been the sole maintainer of these projects for quite a while and it
> has become unsustainable - all of my energy is taken up fielding issues and
> support requests and I haven't been able to even get myself to start
> looking at Django async stuff because of it.
>
> Given that, if nobody else can step forward to take over, I'll have to put
> those three projects (Channels, channels-redis, and Daphne) into an
> explicit maintenance mode where they only accept security requests via the
> normal security@ route, and start the process of retiring them as active
> Django projects, as I don't want to give the impression they're still
> maintained if they're not.
>
> (note: the asgiref project is still fine and should probably move out of
> Django to its own effort at some point giving the growing set of ASGI tools)
>
> If people are willing to take over maintenance, I'm happy to help explain
> some things but I don't have the bandwidth to bring someone completely up
> from scratch, so I can't help mentor someone who is totally new to
> maintaining open-source Python (sorry!).
>
> Once I recover a bit from the burnout I'll be able to come back and help
> with the really complex bugs; the main thing I need out of is the seemingly
> endless support requests and weird WebSocket client bugs.
>
> My personal deadline for this is two weeks, on February 1st. If you want
> to help out, please feel free to reply either here or get in touch with me
> personally to chat about what's involved.
>
> Andrew
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1upnXoLEBcbAuQLqxLTbvrdXzu0FQX4q4aREOQB-uxjurA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Potential suspension of Channels development

[Andrew Godwin]

On Mon, Jan 21, 2019 at 4:34 AM Michael Martinez <
writemichaelmarti...@gmail.com> wrote:

> Hi Andrew
>
> To me, Websockets is the defining use case for using Django Channels. From
> a user POV, saying that Channels is focused on the wrong problem
> (websockets) is like saying Django is too focused on HTTP.
>
> When I have selected Channels (vs other tools), my rationale was not:
>
> "*I need a general purpose async platform and it would be great if it
> worked with Websockets, ZeromQ and played nice with Django...*"
> (therefore Django Channels vs Tornado vs ...)
>
>
> rather my rationale is more like:
>
> "*I need to build real time features with Websockets using Django..*"
> (therefore Django Channels).
>
>
>

Oh, I totally get that, and Channels does well at providing WebSockets -
the problem is that it's still an area with a lot less interest and also
one I personally have no use for at the moment. Those things combined mean
that WebSockets is not something I'm really interested in supporting for
free right now; I'd have to be paid to work on it (as I was with the
Mozilla grant for a lot of Channels' development).

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uq6DHMbpLhyYJNmcWF-9GL_ebTkg3Kvm3V-L5_fG0JnjQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Potential suspension of Channels development

[Andrew Godwin]

On Sat, Jan 19, 2019 at 12:13 PM Carlton Gibson 
wrote:

> Hey Andrew.
>
> I've been thinking a lot about this. You clearly shouldn't be maintaining
> Channels single-handedly indefinitely.
>
> I know Channels started out separately, but, it's time to think about
> what, if anything of channels, is to be brought into core, or become THE
> WAY we do things or... Yes, we need more hands, but there's a bunch of
> people who can help out, and at least part of the problem is your out there
> in the wings by yourself (with not much visibility.)
>
> Q: How does Channels fit into the "Django Async Roadmap"?
>
> To the extent that it does, I think there's a case for asking the board
> and the DSF more widely to throw everything we've got behind making sure
> it's properly resourced.
>

The main problem with Channels, I feel, is that is solves the wrong problem
- it's focused on WebSockets, which is a niche feature that a few people
are happy we provide but most people have no practical use for.

Instead, I think Django should focus on a good async path for HTTP - views,
ORM, templates, and the like. This is what I want to get done if I can get
my time and energy back!


> I presume you DON'T offer support on the issue tracker, and point people
> to other channels? I'd be happy to "straight-bat" obvious tickets away.
> (Michael's offer to recieve is valuable there.)
>

I do try to do this - I wrote a standard page last year to help out with
batting away (https://channels.readthedocs.io/en/latest/support.html) - but
sometimes it's hard to triage the bug from the support request. Your help
there would be most appreciated, and yes, I should add Michael's offer to
that support page.


>
> "...and weird WebSocket client bugs."
>
> Can I ask you to explain what you mean there? (Point to a ticket maybe.)
> What kind of query do you get? Is there a particular knack to working out
> if it's a valid bug or not? (Or is it apparent?)
>

I don't want to single anyone out, as it's not their fault, but bugs like
this: https://github.com/django/daphne/issues/244

This person did the work and figured it out for the most part, but lots of
bugs that complex and confusing come in with just the first part of the
issue and I'm left wondering if it's their browser, their server, their
Python install or actually Channels.

At some point, I wake up every morning to 5-6 emails from the various
projects and it overwhelms, as I'm sure you've encountered. Even the bugs
are nasty enough that I don't feel like fixing them once I've figured out
they're actually bugs.


>
> Thanks for all you work here. Legend. 
>

And thank you for your response and help :)

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1upg1yzYmN2KA989PSUpoPNy-B2Pyo3FmQb9%3D5qA9K6i%2BA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Potential suspension of Channels development

[Andrew Godwin]

Hi Nasir,

I'm looking for help from people with a strong history of open source
maintenance and Django/Python/async contributions, since I can't really
guide people through it and I need to be able to trust those who I give
access to. If you think you meet that, email me personally and I'll outline
what's needed.

Andrew

On Thu, Jan 17, 2019 at 10:10 AM Nasir Hussain 
wrote:

> Hi andrew, I can help in maintaining the projects. Kindly let me know what
> are the next steps.
>
> Thanks
>
> On Thu, Jan 17, 2019, 11:07 PM Andrew Godwin 
>> Hi all,
>>
>> I'm writing to you all to update you on the current situation of Channels
>> and related libraries (channels-redis and Daphne) and potentially ask for
>> help.
>>
>> I've been the sole maintainer of these projects for quite a while and it
>> has become unsustainable - all of my energy is taken up fielding issues and
>> support requests and I haven't been able to even get myself to start
>> looking at Django async stuff because of it.
>>
>> Given that, if nobody else can step forward to take over, I'll have to
>> put those three projects (Channels, channels-redis, and Daphne) into an
>> explicit maintenance mode where they only accept security requests via the
>> normal security@ route, and start the process of retiring them as active
>> Django projects, as I don't want to give the impression they're still
>> maintained if they're not.
>>
>> (note: the asgiref project is still fine and should probably move out of
>> Django to its own effort at some point giving the growing set of ASGI tools)
>>
>> If people are willing to take over maintenance, I'm happy to help explain
>> some things but I don't have the bandwidth to bring someone completely up
>> from scratch, so I can't help mentor someone who is totally new to
>> maintaining open-source Python (sorry!).
>>
>> Once I recover a bit from the burnout I'll be able to come back and help
>> with the really complex bugs; the main thing I need out of is the seemingly
>> endless support requests and weird WebSocket client bugs.
>>
>> My personal deadline for this is two weeks, on February 1st. If you want
>> to help out, please feel free to reply either here or get in touch with me
>> personally to chat about what's involved.
>>
>> Andrew
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/CAFwN1uonedcUeLz8zD%2BK5Ma82gLyAX8g0s58HeT%3Dq-dMgcLxfw%40mail.gmail.com
>> <https://groups.google.com/d/msgid/django-developers/CAFwN1uonedcUeLz8zD%2BK5Ma82gLyAX8g0s58HeT%3Dq-dMgcLxfw%40mail.gmail.com?utm_medium=email_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAB5Hk3yC0MnbJwh%2BMQ63%2BdXha9pnBo6zL_kp2n5wNd8oqkT1UQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/django-developers/CAB5Hk3yC0MnbJwh%2BMQ63%2BdXha9pnBo6zL_kp2n5wNd8oqkT1UQ%40mail.gmail.com?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uoSvEVhffSH2sznNSpXfaraeq8eW9aPDjJkA%3D6rP-5JwQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Potential suspension of Channels development

[Andrew Godwin]

Hi all,

I'm writing to you all to update you on the current situation of Channels
and related libraries (channels-redis and Daphne) and potentially ask for
help.

I've been the sole maintainer of these projects for quite a while and it
has become unsustainable - all of my energy is taken up fielding issues and
support requests and I haven't been able to even get myself to start
looking at Django async stuff because of it.

Given that, if nobody else can step forward to take over, I'll have to put
those three projects (Channels, channels-redis, and Daphne) into an
explicit maintenance mode where they only accept security requests via the
normal security@ route, and start the process of retiring them as active
Django projects, as I don't want to give the impression they're still
maintained if they're not.

(note: the asgiref project is still fine and should probably move out of
Django to its own effort at some point giving the growing set of ASGI tools)

If people are willing to take over maintenance, I'm happy to help explain
some things but I don't have the bandwidth to bring someone completely up
from scratch, so I can't help mentor someone who is totally new to
maintaining open-source Python (sorry!).

Once I recover a bit from the burnout I'll be able to come back and help
with the really complex bugs; the main thing I need out of is the seemingly
endless support requests and weird WebSocket client bugs.

My personal deadline for this is two weeks, on February 1st. If you want to
help out, please feel free to reply either here or get in touch with me
personally to chat about what's involved.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uonedcUeLz8zD%2BK5Ma82gLyAX8g0s58HeT%3Dq-dMgcLxfw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Reactive frontend using Django templates?

[Andrew Godwin]

Hi Brylie,

There are no such efforts at the moment - Django is still focused on being
a backend language for the most part, and we don't really have any of the
pieces in place to do dynamic template rendering. We're more focused on
providing the fundamental components to let other people build that sort of
stuff.

Andrew

On Wed, Nov 21, 2018 at 11:48 AM Brylie Christopher Oxley 
wrote:

> Hello,
> I have been developing with Meteor.js for about four years now. One really
> nice aspect of Meteor is that it streams data to the client, which in turn
> updates the templates (e.g. adding chat messages to a list).
>
> When I look for such a solution with Django, it seems that Channels is a
> solution for the streaming data, but there doesn't seem to be a way to
> reactively update client code generated by Django templates without making
> another server request to render the HTML.
>
> I really like the Django template language, and don't want to switch away
> to a frontend framework, for several reasons. Are there any efforts to
> which I can contribute, with the goal of adding reactivity to client-side
> templates generated with the Django template language?
>
> Best regards,
> Brylie Christopher Oxley
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/82f35996-0663-43dc-951f-f2936670a37c%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uqJpCJeLUV-vksKFL5eiiWyBq%2BBxJxVD0sQzOqHNYim%3DQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Requiring sqlparse for sqlite introspection

[Andrew Godwin]

Adding sqlparse into the introspection code for SQLite specifically means
it's going to be a runtime dependency for anything to do with migrations.

I'm alright having that be the case, but I do think we should make sure the
tests run with SQLite as otherwise a large section of the most complicated
code in migrations won't be tested properly.

Andrew


On Mon, 8 Oct 2018, 00:59 Ian Foote,  wrote:

> Hi all,
>
> On my pull request (https://github.com/django/django/pull/10406)
> refactoring how Django creates database constraints I introduced a
> dependency on sqlparse in the sqlite introspection code. This allows Django
> to correctly read information about constraints on sqlite, particularly the
> name.
>
> When reviewing (
> https://github.com/django/django/pull/10406#issuecomment-424542217) Tim
> Graham raised the question of whether we should make sqlparse a mandatory
> requirement for the sqlite tests or if we should skip those tests that
> require it. In later discussion (
> https://github.com/django/django/pull/10406#issuecomment-427362983), Tim
> raised the point that the introspection code is used by migrations.
>
> I'm not clear myself what the best route forwards here is, so I'm bringing
> it to Django Developers for discussion (as Tim suggested).
>
> Thanks,
> Ian
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAFv-zfKadOeWit8M6GMmx4H2ChUCU6u%3DscHX8F7oBKJkHRbuVg%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1upormR5UjCNMU7xXQiqeLtU%2BP-qBQAdgLx%2Be2Jno%2ByBew%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Channels doesn’t conform to Django middleware specifications

[Andrew Godwin]

Just to say that I've reopened the original issue in question as it's clear
now this needs to be fixed. I won't be able to get to writing a fix for a
bit, though, so if someone else wants to they should.

Andrew

On Tue, Aug 28, 2018 at 5:41 AM John Obelenus  wrote:

> Just finished reading both bug reports and I see the file loader lookup in
> whitenoise. This explains a lot actually. We are still on Channels 1, and
> when we tried an upgrade to Channels 2, but encountered terrible
> performance at load and had to undo it.
>
> On Tuesday, August 28, 2018 at 8:00:17 AM UTC-4, John Obelenus wrote:
>>
>> Can I ask a dumb question? Why did you only discover this with adding
>> Whitenoise, and not the default/common middleware stack that comes in
>> django? Asking that question in another way: What is specific to whitenoise
>> that this constant loading of middleware causes such a perf hit? This
>> question in no way implies this should not be fixed, its just my curiosity
>> (I'm in the exact same situation of running channels and whitenoise)
>>
>> On Monday, August 27, 2018 at 12:46:15 PM UTC-4, Carl-Philip Majgaard
>> wrote:
>>>
>>> After some digging, I was able to find the reason for why adding the
>>> Whitenoise middleware to my Channels-enabled project was causing a massive
>>> performance drop.
>>>
>>>
>>> It appears that for every inbound HTTP request, Channels initializes a
>>> new AsgiHandler. In turn, this means that the entire middleware stack is
>>> being loaded for every HTTP request. See below:
>>>
>>>
>>> https://github.com/django/channels/blob/master/channels/routing.py#L45-L62
>>>
>>> https://github.com/django/channels/blob/master/channels/http.py#L157-L183
>>>
>>>
>>> This has two consequences:
>>>
>>>1. Channels is causing middleware to fall out of conformity with
>>>Django documentation, which states that middleware’s __init__() is only
>>>called *once*, on server startup.
>>>2. Any middleware which performs more-than-trivial work in its
>>>    __init__() causes massive latency on any incoming request.
>>>
>>>
>>> The issue was diagnosed in:
>>>
>>> https://github.com/django/channels/issues/1099
>>>
>>> https://github.com/django/channels/issues/1121
>>>
>>>
>>> In response to the diagnosis in django/channels #1121, Andrew Godwin
>>> says:
>>>
>>> Hmm, ASGI middleware is loaded every request (#1099
>>> <https://github.com/django/channels/issues/1099>) so it's quite
>>> possible I made the same mistake with the Django middleware as the handler
>>> is loaded every request too. If so, someone should be able to patch that
>>> quite easily.
>>>
>>>
>>> However, I feel that a patch to this issue would be non-trivial to
>>> create.
>>>
>>>
>>> If the AsgiHandler is refactored such that it receives *scope* upon
>>> __call__() rather than on __init__() (and is only initialized once)*, *it
>>> no longer conforms to the tests defined in
>>> https://github.com/django/asgiref/blob/master/asgiref/testing.py#L17,
>>> as they require passing scope for initialization. Thus, the AsgiHandler
>>> would no longer fit the shape of an Asgi Application.
>>>
>>>
>>> I would attempt a fix for this, but I am unsure of how to approach it
>>> given the constraints of the problem.
>>>
>>> Perhaps someone more knowledgeable about Channels could chime in with
>>> their $0.02? I would love to devote some time to fixing this, but need some
>>> input before I do.
>>>
>>>
>>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/998d7855-152e-4bc1-b2f2-8fd0f80099b4%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/998d7855-152e-4bc1-b2f2-8fd0f80099b4%40googlegroups.com?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uqyC6a1p0pKRv%2Brh9ZTwYyyu_Q4S7OgvSOfghdSdqCZng%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: django channels test authorization with live server

[Andrew Godwin]

Hi, this is the mailing list for the development of Django itself. I
suspect you will instead want to ask a question like this to the
django-users list!

Andrew

On Fri, Jul 13, 2018 at 5:37 AM Piotrek eF 
wrote:

> Hello,
>
> Normally, when I launch runserver, and login to admin site, I can open js
> console, connect to websocket server, and I'm authorized. I would like to
> realize such scenerio with pytest. Is there such possibility?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/ca28cce1-b683-4fc1-b067-ffae443b2548%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uoEKZD5YQuAQPJpsGZFmAb_UtonwsLrMA8%3D8L6hPV6PiA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: WebSocket connection failed: error during WebSocket handshake: Unexpected response code: 200

[Andrew Godwin]

Hi - this mailing list is for the development of Django itself. You already
posted on django-users (the right one) - please don’t double-post!

I’ll get around to replying to that other one in a day or two.

Andrew

On Sun, 24 Jun 2018 at 18:36,  wrote:

> Hi I am deploying a website with a live discussion feature. On the
> discussion page, people cannot send messages nor receive messages. In the
> console, it shows: "WebSocket connection to 'ws://xxx/room1/' failed:
> Error during WebSocket handshake: Unexpected response code: 200".
> I don't know where the websocket bug is (or maybe bugs from other
> sources?). Could you help me with that? Thanks!!!
> I am attaching some information that I think might be useful. If you think
> I miss anything please let me know and I will post it here!
>
>
> Some information:
> *- server: Ubuntu 14.04.3*
>
> *- Nginx*
> *- the command to run the website*
> uwsgi --socket experiment_platform.sock --module experiment_platform.wsgi
> --chmod-socket=666 --processes=6
> nohup daphne experiment_platform.asgi:channel_layer --port 8000 --bind
> 0.0.0.0 -v2 &
> nohup python manage.py runworker -v2 &
>
> *- the log information of uwsg [it keeps updating whether I was trying to
> send messages or not, trying to establish the connection (I guess?)]*
>  [pid: 29884|app: 0|req: 12/119] 100.15.133.125 () {52 vars in 1094
> bytes} [Mon Jun 25 01:10:57 2018] GET /forum/room1/ => generated 4792 bytes
> in 63 msecs (HTTP/1.1 200) 3 headers in 102 bytes (1 switches on core 0)
>  [pid: 29885|app: 0|req: 23/120] 100.15.133.125 () {52 vars in 1094
> bytes} [Mon Jun 25 01:11:32 2018] GET /forum/room1/ => generated 4792 bytes
> in 59 msecs (HTTP/1.1 200) 3 headers in 102 bytes (1 switches on core 0)
>
> *- the log information of runworker [it didn't update when I did the
> testing on the discussion page trying to send messages]*
>  2018-06-25 00:22:42,827 - INFO - runworker - Running worker against
> channel layer default (asgi_redis.core.RedisChannelLayer)
>  2018-06-25 00:22:42,828 - INFO - worker - Listening on channels
> http.request, websocket.connect, websocket.disconnect, websocket.receive
>
> *- snippets of settings.py*
> CHANNEL_LAYERS = {
> "default": {
> "BACKEND": "asgi_redis.RedisChannelLayer",
> "CONFIG": {
> "hosts": [os.environ.get('REDISTOGO_URL',
> 'redis://localhost:6379')],
> },
> "ROUTING": "experiment_platform.routing.channel_routing",
> #"ROUTING": "waiting_room.routing.channel_routing",
> },
> }
>
> CACHES = {
> "default": {
> "BACKEND": "django_redis.cache.RedisCache",
> "LOCATION": "redis://localhost:6379/1",
> "OPTIONS": {
> "CLIENT_CLASS": "django_redis.client.DefaultClient",
> }
> }
> }
>
> *- forum/consumers.py*
> import re
> import json
> import logging
> from channels import Group
> from channels.sessions import channel_session
> from .models import Discussion, Statement, Vote
> from chat.models import Room
> from experiment.models import ExpUser,TaskUser, Crowd
> from channels.auth import http_session, http_session_user,
> channel_session_user, channel_session_user_from_http
>
>
> log = logging.getLogger(__name__)
>
> @channel_session_user_from_http
> @channel_session
> def ws_connect(message):
> # Extract the discussion-label from the message. This expects
> message.path to be of the
> # form forum/{label}/, and finds a Discussion if the message path is
> applicable,
> # and if the Discussion exists. Otherwise, bails (meaning this is a
> some othersort
> # of websocket). So, this is effectively a version of
> _get_object_or_404.
> try:
> log.debug('In the try block of ws_connect')#added by me
> prefix, label =
> message['path'].decode('ascii').strip('/').split('/')
>
> if prefix != 'forum':
> log.debug('invalid ws path=%s', message['path'])
>
> return
> if prefix == 'forum':
> discussion = Discussion.objects.get(label=label)
>
> except ValueError:
> log.debug('invalid ws path=%s', message['path'])
> return
> except Discussion.DoesNotExist:
> log.debug('ws discussion does not exist label=%s', label)
> return
>
> if prefix == 'forum':
> log.debug('forum connect discussion=%s client=%s:%s',
> discussion.label, message['client'][0], message['client'][1])
> t = TaskUser(user=message.user,crowd=label,time_type='start')
> t.save()
> # Need to be explicit about the channel layer so that testability
> works
> # This may be a FIXME?
> Group('forum-'+label,
> channel_layer=message.channel_layer).add(message.reply_channel)
> message.channel_session['discussion'] = discussion.label
>
> message.reply_channel.send({
> 'accept': True
> })
>
>
> @channel_session_user
> @channel_session
> def ws_receive(message):
> if 'discussion' in 

Re: A Django Async Roadmap

[Andrew Godwin]

On Fri, Jun 8, 2018 at 11:31 PM Josh Smeaton  wrote:

>
> The only real concern I have at the moment is around your comments
> regarding on demand foreign key traversal. If existing code running
> synchronously is going to be impacted, that's going to be very difficult
> for a lot of people. If it's only asynchronous traversal that'll have
> issues, then I have no real concern, as on demand foreign key fetching is
> usually a bug anyway.
>

It would only be for asynchronous contexts - it would still work exactly as
today for synchronous ones. Backwards compatability is VERY important to me
during this whole thing.


>
> Having a brief read through the psycopg asynchronous docs[0], it looks
> like a number of features will be impossible or troublesome to use, like
> transactions, executemany, and named cursors (.iterator() with server side
> cursors). We'd also need to investigate how pgbouncer would work in async
> mode, as most large sites using postgres are also using pgbouncer. I would
> expect support can only further improve, especially if there is a driver
> like django pushing. Fallback would just be to run inside a thread pool
> though, so it's not a blocker for the rest of the proposal.
>
>
Yes, there's a lot of specifics that need to be worked out for each driver
(e.g. does it even make sense with SQLite at all?), but I think providing
that space and framework to experiment in is going to be very beneficial.
The aiomysql package implemented everything in pure Python, as well - it's
possible that similar would be needed for full async PostgreSQL support,
where the non-compiled speed reduction might be outweighed by the
parallelism improvements.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1upvo5suu_7FQXi6cWDau1xK04NdQG2m-sbuX%3D26k1Xoew%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: A Django Async Roadmap

[Andrew Godwin]

Chaim: For references to get up to speed on async concepts, there's not a
lot of great beginner-level stuff out there. I have a practical guide to
async functionality at
https://www.aeracode.org/2018/02/19/python-async-simplified/, but it's not
great at going into detail on Web stuff and why it has big advantages. I'd
love if other people could chime in with resources that helped them (rather
than me, who just started writing async code for fun 8 years ago and have
sort of kept up with things that way)

Tom F:

>  I’m just spitballing here but could we not conditionally return a
future/promise if some conditions are met, then implement some kind of
chaining based on the attribute accesses?

We could, but as you can see from the length of the paragraph you've
written this is incredibly _tricky_ to do correctly, and I'm not a fan of
lazy objects in the first place. Something we should investigate maybe, but
ultimately I'd prefer a simpler solution with less magic.

>  If we go with a `_async` suffix we could reduce a lot of boilerplate
with a bit of magic

Sure, though honestly I wouldn't say it needs magic, just:

def function_async(self, foo):
   ...

function = async_to_sync(function_async)

Tom C:

> Supporting an ASGI/asyncio request-response cycle in 2.2 might be a
perfectly achievable goal with or without funding. Users wouldn't be able
to  use the ORM, or much of the default middleware, but for some
gateway-type services I can still see it being useful.

Indeed. The time I personally spend each week on Channels (around 2 days
fulltime-equivalent) could easily be partially or wholly diverted into this
project, as Channels is pretty stable right now.

That said, my time for that project has been recompensed by the Mozilla OSS
grant they gave us a while back up until now - the funds for that are
finally drying out, and while I could do the work for free, I take a paycut
from my normal work to get the time that I use to work on Django, so
funding would be nice to have - and I don't expect anyone else to work for
free. I suspect the funding level needed would not be too great for the
initial work.

Given we have some MOSS funds left, I may use the last of those to start
getting an async pathway ready for 2.2 once this whole thing gets approved.
It's still within the scope of the original grant.

Jordan:

I'm not sure how we're going to develop this yet, but it'll likely be a
branch in the main Django repo once this proposal is approved (there's
still a little while before that happens - there needs to be consensus here
and/or a technical board vote).

Andrew


On Tue, Jun 5, 2018 at 2:21 PM C. Kirby  wrote:

> Andrew,
>
> I don't know enough about async web to comment directly on your proposal,
> but like most of your proposals it looks well thought out and
> comprehensive. Kudos to you.
>
> As you state, the scope of this project is one that can raise the profile
> of developers in the community - that appeals to me in ways that bug fixing
> has not, mostly because I have not encountered bugs in django.
>
> Towards that end, can you point to any of your preferred references to get
> up to speed on async concepts to help understand the scope and scale of
> this undertaking? Even if I don't work on the project I think intuitively
> that I could make good use of it.
>
> Thanks,
> Chaim
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/7a958407-7dcf-4fd8-b09f-67e28346cb65%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1upLVX6hsH_tgooRUgh9Mtvn%2Bqcn9_9_S1pKOAQ44S-PyQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: A Django Async Roadmap

[Andrew Godwin]

>
>
> I think getting rid of the related field references could be a big issue
> here and cause a lot of headaches for existing applications. Could we do
> this in a backwards compatible way at all? I wonder if PEP 567
>  could help here, could we
> define some kind of ‘Django async context’ and do different things
> depending on if this is true or false?
>
Unfortunately even if you have context variables, you simply can't await
inside of an attribute reference because there's a synchronous call in your
stack. I even chatted to some Python core devs at PyCon US about this and
we couldn't really think of a way out of this problem without some very
serious changes to the language.

What will work, though, is attribute access from a sync context - the
related field references will only error out of they know they're in an
async loop, and we can detect that by looking for an active event loop on
the current thread (no context varaible needed).


> Regarding exposing an async interface alongside a synchronous one: are you
> envisaging something like appending _async to methods or having some kind
> of wrapper class that could be optionally included to go from async->sync?
> I guess it would have to be appending _async, as a wrapper class could be
> used in different contexts.
>

I'm not quite sure on this one. For backwards compatibility, we have to
keep sync methods working with the same name, but there are several options
as to how to separate them - keyword arguments (like "my_method(...,
async=True)"), suffixes ("my_method_async()", as you suggest),
automatically changing based on if you're in an async thread (which is a
bit... magic) or different module namespaces ("from django.db.async import
foo").

I want to sketch out what all of these look like as part of this project
and then work out which is best for Django.


> Async templates seem particularly powerful if we work out the details. We
> could eventually render different parts of the template concurrently, i.e
> example each iteration of a for loop could be it’s own future resolved
> independently, but this is likely a pipe dream.
>

That would very much be a long-term thing, and honestly something I might
consider handing off to something like Jinja (which already has full async
support, I learnt yesterday)

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1upPVQCjHTxgJmzJSUzaevsfQ7evwi89hFt4OvHmMe5m2A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

A Django Async Roadmap

[Andrew Godwin]

Hello everyone,

For a while now I have been working on potential plans for making Django
async-capable, and I finally have a plan I am reasonably happy with and
which I think we can actually do.

This proposed roadmap, in its great length, is here:

https://www.aeracode.org/2018/06/04/django-async-roadmap/

I'd like to invite discussion on this potential plan - including:

 - Do we think async is worth going after? Note that this is just async
HTTP capability, not WebSockets (that would remain in Channels)

 - Can we do this in a reasonable timeframe? If not, is there a way around
that?

 - Are the proposed modifications to how Django runs sensible?

 - How should we fund this?

There's many more potential questions, and I really would love feedback on
this. I'm personally pretty convinced that we can and should do this, but
this is a decision we cannot take lightly, and I would love to hear what
you have to say.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1up4%2BP%2B4bCSSiG6t845idoM8dvktnuTs4dH2R5aAuCz3iA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Channels - Groups and discovery

[Andrew Godwin]

The code that tries to find a handler is just the dispatch method, so it's
actually quite easy to override even at a user level rather than in the
core itself:
https://github.com/django/channels/blob/master/channels/consumer.py#L61

I would like to see a stronger solution for interfaces and type assertions,
but I also think this is something that could be done externally quite
easily (in fact we have a library we use at work for this very thing for
our redis-based message bus: https://github.com/eventbrite/conformity). The
channel layers being just a dumb dictionary transport is quite a nice
facade in terms of keeping them low maintenance, and also in letting people
solve dependency/versioning/schema upgrade issues in the way that fits them
best.

Andrew

On Sun, Mar 25, 2018 at 4:08 PM, Josh Smeaton <josh.smea...@gmail.com>
wrote:

> I see - some kind of python type that can be shared amongst consumers
> might make sense at the application level, but it doesn't at the protocol
> level. Specific patterns may emerge as application level helpers like
> Consumer subclasses that mixin specific Groups, and ASGI won't need to have
> any knowledge of these patterns.
>
> > There's also the idea of adding an easy way to ignore messages you're
> not interested in if you're using groups as a firehose rather than targeted
> broadcast
>
> Something like this would probably solve the biggest issue. At an
> application level I'd imagine it to be something along the lines of
> try_get_handler(message_type, consumer, 
> ignore_missing=consumer.ignore_missing),
> but no idea if that's something that could be permitted by the
> spec/protocol. I'll familiarise myself with the spec and the implementation
> so I can understand what might and might not be possible at each layer
> before proposing any specific ideas.
>
> There's some fantastic stuff in channels though, and was fairly easy to
> hook things together after reading the docs. I'm looking forward to using
> it more.
>
> On Sunday, 25 March 2018 02:24:00 UTC+11, Andrew Godwin wrote:
>>
>> Nobody else has suggested this particular approach yet, and while it
>> would definitely make writing applications much more reliable, there's no
>> particularly easy way to distribute an interface that I can think of (even
>> the ASGI specification ends up being enforced on the receiver side, though
>> ultimately it's the conformance test suites that I rely on for safety).
>>
>> Without a simple solution to that I think it's more the sort of thing I
>> would like to make sure people can hook into themselves to provide
>> validation rather than trying to ship something in the core release.
>>
>> There's also the idea of adding an easy way to ignore messages you're not
>> interested in if you're using groups as a firehose rather than targeted
>> broadcast - that does mean silent failures, though, which is my
>> least-favourite kind. At least with the current design pattern something
>> will complain if it's not working right (even though that thing is on the
>> receiving side not the sending side).
>>
>> Andrew
>>
>> On Sat, Mar 24, 2018 at 4:31 AM, Josh Smeaton <josh.s...@gmail.com>
>> wrote:
>>
>>> I've finally had the chance to use channels for a project (hack day
>>> multiplayer game - hope to release and blog about it some time soon), and I
>>> wanted to document some of the rough edges I hit and ask some questions
>>> about them.
>>>
>>> Specifically though, I find the mapping of payload.type to a method on
>>> the consumer confusing and somewhat brittle.
>>>
>>> The design we went with was to have a PlayerConsumer(AsyncWebsocketConsumer)
>>> and a GameConsumer(SyncConsumer) running as a worker. The GameConsumer
>>> starts the engine in a thread, and lets the engine fetch the channel layer.
>>> The player and engine then communicate like so:
>>>
>>> # PlayerConsumer: receive game state updates
>>> await self.channel_layer.group_add(
>>> self.group_name,
>>> self.channel_name
>>> )
>>>
>>> # PlayerConsumer: publish player joining event to game
>>> await self.channel_layer.send(
>>> 'game_engine',
>>> {
>>> 'type': 'player.new',
>>> 'player': self.user.email,
>>> 'channel': self.channel_name,
>>> }
>>> )
>>>
>>>
>>> # GameConsumer: publish state update
>>> async_to_sync(self.channel_layer.group_send)(
>>> self.group_name,
>>> {
>>> 'type': 'game_update',
>>> 'state': state_json,
>>> }
>>> )
>>>
>

Re: Channels - Groups and discovery

[Andrew Godwin]

Nobody else has suggested this particular approach yet, and while it would
definitely make writing applications much more reliable, there's no
particularly easy way to distribute an interface that I can think of (even
the ASGI specification ends up being enforced on the receiver side, though
ultimately it's the conformance test suites that I rely on for safety).

Without a simple solution to that I think it's more the sort of thing I
would like to make sure people can hook into themselves to provide
validation rather than trying to ship something in the core release.

There's also the idea of adding an easy way to ignore messages you're not
interested in if you're using groups as a firehose rather than targeted
broadcast - that does mean silent failures, though, which is my
least-favourite kind. At least with the current design pattern something
will complain if it's not working right (even though that thing is on the
receiving side not the sending side).

Andrew

On Sat, Mar 24, 2018 at 4:31 AM, Josh Smeaton 
wrote:

> I've finally had the chance to use channels for a project (hack day
> multiplayer game - hope to release and blog about it some time soon), and I
> wanted to document some of the rough edges I hit and ask some questions
> about them.
>
> Specifically though, I find the mapping of payload.type to a method on the
> consumer confusing and somewhat brittle.
>
> The design we went with was to have a PlayerConsumer(AsyncWebsocketConsumer)
> and a GameConsumer(SyncConsumer) running as a worker. The GameConsumer
> starts the engine in a thread, and lets the engine fetch the channel layer.
> The player and engine then communicate like so:
>
> # PlayerConsumer: receive game state updates
> await self.channel_layer.group_add(
> self.group_name,
> self.channel_name
> )
>
> # PlayerConsumer: publish player joining event to game
> await self.channel_layer.send(
> 'game_engine',
> {
> 'type': 'player.new',
> 'player': self.user.email,
> 'channel': self.channel_name,
> }
> )
>
>
> # GameConsumer: publish state update
> async_to_sync(self.channel_layer.group_send)(
> self.group_name,
> {
> 'type': 'game_update',
> 'state': state_json,
> }
> )
>
> This works, provided PlayerConsumer has a method:
>
> async def game_update(self, event):
>
> And the GameConsumer has a method:
>
> def player_new(self, event):
>
> But if these two consumers are in completely different code
> bases/packages, there is no real way to know what the interface is between
> these consumers. Worse, it's extremely easy for a bad actor to crash a
> listening consumer. Either of the following events will crash the consumer
> receiving the message:
>
> await self.channel_layer.send(
> 'game_engine',
> {
> 'type': 'i_am_a_bad_consumer'
> }
> )
>
> async_to_sync(self.channel_layer.group_send)(
> self.group_name,
> {
> 'type': 'gme_update', # typo
> 'msg': 'hi',
> }
> )
>
> I'm not so concerned about arbitrary asgi applications gatecrashing my app
> as I ultimately have control over what is going to run. But each consumer
> participating in a group must support the superset of all message types
> that may be sent to that group if it wants to avoid crashing. And it has to
> support the superset of message types without being able to discover what
> they might be. Oh, and they're strings that **might** match a method I have
> if periods are converted to underscores.
>
> This is all very good for adhoc eventing, but I think at some point you're
> going to want to publish an explicit Interface. A GroupInterface might
> define a collection of message types that are valid for members
> participating in that group, and then a consumer could subscribe to
> **that** rather than just a name. The interface members might be optional
> or required, I haven't thought that far ahead, but at least each member
> would be known. Attempting to publish an unknown message to a group would
> crash the **sender** rather than the **receiver**.
>
> Has anything like this come up before?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/939460e7-45da-4dd0-afb7-
> f386274faea1%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and 

Re: Threads and requests with Channels 2.0

[Andrew Godwin]

Hi Jonathan,

Daphne runs a single asyncio event loop in the main thread which services
all protocol handling (low-level HTTP and websocket) and Channels-level
async-native code (ASGI middleware, routing, async consumers). Any
synchronous calls (e.g. methods on synchronous consumers) are queued up to
run in a threadpool with NUM_CPUS * 5 threads by default.

It's quite different from the thread-handling-one-request model. This isn't
the right venue for a complete explanation of how Python async works in
combination with the GIL, unfortunately - hopefully you can find that
elsewhere if you are interested.

Andrew

On Tue, Mar 20, 2018 at 2:30 PM, Jonathan Stray 
wrote:

> Can anyone offer a brief account of how Channels uses threads to service
> multiple requests?
>
> My understanding is that there are multiple workers, and I imagine each is
> a thread which can handle one request (Django view-style request) at a
> time. Is this correct? I also don't quite understand how this is possible
> in one Python process given the Global Interpreter Lock.
>
> Basically I am trying to reason about race conditions and scaling, and I
> don't have enough understanding to do it.
>
> Apologies if this is document elsewhere, but I've so far not been able to
> find it.
>
>   - Jonathan
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/b2da063e-0fa0-4f35-a86d-
> 536fe210f1b2%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1ureEnSoCYFfON%3DpbWh530GiW4o6NVbS8ugnwPnMOceuMA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: maximum number of connection using django-channels

[Andrew Godwin]

Hi John,

It's impossible to give any accurate estimate of performance numbers as it
depends so heavily on hardware. Instead, I can point out to you likely
bottlenecks:

- Each instance of Daphne can only terminate so many connections. I would
hope this is above 1000 on any reasonable machine.
- Each instance can only handle so many messages in or out per second. This
is a function of how many connections you have combined with how chatty
they are.
- Sending to groups currently scales linearly with group size, so large
groups are slower to send to

Ultimately, you'll have to do your own performance measurements with your
environment and your code to work out what you can do. I would say,
however, don't run "runserver" in production!

Andrew

On Mon, Mar 19, 2018 at 10:54 AM, John Carrell <carrell.j...@gmail.com>
wrote:

> Can I please ask a clarifying question?
>
> You examples above ("if you had 1 clients...") spoke to clients
> sending the messages. I have a use case where the clients are never (at the
> moment) sending messages but rather listening for messages from the server.
> When the server has something to say it does so via Groups. Each client
> will be subscribed to 3 different Consumers. It is likely that there will
> be as many as 30 - 60 total clients each on a Group with no more than a few
> (10s at the very most extreme) other reply_channels.
>
> Can you comment on the limitations of channels in this scenario?
>
> Fair warning: We are currently running Django as a development server in
> "production" (It's an internal tool.) so we're not manually using Daphne
> and a standalone server and WSGI, etc. It's just a single instance of
> "runserver". I'm working on convincing our team that we should change our
> deployment model from its initial prototype but there is not a lot of
> perceived value in that effort at the moment.
>
> Thanks!
>
>
> On Thursday, March 2, 2017 at 11:26:18 AM UTC-6, Andrew Godwin wrote:
>>
>> Hi,
>>
>> There's more about how Channels works in the docs and in talks I've
>> given, but the short version is:
>>
>> * Websockets go into a server called Daphne, which is written in Twisted
>> and so can handle hundreds or potentially thousands of simultaneous
>> connections open at once
>> * Any event on a websocket (connect, message received) is sent onto the
>> channel layer, which is a type of first-in-first-out queue
>> * Django worker processes synchronously take a message from the queue,
>> process it, and then loop back to take another
>>
>> Thus, there are two factors to scaling:
>> * The number of open connections affects how many Daphne instances you run
>> * The throughput of events affects the number of workers you run
>>
>> For example, if you had 1 clients connecting but sending one message
>> every few minutes, you'd want 10 Daphne instances but only a few workers.
>> If you had 100 clients sending 10 messages a second, you'd only need one
>> Daphne instances but lots of workers.
>>
>> The number of connections Daphne can take depends on a combination of CPU
>> speed and kernel limits; it's not going to be RAM-limited except on very
>> low memory systems.
>>
>> Hope that helps.
>>
>> Andrew
>>
>> On Wed, Mar 1, 2017 at 10:01 PM, Gopal <gopalm...@gmail.com> wrote:
>>
>>> Hello everyone,
>>> I am using django=1.10.5 and channels==1.0.3 in my project.
>>> First i will give you little description about my project so that you
>>> all can understand my question properly.
>>> I am getting latitude and longitude of android device(first user) using
>>> websockes. So that android application is connected to my django
>>> server(second user) through websocket (implemented using channels), and i
>>> am storing all latitude of longitude of all connected android devices using
>>> that app. After that i am sending all those locations to web browsers(third
>>> user). So basically i am taking location from android devices (or from
>>> first user), storing into server's database (or to second user) and showing
>>> those location to web browser (or to third user). They all are
>>> communicating using websocket. Now my question is how many clients or user
>>> can connect to that websocket and does number of connections depend upon
>>> free ports or does it depend on RAM and second question is that can some
>>> one explain in detail from OS perspective that how does django-channel
>>> works ???
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>

Re: Thoughts on diff-based migrations with django?

[Andrew Godwin]

Hi,

This was an approach we deliberately avoided during development of the
current migration system - it works poorly with diverging feature branches
and requires that your developers have access to production schema at all
times (and additionally that you don't have divergent production/staging/QA
deploys, like many environments do).

I think it would be a fine addition as an optional third-party tool, but
it's not something that really makes sense to include in Django at this
point I think.

Andrew

On Mon, Feb 19, 2018 at 10:05 PM, djrobstep  wrote:

> Hi all,
>
> Some thoughts on schema migrations that may interest you. By way of
> background, I'm the author of migra (https://github.com/djrobstep/migra),
> a schema comparison/diff tool for Postgres.
>
> I initially wrote this tool because I wanted to be able to generate
> migration scripts automatically, without needing historical migration files.
>
> It means that you can do things like sync your database to your models
> mostly automatically, and also explicitly test for a matching schema.
> Instead of a chain of migration files you only ever need to keep track of
> one, containing any pending changes).
>
> I've used this approach with success on a number of projects now, and it
> seems to work pretty well. I also talked about this approach to migrations
> at the most recent PostgresOpen (https://www.youtube.com/
> watch?v=xr498W8oMRo).
>
> The downside of this approach is that it's slightly more operationally
> complex (you need have a temporary copy of your production schema available
> for comparison purposes, typically a --schema-only dump file) and also that
> it's postgres-only at this point. It's also conceptually quite different to
> the typical Django/Rails style of migrations which represents a barrier to
> entry.
>
> In spite of this, I think doing migrations this way has some compelling
> advantages, that are relevant to users of Django.
>
> I'm interested in your opinions on this topic as Django developers. What
> do you think? Is it something worth offering as an option for managing
> Django migrations? If your feelings are positive, I'm happy to take a look
> at extending the code accordingly.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/5f84157b-071e-42c3-90dd-
> 6f9e3df48ab3%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1urzK%3DwWT0SfS32sg_YhcdJnG_2h9Cbp8u2Uouzs6LaWFg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: binding framework third-party package for Channels 2

[Andrew Godwin]

Hi Daniel,

It was something I was going to do myself once I had the time and the
release support for 2.0 dies down, but if you want to take a crack at it,
please do (I am more than happy to consult where I can).

Andrew

On Sat, Feb 17, 2018 at 1:13 PM, Daniel Gilge  wrote:

> Hi,
>
> I'm not sure where to ask this question because it doesn't seem to fit
> anywhere but I hope this group is the best place.
>
> The Channels 2.0 docs
> 
> say that the binding framework will appear in a third-party
> (non-official-Django) package. What are the plans here? Is there a special
> person/organization in charge? Is there a schedule?
>
> I'm asking because I'll need some kind of bindings and might do it myself
> otherwise.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/dcb7fe1f-c351-4707-9629-
> 7ce013707f0a%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1up6dkVpPs4gSTpC13bNjTLU3SkZK3KrSE0qTWgaj5xomg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Adding a tutorial for Channels

[Andrew Godwin]

I would very much welcome a narrative tutorial like the Django docs - I did
not have the time to personally write and QA one before the 2.0 release,
unfortunately. I'm happy to help out if you want to email me personally to
discuss!

Andrew

On Sun, Feb 4, 2018 at 10:39 PM, David Foster  wrote:

> This weekend I spent several hours getting Channels configured to run a
> simple chat server .
> This was an experiment I wanted to do before trying to integrate Channels
> into a much larger project.
>
> It would have taken me a lot less time if there had been a tutorial in the 
> official
> Channels docs  similar
> in style to the Django tutorial
>  that showed how
> to configure Channels and use it to work with WebSockets while also
> touching on the important Channels concepts and linking to reference
> documentation where appropriate. So I'd like to offer to write such a
> tutorial.
>
> Would this be a welcome contribution?
>
> - David
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/9b68e3cf-76de-4506-9733-
> 00b2d640b1e2%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1upcygqRNkzKmcWXnrxJVm5yR--FB8%2BctqZAmHVee5T0tQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Default to BigAutoField

[Andrew Godwin]

On Fri, Aug 18, 2017 at 5:43 AM, Markus Holtermann <i...@markusholtermann.eu
> wrote:
>
> I'm don't fully agree with the approach. This essentially forces 3rd
> party package authors to make the call about the primary key field size.
> While for small to medium size projects BigAutoField is unlikely
> required and only comes with additional (storage) costs. Given that the
> migrations would need to be part of the 3rd party package there's also
> no (trivial) way for project developers to force or change to
> SmallAutoField for those packages. The same thing holds the other way
> round.
>
> Unfortunately, I don't have another solution at hand.
>
>
This is also true of changing the primary key of third-party packages in
general though - e.g. there's no way I can make everything use UUIDs even
if my database would be way better at those.

I don't see any other solutions that aren't settings doing
spooky-action-at-a-distance to primary keys, and that's something I really
don't want to see.

Andrew


>
> On Thu, Aug 17, 2017 at 02:43:07PM -0700, Andrew Godwin wrote:
>
>> To elaborate on the solution we eventually came up with - we default
>> models
>> to use a new BigAutoField that migrations will pick up on and generate
>> migrations to alter columns to, but for safety reasons for those that
>> don't
>> read release notes, made the migration autodetector ask you if you want to
>> make these migrations with a slowness warning.
>>
>> It also tells you how to preserve the old behaviour and avoid new
>> migrations if you wish (manually set id = SmallAutoField)
>>
>> I like this approach as it means no new settings or Meta options or
>> anything, has a reasonable upgrade path, and won't let people unwittingly
>> wander into giant changes. The downside is that it does add slightly more
>> friction to the upgrade process.
>>
>> Andrew
>>
>> On Thu, Aug 17, 2017 at 2:36 PM, Kenneth Reitz <m...@kennethreitz.org>
>> wrote:
>>
>> I have opened a pull request:
>>>
>>> https://github.com/django/django/pull/8924
>>>
>>> Andrew and I came up with a good solution for migrations, together at
>>> DjangoCon.
>>>
>>> On Wednesday, June 14, 2017 at 7:36:36 AM UTC-7, Melvyn Sopacua wrote:
>>>
>>>>
>>>> On Friday 09 June 2017 15:59:50 Kenneth Reitz wrote:
>>>>
>>>> > However, it should also be noted that those same larger applications
>>>>
>>>> > are the ones that are likely to run into this problem eventually, so
>>>>
>>>> > perhaps forcing the migration is the best path moving forward.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Existing models are the problem. Then again the database knows the
>>>> truth.
>>>> So with a little inspection during apps.get_models we might be able to
>>>> do
>>>> the right thing and even allow migrating in steps.
>>>>
>>>>
>>>>
>>>> Apps is also the place to mark an app as migrated.
>>>>
>>>>
>>>>
>>>> In fact - couldn't an AppConfig grow a method "get_autoid_type()" and
>>>> inject the right one?
>>>>
>>>>
>>>>
>>>> You asked fr thoughts, so there's my 2c stream.
>>>>
>>>> --
>>>>
>>>> Melvyn Sopacua
>>>>
>>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to django-developers+unsubscr...@googlegroups.com.
>>> To post to this group, send email to django-developers@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/django-developers/e3effc41-10e1-42e2-9037-
>>> 84c98217cd91%40googlegroups.com
>>> <https://groups.google.com/d/msgid/django-developers/e3effc4
>>> 1-10e1-42e2-9037-84c98217cd91%40googlegroups.com?utm_medium=
>>> email_source=footer>
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers  (Contributions to Django itself)" group.
>> To unsubscribe

Re: Default to BigAutoField

[Andrew Godwin]

To elaborate on the solution we eventually came up with - we default models
to use a new BigAutoField that migrations will pick up on and generate
migrations to alter columns to, but for safety reasons for those that don't
read release notes, made the migration autodetector ask you if you want to
make these migrations with a slowness warning.

It also tells you how to preserve the old behaviour and avoid new
migrations if you wish (manually set id = SmallAutoField)

I like this approach as it means no new settings or Meta options or
anything, has a reasonable upgrade path, and won't let people unwittingly
wander into giant changes. The downside is that it does add slightly more
friction to the upgrade process.

Andrew

On Thu, Aug 17, 2017 at 2:36 PM, Kenneth Reitz  wrote:

> I have opened a pull request:
>
> https://github.com/django/django/pull/8924
>
> Andrew and I came up with a good solution for migrations, together at
> DjangoCon.
>
> On Wednesday, June 14, 2017 at 7:36:36 AM UTC-7, Melvyn Sopacua wrote:
>>
>> On Friday 09 June 2017 15:59:50 Kenneth Reitz wrote:
>>
>> > However, it should also be noted that those same larger applications
>>
>> > are the ones that are likely to run into this problem eventually, so
>>
>> > perhaps forcing the migration is the best path moving forward.
>>
>>
>>
>>
>>
>> Existing models are the problem. Then again the database knows the truth.
>> So with a little inspection during apps.get_models we might be able to do
>> the right thing and even allow migrating in steps.
>>
>>
>>
>> Apps is also the place to mark an app as migrated.
>>
>>
>>
>> In fact - couldn't an AppConfig grow a method "get_autoid_type()" and
>> inject the right one?
>>
>>
>>
>> You asked fr thoughts, so there's my 2c stream.
>>
>> --
>>
>> Melvyn Sopacua
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/e3effc41-10e1-42e2-9037-
> 84c98217cd91%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uo4Y_pWSf3zAe_4R0GGkDqBv1YGus8Q%2BWPPZ%3DZ6FPwdYQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.