Re: Automatic prefetching in querysets

2017-08-16 Thread Josh Smeaton
> > It won't affect experienced users. They'll read the release notes, see > that this change has been implemented, and either go and delete a bunch of > prefetch_related() calls, grumble a bit and turn auto-prefetch off globally > or just file it away as another fact they know about the Django

Re: Automatic prefetching in querysets

2017-08-16 Thread Adam Johnson
> > I wouldn't want is to give free optimisations to non-ORM pros > Something like 99% of django projects are by non-ORM pros, it can be easy to forget that on a mailing list of experts. On 16 August 2017 at 12:27, Josh Smeaton wrote: > It won't affect experienced users. They'll read the releas

Re: Automatic prefetching in querysets

2017-08-16 Thread Anssi Kääriäinen
On Wednesday, August 16, 2017 at 2:27:11 PM UTC+3, Josh Smeaton wrote: > > It won't affect experienced users. They'll read the release notes, see >> that this change has been implemented, and either go and delete a bunch of >> prefetch_related() calls, grumble a bit and turn auto-prefetch off glo

Re: Automatic prefetching in querysets

2017-08-16 Thread Luke Plant
Hi Josh, On 16/08/17 02:26, Josh Smeaton wrote: I believe we should be optimising for the **common** use case, not expecting everyone to be experts with the ORM. > If I can come up with a single example where it would significantly decrease performance (either memory usage or speed) compared

Re: Should django File wrapper support .next()?

2017-08-16 Thread Cristiano Coelho
I forgot about django dropping support for Python2 so I guess you are right this doesn't make much sense if next() is removed, but for Python2 it would make sense for django's File object to have the same api as the object returned by open(). El martes, 15 de agosto de 2017, 19:21:08 (UTC-3),

Re: Automatic prefetching in querysets

2017-08-16 Thread Marc Tamlyn
As an opt in feature, it does sound quite interesting. If over the course of a few releases, it proves to be reliable and we don't get tons of support requests of it either not working, or causing the opposite problem, we can consider moving it to an opt out feature. This makes it easy for people t

Re: Automatic prefetching in querysets

2017-08-16 Thread Tom Forbes
Some quicker changes that have been brought up here could be implemented in the interim, like adding intelligent warnings. Someone brought up a great point about how the ORM hides most SQL terminology from the user, but still requires the knowledge of the difference between prefetch and select rel

Re: Automatic prefetching in querysets

2017-08-16 Thread Gordon Wrigley
> It's also likely to require a significant patch The guts of it are actually surprisingly clean and small. That's probably a testament to some of the good decisions in the ORM architecture. Covering one2one in both directions, FK's in the forward direction, ignoring tests and various optin/out op

Re: Automatic prefetching in querysets

2017-08-16 Thread 'Tom Evans' via Django developers (Contributions to Django itself)
Is this opt-{in,out} considered to be a global flag, meant to be toggled on or off depending on whether it is an "expert" working on the project or not? I don't think that would be a good idea, almost all of our projects have a mix of skill levels, and people move from team to team on a regular ba

Re: Automatic prefetching in querysets

2017-08-16 Thread Aymeric Augustin
> On 15 Aug 2017, at 11:44, Gordon Wrigley wrote: > > I'd like to discuss automatic prefetching in querysets. Specifically > automatically doing prefetch_related where needed without the user having to > request it. Hello, I'm rather sympathetic to this proposal. Figuring out N + 1 problems

Re: Automatic prefetching in querysets

2017-08-16 Thread Gordon Wrigley
Regarding 2, it does work for reverse one-to-one relations. On Wed, Aug 16, 2017 at 9:17 PM, Aymeric Augustin < aymeric.augus...@polytechnique.org> wrote: > On 15 Aug 2017, at 11:44, Gordon Wrigley wrote: > > I'd like to discuss automatic prefetching in querysets. Specifically > automatically do

Re: Automatic prefetching in querysets

2017-08-16 Thread Shai Berger
First of all, I think making the auto-prefetch feature available in some form is a great idea. As an opt-in, I would have made use of it on several occasions, and cut days of optimization work to minutes. It's true that this wouldn't achieve the best possible optimization in many cases, but it w

Re: Automatic prefetching in querysets

2017-08-16 Thread Josh Smeaton
I think there's a lot right with your suggestions here Shai. It delivers better default behaviour for new projects, does not affect existing deployments, and seems pretty easy to enable/disable selectively at any level of the stack. My only concern would be libraries leaning on this behaviour

Re: Automatic prefetching in querysets

2017-08-16 Thread Gordon Wrigley
I'm not advocating either way on this, but it's probably worth noting that the context manager proposal is compatible with a queryset level optin/out and an object level optout. So you could for example have prefetch_related(auto=None) which can be explicitly set to True / False and takes it's val