I'd like to discuss automatic prefetching in querysets. Specifically
automatically doing prefetch_related where needed without the user having
to request it.
For context consider these three snippets using the Question & Choice
models from the tutorial
<https://docs.djangoproject
My existing code for this is now available as a pypi package
https://github.com/tolomea/django-auto-prefetch
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receivin
which attempt similar magical optimisations?
How do they go about identifying the cases where it is necessary?
On 15 August 2017 at 10:44, Gordon Wrigley wrote:
> I'd like to discuss automatic prefetching in querysets. Specifically
> automatically doing prefetch_related where needed
u know of any other ORMs which attempt similar magical optimisations?
> How do they go about identifying the cases where it is necessary?
>
> On 15 August 2017 at 10:44, Gordon Wrigley
> wrote:
>
>> I'd like to discuss automatic prefetching in querysets. Specifically
>
ng up and
>> maintaining appropriate prefetches and selects is a necessary part of
>> working with an ORM.
>>
>> Do you know of any other ORMs which attempt similar magical
>> optimisations? How do they go about identifying the cases where it is
>> necessary?
>>
&g
iew requires one prefetch, but a subsequent loop in a template requires
>>> some other prefetch? What about nested loops resulting in nested
>>> prefetches? Code like this is almost guaranteed to break unexpectedly in
>>> multiple ways. Personally, I would argue that c
he cases where it is necessary?
>
> On 15 August 2017 at 10:44, Gordon Wrigley
> wrote:
>
>> I'd like to discuss automatic prefetching in querysets. Specifically
>> automatically doing prefetch_related where needed without the user having
>> to request it.
>&g
hat correctly setting up and
>> maintaining appropriate prefetches and selects is a necessary part of
>> working with an ORM.
>>
>> Do you know of any other ORMs which attempt similar magical
>> optimisations? How do they go about identifying the cases where it is
>
nested
>> prefetches? Code like this is almost guaranteed to break unexpectedly in
>> multiple ways. Personally, I would argue that correctly setting up and
>> maintaining appropriate prefetches and selects is a necessary part of
>> working with an ORM.
>>
>> D
e prefetches be identified? What happens when an initial loop in a
>>> view requires one prefetch, but a subsequent loop in a template requires
>>> some other prefetch? What about nested loops resulting in nested
>>> prefetches? Code like this is almost guaranteed to break unexpe
>>
>>> I hope that clarifies my thinking some.
>>>
>>> Regards
>>> G
>>>
>>> On Tue, Aug 15, 2017 at 7:02 PM, Marc Tamlyn >> > wrote:
>>>
>>>> Hi Gordon,
>>>>
>>>> Thanks for the suggestion.
>
al
optimisations? How do they go about identifying the cases where it is
necessary?
On 15 August 2017 at 10:44, Gordon Wrigley <mailto:gordon.wrig...@gmail.com>> wrote:
I'd like to discuss automatic prefetching in querysets.
Specifically automatically doing prefetch_rela
d
> maintaining appropriate prefetches and selects is a necessary part of
> working with an ORM.
>
> Do you know of any other ORMs which attempt similar magical optimisations?
> How do they go about identifying the cases where it is necessary?
>
> On 15 August 2017 at 10:44,
>>> I do not propose touching reverse foreign key or many2many fields as
>>>> the generated queries could request substantially more rows from the DB
>>>> than the original query and it's not at all clear how this mechanism would
>>>> sanely interact
for the authors of those questions.
>>>>>
>>>>> It's worth noting that because these are foreign keys in their "to
>>>>> one" direction each of those queryset results will be at most the same
>>>>> size
>>>>> (in rows) as the
milar magical optimisations?
> How do they go about identifying the cases where it is necessary?
>
> On 15 August 2017 at 10:44, Gordon Wrigley
> wrote:
>
>> I'd like to discuss automatic prefetching in querysets. Specifically
>> automatically doing p
p in a template requires
>> some other prefetch? What about nested loops resulting in nested
>> prefetches? Code like this is almost guaranteed to break unexpectedly in
>> multiple ways. Personally, I would argue that correctly setting up and
>> maintaining appro
ropriate prefetches and selects
is a necessary part of working with an ORM.
Do you know of any other ORMs which attempt similar magical
optimisations? How do they go about identifying the cases
where it is necessary?
On 15 August 2017 at 10:44, Gordon Wr
ue
constraints, locks and race conditions, someone attempting to code
something that's not a blog or hello world really needs to know a bit about
all of that.
El martes, 15 de agosto de 2017, 6:44:19 (UTC-3), Gordon Wrigley escribió:
>
> I'd like to discuss automatic prefetc
es, data denormalization, proper joins here and there, unique
> constraints, locks and race conditions, someone attempting to code
> something that's not a blog or hello world really needs to know a bit about
> all of that.
>
>
>
> El martes, 15 de agosto de 2017, 6:44:
>
> It won't affect experienced users. They'll read the release notes, see
> that this change has been implemented, and either go and delete a bunch of
> prefetch_related() calls, grumble a bit and turn auto-prefetch off globally
> or just file it away as another fact they know about the Django
>
> I wouldn't want is to give free optimisations to non-ORM pros
>
Something like 99% of django projects are by non-ORM pros, it can be easy
to forget that on a mailing list of experts.
On 16 August 2017 at 12:27, Josh Smeaton wrote:
> It won't affect experienced users. They'll read the releas
On Wednesday, August 16, 2017 at 2:27:11 PM UTC+3, Josh Smeaton wrote:
>
> It won't affect experienced users. They'll read the release notes, see
>> that this change has been implemented, and either go and delete a bunch of
>> prefetch_related() calls, grumble a bit and turn auto-prefetch off glo
necessary part of working with an ORM.
Do you know of any other ORMs which attempt similar magical
optimisations? How do they go about identifying the cases
where it is necessary?
On 15 August 2017 at 10:44, Gordon Wrigley
> wrote:
I'
prefetch()` - I'm not sure what I think, I feel like it could get
>>> icky (i.e. increase our technical debt), due to the way it couples things
>>> together. I can't imagine ever wanting to use it, though, I would always
>>> prefer the manual option.
>>&g
>>> `auto_prefetch()` - I'm not sure what I think, I feel like it could get
>>> icky (i.e. increase our technical debt), due to the way it couples things
>>> together. I can't imagine ever wanting to use it, though, I would always
>>> prefer the manua
ons like these you simply cannot know in
>>>> advance whether doing extra queries is going to a be an optimization or a
>>>> pessimization. If I can come up with a single example where it would
>>>> significantly decrease performance (either memory usage or speed
Is this opt-{in,out} considered to be a global flag, meant to be
toggled on or off depending on whether it is an "expert" working on
the project or not?
I don't think that would be a good idea, almost all of our projects
have a mix of skill levels, and people move from team to team on a
regular ba
> On 15 Aug 2017, at 11:44, Gordon Wrigley wrote:
>
> I'd like to discuss automatic prefetching in querysets. Specifically
> automatically doing prefetch_related where needed without the user having to
> request it.
Hello,
I'm rather sympathetic to this proposal. Fi
Regarding 2, it does work for reverse one-to-one relations.
On Wed, Aug 16, 2017 at 9:17 PM, Aymeric Augustin <
aymeric.augus...@polytechnique.org> wrote:
> On 15 Aug 2017, at 11:44, Gordon Wrigley wrote:
>
> I'd like to discuss automatic prefetching in querysets. Specific
First of all, I think making the auto-prefetch feature available in some form
is a great idea. As an opt-in, I would have made use of it on several
occasions, and cut days of optimization work to minutes. It's true that this
wouldn't achieve the best possible optimization in many cases, but it w
I think there's a lot right with your suggestions here Shai.
It delivers better default behaviour for new projects, does not affect
existing deployments, and seems pretty easy to enable/disable selectively
at any level of the stack.
My only concern would be libraries leaning on this behaviour
I'm not advocating either way on this, but it's probably worth noting that
the context manager proposal is compatible with a queryset level optin/out
and an object level optout.
So you could for example have prefetch_related(auto=None) which can be
explicitly set to True / False and takes it's val
of an ORM that works better for all users.
Cheers,
Malcolm
On Wednesday, 16 August 2017 21:17:49 UTC+1, Aymeric Augustin wrote:
>
> On 15 Aug 2017, at 11:44, Gordon Wrigley > wrote:
>
> I'd like to discuss automatic prefetching in querysets. Specifically
> automatically d
for
> all users.
>
> Cheers,
>
> Malcolm
>
> On Wednesday, 16 August 2017 21:17:49 UTC+1, Aymeric Augustin wrote:
>>
>> On 15 Aug 2017, at 11:44, Gordon Wrigley wrote:
>>
>> I'd like to discuss automatic prefetching in querysets. Specifically
>> aut
e first time they see the warning / see a problem.
>> Experts can choose how they migrate - as Adam points out, even experts can
>> miss things.
>>
>> Finally after a suitable warning period, this can become an opt-out
>> feature and we arrive in the sunny world of an ORM
Hello,
> On 17 Aug 2017, at 14:48, Luke Plant wrote:
>
> On 16/08/17 23:17, Aymeric Augustin wrote:
>> It should kick in only when no select_related or prefetch_related is in
>> effect, to avoid interfering with pre-existing optimizations. It's still
>> easy to construct an example where it wo
On Thursday, August 17, 2017 at 11:30:45 PM UTC+3, Aymeric Augustin wrote:
>
> Hello,
>
> > On 17 Aug 2017, at 14:48, Luke Plant >
> wrote:
> >
> > On 16/08/17 23:17, Aymeric Augustin wrote:
> >> It should kick in only when no select_related or prefetch_related is in
> effect, to avoid interf
On Friday 18 August 2017 09:08:11 Anssi Kääriäinen wrote:
> Maybe we should just add the queryset method. This is the smallest atomic
> task that can be done. Even if there's only the queryset method available,
> it's possible to enable prefetches per model by using a Manager.
>
I disagree on both
I like that idea - keep it a private API for now and make it a public API
once people have used it a little bit.
On Fri, Aug 18, 2017 at 4:01 AM, Shai Berger wrote:
> On Friday 18 August 2017 09:08:11 Anssi Kääriäinen wrote:
> > Maybe we should just add the queryset method. This is the smallest
+1 to just add the queryset method it gives a fine granular level of
control. Moreover when not used we could display warnings which can help
people detect these O(n) query cases as Tom Forbes initially suggested.
On Friday, August 18, 2017 at 9:08:11 AM UTC+3, Anssi Kääriäinen wrote:
>
> On Thu
This could work something like the way that ForeignKey `on_delete` works
- you have options that are enumerated as constants, but in reality they
are classes that embody the strategy to be used. We could have something
similar - `on_missing_relation=FETCH | WARN | ERROR | IGNORE ... `
Luke
On
On Sat, Aug 19, 2017 at 1:10 PM, Luke Plant wrote:
> This could work something like the way that ForeignKey `on_delete` works -
> you have options that are enumerated as constants, but in reality they are
> classes that embody the strategy to be used. We could have something
> similar - `on_missi
On Monday 21 August 2017 18:44:35 Tobias McNulty wrote:
> On Sat, Aug 19, 2017 at 1:10 PM, Luke Plant wrote:
> > This could work something like the way that ForeignKey `on_delete` works
> > - you have options that are enumerated as constants, but in reality they
> > are classes that embody the str
W dniu środa, 16 sierpnia 2017 14:48:54 UTC+2 użytkownik Luke Plant napisał:
>
> I completely agree that visibility of this problem is a major issue, and
> would really welcome work on improving this, especially in DEBUG mode. One
> option might be a method that replaces lazy loads with exception
like to discuss automatic prefetching in querysets. Specifically
> automatically doing prefetch_related where needed without the user having
> to request it.
>
> For context consider these three snippets using the Question & Choice
> models from the tutorial
> <https://doc
46 matches
Mail list logo
