#34067: django.core.Paginator wrong query slicing
-------------------------------------+-------------------------------------
Reporter: Hristo Trendafilov | Owner: nobody
Type: Bug | Status: closed
Component: Core (Other) | Version: 3.2
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage:
Paginator,slice,queryset | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Description changed by Hristo Trendafilov:
Old description:
> A have a ListView defined like so:
>
> {{{
> class AllPracticesListView(LoginRequiredMixin, PermissionRequiredMixin,
> ListView, ToolBoxFunctions):
> template_name = 'practice_list.html'
> model = Practice
> paginate_by = 6
> permission_required = 'customauth.access_practices'
>
> def get_queryset(self):
> qs = self.get_all_practices_by_user().order_by(
> 'company_branch__pk',
> 'practice_profile__personal_profile__last_name').prefetch_related('practice_profile')
>
> # method < get_all_practices_by_user > is a toolbox method that
> returns:
> #
> Practice.objects.filter(company_branch__owner=self.request.user).order_by('pk')
>
> return qs
>
> }}}
>
> Models are defined like so:
> {{{
>
> class Practice(models.Model):
> company_branch = models.ForeignKey(
> CompanyBranch,
> on_delete=models.PROTECT,
> related_name='practices',
> )
> # CompanyBranch model has an owner field which ForeignKey
> to the user model
>
> practice_profile = models.ForeignKey(
> PracticeProfile,
> on_delete=models.PROTECT,
> related_name='practices',
> )
>
> class PracticeProfile(models.Model):
> personal_profile = models.ForeignKey(
> PersonalProfile,
> on_delete=models.PROTECT,
> blank=True,
> null=True,
> )
> # PersonalProfile model has a field called <
> last_name >
> }}}
> Or schematically:
> {{{
> Practice object relations:
> -> CompanyBranch
> -> PracticeProfile
> -> PersonalProfile /*NULLABLE RELATION/
>
> }}}
>
> When the code is run, the view does not display results correctly.
>
> I did like so to get the WRONG result:
> Added a breakpoint on {{{Paginator._get_page}}} return statement
>
> Queryset is passed okay and is available in
> {{{Paginator.object_list}}}
>
> But in the args of {{{Paginator._get_page}}} the queryset is totally
> different than expected / should be a slice from the first six elements
> of {{{Paginator.object_list}}} /
>
> I have tried to add a breakpoint like so
> [[Image(https://imgur.com/SdaQUt6)]]
> Then I get this result [[Image(https://imgur.com/5zzLNV0)]].
>
> As you can see, PKs 1632, 1624, etc. are missing, objects are
> different/marked red and purple/, and are actually removed and never
> shown in the view.
>
> If you run slice manually at this point - everything is again fine
> [[Image(https://imgur.com/yvn8KMO)]]
> This happens only on NULL PersonalProfile objects.
>
> I did like so to get OKAY results:
>
> 1. Added a simple breakpoint is added on {{{Paginator.page}}} like so
> [[Image(https://imgur.com/d7J5BMZ)]] and that breakpoint is just run then
> the result is okay [[Image(https://imgur.com/ILDHAMN)]]
>
> 2. changed paginate_by to 10
>
> 3. changed {{{order_by}}} in {{{get_queryset}}} to {{{-pk}}}
>
> 4. call again the {{{page = paginator.page(page_number)}}}
>
> P.S. That is also happening in django 2.2
New description:
A have a ListView defined like so:
{{{
class AllPracticesListView(LoginRequiredMixin, PermissionRequiredMixin,
ListView, ToolBoxFunctions):
template_name = 'practice_list.html'
model = Practice
paginate_by = 6
permission_required = 'customauth.access_practices'
def get_queryset(self):
qs = self.get_all_practices_by_user().order_by(
'company_branch__pk',
'practice_profile__personal_profile__last_name').prefetch_related('practice_profile')
# method < get_all_practices_by_user > is a toolbox method that
returns:
#
Practice.objects.filter(company_branch__owner=self.request.user).order_by('pk')
return qs
}}}
Models are defined like so:
{{{
class Practice(models.Model):
company_branch = models.ForeignKey(
CompanyBranch,
on_delete=models.PROTECT,
related_name='practices',
)
# CompanyBranch model has an owner field which ForeignKey
to the user model
practice_profile = models.ForeignKey(
PracticeProfile,
on_delete=models.PROTECT,
related_name='practices',
)
class PracticeProfile(models.Model):
personal_profile = models.ForeignKey(
PersonalProfile,
on_delete=models.PROTECT,
blank=True,
null=True,
)
# PersonalProfile model has a field called < last_name
>
}}}
Or schematically:
{{{
Practice object relations:
-> CompanyBranch
-> PracticeProfile
-> PersonalProfile /*NULLABLE RELATION/
}}}
When the code is run, the view does not display results correctly.
I did like so to get the WRONG result:
Added a breakpoint on {{{Paginator._get_page}}} return statement
Queryset is passed okay and is available in
{{{Paginator.object_list}}}
But in the args of {{{Paginator._get_page}}} the queryset is totally
different than expected / should be a slice from the first six elements of
{{{Paginator.object_list}}} /
I have tried to add a breakpoint like so
[[Image(https://imgur.com/SdaQUt6)]]
Then I get this result [[Image(https://imgur.com/5zzLNV0)]].
As you can see, PKs 1632, 1624, etc. are missing, objects are
different/marked red and purple/, and are actually removed and never shown
in the view.
If you run slice manually at this point - everything is again fine
[[Image(https://imgur.com/yvn8KMO)]]
This happens only on NULL PersonalProfile objects.
I did like so to get OKAY results:
1. Added a simple breakpoint is added on {{{Paginator.page}}} like so
[[Image(https://imgur.com/d7J5BMZ)]] and that breakpoint is just run then
the result is okay [[Image(https://imgur.com/ILDHAMN)]]
2. changed paginate_by to 10
3. changed {{{order_by}}} in {{{get_queryset}}} to {{{-pk}}}
4. call again the {{{page = paginator.page(page_number)}}}
It is Django fault for those reasons:
* `order_by` might be complicated but is supported by the framework
* Chained `order_by` is also supported
* queryset slicing is a feature of Django queryset API.
* queryset slicing should act like list and string slicing and namely to
provide you with results `[ from : to ]` indexes provided and not to
change or lose data in between. Or in other words - what is passed in the
`Paginator.object_list` should be just sliced `[ from : to ]`. `Paginator`
should never care for the queryset, what is the ordering of the queryset,
what is excluded and so on - it should just work with it.
* changing `paginate_by` to `10` or greater fixes the problem.
`paginate_by` is a standard and documented Django ListView setting.
* changing `order_by` to some other `order_by` also fixes the problem.
`order_by` should work the same or if any limitations those should be well
documented
* removing `order_by` also fixes the problem.
P.S. That is also happening in Django 2.2
--
--
Ticket URL: <https://code.djangoproject.com/ticket/34067#comment:6>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/010701839c961d5d-95ef6060-da73-4015-9bf2-24c9ecca8180-000000%40eu-central-1.amazonses.com.
