#30207: Optimise paginator for tables with massive records
-------------------------------------+-------------------------------------
Reporter: M. Javidan Darugar | Owner: M.
Type: | Javidan Darugar
Cleanup/optimization | Status: new
Component: Core (Other) | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by M. Javidan Darugar):
What do you mean by index? Tables index? for a table with such huge data
index table can be very huge. In our case indexing used almost 18GB of
disk memory. Indexing is very useful method but not always it depends on
the problem and the situation.
Replying to [comment:3 Nikolas]:
> 1. i too use paginator hack for admin, work for Postgres only if not use
any filters, this is can be inbox good optimization
>
> {{{
> class LargeTablePaginator(Paginator):
> """
> Warning: Postgresql only hack
> Overrides the count method of QuerySet objects to get an estimate
instead of actual count when not filtered.
> However, this estimate can be stale and hence not fit for situations
where the count of objects actually matter.
> """
>
> def _get_count(self):
> if getattr(self, '_count', None) is not None:
> return self._count
>
> query = self.object_list.query
> if not query.where:
> try:
> cursor = connection.cursor()
> cursor.execute("SELECT reltuples FROM pg_class WHERE
relname = %s",
> [query.model._meta.db_table])
> self._count = int(cursor.fetchone()[0])
> except:
> self._count = super(LargeTablePaginator,
self)._get_count()
> else:
> self._count = super(LargeTablePaginator, self)._get_count()
>
> return self._count
>
> count = property(_get_count)
> }}}
>
> maximum speed on any sized table
>
> 2. if use only pk order query can serios optimize when have max pk and
wont get page-1, in this case just need select ..... where pk<last_max_id.
In this case will be use index and have maximum speed
--
Ticket URL: <https://code.djangoproject.com/ticket/30207#comment:6>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/067.0ce6505ea9635571ab3b2546ac56ea5e%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
