#21160: in_bulk() fails on SQLite when passing more than 999 ids
-------------------------------------+-------------------------------------
Reporter: NiGhTTraX | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) | Resolution:
Severity: Normal | Triage Stage:
Keywords: in_bulk sqlite 999 | Unreviewed
1000 | Needs documentation: 0
Has patch: 1 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Description changed by EvilDMP:
Old description:
> As we all know, SQLite doesn't allow more than 999 variables in a query,
> failing with the 'too many SQL variables' error if you pass in more. This
> is a problem when using the in_bulk() method.
>
> I've looked through the code and found out that all backends define a
> bulk_batch_size ops feature and it is set to 999 in the SQLite backend.
> https://github.com/django/django/blob/master/django/db/backends/sqlite3/base.py#L140
>
> This ops feature is only used in _batched_insert. I've written a patch
> that makes use of it in the in_bulk method as well. You can check it here
> [link pending]
>
> I've also written two tests that test passing in a large number of ids
> and also the efficiency in this case. The efficiency test runs only on
> databases that don't support 1000 variables in a query.
>
> Any comments are welcome.
New description:
As we all know, SQLite doesn't allow more than 999 variables in a query,
failing with the 'too many SQL variables' error if you pass in more. This
is a problem when using the in_bulk() method.
I've looked through the code and found out that all backends define a
bulk_batch_size ops feature and it is set to 999 in the SQLite backend.
https://github.com/django/django/blob/master/django/db/backends/sqlite3/base.py#L140
This ops feature is only used in _batched_insert. I've written a patch
that makes use of it in the in_bulk method as well. You can check it here:
https://github.com/django/django/pull/1679
I've also written two tests that test passing in a large number of ids and
also the efficiency in this case. The efficiency test runs only on
databases that don't support 1000 variables in a query.
Any comments are welcome.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/21160#comment:3>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/067.9d8bdbe5109accc414507eca286d9783%40djangoproject.com.
For more options, visit https://groups.google.com/groups/opt_out.
