#36430: bulk_batch_size() special-cases single field on SQLite according to
outdated limit
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: Jacob
Type: | Walls
Cleanup/optimization | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
SQLITE_MAX_COMPOUND_SELECT, |
bulk_create |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Sage Abdullah):
Thanks Jacob for reporting this and thanks Sarah for cc-ing me to the
ticket.
As mentioned,
[https://github.com/django/django/pull/19427/files#r2062643293 I did
identify this in my previous PR]. However, I wasn't sure why Django even
uses `SQLITE_MAX_COMPOUND_SELECT ` for a bulk query. I tried to look into
where it might've come from but I didn't find any good leads.
> I think this must date from before bulk inserts used `VALUES` syntax,
which became available in SQLite in 2012, see
[https://github.com/laravel/framework/issues/25262#issuecomment-414836191
discussion in Laravel].
Aaand, I think you're correct! I didn't know `VALUES` wasn't supported
back then. My findings:
- The `500` limit a.k.a. `SQLITE_MAX_COMPOUND_SELECT` was added in
0a0a0d66b316598f7c296e8bf75749a14ce3ac49. As noted in the tests in that
commit, Django used `UNION`ed selects to do the bulk insert for some
reason.
- Related: a27582484cf814554907d2d1ad077852de36963f and #19351
- Looking at c27932ec938217d4fbb0adad23c0d0708f83f690 and #33460, yep.
It's pretty recent.
If it's not needed anymore, I'm in favour of removing this. I'm not 100%
sure, but it seems `bulk_batch_size` is only used for
[https://github.com/django/django/blob/953095d1e603fe0f8f01175b1409ca23818dcff9/django/db/models/query.py#L1909-L1915
bulk_create],
[https://github.com/django/django/blob/953095d1e603fe0f8f01175b1409ca23818dcff9/django/db/models/query.py#L928-L930
bulk_update], and
[https://github.com/django/django/blob/953095d1e603fe0f8f01175b1409ca23818dcff9/django/db/models/deletion.py#L233-L235
deletion], so I don't think any of them needs to consider the "max items
in a `SELECT` statement" limit on SQLite.
--
Ticket URL: <https://code.djangoproject.com/ticket/36430#comment:5>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/d/msgid/django-updates/010701973643a2ee-2651faf6-e64c-4400-8f40-0289dda1da8e-000000%40eu-central-1.amazonses.com.
