#17788: bulk_create() "too many SQL variables" error
-------------------------------------+-------------------------------------
Reporter: alpar | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version:
(models, ORM) | 1.4-beta-1
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by alpar):
Replying to [comment:7 carljm]:
> [...] since it seems likely that the answer "use a better database"
suffices to prevent it being an actual problem in production [...]
My problem is that although we "use a better database" for production, but
we use SQLite during the development, because it is easier to setup, to
create and share snapshots etc.
I believe we are not alone with this development model.
If we accept that "correct behavior" should have priority over
"performance", then SQLite, `bulk_create()` may simply create the object
one-by-one. It is slow (on SQLite), but at least it works, and one may
hope better performance on a "better database". So, it is already much
better than having something unusable.
How does the batch size affects to performance? As far as I understand,
there is an inevitable processing requirement for inserting data, plus the
overhead of processing the SQL query (one-by-one). The feature
`bulk_create()` is about to reduce the second one. Thus, if we use batch
size as small as 10, then it already reduces the overhead to one tenth.
Although I didn't tested it, but it makes me feel, that ''the performance
gain can already be achieved using very small batches'', Large batches
will result in negligible further improvements.
Finally, using reasonable sized batches may be beneficial even on "better
databases", as a huge single SQL insert command may cause problems (such
as timeout) on heavily loaded systems.
--
Ticket URL: <https://code.djangoproject.com/ticket/17788#comment:15>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-updates?hl=en.
