On Sat, Jul 14, 2012 at 03:17:07PM +0100, Simon Slavin wrote:
>
> On 14 Jul 2012, at 3:12pm, Udi Karni <[email protected]> wrote:
>
> > I know
> > nothing about writing DB engines - so I don't know whether adding a 2nd
> > parallel process adds 10K or 10M to the code base.
>
> You've reached the limit of what I know about parallelization. I hope
> someone else can chime in.
Using SQLite's VM architecture, I would guess that adding this sort of
parallelization would be non-trival. You need a parallel VM, significantly
different to the current sequential VM, at at least a way of managing
asynchronous IO, with perhaps a callback mechanism into the VM to handle IO
completion. <shudder>
While not certain, I guess other databases handle this by using tree based
execution plans, where any single execution node can easily be split into
branches to another thread/process/machine, then merged in the parent tree
node, with each branch handling a certain key range.
This might make sense, for example, with a partitioned table, where each
partition is on it's own spindle, so a full table scan can be executed in
parallel on each spindle and merged as a final step. So, for a table scan
between k0 and k3, find intermediate keys to split the query between spindles:
(k0-k3)
/|\
/ | \
/ | \
/ | \
/ | \
(k0-k1] (k1-k2] (k2-k3)
| | |
disk1 disk2 disk3
I sat through an Oracle internals course once, and the instructor gave us an
example of a setup such as this where data was partitioned across 24 disks, and
the resulting full table scans were in fact quicker than index based scans for
the data set they were using.
Of course, the above would be useless for SQLite anyway, being a single file
database. And even with the likes of Oracle, Stripe And Mirror Everything
(SAME) might also largely defeat parallel scans.
All in all, the added bloat would be measured in MB, rather than KB.
Christian
disclaimer: Not a practical DB implementation expert.
_______________________________________________
sqlite-users mailing list
[email protected]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
