I would be very interested to see some benchmark, just to see..... On Mon, Feb 1, 2010 at 5:16 PM, Nicolas Williams <nicolas.willi...@sun.com>wrote:
> On Sat, Jan 30, 2010 at 10:36:56AM +0100, Sylvain Pointeau wrote: > > echo "begin transaction" >> update.sql > > > > sqlite3 -separator $'\t' sample.db 'select rowid, item from foo;' | > > while read rowid item ; do > > status=$(long_running_process "${item}" ) > > echo "update foo set status=${status} where rowid=${rowid} ;" >> > update.sql > > done > > > > echo "commit transaction" >> update.sql > > > > sqlite3 sample.db < update.sql > > More aesthetic (since it seems to matter :) > > ( > echo "begin transaction" > sqlite3 -separator $'\t' sample.db 'select rowid, item from foo;' | > while read rowid item ; do > status=$(long_running_process "${item}" ) > echo "update foo set status=${status} where rowid=${rowid} ;" > done > echo "commit transaction" > ) | sqlite3 sample.db > > Now to parallelize this: > > function par_updates { > typeset -i n > n=$1 > shift > ( > trap "((n++))" CHLD > echo "begin transaction" > sqlite3 -separator $'\t' sample.db 'select rowid, item from foo;' | > while read rowid item > do > while ((n == 0)) > do > sleep 1 > done > (echo "update foo set status=$("$@") where rowid=$rowid;") & > ((n--)) > done > echo "commit transaction" > ) | sqlite3 sample.db > } > > You should run this like so: > > par_updates 20 long_running_program [arguments] > > It should run up to 20 processes to generate the updates. And it should > batch all updates into one transaction. > > I've not tested this, but it should work with recent versions of ksh93 > and bash. (Older versions of ksh93 might not handle the SIGCHLD trap > correctly, IIRC.) > > Nico > -- > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users