I would be very interested to see some benchmark, just to see.....
On Mon, Feb 1, 2010 at 5:16 PM, Nicolas Williams
<[email protected]>wrote:
> On Sat, Jan 30, 2010 at 10:36:56AM +0100, Sylvain Pointeau wrote:
> > echo "begin transaction" >> update.sql
> >
> > sqlite3 -separator $'\t' sample.db 'select rowid, item from foo;' |
> > while read rowid item ; do
> > status=$(long_running_process "${item}" )
> > echo "update foo set status=${status} where rowid=${rowid} ;" >>
> update.sql
> > done
> >
> > echo "commit transaction" >> update.sql
> >
> > sqlite3 sample.db < update.sql
>
> More aesthetic (since it seems to matter :)
>
> (
> echo "begin transaction"
> sqlite3 -separator $'\t' sample.db 'select rowid, item from foo;' |
> while read rowid item ; do
> status=$(long_running_process "${item}" )
> echo "update foo set status=${status} where rowid=${rowid} ;"
> done
> echo "commit transaction"
> ) | sqlite3 sample.db
>
> Now to parallelize this:
>
> function par_updates {
> typeset -i n
> n=$1
> shift
> (
> trap "((n++))" CHLD
> echo "begin transaction"
> sqlite3 -separator $'\t' sample.db 'select rowid, item from foo;' |
> while read rowid item
> do
> while ((n == 0))
> do
> sleep 1
> done
> (echo "update foo set status=$("$@") where rowid=$rowid;") &
> ((n--))
> done
> echo "commit transaction"
> ) | sqlite3 sample.db
> }
>
> You should run this like so:
>
> par_updates 20 long_running_program [arguments]
>
> It should run up to 20 processes to generate the updates. And it should
> batch all updates into one transaction.
>
> I've not tested this, but it should work with recent versions of ksh93
> and bash. (Older versions of ksh93 might not handle the SIGCHLD trap
> correctly, IIRC.)
>
> Nico
> --
> _______________________________________________
> sqlite-users mailing list
> [email protected]
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
[email protected]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
