a good thing would have been to generate one file with all the statements...
if you do that then you run sqlite with this file surrounded by transaction
begin/commit
echo "begin transaction" >> update.sql
sqlite3 -separator $'\t' sample.db 'select rowid, item from foo;' |
while read rowid item ; do
status=$(long_running_process "${item}" )
echo "update foo set status=${status} where rowid=${rowid} ;" >> update.sql
done
echo "commit transaction" >> update.sql
sqlite3 sample.db < update.sql
Best regards,
Sylvain
On Sat, Jan 30, 2010 at 12:04 AM, Robert Citek <[email protected]>wrote:
> Are there some white papers or examples of how to do updates in
> parallel using sqlite?
>
> I have a large dataset in sqlite that I need to process outside of
> sqlite and then update the sqlite database. The process looks
> something like this:
>
> sqlite3 -separator $'\t' sample.db 'select rowid, item from foo;' |
> while read rowid item ; do
> status=$(long_running_process "${item}" )
> sqlite3 sample.db "update foo set status=${status} where rowid=${rowid} ;"
> done
>
> Because long_running_process takes a long time, I could speed up the
> overall time by running more than one long_running_process at the same
> time. One way to do this would be to segment the data and run a
> separate process on each segment. For the update each process would
> collect the status data "outside" of the sample.db, e.g in a separate
> database. When all the processes have finished, the parent process
> would attach the separate databases and update the original database.
> When all is done, the parent process would clean up the ancillary
> databases.
>
> I was just wondering if there are other ways to do this that I may be
> overlooking.
>
> Thanks in advance to pointers to any references.
>
> Regards,
> - Robert
> _______________________________________________
> sqlite-users mailing list
> [email protected]
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
[email protected]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
