Sure. This script can use a lot of aesthetic improvement, but it
highlights processing the data in a single process.
The question would be, how to modify the script to process the data in
with parallel processes?
Regards,
- Robert
On Sat, Jan 30, 2010 at 4:36 AM, Sylvain Pointeau
<sylvain.point...@gmail.com> wrote:
> a good thing would have been to generate one file with all the statements...
> if you do that then you run sqlite with this file surrounded by transaction
> begin/commit
>
> echo "begin transaction" >> update.sql
>
> sqlite3 -separator $'\t' sample.db 'select rowid, item from foo;' |
> while read rowid item ; do
> status=$(long_running_process "${item}" )
> echo "update foo set status=${status} where rowid=${rowid} ;" >> update.sql
> done
>
> echo "commit transaction" >> update.sql
>
> sqlite3 sample.db < update.sql
>
> Best regards,
> Sylvain
>
> On Sat, Jan 30, 2010 at 12:04 AM, Robert Citek <robert.ci...@gmail.com>wrote:
>
>> Are there some white papers or examples of how to do updates in
>> parallel using sqlite?
>>
>> I have a large dataset in sqlite that I need to process outside of
>> sqlite and then update the sqlite database. The process looks
>> something like this:
>>
>> sqlite3 -separator $'\t' sample.db 'select rowid, item from foo;' |
>> while read rowid item ; do
>> status=$(long_running_process "${item}" )
>> sqlite3 sample.db "update foo set status=${status} where rowid=${rowid} ;"
>> done
>>
>> Because long_running_process takes a long time, I could speed up the
>> overall time by running more than one long_running_process at the same
>> time. One way to do this would be to segment the data and run a
>> separate process on each segment. For the update each process would
>> collect the status data "outside" of the sample.db, e.g in a separate
>> database. When all the processes have finished, the parent process
>> would attach the separate databases and update the original database.
>> When all is done, the parent process would clean up the ancillary
>> databases.
>>
>> I was just wondering if there are other ways to do this that I may be
>> overlooking.
>>
>> Thanks in advance to pointers to any references.
>>
>> Regards,
>> - Robert
>> _______________________________________________
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>
> _______________________________________________
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
