Re: [sqlite] Online/Hot backup of WAL journalling mode database

2014-12-12 Thread Clemens Ladisch
Simon Slavin wrote: > On 12 Dec 2014, at 10:27am, Clemens Ladisch wrote: >> If you write your own backup tool that simply calls >> "sqlite3_backup_step(b, -1)", the entire database is copied in >> a single atomic transaction. > > OP's problem is that he runs several processes which are constantly

Re: [sqlite] Online/Hot backup of WAL journalling mode database

2014-12-12 Thread Simon Slavin
On 12 Dec 2014, at 10:27am, Clemens Ladisch wrote: > If you write your own backup tool that simply calls > "sqlite3_backup_step(b, -1)", the entire database is copied in > a single atomic transaction. OP's problem is that he runs several processes which are constantly (every few seconds) writ

Re: [sqlite] Online/Hot backup of WAL journalling mode database

2014-12-12 Thread Clemens Ladisch
Nick wrote: > On 11 Dec 2014, at 20:39, David King wrote: >> Why are you trying to hard to avoid using the backup API? It sounds >> like it does exactly what you want > > Backup API works great if you have periods of no writing. > However, if a process writes during the backup then the API would st

Re: [sqlite] Online/Hot backup of WAL journalling mode database

2014-12-12 Thread Dan Kennedy
On 12/12/2014 03:31 AM, Nick wrote: On 11 Dec 2014, at 10:08, Dan Kennedy wrote: On 12/11/2014 05:49 AM, Nick wrote: On 10 Dec 2014, at 07:35, Dan Kennedy wrote: Strictly speaking the database file may not be well-formed even if there is no ongoing checkpoint. If: a) process A opens a re

Re: [sqlite] Online/Hot backup of WAL journalling mode database

2014-12-11 Thread Nick
On 11 Dec 2014, at 20:39, David King wrote: > Why are you trying to hard to avoid using the backup API? It sounds like it > does exactly what you want Backup API works great if you have periods of no writing. However, if a process writes during the backup then the API would stop and start over

Re: [sqlite] Online/Hot backup of WAL journalling mode database

2014-12-11 Thread David King
Why are you trying to hard to avoid using the backup API? It sounds like it does exactly what you want On 11 Dec 2014, at 12:36, Nick wrote: > > On 11 Dec 2014, at 10:43, Simon Slavin wrote: > >> >> I don't know enough about the internals of SQLite to be sure, but various >> parts of me a

Re: [sqlite] Online/Hot backup of WAL journalling mode database

2014-12-11 Thread Nick
On 11 Dec 2014, at 10:43, Simon Slavin wrote: > > I don't know enough about the internals of SQLite to be sure, but various > parts of me are concerned that this is a bad idea. I don't know what WAL > mode would be like without checkpointing but there has to be a reason for > checkpointing a

Re: [sqlite] Online/Hot backup of WAL journalling mode database

2014-12-11 Thread Nick
On 11 Dec 2014, at 10:08, Dan Kennedy wrote: > On 12/11/2014 05:49 AM, Nick wrote: >> On 10 Dec 2014, at 07:35, Dan Kennedy wrote: >> >>> Strictly speaking the database file may not be well-formed even if there is >>> no ongoing checkpoint. If: >>> >>> a) process A opens a read transaction, >

Re: [sqlite] Online/Hot backup of WAL journalling mode database

2014-12-11 Thread Simon Slavin
On 10 Dec 2014, at 10:40pm, Nick wrote: > All the processes would have automatic checkpointing disabled. Just the > backup process would perform the checkpoint. I don't know enough about the internals of SQLite to be sure, but various parts of me are concerned that this is a bad idea. I don'

Re: [sqlite] Online/Hot backup of WAL journalling mode database

2014-12-11 Thread Dan Kennedy
On 12/11/2014 05:49 AM, Nick wrote: On 10 Dec 2014, at 07:35, Dan Kennedy wrote: Strictly speaking the database file may not be well-formed even if there is no ongoing checkpoint. If: a) process A opens a read transaction, b) process B opens and commits a write transaction to the database

Re: [sqlite] Online/Hot backup of WAL journalling mode database

2014-12-10 Thread Nick
On 10 Dec 2014, at 07:35, Dan Kennedy wrote: > Strictly speaking the database file may not be well-formed even if there is > no ongoing checkpoint. If: > > a) process A opens a read transaction, > b) process B opens and commits a write transaction to the database, > c) process C checkpoints

Re: [sqlite] Online/Hot backup of WAL journalling mode database

2014-12-10 Thread Nick
On 10 Dec 2014, at 02:36, Simon Slavin wrote: > > On 10 Dec 2014, at 12:30am, Nick wrote: > >> That's interesting Simon I didn't expect the database not to be trustworthy. > > The database will be trustworthy at any instant. Your copy of it will be > corrupt because the file will be changin

Re: [sqlite] Online/Hot backup of WAL journalling mode database

2014-12-09 Thread Dan Kennedy
On 12/10/2014 05:06 AM, Simon Slavin wrote: On 9 Dec 2014, at 8:57pm, Nick wrote: Environment is Linux with multiple (c. 4-6) processes accessing a single sqlite database named "test.db". Backup: - New process started using cronjob to initiate application checkpoint until completion. - rsyn

Re: [sqlite] Online/Hot backup of WAL journalling mode database

2014-12-09 Thread Simon Slavin
On 10 Dec 2014, at 12:30am, Nick wrote: > That's interesting Simon I didn't expect the database not to be trustworthy. The database will be trustworthy at any instant. Your copy of it will be corrupt because the file will be changing while you are copying it. > In WAL mode I thought the data

Re: [sqlite] Online/Hot backup of WAL journalling mode database

2014-12-09 Thread Nick
On 9 Dec 2014, at 22:06, Simon Slavin wrote: > > On 9 Dec 2014, at 8:57pm, Nick wrote: > >> Environment is Linux with multiple (c. 4-6) processes accessing a single >> sqlite database named "test.db". >> >> Backup: >> - New process started using cronjob to initiate application checkpoint unt

Re: [sqlite] Online/Hot backup of WAL journalling mode database

2014-12-09 Thread Simon Slavin
On 9 Dec 2014, at 8:57pm, Nick wrote: > Environment is Linux with multiple (c. 4-6) processes accessing a single > sqlite database named "test.db". > > Backup: > - New process started using cronjob to initiate application checkpoint until > completion. > - rsync diff the file "test.db" to ano

[sqlite] Online/Hot backup of WAL journalling mode database

2014-12-09 Thread Nick
Hi, I'd like to check my understanding of Sqlite in WAL journalling mode. With automatic checkpointing turned off would the following psuedo-code result in a online backup approach that allows robust restore of the database with data fresh up to the last checkpoint? Environment is Linux with m