[sqlite] Ideas on how to use backup API for remote, incremental backup?

2015-10-09 Thread David Barrett
Thanks Richard, this is exactly what I was thinking. One question on this: On Tue, Oct 6, 2015 at 10:22 PM, Richard Hipp wrote: > (2) For the source database connection of the backup, use the same > database connection that is used for writing to the database. That > means that when changes ar

[sqlite] Ideas on how to use backup API for remote, incremental backup?

2015-10-09 Thread Richard Hipp
On 10/9/15, David Barrett wrote: > Thanks Richard, this is exactly what I was thinking. One question on this: > > On Tue, Oct 6, 2015 at 10:22 PM, Richard Hipp wrote: > >> (2) For the source database connection of the backup, use the same >> database connection that is used for writing to the da

[sqlite] Ideas on how to use backup API for remote, incremental backup?

2015-10-06 Thread David Barrett
On Tue, Oct 6, 2015 at 8:36 PM, Simon Slavin wrote: > Or copy an existing /open/ database file to the new server using the > SQLite Backup API, [requires other connections to stop modifying the > database for long enough for the copy to be made] > Well the backup API works with WAL mode [1] so i

[sqlite] Ideas on how to use backup API for remote, incremental backup?

2015-10-06 Thread David Barrett
On Tue, Oct 6, 2015 at 2:57 PM, Clemens Ladisch wrote: > It backs up to any disk that you can access. > Do you have a network file system? > Well yes, but I'd like to handle it at the application layer. Basically, we operate a custom replication layer atop sqlite. It replicates individual tran

[sqlite] Ideas on how to use backup API for remote, incremental backup?

2015-10-06 Thread R.Smith
Well, there's the obvious Elephant in the room - SQL. You could just reduce any DB to SQL statements and pass those along at whichever pace/destination/byte-mode you fancy. The target system will have zero trouble turning it into a DB, thanks to the SQLite engine already able to parse SQL. Thi

[sqlite] Ideas on how to use backup API for remote, incremental backup?

2015-10-06 Thread Simon Slavin
On 6 Oct 2015, at 2:44pm, David Barrett wrote: > Regardless, all those solutions require me to wait for the entire backup to > complete before sending the file to the remote host -- my goal is to send > it one page at a time (eg, send the pages as the backup API processes them) > so as to avoid

[sqlite] Ideas on how to use backup API for remote, incremental backup?

2015-10-06 Thread Simon Slavin
On 6 Oct 2015, at 1:52pm, David Barrett wrote: > Well yes, but I'd like to handle it at the application layer. Basically, > we operate a custom replication layer atop sqlite. It replicates > individual transactions great with 2-phase commit, but right now you need > to manually "bootstrap" a n

[sqlite] Ideas on how to use backup API for remote, incremental backup?

2015-10-06 Thread Richard Hipp
On 10/6/15, David Barrett wrote: > sqlite has a cool "online backup" API: https://www.sqlite.org/backup.html > However, it only backs up to a local disk. I'm wondering if anybody can > think on how to use this API to do an incremental backup over a network > connection to a remote host? The net

[sqlite] Ideas on how to use backup API for remote, incremental backup?

2015-10-06 Thread Clemens Ladisch
David Barrett wrote: > sqlite has a cool "online backup" API: https://www.sqlite.org/backup.html > However, it only backs up to a local disk. It backs up to any disk that you can access. Do you have a network file system? > how to use this API to do an incremental backup This API is not increme

[sqlite] Ideas on how to use backup API for remote, incremental backup?

2015-10-06 Thread Charles Leifer
You could mount a directory as a ramdisk. On Tue, Oct 6, 2015 at 7:52 AM, David Barrett wrote: > On Tue, Oct 6, 2015 at 2:57 PM, Clemens Ladisch > wrote: > > > It backs up to any disk that you can access. > > Do you have a network file system? > > > > Well yes, but I'd like to handle it at the

[sqlite] Ideas on how to use backup API for remote, incremental backup?

2015-10-06 Thread David Barrett
sqlite has a cool "online backup" API: https://www.sqlite.org/backup.html However, it only backs up to a local disk. I'm wondering if anybody can think on how to use this API to do an incremental backup over a network connection to a remote host? The networking part is easy. But I can't figure