Thanks for the suggestions. The session extension would work, all tables have rowids. The added flexibility to insert/update into the database copies first independently of each other is valuable.
Tracing the inserts/updates seems also a good idea, available with the present Sqlite version. Copying the WAL files is probably more efficient than the SQL text solutions (considering that roughly 5 GB of binary data are weekly added), and it seems easy to implement, so I'll probably try this first. I guess that simply opening the primary database for read before starting the insert/update process would prevent WAL checkpointing until I have a chance to copy the WAL (and SHM) files? Splitting this kind of scientific databases into smaller files is in my opinion generally not a good idea: typical use cases are that certain patterns/structures in the data are searched for, or some statistical analysis for longer periods is done. Then several database files, eg. containing separate time periods, are a nuisance. As an example, imagine the extra programming work for running a simple moving average, FFT etc, properly across such artifical time boundaries between database files. Sqlite is amazingly fast for this amount of data. Thanks to the log2(size) scaling of selects (on indexed columns) it should not significantly slow down as the database size will still increase. /Stephan
