Re: [sqlite] Transaction behavior with large numbers of inserts

2009-01-28 Thread Ian Frosst
So thinking about this, what would be the impediment (apart from current implementation, of course,) to implementing a disk-based temp file page cache, that could grow arbitrarily large? Operations against the page cache in memory are already lost if your process dies before commission. Are there

Re: [sqlite] Transaction behavior with large numbers of inserts

2009-01-28 Thread vlemaire
Hi, Does your reader threads access to the same table as your writer ? Maybe (I'm not 100% sure it works) you could try : on a single file database, with the PRAGMA read_uncommited=1 on the readers connections (this is not a good idea if the tables are the same between readers and writers, it dep

Re: [sqlite] Transaction behavior with large numbers of inserts

2009-01-28 Thread Roger Binns
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Griggs, Donald wrote: > The INSERT into main.table (with its associated SELECT) would still be > an implicit transaction and of course would require writing to the > database of the million records. > > During the rather long period of time for the IN

Re: [sqlite] Transaction behavior with large numbers of inserts

2009-01-28 Thread Griggs, Donald
-Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Roger Binns Sent: Wednesday, January 28, 2009 1:12 AM To: General Discussion of SQLite Database Subject: Re: [sqlite] Transaction behavior with large numbers of inserts

Re: [sqlite] Transaction behavior with large numbers of inserts

2009-01-27 Thread Roger Binns
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ian Frosst wrote: > My question is; am I assessing the situation correctly, and if so, is there > anything I can do to avoid this rather nasty situation? I would rather not > take the periodic-commit approach, and increasing the page cache size would

Re: [sqlite] Transaction behavior with large numbers of inserts

2009-01-27 Thread Ken
allow readers access. SQLITE does not have any kind of multi versioning of the database pages. Thus you may not have one connection reading and another writing. HTH Ken --- On Tue, 1/27/09, Ian Frosst wrote: > From: Ian Frosst > Subject: [sqlite] Transaction behavior with large numb

[sqlite] Transaction behavior with large numbers of inserts

2009-01-27 Thread Ian Frosst
I have an application which on a single thread writes data to a database. This writing can result in millions of writes to the database in a single transaction (started with BEGIN TRANSACTION.) I also have other threads and processes which read the same database, but never write to it, and never s