Re: [sqlite] sqlite performance, locking & threading

2006-12-28 Thread Roger Binns
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Emerson Clarke wrote: | For architectural reasons the above steps must be performed in that | order. This means that operations cant be separated or queued up in | the way that you suggested. Each step is dependent on the previous | step. I was look

Re: [sqlite] sqlite performance, locking & threading

2006-12-28 Thread Jay Sprenkle
On 12/27/06, Emerson Clarke <[EMAIL PROTECTED]> wrote: The first question is why database locking has been enabled on a per thread basis instead of per process so that the normal methods of thread synchronisation (mutexes, ciritcal sections) could not be used for maintaining consistency within th

Re: [sqlite] sqlite performance, locking & threadin

2006-12-28 Thread Emerson Clarke
Ken, Thanks i understand your example well. What im looking for is the ability to have multiple threads perform a write operation based on my mutex, not some internal sqlite write mutex. If i am managing the concurrency and performing correct synchronisation, why can i not have multiple threads

Re: [sqlite] sqlite performance, locking & threading

2006-12-28 Thread Emerson Clarke
The indexing process works like this. 1.) Open a document and parse its contents. 2.) Look up records in the first database based on the contents of the document, updating records where appropriate and inserting new ones. 3.) Transforming the document based on what was obtained from the first dat

Re: [sqlite] ORDER BY is more than 10 times slower with 3.3.8 compared to 3.3.7

2006-12-28 Thread Joe Wilson
Need more info than that. Schema, sample data, actual slow query, etc. If I were to guess - try doing the same query twice. You've probably got a cold file cache. --- Michael Sizaki <[EMAIL PROTECTED]> wrote: > The following query on a table with 400,000 rows > > SELECT * FROM table where ORD

[sqlite] ORDER BY is more than 10 times slower with 3.3.8 compared to 3.3.7

2006-12-28 Thread Michael Sizaki
Hi, The following query on a table with 400,000 rows SELECT * FROM table where ORDER BY name limit 10; takes less than 3 sec with version 3.3.7 (or 3.3.0) and 35 sec with version 3.3.8. What has changed in 3.3.8 to make it so slow? My application relies on fast sorting Michael ---

Re: [sqlite] sqlite performance, locking & threading

2006-12-28 Thread Ken
Emerson, Is the Database on the same disk as the rest of the file operations? If so is it possible that you are I/O bound and causing seek issues due to i/o access patterns? Take a look at the test_server.c code in the sqlite/src directory. I used that as a basis to build a custom library

Re: [sqlite] sqlite performance, locking & threading

2006-12-28 Thread Roger Binns
Emerson Clarke wrote: The idea is that because i am accessing two databases, and doing several file system operations per document, there should be a large gain by using many threads. There is no actual indexing process, the whole structure is the index, but if anything the database operations t

Re: [sqlite] sqlite performance, locking & threading

2006-12-28 Thread John Stanton
I am curious as to how multiple threads would perform faster inserts into an Sqlite database, which is a single file plus the journal. Are you using a multiple processor machine? Emerson Clarke wrote: Roger, Thanks for the suggestions. I think using a worker thread and a queue would be equiv

Re: [sqlite] sqlite performance, locking & threading

2006-12-28 Thread Emerson Clarke
Roger, Thanks for the suggestions. I think using a worker thread and a queue would be equivalent to just running a single thread since it effectively makes the database operations synchronous. Although i can see what your driving at regarding the transactions every n records. The idea is that

Re: [sqlite] collation with order by bug? (was: multiple order by value bug? )

2006-12-28 Thread kiru
First, sorry for the late reply. I think this is an "Order by" with COLLATION issue. My kereset.datum column in my query sorting well if I'm defined the table withouth COLLATE MAGYAR and modified my query like this: "SELECT szemely.nev, szemely.anyanev, szemely.szulhely, szemely.adoszam, kere

Re: [sqlite] sqlite performance, locking & threading

2006-12-28 Thread drh
"Emerson Clarke" <[EMAIL PROTECTED]> wrote: > > The first question is why database locking has been enabled on a per thread > basis instead of per process so that the normal methods of thread > synchronisation (mutexes, ciritcal sections) could not be used for > maintaining consistency within the