Re: [sqlite] sqlite performance, locking & threading

2007-01-03 Thread Nicolas Williams
On Thu, Jan 04, 2007 at 12:50:01AM +, Emerson Clarke wrote: > My oppologies, your right that explanation had been given. OK. > But i didnt actually take it seriously, i guess i found it hard to > believe that it being the easier option was the only reason why this > limitation was in place.

Re: [sqlite] sqlite performance, locking & threading

2007-01-03 Thread Emerson Clarke
Nicholas, My oppologies, your right that explanation had been given. But i didnt actually take it seriously, i guess i found it hard to believe that it being the easier option was the only reason why this limitation was in place. If this is the case, then surely the fix is simple. Given that i

AW: [sqlite] sqlite performance, locking & threading

2007-01-03 Thread Michael Ruck
07 20:31 An: sqlite-users@sqlite.org Betreff: Re: [sqlite] sqlite performance, locking & threading Michael, Thanks for the advice. During the indexing process i need to select and optionally insert records into a table so i cant ignore the outcomes. Basically the indexing process does compres

Re: [sqlite] sqlite performance, locking & threading

2007-01-03 Thread Emerson Clarke
e [mailto:[EMAIL PROTECTED] Gesendet: Mittwoch, 3. Januar 2007 15:14 An: sqlite-users@sqlite.org Betreff: Re: [sqlite] sqlite performance, locking & threading Michael, Im not sure that atomic operations would be a suitable alternative. The reason why im using events/conditions is so that the clie

Re: [sqlite] sqlite performance, locking & threading

2007-01-03 Thread Nicolas Williams
On Tue, Jan 02, 2007 at 11:56:42PM +, Emerson Clarke wrote: > The single connection multiple thread alternative apparently has > problems with sqlite3_step being active on more than one thread at the > same moment, so cannot easily be used in a safe way. But it is by far > the fastest and simp

AW: [sqlite] sqlite performance, locking & threading

2007-01-03 Thread Michael Ruck
Mittwoch, 3. Januar 2007 15:14 An: sqlite-users@sqlite.org Betreff: Re: [sqlite] sqlite performance, locking & threading Michael, Im not sure that atomic operations would be a suitable alternative. The reason why im using events/conditions is so that the client thread blocks until the serv

Re: [sqlite] sqlite performance, locking & threading

2007-01-03 Thread Emerson Clarke
is started. No thread has ownership of the queue, except may be the main thread. Michael -Ursprüngliche Nachricht- Von: Emerson Clarke [mailto:[EMAIL PROTECTED] Gesendet: Mittwoch, 3. Januar 2007 00:57 An: sqlite-users@sqlite.org Betreff: Re: [sqlite] sqlite performance, locking & thread

AW: [sqlite] sqlite performance, locking & threading

2007-01-03 Thread Michael Ruck
007 00:57 An: sqlite-users@sqlite.org Betreff: Re: [sqlite] sqlite performance, locking & threading Nico, I have implemented all three strategies (thead specific connections, single connection multiple threads, and single thread server with multiple client threads). The problem with using thr

Re: [sqlite] sqlite performance, locking & threading

2007-01-02 Thread drh
"Emerson Clarke" <[EMAIL PROTECTED]> wrote: > > Firstly can i clarify what you mean regarding the same moment. Do you > mean that no two threads can be executing the call, or that no two > threads can be in the middle of stepping through a series of results > using the step function (assuming the

Re: [sqlite] sqlite performance, locking & threading

2007-01-02 Thread Emerson Clarke
Richard, Sorry to bring this up again, but ive just finished testing the alternative strategies. Having coded all 3 versions i still have a question about the sqlite3_step() problem which i think was isolated as the cause of my trouble a few days ago. Firstly can i clarify what you mean regardi

Re: [sqlite] sqlite performance, locking & threading

2007-01-02 Thread Emerson Clarke
Nico, I have implemented all three strategies (thead specific connections, single connection multiple threads, and single thread server with multiple client threads). The problem with using thread specific contexts is that you cant have a single global transaction which wraps all of those contex

Re: [sqlite] sqlite performance, locking & threading

2007-01-02 Thread Nicolas Williams
On Sat, Dec 30, 2006 at 03:34:01PM +, Emerson Clarke wrote: > Technically sqlite is not thread safe. [...] Solaris man pages describe APIs with requirements like SQLite's as "MT-Safe with exceptions" and the exceptions are listed in the man page. That's still MT-Safe, but the caller has to p

Re: [sqlite] sqlite performance, locking & threading

2006-12-30 Thread drh
"Emerson Clarke" <[EMAIL PROTECTED]> wrote: > > "With that and subsequent versions, it is safe to move a connection > handle across threads as long as the connection is not holding any > fcntl() locks. You can safely assume that no locks are being held if > no transaction is pending and all statem

Re: [sqlite] sqlite performance, locking & threading

2006-12-30 Thread Emerson Clarke
Richard, Im getting some mixed signals here, on the one hand i am being told that it should be ok to use a single transaction and connection across multiple threads (assuming that it is correctly synchronised). But on the other hand i am reading statements like that as part of a list of things wh

Re: [sqlite] sqlite performance, locking & threading

2006-12-30 Thread drh
Joe Wilson <[EMAIL PROTECTED]> wrote: > --- [EMAIL PROTECTED] wrote: > > SQLite allows you to have multiple queries running at the > > same time in multiple threads, as long as no two threads are > > running sqlite3_step() at the same moment. > > Do you mean "as long as no two threads are running

Re: [sqlite] sqlite performance, locking & threading

2006-12-30 Thread Doug Currie
On Saturday, December 30, 2006 [EMAIL PROTECTED] wrote: > Isn't there a list of possible causes for SQLITE_MISUSE somewhere. > I seem to remember writing such a list one. Does anybody know where > I put it? This is a forwarded message From:[EMAIL PROTECTED] <[EMAIL PROTECTED]> To: sq

Re: [sqlite] sqlite performance, locking & threading

2006-12-30 Thread Joe Wilson
--- [EMAIL PROTECTED] wrote: > SQLite allows you to have multiple queries running at the > same time in multiple threads, as long as no two threads are > running sqlite3_step() at the same moment. Do you mean "as long as no two threads are running sqlite3_step() _for the same sqlite3* connection_

Re: [sqlite] sqlite performance, locking & threading

2006-12-30 Thread John Stanton
Good advice. If you want to experiment with a certain architecture, write the programs yourself rather than trying to get others to do it for you. Sqlite is a well thought through solution for an embedded database with a wide range of applications, but if you want something more specialised,

Re: [sqlite] sqlite performance, locking & threading

2006-12-30 Thread drh
"Michael Ruck" <[EMAIL PROTECTED]> wrote: > Richard, > > I believe his problem is this: > > "Each query is allowed to complete before the other one starts, but each > thread may have multiple statements or result sets open." > > The open resultsets/multiple started statements are causing him = >

AW: [sqlite] sqlite performance, locking & threading

2006-12-30 Thread Michael Ruck
t- Von: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Gesendet: Samstag, 30. Dezember 2006 17:32 An: sqlite-users@sqlite.org Betreff: Re: [sqlite] sqlite performance, locking & threading "Emerson Clarke" <[EMAIL PROTECTED]> wrote: > Richard, > > Are you sure we are no

AW: [sqlite] sqlite performance, locking & threading

2006-12-30 Thread Michael Ruck
nection can only keep one resultset open or a statement executing (unless that has changed recently.) Mike -Ursprüngliche Nachricht- Von: Emerson Clarke [mailto:[EMAIL PROTECTED] Gesendet: Samstag, 30. Dezember 2006 17:08 An: sqlite-users@sqlite.org Betreff: Re: [sqlite] sqlite perform

Re: [sqlite] sqlite performance, locking & threading

2006-12-30 Thread drh
"Emerson Clarke" <[EMAIL PROTECTED]> wrote: > Richard, > > Are you sure we are not just getting into semantic knots here ? > > Do we have the same definition of "at the same time". I mean > concurrently, so that both threads use the same sqlite3 * structure, > within mutexes. Each query is allow

Re: [sqlite] sqlite performance, locking & threading

2006-12-30 Thread drh
"Emerson Clarke" <[EMAIL PROTECTED]> wrote: > > I have code which creates a transaction on a connection in the parent > thread, then creates several child threads which attempt to use the > same connection and transaction in a synchronised mannor. It does not > work, and by all the documentation

RE: [sqlite] sqlite performance, locking & threading

2006-12-30 Thread Fred Williams
> Sent: Saturday, December 30, 2006 9:34 AM > To: sqlite-users@sqlite.org > Subject: Re: [sqlite] sqlite performance, locking & threading > > > Roger, > > I think sqlite suffers somewhat from a bit of an identity crisis. > Whilst it is both a library and a piece of code wh

Re: [sqlite] sqlite performance, locking & threading

2006-12-30 Thread Emerson Clarke
Richard, Are you sure we are not just getting into semantic knots here ? Do we have the same definition of "at the same time". I mean concurrently, so that both threads use the same sqlite3 * structure, within mutexes. Each query is allowed to complete before the other one starts, but each thre

Re: [sqlite] sqlite performance, locking & threading

2006-12-30 Thread Emerson Clarke
Richard, Well, that has been the whole point of this discussion. I have code which creates a transaction on a connection in the parent thread, then creates several child threads which attempt to use the same connection and transaction in a synchronised mannor. It does not work, and by all the d

Re: [sqlite] sqlite performance, locking & threading

2006-12-30 Thread drh
"Emerson Clarke" <[EMAIL PROTECTED]> wrote: > But why then can i not have a single transaction wrapping a single > connection which is used within multiple threads, obvioulsy not at the > same time. You can. What makes you think you can't? -- D. Richard Hipp <[EMAIL PROTECTED]> --

Re: [sqlite] sqlite performance, locking & threading

2006-12-30 Thread Emerson Clarke
Richard, Ok, im pretty clear on the file locking being the cause of the problems with the sqlite3 * structures, but thanks for confirming it. I understand that on platforms that dont have this issue its not a problem. But why then can i not have a single transaction wrapping a single connection

Re: [sqlite] sqlite performance, locking & threading

2006-12-30 Thread drh
"Emerson Clarke" <[EMAIL PROTECTED]> wrote: > Even on the > platforms where a single sqlite3 * structure can be used on multiple > threads (provided it is not at the same time), it is not possible to > have a transaction which works across these threads. I beg to differ. What makes you think th

Re: [sqlite] sqlite performance, locking & threading

2006-12-30 Thread Emerson Clarke
Roger, I think sqlite suffers somewhat from a bit of an identity crisis. Whilst it is both a library and a piece of code which you embed in a project it is often talked about as though it is some external component. Technically sqlite is not thread safe. Just because the library has explicitly

Re: [sqlite] sqlite performance, locking & threading

2006-12-30 Thread drh
"Emerson Clarke" <[EMAIL PROTECTED]> wrote: > Richard, > > My complaint, if you want to call it that. Was simply that there are > seemingly artificial constraints on what you can and cant do accross > threads. > > If i have a linked list, i can use it across threads if i want to, > provided that

Re: [sqlite] sqlite performance, locking & threading

2006-12-30 Thread Roger Binns
Emerson Clarke wrote: If i have a linked list, i can use it across threads if i want to, provided that i synchronise operations in such a way that the list does not get corrupted. And of course you also have to know about memory barriers and compiler re-ordering. That is highly dependent on t

Re: [sqlite] sqlite performance, locking & threading

2006-12-30 Thread Emerson Clarke
Richard, My complaint, if you want to call it that. Was simply that there are seemingly artificial constraints on what you can and cant do accross threads. If i have a linked list, i can use it across threads if i want to, provided that i synchronise operations in such a way that the list does

Re: [sqlite] sqlite performance, locking & threading

2006-12-30 Thread drh
"Emerson Clarke" <[EMAIL PROTECTED]> wrote: >> > It seemed to me that making a library which only functioned on a per > thread basis was something that you would have to do deliberately and > by design. I'm still trying to understand what your complaint is. -- D. Richard Hipp <[EMAIL PROTECTED]

Re: [sqlite] sqlite performance, locking & threading

2006-12-30 Thread Roger Binns
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Emerson Clarke wrote: | I am left to assume that all other locking mechanisms like ipc and | files have already been tried and been found wanting. I also assume | that priority has been given to making sqlite operate across network | boundaries rathe

Re: [sqlite] sqlite performance, locking & threading

2006-12-30 Thread Emerson Clarke
Roger, My original question was in fact not a statement. I did not want sqlite to work differently. Rather the opposite, sqlite already works differently to the way i, and probably a lot of users assume that it would. So all i wanted to know was why that is the case. It seemed to me that maki

Re: [sqlite] sqlite performance, locking & threading

2006-12-30 Thread Roger Binns
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Emerson Clarke wrote: | I have deliberately tried to avoid giving too much detail on the | architecture of the index since that was not the point and i didnt | want to end up debating it. I don't want to debate your index architecture either :-). Qu

Re: [sqlite] sqlite performance, locking & threading

2006-12-29 Thread Emerson Clarke
Roger, I have deliberately tried to avoid giving too much detail on the architecture of the index since that was not the point and i didnt want to end up debating it. The design of the index is not the issue, suffice to say that i think you are over complicating things. It is a desceptively simp

Re: [sqlite] sqlite performance, locking & threading

2006-12-29 Thread Roger Binns
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 You never answered the bit about whether indexing A can be done at the same time as B. (I get the feeling you have a certain design in mind and insist that SQLite changes to meet that design, rather than change your own design around the constraints

Re: [sqlite] sqlite performance, locking & thread

2006-12-29 Thread Ken
Emereson, And you can't do this with oracle either, That is create a connection and pass it around between multiple threads.. Oracle would be very very unhappy if you did that. Oracle utilizes a context variable to distinguish between threads and utilizes precompiler flags to enable thread su

Re: [sqlite] sqlite performance, locking & threading

2006-12-29 Thread John Stanton
Unless you have a multi processor machine or some form of parallel processing I cannot see how you can do anything other than slow your procesing by adding threading overhead. You can simplify the conception of the whole process by understanding that Sqlite writes to a single file on a single

Re: [sqlite] sqlite performance, locking & thread

2006-12-29 Thread Emerson Clarke
Ken, Yes you cannot have multiple threads within the same transaction, and you cannot pass a connection between threads. I think we have an undestanding about the performance situation, and we are getting to the real heart of the issue, which why it is not possible to have a single transaction,

Re: [sqlite] sqlite performance, locking & threadi

2006-12-29 Thread Ken
The test server.c code is an example that has a single thread that performs the DB interactions each client thread, communicates via a queue interface. Each client will get serialized into the DBserver thread and get its work done. Thus eliminating any lower level locking and mutexing inside th

Re: [sqlite] sqlite performance, locking & threadi

2006-12-29 Thread Emerson Clarke
OK, thanks Joe. I will look up those peculiarities as you suggested, im interested in understanding what they are. I agree with you, DRH has done a great job. Sqlite is a fantastic piece of software and and outstanding example of the way open source should be. It is small, efficient and incred

Re: [sqlite] sqlite performance, locking & threadi

2006-12-29 Thread Emerson Clarke
Ken, Thanks. Ok, i think i did miss the point with your test case. I assumed that it was always possible to perform multiple select, insert, delete, and update statements within a single transaction anyway. Which i guess relates to you last point. I dont see why if i have a single global tran

Re: [sqlite] sqlite performance, locking & threading

2006-12-29 Thread Joe Wilson
This old mailing list thread better describes this proposed algorithm to contain all open/close/lock/unlock activity in a single work thread: http://www.mail-archive.com/sqlite-users@sqlite.org/msg15852.html --- Joe Wilson <[EMAIL PROTECTED]> wrote: > As you already know, it's not just Linux -

Re: [sqlite] sqlite performance, locking & threading

2006-12-29 Thread Emerson Clarke
Richard, First let me say thank you for your earilier suggestion regarding the compile time options and overlaoding of the os routines. It was exactly the kind of information i was searching to for and i am looking into using that strategy at the moment. Sorry if ive managed to offend you, plea

Re: [sqlite] sqlite performance, locking & threadin

2006-12-29 Thread Joe Wilson
My last message cites some of the peculiarities of POSIX fcntl() locking. Search the SQLite mailing list archives for more detailed info as it pertains to SQLite. As for having a single unified (locking) model - SQLite already employs such a strategy as best as is possible given the portable natu

Re: [sqlite] sqlite performance, locking & threading

2006-12-29 Thread Joe Wilson
--- [EMAIL PROTECTED] wrote: > On some older versions of Linux, SQLite is unable to pass > database connections from one thread to another. But this > is a problem with the threading libraries used in those older > linux versions and is outside the control of SQLite. I do not > think this issue c

Re: [sqlite] sqlite performance, locking & threadin

2006-12-29 Thread Emerson Clarke
Joe, Im interested to know what those constraints are and why ? The only reason i mentioned shared memory is because it provides a platform and filesystem agnostic way of handling ipc. Obvioulsy i dont know the ins and outs of the locking process, but i just thought it would make sense to have

Re: [sqlite] sqlite performance, locking & threadin

2006-12-29 Thread Ken
Emerson, You just need to lock the entire transaction using a mutex before hand. That means each thread will have its own sqlite cache. Reread the sqlite locking and concurrency guide, you'll see that SQLITE is NOT a transaction based system such as Postgress/mysql/ oracle. It locks the

Re: [sqlite] sqlite performance, locking & threading

2006-12-29 Thread drh
"Emerson Clarke" <[EMAIL PROTECTED]> wrote: > Im not sure that the current situation of forced thread > incompatibility is better than leaving it up to users to manage the > threading. Usually it is assumed that a library is thread unsafe > unless otherwise specified. > > Developing multithreaded

Re: [sqlite] sqlite performance, locking & threading

2006-12-29 Thread Joe Wilson
--- Emerson Clarke <[EMAIL PROTECTED]> wrote: > Developing multithreaded applications is difficult, i wouldnt dispute > that. But i do dispute the wisdom of actively making a library > incompatible with threads. "Actively"? That's a bit much. There are constraints on the ability to pass SQLite

Re: [sqlite] sqlite performance, locking & threading

2006-12-29 Thread Emerson Clarke
In general worker threads is not an efficient solution to the problem even if parallelism is available. There is nothing to be gained by having one thread handing off A to another set of worker threads because you have to synchronise on whatever queue/list/pipe you use to transfer the information

Re: [sqlite] sqlite performance, locking & threading

2006-12-29 Thread Emerson Clarke
Im not sure that the current situation of forced thread incompatibility is better than leaving it up to users to manage the threading. Usually it is assumed that a library is thread unsafe unless otherwise specified. Developing multithreaded applications is difficult, i wouldnt dispute that. Bu

Re: [sqlite] sqlite performance, locking & threading

2006-12-28 Thread Roger Binns
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Emerson Clarke wrote: | For architectural reasons the above steps must be performed in that | order. This means that operations cant be separated or queued up in | the way that you suggested. Each step is dependent on the previous | step. I was look

Re: [sqlite] sqlite performance, locking & threading

2006-12-28 Thread Jay Sprenkle
On 12/27/06, Emerson Clarke <[EMAIL PROTECTED]> wrote: The first question is why database locking has been enabled on a per thread basis instead of per process so that the normal methods of thread synchronisation (mutexes, ciritcal sections) could not be used for maintaining consistency within th

Re: [sqlite] sqlite performance, locking & threadin

2006-12-28 Thread Emerson Clarke
Ken, Thanks i understand your example well. What im looking for is the ability to have multiple threads perform a write operation based on my mutex, not some internal sqlite write mutex. If i am managing the concurrency and performing correct synchronisation, why can i not have multiple threads

Re: [sqlite] sqlite performance, locking & threading

2006-12-28 Thread Emerson Clarke
The indexing process works like this. 1.) Open a document and parse its contents. 2.) Look up records in the first database based on the contents of the document, updating records where appropriate and inserting new ones. 3.) Transforming the document based on what was obtained from the first dat

Re: [sqlite] sqlite performance, locking & threading

2006-12-28 Thread Ken
Emerson, Is the Database on the same disk as the rest of the file operations? If so is it possible that you are I/O bound and causing seek issues due to i/o access patterns? Take a look at the test_server.c code in the sqlite/src directory. I used that as a basis to build a custom library

Re: [sqlite] sqlite performance, locking & threading

2006-12-28 Thread Roger Binns
Emerson Clarke wrote: The idea is that because i am accessing two databases, and doing several file system operations per document, there should be a large gain by using many threads. There is no actual indexing process, the whole structure is the index, but if anything the database operations t

Re: [sqlite] sqlite performance, locking & threading

2006-12-28 Thread John Stanton
I am curious as to how multiple threads would perform faster inserts into an Sqlite database, which is a single file plus the journal. Are you using a multiple processor machine? Emerson Clarke wrote: Roger, Thanks for the suggestions. I think using a worker thread and a queue would be equiv

Re: [sqlite] sqlite performance, locking & threading

2006-12-28 Thread Emerson Clarke
Roger, Thanks for the suggestions. I think using a worker thread and a queue would be equivalent to just running a single thread since it effectively makes the database operations synchronous. Although i can see what your driving at regarding the transactions every n records. The idea is that

Re: [sqlite] sqlite performance, locking & threading

2006-12-28 Thread drh
"Emerson Clarke" <[EMAIL PROTECTED]> wrote: > > The first question is why database locking has been enabled on a per thread > basis instead of per process so that the normal methods of thread > synchronisation (mutexes, ciritcal sections) could not be used for > maintaining consistency within the

Re: [sqlite] sqlite performance, locking & threading

2006-12-27 Thread Roger Binns
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Emerson Clarke wrote: | modified the api to ensure that each thread was given its own sqlite3 * | structure. I would assume that the actual indexing is the expensive part since it involves a lot of I/O (SQLite page size is 1KB). Why don't you do thi

[sqlite] sqlite performance, locking & threading

2006-12-27 Thread Emerson Clarke
I am developing a multithreaded C++ library which uses sqlite as an embedded database alongside the mysql client as a simple sql api. Both databases share a common interface which supports statements, prepared statements, recordsets, records, and transactions. As part of some research and testin

Re: [sqlite] SQLite Performance Issues

2006-10-23 Thread John Stanton
I built and use an application server which embeds Sqlite and processes web traffic. It is multi-threaded and can handle very many connections. It is very fast because it uses no IPC channels or process creation. It caches threads and reuses them rather than creating and killing them. It can

Re: [sqlite] SQLite Performance Issues

2006-10-23 Thread Christian Smith
James Mills uttered: Hi Richard, When I mean high-traffic I would imagine more than 1 hit/s. I do want to clear something up though (if you could): If a site using sqlite takes 700ms to load and there are two simultaneous (as simultaneous as one can get) hits to the site, say user A and user

Re: [sqlite] SQLite Performance Issues

2006-10-23 Thread James Mills
Hrmm also a couple of other things... In order to make a decision, I would need some way of running tests and simulations so I can come up with some numbers. Then scale that up and use it as an indicator for our decision. Do you have any tools that'll help with this ? cheers James -- -- -"Probl

Re: [sqlite] SQLite Performance Issues

2006-10-23 Thread James Mills
Hi Richard, I appreciate your feedback on the matter. I myself have used SQLite in many of my applications in the past for quite some years now. Most of them do indeed only write to or read from teh database for only fractions of a second. When I mean high-traffic I would imagine more than 1 hit/

Re: [sqlite] SQLite Performance Issues

2006-10-23 Thread drh
James Mills <[EMAIL PROTECTED]> wrote: > Hi Folks, > > I'm wanting to use SQLite in an embedded web application > that will serve as a proxy and possible serve up many > connections at once. I'm talking here of high-traffic > through this web app. > > Question is, how will SQLite perform under th

[sqlite] SQLite Performance Issues

2006-10-22 Thread James Mills
Hi Folks, I'm wanting to use SQLite in an embedded web application that will serve as a proxy and possible serve up many connections at once. I'm talking here of high-traffic through this web app. Question is, how will SQLite perform under these kinds of conditions ? I've been speaking to a few o

RE: [sqlite] sqlite performance questions.

2006-10-18 Thread Robert Simpson
> -Original Message- > From: Mohd Radzi Ibrahim [mailto:[EMAIL PROTECTED] > Sent: Wednesday, October 18, 2006 3:23 PM > To: sqlite-users@sqlite.org > Subject: Re: [sqlite] sqlite performance questions. > > Hi, > Could you tell me what breaks the transaction?

Re: [sqlite] sqlite performance questions.

2006-10-18 Thread Mohd Radzi Ibrahim
Hi, Could you tell me what breaks the transaction? Is create table/index within transaction breaks it? best regards, Radzi. - Original Message - From: "Isaac Raway" <[EMAIL PROTECTED]> To: Sent: Thursday, October 19, 2006 2:07 AM Subject: Re: [sqlite] sqlite perfo

Re: [sqlite] sqlite performance questions.

2006-10-18 Thread Isaac Raway
om: Andrew Cheyne [mailto:[EMAIL PROTECTED] > Sent: Wednesday, October 18, 2006 8:08 AM > To: sqlite-users@sqlite.org > Subject: [sqlite] sqlite performance questions. [snip] > I have then been writing some sample C programs making use > of the C API, > but have been very disappo

RE: [sqlite] sqlite performance questions.

2006-10-18 Thread Robert Simpson
> -Original Message- > From: Andrew Cheyne [mailto:[EMAIL PROTECTED] > Sent: Wednesday, October 18, 2006 8:08 AM > To: sqlite-users@sqlite.org > Subject: [sqlite] sqlite performance questions. [snip] > I have then been writing some sample C programs making use >

[sqlite] sqlite performance questions.

2006-10-18 Thread Andrew Cheyne
Hello, I¹m relatively new to SQLite and have a couple questions centered around performance. I am running SQLite 3.1.3 on Mac OSX 10.4 (this is the version that is pre-installed). I created a very simple database with one Table: sqlite3 test-db "create table Node (NodeId INTEGER PRIMARY KEY,

Re: [sqlite] SQLite performance for 10 Million Records

2006-06-26 Thread Martin Jenkins
Joe Wilson wrote: --- Martin Jenkins <[EMAIL PROTECTED]> wrote: For :memory: databases, long periods were observed where the VM size crept up but I/O write bytes did not, followed by periods where I/O bytes increased. If you use "PRAGMA temp_store=MEMORY" with your :memory: database you will

Re: [sqlite] SQLite performance for 10 Million Records

2006-06-26 Thread Joe Wilson
> I think some other factor is at play here. > SQLite 2.x's memory databases are still twice as fast at batch inserts > than either 3.x's disk-based databases or 2.x's disk-based databases > when the DB size is less than physical machine memory. I did some experimentation with an SQLite 2.8.17 :m

Re: [sqlite] SQLite performance for 10 Million Records

2006-06-26 Thread Joe Wilson
--- Martin Jenkins <[EMAIL PROTECTED]> wrote: > For :memory: databases, long periods > were observed where the VM size crept up but I/O write bytes did not, > followed by periods where I/O bytes increased. If you use "PRAGMA temp_store=MEMORY" with your :memory: database you will have no I/O wh

Re: [sqlite] SQLite performance for 10 Million Records

2006-06-26 Thread Manzoor Ilahi Tamimy
Hi, I am really really thankful to all the members of this group. The Discussion here was really very helpful for me and also for the others. I was not as much experienced as the other members who took part in this dicussion, but i worked hard ad spent a lot of time to find out why i am getti

Re: [sqlite] SQLite performance for 10 Million Records

2006-06-26 Thread Martin Jenkins
Dennis Cote wrote: Joe Wilson wrote: I think some other factor is at play here. Yes there is another factor at work here. [...] I suspect there are optimizations that could be made to the memory I/O routines to speed them up, they should at least be able to run slightly faster than file ba

Re: [sqlite] SQLite performance for 10 Million Records

2006-06-26 Thread Dennis Cote
Joe Wilson wrote: I think some other factor is at play here. SQLite 2.x's memory databases are still twice as fast at batch inserts than either 3.x's disk-based databases or 2.x's disk-based databases when the DB size is less than physical machine memory. Joe, Yes there is another factor

Re: [sqlite] SQLite performance for 10 Million Records

2006-06-23 Thread Joe Wilson
> On 23 Jun 2006, at 14:16, Dennis Cote wrote: > > Most of SQLite's disk I/O is actually going to the memory used for > > the operating system's disk cache, not directly to the disk. Hence > > its speed is not much different when using a disk based database > > than a memory based database. I

Re: [sqlite] SQLite performance for 10 Million Records

2006-06-23 Thread Joe Wilson
--- Manzoor Ilahi Tamimy <[EMAIL PROTECTED]> wrote: > DISK MODE > 3000 000 INSERTS 31 Seconds 96774 INSERTS / Sec > "SELECT * from t" 5 Seconds. > > MEMORY MODE > 3000 000 INSERTS 53 Seconds 56604 INSERTS / Sec > "SELECT * from t" 5 Seconds. > > Can I reduce the TIME of DISK mode or t

Re: [sqlite] SQLite performance for 10 Million Records

2006-06-23 Thread Eric Scouten
On 23 Jun 2006, at 14:16, Dennis Cote wrote: John Stanton wrote: Cacheing will drive you crazy. Very well put. Most of SQLite's disk I/O is actually going to the memory used for the operating system's disk cache, not directly to the disk. Hence its speed is not much different when using

Re: [sqlite] SQLite performance for 10 Million Records

2006-06-23 Thread Dennis Cote
John Stanton wrote: Cacheing will drive you crazy. Very well put. Most of SQLite's disk I/O is actually going to the memory used for the operating system's disk cache, not directly to the disk. Hence its speed is not much different when using a disk based database than a memory based databa

Re: [sqlite] SQLite performance for 10 Million Records

2006-06-23 Thread John Stanton
Cacheing will drive you crazy. CARTER-HITCHIN, David, GBM wrote: This is a modified version of the test code I posted to show that there was a small but definite SLOWDOWN when using :memory: databases compared to a database in a file on disk. It seems strange, but it is true. Use a disk

Re: [sqlite] SQLite performance for 10 Million Records

2006-06-23 Thread Eduardo
At 03:09 23/06/2006, you wrote: #include "stdafx.h" <> samples, t, samples / t); getch(); //* Here you should create index for table t. In your previous example, for hvh itm // Select Time check //*

RE: [sqlite] SQLite performance for 10 Million Records

2006-06-23 Thread CARTER-HITCHIN, David, GBM
> This is a modified version of the test code I posted to show > that there > was a small but definite SLOWDOWN when using :memory: > databases compared > to a database in a file on disk. It seems strange, but it is > true. Use a > disk file for best speed. If true, this is crazy. Memory

Re: [sqlite] SQLite performance for 10 Million Records

2006-06-23 Thread Dennis Cote
Manzoor Ilahi Tamimy wrote: I have tested my Code with the following PRAGMA and still not getting any change in time. //-- sqlite3_exec(db, "PRAGMA temp_store=2", NULL, NULL, NULL); sqlite3_exec(db, "PRAGMA synchronous=0", NULL, NULL,

Re: [sqlite] SQLite performance for 10 Million Records

2006-06-23 Thread Nikki Locke
Manzoor Ilahi Tamimy wrote: >sqlite3_exec(db, "create table t (a integer, b float, c text,d integer, > e float, f text, g float, h text)", NULL, NULL, NULL); > >sqlite3_exec(db, "begin transaction", NULL, NULL, NULL); > >sqlite3_prepare(db, "insert into t values (?, ?, ?, ?, ?

Re: [sqlite] SQLite performance for 10 Million Records

2006-06-22 Thread Manzoor Ilahi Tamimy
I have tested my Code with the following PRAGMA and still not getting any change in time. //-- sqlite3_exec(db, "PRAGMA temp_store=2", NULL, NULL, NULL); sqlite3_exec(db, "PRAGMA synchronous=0", NULL, NULL, NULL); sqlite3_exec(db, "PRAG

Re: [sqlite] SQLite performance for 10 Million Records

2006-06-20 Thread Dennis Cote
Manzoor Ilahi Tamimy wrote: Here Is The Schema For these Tables. CREATE TABLE HVH ( Field1 VARCHAR(8), IDC VARCHAR(4), Field3 VARCHAR(2), Field4 VARCHAR(4), Field5 VARCHAR(7), Field6 VARCHAR(8), Field7 VARCHAR(1), Field8 FLOAT); CREATE TABLE ITM( IDC VARCHAR(4),ITEMNAME VARCHAR(2

Re: [sqlite] SQLite performance for 10 Million Records

2006-06-20 Thread Eduardo
At 03:30 20/06/2006, you wrote: Here Is The Schema For these Tables. CREATE TABLE HVH ( Field1 VARCHAR(8), IDC VARCHAR(4), Field3 VARCHAR(2), Field4 VARCHAR(4), Field5 VARCHAR(7), Field6 VARCHAR(8), Field7 VARCHAR(1), Field8 FLOAT); CREATE TABLE ITM( IDC VARCHAR(4),

Re: [sqlite] SQLite performance for 10 Million Records

2006-06-20 Thread Derrell . Lipman
"Manzoor Ilahi Tamimy" <[EMAIL PROTECTED]> writes: > Here Is The Schema For these Tables. > > CREATE TABLE HVH ( > Field1 VARCHAR(8),IDC VARCHAR(4), > Field3 VARCHAR(2),Field4 VARCHAR(4), > Field5 VARCHAR(7),Field6 VARCHAR(8), > Field7 VARCHAR(1),Field8 FLOAT); > > CREATE TABL

Re: [sqlite] SQLite performance for 10 Million Records

2006-06-19 Thread Manzoor Ilahi Tamimy
Here Is The Schema For these Tables. CREATE TABLE HVH ( Field1 VARCHAR(8), IDC VARCHAR(4), Field3 VARCHAR(2), Field4 VARCHAR(4), Field5 VARCHAR(7), Field6 VARCHAR(8), Field7 VARCHAR(1), Field8 FLOAT); CREATE TABLE ITM( IDC VARCHAR(4),ITEMNAME VARCHAR(20), COLUMN3

Re: [sqlite] SQLite performance for 10 Million Records

2006-06-19 Thread Brett Wilson
count(*) is pretty slow in sqlite because it basically does select * and then counts the results. This means it's looking through your whole big file. You can come up with some tricks like keeping a separate count up-to-date with triggers. There have been some old threads on optimizing count that

Re: [sqlite] SQLite performance for 10 Million Records

2006-06-19 Thread Manzoor Ilahi Tamimy
Dear ALL, I am really thankful to Bill King, Micha Bieber , Derrell for your valuable suggestions. I was really confused that which way should I follow now, because I was sure that SQLite will work much better. when I got the suggestion about Firebird then again I went to the comparison page

Re: [sqlite] SQLite performance for 10 Million Records

2006-06-19 Thread Derrell . Lipman
Bill King <[EMAIL PROTECTED]> writes: > Manzoor Ilahi Tamimy wrote: > >> We are Using SQLite for one of our project. >> >>The Database Size is more than 500 MB. >>It contain one table and about 10 million Records. > Err, for that size, I'd recommend going something heavier, like > firebird. This

<    1   2   3   >