Re: [sqlite] NFS Query Performance

2005-04-21 Thread William Hachfeld
On Thu, Apr 21, 2005 at 11:49:21AM +0100, Christian Smith wrote: > I know the snippet above is from a test program, but don't ever put things > like sqlite3_exec() inside an assert()! I hope you're (OP) not doing this in > your application. I realize that evaluation of assert() is short-circuited

Re: [sqlite] NFS Query Performance

2005-04-21 Thread Christian Smith
On Wed, 20 Apr 2005, Klint Gore wrote: >On Tue, 19 Apr 2005 12:40:51 -0500, William Hachfeld <[EMAIL PROTECTED]> wrote: >> uint64_t t_stop = Now(); >> >> assert(sqlite3_exec(handle, "COMMIT TRANSACTION;", >> NULL, NULL, NULL) == SQLITE_OK); > >What happens to your inse

Re: [sqlite] NFS Query Performance

2005-04-20 Thread William Hachfeld
Good suggestions, guys! First, I modified the previously-posted test program to time the insertion rate outside of the transaction (i.e. Kilnt's suggestion). I left the creation of the table, however, outside of the timed insertion loop. In other words: Create Table Begin Timing Begi

Re: [sqlite] NFS Query Performance

2005-04-19 Thread Klint Gore
On Tue, 19 Apr 2005 12:40:51 -0500, William Hachfeld <[EMAIL PROTECTED]> wrote: > uint64_t t_stop = Now(); > > assert(sqlite3_exec(handle, "COMMIT TRANSACTION;", > NULL, NULL, NULL) == SQLITE_OK); What happens to your insert times if you swap the above 2 lines? If y

Re: [sqlite] NFS Query Performance

2005-04-19 Thread Dan Kennedy
What happens if you wrap the SELECT statements in a transaction in the same way as you have the INSERTs? --- William Hachfeld <[EMAIL PROTECTED]> wrote: > > Heh everyone! > > Can anyone explain to me why SQLite shows such poor query performance when the > database is located on an NFS-mounted

RE: [sqlite] NFS Query Performance

2005-04-19 Thread Cariotoglou Mike
mailto:[EMAIL PROTECTED] > Sent: Tuesday, April 19, 2005 10:18 PM > To: sqlite-users@sqlite.org > Subject: Re: [sqlite] NFS Query Performance > > On Tue, Apr 19, 2005 at 11:58:11AM -0700, Ted Unangst wrote: > > If you don't need locking, and it sounds like you don't

Re: [sqlite] NFS Query Performance

2005-04-19 Thread William Hachfeld
On Tue, Apr 19, 2005 at 11:58:11AM -0700, Ted Unangst wrote: > If you don't need locking, and it sounds like you don't, just neuter the > fcntl calls in os_unix.c. In fact, there's a #define and comment for DJGPP > that does exactly what you want. Hmmm. Thanks for the suggestion, Ted. I think I'l

Re: [sqlite] NFS Query Performance

2005-04-19 Thread Ted Unangst
William Hachfeld wrote: Nope. that isn't going to be an option for me. Preserving the "zero configuration" and "single file" environment is of utmost importance in our application. If you don't need locking, and it sounds like you don't, just neuter the fcntl calls in os_unix.c. In fact, there's

Re: [sqlite] NFS Query Performance

2005-04-19 Thread Jay Sprenkle
Any chance you can write a really simple shell script/C/Perl program to do a bunch of lock/unlocks and benchmark it? > Possibly. Although as I noted, our "real" application is showing a speed > differential of about 50x. Much more than can be attributed to simple transfer > rates. But it sounds li

Re: [sqlite] NFS Query Performance

2005-04-19 Thread William Hachfeld
On Tue, Apr 19, 2005 at 02:23:44PM -0400, Griggs, Donald wrote: > I'm not sure how your network is set up, but if you're using 100mbps > ethernet, the full- table-scan queries are only 4-5 times slower over the > net. Wouldn't that be the same order of magnitude as the ratio of link > speed to AT

Re: [sqlite] NFS Query Performance

2005-04-19 Thread William Hachfeld
On Tue, Apr 19, 2005 at 01:20:35PM -0500, Kurt Welgehausen wrote: > SQLite is not a client-server database. Yup. I realize that. > In the worst case, when you query a database file over NFS, all the tables in > your query are sent over the network to be processed on your cpu; so if you > want to

Re: [sqlite] NFS Query Performance

2005-04-19 Thread Jay Sprenkle
On 4/19/05, Kurt Welgehausen <[EMAIL PROTECTED]> wrote: > SQLite is not a client-server database. In the worst case, > when you query a database file over NFS, all the tables in > your query are sent over the network to be processed on > your cpu; so if you want to select one row from a table of >

RE: [sqlite] NFS Query Performance

2005-04-19 Thread Griggs, Donald
I'm not sure how your network is set up, but if you're using 100mbps ethernet, the full- table-scan queries are only 4-5 times slower over the net. Wouldn't that be the same order of magnitude as the ratio of link speed to ATA local disk bus speed? As for the inserts, if you're achieving over 36

Re: [sqlite] NFS Query Performance

2005-04-19 Thread Kurt Welgehausen
SQLite is not a client-server database. In the worst case, when you query a database file over NFS, all the tables in your query are sent over the network to be processed on your cpu; so if you want to select one row from a table of a million rows, the million rows are retrieved over the network (i

Re: [sqlite] NFS Query Performance

2005-04-19 Thread Jay Sprenkle
That sounds right. I might not understand what you're doing so it's just a SWAG ;) On 4/19/05, William Hachfeld <[EMAIL PROTECTED]> wrote: > On Tue, Apr 19, 2005 at 01:07:03PM -0500, Jay Sprenkle wrote: > > On a system we developed several years ago (non sqlite) we noted that > > obtaining locks

Re: [sqlite] NFS Query Performance

2005-04-19 Thread William Hachfeld
On Tue, Apr 19, 2005 at 01:07:03PM -0500, Jay Sprenkle wrote: > On a system we developed several years ago (non sqlite) we noted that > obtaining locks on an NFS mounted file was VERY slow. I *think* it's trying > to establish a lock then waiting at least the propagation delay of your > network to

Re: [sqlite] NFS Query Performance

2005-04-19 Thread Jay Sprenkle
On a system we developed several years ago (non sqlite) we noted that obtaining locks on an NFS mounted file was VERY slow. I *think* it's trying to establish a lock then waiting at least the propagation delay of your network to ensure there was no collision with another process. Doing anything ov

[sqlite] NFS Query Performance

2005-04-19 Thread William Hachfeld
Heh everyone! Can anyone explain to me why SQLite shows such poor query performance when the database is located on an NFS-mounted file system? When running with a single process accessing the database, I'm finding that my insertion rates are similar to local disk, but my query performance is muc