Re: [sqlite] sqlite performance problem

2005-04-12 Thread Gé Weijers
Maksim Yevmenkin wrote: > >>>so, just with plain ascii file i get four times the speed i get with >>>sqlite. note that my c program will scale linearly with the size of >>>dataset (just like i see with sqlite). >>> >>> >> With anything related to computers, there are always tradeoffs - mos

Re: [sqlite] sqlite performance problem

2005-04-12 Thread Maksim Yevmenkin
John. > >i think, i know what is going on here. the problem is that every time > >i do an indexed scan sqlite has to > > > >1) fetch index pages > > > >2) fetch data pages that match "where" condition > > > >because both index and data are in the same file sqlite has to perform > >insane amount of

[sqlite] Copying a table between databases

2005-04-12 Thread Dennis Volodomanov
Hello all, What is the best way to copy (not move) a table between an existing database and a blank, attached database using SQLite v3? The COPY command is gone, what else can I use? Thank you. Dennis

Re: [sqlite] sqlite performance problem

2005-04-12 Thread Maksim Yevmenkin
> > SELECT count(*) FROM (SELECT a,n1 FROM data WHERE a <= 18234721 GROUP BY a); > > > > > time sqlite3 db < test.sql > > 30 > > 1024 > > 1417 > > 13.14u 1.06s 0:14.40 98.6% > > Have you tried doing the query like this: > > SELECT count(*) FROM (SELECT a,nl FROM data WHERE a-18234721<=0 GRO

Re: [sqlite] sqlite performance problem

2005-04-12 Thread John LeSueur
Maksim Yevmenkin wrote: i think, i know what is going on here. the problem is that every time i do an indexed scan sqlite has to 1) fetch index pages 2) fetch data pages that match "where" condition because both index and data are in the same file sqlite has to perform insane amount of seek() calls

Re: [sqlite] sqlite performance problem

2005-04-12 Thread D. Richard Hipp
On Tue, 2005-04-12 at 16:17 -0700, Maksim Yevmenkin wrote: > SELECT count(*) FROM (SELECT a,n1 FROM data WHERE a <= 18234721 GROUP BY a); > > > time sqlite3 db < test.sql > 30 > 1024 > 1417 > 13.14u 1.06s 0:14.40 98.6% > Have you tried doing the query like this: SELECT count(*) FROM (SEL

Re: [sqlite] sqlite performance problem

2005-04-12 Thread Maksim Yevmenkin
Thomas, > > with sum(n1) added query runs twice as slow. as i was told its because > > sqlite has to fetch data row. fine, but why its soo slow?! and it > >Because for each row it has to compute the aggregate key, find the > aggregator for that key and increment the sum for that aggregate key

Re: [sqlite] determing the primary key

2005-04-12 Thread Will Leshner
On 4/12/05, Thomas Briggs <[EMAIL PROTECTED]> wrote: > >Aliases rowid to rowid seems to work for me, i.e. Aha. I forgot about that trick. Thanks!

RE: [sqlite] determing the primary key

2005-04-12 Thread Thomas Briggs
Aliases rowid to rowid seems to work for me, i.e. SELECT rowid as rowid, primary_key_col -Tom > -Original Message- > From: Will Leshner [mailto:[EMAIL PROTECTED] > Sent: Tuesday, April 12, 2005 5:36 PM > To: sqlite-users@sqlite.org > Subject: [sqlite] determing the primary key >

RE: [sqlite] sqlite performance problem

2005-04-12 Thread Thomas Briggs
> with sum(n1) added query runs twice as slow. as i was told its because > sqlite has to fetch data row. fine, but why its soo slow?! and it Because for each row it has to compute the aggregate key, find the aggregator for that key and increment the sum for that aggregate key. That's a lot mo

Re: [sqlite] SQLITE 3 Make install errors -- DESTDIR variableunpopulated,...

2005-04-12 Thread Tiago Dionizio
On Apr 12, 2005 8:10 PM, Reid Thompson <[EMAIL PROTECTED]> wrote: > D. Richard Hipp wrote: > > On Tue, 2005-04-12 at 11:53 -0400, Reid Thompson wrote: > >> This issues still exists: > >> http://www.mail-archive.com/sqlite-users@sqlite.org/msg06857.html > >> > >> reid > >> > > > > Perhaps someone wh

[sqlite] determing the primary key

2005-04-12 Thread Will Leshner
I realize that in sqlite 3, if I construct a query with 'rowid' as one of the columns, and the table I am querying explicitly declares an integer primary key column, my results have the table's name for the column and not 'rowid'. What I'm wondering is if there is a good way to determine which of t

Re: [sqlite] sqlite performance problem

2005-04-12 Thread Maksim Yevmenkin
Dear SQLite users, consider this schema - sqlite> .schema data CREATE TABLE data( a INTEGER, b INTEGER, c CHAR, d INTEGER, e INTEGER, n1 FLOAT, n2 FLOAT ); CREATE INDEX data_by_a on data (a); data set - sqlite> select count(*) from data where a <= 18234721; 92

RE: [sqlite] SQLITE 3 Make install errors -- DESTDIR variableunpopulated,...

2005-04-12 Thread Reid Thompson
D. Richard Hipp wrote: > On Tue, 2005-04-12 at 11:53 -0400, Reid Thompson wrote: >> This issues still exists: >> http://www.mail-archive.com/sqlite-users@sqlite.org/msg06857.html >> >> reid >> > > Perhaps someone who has access to cygwin on Windows XP can > suggest a fix. (I do not own a copy o

Re: [sqlite] sqlite performance problem

2005-04-12 Thread Maksim Yevmenkin
Gé, thanks for the suggestion. unfortunately it did not make any difference :( below is the results. as you can see it takes 7+ seconds to "group by" 333,392 records and i'm grouping by column on which i have index. again, i'm not a database guy, but i think that is slow. perhaps someone can comm

Re: [sqlite] SQLITE 3 Make install errors -- DESTDIR variable unpopulated,...

2005-04-12 Thread D. Richard Hipp
On Tue, 2005-04-12 at 11:53 -0400, Reid Thompson wrote: > This issues still exists: > http://www.mail-archive.com/sqlite-users@sqlite.org/msg06857.html > > reid > Perhaps someone who has access to cygwin on Windows XP can suggest a fix. (I do not own a copy of WinXP so it would be rather diffic

[sqlite] SQLITE 3 Make install errors -- DESTDIR variable unpopulated,...

2005-04-12 Thread Reid Thompson
This issues still exists: http://www.mail-archive.com/sqlite-users@sqlite.org/msg06857.html reid

Re: [sqlite] sqlite performance problem

2005-04-12 Thread Gé Weijers
Maksim, Some things you could try: 1) increase cache memory You may be causing a lot of cache misses if the size of the query result is very large compared to the size of the cache. Index-based searches can cause multiple reloads of the same page because of a lack of locality in the cache. An in

Re: Fwd: [sqlite] assertion failing in pager.c :(

2005-04-12 Thread Dan Kennedy
The comment that was added at the same time as the assertion was commented out explains it pretty well. If behaviour when the system fails to allocate memory is important to you, I recommend upgrading to version 3. Version 3 handles this condition much more reliably than version 2 does. It do