Re: [sqlite] Improving select distinct performance

2009-10-08 Thread John Elrick
Jason Freshwater wrote: > If this comes out garbled again the answer is java 1.6 on 1.8Ghz windows xp > > i get about 500,000 rows/second reading single character values via xerial > jdbc > Thanks, I did some additional experiments with only integers and got similar results. I had always bee

Re: [sqlite] Improving select distinct performance

2009-10-08 Thread Jason Freshwater
If this comes out garbled again the answer is java 1.6 on 1.8Ghz windows xp i get about 500,000 rows/second reading single character values via xerial jdbc > From: john.elr...@fenestra.com > Jason Freshwater wrote: > > 2. Read every record into my application and use a hashtable to > > do my own

Re: [sqlite] Improving select distinct performance

2009-10-08 Thread Jason Freshwater
<4acdf22b.2080...@fenestra.com> Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 > From: john.elr...@fenestra.com >> 2. Read every record into my application and use a hashtable to >> do my own select distinct. This is pretty good=2C a

Re: [sqlite] Improving select distinct performance

2009-10-08 Thread John Elrick
Jason Freshwater wrote: > 2. Read every record into my application and use a hashtable to > do my own select distinct. This is pretty good, about 0.6s to > achieve the distinct list (I can count each distinct value > along the way which is a useful side effect). Would you mind my asking what langu

Re: [sqlite] Improving select distinct performance

2009-10-08 Thread Jason Freshwater
> From: itandet...@mvps.org > Jason Freshwater wrote: > > For a 1 million row table with 8 distinct values of c the query > > time on my system (pretty typical modern laptop using precompiled > > sqlite 3.6.18) is about 2.8 seconds. For my requirements this is > > a bit too slow... > > I believe

Re: [sqlite] Improving select distinct performance

2009-10-08 Thread Igor Tandetnik
Jason Freshwater wrote: > The problem I keep running into is with expressions of the form > > select distinct c from t1 > > For a 1 million row table with 8 distinct values of c the query > time on my system (pretty typical modern laptop using precompiled > sqlite 3.6.18) is about 2.8 seconds

[sqlite] Improving select distinct performance

2009-10-08 Thread Jason Freshwater
Hi (Apologies for the long post), I am using sqlite for storing application data and settings. A typical simple scenario is to import a CSV file into sqlite and then present the user with various summaries of the data that it contains. Generally everything works really well; I can insert data