Re: [sqlite] 50% faster than 3.7.17

2014-10-09 Thread David Woodhouse
On Thu, 2014-09-25 at 11:13 +0100, David Woodhouse wrote: I suggested a couple of specific optimisations which the query planner might be able to make, which should hopefully have benefits wider than just my own use case. Are those not viable? I'm preparing to commit a workaround to

Re: [sqlite] 50% faster than 3.7.17

2014-10-09 Thread Dan Kennedy
On 10/09/2014 04:38 PM, David Woodhouse wrote: On Thu, 2014-09-25 at 11:13 +0100, David Woodhouse wrote: I suggested a couple of specific optimisations which the query planner might be able to make, which should hopefully have benefits wider than just my own use case. Are those not viable? I'm

Re: [sqlite] 50% faster than 3.7.17

2014-09-30 Thread jose isaias cabrera
Simon Slavin wrote... On 24 Sep 2014, at 2:13pm, jose isaias cabrera jic...@cinops.xerox.com wrote: This would be a nice set of options. On my case, I would set all connections to our project to be max_performance, as it is what we need. Just thinking out loud. How much max is max ?

Re: [sqlite] 50% faster than 3.7.17

2014-09-25 Thread Stephen Chrzanowski
In the vein of configurability, and in a day dream I just had, it would be nice (But probably not possible as there could be compiler directives you can't use at the same time) that we could have a single output DLL/SO/whatever dumped from the compiler that had everything available, then, via

Re: [sqlite] 50% faster than 3.7.17

2014-09-25 Thread David Woodhouse
On Wed, 2014-09-24 at 19:36 -0600, Keith Medcalf wrote: Interesting. From that code you might want to try something like this: SELECT uid, vcard, bdata FROM folder_id WHERE uid in ( select uid FROM email_list where value like 'p%' union select uid from

Re: [sqlite] 50% faster than 3.7.17

2014-09-25 Thread David Woodhouse
On Tue, 2014-09-23 at 17:48 +0100, David Woodhouse wrote: That looks really promising; thanks for all this work. Tristan, you have a comprehensive set of benchmarks for Evolution's addressbook; is it possible for someone else to run those or would it take more of your time to babysit than it

Re: [sqlite] 50% faster than 3.7.17

2014-09-24 Thread David Woodhouse
On Fri, 2014-09-19 at 21:14 -0400, Richard Hipp wrote: The 50% faster number above is not about better query plans. Speaking of better query plans, though... here's a query which takes about 1700ms on my data set, followed by a couple of optimisations which seem like they might be generically

Re: [sqlite] 50% faster than 3.7.17

2014-09-24 Thread Keith Medcalf
Would it not be more efficient to skip the join altogether since all you want is the list of uid's, and assuming that you have maintained the referential integrity of your database mail_list(list_uid) references main(uid)? SELECT list_uid FROM mail_list WHERE email LIKE 'al%' UNION SELECT

Re: [sqlite] 50% faster than 3.7.17

2014-09-24 Thread jose isaias cabrera
Roger Binns wrote... On 22/09/14 10:48, Richard Hipp wrote: But if you have any new ideas on how we can further reduce the I/O, we'd love to hear from you. The single biggest problem for me is defaults. SQLite supports memory mapped i/o which has many advantages. The stat4 analyze does a

Re: [sqlite] 50% faster than 3.7.17

2014-09-24 Thread Simon Slavin
On 24 Sep 2014, at 2:13pm, jose isaias cabrera jic...@cinops.xerox.com wrote: This would be a nice set of options. On my case, I would set all connections to our project to be max_performance, as it is what we need. Just thinking out loud. How much max is max ? Are you willing to give up

Re: [sqlite] 50% faster than 3.7.17

2014-09-24 Thread David Woodhouse
On Wed, 2014-09-24 at 06:13 -0600, Keith Medcalf wrote: Would it not be more efficient to skip the join altogether since all you want is the list of uid's, and assuming that you have maintained the referential integrity of your database mail_list(list_uid) references main(uid)? SELECT

Re: [sqlite] 50% faster than 3.7.17

2014-09-24 Thread Roger Binns
On 24/09/14 06:19, Simon Slavin wrote: How much max is max ? Not giving up ACID. But for example stat4 is better than the default stat1. Memory mapping (especially on 64 bit) is great. So is WAL. All are off by default. If you want to give up ACID then you should really be on your own to

Re: [sqlite] 50% faster than 3.7.17

2014-09-24 Thread Keith Medcalf
] Sent: Wednesday, 24 September, 2014 07:25 To: Keith Medcalf Cc: General Discussion of SQLite Database Subject: Re: [sqlite] 50% faster than 3.7.17 On Wed, 2014-09-24 at 06:13 -0600, Keith Medcalf wrote: Would it not be more efficient to skip the join altogether since all you want is the list

Re: [sqlite] 50% faster than 3.7.17

2014-09-23 Thread Donald Shepherd
Are any of these improvements specifically in the area of the online backup API, or are they more in the general running of SQLite? On 20 September 2014 11:14, Richard Hipp d...@sqlite.org wrote: The latest SQLite 3.8.7 alpha version (available on the download page

Re: [sqlite] 50% faster than 3.7.17

2014-09-23 Thread Richard Hipp
On Tue, Sep 23, 2014 at 2:33 AM, Donald Shepherd donald.sheph...@gmail.com wrote: Are any of these improvements specifically in the area of the online backup API, or are they more in the general running of SQLite? The speedtest1 benchmark does not use the backup API. However, many of the

Re: [sqlite] 50% faster than 3.7.17

2014-09-23 Thread David Woodhouse
On Fri, 2014-09-19 at 21:14 -0400, Richard Hipp wrote: The latest SQLite 3.8.7 alpha version (available on the download page http://www.sqlite.org/download.html) is 50% faster than the 3.7.17 release from 16 months ago. That is to say, it does 50% more work using the same number of CPU

Re: [sqlite] 50% faster than 3.7.17

2014-09-23 Thread Roger Binns
On 22/09/14 10:48, Richard Hipp wrote: But if you have any new ideas on how we can further reduce the I/O, we'd love to hear from you. The single biggest problem for me is defaults. SQLite supports memory mapped i/o which has many advantages. The stat4 analyze does a really good job. WAL

Re: [sqlite] 50% faster than 3.7.17

2014-09-22 Thread Valentin Davydov
On Fri, Sep 19, 2014 at 09:14:17PM -0400, Richard Hipp wrote: The latest SQLite 3.8.7 alpha version (available on the download page http://www.sqlite.org/download.html) is 50% faster than the 3.7.17 release from 16 months ago. That is to say, it does 50% more work using the same number of

Re: [sqlite] 50% faster than 3.7.17

2014-09-22 Thread big stone
Hi, This 3.8.7alpha release seems to bring about 5% win from 3.8.6 , on my particular SQL test case. Question : PRAGMA threads=2 didn't bring any speed-up on my 2 true cores machine. Did I miss a compilation option ? ___ sqlite-users mailing list

Re: [sqlite] 50% faster than 3.7.17

2014-09-22 Thread Richard Hipp
On Mon, Sep 22, 2014 at 1:43 PM, big stone stonebi...@gmail.com wrote: Hi, This 3.8.7alpha release seems to bring about 5% win from 3.8.6 , on my particular SQL test case. Question : PRAGMA threads=2 didn't bring any speed-up on my 2 true cores machine. Did I miss a compilation option ?

Re: [sqlite] 50% faster than 3.7.17

2014-09-22 Thread Richard Hipp
On Mon, Sep 22, 2014 at 8:29 AM, Valentin Davydov sqlite-u...@soi.spb.ru wrote: On Fri, Sep 19, 2014 at 09:14:17PM -0400, Richard Hipp wrote: The latest SQLite 3.8.7 alpha version (available on the download page http://www.sqlite.org/download.html) is 50% faster than the 3.7.17 release

Re: [sqlite] 50% faster than 3.7.17

2014-09-22 Thread big stone
ok, Nearly all the time is spent in a big 'CTE' select. So maybe this sort of ugly CTE query is not threadable. with f0_k as (SELECT f.rowid as nof2, f.*, p.Dn, p.g1, p.g2, p.w, p.other FROM F0 AS f, Pt AS p WHERE f.Io =p.Io) ,F0_mcalc as (SELECT f.*, p.*, (case when Priority=999 then Cg

Re: [sqlite] 50% faster than 3.7.17

2014-09-20 Thread tonyp
In trying to see if the new version breaks any of my queries, I ran several of them repeatedly, and they all appear to have produced the expected output. The only thing I noticed which maybe of interest in relation to speed performance was (with .timer on) that although the first two run time

Re: [sqlite] 50% faster than 3.7.17

2014-09-20 Thread Richard Hipp
On Sat, Sep 20, 2014 at 1:34 PM, to...@acm.org wrote: In trying to see if the new version breaks any of my queries, I ran several of them repeatedly, and they all appear to have produced the expected output. The only thing I noticed which maybe of interest in relation to speed performance

[sqlite] 50% faster than 3.7.17

2014-09-19 Thread Richard Hipp
The latest SQLite 3.8.7 alpha version (available on the download page http://www.sqlite.org/download.html) is 50% faster than the 3.7.17 release from 16 months ago. That is to say, it does 50% more work using the same number of CPU cycles. This performance gain is over and above the query

Re: [sqlite] 50% faster than 3.7.17

2014-09-19 Thread jose isaias cabrera
Richard Hipp wrote... The latest SQLite 3.8.7 alpha version (available on the download page http://www.sqlite.org/download.html) is 50% faster than the 3.7.17 release from 16 months ago. That is to say, it does 50% more work using the same number of CPU cycles. This performance gain is over

Re: [sqlite] 50% faster than 3.7.17

2014-09-19 Thread Stephen Chrzanowski
I, as well, wish to thank you for this tool Dr. Hipp. I've never published a public application using this engine, but, at my place of employment, where my primary responsibility is to just monitor servers world wide, I've coded a few tidbit web and desktop applications that have made my job SO