Re: [sqlite] Speeding up index creation

2011-09-05 Thread Baruch Burstein
This is unrelated, but I have never done any serious testing, and am wondering what the user/sys results are in all these performance tests. Is user I/O time and sys CPU time? On Sun, Sep 4, 2011 at 11:43 AM, Filip Navara filip.nav...@gmail.comwrote: Updated results against current trunk: CPU

Re: [sqlite] Speeding up index creation

2011-09-05 Thread Filip Navara
On Mon, Sep 5, 2011 at 9:39 AM, Baruch Burstein bmburst...@gmail.com wrote: This is unrelated, but I have never done any serious testing, and am wondering what the user/sys results are in all these performance tests. Is user I/O time and sys CPU time? User is the time spent in user-mode code,

Re: [sqlite] Speeding up index creation

2011-09-05 Thread Jaco Breitenbach
Dear all, Any speculation on if and how the new code would affect inserts into an indexed table? Jaco On 5 September 2011 09:15, Filip Navara filip.nav...@gmail.com wrote: On Mon, Sep 5, 2011 at 9:39 AM, Baruch Burstein bmburst...@gmail.com wrote: This is unrelated, but I have never done

Re: [sqlite] Speeding up index creation

2011-09-05 Thread Filip Navara
On Mon, Sep 5, 2011 at 1:18 PM, Jaco Breitenbach jjbreitenb...@gmail.com wrote: Dear all, Any speculation on if and how the new code would affect inserts into an indexed table? Jaco It doesn't affect them at all. The optimization specifically targets sorting large amount of data. Best

Re: [sqlite] Speeding up index creation

2011-09-04 Thread Filip Navara
Updated results against current trunk: CPU Time: user 82.009726 sys 129.636831 I rerun the profiler and looked at the results. Most of them looked sensible, but one thing still struck me. Once the new index is created in the VDBE Sorter it gets copied back into the new Btree using a loop with

Re: [sqlite] Speeding up index creation

2011-09-03 Thread Filip Navara
On Fri, Sep 2, 2011 at 6:04 PM, Dan Kennedy danielk1...@gmail.com wrote: On 09/02/2011 07:32 PM, Filip Navara wrote: On Fri, Sep 2, 2011 at 11:04 AM, Filip Navarafilip.nav...@gmail.com  wrote: *snip* The time to create an index on my 266 Mb experimental database is more than 9 minutes.

[sqlite] Speeding up index creation

2011-09-02 Thread Filip Navara
Hi, I'm experimenting with the new code in SQLite's trunk and measuring the improvements in index creation. The I/O required to create index and also the disk space requirements has reduced significantly, so the CREATE INDEX is now dominated by CPU time. Almost no memory is used for cache though,

Re: [sqlite] Speeding up index creation

2011-09-02 Thread Filip Navara
On Fri, Sep 2, 2011 at 11:04 AM, Filip Navara filip.nav...@gmail.com wrote: *snip* The time to create an index on my 266 Mb experimental database is more than 9 minutes. *snip* I erroneously measured the time with DEBUG build, so I've rerun the experiment with several SQLite versions:

Re: [sqlite] Speeding up index creation

2011-09-02 Thread Simon Slavin
On 2 Sep 2011, at 10:04am, Filip Navara wrote: The time to create an index on my 266 Mb experimental database is more than 9 minutes. The database is available at http://www.emclient.com/temp/mail_index.zip and the command I use to create the index is CREATE INDEX

Re: [sqlite] Speeding up index creation

2011-09-02 Thread Simon Slavin
On 2 Sep 2011, at 3:24pm, Simon Slavin wrote: On 2 Sep 2011, at 10:04am, Filip Navara wrote: The time to create an index on my 266 Mb experimental database is more than 9 minutes. The database is available at http://www.emclient.com/temp/mail_index.zip and the command I use to create the

Re: [sqlite] Speeding up index creation

2011-09-02 Thread Filip Navara
On Fri, Sep 2, 2011 at 5:23 PM, Simon Slavin slav...@bigfraud.org wrote: On 2 Sep 2011, at 3:24pm, Simon Slavin wrote: On 2 Sep 2011, at 10:04am, Filip Navara wrote: The time to create an index on my 266 Mb experimental database is more than 9 minutes. The database is available at

Re: [sqlite] Speeding up index creation

2011-09-02 Thread Simon Slavin
On 2 Sep 2011, at 4:38pm, Filip Navara wrote: With SQLite 3.7.7 and older the index creation takes eons since the file is in WAL mode and the journal file grows uncontrollably. Yeah. It was 5.1Gig, and the underlying database was only 266Meg. Heh. Simon.

Re: [sqlite] Speeding up index creation

2011-09-02 Thread Dan Kennedy
On 09/02/2011 09:24 PM, Simon Slavin wrote: On 2 Sep 2011, at 10:04am, Filip Navara wrote: The time to create an index on my 266 Mb experimental database is more than 9 minutes. The database is available at http://www.emclient.com/temp/mail_index.zip and the command I use to create the index

Re: [sqlite] Speeding up index creation

2011-09-02 Thread Dan Kennedy
On 09/02/2011 07:32 PM, Filip Navara wrote: On Fri, Sep 2, 2011 at 11:04 AM, Filip Navarafilip.nav...@gmail.com wrote: *snip* The time to create an index on my 266 Mb experimental database is more than 9 minutes. *snip* I erroneously measured the time with DEBUG build, so I've rerun the

Re: [sqlite] Speeding up index creation?

2008-05-24 Thread Stefan Arentz
On Fri, May 23, 2008 at 6:37 PM, Nemanja Čorlija [EMAIL PROTECTED] wrote: On Fri, May 23, 2008 at 4:20 PM, Stefan Arentz [EMAIL PROTECTED] wrote: I have an interesting problem. I need to generate a large table periodically. The table contains a unique SHA1 hash code and 6 integer values and

[sqlite] Speeding up index creation?

2008-05-23 Thread Stefan Arentz
I have an interesting problem. I need to generate a large table periodically. The table contains a unique SHA1 hash code and 6 integer values and has about 6 million rows. Generating this table is fast. I can set it up in less than 90 seconds on a slow iMac (2.16 Ghz and slow disk). The thing that

Re: [sqlite] Speeding up index creation?

2008-05-23 Thread Jay A. Kreibich
On Fri, May 23, 2008 at 10:20:45AM -0400, Stefan Arentz scratched on the wall: I have an interesting problem. I need to generate a large table periodically. The table contains a unique SHA1 hash code and 6 integer values and has about 6 million rows. Generating this table is fast. I can set it

Re: [sqlite] Speeding up index creation?

2008-05-23 Thread Nemanja Čorlija
On Fri, May 23, 2008 at 4:20 PM, Stefan Arentz [EMAIL PROTECTED] wrote: I have an interesting problem. I need to generate a large table periodically. The table contains a unique SHA1 hash code and 6 integer values and has about 6 million rows. Generating this table is fast. I can set it up in