Re: [sqlite] fts3 database grows

2008-09-17 Thread Holger Lembke
Scott, thanks again for the many lines of good hints and help. I think I will simply ignore the problem until I have real end user experience and see where it will or could end. The FullTextModul more or less has no whatever specific database interface, it is simply an abstract black hole that do

Re: [sqlite] fts3 database grows

2008-09-17 Thread Scott Hess
On Wed, Sep 17, 2008 at 12:15 AM, Holger Lembke <[EMAIL PROTECTED]> wrote: >>> So it reads: grows from 3 MB to 85 MB. >> That IS an important distinction! > > So what to do? > > a.) let it grow. > b.) do optimize() (but how often and how to detect when it is needed?) > c.) create a new database by

Re: [sqlite] fts3 database grows

2008-09-17 Thread Holger Lembke
Scott, >> So it reads: grows from 3 MB to 85 MB. > That IS an important distinction! So what to do? a.) let it grow. b.) do optimize() (but how often and how to detect when it is needed?) c.) create a new database by reindexing my data. (same problems as b.) d.) ignore the problem until complain

Re: [sqlite] fts3 database grows

2008-09-16 Thread Scott Hess
On Tue, Sep 16, 2008 at 12:02 AM, Holger Lembke <[EMAIL PROTECTED]> wrote: > [EMAIL PROTECTED] wrote: >> Database will grow from about 3.615 KB and continue growing. >> After 100 more FILLs (40.000 delete+insert actions.) it >> reaches a size of 85.683 KB. > > Just in case we are confused due to Eu

Re: [sqlite] fts3 database grows

2008-09-16 Thread Holger Lembke
[EMAIL PROTECTED] wrote: > Database will grow from about 3.615 KB and continue growing. > After 100 more FILLs (40.000 delete+insert actions.) it > reaches a size of 85.683 KB. Just in case we are confused due to European and American dots: those are European dots. So it reads: grows from 3 MB t

Re: [sqlite] fts3 database grows

2008-09-15 Thread Holger Lembke
Scott, > If you're worried about things at the level of kilobytes, > then you may be outside the core target for fts. optimize() > should work reasonably well for that size of dataset. I don't know if and about what I'm worried. I just noticed it and then thought about my poor users, that do use

Re: [sqlite] fts3 database grows

2008-09-15 Thread Scott Hess
The optimize() function was mainly written for privacy reasons (it drops all of the unused terms from the index). I'm a little concerned about promoting it as a general-purpose solution for size and performance, because for larger databases it can be a VERY expensive operation. I would rather fin

[sqlite] fts3 database grows

2008-09-15 Thread Holger Lembke
Scott, Thanks for the "full table scan hints", I'll change my design. > You're seeing two effects. I set up a small test. Its a bunch of windows shell scripts (good old dos stuff). http://www.lembke.eu/fts3.zip For security reasons the sqlite3.exe is missing. Run CREATE than FILL. Repeat FILL.

Re: [sqlite] fts3 database grows

2008-09-15 Thread Scott Hess
On Sun, Sep 14, 2008 at 11:01 AM, Holger Lembke <[EMAIL PROTECTED]> wrote: > (Hello all, new user... refreshing my rusty sql knowledge) > > I have a simple table created with > > create virtual table ft using fts3(id, content); > > and store id/text-pairs with > > delete from ft where (id="theid"

[sqlite] fts3 database grows

2008-09-14 Thread Holger Lembke
Hi, (Hello all, new user... refreshing my rusty sql knowledge) I have a simple table created with create virtual table ft using fts3(id, content); and store id/text-pairs with delete from ft where (id="theid"); insert into ft values ("theid", "sometext"); To test the table I have 400 fi