Re: [sqlite] FTS3/4 merge function behaviour when deleting rows

2014-05-13 Thread andrewmo
On 11 May 2014 00:01, Scott Robison-2 [via SQLite] ml-node+s1065341n75608...@n5.nabble.com wrote: I suspect the primary use case it was designed and tested for (and in fact the way we use it at my place of employment) was more for only growing datasets and less for an environment where stuff

Re: [sqlite] FTS3/4 merge function behaviour when deleting rows

2014-05-10 Thread Scott Robison
On Fri, May 9, 2014 at 4:20 AM, Andrew Moss curioussq...@googlemail.comwrote: I am really surprised that FTS behaves this way. To my mind this is a bug in the FTS extension that makes it unusable for many applications. Was anyone else aware of this problem or made attempts at resolving it? I

Re: [sqlite] FTS3/4 merge function behaviour when deleting rows

2014-05-09 Thread Andrew Moss
I am now totally convinced that FTS3/4 does not work for this usage model. If you are deleting and inserting documents, the size of the FTS index will grow in a linear manner with no limit no matter what you do with the merge command (when you run it, what parameters you provide). I have

Re: [sqlite] FTS3/4 merge function behaviour when deleting rows

2014-05-05 Thread Andrew Moss
On 2 May 2014 10:22, Dan Kennedy danielk1...@gmail.com wrote: A query like: SELECT level, count(*) AS ntree FROM yourftstablename_segdir; will tell you how many b-trees there currently are at each level. Which might help you figure out what is going on and when you might expect a merge

Re: [sqlite] FTS3/4 merge function behaviour when deleting rows

2014-05-02 Thread Dan Kennedy
On 05/01/2014 03:30 PM, andrewmo wrote: We are using the FTS3 extension to sqlite to store large numbers of short (~300 byte) documents. This is working very well and providing us with very fast text search, but the behaviour around deletion of documents has me confused. Our system must control

Re: [sqlite] FTS3/4 merge function behaviour when deleting rows

2014-05-02 Thread Andrew Moss
On 2 May 2014 07:57, Dan Kennedy danielk1...@gmail.com wrote: On 05/01/2014 03:30 PM, andrewmo wrote: We are using the FTS3 extension to sqlite to store large numbers of short (~300 byte) documents. This is working very well and providing us with very fast text search, but the behaviour

Re: [sqlite] FTS3/4 merge function behaviour when deleting rows

2014-05-02 Thread Dan Kennedy
On 05/02/2014 04:13 PM, Andrew Moss wrote: On 2 May 2014 07:57, Dan Kennedy danielk1...@gmail.com wrote: On 05/01/2014 03:30 PM, andrewmo wrote: We are using the FTS3 extension to sqlite to store large numbers of short (~300 byte) documents. This is working very well and providing us with

Re: [sqlite] FTS3/4 merge function behaviour when deleting rows

2014-05-02 Thread Stadin, Benjamin
Is the delete marker also set on old keys on UPDATE? Or just DELETE - INSERT? I ran into the ever-growing FTS index issue last year. I’m creating DB diffs which also contain some FTS3/4 tables. The tables get constantly updated for the checksum. The DBs were always vacuum’ed, but the growing FTS

Re: [sqlite] FTS3/4 merge function behaviour when deleting rows

2014-05-02 Thread Dan Kennedy
On 05/02/2014 07:30 PM, Stadin, Benjamin wrote: Is the delete marker also set on old keys on UPDATE? Or just DELETE - INSERT? Internally, FTS implements an UPDATE as a DELETE followed by an INSERT. Dan. I ran into the ever-growing FTS index issue last year. I’m creating DB diffs which

[sqlite] FTS3/4 merge function behaviour when deleting rows

2014-05-01 Thread andrewmo
We are using the FTS3 extension to sqlite to store large numbers of short (~300 byte) documents. This is working very well and providing us with very fast text search, but the behaviour around deletion of documents has me confused. Our system must control the the size of the database and will