Re: [sqlite] FTS simple tokenizer with custom delimeters

[Richard Hipp]

On Sun, May 6, 2012 at 5:50 AM, Jos Groot Lipman  wrote:

> While looking around in the source of the simple tokenizer I found code
> that
> suggests custom delimeters can be specified (I want to exclude the
> underscore).
>
>
>
> http://www.sqlite.org/src/artifact/5c98225a53705e5ee34824087478cf477bdb7004?
> ln=76-87
>
> An indeed:
>  CREATE VIRTUAL TABLE ft USING fts3(title, body, tokenize=simple XX
> [&'\" *()./\\=,:;%<>-?!])
> seems to work fine.
>
> As far as I can tell this feature is undocumented which means I am not
> suppose to use it.
> Is this:
> - An oversight
> - For good reason as it is unstable
> - or: because the syntax might change in the near future?
>

Likely the reason is that we forgot that this feature even exists.  It
seems to have existed in the simple tokenizer, unchanged, since the
original introduction of FTS1 back in 2006.  The fact that it uses argv[1]
instead of argv[0]


>
> Also: I need to include the dummy XX as the delimeters are searched in
> argv[1] in stead of argv[0]. I cannot find what the argv[0] is supposed to
> do here. Any reason?
>

Probably this is a bug.  As far as I can tell, the alternative delimiter
feature of FTS1/2/3 has never been tested.  (We do not (yet) do
full-coverage testing of FTS.)  You may be the first person to ever use
this feature.  But it has been in the code for so long now that we dare not
change it for fear of breaking long-established programs.



> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] FTS simple tokenizer with custom delimeters

[Jos Groot Lipman]

While looking around in the source of the simple tokenizer I found code that
suggests custom delimeters can be specified (I want to exclude the
underscore).
 
 
http://www.sqlite.org/src/artifact/5c98225a53705e5ee34824087478cf477bdb7004?
ln=76-87
 
An indeed:
  CREATE VIRTUAL TABLE ft USING fts3(title, body, tokenize=simple XX
[&'\" *()./\\=,:;%<>-?!])
seems to work fine.
 
As far as I can tell this feature is undocumented which means I am not
suppose to use it.
Is this:
- An oversight
- For good reason as it is unstable
- or: because the syntax might change in the near future?
 
Also: I need to include the dummy XX as the delimeters are searched in
argv[1] in stead of argv[0]. I cannot find what the argv[0] is supposed to
do here. Any reason?
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users