Ah, Thanks. That makes sense. My tokenizer extracts text-node values from XHTML content and delegates the tokenizing of each node value to the Porter tokenizer. Since the RHS of the MATCH operator will never be XHTML, I should be able to just defer the tokenizing of that input directly to the Porter tokenizer? And I can do that by registering a custom tokenizer-module in my read-only application that specifies the Porter functions, but has the name of the custom tokenizer I used to index the XHTML data.
Does that seem reasonable? Nick Hodapp D. Richard Hipp wrote: > > > On Dec 16, 2009, at 11:15 AM, D. Richard Hipp wrote: > >> >> On Dec 16, 2009, at 11:12 AM, Nick Hodapp wrote: >> >>> I wrote a custom tokenizer for fts3, and I intend it to be used >>> when I >>> populate my database. >>> >>> I'd rather not ship the tokenizer in my embedded application which >>> accesses >>> the database in read-only mode. But it seems that fts3 checks for >>> the >>> existence of the tokenizer even when accessing a fts3 table for a >>> read-only >>> query. >>> >>> Is it possible to change this behavior (or, is the tokenizer actually >>> required for a read-only query?) >>> >>> If I register a dummy tokenizer with the same name in my read-only >>> application, would that work? >> >> >> The tokenizer is needed for both reading and writing. It is needing >> to parse the LHS of the MATCH operator when reading. > > Correction: ...the RHS of the MATCH operator.... > >> >> You *must* register exactly the same tokenizer when reading as you >> used for writing or your queries will not work. >> >> D. Richard Hipp >> d...@hwaci.com >> >> >> >> _______________________________________________ >> sqlite-users mailing list >> sqlite-users@sqlite.org >> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > > D. Richard Hipp > d...@hwaci.com > > > > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > > -- View this message in context: http://old.nabble.com/custom-fts3-tokenizer%2C-used-in-read-only--tp26813756p26817624.html Sent from the SQLite mailing list archive at Nabble.com. _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
