Good day, Alexey. I thank you for your post as it sent me on a bit of a learning endeavor.
I have found that there is a framework to support user-developed stemmers already, with one built-in stemmer (Porter) coded into the DBMS by default. SQLite v3.7.4 facilitates the use FTS3 or FTS4. The Porter (stemmer) tokenizer does work well with FTS4 on v3.7.4 as described in the docs. Relevant info is detailed here: http://www.sqlite.org/fts3.html#tokenizer. The details on user implemented tokenizers is detailed here: http://www.sqlite.org/fts3.html#section_5_1. I was confused by one element in the docs, and will help to clarify it here, in case someone else struggles with it in the future. My existing code contained column names as parameters to the FTS creation constructs. [code] ... DBMS_Toss_Error( "Rebuilding FTS Tables as they differ from Production Table...", 0, 0, 0, 0) SQLite_Query := "Drop Table API_FTS_DB;`n" SQLite_Query .= "CREATE VIRTUAL TABLE API_FTS_DB USING FTS4 (Target_Name, Target_Context, Target_Description, Target_Content, Target_Link, Parent_Short, Parent_Name, Parent_Link, API_Version);`n" SQLite_Query .= "INSERT INTO API_FTS_DB SELECT * FROM API_DB;`n" SQLite_Query .= "SELECT count( * ) AS total_records FROM API_FTS_DB;`n" ... [/code] In the docs pointed to above, to implement a tokenizer, one's construct must resemble this: [code] ... CREATE VIRTUAL TABLE porter USING fts3(tokenize=porter); ... [/code] What I ended up getting working was this: [code]... SQLite_Query .= "CREATE VIRTUAL TABLE API_FTS_DB USING FTS4 (Target_Name, Target_Context, Target_Description, Target_Content, Target_Link, Parent_Short, Parent_Name, Parent_Link, API_Version, tokenize=porter);`n" ...[/code] In other words, the use of the tokenize parameter, along with column names parameters, is supported. Thanks, again, Alexey. Take care. -t -----Original Message----- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Alexey Pechnikov Sent: Wednesday, January 19, 2011 09:13 AM To: General Discussion of SQLite Database Subject: [sqlite] Patch: FTS3 Snawball Stemmer support Hello! May be this will be useful for somebody too. The code is not well tested yet but demonstrate the solution. http://sqlite.mobigroup.ru/wiki?name=FTS3+Snowball+Stemmer -- Best regards, Alexey Pechnikov. http://pechnikov.tel/ _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
