On Sat, Jul 23, 2011 at 11:00 PM, Richard Hipp <[email protected]> wrote:
> On Sat, Jul 23, 2011 at 1:01 PM, Abhinav Upadhyay <
> [email protected]> wrote:
>
>> Hi,
>>
>> I am using the Sqlite3 amalgamation. I am trying to register a custom
>> tokenizer with sqlite for my FTS application. The custom tokenizer is
>> in it's separate source file. I have included sqlite3.h header with
>> the tokenizer source but sqlite3.h does not contain the declaration of
>> the various structs like sqlite3_tokenizer_module etc. So what is the
>> usual way to resolve this ? These declarations are also not there in
>> sqlite3ext.h . Although I see them there in sqlite3.c but I am not
>> sure I want to include it ? What is the usual way to resolve this ?
>> May be import fts3_tokenizer.h from the sqlite3 source ?
>>
>
> Yes. Use fts3_tokenizer.h.
Thanks, That worked. Now, I am able to compile everything.
Next, I am having problem with using this tokenizer. I followed the
exampple from the FTS3 documentation page on the website and
registered the tokenizer using code like this:
const sqlite3_tokenizer_module *stopword_tokenizer_module;
sqlstr = "select fts3_tokenizer(:tokenizer_name, :tokenizer_ptr)";
rc = sqlite3_prepare_v2(db, sqlstr, -1, &stmt, NULL);
if (rc != SQLITE_OK) {
sqlite3_close(db);
sqlite3_shutdown();
return -1;
}
idx = sqlite3_bind_parameter_index(stmt, ":tokenizer_name");
rc = sqlite3_bind_text(stmt, idx, "my_tokenizer", -1, NULL);
if (rc != SQLITE_OK) {
fprintf(stderr, "%s\n", sqlite3_errmsg(db));
sqlite3_finalize(stmt);
return -1;
}
sqlite3Fts3MyTokenizerModule((const sqlite3_tokenizer_module
**)&my_tokenizer_module);
idx = sqlite3_bind_parameter_index(stmt, ":tokenizer_ptr");
rc = sqlite3_bind_blob(stmt, idx, &my_tokenizer_module,
sizeof(my_tokenizer_module), SQLITE_STATIC);
if (rc != SQLITE_OK) {
fprintf(stderr, "%s\n", sqlite3_errmsg(db));
sqlite3_finalize(stmt);
return -1;
}
rc = sqlite3_step(stmt);
if (rc != SQLITE_ROW) {
fprintf(stderr, "%s\n", sqlite3_errmsg(db));
sqlite3_finalize(stmt);
return -1;
}
sqlite3_finalize(stmt);
It is working fine till above. After executing the above statements, I
try to create an FTS table using this custom tokenizer, which also
seem to be getting created. The problem is coming up when I try to
insert data in the table. A simple statement like "insert into
my_table values(...)" is giving out errors:
unknown tokenizer: my_tokenizer
I am sure I am missing something here, but don't know what ?
Thanks
_______________________________________________
sqlite-users mailing list
[email protected]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
