On Sat, Jul 23, 2011 at 11:00 PM, Richard Hipp <d...@sqlite.org> wrote: > On Sat, Jul 23, 2011 at 1:01 PM, Abhinav Upadhyay < > er.abhinav.upadh...@gmail.com> wrote: > >> Hi, >> >> I am using the Sqlite3 amalgamation. I am trying to register a custom >> tokenizer with sqlite for my FTS application. The custom tokenizer is >> in it's separate source file. I have included sqlite3.h header with >> the tokenizer source but sqlite3.h does not contain the declaration of >> the various structs like sqlite3_tokenizer_module etc. So what is the >> usual way to resolve this ? These declarations are also not there in >> sqlite3ext.h . Although I see them there in sqlite3.c but I am not >> sure I want to include it ? What is the usual way to resolve this ? >> May be import fts3_tokenizer.h from the sqlite3 source ? >> > > Yes. Use fts3_tokenizer.h.
Thanks, That worked. Now, I am able to compile everything. Next, I am having problem with using this tokenizer. I followed the exampple from the FTS3 documentation page on the website and registered the tokenizer using code like this: const sqlite3_tokenizer_module *stopword_tokenizer_module; sqlstr = "select fts3_tokenizer(:tokenizer_name, :tokenizer_ptr)"; rc = sqlite3_prepare_v2(db, sqlstr, -1, &stmt, NULL); if (rc != SQLITE_OK) { sqlite3_close(db); sqlite3_shutdown(); return -1; } idx = sqlite3_bind_parameter_index(stmt, ":tokenizer_name"); rc = sqlite3_bind_text(stmt, idx, "my_tokenizer", -1, NULL); if (rc != SQLITE_OK) { fprintf(stderr, "%s\n", sqlite3_errmsg(db)); sqlite3_finalize(stmt); return -1; } sqlite3Fts3MyTokenizerModule((const sqlite3_tokenizer_module **)&my_tokenizer_module); idx = sqlite3_bind_parameter_index(stmt, ":tokenizer_ptr"); rc = sqlite3_bind_blob(stmt, idx, &my_tokenizer_module, sizeof(my_tokenizer_module), SQLITE_STATIC); if (rc != SQLITE_OK) { fprintf(stderr, "%s\n", sqlite3_errmsg(db)); sqlite3_finalize(stmt); return -1; } rc = sqlite3_step(stmt); if (rc != SQLITE_ROW) { fprintf(stderr, "%s\n", sqlite3_errmsg(db)); sqlite3_finalize(stmt); return -1; } sqlite3_finalize(stmt); It is working fine till above. After executing the above statements, I try to create an FTS table using this custom tokenizer, which also seem to be getting created. The problem is coming up when I try to insert data in the table. A simple statement like "insert into my_table values(...)" is giving out errors: unknown tokenizer: my_tokenizer I am sure I am missing something here, but don't know what ? Thanks _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users