[sqlite] sqlite3_auto_extension + custom FTS5 tokenizer
I?m currently implementing a custom FTS5 tokenizer which I?d like to automatically register for each db connection. So I tried to register an extension hook with sqlite3_auto_extension but when my code is called the FTS_* modules have not been initialized, because sqlite3Fts5Init() is called *after* sqlite3AutoLoadExtensions(db). Therefore registering the tokenizer is not possible at this stage, since the fts5() function is not defined yet. Is there another way? I can?t use dynamic extensions so need to use sqlite3_auto_extension or something similar. thanks, ? ?Jan
[sqlite] sqlite3_auto_extension + custom FTS5 tokenizer
On 05/11/2016 05:24 PM, Jan Berkel wrote: > I?m currently implementing a custom FTS5 tokenizer which I?d like to > automatically register for each db connection. > > So I tried to register an extension hook with sqlite3_auto_extension but when > my code is called the FTS_* modules have not been initialized, because > sqlite3Fts5Init() is called *after* sqlite3AutoLoadExtensions(db). > > Therefore registering the tokenizer is not possible at this stage, since the > fts5() function is not defined yet. > > Is there another way? I can?t use dynamic extensions so need to use > sqlite3_auto_extension or something similar. That sounds like a problem. I think you could: * build SQLite without SQLITE_ENABLE_FTS5, * include the file "fts5.c" in the application build, and * call sqlite3_fts5_init() from within the auto_extension() callback before creating the tokenizer. To generate fts5.c, grab the full source package (either from here - http://sqlite.org/2016/sqlite-src-3120200.zip - or from fossil) and run "./configure && make fts5.c". You can't just call sqlite3_fts5_init() from the auto_extension() callback in an SQLITE_ENABLE_FTS5 build, as the call to sqlite3Fts5Init() made later will replace the existing fts5 module with a new one - one that does not have your custom tokenizer registered with it. I think we should probably change that last bit so that calling sqlite3_fts5_init() on a db handle that already has an fts5 extension registered with it is a no-op. That might not happen until after the 3.13 release though. Dan.
[sqlite] sqlite3_auto_extension + custom FTS5 tokenizer
> I?m currently implementing a custom FTS5 tokenizer which I?d like to > automatically register for each db connection. > > So I tried to register an extension hook with sqlite3_auto_extension but when > my code is called the FTS_* modules have not been initialized, because > sqlite3Fts5Init() is called *after* sqlite3AutoLoadExtensions(db). > > Therefore registering the tokenizer is not possible at this stage, since the > fts5() function is not defined yet. > > Is there another way? I can?t use dynamic extensions so need to use > sqlite3_auto_extension or something similar. That sounds like a problem. I think you could: * build SQLite without SQLITE_ENABLE_FTS5, * include the file "fts5.c" in the application build, and * call sqlite3_fts5_init() from within the auto_extension() callback before creating the tokenizer. You can't just call sqlite3_fts5_init() from the auto_extension() callback in an SQLITE_ENABLE_FTS5 build, as the call to sqlite3Fts5Init() made later will replace the existing fts5 module with a new one - one that does not have your custom tokenizer registered with it. Dan, thank you for your suggestions. I had trouble compiling fts5.c without also defining -DSQLITE_CORE which meant that sqlite3_fts5_init was not defined so I just defined and called?sqlite3Fts5Init(), but everything worked as expected. I think we should probably change that last bit so that calling? sqlite3_fts5_init() on a db handle that already has an fts5 extension? registered with it is a no-op. That might not happen until after the? 3.13 release though.? I?m happy to contribute a patch but can't quite figure out the development process / guidelines. Also, wouldn?t it be easier to defer sqlite3AutoLoadExtensions() after all built-in extensions have been loaded? ? ?Jan ? ? ?
