[sqlite] sqlite3_auto_extension + custom FTS5 tokenizer

[Jan Berkel]

> I?m currently implementing a custom FTS5 tokenizer which I?d like to 
> automatically register for each db connection.  
>  
> So I tried to register an extension hook with sqlite3_auto_extension but when 
> my code is called the FTS_* modules have not been initialized, because 
> sqlite3Fts5Init() is called *after* sqlite3AutoLoadExtensions(db).  
>  
> Therefore registering the tokenizer is not possible at this stage, since the 
> fts5() function is not defined yet.  
>  
> Is there another way? I can?t use dynamic extensions so need to use 
> sqlite3_auto_extension or something similar.  

That sounds like a problem.  

I think you could:  

* build SQLite without SQLITE_ENABLE_FTS5,  
* include the file "fts5.c" in the application build, and  
* call sqlite3_fts5_init() from within the auto_extension() callback  
before creating the tokenizer.  

You can't just call sqlite3_fts5_init() from the auto_extension()  
callback in an SQLITE_ENABLE_FTS5 build, as the call to  
sqlite3Fts5Init() made later will replace the existing fts5 module with  
a new one - one that does not have your custom tokenizer registered with it.  

Dan, thank you for your suggestions. I had trouble compiling fts5.c without 
also defining -DSQLITE_CORE which meant that sqlite3_fts5_init was not defined 
so I just defined and called?sqlite3Fts5Init(), but everything worked as 
expected.

I think we should probably change that last bit so that calling?
sqlite3_fts5_init() on a db handle that already has an fts5 extension?
registered with it is a no-op. That might not happen until after the?
3.13 release though.?
I?m happy to contribute a patch but can't quite figure out the development 
process / guidelines.

Also, wouldn?t it be easier to defer sqlite3AutoLoadExtensions() after all 
built-in extensions have been loaded?

? ?Jan

? ? ?

[sqlite] sqlite3_auto_extension + custom FTS5 tokenizer

[Dan Kennedy]

On 05/11/2016 05:24 PM, Jan Berkel wrote:
> I?m currently implementing a custom FTS5 tokenizer which I?d like to 
> automatically register for each db connection.
>
> So I tried to register an extension hook with sqlite3_auto_extension but when 
> my code is called the FTS_* modules have not been initialized, because 
> sqlite3Fts5Init() is called *after* sqlite3AutoLoadExtensions(db).
>
> Therefore registering the tokenizer is not possible at this stage, since the 
> fts5() function is not defined yet.
>
> Is there another way? I can?t use dynamic extensions so need to use 
> sqlite3_auto_extension or something similar.

That sounds like a problem.

I think you could:

* build SQLite without SQLITE_ENABLE_FTS5,
* include the file "fts5.c" in the application build, and
* call sqlite3_fts5_init() from within the auto_extension() callback 
before creating the tokenizer.

To generate fts5.c, grab the full source package (either from here - 
http://sqlite.org/2016/sqlite-src-3120200.zip - or from fossil) and run 
"./configure && make fts5.c".

You can't just call sqlite3_fts5_init() from the auto_extension() 
callback in an SQLITE_ENABLE_FTS5 build, as the call to 
sqlite3Fts5Init() made later will replace the existing fts5 module with 
a new one - one that does not have your custom tokenizer registered with it.

I think we should probably change that last bit so that calling 
sqlite3_fts5_init() on a db handle that already has an fts5 extension 
registered with it is a no-op. That might not happen until after the 
3.13 release though.

Dan.

[sqlite] sqlite3_auto_extension + custom FTS5 tokenizer

[Jan Berkel]

I?m currently implementing a custom FTS5 tokenizer which I?d like to 
automatically register for each db connection.

So I tried to register an extension hook with sqlite3_auto_extension but when 
my code is called the FTS_* modules have not been initialized, because 
sqlite3Fts5Init() is called *after* sqlite3AutoLoadExtensions(db).

Therefore registering the tokenizer is not possible at this stage, since the 
fts5() function is not defined yet.

Is there another way? I can?t use dynamic extensions so need to use 
sqlite3_auto_extension or something similar.

thanks,

? ?Jan

Re: [sqlite] sqlite3_auto_extension - unloaded DLL issue

[Dušan Paulovič]

Thank you MR. Hipp...

Dušan



2013/7/15 Richard Hipp 

> On Sat, Jul 13, 2013 at 5:14 AM, Dušan Paulovič 
> wrote:
>
> > Hello,
> > we are currently facing problem with Access violation exception caused by
> > function sqlite3_open_v2 trying to load extensions from unloaded DLLs.
> >
>
> New interface added: sqlite3_cancel_auto_extension(X).  You can use this to
> cancel prior calls to sqlite3_auto_extension() before unloading the DLL.
>
> http://www.sqlite.org/draft/c3ref/cancel_auto_extension.html
>
> Download snapshots containing this change from
>
> http://www.sqlite.org/draft/download.html
>
>
>
> >
> > How it happens:
> > - There are 2 (or more) plugin DLLs in main application which are linked
> to
> > SQLite.
> > - These plugins can be loaded and uloaded by user.
> > - Main application itself does not use SQLite, so each plugin using
> SQLite
> > must be linked to it.
> > - When any plugin is loaded, it registers its entry point using interface
> > sqlite3_auto_extension.
> > - So when 2 (or more) plugins are loaded, each registers its entry
> point(s)
> > to the same SQLite module instance.
> > - When one of these plugins is going to be unloaded by user, it can not
> use
> > sqlite3_reset_auto_extension,
> >   because that would uninstall also all entry points of another plugins.
> >   So it happens, that SQlite holds entry point(s) to DLL which is
> unloaded.
> > - Now, if user executes any command from remaining DLL, it invokes
> > sqlite3_open_v2,
> >   SQLite invokes all registered entry points including those pointing to
> > released memory,
> >   so system raises Access Viloation exception.
> >
> > Possible solution of this issue would be a way to 'uninstall' entry
> point.
> > something like:
> > int sqlite3_remove_auto_extension(void (*xEntryPoint)(void));
> >
> > Also it would be fine to be able to load static extensions to separate
> > connections:
> > something like:
> > int sqlite3_load_static_extension(sqlite3 *db, void
> (*xEntryPoint)(void));
> >
> > If you would need any additional information about issue,
> > please contact me at: paulo...@gisoft.cz
> >
> > Additional info:
> > SQLite Version: 3.7.14.1, Source ID: 2012-10-04 19:37:12
> > 091570e46d04e84b67228e0bdbcd6e1fb60c6bdb
> >
> > Perhaps irrelevant in this case:
> > OS: Windows Vista
> > Main Application: Bentley MicroStation V8i
> > Plugins: MDL (MicroStation Development Library)
> >
> > Regards,
> > Dusan Paulovic
> > ___
> > sqlite-users mailing list
> > sqlite-users@sqlite.org
> > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
> >
>
>
>
> --
> D. Richard Hipp
> d...@sqlite.org
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] sqlite3_auto_extension - unloaded DLL issue

[Richard Hipp]

On Sat, Jul 13, 2013 at 5:14 AM, Dušan Paulovič  wrote:

> Hello,
> we are currently facing problem with Access violation exception caused by
> function sqlite3_open_v2 trying to load extensions from unloaded DLLs.
>

New interface added: sqlite3_cancel_auto_extension(X).  You can use this to
cancel prior calls to sqlite3_auto_extension() before unloading the DLL.

http://www.sqlite.org/draft/c3ref/cancel_auto_extension.html

Download snapshots containing this change from

http://www.sqlite.org/draft/download.html



>
> How it happens:
> - There are 2 (or more) plugin DLLs in main application which are linked to
> SQLite.
> - These plugins can be loaded and uloaded by user.
> - Main application itself does not use SQLite, so each plugin using SQLite
> must be linked to it.
> - When any plugin is loaded, it registers its entry point using interface
> sqlite3_auto_extension.
> - So when 2 (or more) plugins are loaded, each registers its entry point(s)
> to the same SQLite module instance.
> - When one of these plugins is going to be unloaded by user, it can not use
> sqlite3_reset_auto_extension,
>   because that would uninstall also all entry points of another plugins.
>   So it happens, that SQlite holds entry point(s) to DLL which is unloaded.
> - Now, if user executes any command from remaining DLL, it invokes
> sqlite3_open_v2,
>   SQLite invokes all registered entry points including those pointing to
> released memory,
>   so system raises Access Viloation exception.
>
> Possible solution of this issue would be a way to 'uninstall' entry point.
> something like:
> int sqlite3_remove_auto_extension(void (*xEntryPoint)(void));
>
> Also it would be fine to be able to load static extensions to separate
> connections:
> something like:
> int sqlite3_load_static_extension(sqlite3 *db, void (*xEntryPoint)(void));
>
> If you would need any additional information about issue,
> please contact me at: paulo...@gisoft.cz
>
> Additional info:
> SQLite Version: 3.7.14.1, Source ID: 2012-10-04 19:37:12
> 091570e46d04e84b67228e0bdbcd6e1fb60c6bdb
>
> Perhaps irrelevant in this case:
> OS: Windows Vista
> Main Application: Bentley MicroStation V8i
> Plugins: MDL (MicroStation Development Library)
>
> Regards,
> Dusan Paulovic
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] sqlite3_auto_extension - unloaded DLL issue

[Jay A. Kreibich]

On Sat, Jul 13, 2013 at 11:14:51AM +0200, Du?an Paulovi? scratched on the wall:
> Hello,
> we are currently facing problem with Access violation exception caused by
> function sqlite3_open_v2 trying to load extensions from unloaded DLLs.


> Also it would be fine to be able to load static extensions to separate
> connections:
> something like:
> int sqlite3_load_static_extension(sqlite3 *db, void (*xEntryPoint)(void));

  That would be sqlite3_load_extension():

 http://sqlite.org/c3ref/load_extension.html

  You should be able to pass a NULL in for the filename to have the
  system search the current symbol table context (i.e. the plug-in DLL),
  rather than try to load another library.


  If this is an acceptable solution, I assume it works because each
  plugin manages its own connections to whatever SQLite databases
  you're using, and that the plugins do not cross-utilize extensions.
  In other words, the SQLite extensions used by a plugin are only used
  by that specific plugin, and that plugin "A" does not depend on
  SQLite extensions in plugin "B".  In that respect, having all the
  extensions always auto-load is a bit of overkill, since each
  extension will have access to the SQLite extensions in every loaded
  plugin (and hence the issues with unloading).

  If that's the case, another solution is simply to include a copy of
  SQLite in each plugin.  If each plugin has its own private copy of
  SQLite, then a call to sqlite3_auto_extension() will cause the
  plugin's extensions to only be installed in *that* plugin's database
  connections.  It adds a bit of bulk to every plugin, but on any
  type of desktop system, it shouldn't be a big deal.  You just need to
  integrate the SQLite code directly into the plugin, and make sure it
  is built in a way that the SQLite APIs are not exported from the
  plugin's DLL (otherwise the different plugins will clash).


-j

-- 
Jay A. Kreibich < J A Y  @  K R E I B I.C H >

"Intelligence is like underwear: it is important that you have it,
 but showing it to the wrong people has the tendency to make them
 feel uncomfortable." -- Angela Johnson
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] sqlite3_auto_extension - unloaded DLL issue

[Simon Slavin]

On 13 Jul 2013, at 3:33pm, Jay A. Kreibich  wrote:

>  That would not actually solve the problem.  No matter if SQLite is in
>  a DLL or linked statically in the app, if there is sone master copy
>  of SQLite you're going to have the same issues as application plugins
>  are loaded and unloaded.

So could you have a third plugin which has the job of just making sure SQLite 
is available ?  You require this plugin to be present if either of the other 
ones are to work.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] sqlite3_auto_extension - unloaded DLL issue

[Stephan Beal]

On Sat, Jul 13, 2013 at 11:14 AM, Dušan Paulovič  wrote:

>   because that would uninstall also all entry points of another plugins.
>   So it happens, that SQlite holds entry point(s) to DLL which is unloaded.
> - Now, if user executes any command from remaining DLL, it invokes
> sqlite3_open_v2,
>   SQLite invokes all registered entry points including those pointing to
> released memory,
>   so system raises Access Viloation exception.
>

Generically speaking it is _never_ safe to unload DLLs. Opening of a DLL
can have side-effects, e.g. memory allocation and loading of _other_
external resources, and it's impossible for an application to know
if/when/where that has happened. In C this is easier to control than in C++
(where static initializers can call functions and import new subtypes of
app-side classes), but it's nonetheless not possible to generically close
DLLs with 100% safety. e.g. a C++-built DLL registered a function with
sqlite via a "self-registering" classloader mechanism[1]. Then you close
the DLL. The function registration in sqlite now points back is dead and it
has no way of knowing that.


[1] = http://wanderinghorse.net/computing/papers/#classloading_cpp
-- 
- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] sqlite3_auto_extension - unloaded DLL issue

[Jay A. Kreibich]

On Sat, Jul 13, 2013 at 01:28:28PM +0100, Simon Slavin scratched on the wall:
> 
> On 13 Jul 2013, at 10:14am, Dušan Paulovič  wrote:
> 
> > - These plugins can be loaded and uloaded by user.
> > - Main application itself does not use SQLite, so each plugin using SQLite
> > must be linked to it.
> 
> This is a defect in the way plugins are implemented. The easiest cure
> would be to have Bentley incorporate the SQLite API into its product
> even if it wasn't called in its product.  But you are not Bentley
> and they may not do this just because you asked them.

  That would not actually solve the problem.  No matter if SQLite is in
  a DLL or linked statically in the app, if there is sone master copy
  of SQLite you're going to have the same issues as application plugins
  are loaded and unloaded.

  I'd also point out that getting a DLL to link back against APIs that
  are part of an application is a bit of a trick.  Although this is
  somewhat common place in Unix style systems, windows really likes to
  link "downstream" with DLLs.  The common wisdom to do something like
  this would require extracting SQLite into a DLL so that both the
  application and the plugins could link it in.  This is part of why so
  many Windows applications end up having dozens of DLLs, even when the
  code is part of their core.  It is possible to get a dynamically
  loaded DLL to link "upstream" without passing a block of function
  pointers or some such nonesense, but it isn't the usual way of
  getting things done.

 
-j

-- 
Jay A. Kreibich < J A Y  @  K R E I B I.C H >

"Intelligence is like underwear: it is important that you have it,
 but showing it to the wrong people has the tendency to make them
 feel uncomfortable." -- Angela Johnson
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] sqlite3_auto_extension - unloaded DLL issue

[Simon Slavin]

On 13 Jul 2013, at 10:14am, Dušan Paulovič  wrote:

> - These plugins can be loaded and uloaded by user.
> - Main application itself does not use SQLite, so each plugin using SQLite
> must be linked to it.

This is a defect in the way plugins are implemented. The easiest cure would be 
to have Bentley incorporate the SQLite API into its product even if it wasn't 
called in its product.  But you are not Bentley and they may not do this just 
because you asked them.

SQLite is implemented as one .h and one .c file.  They are both tiny.  The idea 
that it should be used as a DLL was invented by Windows users who seem to 
expect everything to be a DLL.  Implementing as DLLs brings about its own 
confusion of dependencies and version conflicts.

One cure you can control is to have the plugins incorporate SQLite by including 
the amalgamation source code (the .h and .c files) rather than trying to have 
them each use the same DLL.  You can find the amalgamation source code here:



Another is to have each plugin link to their own SQLite DLL, with an 
appropriate name, even if those DLLs are identical.  I /think/ this will work 
under Windows.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] sqlite3_auto_extension - unloaded DLL issue

[Dušan Paulovič]

Hello,
we are currently facing problem with Access violation exception caused by
function sqlite3_open_v2 trying to load extensions from unloaded DLLs.

How it happens:
- There are 2 (or more) plugin DLLs in main application which are linked to
SQLite.
- These plugins can be loaded and uloaded by user.
- Main application itself does not use SQLite, so each plugin using SQLite
must be linked to it.
- When any plugin is loaded, it registers its entry point using interface
sqlite3_auto_extension.
- So when 2 (or more) plugins are loaded, each registers its entry point(s)
to the same SQLite module instance.
- When one of these plugins is going to be unloaded by user, it can not use
sqlite3_reset_auto_extension,
  because that would uninstall also all entry points of another plugins.
  So it happens, that SQlite holds entry point(s) to DLL which is unloaded.
- Now, if user executes any command from remaining DLL, it invokes
sqlite3_open_v2,
  SQLite invokes all registered entry points including those pointing to
released memory,
  so system raises Access Viloation exception.

Possible solution of this issue would be a way to 'uninstall' entry point.
something like:
int sqlite3_remove_auto_extension(void (*xEntryPoint)(void));

Also it would be fine to be able to load static extensions to separate
connections:
something like:
int sqlite3_load_static_extension(sqlite3 *db, void (*xEntryPoint)(void));

If you would need any additional information about issue,
please contact me at: paulo...@gisoft.cz

Additional info:
SQLite Version: 3.7.14.1, Source ID: 2012-10-04 19:37:12
091570e46d04e84b67228e0bdbcd6e1fb60c6bdb

Perhaps irrelevant in this case:
OS: Windows Vista
Main Application: Bentley MicroStation V8i
Plugins: MDL (MicroStation Development Library)

Regards,
Dusan Paulovic
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] sqlite3_auto_extension() memory leaks

[刘以舟]

static void init_db(sqlite3* db, char** pzErrMsg, const struct 
sqlite3_api_routines* *pThunk)
{
.
}
 
BOOL CMyDlg::OnInitDialog()
{
.
sqlite3_auto_extension ((void (*)(void)) init_db);

}
 
 

Detected memory leaks!
Dumping objects ->
{428} normal block at 0x0039C2F8, 16 bytes long.
 Data: <  F > 08 00 00 00 00 00 00 00 E0 02 46 00 CD CD CD CD
Object dump complete.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] sqlite3_auto_extension

[Jean-Christophe Deschamps]

Hi Nico,

Thank you for your answer.

>It's obvious from the function prototype (no DLL/DSO file name
>argument).

Yes, of course.  I'm not confusing a string holding a filename and a 
function pointer!

>However, you can use sqlite3_load_extension() instead per-DB connection,
>OR, you can even do the dlopen(DSO)/dlsym(handle, 
>"sqlite3_extension_init")
>(or Windows equivalent) yourself to get the xEntryPoint argument for
>sqlite3_auto_extension().

Here's what I'm doing from within my extension.  First, remember that 
my problem is having extensions loaded for use with a 3rd-party 
manager.  I can't modify the code of this program, the only code I can 
act upon is the .dll extension itself.  sqlite3_auto_extension gets 
called indirectly thru a select statement to have the default entry 
point (sqlite3_extension_init) called.

select load_extension('filename.dll');

Here's a snippet of my code:

// my registration code
DLL_EXPORT int jcdext_init(sqlite3 *db){
...
// register functions with sqlite3_create_function(...)
}

DLL_EXPORT int sqlite3_extension_init(
   sqlite3 *db,
   char **pzErrMsg,
   const sqlite3_api_routines *pApi
){
   SQLITE_EXTENSION_INIT2(pApi)
   return sqlite3_auto_extension((void*)jcdext_init);
}

At first I thought that the registration code wasn't invoked but it is. 
The catch is that SQLite is still executing a SELECT statement; that 
must be the reason why, when overloading existing functions (like, 
upper, lower, nocase, ...), I receive a return code = SQLITE_BUSY.

New (proprietary) functions register without problem but only for the 
current DB connection: they remain invisible to new connections 
(althought having been registered using auto_extension).

What should I try next?


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] sqlite3_auto_extension

[Nicolas Williams]

On Thu, Jun 18, 2009 at 04:53:13PM +0200, Jean-Christophe Deschamps wrote:
> Having spent "some" time trying to have this function work with dll 
> extensions, I've come to the [I hope 'wrong'] conclusion that it's 
> unusable in such case.  Indeed, the docs mentions "statically linked 
> extension".  Can someone confirm it won't auto load extensions when 
> they reside in dll(s)?

It's obvious from the function prototype (no DLL/DSO file name
argument).

However, you can use sqlite3_load_extension() instead per-DB connection,
OR, you can even do the dlopen(DSO)/dlsym(handle, "sqlite3_extension_init")
(or Windows equivalent) yourself to get the xEntryPoint argument for
sqlite3_auto_extension().

Nico
-- 
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] sqlite3_auto_extension

[Jean-Christophe Deschamps]

Hi,

Having spent "some" time trying to have this function work with dll 
extensions, I've come to the [I hope 'wrong'] conclusion that it's 
unusable in such case.  Indeed, the docs mentions "statically linked 
extension".  Can someone confirm it won't auto load extensions when 
they reside in dll(s)?

I find this very unfortunate because, as I already said in an previous 
message that nobody answered, this leaves users of third-party database 
managers in the mud, to keep it polite.

The rationale behind the design choice seems a little backward to 
me.  Again I would like to be wrong but, as I understand it, the auto 
loading of extension(s) is reserved to compiled programs having the 
extension(s) statically linked within their code.  If it is actually 
the case then I see little point in the function, since the main 
program knows which extension(s) it embeds, how to load and could do so 
when openning any new database.
At the same time, those who have to rely on shared modules (.dll or .so 
or whatever) can't use it and must register each extension for each 
attached database.  The worst case is for third-party DB manager: then 
it belongs to the users to load every extension needed for every base 
they use.

Is this something that can be addressed, either today using a smart 
workaround or in some release in the near future, because I find it to 
be an utterly irritating inconvenience.


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] sqlite3_auto_extension

[Maurí­cio]

Hi,

What is sqlite3_auto_extension used for? What
kind of extensions can it hold; and what are
extensions?

Thanks,
Maurício

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users