Re: [sqlite] Thread handling in Windows
[EMAIL PROTECTED] wrote: Every Windows DLL can provide a DllMain function. This function gets called whenever a thread gets created or destroyed. http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/dllmain.asp Except that not all of us use Sqlite as a DLL. We statically link it directly into our multi-threaded application. Fortunately for us, we create a fixed number of threads at startup and after that, do not create nor destroy any threads that use sqlite until shutdown.
Re: [sqlite] Thread handling in Windows
Imagine how different the IT world would be if Microsoft had hired computer scientists instead of lawyers. We can but dream. [EMAIL PROTECTED] wrote: "Robert Simpson" <[EMAIL PROTECTED]> wrote: Not quite so straightforward in Windows: Seems like nothing ever is. (sigh) My life would be so much simpler if Bill Gates has stayed in school long enough to take just one course in operating systems... -- D. Richard Hipp <[EMAIL PROTECTED]>
Re: [sqlite] Thread handling in Windows
Rats! It figures that I'd read this right after hitting the send button: (From MSDN) If a DLL declares any nonlocal data or object as __declspec( thread ), it can cause a protection fault if dynamically loaded. After the DLL is loaded with LoadLibrary, it causes system failure whenever the code references the nonlocal __declspec( thread ) data. Because the global variable space for a thread is allocated at run time, the size of this space is based on a calculation of the requirements of the application plus the requirements of all of the DLLs that are statically linked. When you use LoadLibrary, there is no way to extend this space to allow for the thread local variables declared with __declspec( thread ). Use the TLS APIs, such as TlsAlloc, in your DLL to allocate TLS if the DLL might be loaded with LoadLibrary.
Re: [sqlite] Thread handling in Windows
- Original Message - From: <[EMAIL PROTECTED]> It makes the code smaller and faster to set a flag when malloc fails, rather than trying to test for the failure all the way back up the stack. The flag used to be a global variable. But that was causing problems for embedded device manufacturers. Doesn't *nix have the __declspec(thread) modifier? Your malloc failure flag could still be a global variable with those attributes ...
Re: [sqlite] Thread handling in Windows
Teg <[EMAIL PROTECTED]> wrote: > Hello drh, > > Could you perhaps give us the rational for using TLS? How is it that > TLS is needed on cases where you run out of memory? It makes the code smaller and faster to set a flag when malloc fails, rather than trying to test for the failure all the way back up the stack. The flag used to be a global variable. But that was causing problems for embedded device manufacturers. Specifically, embedded devices run out of memory quite often - a malloc() failure is routine. It is not acceptable to shut down the whole process just because malloc failed. But if the failure flag is in a global variable, there is no way to know which thread took the failure and hence no way to recover gracefully. Everything has to be stopped and restarted. By moving the malloc failure flag into TLS, we can recover gracefully from OOM without impacting other threads at all. TLS is also used to support new features of 3.3.0: Share caches and memory management. Both are off by default. But if you want to use them you will need TLS. -- D. Richard Hipp <[EMAIL PROTECTED]>
Re: [sqlite] Thread handling in Windows
Hello drh, Could you perhaps give us the rational for using TLS? How is it that TLS is needed on cases where you run out of memory? By using TLS you're essentially making SQlite thread aware where I'd prefer it to know nothing about thread contexts or whether it's running in a thread or not. How is TLS better than a simple malloc and free in this instance? To me the proper solution would be to avoid using TLS in the first place since, you (sqlite) neither create nor destroy the thread. C Wednesday, January 11, 2006, 10:00:02 AM, you wrote: dhc> In the unix world using pthreads, when you allocate dhc> thread-local storage, you specify a callback function dhc> to delete the storage when the thread terminates. dhc> This callback is the second argument to dhc> pthread_key_create(), See, for example, dhc> http://www.mkssoftware.com/docs/man3/pthread_key_create.3.asp dhc> Question: How do you do the same thing on windows? dhc> How do you get a thread to clean up its thread-local-storage dhc> obtained using TlsAlloc() and TlsSetValue() when the dhc> thread terminates? dhc> I need an answer to this question so that I can fix dhc> ticket #1601: http://www.sqlite.org/cvstrac/tktview?tn=1601 dhc> -- dhc> D. Richard Hipp <[EMAIL PROTECTED]> -- Best regards, Tegmailto:[EMAIL PROTECTED]
Re: [sqlite] Thread handling in Windows
- Original Message - From: <[EMAIL PROTECTED]> No. Thread-local storage is needed to recover from out-of-memory errors. It is not optional. There is one way to do this, but it might be ugly. The way I see it, the problem lies in the fact that you are never notified when a thread terminates. The only way to detect if a thread has terminated is to call GetExitCodeThread() and if the status is not STILL_ACTIVE, then the thread is dead. In order to call that function, you need a thread handle. Which means that whenever you allocate new tls data for a given thread, you must call DuplicateHandle(GetCurrentThread()) and store the copy of the thread handle in a linked list. So that leaves you with periodically scanning through the list looking for threads that have disappeared, and once gone, you raise the callback routine and destroy their tls data and close the thread handle. Perhaps it could be done during sqlite_reset, or sqlite_finalize or sqlite_prepare ... but its not quite as elegant as the *nix way. Robert
Re: [sqlite] Thread handling in Windows
On Jan 11, 2006, at 7:43 AM, [EMAIL PROTECTED] wrote: Seems like nothing ever is. (sigh) My life would be so much simpler if Bill Gates has stayed in school long enough to take just one course in operating systems... LOL! That's about the funniest thing I've ever heard. Do we have permission to use this?
Re: [sqlite] Thread handling in Windows
I'm the one who posted ticket 1601. I have a multithreaded sqlite tcp server, and it ran fine with 2.8.x and 3.x versions, but this "feature" is a stopper for me. I think there is no difference using a dll or not, there is no way to free that thread memory allocated, since that function it uses static variables. Some have stated using DLL_THREAD_DETACH, but this only works (in case it does work) using the dll, if linked statically it keeps leaking memory. As drh stated: "In the unix world using pthreads, when you allocate thread-local storage, you specify a callback function to delete the storage when the thread terminates. This callback is the second argument..." Ok, and what is it suppose to do that callback being that info internal to sqlite, and thus not exposed? The problem arises when sqlite uses (malloc) thread memory but itself does not create/terminate threads. An application that uses sqlite is the one that knows about threads, modules, libraries, etc. ie., when they are created/terminated, thread id, priority, library (dll) loading/unloading, and so on. I've always thought (since I began using sqlite) about a couple of public functions (and exported in the "def" file) like: InitSqliteLib and DeInitSqliteLib. These would take care of all the public variables, hash lists and so forth to make a clean start and exit. Extending this, maybe a couple more but for threads, like (exported in the "def" file): InitSqliteThread and DeInitSqliteThread. So right after creating a thread that is going to use sqlite functions, it would call InitSqliteThread, which would allocate, initialize or whatever it needs for the current thread, and right before terminating the thread it would call DeInitSqliteThread. Same for the InitSqliteLib and DeInitSqliteLib. This way, it would be possible to get it working using pthreads, Windows threads and others.
Re: [sqlite] Thread handling in Windows
"Robert Simpson" <[EMAIL PROTECTED]> wrote: > > I haven't looked too closely at the thread local data stuff recently added > to SQLite, however, is there a way to have it default to off, and opt-in at > runtime? No. Thread-local storage is needed to recover from out-of-memory errors. It is not optional. -- D. Richard Hipp <[EMAIL PROTECTED]>
Re: [sqlite] Thread handling in Windows
- Original Message - From: <[EMAIL PROTECTED]> In the unix world using pthreads, when you allocate thread-local storage, you specify a callback function to delete the storage when the thread terminates. This callback is the second argument to pthread_key_create(), See, for example, http://www.mkssoftware.com/docs/man3/pthread_key_create.3.asp Question: How do you do the same thing on windows? How do you get a thread to clean up its thread-local-storage obtained using TlsAlloc() and TlsSetValue() when the thread terminates? I need an answer to this question so that I can fix ticket #1601: http://www.sqlite.org/cvstrac/tktview?tn=1601 I haven't looked too closely at the thread local data stuff recently added to SQLite, however, is there a way to have it default to off, and opt-in at runtime? That way if your threads failed to call the "sqlite3_uber_cleanup()" function and leak memory on exit, it'd be your own damn fault and not SQLite's. Robert
RE: [sqlite] Thread handling in Windows
Damn! Then he would have ALL the money, not just a few billion! :-) > -Original Message- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] > Sent: Wednesday, January 11, 2006 9:44 AM > To: sqlite-users@sqlite.org > Subject: Re: [sqlite] Thread handling in Windows > > > "Robert Simpson" <[EMAIL PROTECTED]> wrote: > > > > Not quite so straightforward in Windows: > > > > Seems like nothing ever is. (sigh) My life would be so much > simpler if Bill Gates has stayed in school long enough > to take just one course in operating systems... > > -- > D. Richard Hipp <[EMAIL PROTECTED]> >
Re: [sqlite] Thread handling in Windows
- Original Message - From: <[EMAIL PROTECTED]> "Robert Simpson" <[EMAIL PROTECTED]> wrote: Not quite so straightforward in Windows: Seems like nothing ever is. (sigh) My life would be so much simpler if Bill Gates has stayed in school long enough to take just one course in operating systems... Imagine if Bill had taken that OS course instead of that business course ... then imagine Bill as Linus Torvalds. Then imagine Commodore rules the world! Amiga Forever! Robert
Re: [sqlite] Thread handling in Windows
"Robert Simpson" <[EMAIL PROTECTED]> wrote: > > Not quite so straightforward in Windows: > Seems like nothing ever is. (sigh) My life would be so much simpler if Bill Gates has stayed in school long enough to take just one course in operating systems... -- D. Richard Hipp <[EMAIL PROTECTED]>
RE: [sqlite] Thread handling in Windows
>Question: How do you do the same thing on windows? >How do you get a thread to clean up its thread-local-storage >obtained using TlsAlloc() and TlsSetValue() when the >thread terminates? You may find this example useful: http://www.codeproject.com/threads/tls.asp Regards Nick This email and any attachments are confidential to the intended recipient and may also be privileged. If you are not the intended recipient please delete it from your system and notify the sender. You should not copy it or use it for any purpose nor disclose or distribute its contents to any other person.
Re: [sqlite] Thread handling in Windows
- Original Message - From: <[EMAIL PROTECTED]> In the unix world using pthreads, when you allocate thread-local storage, you specify a callback function to delete the storage when the thread terminates. This callback is the second argument to pthread_key_create(), See, for example, http://www.mkssoftware.com/docs/man3/pthread_key_create.3.asp Question: How do you do the same thing on windows? How do you get a thread to clean up its thread-local-storage obtained using TlsAlloc() and TlsSetValue() when the thread terminates? I need an answer to this question so that I can fix ticket #1601: http://www.sqlite.org/cvstrac/tktview?tn=1601 Not quite so straightforward in Windows: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/thread_local_storage.asp Excerpt of #4: When each thread no longer needs to use a TLS index, it must free the memory whose pointer is stored in the TLS slot. When all threads have finished using a TLS index, use the TlsFree function to free the index. For example, use the following code in your DllMain during DLL_THREAD_DETACH: lpvBuffer = TlsGetValue(gdwTlsIndex); LocalFree((HLOCAL) lpvBuffer); and the following code during DLL_PROCESS_DETACH: TlsFree(gdwTlsIndex);
Re: [sqlite] Thread handling in Windows
Every Windows DLL can provide a DllMain function. This function gets called
whenever a thread gets created or destroyed.
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/dllmain.asp
Some example code:
/**
\brief
*/
BOOL WINAPI DllMain(
HINSTANCE hinstDLL,
DWORD fdwReason,
LPVOID /*lpReserved*/)
{
// Perform actions based on the reason for calling.
switch( fdwReason )
{
case DLL_PROCESS_ATTACH:
// Initialize once for each new process.
// Return FALSE to fail DLL load.
myDllHandle = hinstDLL;
// Allocate a TLS index.
if((myTlsIndex = TlsAlloc()) == 0x)
return FALSE;
break;
case DLL_THREAD_ATTACH:
// Do thread-specific initialization.
break;
case DLL_THREAD_DETACH:
// Do thread-specific cleanup.
// Release the allocated memory for this thread.
myFreeUserMessageBufferOfThread();
break;
case DLL_PROCESS_DETACH:
NI_DEBUG_ASSERT(myRefCount == 0); // usi not correctly
deinitialized
myFreeUserMessageBufferOfThread();
TlsFree(myTlsIndex);
myTlsIndex = myTlsIndexInvalid;
break;
}
return TRUE; // Successful DLL_PROCESS_ATTACH.
}
[EMAIL PROTECTED]
11.01.2006 16:00 To
sqlite-users@sqlite.org
cc
Please respond to
[EMAIL PROTECTED] Subject
te.org [sqlite] Thread handling in Windows
In the unix world using pthreads, when you allocate
thread-local storage, you specify a callback function
to delete the storage when the thread terminates.
This callback is the second argument to
pthread_key_create(), See, for example,
http://www.mkssoftware.com/docs/man3/pthread_key_create.3.asp
Question: How do you do the same thing on windows?
How do you get a thread to clean up its thread-local-storage
obtained using TlsAlloc() and TlsSetValue() when the
thread terminates?
I need an answer to this question so that I can fix
ticket #1601: http://www.sqlite.org/cvstrac/tktview?tn=1601
--
D. Richard Hipp <[EMAIL PROTECTED]>
