Re: [sqlite] threading and grand central dispatch (OS X/iOS)
> But apparently your "not 100% sure" is actually "it's not > true" in this case (as your tests showed). No, I think we're good. As long as one is very careful (as Roger mentioned), it seems to work fine under GCD. I was kinda reaching, thinking that moving across threads mattered. I was being very careful right up to completely forgetting about a wal checkpoint thread which I haven't thought about in months. And which was using the same connection that "belongs" to another thread. Thanks for the comments, all. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] threading and grand central dispatch (OS X/iOS)
> I'm still not 100% sure if there's a problem sharing connections across > threads with SQLITE_OPEN_NOMUTEX as long as I guarantee that they aren't > concurrent. I suspect there aren't, but I'm not 100% sure. Any case where > sqlite3 would be less than happy that pthread_self wasn't always the same? As long as you guarantee that the same connection is not used concurrently in different threads SQLite will be happy with any mix of threads. But apparently your "not 100% sure" is actually "it's not true" in this case (as your tests showed). Pavel On Sun, May 15, 2011 at 4:51 PM, Steven Parkes wrote: > Ah, crud. I kinda forgot I had a WAL checkpoint thread running. I thought I > had disabled that a long time ago. And it was sharing a connection on a > different thread. > > Fixing that, things seem stable under GCD with SQLITE_OPEN_NOMUTEX. > > I'm still not 100% sure if there's a problem sharing connections across > threads with SQLITE_OPEN_NOMUTEX as long as I guarantee that they aren't > concurrent. I suspect there aren't, but I'm not 100% sure. Any case where > sqlite3 would be less than happy that pthread_self wasn't always the same? > > Note that this isn't me redoing what sqlite3 is doing: GCD provides a higher > level abstraction that guarantees (modulo my stupidity) non-concurrency. It > should be enough for SQLITE_OPEN_NOMUTEX while being (at least slightly) more > concurrent than SQLITE_OPEN_FULLMUTEX. > > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] threading and grand central dispatch (OS X/iOS)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/15/2011 11:14 AM, Steven Parkes wrote: >> Isn't that exactly what NOMUTEX does? > > The docs (http://www.sqlite.org/threadsafe.html) say > > The SQLITE_OPEN_NOMUTEX flag causes the database connection to be in the > multi-thread mode and > the SQLITE_OPEN_FULLMUTEX flag causes the connection to be in serialized > mode. > > Is this wrong? It is misleading but the documentation is consistent. See http://www.sqlite.org/threadsafe.html Multi-thread mode means that *you* have to ensure it is not used in more than one thread concurrently. ie mutexes that normally ensure SQLite is safe to use in multiple threads concurrently are not used and ensuring that multi-thread safety is entirely up to the caller. Which is why you'll generally want normal/serialized/fullmutex since SQLite will then take complete care of itself in a multi-threaded environment. If you disable that then you will have to ensure you have paid the same attention to detail, design and testing and any bug reports would need to eliminate your code taking over responsibility for thread safety as the issue. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk3QUeEACgkQmOOfHg372QR59QCeMJdal/su37np9x+me0oRYjCB uwUAoKzkOH+1SUK1YghzBA2C+UAlSKtD =xBUF -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] threading and grand central dispatch (OS X/iOS)
Ah, crud. I kinda forgot I had a WAL checkpoint thread running. I thought I had disabled that a long time ago. And it was sharing a connection on a different thread. Fixing that, things seem stable under GCD with SQLITE_OPEN_NOMUTEX. I'm still not 100% sure if there's a problem sharing connections across threads with SQLITE_OPEN_NOMUTEX as long as I guarantee that they aren't concurrent. I suspect there aren't, but I'm not 100% sure. Any case where sqlite3 would be less than happy that pthread_self wasn't always the same? Note that this isn't me redoing what sqlite3 is doing: GCD provides a higher level abstraction that guarantees (modulo my stupidity) non-concurrency. It should be enough for SQLITE_OPEN_NOMUTEX while being (at least slightly) more concurrent than SQLITE_OPEN_FULLMUTEX. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] threading and grand central dispatch (OS X/iOS)
> Isn't that exactly what NOMUTEX does? The docs (http://www.sqlite.org/threadsafe.html) say The SQLITE_OPEN_NOMUTEX flag causes the database connection to be in the multi-thread mode and the SQLITE_OPEN_FULLMUTEX flag causes the connection to be in serialized mode. Is this wrong? ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] threading and grand central dispatch (OS X/iOS)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/15/2011 10:45 AM, Steven Parkes wrote: > Not trying to throw them away ... Isn't that exactly what NOMUTEX does? > trying to switch from Serialized to Multi-thread. My reading of the docs says that that's what SQLITE_OPEN_NOMUTEX does. Did I misinterpret something? The default compile of SQLite is threadsafe. You can use the same sqlite3 pointer concurrently from as many threads as you want. It will just work. The reason for this is that SQLite uses mutexes on each relevant API to ensure that only one thread at a time is actually using the connection. If you remove those mutexes then you will crash and burn. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk3QFtQACgkQmOOfHg372QQ8UwCfbStQaZI/Mx4RBZOKpONXQtNJ 3hkAn3jKBqAs9ACQp8tFG1gGIxNEQ8Fy =AeQo -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] threading and grand central dispatch (OS X/iOS)
>> I've been using sqlite3 under iOS fine in FULLMUTEX mode, but it dies >> horribly with NOMUTEX > > What problem are you trying to solve by getting rid of SQLite's mutexes? > Even if you remove them you still have to add the same thing back in again > to ensure you don't use SQLite internals concurrently. Not trying to throw them away ... trying to switch from Serialized to Multi-thread. My reading of the docs says that that's what SQLITE_OPEN_NOMUTEX does. Did I misinterpret something? ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] threading and grand central dispatch (OS X/iOS)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/15/2011 09:37 AM, Steven Parkes wrote: > I've been using sqlite3 under iOS fine in FULLMUTEX mode, but it dies > horribly with NOMUTEX What problem are you trying to solve by getting rid of SQLite's mutexes? Even if you remove them you still have to add the same thing back in again to ensure you don't use SQLite internals concurrently. If you don't then things will horribly die. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk3QEBsACgkQmOOfHg372QQlmQCgg5RDD/XirQ9fVVAbWt4JvsID VlcAoM6Ow5aOAwngLzpjoKAFNmt/j9Yj =S09H -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] threading and grand central dispatch (OS X/iOS)
This is a little out there but I've been thinking about it and wondered if anyone had any comments: I've been using sqlite3 under iOS fine in FULLMUTEX mode, but it dies horribly with NOMUTEX and I'm pretty sure this is because of my use of Grand Central Dispatch (GCD). GCD is related to threads but has some twists. In my simplified model, I create a small number of GCD queues and those are essentially one-to-one to threads. Each queue has its own db connection and they aren't shared. But there's one twist, which is the ability in GCD to make a synchronous call from one queue/thread to another. So code executing on queue A in thread A can make a synchronous call to queue B. GCD will make sure that queue B (and queue B's thread) is inactive. And my code will make sure that for the duration of this call, I'm using queue B's db connection. But it's "borrowing" queue A's thread. So if sqlite3 does a pthread_self() during this call, it's going to get A's threadid, not B's. My guess is this confuses it. So if I wanted to "fix" this, somehow I'd provide a GCD thread provider that was essentially posix threads but returns GCDs queue id instead of the pthreadid. Comments? Anybody looked at this? ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users