Re: [sqlite] threading and grand central dispatch (OS X/iOS)

2011-05-15 Thread Steven Parkes
> But apparently your "not 100% sure" is actually "it's not
> true" in this case (as your tests showed).

No,  I think we're good. As long as one is very careful (as Roger mentioned), 
it seems to work fine under GCD. I was kinda reaching, thinking that moving 
across threads mattered.

I was being very careful right up to completely forgetting about a wal 
checkpoint thread which I haven't thought about in months. And which was using 
the same connection that "belongs" to another thread.

Thanks for the comments, all.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] threading and grand central dispatch (OS X/iOS)

2011-05-15 Thread Pavel Ivanov
> I'm still not 100% sure if there's a problem sharing connections across 
> threads with SQLITE_OPEN_NOMUTEX as long as I guarantee that they aren't 
> concurrent. I suspect there aren't, but I'm not 100% sure. Any case where 
> sqlite3 would be less than happy that pthread_self wasn't always the same?

As long as you guarantee that the same connection is not used
concurrently in different threads SQLite will be happy with any mix of
threads. But apparently your "not 100% sure" is actually "it's not
true" in this case (as your tests showed).


Pavel


On Sun, May 15, 2011 at 4:51 PM, Steven Parkes  wrote:
> Ah, crud. I kinda forgot I had a WAL checkpoint thread running. I thought I 
> had disabled that a long time ago. And it was sharing a connection on a 
> different thread.
>
> Fixing that, things seem stable under GCD with SQLITE_OPEN_NOMUTEX.
>
> I'm still not 100% sure if there's a problem sharing connections across 
> threads with SQLITE_OPEN_NOMUTEX as long as I guarantee that they aren't 
> concurrent. I suspect there aren't, but I'm not 100% sure. Any case where 
> sqlite3 would be less than happy that pthread_self wasn't always the same?
>
> Note that this isn't me redoing what sqlite3 is doing: GCD provides a higher 
> level abstraction that guarantees (modulo my stupidity) non-concurrency. It 
> should be enough for SQLITE_OPEN_NOMUTEX while being (at least slightly) more 
> concurrent than SQLITE_OPEN_FULLMUTEX.
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] threading and grand central dispatch (OS X/iOS)

2011-05-15 Thread Roger Binns
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 05/15/2011 11:14 AM, Steven Parkes wrote:
>> Isn't that exactly what NOMUTEX does?
> 
> The docs (http://www.sqlite.org/threadsafe.html) say 
> 
>   The SQLITE_OPEN_NOMUTEX flag causes the database connection to be in the 
> multi-thread mode and
>   the SQLITE_OPEN_FULLMUTEX flag causes the connection to be in serialized 
> mode.
> 
> Is this wrong?

It is misleading but the documentation is consistent.  See
http://www.sqlite.org/threadsafe.html

Multi-thread mode means that *you* have to ensure it is not used in more
than one thread concurrently.  ie mutexes that normally ensure SQLite is
safe to use in multiple threads concurrently are not used and ensuring that
multi-thread safety is entirely up to the caller.

Which is why you'll generally want normal/serialized/fullmutex since SQLite
will then take complete care of itself in a multi-threaded environment.  If
you disable that then you will have to ensure you have paid the same
attention to detail, design and testing and any bug reports would need to
eliminate your code taking over responsibility for thread safety as the issue.

Roger
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk3QUeEACgkQmOOfHg372QR59QCeMJdal/su37np9x+me0oRYjCB
uwUAoKzkOH+1SUK1YghzBA2C+UAlSKtD
=xBUF
-END PGP SIGNATURE-
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] threading and grand central dispatch (OS X/iOS)

2011-05-15 Thread Steven Parkes
Ah, crud. I kinda forgot I had a WAL checkpoint thread running. I thought I had 
disabled that a long time ago. And it was sharing a connection on a different 
thread.

Fixing that, things seem stable under GCD with SQLITE_OPEN_NOMUTEX.

I'm still not 100% sure if there's a problem sharing connections across threads 
with SQLITE_OPEN_NOMUTEX as long as I guarantee that they aren't concurrent. I 
suspect there aren't, but I'm not 100% sure. Any case where sqlite3 would be 
less than happy that pthread_self wasn't always the same?

Note that this isn't me redoing what sqlite3 is doing: GCD provides a higher 
level abstraction that guarantees (modulo my stupidity) non-concurrency. It 
should be enough for SQLITE_OPEN_NOMUTEX while being (at least slightly) more 
concurrent than SQLITE_OPEN_FULLMUTEX.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] threading and grand central dispatch (OS X/iOS)

2011-05-15 Thread Steven Parkes
> Isn't that exactly what NOMUTEX does?

The docs (http://www.sqlite.org/threadsafe.html) say 

  The SQLITE_OPEN_NOMUTEX flag causes the database connection to be in the 
multi-thread mode and
  the SQLITE_OPEN_FULLMUTEX flag causes the connection to be in serialized mode.

Is this wrong?
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] threading and grand central dispatch (OS X/iOS)

2011-05-15 Thread Roger Binns
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 05/15/2011 10:45 AM, Steven Parkes wrote:
> Not trying to throw them away ...

Isn't that exactly what NOMUTEX does?

> trying to switch from Serialized to Multi-thread. My reading of the docs
says that that's what SQLITE_OPEN_NOMUTEX does. Did I misinterpret something?

The default compile of SQLite is threadsafe.  You can use the same sqlite3
pointer concurrently from as many threads as you want.  It will just work.
The reason for this is that SQLite uses mutexes on each relevant API to
ensure that only one thread at a time is actually using the connection.  If
you remove those mutexes then you will crash and burn.

Roger
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk3QFtQACgkQmOOfHg372QQ8UwCfbStQaZI/Mx4RBZOKpONXQtNJ
3hkAn3jKBqAs9ACQp8tFG1gGIxNEQ8Fy
=AeQo
-END PGP SIGNATURE-
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] threading and grand central dispatch (OS X/iOS)

2011-05-15 Thread Steven Parkes
>> I've been using sqlite3 under iOS fine in FULLMUTEX mode, but it dies 
>> horribly with NOMUTEX 
> 
> What problem are you trying to solve by getting rid of SQLite's mutexes?
> Even if you remove them you still have to add the same thing back in again
> to ensure you don't use SQLite internals concurrently. 

Not trying to throw them away ... trying to switch from Serialized to 
Multi-thread. My reading of the docs says that that's what SQLITE_OPEN_NOMUTEX 
does. Did I misinterpret something?
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] threading and grand central dispatch (OS X/iOS)

2011-05-15 Thread Roger Binns
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 05/15/2011 09:37 AM, Steven Parkes wrote:
> I've been using sqlite3 under iOS fine in FULLMUTEX mode, but it dies 
> horribly with NOMUTEX 

What problem are you trying to solve by getting rid of SQLite's mutexes?
Even if you remove them you still have to add the same thing back in again
to ensure you don't use SQLite internals concurrently.  If you don't then
things will horribly die.

Roger
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk3QEBsACgkQmOOfHg372QQlmQCgg5RDD/XirQ9fVVAbWt4JvsID
VlcAoM6Ow5aOAwngLzpjoKAFNmt/j9Yj
=S09H
-END PGP SIGNATURE-
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


[sqlite] threading and grand central dispatch (OS X/iOS)

2011-05-15 Thread Steven Parkes
This is a little out there but I've been thinking about it and wondered if 
anyone had any comments:

I've been using sqlite3 under iOS fine in FULLMUTEX mode, but it dies horribly 
with NOMUTEX and I'm pretty sure this is because of my use of Grand Central 
Dispatch (GCD).

GCD is related to threads but has some twists. In my simplified model, I create 
a small number of GCD queues and those are essentially one-to-one to threads. 
Each queue has its own db connection and they aren't shared.

But there's one twist, which is the ability in GCD to make a synchronous call 
from one queue/thread to another. So code executing on queue A in thread A can 
make a synchronous  call to queue B. GCD will make sure that queue B (and queue 
B's thread) is inactive. And my code will make sure that for the duration of 
this call, I'm using queue B's db connection.

But it's "borrowing" queue A's thread. So if sqlite3 does a pthread_self() 
during this call, it's going to get A's threadid, not B's. My guess is this 
confuses it.

So if I wanted to "fix" this, somehow I'd provide a GCD thread provider that 
was essentially posix threads but returns GCDs queue id instead of the 
pthreadid.

Comments? Anybody looked at this?
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users