Re: [sqlite] Multi-threading Common Problem
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/23/2011 09:12 PM, John Deal wrote: I guess I am lost on how to obtain a many reader or one writer mutex in SQLite. You are confusing locks on the database and locks in the library on a sqlite3 pointer. The latter is what the mutex alloc function you reference is about and there there is no reader/writer mechanism. Access has to be serialized. You are correct in the locking article referenced I want a mutex that can have the lock states of shared, pending, and exclusive. Those are locks on the database which you get through regular operations and transactions. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk3bWeIACgkQmOOfHg372QQf8QCgjlawQMJWJ1I3/6OqMkczXswk VWQAmgLzGifXbh9UJpuEdUTTZl8e8xYp =rXCY -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Multi-threading Common Problem
Hello Roger, Sorry to be so brain-dead but I am still confused. I have multiple threads, each with their own DB connection. I want to allow multiple readers accessing the DB at the same time since nothing is changing. However, if a writes is to take place, I want all readers to finish their reads and give the writer exclusive access. Once the writer is done, the readers can come back in. I have all writes in transactions. If I deactivate my pthread_rwlock() that enforce the above, several writes fail with a database locked error (I assume it is returning SQLITE_BUSY). With my pthread_rwlock(), I have multiple threads reading the DB and my writes get the exclusive access they need. Now I could loop on the write until it gets in but that seems very wasteful. So how do I implement the equivalent of a pthread_rwlock() using SQLite mechinisms? Thanks, John --- On Tue, 5/24/11, Roger Binns rog...@rogerbinns.com wrote: From: Roger Binns rog...@rogerbinns.com Subject: Re: [sqlite] Multi-threading Common Problem To: General Discussion of SQLite Database sqlite-users@sqlite.org Date: Tuesday, May 24, 2011, 3:10 AM -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/23/2011 09:12 PM, John Deal wrote: I guess I am lost on how to obtain a many reader or one writer mutex in SQLite. You are confusing locks on the database and locks in the library on a sqlite3 pointer. The latter is what the mutex alloc function you reference is about and there there is no reader/writer mechanism. Access has to be serialized. You are correct in the locking article referenced I want a mutex that can have the lock states of shared, pending, and exclusive. Those are locks on the database which you get through regular operations and transactions. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk3bWeIACgkQmOOfHg372QQf8QCgjlawQMJWJ1I3/6OqMkczXswk VWQAmgLzGifXbh9UJpuEdUTTZl8e8xYp =rXCY -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Multi-threading Common Problem
On 24 May 2011, at 12:43pm, John Deal wrote: Sorry to be so brain-dead but I am still confused. I have multiple threads, each with their own DB connection. Read http://www.sqlite.org/threadsafe.html If it's not clear to you please ask specific questions about what's on that page, since your questions will help us to work out how to improve it. I want to allow multiple readers accessing the DB at the same time since nothing is changing. However, if a writes is to take place, I want all readers to finish their reads and give the writer exclusive access. Once the writer is done, the readers can come back in. I have all writes in transactions. If I deactivate my pthread_rwlock() that enforce the above, several writes fail with a database locked error (I assume it is returning SQLITE_BUSY). The documentation on how to handle this is a little lacking, but if your app has no principles of its own, just loop until the problem goes away. With my pthread_rwlock(), I have multiple threads reading the DB and my writes get the exclusive access they need. Now I could loop on the write until it gets in but that seems very wasteful. So how do I implement the equivalent of a pthread_rwlock() using SQLite mechinisms? You use transactions correctly. Pay especial attention to the differences between BEGIN BEGIN IMMEDIATE BEGIN EXCLUSIVE BEGIN DEFERRED See the following page http://www.sqlite.org/lang_transaction.html However, you may have done something to defeat SQLite's built in mechanism. Have you done any of the following: Have you set any of SQLite's compiler directives in your compilation ? I'm thinking especially of SQLITE_THREADSAFE. I'm not completely familiar with SQLite's model, but it seems to me as if you should be using '-DSQLITE_THREADSAFE=2'. The explanation of this is on the first web page I pointed to above. Have you set any PRAGMAs which are related to threading ? Especially 'PRAGMA temp_store_directory' and 'PRAGMA read_uncommitted' ? If so, try removing your PRAGMAs and see if this fixes your problem. Simon. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Multi-threading Common Problem
I have all writes in transactions. If I deactivate my pthread_rwlock() that enforce the above, several writes fail with a database locked error (I assume it is returning SQLITE_BUSY). So how do I implement the equivalent of a pthread_rwlock() using SQLite mechinisms? When SQLITE_BUSY in a reader transaction is returned just wait a little bit and try again. Also you can benefit from sqlite3_busy_timeout (http://www.sqlite.org/c3ref/busy_timeout.html). Another question is why do you want to get read of pthread_rwlock if it works for you? Pavel On Tue, May 24, 2011 at 7:43 AM, John Deal bassd...@yahoo.com wrote: Hello Roger, Sorry to be so brain-dead but I am still confused. I have multiple threads, each with their own DB connection. I want to allow multiple readers accessing the DB at the same time since nothing is changing. However, if a writes is to take place, I want all readers to finish their reads and give the writer exclusive access. Once the writer is done, the readers can come back in. I have all writes in transactions. If I deactivate my pthread_rwlock() that enforce the above, several writes fail with a database locked error (I assume it is returning SQLITE_BUSY). With my pthread_rwlock(), I have multiple threads reading the DB and my writes get the exclusive access they need. Now I could loop on the write until it gets in but that seems very wasteful. So how do I implement the equivalent of a pthread_rwlock() using SQLite mechinisms? Thanks, John --- On Tue, 5/24/11, Roger Binns rog...@rogerbinns.com wrote: From: Roger Binns rog...@rogerbinns.com Subject: Re: [sqlite] Multi-threading Common Problem To: General Discussion of SQLite Database sqlite-users@sqlite.org Date: Tuesday, May 24, 2011, 3:10 AM -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/23/2011 09:12 PM, John Deal wrote: I guess I am lost on how to obtain a many reader or one writer mutex in SQLite. You are confusing locks on the database and locks in the library on a sqlite3 pointer. The latter is what the mutex alloc function you reference is about and there there is no reader/writer mechanism. Access has to be serialized. You are correct in the locking article referenced I want a mutex that can have the lock states of shared, pending, and exclusive. Those are locks on the database which you get through regular operations and transactions. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk3bWeIACgkQmOOfHg372QQf8QCgjlawQMJWJ1I3/6OqMkczXswk VWQAmgLzGifXbh9UJpuEdUTTZl8e8xYp =rXCY -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Multi-threading Common Problem
Hello Pavel, I don't want per-say to remove my pthread_rwlock() but the main point of this discussion is I should not have to use pthread_rwlock(). Others have mentioned I should be using SQLite-specific mechanisms to achieve the same results. I am just trying to understand how to do that. Pthread_rwlock() works fine. Thanks, John --- On Tue, 5/24/11, Pavel Ivanov paiva...@gmail.com wrote: From: Pavel Ivanov paiva...@gmail.com Subject: Re: [sqlite] Multi-threading Common Problem To: General Discussion of SQLite Database sqlite-users@sqlite.org Date: Tuesday, May 24, 2011, 9:51 AM I have all writes in transactions. If I deactivate my pthread_rwlock() that enforce the above, several writes fail with a database locked error (I assume it is returning SQLITE_BUSY). So how do I implement the equivalent of a pthread_rwlock() using SQLite mechinisms? When SQLITE_BUSY in a reader transaction is returned just wait a little bit and try again. Also you can benefit from sqlite3_busy_timeout (http://www.sqlite.org/c3ref/busy_timeout.html). Another question is why do you want to get read of pthread_rwlock if it works for you? Pavel On Tue, May 24, 2011 at 7:43 AM, John Deal bassd...@yahoo.com wrote: Hello Roger, Sorry to be so brain-dead but I am still confused. I have multiple threads, each with their own DB connection. I want to allow multiple readers accessing the DB at the same time since nothing is changing. However, if a writes is to take place, I want all readers to finish their reads and give the writer exclusive access. Once the writer is done, the readers can come back in. I have all writes in transactions. If I deactivate my pthread_rwlock() that enforce the above, several writes fail with a database locked error (I assume it is returning SQLITE_BUSY). With my pthread_rwlock(), I have multiple threads reading the DB and my writes get the exclusive access they need. Now I could loop on the write until it gets in but that seems very wasteful. So how do I implement the equivalent of a pthread_rwlock() using SQLite mechinisms? Thanks, John --- On Tue, 5/24/11, Roger Binns rog...@rogerbinns.com wrote: From: Roger Binns rog...@rogerbinns.com Subject: Re: [sqlite] Multi-threading Common Problem To: General Discussion of SQLite Database sqlite-users@sqlite.org Date: Tuesday, May 24, 2011, 3:10 AM -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/23/2011 09:12 PM, John Deal wrote: I guess I am lost on how to obtain a many reader or one writer mutex in SQLite. You are confusing locks on the database and locks in the library on a sqlite3 pointer. The latter is what the mutex alloc function you reference is about and there there is no reader/writer mechanism. Access has to be serialized. You are correct in the locking article referenced I want a mutex that can have the lock states of shared, pending, and exclusive. Those are locks on the database which you get through regular operations and transactions. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk3bWeIACgkQmOOfHg372QQf8QCgjlawQMJWJ1I3/6OqMkczXswk VWQAmgLzGifXbh9UJpuEdUTTZl8e8xYp =rXCY -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Multi-threading Common Problem
I don't want per-say to remove my pthread_rwlock() but the main point of this discussion is I should not have to use pthread_rwlock(). I'd say pthread_rwlock and SQLite-specific mechanisms work completely differently and you should choose depending on what you want to do. As you saw to use SQLite's mechanisms you should write additional code waiting when write lock is released. As you said it's not effective and prone to starvation. But it works across process boundaries when pthread_rwlock works only inside your process although it does all waiting very effectively on a kernel level. So if you will ever want to connect to your database with sqlite3 command line tool for example while your application is running, and you will do some manipulations with the database, then your pthread_rwlock won't work and you will still get SQLITE_BUSY. And now you decide what mechanism is better for you. Pavel On Tue, May 24, 2011 at 10:11 AM, John Deal bassd...@yahoo.com wrote: Hello Pavel, I don't want per-say to remove my pthread_rwlock() but the main point of this discussion is I should not have to use pthread_rwlock(). Others have mentioned I should be using SQLite-specific mechanisms to achieve the same results. I am just trying to understand how to do that. Pthread_rwlock() works fine. Thanks, John --- On Tue, 5/24/11, Pavel Ivanov paiva...@gmail.com wrote: From: Pavel Ivanov paiva...@gmail.com Subject: Re: [sqlite] Multi-threading Common Problem To: General Discussion of SQLite Database sqlite-users@sqlite.org Date: Tuesday, May 24, 2011, 9:51 AM I have all writes in transactions. If I deactivate my pthread_rwlock() that enforce the above, several writes fail with a database locked error (I assume it is returning SQLITE_BUSY). So how do I implement the equivalent of a pthread_rwlock() using SQLite mechinisms? When SQLITE_BUSY in a reader transaction is returned just wait a little bit and try again. Also you can benefit from sqlite3_busy_timeout (http://www.sqlite.org/c3ref/busy_timeout.html). Another question is why do you want to get read of pthread_rwlock if it works for you? Pavel On Tue, May 24, 2011 at 7:43 AM, John Deal bassd...@yahoo.com wrote: Hello Roger, Sorry to be so brain-dead but I am still confused. I have multiple threads, each with their own DB connection. I want to allow multiple readers accessing the DB at the same time since nothing is changing. However, if a writes is to take place, I want all readers to finish their reads and give the writer exclusive access. Once the writer is done, the readers can come back in. I have all writes in transactions. If I deactivate my pthread_rwlock() that enforce the above, several writes fail with a database locked error (I assume it is returning SQLITE_BUSY). With my pthread_rwlock(), I have multiple threads reading the DB and my writes get the exclusive access they need. Now I could loop on the write until it gets in but that seems very wasteful. So how do I implement the equivalent of a pthread_rwlock() using SQLite mechinisms? Thanks, John --- On Tue, 5/24/11, Roger Binns rog...@rogerbinns.com wrote: From: Roger Binns rog...@rogerbinns.com Subject: Re: [sqlite] Multi-threading Common Problem To: General Discussion of SQLite Database sqlite-users@sqlite.org Date: Tuesday, May 24, 2011, 3:10 AM -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/23/2011 09:12 PM, John Deal wrote: I guess I am lost on how to obtain a many reader or one writer mutex in SQLite. You are confusing locks on the database and locks in the library on a sqlite3 pointer. The latter is what the mutex alloc function you reference is about and there there is no reader/writer mechanism. Access has to be serialized. You are correct in the locking article referenced I want a mutex that can have the lock states of shared, pending, and exclusive. Those are locks on the database which you get through regular operations and transactions. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk3bWeIACgkQmOOfHg372QQf8QCgjlawQMJWJ1I3/6OqMkczXswk VWQAmgLzGifXbh9UJpuEdUTTZl8e8xYp =rXCY -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Multi-threading Common Problem
Hello Pavel, Thanks. This is the conclusion I was arriving at. I do use the Sqlite3 utility but only for looking at test results when the server is in an inactive state. Thanks, John --- On Tue, 5/24/11, Pavel Ivanov paiva...@gmail.com wrote: From: Pavel Ivanov paiva...@gmail.com Subject: Re: [sqlite] Multi-threading Common Problem To: General Discussion of SQLite Database sqlite-users@sqlite.org Date: Tuesday, May 24, 2011, 10:32 AM I don't want per-say to remove my pthread_rwlock() but the main point of this discussion is I should not have to use pthread_rwlock(). I'd say pthread_rwlock and SQLite-specific mechanisms work completely differently and you should choose depending on what you want to do. As you saw to use SQLite's mechanisms you should write additional code waiting when write lock is released. As you said it's not effective and prone to starvation. But it works across process boundaries when pthread_rwlock works only inside your process although it does all waiting very effectively on a kernel level. So if you will ever want to connect to your database with sqlite3 command line tool for example while your application is running, and you will do some manipulations with the database, then your pthread_rwlock won't work and you will still get SQLITE_BUSY. And now you decide what mechanism is better for you. Pavel On Tue, May 24, 2011 at 10:11 AM, John Deal bassd...@yahoo.com wrote: Hello Pavel, I don't want per-say to remove my pthread_rwlock() but the main point of this discussion is I should not have to use pthread_rwlock(). Others have mentioned I should be using SQLite-specific mechanisms to achieve the same results. I am just trying to understand how to do that. Pthread_rwlock() works fine. Thanks, John --- On Tue, 5/24/11, Pavel Ivanov paiva...@gmail.com wrote: From: Pavel Ivanov paiva...@gmail.com Subject: Re: [sqlite] Multi-threading Common Problem To: General Discussion of SQLite Database sqlite-users@sqlite.org Date: Tuesday, May 24, 2011, 9:51 AM I have all writes in transactions. If I deactivate my pthread_rwlock() that enforce the above, several writes fail with a database locked error (I assume it is returning SQLITE_BUSY). So how do I implement the equivalent of a pthread_rwlock() using SQLite mechinisms? When SQLITE_BUSY in a reader transaction is returned just wait a little bit and try again. Also you can benefit from sqlite3_busy_timeout (http://www.sqlite.org/c3ref/busy_timeout.html). Another question is why do you want to get read of pthread_rwlock if it works for you? Pavel On Tue, May 24, 2011 at 7:43 AM, John Deal bassd...@yahoo.com wrote: Hello Roger, Sorry to be so brain-dead but I am still confused. I have multiple threads, each with their own DB connection. I want to allow multiple readers accessing the DB at the same time since nothing is changing. However, if a writes is to take place, I want all readers to finish their reads and give the writer exclusive access. Once the writer is done, the readers can come back in. I have all writes in transactions. If I deactivate my pthread_rwlock() that enforce the above, several writes fail with a database locked error (I assume it is returning SQLITE_BUSY). With my pthread_rwlock(), I have multiple threads reading the DB and my writes get the exclusive access they need. Now I could loop on the write until it gets in but that seems very wasteful. So how do I implement the equivalent of a pthread_rwlock() using SQLite mechinisms? Thanks, John --- On Tue, 5/24/11, Roger Binns rog...@rogerbinns.com wrote: From: Roger Binns rog...@rogerbinns.com Subject: Re: [sqlite] Multi-threading Common Problem To: General Discussion of SQLite Database sqlite-users@sqlite.org Date: Tuesday, May 24, 2011, 3:10 AM -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/23/2011 09:12 PM, John Deal wrote: I guess I am lost on how to obtain a many reader or one writer mutex in SQLite. You are confusing locks on the database and locks in the library on a sqlite3 pointer. The latter is what the mutex alloc function you reference is about and there there is no reader/writer mechanism. Access has to be serialized. You are correct in the locking article referenced I want a mutex that can have the lock states of shared, pending, and exclusive. Those are locks on the database which you get through regular operations and transactions. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk3bWeIACgkQmOOfHg372QQf8QCgjlawQMJWJ1I3/6OqMkczXswk VWQAmgLzGifXbh9UJpuEdUTTZl8e8xYp =rXCY -END PGP SIGNATURE- ___ sqlite
Re: [sqlite] Multi-threading Common Problem
Hello Roger, Sorry for the long delay. I did not see this in the swamp of recent email. I guess I am lost on how to obtain a many reader or one writer mutex in SQLite. I interpret the mutex returned by sqlite3_mutex_alloc() as being an exclusive mutex since the documentation states The sqlite3_mutex_enter() and sqlite3_mutex_try() routines attempt to enter a mutex. If another thread is already within the mutex, sqlite3_mutex_enter() will block and sqlite3_mutex_try() will return SQLITE_BUSY. If I interpret this correctly, this is to ensure serial access to the DB in a multi-connection/multi-thread environment. It seems if a recursive mutex is requested, the same thread can enter the mutex multiple times if it also frees the mutex the same amount of time. I don't understand how to implement a multi-reader or one writer mutex with sqlite3_mutex_alloc(). Should I be looking at something else? You are correct in the locking article referenced I want a mutex that can have the lock states of shared, pending, and exclusive. In other words, the same functionality as a pthread_rwlock() OS call. How do I get this? Sorry if I am missing something obvious. Thanks, John --- On Fri, 5/13/11, Roger Binns rog...@rogerbinns.com wrote: From: Roger Binns rog...@rogerbinns.com Subject: Re: [sqlite] Multi-threading Common Problem To: sqlite-users@sqlite.org Date: Friday, May 13, 2011, 2:00 AM -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/12/2011 01:26 PM, John Deal wrote: Good question. Very possible my understanding is not complete. This document has the full details: http://www.sqlite.org/lockingv3.html I have basically read and write transactions, each potentially with several accesses to the DB. I want to ensure that if a write transaction is happening, no read transactions are in progress You do know you can use transactions for reads? Or use multiple database connections to get isolation. If you worry about the efficiency of the latter then don't - ie get your code correct and then worry about performance. I recommend against the use of shared cache mode on general purpose computers (as opposed to embedded devices with trivial amounts of memory) because it changes some API behaviour (especially busy handling) and the amount of memory wasted is a rounding error. In any event it looks like I am not understanding some deeper aspect of what you are doing. My underlying point remains - there is absolutely no need to remove or workaround SQLite's builtin mutexes. They ensure that threaded code does not screw things up and are thoroughly tested/developed. Any question that starts with so I removed/changed/subverted SQLite's existing mutexes will be followed with a response where you'll need to prove that doing so isn't the cause. It is ok to augment them with your own locking but even that should not be necessary. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk3MyPkACgkQmOOfHg372QTTXgCcCa2bDbYH9WKQ2J2fPYhKLHPX DBgAoLoj+uRJ3GDIHWGU7TfgNXxDAuAH =exlM -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Multi-threading Common Problem
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/12/2011 01:26 PM, John Deal wrote: Good question. Very possible my understanding is not complete. This document has the full details: http://www.sqlite.org/lockingv3.html I have basically read and write transactions, each potentially with several accesses to the DB. I want to ensure that if a write transaction is happening, no read transactions are in progress You do know you can use transactions for reads? Or use multiple database connections to get isolation. If you worry about the efficiency of the latter then don't - ie get your code correct and then worry about performance. I recommend against the use of shared cache mode on general purpose computers (as opposed to embedded devices with trivial amounts of memory) because it changes some API behaviour (especially busy handling) and the amount of memory wasted is a rounding error. In any event it looks like I am not understanding some deeper aspect of what you are doing. My underlying point remains - there is absolutely no need to remove or workaround SQLite's builtin mutexes. They ensure that threaded code does not screw things up and are thoroughly tested/developed. Any question that starts with so I removed/changed/subverted SQLite's existing mutexes will be followed with a response where you'll need to prove that doing so isn't the cause. It is ok to augment them with your own locking but even that should not be necessary. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk3MyPkACgkQmOOfHg372QTTXgCcCa2bDbYH9WKQ2J2fPYhKLHPX DBgAoLoj+uRJ3GDIHWGU7TfgNXxDAuAH =exlM -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Multi-threading Common Problem
Hello All, I have been using SQLite for a couple of years but have never posted to this list before. I am sure my problem is common and am looking for ideas to solve it. I have used SQLite extensively single-threaded with no problems (other than my own!). I am currently working on another project adding SQLite functionality to a multi-threaded environment. Here is my situation. Ubuntu 10.04 64-bit. Have used the SQLite3 library and compiled from source directly into the application. Same results. Multi-threaded compile flag configuration kept as the default and forced with sqlite3_config() to SQLITE_CONFIG_MULTITHREAD with no error reported. Open is via SQLITE_OPEN_READWRITE | SQLITE_OPEN_NOMUTEX | SQLITE_OPEN_SHAREDCACHE flags. I have tried this with and without shared cache. When I use the above configuration multi-threaded but using only one DB connection in which only one thread is accessing the DB at a time, it works fine for both read and write. I use OS read/write mutex in which all threads must obtain a write lock to get at the DB effectively forcing only one thread accessing the DB at one time. When I allow multiple readers with each thread using a different DB connection (open with the same flags) and each thread having exclusive use of its DB connection (no sharing of connections) and if more than one thread is reading the DB at the same time, the DB becomes locked for writing even when all the reads are finished. The DB is locked, not the OS mutex. There are no DB writes. How can the DB be locked for writes in this situation? I test this with the sqlite3 program and opening the database while the application is running and try to do an insert. I have been working for weeks on this and I feel there must be something simple I am overlooking. Thanks for any help. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Multi-threading Common Problem
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/12/2011 09:38 AM, John Deal wrote: I have been working for weeks on this and I feel there must be something simple I am overlooking. Why are you discarding SQLite's builtin and tested mutexes and then effectively reimplementing your own to get the same effect? Or bigger picture question what is it you are trying to achieve in the first place? Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk3MPIkACgkQmOOfHg372QQzjgCg3106pWiiUMuOQay+2ONv3G0c ZvQAnAvBFXI+A8ae8tV9yXRmz7IZgid6 =jehy -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Multi-threading Common Problem
Good question. Very possible my understanding is not complete. I have basically read and write transactions, each potentially with several accesses to the DB. I want to ensure that if a write transaction is happening, no read transactions are in progress since it would be possible to have obtain incomplete data (mixture of some reads being valid but other no longer valid because the write transaction changed them). In other words, a read transaction (I do not use a transaction for the reads) consists of multiple pieces of data that makeup a set that I want to ensure is valid as a set. It is my understanding that a transaction (which I do use for the write transaction which is also a set) locks the DB for writes but not reads. If a transaction does lock the DB for exclusive access then you are correct, I do not need the OS mutexes. Maybe I do not understand the following: After a BEGIN EXCLUSIVE, no other database connection except for read_uncommitted connections will be able to read the database and no other connection without exception will be able to write the database until the transaction is complete. This tells me that reads outside of a transaction would be permitted while an exclusive transaction is taking place. If a write transaction is not taking place, I want to allow multiple reads which the OS rwlock allows. Any enlightenment would be welcomed. Thanks. --- On Thu, 5/12/11, Roger Binns rog...@rogerbinns.com wrote: From: Roger Binns rog...@rogerbinns.com Subject: Re: [sqlite] Multi-threading Common Problem To: sqlite-users@sqlite.org Date: Thursday, May 12, 2011, 4:01 PM -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/12/2011 09:38 AM, John Deal wrote: I have been working for weeks on this and I feel there must be something simple I am overlooking. Why are you discarding SQLite's builtin and tested mutexes and then effectively reimplementing your own to get the same effect? Or bigger picture question what is it you are trying to achieve in the first place? Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk3MPIkACgkQmOOfHg372QQzjgCg3106pWiiUMuOQay+2ONv3G0c ZvQAnAvBFXI+A8ae8tV9yXRmz7IZgid6 =jehy -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Multi-threading Common Problem
After a BEGIN EXCLUSIVE, no other database connection except for read_uncommitted connections will be able to read the database and no other connection without exception will be able to write the database until the transaction is complete. This tells me that reads outside of a transaction would be permitted while an exclusive transaction is taking place. This works only when shared cache mode is turned on and only within the same process. Nothing uncommitted can be read between processes or between different connections in the same process when shared cache is turned off. Pavel On Thu, May 12, 2011 at 4:26 PM, John Deal bassd...@yahoo.com wrote: Good question. Very possible my understanding is not complete. I have basically read and write transactions, each potentially with several accesses to the DB. I want to ensure that if a write transaction is happening, no read transactions are in progress since it would be possible to have obtain incomplete data (mixture of some reads being valid but other no longer valid because the write transaction changed them). In other words, a read transaction (I do not use a transaction for the reads) consists of multiple pieces of data that makeup a set that I want to ensure is valid as a set. It is my understanding that a transaction (which I do use for the write transaction which is also a set) locks the DB for writes but not reads. If a transaction does lock the DB for exclusive access then you are correct, I do not need the OS mutexes. Maybe I do not understand the following: After a BEGIN EXCLUSIVE, no other database connection except for read_uncommitted connections will be able to read the database and no other connection without exception will be able to write the database until the transaction is complete. This tells me that reads outside of a transaction would be permitted while an exclusive transaction is taking place. If a write transaction is not taking place, I want to allow multiple reads which the OS rwlock allows. Any enlightenment would be welcomed. Thanks. --- On Thu, 5/12/11, Roger Binns rog...@rogerbinns.com wrote: From: Roger Binns rog...@rogerbinns.com Subject: Re: [sqlite] Multi-threading Common Problem To: sqlite-users@sqlite.org Date: Thursday, May 12, 2011, 4:01 PM -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/12/2011 09:38 AM, John Deal wrote: I have been working for weeks on this and I feel there must be something simple I am overlooking. Why are you discarding SQLite's builtin and tested mutexes and then effectively reimplementing your own to get the same effect? Or bigger picture question what is it you are trying to achieve in the first place? Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk3MPIkACgkQmOOfHg372QQzjgCg3106pWiiUMuOQay+2ONv3G0c ZvQAnAvBFXI+A8ae8tV9yXRmz7IZgid6 =jehy -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Multi-threading Common Problem
Hello Pavel, This makes sense but I have shared cache on. Thanks. --- On Thu, 5/12/11, Pavel Ivanov paiva...@gmail.com wrote: From: Pavel Ivanov paiva...@gmail.com Subject: Re: [sqlite] Multi-threading Common Problem To: General Discussion of SQLite Database sqlite-users@sqlite.org Date: Thursday, May 12, 2011, 5:55 PM After a BEGIN EXCLUSIVE, no other database connection except for read_uncommitted connections will be able to read the database and no other connection without exception will be able to write the database until the transaction is complete. This tells me that reads outside of a transaction would be permitted while an exclusive transaction is taking place. This works only when shared cache mode is turned on and only within the same process. Nothing uncommitted can be read between processes or between different connections in the same process when shared cache is turned off. Pavel On Thu, May 12, 2011 at 4:26 PM, John Deal bassd...@yahoo.com wrote: Good question. Very possible my understanding is not complete. I have basically read and write transactions, each potentially with several accesses to the DB. I want to ensure that if a write transaction is happening, no read transactions are in progress since it would be possible to have obtain incomplete data (mixture of some reads being valid but other no longer valid because the write transaction changed them). In other words, a read transaction (I do not use a transaction for the reads) consists of multiple pieces of data that makeup a set that I want to ensure is valid as a set. It is my understanding that a transaction (which I do use for the write transaction which is also a set) locks the DB for writes but not reads. If a transaction does lock the DB for exclusive access then you are correct, I do not need the OS mutexes. Maybe I do not understand the following: After a BEGIN EXCLUSIVE, no other database connection except for read_uncommitted connections will be able to read the database and no other connection without exception will be able to write the database until the transaction is complete. This tells me that reads outside of a transaction would be permitted while an exclusive transaction is taking place. If a write transaction is not taking place, I want to allow multiple reads which the OS rwlock allows. Any enlightenment would be welcomed. Thanks. --- On Thu, 5/12/11, Roger Binns rog...@rogerbinns.com wrote: From: Roger Binns rog...@rogerbinns.com Subject: Re: [sqlite] Multi-threading Common Problem To: sqlite-users@sqlite.org Date: Thursday, May 12, 2011, 4:01 PM -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/12/2011 09:38 AM, John Deal wrote: I have been working for weeks on this and I feel there must be something simple I am overlooking. Why are you discarding SQLite's builtin and tested mutexes and then effectively reimplementing your own to get the same effect? Or bigger picture question what is it you are trying to achieve in the first place? Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk3MPIkACgkQmOOfHg372QQzjgCg3106pWiiUMuOQay+2ONv3G0c ZvQAnAvBFXI+A8ae8tV9yXRmz7IZgid6 =jehy -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Multi-threading Common Problem
Hello Roger, OK I see your point now. I could most likely remove the OS mutexes. Thanks, John --- On Thu, 5/12/11, Roger Binns rog...@rogerbinns.com wrote: From: Roger Binns rog...@rogerbinns.com Subject: Re: [sqlite] Multi-threading Common Problem To: sqlite-users@sqlite.org Date: Thursday, May 12, 2011, 4:01 PM -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/12/2011 09:38 AM, John Deal wrote: I have been working for weeks on this and I feel there must be something simple I am overlooking. Why are you discarding SQLite's builtin and tested mutexes and then effectively reimplementing your own to get the same effect? Or bigger picture question what is it you are trying to achieve in the first place? Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk3MPIkACgkQmOOfHg372QQzjgCg3106pWiiUMuOQay+2ONv3G0c ZvQAnAvBFXI+A8ae8tV9yXRmz7IZgid6 =jehy -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] multi-threading multi-db problem
Hello! I use the latest (3.6.11) version of the SQLite dll in a WinXP Application. The main task of this application is to store values from a sensor-hardware in multiple databases. Each DB contains a subset from the Values. I tested serveral design options. One is to read all values from the senor hardware and split them to the different DBs. operational sequence: --main-- create and init 2 databases in the main-function. --main-- create 2 Threads and pass the 2 DB-handle parameter (one thread per DB!!!) --thread-- fill the DBs with the values In the last step occurs an Error. Sometimes the threads dont write in the correct DB!!! I inserted 2 values in one DB. DB0.db have to look like this: index | subid | time | 0.0 --- DB1.db have to look like this: index | subid | time | 1.0 --- What i got was: DB0.db: sqlite SELECT * FROM ArchiveLog WHERE Value==1; 11438|37|33:10.897|1.0 14180|19|33:13.647|1.0 19854|29|33:19.241|1.0 DB1.db sqlite SELECT * FROM ArchiveLog WHERE Value==0; 18719|16|33:18.101|0.0 The other Values in the DBs are correct. Best regards! ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Multi-threading problem!
Hello! I use the latest (3.6.10) version of SQLite library. I was trying to perform query select from one table inside single database connection from different threads(up to 8 threads). In summary I had 50% CPU Usage( 2 core CPU)-this is a problem! I tried to find the reason in Debug mode and I've found that the most of the execution time one thread was executing when other threads waited him.. So I have at least two questions: 1.What may be a reason for this lack of performance? 2.May be I there is a RIGHT way for multi-thread using of SQLite? Best regards! ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Multi-threading problem!
Create a connection for each thread. --- On Wed, 1/28/09, Anatoly Stepanov anatoly.stepa...@kaspersky.com wrote: From: Anatoly Stepanov anatoly.stepa...@kaspersky.com Subject: [sqlite] Multi-threading problem! To: sqlite-users@sqlite.org sqlite-users@sqlite.org Date: Wednesday, January 28, 2009, 7:58 AM Hello! I use the latest (3.6.10) version of SQLite library. I was trying to perform query select from one table inside single database connection from different threads(up to 8 threads). In summary I had 50% CPU Usage( 2 core CPU)-this is a problem! I tried to find the reason in Debug mode and I've found that the most of the execution time one thread was executing when other threads waited him.. So I have at least two questions: 1.What may be a reason for this lack of performance? 2.May be I there is a RIGHT way for multi-thread using of SQLite? Best regards! ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Multi-threading.
On Sat, 2005-07-30 at 14:30 +0200, Jan-Eric Duden wrote: Win9X doesn't support the API async file operations. WinNT/2K/XP does support it. It supports everything it needs to: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/waitformultipleobjects.asp it doesn't support async file NOTIFICATIONS, (afaik), but those aren't necessary... Regarding threads: Threads complicate things. People think they understand threads long before they actually do. Goto complicates things. People think they understand goto long before they actually do. That's a completely normal property of human nature. You should get used to it. This behavior is not limited to the issue of using thread or goto. It applies to everything that needs some skill. Most people just don't know what they can do or can't do until they fail. Even if they fail they think they succeeded. To make things worse, once you taught someone to know better, then someone else will take his/her place. Agreed.
Re: [sqlite] Multi-threading.
Mrs. Brisby wrote: On Sat, 2005-07-30 at 14:30 +0200, Jan-Eric Duden wrote: Win9X doesn't support the API async file operations. WinNT/2K/XP does support it. It supports everything it needs to: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/waitformultipleobjects.asp it doesn't support async file NOTIFICATIONS, (afaik), but those aren't necessary... How should waitformultipleobjects help with aync file io? For async file io on Win32 you would open a file with CreateFile and pass the FILE_FLAG_OVERLAPPED flag. After that, functions like WriteFile and ReadFile work asynchronously. Unfortunately, on Win9X/ME it is not supported for files on disk. For example, see the docs of WriteFile/ReadFile/WriteFileEx/ReadFileEx: *Windows 95/98/Me: *For asynchronous write operations, /hFile/ can be a communications resource opened with the FILE_FLAG_OVERLAPPED flag by *CreateFile*, or a socket handle returned by *socket* or *accept*. You cannot perform asynchronous write operations on mailslots, named pipes, or *disk files*. Regarding threads: Threads complicate things. People think they understand threads long before they actually do. Goto complicates things. People think they understand goto long before they actually do. That's a completely normal property of human nature. You should get used to it. This behavior is not limited to the issue of using thread or goto. It applies to everything that needs some skill. Most people just don't know what they can do or can't do until they fail. Even if they fail they think they succeeded. To make things worse, once you taught someone to know better, then someone else will take his/her place. Agreed.
Re: [sqlite] Multi-threading.
On Sat, 2005-07-30 at 20:29 +0200, Jan-Eric Duden wrote: Mrs. Brisby wrote: On Sat, 2005-07-30 at 14:30 +0200, Jan-Eric Duden wrote: Win9X doesn't support the API async file operations. WinNT/2K/XP does support it. It supports everything it needs to: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/waitformultipleobjects.asp it doesn't support async file NOTIFICATIONS, (afaik), but those aren't necessary... How should waitformultipleobjects help with aync file io? It doesn't. Read below. For async file io on Win32 you would open a file with CreateFile and pass the FILE_FLAG_OVERLAPPED flag. After that, functions like WriteFile and ReadFile work asynchronously. Unfortunately, on Win9X/ME it is not supported for files on disk. For example, see the docs of WriteFile/ReadFile/WriteFileEx/ReadFileEx: *Windows 95/98/Me: *For asynchronous write operations, /hFile/ can be a communications resource opened with the FILE_FLAG_OVERLAPPED flag by *CreateFile*, or a socket handle returned by *socket* or *accept*. You cannot perform asynchronous write operations on mailslots, named pipes, or *disk files*. No you don't. It's not just that Windows 3.11/95/98/ME don't support FILE_FLAG_OVERLAPPED - its that their filesystem isn't reentrant. It cannot do a context switch during that either. You use a large read-size (bigger than the disk-block) and it can context-switch inbetween disk operations. Threads look like a win here! You can also use the disk-block reads. The operating system you're talking about is uniprocessor only. Process control returns to your program just as fast as a thread-switch can occur, and you've saved a context-switch. Use waitformultipleobjects to check your event pump after each readFile or writeFile operation. [[ as a side note: no, it's not really that easy. If the disk is asleep, spun down, or if you run two programs you need to do extra work: * wake up the disk * lock the underlying physical disk Because your code is simpler, these things aren't such a big deal. ]] I will personally advocate splitting processes into a worker-master situation- I just don't recommend pool-threading. That's a lie. I recommend against pool-threading.
Re: [sqlite] Multi-threading.
Just FYI... All you people posting to comp.sys.15yearoldarguments - you know you're also cross-posting to the sqlite mailing list, right? Tim Mrs. Brisby wrote: On Sat, 2005-07-30 at 20:29 +0200, Jan-Eric Duden wrote: Mrs. Brisby wrote: On Sat, 2005-07-30 at 14:30 +0200, Jan-Eric Duden wrote: Win9X doesn't support the API async file operations. WinNT/2K/XP does support it. It supports everything it needs to: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/waitformultipleobjects.asp it doesn't support async file NOTIFICATIONS, (afaik), but those aren't necessary... How should waitformultipleobjects help with aync file io? It doesn't. Read below. For async file io on Win32 you would open a file with CreateFile and pass the FILE_FLAG_OVERLAPPED flag. After that, functions like WriteFile and ReadFile work asynchronously. Unfortunately, on Win9X/ME it is not supported for files on disk. For example, see the docs of WriteFile/ReadFile/WriteFileEx/ReadFileEx: *Windows 95/98/Me: *For asynchronous write operations, /hFile/ can be a communications resource opened with the FILE_FLAG_OVERLAPPED flag by *CreateFile*, or a socket handle returned by *socket* or *accept*. You cannot perform asynchronous write operations on mailslots, named pipes, or *disk files*. No you don't. It's not just that Windows 3.11/95/98/ME don't support FILE_FLAG_OVERLAPPED - its that their filesystem isn't reentrant. It cannot do a context switch during that either. You use a large read-size (bigger than the disk-block) and it can context-switch inbetween disk operations. Threads look like a win here! You can also use the disk-block reads. The operating system you're talking about is uniprocessor only. Process control returns to your program just as fast as a thread-switch can occur, and you've saved a context-switch. Use waitformultipleobjects to check your event pump after each readFile or writeFile operation. [[ as a side note: no, it's not really that easy. If the disk is asleep, spun down, or if you run two programs you need to do extra work: * wake up the disk * lock the underlying physical disk Because your code is simpler, these things aren't such a big deal. ]] I will personally advocate splitting processes into a worker-master situation- I just don't recommend pool-threading. That's a lie. I recommend against pool-threading.
Re: [sqlite] Multi-threading.
On Thu, 2005-07-28 at 08:44 -0400, Eli Burke wrote: Mrs. Brisby wrote: My PII-350 easily handles 2 million local deliveries per hour, and it fork()s for each one. As has been mentioned: If you think threads are going to make your program fast, you don't know what you're doing. Like it or not, the world is bigger than just Unix. Some of us have to write code for other platforms, or even cross-platform. You can preach until you turn green about the dangers of threads in the hands of all the rest of us dumb idiots, but that doesn't change the fact that sometimes they are an appropriate (or the only) solution. Like it or not, you still don't have any idea what you are doing. They [threads] are never the only solution, and they are rarely appropriate. I'm certain I'm not being entirely clear on just how rare it is: Threads so rarely help that the programs that they CAN help don't often use them, and the programs that it DOESN'T HELP- nay actually hurts, are the ones using threads. 20 years ago, people said the exact same thing about goto- the places where it can help are in programs that don't often use them, but the places where it DOESN'T HELP- and actually hurts, are the ones using goto. Now, if you're feeling like you're on a high horse, go ahead: point out a single common platform where threads are the necessity. Here's some hints: * it's not windows * it's not unix * it's not java (closures) * it's not lisp (closures) * it's not forth (coroutines) * it's not macos (see unix) * it's not gtk+ (closures/event pump) That said: Writing kernel processes for the linux kernel requires a lot of thread-think in that you have to keep all of your memory access private. I generally suspect, however, that the people having problems with threads _there_ aren't the same people who are having problems with threads _here_. Mrs. Brisby wrote: Maybe people think that SQLite should serialize its own internals. Maybe that's why people keep saying this. [snip] meanwhile, two threads attempting to multiplex access to a single sqlite handle are a) bound to be confused, and b) need those resource locks anyway. I don't think anyone is asking for SQLite to be serialized, or for it to work properly when a db handle is reused by multiple threads. Yes, there are a lot of questions about threads on the mailing list. Maybe it is because Kervin Pierre is right-- the documentation RE threads is poor. SQLite doesn't have anything to do with threads. That's the problem. What people miss is that they either think memory-chunk serialization is automatic, or automatically provided by SQLite. But SQLite isn't different from the majority of third-party libraries out there, and if _THIS_ is what is tripping people up, then it should be all the more evident that thread-think makes people dumb. [[ personally: I don't think this is what's tripping people up. _I_ think that they really don't know how to use threads in the first place. ]] It's no big secret that Dr. Hipp is in the threads are bad camp, and so getting helpful information means spending hours reading through old mailing list posts, sorting through the chaff, trying to figure out what behavior applies to the current software version. (a number of significant threading issues have been resolved since 3.0 was released). There is a short article in the wiki on threads, but you have to know to look for it. No. In order to use threads with SQLite you need to know how to use threads. In order to use threads with ImageMagick, you need to know how to use threads. In order to use threads with PostgreSQL you need to know how to use threads. In order to use threads with libmodplug you need to know how to use threads. Notice a common theme? All of these libraries require the user do their own serialization, and yet people continue to ask how do I make OpenLDAP work with threads when they really mean how do I use threads. There isn't any grey area here. Threads are complicated- so complicated that they screw up otherwise promising programmers into thinking that they're necessary and/or easy, when they're clearly neither. Consider the lone FAQ entry on threading. It contains two helpful bits of advice. 1) Don't reuse db handles between threads. 2) Don't reuse db handles after fork(). Now imagine if that information was actually in a useful place, namely in the sqlite3_open API reference. Perhaps that would cut down on some new user confusion? I doubt it. It's worth a shot though.
Re: [sqlite] Multi-threading.
On Thu, 2005-07-28 at 07:42 -0500, Dennis Jenkins wrote: Mrs. Brisby wrote: meanwhile, two threads attempting to multiplex access to a single sqlite handle are a) bound to be confused, and b) need those resource locks anyway. (background: I've been using threads on win32 since 1998. Sometimes to spread load across multiple CPUs, but usually just because I find it simpler than state logic to do everything in one thread. I'm not entering into this debate on threads vs. non-threads.) I find I use goto a lot. I find it's simpler than lots of if/break scenarios. I think you'd be surprised to find out that you might actually agree with me :) I just want to know why anyone would code using multiple threads accessing the same SQLITE connection object ( or handle or whatever you want to call it). I allocate a separate connection object for each thread and have ZERO troubles with locking so long as I use transactions properly. Assuming a multi-threaded (with in the same process) environment, what benefits are there to use a single (global?) SQLITE object (protected by a mutex, semaphore, critical section, whatever)? It seems so much more complicated and brittle. The reason they would write their code this way is because they don't know how to use threads. They think threads work differently than they do. And folks, this here is my whole point. Coming from the position of someone who knows how to use threads: Why would anyone chose to use them incorrectly. The only reasonable answer is that they simply don't know how to use threads. Threads complicate things. People think they understand threads long before they actually do. Goto complicates things. People think they understand goto long before they actually do. They think they know the dangers of threads, and they think they know the dangers of goto. And yet, there are folks who vehemently avoid goto as if it were a plague, that simply don't understand why people would just as aggressively reject threads.
RE: [sqlite] Multi-threading.
How 'bout Y'all take this off-line before I have to track you down and lay some Texas justice on both you... Mentioning SQLite every few sentences does not mask this religious warfare. -Original Message- From: Mrs. Brisby [mailto:[EMAIL PROTECTED] Sent: Friday, July 29, 2005 3:15 PM To: sqlite-users@sqlite.org Subject: Re: [sqlite] Multi-threading. On Thu, 2005-07-28 at 07:42 -0500, Dennis Jenkins wrote: Mrs. Brisby wrote: meanwhile, two threads attempting to multiplex access to a single sqlite handle are a) bound to be confused, and b) need those resource locks anyway. ...
Re: [sqlite] Multi-threading.
Like it or not, you still don't have any idea what you are doing. They [threads] are never the only solution, and they are rarely appropriate. I'm certain I'm not being entirely clear on just how rare it is: Threads so rarely help that the programs that they CAN help don't often use them, and the programs that it DOESN'T HELP- nay actually hurts, are the ones using threads. I wouldn't generalize to *quite* this extreme, being as I have not written nor reviewed even a small fraction of a percent of the millions of programs written on various platforms. However, I will say this: Threads are a tool ... sortof like a chainsaw. Used properly, they can save you time and accomplish your goal. In the hands of an expert, you can even make some impressive art. Well, perhaps the latter only applies to chainsaws, but you get the idea. In the hands of the ignorant or inexperienced, you can easily lop off an arm or a leg, or getting back to threading, seriously undermine the integrity of your code. Threading, like chainsaws, can be dangerous if not used properly. 20 years ago, people said the exact same thing about goto- the places where it can help are in programs that don't often use them, but the places where it DOESN'T HELP- and actually hurts, are the ones using goto. I used goto's back when I was programming on the Vic-20 and I only had 3k of RAM to work with -- every byte was valuable back then. I also couldn't understand the various branches of my code a week later when I went to debug it, so I had a love-hate relationship with gotos. I don't use them primarily because they, like threads, are easily abused and can potentially make your code extremely hard to follow. Case in point, one of the older programmers in my office uses goto's liberally (on one project we discovered a 1:9 ratio of goto's in his code -- meaning 1 goto for every 9 lines of code on average). His code is unintelligible. He has labels inside his for() loops and goto's jumping in and out of them. He's got code leapfrogging all over creation, and its the sickest ugliest thing I've ever witnessed. Who in hell even knows if the stuff works -- its all but impossible to understand without running it. Now, if you're feeling like you're on a high horse, go ahead: point out a single common platform where threads are the necessity. That argument is silly. There exist no *platforms* where multithreading is a necessity. The platform doesn't dictate whether threading is useful or warranted for a given application. The programmer decides that. [snip] No. In order to use threads with SQLite you need to know how to use threads. In order to use threads with ImageMagick, you need to know how to use threads. In order to use threads with PostgreSQL you need to know how to use threads. In order to use threads with libmodplug you need to know how to use threads. Notice a common theme? All of these libraries require the user do their own serialization, and yet people continue to ask how do I make OpenLDAP work with threads when they really mean how do I use threads. The real problem many coders encounter when writing multithreaded applications is that some of the underlying libraries they may be using may include optimizations that make the library inherently unsafe in a multithreaded environment. For example, a library that uses a global variable to store a handle to a database file, and all instances of the object simply refer back to that global handle. You could code all the synchronization objects you like and make sure all the object instances never cross a thread boundary, but in the end if the underlying library's objects are all sharing a global variable unbenknownst to you, then your code is still going to have a potentially lethal race condition. I think *this* is the major issue people are concerned about when they come to the SQLite list asking if SQLite is threadsafe. There isn't any grey area here. Threads are complicated- so complicated that they screw up otherwise promising programmers into thinking that they're necessary and/or easy, when they're clearly neither. I absolutely 100% agree. Threads are dangerous and their benefits should be weighed before applying them to your application(s). Personally, I've written quite a few programs that have benefitted significantly from multithreading, most of them server-based. Those particular apps would've crashed and burned horribly when placed under a heavy load without using threads, so one could argue that they were not only beneficial but necessary to their success. Robert Simpson Programmer at Large
Re: [sqlite] Multi-threading.
Some class-libraries make multithreading programming really easy, piece of cake, even when there are points to synchronize again to a single thread (f.e. interface), it is easy. You do not need to be a genius to write multithreading/multithreaded code kind regards Bert Verhees
Re: [sqlite] Multi-threading.
Mrs. Brisby wrote: Now, if you're feeling like you're on a high horse, go ahead: point out a single common platform where threads are the necessity. Here's some hints: * it's not windows * it's not unix * it's not java (closures) * it's not lisp (closures) * it's not forth (coroutines) * it's not macos (see unix) * it's not gtk+ (closures/event pump) Argh... gtk+ uses threads on windows. The g_io_channel async api is implemented this way. This is done because windows has no async file io api. There may be other places/platforms where glib/gtk+ uses threads. signature.asc Description: OpenPGP digital signature
RE: [sqlite] Multi-threading.
Reminds me alt.C in the News Groups, arguing over curly bracket placement :-( -Original Message- From: Puneet Kishor [mailto:[EMAIL PROTECTED] Sent: Friday, July 29, 2005 5:23 PM To: sqlite-users@sqlite.org Subject: Re: [sqlite] Multi-threading. Fred Williams wrote: How 'bout Y'all take this off-line before I have to track you down and lay some Texas justice on both you... Mentioning SQLite every few sentences does not mask this religious warfare. this 'thread' is not going to die, no matter where you tell them to 'goto' *cough* ...
Re: [sqlite] Multi-threading.
On Fri, 2005-07-29 at 16:22 -0600, K. Haley wrote: Mrs. Brisby wrote: Now, if you're feeling like you're on a high horse, go ahead: point out a single common platform where threads are the necessity. Here's some hints: * it's not windows * it's not unix * it's not java (closures) * it's not lisp (closures) * it's not forth (coroutines) * it's not macos (see unix) * it's not gtk+ (closures/event pump) Argh... gtk+ uses threads on windows. The g_io_channel async api is implemented this way. This is done because windows has no async file io api. There may be other places/platforms where glib/gtk+ uses threads. Err, windows most certainly does have Async API.
Re: [sqlite] Multi-threading.
On Fri, 2005-07-29 at 14:29 -0700, Robert Simpson wrote: Threads are a tool ... sortof like a chainsaw. Used properly, they can save you time and accomplish your goal. In the hands of an expert, you can even make some impressive art. Well, perhaps the latter only applies to chainsaws, but you get the idea. In the hands of the ignorant or inexperienced, you can easily lop off an arm or a leg, or getting back to threading, seriously undermine the integrity of your code. Threading, like chainsaws, can be dangerous if not used properly. Agreed. 20 years ago, people said the exact same thing about goto- the places where it can help are in programs that don't often use them, but the places where it DOESN'T HELP- and actually hurts, are the ones using goto. I used goto's back when I was programming on the Vic-20 and I only had 3k of RAM to work with -- every byte was valuable back then. I also couldn't understand the various branches of my code a week later when I went to debug it, so I had a love-hate relationship with gotos. Like chainsaws, gotos aren't useful for every task. I don't use them primarily because they, like threads, are easily abused and can potentially make your code extremely hard to follow. Case in point, one of the older programmers in my office uses goto's liberally (on one project we discovered a 1:9 ratio of goto's in his code -- meaning 1 goto for every 9 lines of code on average). His code is unintelligible. He has labels inside his for() loops and goto's jumping in and out of them. He's got code leapfrogging all over creation, and its the sickest ugliest thing I've ever witnessed. Who in hell even knows if the stuff works -- its all but impossible to understand without running it. Agreed. Now, if you're feeling like you're on a high horse, go ahead: point out a single common platform where threads are the necessity. That argument is silly. There exist no *platforms* where multithreading is a necessity. The platform doesn't dictate whether threading is useful or warranted for a given application. The programmer decides that. Hence the point. Look at the parent and grandparent. There are many people posting on this thread that _believe_ that threads are a necessity. The real problem many coders encounter when writing multithreaded applications is that some of the underlying libraries they may be using may include optimizations that make the library inherently unsafe in a multithreaded environment. For example, a library that uses a global variable to store a handle to a database file, and all instances of the object simply refer back to that global handle. You could code all the synchronization objects you like and make sure all the object instances never cross a thread boundary, but in the end if the underlying library's objects are all sharing a global variable unbenknownst to you, then your code is still going to have a potentially lethal race condition. I think *this* is the major issue people are concerned about when they come to the SQLite list asking if SQLite is threadsafe. Right. But they don't ask that. They say I found a bug in SQLite when using threads.
Re: [sqlite] Multi-threading.
Mrs. Brisby wrote: meanwhile, two threads attempting to multiplex access to a single sqlite handle are a) bound to be confused, and b) need those resource locks anyway. (background: I've been using threads on win32 since 1998. Sometimes to spread load across multiple CPUs, but usually just because I find it simpler than state logic to do everything in one thread. I'm not entering into this debate on threads vs. non-threads.) I just want to know why anyone would code using multiple threads accessing the same SQLITE connection object ( or handle or whatever you want to call it). I allocate a separate connection object for each thread and have ZERO troubles with locking so long as I use transactions properly. Assuming a multi-threaded (with in the same process) environment, what benefits are there to use a single (global?) SQLITE object (protected by a mutex, semaphore, critical section, whatever)? It seems so much more complicated and brittle.
Re: [sqlite] Multi-threading.
Mrs. Brisby wrote: My PII-350 easily handles 2 million local deliveries per hour, and it fork()s for each one. As has been mentioned: If you think threads are going to make your program fast, you don't know what you're doing. Like it or not, the world is bigger than just Unix. Some of us have to write code for other platforms, or even cross-platform. You can preach until you turn green about the dangers of threads in the hands of all the rest of us dumb idiots, but that doesn't change the fact that sometimes they are an appropriate (or the only) solution. Mrs. Brisby wrote: Maybe people think that SQLite should serialize its own internals. Maybe that's why people keep saying this. [snip] meanwhile, two threads attempting to multiplex access to a single sqlite handle are a) bound to be confused, and b) need those resource locks anyway. I don't think anyone is asking for SQLite to be serialized, or for it to work properly when a db handle is reused by multiple threads. Yes, there are a lot of questions about threads on the mailing list. Maybe it is because Kervin Pierre is right-- the documentation RE threads is poor. It's no big secret that Dr. Hipp is in the threads are bad camp, and so getting helpful information means spending hours reading through old mailing list posts, sorting through the chaff, trying to figure out what behavior applies to the current software version. (a number of significant threading issues have been resolved since 3.0 was released). There is a short article in the wiki on threads, but you have to know to look for it. Consider the lone FAQ entry on threading. It contains two helpful bits of advice. 1) Don't reuse db handles between threads. 2) Don't reuse db handles after fork(). Now imagine if that information was actually in a useful place, namely in the sqlite3_open API reference. Perhaps that would cut down on some new user confusion? -Eli
Re: [sqlite] Multi-threading.
At 03:21 27/07/2005, Mrs. Brisby wrote: On Mon, 2005-07-25 at 09:48 -0500, Jay Sprenkle wrote: The theory has been proposed that threads aren't better than separate processes, or application implemented context switching. Does anyone have an experiment that will prove the point either way? It will have to be OS specific though, since I'm sure not all thread implementations are equal. This page might be interesting.. http://john.redmood.com/osfastest.html It shows (pretty conclusively), that 'one process per task' is not the way to go if you want any sort of performance. One thread per task is very good One thread for many tasks is slightly better, with the benefit growing as more threads are created (up to 300-500 tasks there's not a massive difference, by the time you get to 1000 tasks there's a 35% benefit to using one thread for many tasks. Of course, this assumes a well designed architecture... But, one process per task is very poor in comparison to the other ways (on all the platforms they tested) handling only about 5% of load of the 'one thread for many tasks' architecture. PaulVPOP3 - Internet Email Server/Gateway [EMAIL PROTECTED] http://www.pscs.co.uk/
Re: [sqlite] Multi-threading.
On Wed, 27 Jul 2005, Paul Smith wrote: At 03:21 27/07/2005, Mrs. Brisby wrote: On Mon, 2005-07-25 at 09:48 -0500, Jay Sprenkle wrote: The theory has been proposed that threads aren't better than separate processes, or application implemented context switching. Does anyone have an experiment that will prove the point either way? It will have to be OS specific though, since I'm sure not all thread implementations are equal. This page might be interesting.. http://john.redmood.com/osfastest.html This whole article is pretty flawed. The performance of creating and deleting files between the different systems can be explained by the relative safety of the different systems. FreeBSD and Solaris both order metadata writes to minimize FS damage on OS or power failure. Ext2 under Linux is well known for foregoing these safe guards by default. Using a journalled filesystem (Solaris 7+ has journalled UFS, ext3 on Linux) or soft-updates on FreeBSD should probably level this playing field. Don't know about Windows, though. It shows (pretty conclusively), that 'one process per task' is not the way to go if you want any sort of performance. One process per request, from the article. That's HTTP 1.0. Most systems don't handle one request then exit, especially database systems. Oracle, PostgreSQL and probably others have a process per session, using shared memory and IPC for synchronisation, and scale very nicely thank you. The other problem with the test is that the example is not very representitive of real work. Simply writing a 405 - too busy to a request does not constitute actual work, and as soon as the single threaded, async implementation does any work in that thread, throughput will go down and latency will increase. On the other hand, the threaded implementation should maintain throughput even if requests result in blocking, or the machine has more than 1 CPU and the threads can operate concurrently. Of course, I'm advocating neither one method or another, it's all a bit grey, and can see both sides of the argument. It would be nice if SQLite played nicer with threads, but it doesn't at the moment. One thread per task is very good One thread for many tasks is slightly better, with the benefit growing as more threads are created (up to 300-500 tasks there's not a massive difference, by the time you get to 1000 tasks there's a 35% benefit to using one thread for many tasks. For real work, the drop off of scalability in the threaded instance would probably not be as great, as the given test represents about the worst case for threads (little concurrency, IO bound, context thrashing) Of course, this assumes a well designed architecture... The test was far from it... But, one process per task is very poor in comparison to the other ways (on all the platforms they tested) handling only about 5% of load of the 'one thread for many tasks' architecture. As I said above, the test from the article is loaded against process per task architectures by the HTTP 1.0 nature of the task. The numbers will not be representitive of real situations. Perhaps a better test would be to use the different MPMs of Apache against each other: http://httpd.apache.org/docs/2.0/mpm.html There is no MPM for single thread, async IO, presumably because async IO is not an option once you start using third-party plugins. My 2c Christian -- /\ \ /ASCII RIBBON CAMPAIGN - AGAINST HTML MAIL X - AGAINST MS ATTACHMENTS / \
Re: [sqlite] Multi-threading.
On Tue, Jul 26, 2005 at 10:21:22PM -0400, Mrs. Brisby wrote: That's incorrect. Threading increases development time and produces less stable applications. In fairness: it's the skill level of the engineer Mrs. Brisby, that is probably quite correct in at least one particular sense, but since you're so eager to bring science into the discussion, you should define your terms more carefully. When people - even smart and knowledgeable people - say thread, they typically mean one of two related but quite distinct concepts, and they often don't say precisely which they mean: They may mean thread of control, or they may mean, threads as implemented by system X, where X usually means either POSIX or Win32 threads. It would be nice if there were clearly distinct names for those two things, but I don't know of any. A thread of control is a conceptual entity, and is usually contrasted to event-driven state machines, both of which are used for concurrent programming. Using 5 POSIX threads and using 5 Unix processes are BOTH examples of using 5 threads of control. In the conceptual sense they are BOTH using threads in contrast to events. I see that sense of thread of control in the academic computer science literature a lot. They mostly seem convinved that mutiple threads of control are usually much easier to program than state machines. Thus they like to write papers on different ways of minimizing the overhead of threads vs. state machines on a single CPU, how to best take advantage of multiple CPUs, implementing lightweight threads on top of an event driven core, etc. etc. Often, whether memory is default-shared or default-private seems to be an implementation detail of little direct interest to those researchers. Working programmers who want to write applications, not their own operating system, compiler, or related libraries, don't much care about that. Their OS and other tools typically support and encourage a TINY handful of tools for concurrent programming - many of which often suck. The danger is that plenty of those working programmers know nothing about the other 95% of the possibilities their particular OS does NOT give them, so they make lots of overly broad generalizations. And a serious computing SCIENTIST certainly should not allow himself (or herself) to make that error. Now here's the interesting part: When it comes to different styles of implementing threads of control (processes vs. POSX threads, etc.), like lots of things with computers, it's often not that hard to build one style on top of an implemention originally intended for another style - sometimes even with decent efficiency. For example, it is quite possible - and sometimes useful - to implement a private-mostly, optionally shared process-like memory model on top of shared-everything POSIX or Win32 threads. E.g., AOLserver and Tcl do that. Erlang does not, but could. (And they each have their reasons for that.) I doubt that it would be ever be useful to implement a shared-mostly model on top of multiple Unix processes and System V shared memory, but you could if you really wanted to. thread vs. processes just isn't a terribly well posed argument. It is an ugly smoosheed together mess of several different orthogonal concepts: 1. Mulitple threads of control vs. an event-driven state machine. 2. Default-shared vs. default-private memory. 3. Shared memory vs. message passing. And those are still just a few dimensions in the space of techniques for concurrent programming. Here's at least ONE other important dimension that everyone on the SQLite list is probably familiar with: transactions vs. explicit locking. And, aha, if you had good support for TRANSACTIONS in-memory in your programming language environement (aka, transactional memory), how much would it then matter to you whether your concurrent transactions were implemented default-shared or default-private memory underneath? Not much! Think: Do I as a user care that Oracle is implemented multi-process w/ System V shared memory while SQL Server uses Win32 threads? Only slightly, if at all. Why? Because transactions are a higher level abstraction, and as abstractons go it's pretty wateright, not very leaky at all. (I am MUCH more interested in the fact that Oracle uses MVCC while SQL Server uses pessimistic locking - different flavors of transactions.) My personal suspicion is that there probably many more such dimensions to concurrent programming technique, mostly poorly investigated or poorly known (or just plain not invented yet at all), some of which are hiding vastly larger productivity gains than belaboring of threads vs. processes. Related to that, although I haven't read it yet, this book was recently highly recommended to me: Concepts, Techniques, and Models of Computer Programming by Peter Van Roy, Seif Haridi http://www.amazon.com/exec/obidos/tg/detail/-/0262220695/104-1093447-6927163 A reasonable analogy to threads vs. processes
Re: [sqlite] Multi-threading.
On Tue, 2005-07-26 at 23:20 -0400, Kervin L. Pierre wrote: We get all those threads related questions because SQLite is difficult to use with threads and has sparse sometimes conflicting documentation on the subject. Maybe people think that SQLite should serialize its own internals. Maybe that's why people keep saying this. It shouldn't. Those serialization steps are expensive and not always necessary. A programmer using thread-think should be able to create the appropriate locks WHEN appropriate- e.g. a gui application with an SQLite worker thread won't need any locks on sqlite's data structures because only one thread will be accessing them. meanwhile, two threads attempting to multiplex access to a single sqlite handle are a) bound to be confused, and b) need those resource locks anyway. Don't get me wrong, again I am not complaining, it is *way* better than the other options out there. But you can't expect someone to magically figure out an API based on one or two sentence definitions and comments in a header file without asking a few questions and making a few mistakes. I _NEVER_ said that I expect people to figure out an API through magical means. I _would_ however expect people trying to use a third-party library to already have isolated their problem _to_ that third party library. I resent people saying: I'm writing a program using threads and SQLite doesn't work with it, when what they should be saying is I don't know how to use threads.
Re: [sqlite] Multi-threading.
On Wed, 2005-07-27 at 09:08 +0100, Paul Smith wrote: At 03:21 27/07/2005, Mrs. Brisby wrote: On Mon, 2005-07-25 at 09:48 -0500, Jay Sprenkle wrote: The theory has been proposed that threads aren't better than separate processes, or application implemented context switching. Does anyone have an experiment that will prove the point either way? It will have to be OS specific though, since I'm sure not all thread implementations are equal. This page might be interesting.. http://john.redmood.com/osfastest.html My PII-350 easily handles 2 million local deliveries per hour, and it fork()s for each one. As has been mentioned: If you think threads are going to make your program fast, you don't know what you're doing. Unit testing and a profiler will give much greater performance gains than trying to emulate piece of shit applications like MailEngine.
Re: [sqlite] Multi-threading.
On Wed, 2005-07-27 at 13:31 -0400, Andrew Piskorski wrote: On Tue, Jul 26, 2005 at 10:21:22PM -0400, Mrs. Brisby wrote: That's incorrect. Threading increases development time and produces less stable applications. In fairness: it's the skill level of the engineer ... When people - even smart and knowledgeable people - say thread, they typically mean one of two related but quite distinct concepts, and they often don't say precisely which they mean: [snip] I don't think there's any confusion here: We certainly are referring to shared memory concurrent multiprogramming. If someone was accidentally referring to something else, then they can be safely excused. This discussion was never academic, but practical. There aren't any transactional memory systems being used here. Javascript, LISP, Perl, TCL and Erlang aren't relevant, because people using SQLite on those platforms with those languages aren't having the multi-threading problems that spawned this discussion. People using those languages to do real work aren't running into the problems that we're talking about. You brought up some excellent examples of highly successful programs that use threads or thread-think. I commend that. It's also completely irrelevant. It's not that nobody can use threads safely- it's that those who can rarely chose to use threads because the skillset that comes with knowing how to use threads safely is axillary to the skillset that comes from knowing how to use goto safely, and how to write serialized event-driven applications easily. I keep using terms like thread-think and process-think and event-think: It's evident that one can write event-driven programs with threads (Amoeba), or processes-think programs with threads (perl) or process-think programs with events (GNU Pth) or event-think programs with processes (yay signals). thread-think is the idea of having two basic constructs, UP and DOWN- without these two constructs (and having their accepted meaning) no memory access nor IPC is atomic. process-think considers all memory access atomic, but not necessarily IPC. meanwhile, event-think considers memory access AND IPC access atomic. This is as close to transactional memory as exists and is useful. It's also the mindset that I suggest to new programmers, and the one I myself find myself using often. Sometimes it _does_ matter. Sometimes, with the aid of a profiler, someone can find out exactly how to squeeze a little bit more performance out of this application. But: nobody on this list having problems with threads is using threads for that reason. Or rather, if they are, they're not complaining.
Re: [sqlite] Multi-threading.
On Mon, 2005-07-25 at 09:48 -0500, Jay Sprenkle wrote: The theory has been proposed that threads aren't better than separate processes, or application implemented context switching. Does anyone have an experiment that will prove the point either way? It will have to be OS specific though, since I'm sure not all thread implementations are equal. Define better. It's evident that many bugs are more common, and some kinds of bugs only appear in threading applications. It's also debatable which is easier to program. I think the only useful measure might be context-switch performance. But note that if two processes (A and B) are doing rapid context switches between eachother, if their pages are all COW the page tables don't need to be updated. Your overhead would at most be comparison time. This behavior occurs quite frequently on unix systems. The question one postulates at this point is comparing page pointers more expensive than locks/system calls. The answer seems to be usually not although I have indeed run into cases where the answer is yes. On other systems, where we have _three_ processes; two that COW their page tables and a third that has it's own address space, rapid context switches between all three if the third process goes between each of the other two (A-c-B-c-A-etc) - such as it is under Windows NT kernels that move many operations like network I/O onto a separate process - complete with its own address space and etc. On those systems, locks are definitely cheaper. However: Note that even on those systems, scheduling tasks like that are generally better optimized by moving both A and B's tasks into a single event-driven loop (and let c use the other processor). As a side note, many unix applications can be developed for a single event-driven loop and then with a few well placed fork()s be able to take advantage of multiprocessor systems immediately. If you take memory from your stack- you could even use pthread_create(). In this situation, however, I'd like to point out that we're not using a thread-programming-think even if the underlying technology might be called a thread.
Re: [sqlite] Multi-threading.
On Mon, 2005-07-25 at 09:00 +0100, Ben Clewett wrote: I like what you say about computer science being a Science. This is exactly my point. A science is a collection of theories which hold true within a domain until otherwise dissproven. For instance Isac Newtons's law of gravety and Einstain's law. Both are true within a fixed domain. Both are used today. Neither are truelly corrrect. There will be another more complex theroy in time. You have that backwards. They remain theory until proven. Not the other way around. This is the same with Threading. There may be places where this is useful. There may be places where it should not be used. Both are theories within computer science which hold true in a limited domain. So when I said this was my opinion, I should have stated this is my theory within a fixed domain. However when you stated that I was wrong, I don't think this is compatible of your statement that Computer Science is a Science. Nothing in any science is either right or wrong. That's a Religion. You have that backwards. Science most certainly has right and wrong. We also find it useful to examine things that aren't known to be right or wrong, but it is that that makes science different than religion. I don't want to spend ages on the argument, I can see there are passionate views. I only want to explore the idea that threading is a viable strategy for a modern high level language and can produce stable applications with low development time. That's incorrect. Threading increases development time and produces less stable applications. In fairness: it's the skill level of the engineer that does this, but then, they _did_ chose to use a threaded development model. http://www.google.com/search?hl=enlr=q=problems+with +multithreadingbtnG=Search Maybe you're special. Most people aren't special and as soon as you throw threads into the equation they can't find their ass with both hands. For instance a call to a TCP blocking Wait. It's perfectly possible for one thread to cycle round handing GUI events, then checking on the port for new packets. But an alternate method is to partition the code space into two autonomous threads using their own variable set. Two simple sets of code which are not coupled, and remain highly cohesive and encapsulated. That's why we have the select() system call. That's why GTK uses glib to encapsulate file descriptors (including those of sockets) so that event programming can allow the GUI events to be processed immediately and then handle other things without worrying about threads. If you use a language like Java or Javascript (or lisp or perl or etc, etc) you get something called a closure - and they make using event-programming-think a real pleasure, because you don't have to worry about locking, or surprises because race conditions simply won't occur. http://www.kegel.com/c10k.html Should be read by anyone thinking they need threads for performance reasons. Communicating through a thread-safe object. Such code is fast to write, easy to read, robust and does the job. I can complete this in c# in ten minutes, and it will not break. With large amounts of time, better methods may be available. But this theory is viable within its domain. This is speculative, and doesn't provide anything useful to anyone reading this thread. I wasn't about to consider Windows a modern GUI system :) Are you saying it's not modern, or not a GUI? It may not be prefect and it is very complex. But it's not about to go away, and it's installed on modern computers. More importantly, my views on threads remain. If you give the GUI it's own thread, you have implemented a simple solution to ensure that the correct cursor and mouse events are honoured. Then a worker thread is free to do what ever it likes without being coupled to the GUI. Simple robust coding using thread-safe objects. It's a joke, and it's moot anyway. As Richard pointed out, Windows doesn't demand threads from its programmers either. I am also interested in peoples views that threading is slower. Since all processors are likely to be hyperthreaded, multicore, or both. I can see no place where this will be true in the future. Threading is often slower, but if you're worried about the performance drops that _threads_ cause, you wouldn't use C#. Threading is _harder_to_program_. It makes buggier programs. Fewer people can follow and debug it as the program gets larger. And worst of all: people are continually encouraged to use it for exactly the opposite reason. Java uses references, not pointers. Is there any important difference? My point was not about language. It was a question to Dr Hipp about what he feels is missing from the Java pointer (reference) model. Perhaps I should have explained better. I brought this up. Pointers have their own value. References don't. The compiler can
Re: [sqlite] Multi-threading.
Mrs. Brisby wrote: chances are you can't use threads correctly either. This mailing list is an excellent example of how many professional programmers simply can't deal with threads- every problem they run into, it's how do I make sqlite work with threads. If you have to ask that question, you simply have no idea what you're doing. I wasn't going to drag this thread on any longer, but just in case this view is popular... SQLite is a great library, but calling it multi-threaded is stretching the common definition of the term. I consider SQLite multi-thread tolerant :) It detects and allows you to deal with potential threading related issues as opposed to dealing with those issues for the developer ( real thread support in my view ). Given the good doctor's stance on threads, I am grateful for what thread tolerance is in the library :) We get all those threads related questions because SQLite is difficult to use with threads and has sparse sometimes conflicting documentation on the subject. Don't get me wrong, again I am not complaining, it is *way* better than the other options out there. But you can't expect someone to magically figure out an API based on one or two sentence definitions and comments in a header file without asking a few questions and making a few mistakes. Regards, Kervin
Re: [sqlite] Multi-threading.
Dear Mrs Brisby, Thanks for your passionate replies to my original posting. You have much information here. It's obvious I don't know everything about threading. I like what you say about computer science being a Science. This is exactly my point. A science is a collection of theories which hold true within a domain until otherwise dissproven. For instance Isac Newtons's law of gravety and Einstain's law. Both are true within a fixed domain. Both are used today. Neither are truelly corrrect. There will be another more complex theroy in time. This is the same with Threading. There may be places where this is useful. There may be places where it should not be used. Both are theories within computer science which hold true in a limited domain. So when I said this was my opinion, I should have stated this is my theory within a fixed domain. However when you stated that I was wrong, I don't think this is compatible of your statement that Computer Science is a Science. Nothing in any science is either right or wrong. That's a Religion. I don't want to spend ages on the argument, I can see there are passionate views. I only want to explore the idea that threading is a viable strategy for a modern high level language and can produce stable applications with low development time. For instance a call to a TCP blocking Wait. It's perfectly possible for one thread to cycle round handing GUI events, then checking on the port for new packets. But an alternate method is to partition the code space into two autonomous threads using their own variable set. Two simple sets of code which are not coupled, and remain highly cohesive and encapsulated. Communicating through a thread-safe object. Such code is fast to write, easy to read, robust and does the job. I can complete this in c# in ten minutes, and it will not break. With large amounts of time, better methods may be available. But this theory is viable within its domain. I wasn't about to consider Windows a modern GUI system :) Are you saying it's not modern, or not a GUI? It may not be prefect and it is very complex. But it's not about to go away, and it's installed on modern computers. More importantly, my views on threads remain. If you give the GUI it's own thread, you have implemented a simple solution to ensure that the correct cursor and mouse events are honoured. Then a worker thread is free to do what ever it likes without being coupled to the GUI. Simple robust coding using thread-safe objects. I am also interested in peoples views that threading is slower. Since all processors are likely to be hyperthreaded, multicore, or both. I can see no place where this will be true in the future. Java uses references, not pointers. Is there any important difference? My point was not about language. It was a question to Dr Hipp about what he feels is missing from the Java pointer (reference) model. Perhaps I should have explained better. Yes, I understand that 'c' can make use of goto, and that goto is fast. There are also very very bad places to use goto. Exceptions, breaks and continue statements are linear code, easy to follow and more robust to code changes. Goto is a legacy of assembler programming. I don't think it's time to teach new coders about goto. Which is my theory within a fixed domain. Regards, Ben. Mrs. Brisby wrote: On Wed, 2005-07-20 at 17:26 +0100, Ben Clewett wrote: Dr Hipp, I am just playing devils advocate here because I have completed much Java programming in a multi-threaded application. :) I understand the problems of multi-threading. I am reminded that it took nearly 20 years of development to get multi-processor support in a modern OS stable. Much success for this can be attributed to Semaphore Flags. With CPU hardware support to ensure that the semaphore it's self cannot be controlled by more than one process. ITC in 1970 supported multiple threads trivially. Multi-thread applications suffer the same problems. Without semaphore flags or 20 years of development. A novice programmer can happily create a second thread and quickly create terribly applications. However the need for multi-threads is compelling. Especially in a GUI environment. For instance a Mail reader. Where one thread is needed to ensure the GUI is drawn correctly and respond to GUI events. Another to download and dispatch mail. (My Thunderbird has 10 threads. This may be a bit of overkill :) No. Threads are not a need. They allow you to use blocking system calls in parallel without extra page table loads. History has demonstrated that programmers building multithreaded applications tend to produce buggier code, and code that touches more pages than a non-threaded version. As a result, the non-threaded version is easier to write, safer, and runs faster. As another user also mentioned, a Windows system works better with few processes
RE: [sqlite] Multi-threading.
I think the point about multi-threaded apps is not that there might be a few instances where they are useful, but how easy is it to debug and support them and is it worth the price for a little more concurrency? In my experience, most debugging IDE's can't handle multiple threads and actually turn the application into a pseudo threaded version prior to running it. I wouldn't mind betting that's what's going on in Java (it's worth pointing out that Java didn't used to even support native threads and I bet most Java programmers have no idea whether they are actually using Native or pseudo threads) Because of this, you get the frustrating situation where an application works fine in debug but fails sporadically in production. Another good real world example of this is the IIS/ASP server side debugging. IIS allocates a new thread per request which are autonomous but do some semaphoring to share the Session object (I think). There's a handy switch you can set on your IIS virtual directory that turns on server side debugging - great! . only, not so great!! Suddenly IIS starts allocating the same thread to every request i.e. it queues your requests and the very problem you were trying to solve goes away! The moral of the story is as Richard says, if there isn't a burning need for multiple threads then don't use them. By the way, Mrs Brisby is being facetious when he says that he doesn't regard Windows as a modern GUI - there's no such thing! They've all got their origins in concepts put forward over 20 years ago. Steve -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] rg]On Behalf Of Ben Clewett Sent: 25 July 2005 09:01 To: sqlite-users@sqlite.org Subject: Re: [sqlite] Multi-threading. Dear Mrs Brisby, Thanks for your passionate replies to my original posting. You have much information here. It's obvious I don't know everything about threading. I like what you say about computer science being a Science. This is exactly my point. A science is a collection of theories which hold true within a domain until otherwise dissproven. For instance Isac Newtons's law of gravety and Einstain's law. Both are true within a fixed domain. Both are used today. Neither are truelly corrrect. There will be another more complex theroy in time. This is the same with Threading. There may be places where this is useful. There may be places where it should not be used. Both are theories within computer science which hold true in a limited domain. So when I said this was my opinion, I should have stated this is my theory within a fixed domain. However when you stated that I was wrong, I don't think this is compatible of your statement that Computer Science is a Science. Nothing in any science is either right or wrong. That's a Religion. I don't want to spend ages on the argument, I can see there are passionate views. I only want to explore the idea that threading is a viable strategy for a modern high level language and can produce stable applications with low development time. For instance a call to a TCP blocking Wait. It's perfectly possible for one thread to cycle round handing GUI events, then checking on the port for new packets. But an alternate method is to partition the code space into two autonomous threads using their own variable set. Two simple sets of code which are not coupled, and remain highly cohesive and encapsulated. Communicating through a thread-safe object. Such code is fast to write, easy to read, robust and does the job. I can complete this in c# in ten minutes, and it will not break. With large amounts of time, better methods may be available. But this theory is viable within its domain. I wasn't about to consider Windows a modern GUI system :) Are you saying it's not modern, or not a GUI? It may not be prefect and it is very complex. But it's not about to go away, and it's installed on modern computers. More importantly, my views on threads remain. If you give the GUI it's own thread, you have implemented a simple solution to ensure that the correct cursor and mouse events are honoured. Then a worker thread is free to do what ever it likes without being coupled to the GUI. Simple robust coding using thread-safe objects. I am also interested in peoples views that threading is slower. Since all processors are likely to be hyperthreaded, multicore, or both. I can see no place where this will be true in the future. Java uses references, not pointers. Is there any important difference? My point was not about language. It was a question to Dr Hipp about what he feels is missing from the Java pointer (reference) model. Perhaps I should have explained better. Yes, I understand that 'c' can make use of goto, and that goto is fast. There are also very very bad places to use goto. Exceptions, breaks and continue statements are linear code, easy to follow and more robust to code changes. Goto is a legacy of assembler
Re: [sqlite] Multi-threading.
Steve O'Hara wrote: I think the point about multi-threaded apps is not that there might be a few instances where they are useful, but how easy is it to debug and support them and is it worth the price for a little more concurrency? Good point. The ones I have used are terrible. I hope things will improve in time. This is hopefully not a reason to give up. In my experience, most debugging IDE's can't handle multiple threads and actually turn the application into a pseudo threaded version prior to running it. I wouldn't mind betting that's what's going on in Java (it's worth pointing out that Java didn't used to even support native threads and I bet most Java programmers have no idea whether they are actually using Native or pseudo threads) Because of this, you get the frustrating situation where an application works fine in debug but fails sporadically in production. Another good real world example of this is the IIS/ASP server side debugging. IIS allocates a new thread per request which are autonomous but do some semaphoring to share the Session object (I think). There's a handy switch you can set on your IIS virtual directory that turns on server side debugging - great! . only, not so great!! Suddenly IIS starts allocating the same thread to every request i.e. it queues your requests and the very problem you were trying to solve goes away! :) The moral of the story is as Richard says, if there isn't a burning need for multiple threads then don't use them. I have noticed something. There are two lines of thought here. The two ideas may not be too different. (Here is where I am sure I will be flamed, but I am sure I'll learn something from it :) - The single-thread multi-task option. - The multiple-thread single thread-task. The first option involves building a context scheduler into our own programs. So GUI events, TCP listen, and everything else can occur without blocking. A virtual thread environment. As you say, some Java and ISS can sometimes be this. The second option involves letting the OS do the context changes. Letting the programmer just code each thread in isolation. (Programming in a thread-safe way.) So in essence, the two become the same. A switch could even be added to use own context switch, or use the OS. The OS could be virtual or physical. Externally it's not important: any option should yields the same function. Therefore, programming ones own context scheduler is a bit like not using SQL and accessing the data file directly. It may be faster, you may have more control. But you may just be giving your self a lot of work where SQL does it all for you. This is why we all like SQLite so much. I do not know if this is the case today. My assumption is that thread safe programming is easy, by using good tools and good methodology. I truly believe this, and I have not experienced some of the thread problems other people have reported. Considering the future of processor design, should we be getting more excited about threads and how we can make them work for us? By the way, Mrs Brisby is being facetious when he says that he doesn't regard Windows as a modern GUI - there's no such thing! They've all got their origins in concepts put forward over 20 years ago. :) Ben. Steve -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] rg]On Behalf Of Ben Clewett Sent: 25 July 2005 09:01 To: sqlite-users@sqlite.org Subject: Re: [sqlite] Multi-threading. Dear Mrs Brisby, Thanks for your passionate replies to my original posting. You have much information here. It's obvious I don't know everything about threading. I like what you say about computer science being a Science. This is exactly my point. A science is a collection of theories which hold true within a domain until otherwise dissproven. For instance Isac Newtons's law of gravety and Einstain's law. Both are true within a fixed domain. Both are used today. Neither are truelly corrrect. There will be another more complex theroy in time. This is the same with Threading. There may be places where this is useful. There may be places where it should not be used. Both are theories within computer science which hold true in a limited domain. So when I said this was my opinion, I should have stated this is my theory within a fixed domain. However when you stated that I was wrong, I don't think this is compatible of your statement that Computer Science is a Science. Nothing in any science is either right or wrong. That's a Religion. I don't want to spend ages on the argument, I can see there are passionate views. I only want to explore the idea that threading is a viable strategy for a modern high level language and can produce stable applications with low development time. For instance a call to a TCP blocking Wait. It's perfectly possible for one thread to cycle round handing GUI events, then checking on the port for new
Re: [sqlite] Multi-threading.
Computer _Science_ is just that: Science. It's not opinion. What they taught me at university was the current collection of best practices for solving problems. Here's a common problem, here are the known good algorithms for solving it. Here are the techniques we use for creating 'good' programs. You are now educated. They didn't teach us the scientific method for solving problems. The theory has been proposed that threads aren't better than separate processes, or application implemented context switching. Does anyone have an experiment that will prove the point either way? It will have to be OS specific though, since I'm sure not all thread implementations are equal.
Re: [sqlite] Multi-threading.
On Wed, 2005-07-20 at 17:26 +0100, Ben Clewett wrote: Dr Hipp, I am just playing devils advocate here because I have completed much Java programming in a multi-threaded application. :) I understand the problems of multi-threading. I am reminded that it took nearly 20 years of development to get multi-processor support in a modern OS stable. Much success for this can be attributed to Semaphore Flags. With CPU hardware support to ensure that the semaphore it's self cannot be controlled by more than one process. ITC in 1970 supported multiple threads trivially. Multi-thread applications suffer the same problems. Without semaphore flags or 20 years of development. A novice programmer can happily create a second thread and quickly create terribly applications. However the need for multi-threads is compelling. Especially in a GUI environment. For instance a Mail reader. Where one thread is needed to ensure the GUI is drawn correctly and respond to GUI events. Another to download and dispatch mail. (My Thunderbird has 10 threads. This may be a bit of overkill :) No. Threads are not a need. They allow you to use blocking system calls in parallel without extra page table loads. History has demonstrated that programmers building multithreaded applications tend to produce buggier code, and code that touches more pages than a non-threaded version. As a result, the non-threaded version is easier to write, safer, and runs faster. As another user also mentioned, a Windows system works better with few processes with many threads. Windows uses threads because x86 page tables are expensive to load. It doesn't help: the system-call method Windows uses eats any benefit that it has, again producing net-zero. I believe the issue is not whether to use threads, but to use them correctly. Which is not a hard thing to do with a little support. This is where Java (and .NET) work well. If you use them correctly. They provide thread-safe objects. Which have been designed to use semaphore flags internally. If the programmer uses these thread-safe objects correctly, they will not encounter thread issues. For instance, all communication between threads should be exclusively through these thread-safe objects. Java uses threads poorly. They're expensive to set up, and many Java programmers yield to non-blocking methods as Java closures tend to be easier to program, and faster too. Further, Java and .NET provide Sycronisation methods. The defining of a method to be synchronised automatically creates the locks to ensure thread safe access. ... I am also interested in your comments on Pointers and GoTo. I note that Java is 100% pointers. Apart from basic types, all object access is by pointer. Java uses references, not pointers. Using Exceptions correctly, I have never felt the need for a GoTo. Exceptions do the same as GoTo, accept, maybe, in a slightly more developed and useful way. Exceptions are slower than goto. They are also less straightforward when deeply nested (long chains of throws XYZ come to mind...) These are just my opinions :) They are wrong.
Re: [sqlite] Multi-threading.
- Original Message - From: Ben Clewett [EMAIL PROTECTED] To: sqlite-users@sqlite.org Sent: Wednesday, July 20, 2005 12:26 PM Subject: Re: [sqlite] Multi-threading. Dr Hipp, I am just playing devils advocate here because I have completed much Java programming in a multi-threaded application. :) I understand the problems of multi-threading. I am reminded that it took nearly 20 years of development to get multi-processor support in a modern OS stable. Much success for this can be attributed to Semaphore Flags. With CPU hardware support to ensure that the semaphore it's self cannot be controlled by more than one process. whatcha talkin bout willis? all you need to fence access to a shared resource is a set of atomic operations which carry the semantics you are looking for. this has been around for quite some time. Multi-thread applications suffer the same problems. Without semaphore flags or 20 years of development. say what? A novice programmer can happily create a second thread and quickly create terribly applications. yes. non-novices too, hence richard's advice. As another user also mentioned, a Windows system works better with few processes with many threads. this has to do with process creation being a very expensive operation in windows. just because a certain platform cripples something doesn't make it bad. I believe the issue is not whether to use threads, but to use them correctly. Which is not a hard thing to do with a little support. you've got no idea. there are *very* few threading interfaces which make writing correct code reasonably easy. java and .net are certainly not among them. erlang is, but the trouble is they're implemented as microthreads, so you're still stuck with problems making blocking syscalls. This is where Java (and .NET) work well. If you use them correctly. They provide thread-safe objects. Which have been designed to use semaphore flags internally. If the programmer uses these thread-safe objects correctly, they will not encounter thread issues. For instance, all communication between threads should be exclusively through these thread-safe objects. stop misusing 'semaphore' already, semaphore is not a synonym for a synchronization object - it is a kind of a synchronization object and has a very narrow definition. Further, Java and .NET provide Sycronisation methods. The defining of a method to be synchronised automatically creates the locks to ensure thread safe access. you don't get the point. this (as opposed to more intuitive primitives *around* threading, such as actors) is just a couple lines of code here and there in a language which does not have syntax for it. coders have to understand the issues and use the locking primitives they are given correctly. i'd say your opinions, as expressed in your message, demonstrate that the mere presence of higher level sycnhronization primitives in a development platform does not magically make users of said platform understand threading issues. I am also interested in your comments on Pointers and GoTo. I note that Java is 100% pointers. Apart from basic types, all object access is by pointer. this is why teaching java in cs 101 is a bad idea, people have no understanding of how things work 'under the hood' anymore. java uses references, which are implemented as primitives in the jvm itself. Using Exceptions correctly, I have never felt the need for a GoTo. Exceptions do the same as GoTo, accept, maybe, in a slightly more developed and useful way. then you haven't coded anything complex enough to require them. i can tell you that they are an absolute necessity when dealing with a lot of nesting, which may or may not deal with error handling. dijkstra was preaching against using goto to create spaghetti code; many experienced coders use gotos to *improve* readability and hence maintainability of their code. 'using'-like constructs give you *some* of this, but you will still want gotos even in languages where both exceptions and using are available when dealing with non-trivial logic. These are just my opinions :) no offense, but you probably want to have the correct data from which to derive your opinions ;) -p
Re: [sqlite] Multi-threading.
On 7/22/05, Paul G [EMAIL PROTECTED] wrote: Using Exceptions correctly, I have never felt the need for a GoTo. Exceptions do the same as GoTo, accept, maybe, in a slightly more developed and useful way. then you haven't coded anything complex enough to require them. i can tell you that they are an absolute necessity when dealing with a lot of nesting, I've found the state machine to be a wonderful concept for eliminating the need for gotos.
RE: [sqlite] Multi-threading.
-Original Message- From: Mrs. Brisby [mailto:[EMAIL PROTECTED] Sent: Friday, July 22, 2005 3:07 PM To: sqlite-users@sqlite.org Subject: Re: [sqlite] Multi-threading. However the need for multi-threads is compelling. Especially in a GUI environment. For instance a Mail reader. Where one thread is needed to ensure the GUI is drawn correctly and respond to GUI events. Another to download and dispatch mail. (My Thunderbird has 10 threads. This may be a bit of overkill :) No. Threads are not a need. They allow you to use blocking system calls in parallel without extra page table loads. History has demonstrated that programmers building multithreaded applications tend to produce buggier code, and code that touches more pages than a non-threaded version. As a result, the non-threaded version is easier to write, safer, and runs faster. So, what's your point? That writing things the easy way leads to safer, less buggy, faster code? That's hardly a point. The original poster presented one of the more compelling reasons for multi-threading in modern apps, the GUI. It is hard, if not impossible, with modern GUI systems to write any relatively complex app that is both performant and graphically responsive without using threads. At least for the short term, Moore's Law is slowing down, we might as well start calling it Moore's Dream. All main CPUs are going multicore, even game consoles, and one of the only realistic ways to take advantage of that is through multi-threading. Saying it is hard doesn't change reality. As another user also mentioned, a Windows system works better with few processes with many threads. Windows uses threads because x86 page tables are expensive to load. It doesn't help: the system-call method Windows uses eats any benefit that it has, again producing net-zero. This being THE reason Windows emphasizes threads over processes is hard to swallow. I am also interested in your comments on Pointers and GoTo. I note that Java is 100% pointers. Apart from basic types, all object access is by pointer. Java uses references, not pointers. This is purely semantic nit picking. Using Exceptions correctly, I have never felt the need for a GoTo. Exceptions do the same as GoTo, accept, maybe, in a slightly more developed and useful way. Exceptions are slower than goto. They are also less straightforward when deeply nested (long chains of throws XYZ come to mind...) I would agree that exceptions are not a good replacement for gotos. However, having been a professional C++ programmer for over 10 years, I have never needed a goto. This probably stems more from the fact that with C++/Java/C# you don't really need gotos, but with C/Basic/etc there are arguably things that you can't do, or would be quite hard to do, without gotos. These are just my opinions :) They are wrong. I hope there was a hint of sarcasm in that last comment. The original poster obviously didn't hit everything on the nail, but there is a whole world of gray between right and wrong. Tim
RE: [sqlite] Multi-threading.
On Fri, 2005-07-22 at 14:00 -0700, Tim McDaniel wrote: It is hard, if not impossible, with modern GUI systems to write any relatively complex app that is both performant and graphically responsive without using threads. Not true. The event model works just fine for GUI programming. The bulk of my consulting practice (for the past 13 years) has been writing high-performance GUIs running in front of numerically intensive scientific and engineering applications. Most of these run on windows - at least over the past 5 years. None of them have ever used more than a single thread. -- D. Richard Hipp [EMAIL PROTECTED]
RE: [sqlite] Multi-threading.
On Fri, 2005-07-22 at 14:00 -0700, Tim McDaniel wrote: History has demonstrated that programmers building multithreaded applications tend to produce buggier code, and code that touches more pages than a non-threaded version. As a result, the non-threaded version is easier to write, safer, and runs faster. So, what's your point? That writing things the easy way leads to safer, less buggy, faster code? That's hardly a point. The original poster presented one of the more compelling reasons for multi-threading in modern apps, the GUI. It is hard, if not impossible, with modern GUI systems to write any relatively complex app that is both performant and graphically responsive without using threads. I have no problems writing GTK applications without threads. I suspect the toolkits you are familiar with are inadequate. Most GUI toolkits rely heavily on callback interfaces. Closures and signal-based event notification are much more beneficial than threads: they produce fewer bugs, and are easier to program still. But this isn't an argument for closures and signal-based event notification. This is a general argument against threads. At least for the short term, Moore's Law is slowing down, we might as well start calling it Moore's Dream. All main CPUs are going multicore, even game consoles, and one of the only realistic ways to take advantage of that is through multi-threading. Saying it is hard doesn't change reality. The other realistic way is multiple processes, which is the point I argue for. Threads can be useful, but they're only faster than processes if they touch less pages. Threads require more memory (synchronization primitives, etc) and many languages that encourage threading provide no mechanism to isolate those privileges onto a few pages. Those page hits tend to cause MORE page-table access than similarly-written programs than use processes. I use threads sparingly, and try to keep all memory access on the stack when I do use threads so that I am NOT touching any heap-memory (rather as little as possible), and therefore, require few synchronization primitives. I tend to not see very many programmers doing this. I suspect this is because programmers believe threads are faster and cheaper, so everyone should use them. In general, threads don't solve problems because the problems that they CAN solve, people rarely face, and the problems that they are ATTEMPTING to solve can be solved in better, cleaner ways. As another user also mentioned, a Windows system works better with few processes with many threads. Windows uses threads because x86 page tables are expensive to load. It doesn't help: the system-call method Windows uses eats any benefit that it has, again producing net-zero. This being THE reason Windows emphasizes threads over processes is hard to swallow. What? That threads are cheaper than processes? If Windows has another reason for using threads instead of processes I'd love to hear it. With all of the downsides to threads, the only reasonable one is that they _may_ be cheaper than processes. Unfortunately, that textbook _may_ is a reality _isn't_ in most cases. Nevertheless, my point to this statement was lost: Windows could optimize process creation, but it could also optimize system calls. Those steps would make the parent-threads' argument moot (not to mix metaphors). I am also interested in your comments on Pointers and GoTo. I note that Java is 100% pointers. Apart from basic types, all object access is by pointer. Java uses references, not pointers. This is purely semantic nit picking. That's your opinion. Other languages have both. When they do, the distinction is more important. Using Exceptions correctly, I have never felt the need for a GoTo. Exceptions do the same as GoTo, accept, maybe, in a slightly more developed and useful way. Exceptions are slower than goto. They are also less straightforward when deeply nested (long chains of throws XYZ come to mind...) I would agree that exceptions are not a good replacement for gotos. However, having been a professional C++ programmer for over 10 years, I have never needed a goto. This probably stems more from the fact that with C++/Java/C# you don't really need gotos, but with C/Basic/etc there are arguably things that you can't do, or would be quite hard to do, without gotos. I'm glad I gave you an opportunity to post your VC. Meanwhile, while C doesn't _need_ gotos: for (i = 0; i n; i++) for (j = 0;j m; j++) if (q(i,j)) goto TAIL; TAIL: looks better than the alternative. Just because you don't _need_ a condom doesn't mean it's not a good idea. I think that avoiding goto is good for the same reason that avoiding threads are good. Once you've gotten used to knowing why they're bad is when you can begin to safely start using them for the right thing. Maybe in another 10 years you'll be ready to
RE: [sqlite] Multi-threading.
On Fri, 2005-07-22 at 17:23 -0400, D. Richard Hipp wrote: On Fri, 2005-07-22 at 14:00 -0700, Tim McDaniel wrote: It is hard, if not impossible, with modern GUI systems to write any relatively complex app that is both performant and graphically responsive without using threads. Not true. The event model works just fine for GUI programming. The bulk of my consulting practice (for the past 13 years) has been writing high-performance GUIs running in front of numerically intensive scientific and engineering applications. Most of these run on windows - at least over the past 5 years. None of them have ever used more than a single thread. I wasn't about to consider Windows a modern GUI system :)
Re: [sqlite] Multi-threading.
Dr Hipp, I am just playing devils advocate here because I have completed much Java programming in a multi-threaded application. :) I understand the problems of multi-threading. I am reminded that it took nearly 20 years of development to get multi-processor support in a modern OS stable. Much success for this can be attributed to Semaphore Flags. With CPU hardware support to ensure that the semaphore it's self cannot be controlled by more than one process. Multi-thread applications suffer the same problems. Without semaphore flags or 20 years of development. A novice programmer can happily create a second thread and quickly create terribly applications. However the need for multi-threads is compelling. Especially in a GUI environment. For instance a Mail reader. Where one thread is needed to ensure the GUI is drawn correctly and respond to GUI events. Another to download and dispatch mail. (My Thunderbird has 10 threads. This may be a bit of overkill :) As another user also mentioned, a Windows system works better with few processes with many threads. I believe the issue is not whether to use threads, but to use them correctly. Which is not a hard thing to do with a little support. This is where Java (and .NET) work well. If you use them correctly. They provide thread-safe objects. Which have been designed to use semaphore flags internally. If the programmer uses these thread-safe objects correctly, they will not encounter thread issues. For instance, all communication between threads should be exclusively through these thread-safe objects. Further, Java and .NET provide Sycronisation methods. The defining of a method to be synchronised automatically creates the locks to ensure thread safe access. I am also interested in your comments on Pointers and GoTo. I note that Java is 100% pointers. Apart from basic types, all object access is by pointer. Using Exceptions correctly, I have never felt the need for a GoTo. Exceptions do the same as GoTo, accept, maybe, in a slightly more developed and useful way. These are just my opinions :) Regards, Ben Clewett. D. Richard Hipp wrote: On Fri, 2005-07-15 at 16:41 +0530, Roushan Ali wrote: Hello all, Can we use single sqlite_open handle(global) across threads( if all database operations are serialized by using semaphore) ? Please help. Opening a database connection in one thread and using it in another will work on some operating systems but not on others. You are advised not to do it. See http://www.sqlite.org/cvstrac/tktview?tn=1272 and http://www.sqlite.org/cvstrac/chngview?cn=2521. Actually, this seems like a good opportunity to repeat my oft-ignored advice to not use more than one thread in a single address space. If you need multiple threads, create multiple processes. This has nothing to do with SQLite = it is just good programming advice. I have worked on countless multi- threaded programs over the years, and I have yet to see a single one that didn't contain subtle, hard to reproduce, and very hard to troubleshoot bugs related to threading issues. I am constantly amazed at the prevailing idea (exemplified by Java) that software should be strongly typed and should not use goto statement or pointers - all in the name of reducing bugs - but that it is OK to use multiple threads within the same address space. Strong typing helps prevent only bugs that are trivially easy to locate and fix. The use of goto statements and pointers likewise results in deterministic problems that are easy to test for and relatively easy to track down and correct. But threading bugs tend to manifest themselves as timing-dependent glitches and lock-ups that are hardware and platform dependent, that never happen the same way twice, and that only appear for customers after deployment and never in a testing environment.
Re: [sqlite] Multi-threading.
Paul G wrote: richard's advice is solid. use async io/event loops if possible, separate processes if possible, threads as a last resort, in that order. the grey area is the 'if possible' test, since it's a multi-way tradeoff between performance, simplicity and provable (to an extent) correctness. i fully expect that a lot of folks *do* need to use threads and the probability of that being the case on windows is much higher than on posixish platforms. I agree with you, but it doesn't seem like you're exactly concurring with what DRH said though. I'm guessing that that 'if possible' test is huge, and very frequently will fail. Why suffer the context switch when you don't have to? Would you write a non-trivial GUI program today without using threads? Why subject the users to the added latency IPC is going to introduce? The funny thing is eventually multi-process apps go to MMap for IPC, for performance, and then run into a lot of the same issues they'd have to deal with if they were threaded. And as far as the 'threads introduce difficult to debug' errors; Isn't that the age-old performance versus complexity trade- off? Processes are easier to use, but very often perform worse under the same conditions as the more complex threaded application. That is a fact many of us can not look past easily. PS. It's funny, this discussion seems like its been taked right from a early '90s newsgroup :) Regards, Kervin
[sqlite] Multi-threading.
Hello all, Can we use single sqlite_open handle(global) across threads( if all database operations are serialized by using semaphore) ? Please help. Regards, Roushan
Re: [sqlite] Multi-threading.
On Fri, 2005-07-15 at 16:41 +0530, Roushan Ali wrote: Hello all, Can we use single sqlite_open handle(global) across threads( if all database operations are serialized by using semaphore) ? Please help. Opening a database connection in one thread and using it in another will work on some operating systems but not on others. You are advised not to do it. See http://www.sqlite.org/cvstrac/tktview?tn=1272 and http://www.sqlite.org/cvstrac/chngview?cn=2521. Actually, this seems like a good opportunity to repeat my oft-ignored advice to not use more than one thread in a single address space. If you need multiple threads, create multiple processes. This has nothing to do with SQLite = it is just good programming advice. I have worked on countless multi- threaded programs over the years, and I have yet to see a single one that didn't contain subtle, hard to reproduce, and very hard to troubleshoot bugs related to threading issues. I am constantly amazed at the prevailing idea (exemplified by Java) that software should be strongly typed and should not use goto statement or pointers - all in the name of reducing bugs - but that it is OK to use multiple threads within the same address space. Strong typing helps prevent only bugs that are trivially easy to locate and fix. The use of goto statements and pointers likewise results in deterministic problems that are easy to test for and relatively easy to track down and correct. But threading bugs tend to manifest themselves as timing-dependent glitches and lock-ups that are hardware and platform dependent, that never happen the same way twice, and that only appear for customers after deployment and never in a testing environment. -- D. Richard Hipp [EMAIL PROTECTED]
RE: [sqlite] Multi-threading.
which gives me the opportunity to repear my oft-ignored reply :) what you say is true, provided the OS is geared towards multiple processes. which is not true for windows, but is true for *ix, as400 and other environments. if you need, say, a server that handles 500 sessions, and attempt to do this with spawning processes on windows, you are probably dead, memory and cpu-wise. on Linux, this is nothing, it can handle it easily. otoh, 500 threads for windows is business as usual, but threading on Linux, is , I hear, iffy at best. so, although this is good advice, it is not unconditionally applicable. -Original Message- From: D. Richard Hipp [mailto:[EMAIL PROTECTED] Sent: Friday, July 15, 2005 3:27 PM To: sqlite-users@sqlite.org Subject: Re: [sqlite] Multi-threading. On Fri, 2005-07-15 at 16:41 +0530, Roushan Ali wrote: Hello all, Can we use single sqlite_open handle(global) across threads( if all database operations are serialized by using semaphore) ? Please help. Opening a database connection in one thread and using it in another will work on some operating systems but not on others. You are advised not to do it. See http://www.sqlite.org/cvstrac/tktview?tn=1272 and http://www.sqlite.org/cvstrac/chngview?cn=2521. Actually, this seems like a good opportunity to repeat my oft-ignored advice to not use more than one thread in a single address space. If you need multiple threads, create multiple processes. This has nothing to do with SQLite = it is just good programming advice. I have worked on countless multi- threaded programs over the years, and I have yet to see a single one that didn't contain subtle, hard to reproduce, and very hard to troubleshoot bugs related to threading issues. I am constantly amazed at the prevailing idea (exemplified by Java) that software should be strongly typed and should not use goto statement or pointers - all in the name of reducing bugs - but that it is OK to use multiple threads within the same address space. Strong typing helps prevent only bugs that are trivially easy to locate and fix. The use of goto statements and pointers likewise results in deterministic problems that are easy to test for and relatively easy to track down and correct. But threading bugs tend to manifest themselves as timing-dependent glitches and lock-ups that are hardware and platform dependent, that never happen the same way twice, and that only appear for customers after deployment and never in a testing environment. -- D. Richard Hipp [EMAIL PROTECTED]
Re: [sqlite] Multi-threading.
Thanks Richard for your reply. Actually, we have written a windows application which uses four threads. Each thread may have to add/delete thousands of entries in the database( for performance reason , we don't want to open/close the database for each insertion/deletion) .If we use different sqlite_open handle for each thread , then one thread has to do busy looping until other threads complete their operation, which is not desirable according to the application requirement. That's why we opened global database handle for the lifetime of the application and each thread used the handle serially and it worked. Thanking you again, Roushan On Fri, 2005-07-15 at 17:57, D. Richard Hipp wrote: On Fri, 2005-07-15 at 16:41 +0530, Roushan Ali wrote: Hello all, Can we use single sqlite_open handle(global) across threads( if all database operations are serialized by using semaphore) ? Please help. Opening a database connection in one thread and using it in another will work on some operating systems but not on others. You are advised not to do it. See http://www.sqlite.org/cvstrac/tktview?tn=1272 and http://www.sqlite.org/cvstrac/chngview?cn=2521. Actually, this seems like a good opportunity to repeat my oft-ignored advice to not use more than one thread in a single address space. If you need multiple threads, create multiple processes. This has nothing to do with SQLite = it is just good programming advice. I have worked on countless multi- threaded programs over the years, and I have yet to see a single one that didn't contain subtle, hard to reproduce, and very hard to troubleshoot bugs related to threading issues. I am constantly amazed at the prevailing idea (exemplified by Java) that software should be strongly typed and should not use goto statement or pointers - all in the name of reducing bugs - but that it is OK to use multiple threads within the same address space. Strong typing helps prevent only bugs that are trivially easy to locate and fix. The use of goto statements and pointers likewise results in deterministic problems that are easy to test for and relatively easy to track down and correct. But threading bugs tend to manifest themselves as timing-dependent glitches and lock-ups that are hardware and platform dependent, that never happen the same way twice, and that only appear for customers after deployment and never in a testing environment.
Re: [sqlite] Multi-threading.
Roushan Ali wrote: Thanks Richard for your reply. Actually, we have written a windows application which uses four threads. Each thread may have to add/delete thousands of entries in the database( for performance reason , we don't want to open/close the database for each insertion/deletion) .If we use different sqlite_open handle for each thread , then one thread has to do busy looping until other threads complete their operation, which is not desirable according to the application requirement. That's why we opened global database handle for the lifetime of the application and each thread used the handle serially and it worked. We have a multi-threaded windows application with four threads. Three threads need access to the database (all three are producers and consumers), but one thread is the GUI thread and wants to access the database while handling WM_TIMER messages (re-entrency issues). So we allocate 4 database connections during initialization. Each section of our code uses its own connection. We have a special stress test mode that we can enable. The program remains stable after hours of operation under the stress test. The program will slow down because of the database locking mechanism (especially during large transactions), but it has never crashed due to multiple threads accessing the database used _different_ connection objects. If you are going to be multi-threaded, then why not just use multiple connection objects (structs - ours are wrapped in a C++ class)?
Re: [sqlite] Multi-threading.
On Fri, Jul 15, 2005 at 08:27:14AM -0400, D. Richard Hipp wrote: [...] I am constantly amazed at the prevailing idea (exemplified by Java) that software should be strongly typed and should not use goto statement or pointers - all in the name of reducing bugs - but that it is OK to use multiple threads within the same address space. Strong typing helps prevent only bugs that are trivially easy to locate and fix. The use of goto statements and pointers likewise results in deterministic problems that are easy to test for and relatively easy to track down and correct. But threading bugs tend to manifest themselves as timing-dependent glitches and lock-ups that are hardware and platform dependent, that never happen the same way twice, and that only appear for customers after deployment and never in a testing environment. My quote of the week :-) -- Gerhard -- Gerhard Häring - [EMAIL PROTECTED] - Python, web database development signature.asc Description: Digital signature
Re: [sqlite] Multi-threading.
Hi, Thanks for your response. I don't have any idea how multiple connection objects work. Can you please tell us something about that. Thanks, Roushan On Fri, 2005-07-15 at 20:15, Dennis Jenkins wrote: Roushan Ali wrote: Thanks Richard for your reply. Actually, we have written a windows application which uses four threads. Each thread may have to add/delete thousands of entries in the database( for performance reason , we don't want to open/close the database for each insertion/deletion) .If we use different sqlite_open handle for each thread , then one thread has to do busy looping until other threads complete their operation, which is not desirable according to the application requirement. That's why we opened global database handle for the lifetime of the application and each thread used the handle serially and it worked. We have a multi-threaded windows application with four threads. Three threads need access to the database (all three are producers and consumers), but one thread is the GUI thread and wants to access the database while handling WM_TIMER messages (re-entrency issues). So we allocate 4 database connections during initialization. Each section of our code uses its own connection. We have a special stress test mode that we can enable. The program remains stable after hours of operation under the stress test. The program will slow down because of the database locking mechanism (especially during large transactions), but it has never crashed due to multiple threads accessing the database used _different_ connection objects. If you are going to be multi-threaded, then why not just use multiple connection objects (structs - ours are wrapped in a C++ class)?
Re: [sqlite] Multi-threading.
Roushan Ali wrote: Hi, Thanks for your response. I don't have any idea how multiple connection objects work. Can you please tell us something about that. I wrappered the C interface to SQLite3 via a C++ Class called CSqlite3. The constructor does NOT open the database, it just allocates the sqlite struct. I declared 4 global instances of this class. The constructors get called before my WinMain(). In my initialization code (called before any threads are created), I open the database 4 times. I do an integrity check (and some other logic) after the first open. Like this: g_DbMain.Open(szFilename); CheckDatabase(g_DbMain); // pragma integrity_check, create missing tables / schema updates, vacuum g_DbTimer.Open(szFilename); g_DbThread2.Open(szFilename); g_DbThread3.Open(szFilename); I then create the worker threads. One of my threads does NOT use any database, so we'll ignore it. Another thread (main / gui) already exists, so I am really only creating threads #2 and #3. The thread function uses the database object as needed. After the worker threads terminate, the main thread closes all four database objects. The object's destructor is called when the application exits. I do not create new connections to the database while the executing. Please note that my solution is NOT appropriate if I wanted to create arbitrary threads at arbitrary times. If I were doing that, then each thread would create it's own database object on it's own TLS (thread local storage) or stack. I created all of my database Open() code into the main thread just to keep it all together. Each of my threads does a very specific function that is totally unique to that thread: 1. The main thread uses it's database connection to respond to user initiated GUI events. 2. The main thread also uses the timer database connection to handle WM_TIMER messages to update a status display synchronously (kinda). Because this function can be invoked while the thread has a transaction on the main connection, I need to use a different connection. One thread, but it must be fully re-entrant. 3. Thread #2 is a producer. It gathers data and inserts it into the database. 4. Thread #3 is a consumer. It takes data from the database and does stuff with them. It updates those rows. The timer connection only executes select to update the GUI. The main connection is used to query the database, update the database and to delete from the database. The application is what it is. I make no public claims about it being the best designed thing ever, but it does work well under stress.
Re: [sqlite] Multi-threading.
D. Richard Hipp wrote: Actually, this seems like a good opportunity to repeat my oft-ignored advice to not use more than one thread in a single address space. If you need multiple threads, create multiple processes. I think its not really an acceptable option for those who are on Windows :( best regards, Alex
Re: [sqlite] Multi-threading.
- Original Message - From: Andrew Piskorski [EMAIL PROTECTED] To: sqlite-users@sqlite.org Sent: Friday, July 15, 2005 1:05 PM Subject: Re: [sqlite] Multi-threading. On Fri, Jul 15, 2005 at 04:21:05PM +0300, Cariotoglou Mike wrote: memory and cpu-wise. on Linux, this is nothing, it can handle it easily. otoh, 500 threads for windows is business as usual, but threading on Linux, is , I hear, iffy at best. Linux runs multi-threaded apps (e.g., AOLserver) quite well, and has for many years - since at least 2000 or so, probably earlier. My understanding is that the old LinuxThreads implementation had some pretty ugly bits, but it worked. NPTL is much better, and is standard with the Linux 2.6.x kernels. the issue wasn't necessarily the thread implementation per se, but the fact that threads were treated as processes for scheduling purposes and hence scheduled with the regular process scheduler, which was not efficient for a large number of processes. these problems went away when ingo molnar's O(1) scheduler was adopted (not sure when it was merged into linus' 2.4, but distros adopted it quite fast). -p
Re: [sqlite] Multi-threading.
Andrew Piskorski wrote: On Fri, Jul 15, 2005 at 04:21:05PM +0300, Cariotoglou Mike wrote: memory and cpu-wise. on Linux, this is nothing, it can handle it easily. otoh, 500 threads for windows is business as usual, but threading on Linux, is , I hear, iffy at best. Linux runs multi-threaded apps (e.g., AOLserver) quite well, and has for many years - since at least 2000 or so, probably earlier. My understanding is that the old LinuxThreads implementation had some pretty ugly bits, but it worked. NPTL is much better, and is standard with the Linux 2.6.x kernels. Some architectures permit, or even encourage, multi-threaded design. It can be done obviously. However, Dr. Hipp still has a point. One thread can trash another's address space. They share code, global data, the heap (generally) and system object handles (files, sockets, IPC devices ( and weird crap like Desktop and Mutants on windows). The only non-shared things are the stack, TLS (thread local storage) and per-thread CPU context. Even then all of those things can be trashed by other threads in the same process. Unless you can _prove_ that your code won't do this (and all code that you call, including system DLLs / SOs) then you are taking a risk. Personally, I prefer multi-threaded code. I like to write it and I like to debug it. I ship it to customers. Your millage may vary. And yes, Linux threads used to be very unstable. I've only used Linux threads once, and it was on a recent 2.6 kernel, so I never experienced the problem(s).
Re: [sqlite] Multi-threading.
D. Richard Hipp wrote: On Fri, 2005-07-15 at 16:41 +0530, Roushan Ali wrote: Hello all, Can we use single sqlite_open handle(global) across threads( if all database operations are serialized by using semaphore) ? Please help. Opening a database connection in one thread and using it in another will work on some operating systems but not on others. You are advised not to do it. See http://www.sqlite.org/cvstrac/tktview?tn=1272 and http://www.sqlite.org/cvstrac/chngview?cn=2521. Good sound advice, to a point. Multiple threads accessing the same connection *can* be done, its a design time issue that needs to be addressed before even a single line of code is written. I do it with my mail server using SQLite for logging purposes, I use mutex semaphores to handle the nitty gritty details. Its the usual issue of knowing what you are stepping into before you step, because some of what you step into stinks. -- Craig Morrison =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= http://www.mtsprofessional.com/ A Win32 email server that works for You.
Re: [sqlite] Multi-threading.
On Fri, Jul 15, 2005 at 01:04:50PM -0400, Paul G wrote: the issue wasn't necessarily the thread implementation per se, but the fact that threads were treated as processes for scheduling purposes and hence scheduled with the regular process scheduler, which was not efficient for a large number of processes. these problems went away when ingo molnar's O(1) scheduler was adopted (not sure when it was merged into linus' 2.4, but Interesting. Then that may explain why I never heard any first hand reports of notable Linux threading problems. The people I tended to talk to were probably all running with well under 100 threads per process, and only 2 or 3 such processes at most. Presumably even the earlier lousy process scheduler could handle that ok. -- Andrew Piskorski [EMAIL PROTECTED] http://www.piskorski.com/
Re: [sqlite] multi threading
My advice to *all* programmers is to never use more than one thread in the same address space. I have never in 20 years worked on a multiple threaded program that actually got all of the threading issues right. There are always subtle bugs that cause error that are very difficult to reproduce and fix. Multithreading is the fastest road to buggy code that I know of. Avoid it. If you absolutely, positively must have multiple threads of control, put each thread in its own address space (make it a process.) I second the motion!
Re: [sqlite] multi threading
Jay Sprenkle wrote: My advice to *all* programmers is to never use more than one thread in the same address space. I have never in 20 years worked on a multiple threaded program that actually got all of the threading issues right. There are always subtle bugs that cause error that are very difficult to reproduce and fix. Multithreading is the fastest road to buggy code that I know of. Avoid it. If you absolutely, positively must have multiple threads of control, put each thread in its own address space (make it a process.) I second the motion! I agree that threads have to be treated with caution. Locking must be used everywhere. Expect some difficult bugs. Protect your buffers etc... But I think there is a valid place... For instance in event-driver programming. Programming a single thread to handle events without sticking on one event, can arguably be as hard and buggy as giving each event it's own thread. See Win95 :) This can be helped by using objects which have been designed to be thread safe. For instance, Java's excellent thread-safe hash-table. Threads also find a use where TCP/IP blocking may be encountered. Also where a UI is used at the same time as some back-processing. Eg, downloading email on a slow link, whilst writing and sending another email. With a single thread, this would be extremely hard to program well. I also note the performance gains which can be obtained from modern hyperthreaded CPU's. But these are my opinions, and I have been programming a lot less than 20 years, as well as being new to this list, so ignore please my ramblings! Ben
Re: [sqlite] multi threading
I also note the performance gains which can be obtained from modern hyperthreaded CPU's. Does Linux/Windows make each process a thread on these beasties? If so, wouldn't making each task a process end up being the same thing?
Re: [sqlite] multi threading
Jay Sprenkle wrote: I also note the performance gains which can be obtained from modern hyperthreaded CPU's. Does Linux/Windows make each process a thread on these beasties? If so, wouldn't making each task a process end up being the same thing? Each process is a collection of one or more threads. Since each thread has access to the same memory and the same code, a hyperthreaded CPU can run more than one thread literally at the same time. As long as they are in the same process. So if you split your processing between a collection of threads, your application will run much faster. Since this is at CPU level, this works on Linux, Windows and any other OS. (Accept SCO, it has no thread ability.) This of course added to the many forms of bug available to badly programmed threaded applications... Ben
Re: [sqlite] multi threading
Thank you for your answer, Finally I can use sqlite very well on a multi-threading environment. I can run six threads with an maximum of 25% of processor busy. For each thread I have a new pointer for my wrapper class over sqlite(another connection.. etc...). Thank you. but, I have another question... It is possible this kind of statement? SELECT id, code FROM a WHERE (code IN (SELECT code FROM (SELECT code, COUNT(code) AS c FROM a GROUP BY code) AS aaa WHERE c 1) ) and ORDER BY code so this query check if the code appears more than once a time in the table -A- I run this kind of query but i got an error. The query is functional, cause I run the same query in sql server, in a database with the same structure. Thank you a lot. - Original Message - From: D. Richard Hipp [EMAIL PROTECTED] To: sqlite-users@sqlite.org Sent: Saturday, April 16, 2005 2:11 AM Subject: Re: [sqlite] multi threading On Fri, 2005-04-15 at 14:44 +0300, Cosmin Vlasiu wrote: Hello to everyone, I have a question... regarding multi-threading... the question is for microsoft windows (a visual c++ application)... I saw the documentation and I understood that for microsoft OS, the multi-threading is enabled by default. So, of course I start two threads, both of them make a loop into a recordset. the first thread it works good, but the second, no way. I made kind of research in the documentation and I observed that I have to make new connections for each thread. All donne. But when I run the application I got the same problem. In the second thread the sqlite3_prepare function always return the 21 value that means SQLITE_MISUSE, because sqlite3SafetyOn retun 1. Can somebody tells me if there is a solution for that problem? Two separate threads may not use the same database handle at the same time. If they do, the SQLITE_MISUSE value is often returned. Separate threads should have their own database handles. I know you said above that you are using separate database handles in your two threads. But probably there is a bug in your code that is preventing this from happening really. My advice to *all* programmers is to never use more than one thread in the same address space. I have never in 20 years worked on a multiple threaded program that actually got all of the threading issues right. There are always subtle bugs that cause error that are very difficult to reproduce and fix. Multithreading is the fastest road to buggy code that I know of. Avoid it. If you absolutely, positively must have multiple threads of control, put each thread in its own address space (make it a process.) -- D. Richard Hipp [EMAIL PROTECTED]
Re: [sqlite] multi threading
SELECT id, code FROM a WHERE (code IN (SELECT code FROM (SELECT code, COUNT(code) AS c FROM a GROUP BY code) AS aaa WHERE c 1) ) and ORDER BY code The and in your code is illegal, but it is better to use select id, code from a where code in (select code from a group by code having count(code) 1) order by code Regards
Re: [sqlite] multi threading
yes it is illegal, cause I've copy/paste then I adapt by replacing th name of table, i've extract irrelevant info and I forgot the and but, your statement it's very well thanks and regards - Original Message - From: Kurt Welgehausen [EMAIL PROTECTED] To: sqlite-users@sqlite.org Sent: Sunday, April 17, 2005 11:20 PM Subject: Re: [sqlite] multi threading SELECT id, code FROM a WHERE (code IN (SELECT code FROM (SELECT code, COUNT(code) AS c FROM a GROUP BY code) AS aaa WHERE c 1) ) and ORDER BY code The and in your code is illegal, but it is better to use select id, code from a where code in (select code from a group by code having count(code) 1) order by code Regards
[sqlite] multi threading
Hello to everyone, I have a question... regarding multi-threading... the question is for microsoft windows (a visual c++ application)... I saw the documentation and I understood that for microsoft OS, the multi-threading is enabled by default. So, of course I start two threads, both of them make a loop into a recordset. the first thread it works good, but the second, no way. I made kind of research in the documentation and I observed that I have to make new connections for each thread. All donne. But when I run the application I got the same problem. In the second thread the sqlite3_prepare function always return the 21 value that means SQLITE_MISUSE, because sqlite3SafetyOn retun 1. Can somebody tells me if there is a solution for that problem? Thanks in advance Cosmin
Re: [sqlite] multi threading
On Fri, 2005-04-15 at 14:44 +0300, Cosmin Vlasiu wrote: Hello to everyone, I have a question... regarding multi-threading... the question is for microsoft windows (a visual c++ application)... I saw the documentation and I understood that for microsoft OS, the multi-threading is enabled by default. So, of course I start two threads, both of them make a loop into a recordset. the first thread it works good, but the second, no way. I made kind of research in the documentation and I observed that I have to make new connections for each thread. All donne. But when I run the application I got the same problem. In the second thread the sqlite3_prepare function always return the 21 value that means SQLITE_MISUSE, because sqlite3SafetyOn retun 1. Can somebody tells me if there is a solution for that problem? Two separate threads may not use the same database handle at the same time. If they do, the SQLITE_MISUSE value is often returned. Separate threads should have their own database handles. I know you said above that you are using separate database handles in your two threads. But probably there is a bug in your code that is preventing this from happening really. My advice to *all* programmers is to never use more than one thread in the same address space. I have never in 20 years worked on a multiple threaded program that actually got all of the threading issues right. There are always subtle bugs that cause error that are very difficult to reproduce and fix. Multithreading is the fastest road to buggy code that I know of. Avoid it. If you absolutely, positively must have multiple threads of control, put each thread in its own address space (make it a process.) -- D. Richard Hipp [EMAIL PROTECTED]
