In general worker threads is not an efficient solution to the problem even if parallelism is available. There is nothing to be gained by having one thread handing off A to another set of worker threads because you have to synchronise on whatever queue/list/pipe you use to transfer the information between threads anyway. So you may as well just have multiple threads all performing their A,B, and C operations in parallel and they will naturally interleave their operations based on the synchronisation that you use.
Typically worker threads are useful when you want something to be done asynchronously in the background. I dont think its a suitable design for this task though because it does not do anything that multiple threads with mutexes cant do in the foreground and only introduces overhead and delay in the queueing process and context switching. On the second point, i cant see any reason why the VDBE design would be a problem for what i want to do. Taking out a table wide lock is fine, so long as i can make changes to that table from multiple threads using the same connection and lock. The only danger occurs if those threads are not synchronised and two or more attempt to modify the table at once, but of course, thats what mutexes are for. Emerson On 12/29/06, Roger Binns <[EMAIL PROTECTED]> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Emerson Clarke wrote: | For architectural reasons the above steps must be performed in that | order. This means that operations cant be separated or queued up in | the way that you suggested. Each step is dependent on the previous | step. I was looking for parallelism expecting that indexing one document has no dependencies on other documents. Lets say that you have 3 documents A, B and C. Is it impossible to index B until after A's indexing is done? If that is the case then your indexing process is imposing serialization and there is nothing SQLite can do about that. If the indexing of A, B & C can be done in parallel, then you can have the indexing workers send updates to a single thread/process. That can aggregate information and then dump it all in a single transaction quickly. | Of course i know the simple answer, which is that the current api does | not support this. But im wondering why, and if there are any other | ways to achieve the desired performance. It is actually the internal design of SQLite that doesn't support this. ~ (Look at VDBE instructions for queries sometime). SQLite locks the whole database, rather than tables or rows. But the locks are held for the shortest time period possible. There are other database systems out there that have finer granularity locking, but that is more complicated. ~ There are also potential patent issues and SQLite deliberately uses old techniques to ensure no patent violation. SQLite's techniques have to work without centrally coordinating mechanism and within the confines of lowest common denominator functionality available on a variety of operating systems. Quite frankly if you cannot change the way you work with your data set then you may want to try a different database system such as Postgres. Of course that internally solves the problems by having a single process and doesn't have to worry about any other process/thread stepping on its toes. Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFFlLG+mOOfHg372QQRAn+sAKCrImPgr0Ex1xo3XOXgWg0E7t3wWQCgtAAQ ZB5bs3fFntT4u3rABIGcOik= =c2pB -----END PGP SIGNATURE----- ----------------------------------------------------------------------------- To unsubscribe, send email to [EMAIL PROTECTED] -----------------------------------------------------------------------------
----------------------------------------------------------------------------- To unsubscribe, send email to [EMAIL PROTECTED] -----------------------------------------------------------------------------
