[sqlite] SQLITE_SCHEMA error and pragmas
Hello, After some time, I've gone back to working with precompiled queries and found out why I was consistently getting SQLITE_SCHEMA errors. In my code, I was calling two pragmas after I compiled the query and before I executed it. Removing the pargmas removed the problem. The pargmas were: PRAGMA short_column_names = OFF PRAGMA full_column_names = ON My question is, do these queries change the schema? Is this the expected behaviour? It's not a problem either way, I'm just curious. If these (or other/all?) pragmas will have this effect, it might be useful to add this to the FAQ (http://www.sqlite.org/faq.html#q17). Cheers, Demitri
Re: [sqlite] Multi-threading.
- Original Message - From: "Kervin L. Pierre" <[EMAIL PROTECTED]> To: Sent: Saturday, July 16, 2005 9:31 AM Subject: Re: [sqlite] Multi-threading. > Paul G wrote: > > > > richard's advice is solid. use async io/event loops if possible, separate > > processes if possible, threads as a last resort, in that order. the grey > > area is the 'if possible' test, since it's a multi-way tradeoff between > > performance, simplicity and provable (to an extent) correctness. i fully > > expect that a lot of folks *do* need to use threads and the probability of > > that being the case on windows is much higher than on posixish platforms. > > > > I agree with you, but it doesn't seem like > you're exactly concurring with what DRH > said though. right, i'm not concurring with *everything* he wrote. eg. structured programming wasn't meant to reduce buggage directly, but rather to improve maintainability of code (which has the indirect effect of reducing buggage). however, his main point that arbitrary use of a shared address space with concurrent access is a bit like drinking and skydiving (ie a bad idea which you come to regret very fast but never get the chance to regret for very long ;) is indeed very cogent. the key is understanding that nothing is ever black and white. the more restrictions you place on usage of shared data structures, the less concurrency-related issues you are going to have, but take it far enough and you've got the equivalent of ipc api-wise. > I'm guessing that that 'if possible' test > is huge, and very frequently will fail. right. event loops are great for i/o heavy apps, but you will still need to farm compute heavy units of work to a thread pool (this is a very common pattern), not to mention the fact that a lot of external dependencies (ie db client libs) do not provide an async api. separate processes are fine for a certain range of tasks, but as soon as you need to move a lot of data back and forth or spawn a lot of them, your performance tanks. threads have neither of the above issues, but you then need to figure out why the thing crashes when jupiter is in saggitarius and the moon is full (hint: many threadsafe libs aren't, pain invariably ensues ;). when you're making the call on which way to go, you invariably consider all the tradeoffs and go with the least worst of your options given your constraints (which model fits the largest part of the app best, things you need to interface with, capabilities of your team, deadlines, whether you're going to still be there to take the flogging when the fit hits the shan etc). > Why suffer the context switch when you don't > have to? Would you write a non-trivial GUI > program today without using threads? Why > subject the users to the added latency IPC > is going to introduce? if the language i'm writing it in has a good async io framework, the app is not compute bound and the windowing toolkit supports it, most definitely. consider the fact that the gui handling is (in most modern windowing toolkits) an event loop and compute heavy tasks are farmed off to a thread pool and you'll start to see the big picture. > The funny thing is eventually multi-process > apps go to MMap for IPC, for performance, and > then run into a lot of the same issues they'd > have to deal with if they were threaded. performance must suck enough for this to happen; this is not always the case. beyond that, there is a big difference between taking a shared address space for granted and being forced to think about access to shared data structures from a psychological point of view. i must confess, however, that i usually oscillate between the pure event loop, event loop + task queue + thread pool and pure multithreading for the things i write (no separate processes included here, with the hybrid model being the most common by far), but that is down to the things i usually need to do and the fact that i generally trust myself to do a decent job of fencing my shared data access given the relative rigidity of the task queue + thread pool pattern. > And as far as the 'threads introduce > difficult to debug' errors; Isn't that the > age-old performance versus complexity trade- > off? not necessarily. i routinely see multithreaded applications which would have a simpler and more performant event loop implementation. this is a bit of a contradiction given my previous statement that the two are isomorphic, but in the real world mt apps are often suboptimally implemented due to the complexity of an optimal implementation. iow, given locking which is fine grained enough, the overhead of locking shouldn't differ significantly from the overhead of the event loop, but locking which is fine grained enough is too tough and you see lock contention (very hard to debug) and inefficiencies elsewhere because the deadline has come and passed while everyone was debugging elusive random data corruption/memory leak/what-have-you somewhere. i'm sure you've seen apps which
Re: [sqlite] Multi-threading.
Paul G wrote: richard's advice is solid. use async io/event loops if possible, separate processes if possible, threads as a last resort, in that order. the grey area is the 'if possible' test, since it's a multi-way tradeoff between performance, simplicity and provable (to an extent) correctness. i fully expect that a lot of folks *do* need to use threads and the probability of that being the case on windows is much higher than on posixish platforms. I agree with you, but it doesn't seem like you're exactly concurring with what DRH said though. I'm guessing that that 'if possible' test is huge, and very frequently will fail. Why suffer the context switch when you don't have to? Would you write a non-trivial GUI program today without using threads? Why subject the users to the added latency IPC is going to introduce? The funny thing is eventually multi-process apps go to MMap for IPC, for performance, and then run into a lot of the same issues they'd have to deal with if they were threaded. And as far as the 'threads introduce difficult to debug' errors; Isn't that the age-old performance versus complexity trade- off? Processes are easier to use, but very often perform worse under the same conditions as the more complex threaded application. That is a fact many of us can not look past easily. PS. It's funny, this discussion seems like its been taked right from a early '90s newsgroup :) Regards, Kervin
Re: [sqlite] Multi-threading.
- Original Message - From: "Andrew Piskorski" <[EMAIL PROTECTED]> To: Sent: Friday, July 15, 2005 1:27 PM Subject: Re: [sqlite] Multi-threading. > On Fri, Jul 15, 2005 at 01:04:50PM -0400, Paul G wrote: > > > the issue wasn't necessarily the thread implementation per se, but the fact > > that threads were treated as processes for scheduling purposes and hence > > scheduled with the regular process scheduler, which was not efficient for a > > large number of processes. these problems went away when ingo molnar's O(1) > > scheduler was adopted (not sure when it was merged into linus' 2.4, but > > Interesting. Then that may explain why I never heard any first hand > reports of notable Linux threading problems. The people I tended to > talk to were probably all running with well under 100 threads per > process, and only 2 or 3 such processes at most. Presumably even the > earlier lousy process scheduler could handle that ok. all depends on what these threads were doing and the activation pattern - a bunch of compute intensive, fixed work slice threads in a constant pool are a completely different animal to, say, threads handling network and disk i/o with variable workload in dynamically growing/shrinking thread pools. another notable issue was the so-called 'thundering herd' problem - processes (traditional or threads) waiting on an event being *all* woken up when the event comes in. this would happen on accept(), for example, and the scheduler would thrash. windows had an elegant solution for this (since at least nt5) in completion ports. linux 2.6 has epoll, which has been backported to 2.4 and available in distro kernels at least for quite some time. we're getting a bit too off-topic here. in short, multi-threading and event driven models are equivalent (this has been demonstrated formally) for ideal implementations of both. the trouble is, if w're talking c, that while there are many good event driven model implementations, threads invariably suck. they are more intuitive from a comprehension standpoint, but the devil is in the details, namely concurrency/locking issues. for bonus points, use both at the same time and watch your brain (and debugger) melt. practically speaking, threads are non-trivial to do right in non-trivial applications without higher-level primitives, cf. erlang and various actor-based languages. richard's advice is solid. use async io/event loops if possible, separate processes if possible, threads as a last resort, in that order. the grey area is the 'if possible' test, since it's a multi-way tradeoff between performance, simplicity and provable (to an extent) correctness. i fully expect that a lot of folks *do* need to use threads and the probability of that being the case on windows is much higher than on posixish platforms. -p
