[sqlite] Database Corrupted!

2013-05-27 Thread Woody Wu
Hi, List Probably this is another case of database corrupted. I read the documents about this topic and think I did not make same mistakes described in that 'how to corrupt ...' documentation. I have a testing code, attached in this email, if continuously run it for 20 - 40 hours, the sqlite data

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Woody Wu
On Mon, May 27, 2013 at 11:08:57AM +0800, Woody Wu wrote: > Hi, List > > Probably this is another case of database corrupted. I read the > documents about this topic and think I did not make same mistakes > described in that 'how to corrupt ...' documentation. > > I have a testing code, attached

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Clemens Ladisch
Woody Wu wrote: > I have a testing code, attached in this email, if continuously run it for > 20 - 40 hours, the sqlite database will be corrupted. > > The application is running on an ARM Linux system with Yaffs2 filesystem > on NAND flashes. I'd guess that the flash is not very reliable. Does t

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Simon Slavin
On 27 May 2013, at 1:25pm, Clemens Ladisch wrote: > Woody Wu wrote: >> I have a testing code, attached in this email, if continuously run it for >> 20 - 40 hours, the sqlite database will be corrupted. >> >> The application is running on an ARM Linux system with Yaffs2 filesystem >> on NAND fla

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Richard Hipp
On Mon, May 27, 2013 at 8:28 AM, Simon Slavin wrote: > > On 27 May 2013, at 1:25pm, Clemens Ladisch wrote: > > > Woody Wu wrote: > >> I have a testing code, attached in this email, if continuously run it > for > >> 20 - 40 hours, the sqlite database will be corrupted. > >> > >> The application i

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Woody Wu
Richard, If Yaffs2 is the cause, how can I write an effective test to exposure it? I straced my test sqlite test program and wanted to understand its IO behavior pattern. The difficulty is that I cannot run strace on my ARM target too long since the log will fill the limited memory, but if I don

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Simon Slavin
On 27 May 2013, at 4:22pm, Woody Wu wrote: > If Yaffs2 is the cause, how can I write an effective test to exposure it? Do you have an opportunity to format the same drive in a different format ? I'm not telling you to change your long-term practises, just to try a different format for testin

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Richard Hipp
On Mon, May 27, 2013 at 11:22 AM, Woody Wu wrote: > Richard, > > If Yaffs2 is the cause, how can I write an effective test to exposure it? > Do you really need to prove that yaffs2 is at fault? Try this experiment: mention that you might be having problems with yaffs2 to anybody else who has do

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Richard Hipp
On Mon, May 27, 2013 at 11:31 AM, Simon Slavin wrote: > > > On one hand, I am not > > sure if this is caused by sqlite or Yaffs2 itself, on the other hand, I > > also cannot prove this really means bad things since the program at that > > moment was still running fine. > > The chances that this

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Thanumalayan Sankaranarayana Pillai
Hi Woody, If the log messages that you see are "chunk nnn not erased", it might probably be an error between your NAND device and YAFFS2 (according to a couple of Google searches). Ref: http://osdir.com/ml/linux.file-systems.yaffs/2006-09/msg00033.html -- Thanu On Mon, May 27, 2013 at 10:22 AM

[sqlite] dramatic performance drop somewhere between 3.7.09.0 and 3.7.17.0

2013-05-27 Thread Wolfgang Enzinger
Hallo list, I detected a dramatic performance drop with a certain query which occurred somewhere between versions 3.7.09.0 and 3.7.17.0. Now this is not a real showstopper because I also noticed than an ANALYZE cures the issue in the latter version immediately. However, the drop without ANALYZE is

Re: [sqlite] dramatic performance drop somewhere between 3.7.09.0 and 3.7.17.0

2013-05-27 Thread Wolfgang Enzinger
Richard Hipp writes: > However, once we get the new query planner running and ready to beta-test, > we really, really would appreciate you giving it a spin and letting us know > if you encounter any problems with it. Alright, will do. Thanks for the great work! Wolfgang __

Re: [sqlite] dramatic performance drop somewhere between 3.7.09.0 and 3.7.17.0

2013-05-27 Thread Richard Hipp
On Mon, May 27, 2013 at 2:36 PM, Wolfgang Enzinger wrote: > Hallo list, > > I detected a dramatic performance drop with a certain query which occurred > somewhere between versions 3.7.09.0 and 3.7.17.0. Now this is not a real > showstopper because I also noticed than an ANALYZE cures the issue in

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Woody Wu
On Mon, May 27, 2013 at 04:31:25PM +0100, Simon Slavin wrote: > > On 27 May 2013, at 4:22pm, Woody Wu wrote: > > > If Yaffs2 is the cause, how can I write an effective test to exposure it? > > Do you have an opportunity to format the same drive in a different > format ? I'm not telling you to

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Woody Wu
On Mon, May 27, 2013 at 11:33:08AM -0400, Richard Hipp wrote: > On Mon, May 27, 2013 at 11:22 AM, Woody Wu wrote: > > > Richard, > > > > If Yaffs2 is the cause, how can I write an effective test to exposure it? > > > > Do you really need to prove that yaffs2 is at fault? Try this experiment: >

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Woody Wu
On Mon, May 27, 2013 at 10:59:25AM -0500, Thanumalayan Sankaranarayana Pillai wrote: > Hi Woody, > > If the log messages that you see are "chunk nnn not erased", it might > probably be an error between your NAND device and YAFFS2 (according to a > couple of Google searches). > > Ref: http://osdi

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Woody Wu
On Mon, May 27, 2013 at 04:31:25PM +0100, Simon Slavin wrote: > > On 27 May 2013, at 4:22pm, Woody Wu wrote: > > > If Yaffs2 is the cause, how can I write an effective test to exposure it? > > Do you have an opportunity to format the same drive in a different format ? > I'm not telling you to

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Simon Slavin
On 28 May 2013, at 2:37am, Woody Wu wrote: > How do you guys think about this: if NAND has an > IO problem, Yaffs2 should recover it or forward the error to > applications, right? Arguably. The file system can send an error back to the application. If something does that to SQLite3 SQLite3 w

Re: [sqlite] Best way to implement Find commands

2013-05-27 Thread James K. Lowden
On Fri, 24 May 2013 12:30:58 + (UTC) ven...@intouchmi.com wrote: > I was hoping that by making the exact same query to build a temporary > table, SQLite would go through the same procedure thereby loading the > temp table in the same order that stepping would have generated. If > this hope is

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Woody Wu
On Tue, May 28, 2013 at 03:08:55AM +0100, Simon Slavin wrote: > > On 28 May 2013, at 2:37am, Woody Wu wrote: > > > How do you guys think about this: if NAND has an > > IO problem, Yaffs2 should recover it or forward the error to > > applications, right? > > Arguably. The file system can send a

Re: [sqlite] Is pragma foreign_key_check fast enough to be used in normal application flow?

2013-05-27 Thread James K. Lowden
On Fri, 24 May 2013 13:47:08 +0200 Clemens Ladisch wrote: > Foreign key checks are not intended to prevent user errors but to > prevent programming errors. In other words, your program is > responsible for preventing the user from trying to delete some X that > is still referenced by some Y (by