I might have missed something, but: What are you compressing? ehm... Are compressing the database file while sqlite is running ?
> The problem is happening in the field and the program deals with a corrupt > database by renaming it, logging the error and creating a new database. > So > I only end up with the corrupt one. > > I would GUESS that the database doesn't have any free pages except those > at > the end since it's an auditing product so we never delete anything...but I > could be wrong. > > The only reason I think I'm 'dropping a byte' is that all the > compression/encryption code is byte based and the database gets corrupt > (invalid pages) but I'm able to .dump it successfully. That and I try > to > take the blame first, since it's usually my fault. (just ask my wife!) > > Right now I'm running a large test on the encryption/compression code to > see > if I can get it to break. > > -----Original Message----- > From: sqlite-users-boun...@sqlite.org > [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Marcus Grimm > Sent: Wednesday, May 27, 2009 5:09 PM > To: 'General Discussion of SQLite Database' > Subject: Re: [sqlite] corrupt database recovery > > sorry, I don't know where sqlite stores any kind of data like > index or raw table, my suspect is that, since sqlite > usually looks for a free page, the pages are randomly > distributed over the file or added at the end if no free > page has been found. > > what I don't understand: > what makes you thinking that you may miss some bytes of a tree > (and thus of a page) ? > That is sqlite internal, except you are using your own > vfs, I guess, ...do you ? > > by the way: did your application crash prior or nearby you had this > corrupted DB ? > >> I've reviewed the code over and over again, but am yet to find it. But >> it >> is a good bit of very complicated code (blowfish and compression code). >> >> That's why I was asking about the format of the database. Since I'm >> able >> to >> successfully do a .dump but the integrity_check whines, I'm thinking >> that >> maybe if I focus on where what could cause that, it might give me a >> clue. >> >> For example, if the tree is stored at the end of file maybe I'm falling >> out >> the loop without writing the final bytes. But if the tree is not >> localized >> to one area of the file, I'll have to rethink my approach. >> >> >> -----Original Message----- >> From: sqlite-users-boun...@sqlite.org >> [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Marcus Grimm >> Sent: Wednesday, May 27, 2009 2:44 PM >> To: General Discussion of SQLite Database >> Subject: Re: [sqlite] corrupt database recovery >> >> so, if you think it is a coding error on your side it will >> be a hard work to find the source. I can only image side effects >> on an used API like sqlite3 with the classics: >> >> - stack frame overload in a function that calls sqlite functions >> - using a local stack variable as a return pointer and reuse >> this external. >> - memory overwriting on malloc'ed pointers or reusing of allready >> free'd memory pages. >> >> one and two might be found using a code review and identify >> local variables or particular arrays that are written: >> So wherever you call anything from sqlite check the stack >> declaration and review the usage of these variables... >> of course just don't do number two... ;) >> >> memory errors might be detected using special debug versions of the >> malloc/free library, by code review, or manually by adding some >> test variables whereever you malloc or free a pointer. >> >> I'm sure you allready did some of these... anyway, good luck >> >> are you using threads? would be another beautiful issue to trace...;) >> >> Marcus >> >> >> >>> It is a server class machine running Windows 2003 with 4 GB. No, it's >>> a >>> local drive with 20GB free on it. >>> >>> I'm sure that it's a coding error on my part. SQLite is very stable, >>> in >>> my >>> opinion. I'm just trying to get a rough idea on where I'm screwing up >>> the >>> database. >>> >>> >>> -----Original Message----- >>> From: sqlite-users-boun...@sqlite.org >>> [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of John Elrick >>> Sent: Wednesday, May 27, 2009 12:58 PM >>> To: General Discussion of SQLite Database >>> Subject: Re: [sqlite] corrupt database recovery >>> >>> What platform? Any chance they are using a network drive? >>> >>> >>> John >>> >>> Gene wrote: >>>> My code is outside the database layer. So I do all my database work, >>>> then >>>> compress and encrypt it. No errors are returned anywhere. I'm >>>> guessing >>>> that it's going to be an uninitialized variable or byte alignment >>>> problems >>>> somewhere. >>>> >>>> This code is running on hundreds of machines without a problem and >>>> I've >>>> never reproduced it but every now and again I get a support ticket >>>> showing >>> a >>>> corrupt database. So I'm trying to figure out WHERE to look. >>>> >>>> -----Original Message----- >>>> From: sqlite-users-boun...@sqlite.org >>>> [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of John Elrick >>>> Sent: Wednesday, May 27, 2009 10:59 AM >>>> To: General Discussion of SQLite Database >>>> Subject: Re: [sqlite] corrupt database recovery >>>> >>>> Gene Allen wrote: >>>> >>>>> Ok...it's happened again and I've decided that I need to track this >>>>> down >>>>> once and for all! >>>>> >>>>> Here is what I'm seeing: I get errors when I do a integrity_check >>>>> (see >>>>> below), but I can .dump it to a text file and then .read it into >>>>> another >>>>> database ok. >>>>> >>>>> It seems to me that I'm screwing up an index or something. Are >>>>> indexes >>>>> stored at the end of the database file? All I can think of is that >>>>> my >>>>> compression/encryption routines are messing something up and I'm >>>>> trying >>> to >>>>> figure out 'where' to look. >>>>> >>>>> I guess the real question is, what would I have to do to make an >>>>> integrity_check fail, but still let a dump work correctly? >>>>> >>>>> Many thanks for any advice on tracking down this ugliness. >>>>> >>>>> >>>> >>>> SNIP >>>> >>>> Personally, I'd refactor the code to allow me to verify the operation >>>> of >>>> the compression/encryption routines independently of the database >>>> operation. How are you injecting the compression/encryption into the >>>> database layer? >>>> >>>> >>>> John >>>> _______________________________________________ >>>> sqlite-users mailing list >>>> sqlite-users@sqlite.org >>>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users >>>> >>>> _______________________________________________ >>>> sqlite-users mailing list >>>> sqlite-users@sqlite.org >>>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users >>>> >>>> ------------------------------------------------------------------------ >>>> >>>> >>>> No virus found in this incoming message. >>>> Checked by AVG - www.avg.com >>>> Version: 8.0.238 / Virus Database: 270.12.39/2133 - Release Date: >>>> 05/25/09 >>> 08:16:00 >>>> >>>> >>> >>> _______________________________________________ >>> sqlite-users mailing list >>> sqlite-users@sqlite.org >>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users >>> >>> _______________________________________________ >>> sqlite-users mailing list >>> sqlite-users@sqlite.org >>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users >>> >> >> >> _______________________________________________ >> sqlite-users mailing list >> sqlite-users@sqlite.org >> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users >> >> > > > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > > _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
