I'm sorry that I can't help with the SQLite part, but couldn't you write a simple test harness that would create a file (of random size?) with random contents, encrypt/compress to a secondary file, then decompress/decrypt to a third file. Compare first and third files. If they match, delete and run test again. If they don't match you now have your debugging input. I'd think letting something like that run for a day or so would flush out boundary/buffer errors. It's not deterministic, but should be easy enough to bang out and let run on a secondary machine while you continue your main search.
Good luck either way. Doug > -----Original Message----- > From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users- > boun...@sqlite.org] On Behalf Of Gene Allen > Sent: Wednesday, May 27, 2009 4:52 PM > To: mgr...@medcom-online.de; 'General Discussion of SQLite Database' > Subject: Re: [sqlite] corrupt database recovery > > I've reviewed the code over and over again, but am yet to find it. But > it > is a good bit of very complicated code (blowfish and compression code). > > That's why I was asking about the format of the database. Since I'm > able to > successfully do a .dump but the integrity_check whines, I'm thinking > that > maybe if I focus on where what could cause that, it might give me a > clue. > > For example, if the tree is stored at the end of file maybe I'm falling > out > the loop without writing the final bytes. But if the tree is not > localized > to one area of the file, I'll have to rethink my approach. > > > -----Original Message----- > From: sqlite-users-boun...@sqlite.org > [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Marcus Grimm > Sent: Wednesday, May 27, 2009 2:44 PM > To: General Discussion of SQLite Database > Subject: Re: [sqlite] corrupt database recovery > > so, if you think it is a coding error on your side it will > be a hard work to find the source. I can only image side effects > on an used API like sqlite3 with the classics: > > - stack frame overload in a function that calls sqlite functions > - using a local stack variable as a return pointer and reuse > this external. > - memory overwriting on malloc'ed pointers or reusing of allready > free'd memory pages. > > one and two might be found using a code review and identify > local variables or particular arrays that are written: > So wherever you call anything from sqlite check the stack > declaration and review the usage of these variables... > of course just don't do number two... ;) > > memory errors might be detected using special debug versions of the > malloc/free library, by code review, or manually by adding some > test variables whereever you malloc or free a pointer. > > I'm sure you allready did some of these... anyway, good luck > > are you using threads? would be another beautiful issue to trace...;) > > Marcus > > > > > It is a server class machine running Windows 2003 with 4 GB. No, > it's a > > local drive with 20GB free on it. > > > > I'm sure that it's a coding error on my part. SQLite is very stable, > in > > my > > opinion. I'm just trying to get a rough idea on where I'm screwing > up the > > database. > > > > > > -----Original Message----- > > From: sqlite-users-boun...@sqlite.org > > [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of John Elrick > > Sent: Wednesday, May 27, 2009 12:58 PM > > To: General Discussion of SQLite Database > > Subject: Re: [sqlite] corrupt database recovery > > > > What platform? Any chance they are using a network drive? > > > > > > John > > > > Gene wrote: > >> My code is outside the database layer. So I do all my database > work, > >> then > >> compress and encrypt it. No errors are returned anywhere. I'm > guessing > >> that it's going to be an uninitialized variable or byte alignment > >> problems > >> somewhere. > >> > >> This code is running on hundreds of machines without a problem and > I've > >> never reproduced it but every now and again I get a support ticket > >> showing > > a > >> corrupt database. So I'm trying to figure out WHERE to look. > >> > >> -----Original Message----- > >> From: sqlite-users-boun...@sqlite.org > >> [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of John Elrick > >> Sent: Wednesday, May 27, 2009 10:59 AM > >> To: General Discussion of SQLite Database > >> Subject: Re: [sqlite] corrupt database recovery > >> > >> Gene Allen wrote: > >> > >>> Ok...it's happened again and I've decided that I need to track this > >>> down > >>> once and for all! > >>> > >>> Here is what I'm seeing: I get errors when I do a integrity_check > (see > >>> below), but I can .dump it to a text file and then .read it into > >>> another > >>> database ok. > >>> > >>> It seems to me that I'm screwing up an index or something. Are > indexes > >>> stored at the end of the database file? All I can think of is that > my > >>> compression/encryption routines are messing something up and I'm > trying > > to > >>> figure out 'where' to look. > >>> > >>> I guess the real question is, what would I have to do to make an > >>> integrity_check fail, but still let a dump work correctly? > >>> > >>> Many thanks for any advice on tracking down this ugliness. > >>> > >>> > >> > >> SNIP > >> > >> Personally, I'd refactor the code to allow me to verify the > operation of > >> the compression/encryption routines independently of the database > >> operation. How are you injecting the compression/encryption into > the > >> database layer? > >> > >> > >> John > >> _______________________________________________ > >> sqlite-users mailing list > >> sqlite-users@sqlite.org > >> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > >> > >> _______________________________________________ > >> sqlite-users mailing list > >> sqlite-users@sqlite.org > >> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > >> > >> -------------------------------------------------------------------- > ---- > >> > >> > >> No virus found in this incoming message. > >> Checked by AVG - www.avg.com > >> Version: 8.0.238 / Virus Database: 270.12.39/2133 - Release Date: > >> 05/25/09 > > 08:16:00 > >> > >> > > > > _______________________________________________ > > sqlite-users mailing list > > sqlite-users@sqlite.org > > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > > > > _______________________________________________ > > sqlite-users mailing list > > sqlite-users@sqlite.org > > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > > > > > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
