I have not! but I just might...thanks for the tip. I think I found my defect: my old stress tests was based on doing compression/encryptions/decryption/decompression passes on files of random sizes; so I would do about a 10 million passes or so and say...that's pretty good.
Well...a more structured test exposed the problem and it was this: The feof() does return true until you attempt to read PAST the end of a file. So the code worked great until the file's length was a multiple of the buffer size (in my case 262,144 bytes). As you can imagine that doesn't happen too often in the real world. Since I assumed that a feof would return true where there wasn't any more data in the file, I would start another pass at reading a chunk of data (which wouldn't find anything) and run thru the compression/encryption code. The compression code worked handled it correctly, but the encryption required that a DWORD boundary (blowfish) and since 0 is on such a boundary but at the wrong end...it would fail. I fixed the code to test for 0 bytes being read, instead of solely relying on feof() and all seems well. :) I would like to say thank you for all your help and advice. Gene -----Original Message----- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Ken Sent: Thursday, May 28, 2009 11:32 AM To: General Discussion of SQLite Database Subject: Re: [sqlite] corrupt database recovery Gene, Im sure others have suggested, but have you tried running your code through valgrind? Can you remove the custom VFS ? --- On Wed, 5/27/09, Gene Allen <g...@bystorm.com> wrote: > From: Gene Allen <g...@bystorm.com> > Subject: Re: [sqlite] corrupt database recovery > To: mgr...@medcom-online.de, "'General Discussion of SQLite Database'" <sqlite-users@sqlite.org> > Date: Wednesday, May 27, 2009, 4:51 PM > I've reviewed the code over and over > again, but am yet to find it. But it > is a good bit of very complicated code (blowfish and > compression code). > > That's why I was asking about the format of the > database. Since I'm able to > successfully do a .dump but the integrity_check whines, I'm > thinking that > maybe if I focus on where what could cause that, it might > give me a clue. > > For example, if the tree is stored at the end of file maybe > I'm falling out > the loop without writing the final bytes. But if the > tree is not localized > to one area of the file, I'll have to rethink my approach. > > > > -----Original Message----- > From: sqlite-users-boun...@sqlite.org > [mailto:sqlite-users-boun...@sqlite.org] > On Behalf Of Marcus Grimm > Sent: Wednesday, May 27, 2009 2:44 PM > To: General Discussion of SQLite Database > Subject: Re: [sqlite] corrupt database recovery > > so, if you think it is a coding error on your side it will > be a hard work to find the source. I can only image side > effects > on an used API like sqlite3 with the classics: > > - stack frame overload in a function that calls sqlite > functions > - using a local stack variable as a return pointer and > reuse > this external. > - memory overwriting on malloc'ed pointers or reusing of > allready > free'd memory pages. > > one and two might be found using a code review and > identify > local variables or particular arrays that are written: > So wherever you call anything from sqlite check the stack > declaration and review the usage of these variables... > of course just don't do number two... ;) > > memory errors might be detected using special debug > versions of the > malloc/free library, by code review, or manually by adding > some > test variables whereever you malloc or free a pointer. > > I'm sure you allready did some of these... anyway, good > luck > > are you using threads? would be another beautiful issue to > trace...;) > > Marcus > > > > > It is a server class machine running Windows 2003 with > 4 GB. No, it's a > > local drive with 20GB free on it. > > > > I'm sure that it's a coding error on my part. > SQLite is very stable, in > > my > > opinion. I'm just trying to get a rough idea on > where I'm screwing up the > > database. > > > > > > -----Original Message----- > > From: sqlite-users-boun...@sqlite.org > > [mailto:sqlite-users-boun...@sqlite.org] > On Behalf Of John Elrick > > Sent: Wednesday, May 27, 2009 12:58 PM > > To: General Discussion of SQLite Database > > Subject: Re: [sqlite] corrupt database recovery > > > > What platform? Any chance they are using a > network drive? > > > > > > John > > > > Gene wrote: > >> My code is outside the database layer. So I > do all my database work, > >> then > >> compress and encrypt it. No errors are > returned anywhere. I'm guessing > >> that it's going to be an uninitialized variable or > byte alignment > >> problems > >> somewhere. > >> > >> This code is running on hundreds of machines > without a problem and I've > >> never reproduced it but every now and again I get > a support ticket > >> showing > > a > >> corrupt database. So I'm trying to figure > out WHERE to look. > >> > >> -----Original Message----- > >> From: sqlite-users-boun...@sqlite.org > >> [mailto:sqlite-users-boun...@sqlite.org] > On Behalf Of John Elrick > >> Sent: Wednesday, May 27, 2009 10:59 AM > >> To: General Discussion of SQLite Database > >> Subject: Re: [sqlite] corrupt database recovery > >> > >> Gene Allen wrote: > >> > >>> Ok...it's happened again and I've decided that > I need to track this > >>> down > >>> once and for all! > >>> > >>> Here is what I'm seeing: I get errors when I > do a integrity_check (see > >>> below), but I can .dump it to a text file and > then .read it into > >>> another > >>> database ok. > >>> > >>> It seems to me that I'm screwing up an index > or something. Are indexes > >>> stored at the end of the database file? > All I can think of is that my > >>> compression/encryption routines are messing > something up and I'm trying > > to > >>> figure out 'where' to look. > >>> > >>> I guess the real question is, what would I > have to do to make an > >>> integrity_check fail, but still let a dump > work correctly? > >>> > >>> Many thanks for any advice on tracking down > this ugliness. > >>> > >>> > >> > >> SNIP > >> > >> Personally, I'd refactor the code to allow me to > verify the operation of > >> the compression/encryption routines independently > of the database > >> operation. How are you injecting the > compression/encryption into the > >> database layer? > >> > >> > >> John > >> _______________________________________________ > >> sqlite-users mailing list > >> sqlite-users@sqlite.org > >> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > >> > >> _______________________________________________ > >> sqlite-users mailing list > >> sqlite-users@sqlite.org > >> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > >> > >> > ------------------------------------------------------------------------ > >> > >> > >> No virus found in this incoming message. > >> Checked by AVG - www.avg.com > >> Version: 8.0.238 / Virus Database: 270.12.39/2133 > - Release Date: > >> 05/25/09 > > 08:16:00 > >> > >> > > > > _______________________________________________ > > sqlite-users mailing list > > sqlite-users@sqlite.org > > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > > > > _______________________________________________ > > sqlite-users mailing list > > sqlite-users@sqlite.org > > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > > > > > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
