On Tue, Feb 7, 2012 at 5:19 PM, David Barrett <dbarr...@expensify.com>wrote:
> Hi! We're seeing the dreaded "Error: database disk image is malformed"
> message, even though the database is actually fine. I've read a few other
> threads on it and I'd previously concluded that it was just an outstanding
> bug. But I'm wondering if it's actually due to a combination of disabling
> synchronous writing AND disabling the shared cache, and then accessing it
> from a second process. Have you heard of this? More details follow:
>
> 1) First, we have a server with two threads: the primary thread doing the
> main activity, and a second thread doing checkpointing.
>
> 2) However, we get erratic behavior when using the sqlite3 command-line
> tool to just do a basic select on the database: sometimes it works,
> sometimes it returns "Error: database disk image malformed". Sometimes we
> just run the same command many times until it works.
>
As the very first thing you do in the command-line tool, enter this command:
.log stdout
That will cause additional error diagnostics to appear on standard output.
Then do your commands that provoke the malformed error, and let us know
what you see as output.
>
> 3) The main server thread opens the database like:
> sqlite3_config( SQLITE_CONFIG_MULTITHREAD )
> sqlite3_enable_shared_cache( 0 )
> #define DB_OPEN_FLAGS SQLITE_OPEN_READWRITE | \
> SQLITE_OPEN_CREATE | \
> SQLITE_OPEN_NOMUTEX
> sqlite3_open_v2( filename.c_str(), &_db, DB_OPEN_FLAGS, NULL )
> PRAGMA legacy_file_format = OFF
> PRAGMA synchronous = OFF
> PRAGMA count_changes = OFF
> PRAGMA temp_store = MEMORY
> PRAGMA journal_mode = WAL
> PRAGMA wal_autocheckpoint = 0
>
> 4) All this works great for server threads, and we have the same exact
> commits happening in the same exact order on 5 different servers (using a
> custom distributed transaction layer).
>
> 5) And this has worked great even with the command-line access for a very
> long time -- years now, in fact. But as the database grows (>40GB) and
> activity increases, we've started to get the malformed error more and more
> frequently -- to the point that it's also impossible to query the database
> from the command line anymore.
>
> 6) I had just accepted this, but recently we did queries on two servers
> via the sqlite3 command-line, and found they were giving back *very*
> different results -- for several minutes. Then I'm guessing both flushed
> their changes to disk, after which they gave the same result. This made me
> think "Hm, maybe the command line process isn't able to access the
> in-memory cached data of the server process, and maybe that's what's
> causing the malformed error?"
>
> Anyway, that's the background -- has anybody encountered this before? I'd
> welcome any advice. Thanks!
>
> -david
> Founder and CEO of Expensify
> Follow us at http://twitter.com/expensify
> ______________________________**_________________
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-**bin/mailman/listinfo/sqlite-**users<http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users>
>
--
D. Richard Hipp
d...@sqlite.org
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
