Re: [sqlite] DatabaseError: database disk image is malformed

[Larry Knibb]

Just wanted to say thanks to everyone for the help... I've decided to
use MySQL for this particular setup rather than go down the route of
custom builds. SQLite remains my preference for exclusive scenarios
and it's only because I have to support scaling to simultaneous
connections that I'm compromising this time.

Always a pleasure to feel the high-engagement of this list. Thanks again.

Cheers,
Larry


On 16 October 2012 17:10, Clemens Ladisch  wrote:
> Keith Medcalf wrote:
>> Note that according to the Microsoft documentation opportunistic
>> locking is only used when overlapped I/O is enabled.
>
> That applies only to oplocks that are requested manually by
> an application through FSCTL_ control codes:
> http://msdn.microsoft.com/en-us/library/windows/desktop/aa365438(v=vs.85).aspx
>
> Windows can also request oplocks automatically, and this happens for
> both synchronous and asynchronous I/O.  (Internally, even synchronous
> operations are implemented using overlapped I/O:
> .)
>
>
> On OSes before Vista/Server 2008, oplocks were incompatible with byte
> range locks (which SQLite uses), but this is unlikely to happen
> nowadays.
>
>
> Regards,
> Clemens
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] DatabaseError: database disk image is malformed

[Clemens Ladisch]

Keith Medcalf wrote:
> Note that according to the Microsoft documentation opportunistic
> locking is only used when overlapped I/O is enabled.

That applies only to oplocks that are requested manually by
an application through FSCTL_ control codes:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa365438(v=vs.85).aspx

Windows can also request oplocks automatically, and this happens for
both synchronous and asynchronous I/O.  (Internally, even synchronous
operations are implemented using overlapped I/O:
.)


On OSes before Vista/Server 2008, oplocks were incompatible with byte
range locks (which SQLite uses), but this is unlikely to happen
nowadays.


Regards,
Clemens
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] DatabaseError: database disk image is malformed

[Keith Medcalf]

Note that according to the Microsoft documentation opportunistic locking is 
only used when overlapped I/O is enabled.  Of course, this is from Microsoft 
documentation, so it has to be taken with a really huge boulder of salt.  
Anyway, long and short is that you may be able to get windows to help less by 
adding some flags to the file open.

Perhaps try the following patch and see if it helps:

--- sqlite3.c   Sun Aug 19 12:32:41 2012
+++ ..\sqlite3.cSun Aug 19 11:54:19 2012
@@ -33446,8 +33448,14 @@
   }
   /* Reports from the internet are that performance is always
   ** better if FILE_FLAG_RANDOM_ACCESS is used.  Ticket #2699. */
-#if SQLITE_OS_WINCE
+#if SQLITE_OS_WINCE || SQLITE_WIN32_FILE_RANDOM
   dwFlagsAndAttributes |= FILE_FLAG_RANDOM_ACCESS;
+#elif SQLITE_WIN32_FILE_SEQUENTIAL
+  dwFlagsAndAttributes |= FILE_FLAG_SEQUENTIAL_SCAN;
+#endif
+
+#ifdef SQLITE_WIN32_FILE_WRITETHROUGH
+  dwFlagsAndAttributes |= FILE_FLAG_WRITE_THROUGH;
 #endif

   if( isNT() ){

Then set SQLITE_WIN32_FILE_RANDOM and SQLITE_WIN32_FILE_WRITETHROUGH when 
compiling the amalgamation (note the line numbers in the patch are not correct, 
they are for an earlier version of the amalgamation).  The former will stop 
windows from helping so much, and the latter will ensure that writes to sqlite 
opened files do not get cached.  This may or may not defeat the ill-conceived 
behaviour of Windows.  Then again, it might not.  Windows is pretty hopeless 
when it comes to I/O and it always has been thus.

---
()  ascii ribbon campaign against html e-mail
/\  www.asciiribbon.org




___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] DatabaseError: database disk image is malformed

[Yuriy Kaminskiy]

Larry Knibb wrote:
> On 15 October 2012 12:32, Keith Medcalf  wrote:
>> Define "clients".  Do you mean multiple client processes running on a single 
>> computer against a database hosted on an attached local device, such as on a 
>> Terminal Server for example?  Or do you mean multiple clients connecting 
>> over LANMAN/CIFS/NFS to a database file sitting on a remote fileserver?
> 
> It's the second one. The database file is being accessed over the network.
> 
>> If file locking is working correctly, you cannot have multiple writes to a 
>> database.  Write operations to the database are exclusive.
> 
> As Simon correctly guessed, the database isn't actually being
> corrupted; I'm just getting an error that suggests that it is. So I

Maybe it is not really corrupted now, but if
1) locking is not working properly (when database file is on networked fs), and
2) two writers will modify db at same time,
you *will* corrupt your database.

> suspect the writes are not conflicting, but one is failing (possibly)
> due to another happening at the same time and whatever locking/waiting
> not being observed properly results in this 'malformed' error rather
> than a blocking error.
> 
>> You may wish to try forcing locks to be acquired sooner when updating the 
>> database by using BEGIN IMMEDIATE or perhaps even BEGIN EXCLUSIVE before 
>> updating the database, and COMMIT when you are done.  You will then also 
>> need a busy-timeout so that other readers/writers will wait for that 
>> operation to complete.  You can set the timeout when creating the connection 
>> with sqlite3 in python, or by using PRAGMA busy_timeout
> 
> Thanks - I'll give that a shot.
> 
>> Latency of locking operations over network connections can play havoc, 
>> however.  It is possible if you are using a network mounted database file 
>> that the write operations are not being flushed properly and or the locks 
>> are not propagating in a timely fashion.
>>
>> http://www.sqlite.org/lang_transaction.html
>> http://www.sqlite.org/lockingv3.html
> 
> Reading up now...

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] DatabaseError: database disk image is malformed

[Clemens Ladisch]

Larry Knibb wrote:
> On 15 October 2012 16:35, Clemens Ladisch  wrote:
>> Which network protocol?  And what are the OSes on the clients and on the
>> file server?
>
> The server and clients are Windows machines so I guess (not my area of
> expertise) that the protocol is SMB/CIFS?

Yes.  The actual protocol (version) depends on the OS version.

In theory, locking on Windows works if the server and all clients have
the latest service packs, if no computer hangs or crashes, and if the
network is reliable.

SMB has a clever caching feature called opportunistic locking.
It is common pratice do disable it when one sees data corruption; see
.


Regards,
Clemens
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] DatabaseError: database disk image is malformed

[Larry Knibb]

On 15 October 2012 16:35, Clemens Ladisch  wrote:
> Which network protocol?  And what are the OSes on the clients and on the
> file server?

The server and clients are Windows machines so I guess (not my area of
expertise) that the protocol is SMB/CIFS?

Cheers,
Larry
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] DatabaseError: database disk image is malformed

[Clemens Ladisch]

Larry Knibb wrote:
> On 15 October 2012 12:32, Keith Medcalf  wrote:
>> Define "clients".  Do you mean multiple client processes running on
>> a single computer against a database hosted on an attached local
>> device, such as on a Terminal Server for example?  Or do you mean
>> multiple clients connecting over LANMAN/CIFS/NFS to a database file
>> sitting on a remote fileserver?
>
> It's the second one. The database file is being accessed over the
> network.

Which network protocol?  And what are the OSes on the clients and on the
file server?


Regards,
Clemens
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] DatabaseError: database disk image is malformed

[Larry Knibb]

On 15 October 2012 12:32, Keith Medcalf  wrote:
>
> Define "clients".  Do you mean multiple client processes running on a single 
> computer against a database hosted on an attached local device, such as on a 
> Terminal Server for example?  Or do you mean multiple clients connecting over 
> LANMAN/CIFS/NFS to a database file sitting on a remote fileserver?

It's the second one. The database file is being accessed over the network.

> If file locking is working correctly, you cannot have multiple writes to a 
> database.  Write operations to the database are exclusive.

As Simon correctly guessed, the database isn't actually being
corrupted; I'm just getting an error that suggests that it is. So I
suspect the writes are not conflicting, but one is failing (possibly)
due to another happening at the same time and whatever locking/waiting
not being observed properly results in this 'malformed' error rather
than a blocking error.

> You may wish to try forcing locks to be acquired sooner when updating the 
> database by using BEGIN IMMEDIATE or perhaps even BEGIN EXCLUSIVE before 
> updating the database, and COMMIT when you are done.  You will then also need 
> a busy-timeout so that other readers/writers will wait for that operation to 
> complete.  You can set the timeout when creating the connection with sqlite3 
> in python, or by using PRAGMA busy_timeout

Thanks - I'll give that a shot.

> Latency of locking operations over network connections can play havoc, 
> however.  It is possible if you are using a network mounted database file 
> that the write operations are not being flushed properly and or the locks are 
> not propagating in a timely fashion.
>
> http://www.sqlite.org/lang_transaction.html
> http://www.sqlite.org/lockingv3.html

Reading up now...

Cheers,
Larry
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] DatabaseError: database disk image is malformed

[Larry Knibb]

On 15 October 2012 11:54, Simon Slavin  wrote:
>
> I think you need someone who is more familiar with the 'malformed' error 
> indication than I am.  I really can't guess what's faking this response, 
> assuming that it is a byproduct of networking and the database really isn't 
> malformed.

What's interesting is that you're right about it being a fake response
(not sure how you knew that... you didn't say). The database is NOT
actually corrupted and re-running a job will more-often-than-not just
succeed on the second time of trying. So something in my setup is
causing erroneous malformed errors and failing INSERTs.

Looks like Keith had some tips in the next response. Thanks for your time Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] DatabaseError: database disk image is malformed

[Keith Medcalf]

> On Sunday, 14 October, 2012, at 20:28, Larry Knibb said:

> I'm trialling a centralised SQLite database to store process
> automation metadata with a known, small, fixed number clients (around
> 10) as part of a Jenkins CI system running on Windows. The clients
> connect occasionally and run simple, quick (sub-second) SELECT queries
> and even less frequent INSERTs using Python APIs. Due to the nature of
> the workflow, I'd estimate the chances of concurrent access are pretty
> low, although not impossible by any means. Recently, I've started
> seeing these errors related to some of the INSERT calls:

> DatabaseError: database disk image is malformed
> StorageError: database disk image is malformed

Define "clients".  Do you mean multiple client processes running on a single 
computer against a database hosted on an attached local device, such as on a 
Terminal Server for example?  Or do you mean multiple clients connecting over 
LANMAN/CIFS/NFS to a database file sitting on a remote fileserver?

> Given the scenario (and being familiar with
> http://www.sqlite.org/whentouse.html), I suspect this almost certainly
> an issue caused by concurrent writes to the database, but I'd like
> some advice before move to MySQL or similar.

If file locking is working correctly, you cannot have multiple writes to a 
database.  Write operations to the database are exclusive.
 
> Would using TRANSACTIONs on the INSERTs prevent this issue? If
> necessary, by combining this with a delay-retry mechanism in the
> client code.

You may wish to try forcing locks to be acquired sooner when updating the 
database by using BEGIN IMMEDIATE or perhaps even BEGIN EXCLUSIVE before 
updating the database, and COMMIT when you are done.  You will then also need a 
busy-timeout so that other readers/writers will wait for that operation to 
complete.  You can set the timeout when creating the connection with sqlite3 in 
python, or by using PRAGMA busy_timeout
 
> Any other tips or tricks to shore-up the robustness, or should I just
> limber up my throwing arm and call for the towel?
> 
> Cheers,
> Larry
> 
> P.S. just fmi, I'd also love to know -->definitively<-- if concurrent
> SELECT calls would ever be an issue. 

Multiple connections can concurrently read (as in SELECT).  Writing is 
exclusive.  For multiple processes on the same machine, WAL mode permits 
multiple readers and a single writer where readers do not block the writer and 
the writer does not block readers.

Latency of locking operations over network connections can play havoc, however. 
 It is possible if you are using a network mounted database file that the write 
operations are not being flushed properly and or the locks are not propagating 
in a timely fashion.

http://www.sqlite.org/lang_transaction.html
http://www.sqlite.org/lockingv3.html

---
()  ascii ribbon campaign against html e-mail
/\  www.asciiribbon.org




___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] DatabaseError: database disk image is malformed

[Simon Slavin]

On 15 Oct 2012, at 3:58am, Larry Knibb  wrote:

> On 15 October 2012 10:48, Simon Slavin  wrote:
>> 
> 
> Thanks Simon.
> 
> I'm new to Python so not sure how to call this C API from my Python
> script. http://docs.python.org/library/sqlite3.html doesn't offer the
> answer

You just need to know where to look because the Python interface for SQLite3 is 
weird.  It's an optional second parameter to the sqlite3.connect() routine.

But having found that and read it, the default for Python is 5 seconds, unlike 
the default for SQLite3 which is to immediately present an error.  And if 
Python really is giving you a reasonable timeout like 5 seconds this is 
probably nothing to do with the problem you're getting.

I think you need someone who is more familiar with the 'malformed' error 
indication than I am.  I really can't guess what's faking this response, 
assuming that it is a byproduct of networking and the database really isn't 
malformed.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] DatabaseError: database disk image is malformed

[Larry Knibb]

On 15 October 2012 10:48, Simon Slavin  wrote:
>
> On 15 Oct 2012, at 3:28am, Larry Knibb  wrote:
>
What you can do is introduce your own timeout and see if this changes
anything, just as something to try.
>
> 

Thanks Simon.

I'm new to Python so not sure how to call this C API from my Python
script. http://docs.python.org/library/sqlite3.html doesn't offer the
answer; similarly a Google search for
http://www.google.co.uk/search?q=call+sqlite+c+functions+from+python
is likewise uninspiring.

Do you know where I can read about this?

Cheers,
Larry
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] DatabaseError: database disk image is malformed

[Simon Slavin]

On 15 Oct 2012, at 3:28am, Larry Knibb  wrote:

> Recently, I've started
> seeing these errors related to some of the INSERT calls:
> 
> DatabaseError: database disk image is malformed
> StorageError: database disk image is malformed
> 
> Given the scenario (and being familiar with
> http://www.sqlite.org/whentouse.html), I suspect this almost certainly
> an issue caused by concurrent writes to the database, but I'd like
> some advice before move to MySQL or similar.
> 
> Would using TRANSACTIONs on the INSERTs prevent this issue? If
> necessary, by combining this with a delay-retry mechanism in the
> client code.

No.  Specifying an INSERT without a transaction just makes SQL wrap it in a 
transaction of its own.  I don't think that will improve anything.  What you 
can do is introduce your own timeout and see if this changes anything, just as 
something to try.



> Any other tips or tricks to shore-up the robustness, or should I just
> limber up my throwing arm and call for the towel?

Are you connecting across a network ?  Even if it's just one server and a 
network-mounted disk.  Generally, network file systems do not implement locking 
properly and this can present problems.

Are you using any PRAGMAs ?  I'm not recommending them -- in fact lack of 
PRAGMAs is probably the least likely to generate errors -- but it's important 
for that kind of problem.  If you're using any, try it without.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] DatabaseError: database disk image is malformed

[Larry Knibb]

I'm trialling a centralised SQLite database to store process
automation metadata with a known, small, fixed number clients (around
10) as part of a Jenkins CI system running on Windows. The clients
connect occasionally and run simple, quick (sub-second) SELECT queries
and even less frequent INSERTs using Python APIs. Due to the nature of
the workflow, I'd estimate the chances of concurrent access are pretty
low, although not impossible by any means. Recently, I've started
seeing these errors related to some of the INSERT calls:

DatabaseError: database disk image is malformed
StorageError: database disk image is malformed

Given the scenario (and being familiar with
http://www.sqlite.org/whentouse.html), I suspect this almost certainly
an issue caused by concurrent writes to the database, but I'd like
some advice before move to MySQL or similar.

Would using TRANSACTIONs on the INSERTs prevent this issue? If
necessary, by combining this with a delay-retry mechanism in the
client code.

Any other tips or tricks to shore-up the robustness, or should I just
limber up my throwing arm and call for the towel?

Cheers,
Larry

P.S. just fmi, I'd also love to know -->definitively<-- if concurrent
SELECT calls would ever be an issue.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] 'DatabaseError: database disk image is malformed'

[Simon Slavin]

On 10 Apr 2010, at 3:32pm, D. Richard Hipp wrote:

> none of that should be in the first page of the database file.  Nor  
> does that text appear to be in a format that would appear anywhere in  
> a valid SQLite database.  So I'm guessing that some other process has  
> decided to open the SQLite database and overwrite it with log file  
> information.

What that text is, is a copy of an HTTP conversation between a web browser and 
a web server.  Looking at the conversation, it's a copy of what happens when a 
client talks to the trac server itself.  It's not some completely irrelevant 
thing, but probably something happening when someone tried to download those 
very files.

This suggests to me that some sort of corruption has occurred on the server 
itself, either low-level hard disk corruption or the trac server has bad code 
in and is overwriting the files its meant to be serving.  Either way, if I was 
the admin of that server, I would be switching to panic mode and doing a 
widespread investigation into how many of my files were corrupt.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] 'DatabaseError: database disk image is malformed'

[exarkun]

On 02:21 pm, rut...@cs.vu.nl wrote:
>[I first posted this on the Trac user list. There, I was referred to 
>the
>sqlite mailing lists.]
>
>I am running 3 trac instances on a FreeBSD server; trac 0.11b, sqlite3
>3.4.1, pysqlite-2.3.5. More or less simultaneously (at least within a
>few days) all three trac databases got corrupted. You can check for
>yourself at e.g. http://trac.rfidguardian.org:8000/trac-0-11b, it shows
>a python stack trace with at bottom a database disk image is malformed.
>
>When I have a look at one of the sqlite3 databases, the header block is
>obviously corrupted:
>
>000  53 51 4C 69 74 65 20 66 6F 72 6D 61 74 20 33 00 SQLite format 3.
>010  04 00 01 01 00 40 20 20 00 01 D5 6D 00 00 00 00 .@  ...m
>020  00 00 24 AA 00 00 04 9A 38 32 2E 39 35 2E 31 35 ..$.82.95.15
>030  37 2E 32 31 20 2D 20 2D 20 5B 30 36 2F 41 70 72 7.21 - - [06/Apr
>040  2F 32 30 31 30 20 31 34 3A 33 30 3A 31 30 5D 20 /2010 14:30:10]
>050  22 47 45 54 20 2F 74 72 61 63 2D 30 2D 31 31 62 "GET /trac-0-11b
>060  20 48 54 54 50 2F 31 2E 31 22 20 35 30 30 20 2D  HTTP/1.1" 500 -
>070  0A 74 72 75 6E 6B 2F 73 72 63 2F 75 69 2F 73 74 .trunk/src/ui/st
>080  64 69 6E 20 48 54 54 50 2F 31 2E 31 22 20 35 30 din HTTP/1.1" 50
>090  30 20 2D 0A 39 39 34 20 48 54 54 50 2F 31 2E 31 0 -.994 HTTP/1.1
>0A0  22 20 35 30 30 20 2D 0A 2E 30 22 20 35 30 30 20 " 500 -..0" 500
>0B0  2D 0A 20 35 30 30 20 2D 0A 54 50 2F 31 2E 30 22 -. 500 -.TP/1.0"
>0C0  20 35 30 30 20 2D 0A 22 20 35 30 30 20 2D 0A 50  500 -." 500 -.P
>0D0  2F 31 2E 31 22 20 35 30 30 20 2D 0A 31 2E 30 22 /1.1" 500 -.1.0"
>0E0  20 35 30 30 20 2D 0A 30 30 39 2D 30 37 2D 32 34  500 -.009-07-24
>0F0  54 32 30 25 33 41 35 30 25 33 41 35 35 5A 25 32 T20%3A50%3A55Z%2
>
>E.g. bytes 0x01c..0x01f should give the database size in 1K pages; that
>should be 1408 / 1024 = 13750 = 0x35b6. It is 0x00. And
>other fields are as obviously broken.
>
>What can be the cause of this simultaneous corruption? Is there a way 
>to
>recover the database?

I've seen trac write random garbage (request logs, debug logs, etc) to 
random files (mostly password files, never trac.db itself though).

Perhaps that's what happened here, trac got confused, though the 
database file was a log file (either by name or perhaps by file 
descriptor) and dumped some access log bytes into it.

Jean-Paul
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] 'DatabaseError: database disk image is malformed'

[D. Richard Hipp]

On Apr 10, 2010, at 10:21 AM, Rutger Hofman wrote:

> [I first posted this on the Trac user list. There, I was referred to  
> the
> sqlite mailing lists.]
>
> I am running 3 trac instances on a FreeBSD server; trac 0.11b, sqlite3
> 3.4.1, pysqlite-2.3.5. More or less simultaneously (at least within a
> few days) all three trac databases got corrupted. You can check for
> yourself at e.g. http://trac.rfidguardian.org:8000/trac-0-11b, it  
> shows
> a python stack trace with at bottom a database disk image is  
> malformed.
>
> When I have a look at one of the sqlite3 databases, the header block  
> is
> obviously corrupted:
>
> 000  53 51 4C 69 74 65 20 66 6F 72 6D 61 74 20 33 00 SQLite format 3.
> 010  04 00 01 01 00 40 20 20 00 01 D5 6D 00 00 00 00 .@  ...m
> 020  00 00 24 AA 00 00 04 9A 38 32 2E 39 35 2E 31 35 ..$.82.95.15
> 030  37 2E 32 31 20 2D 20 2D 20 5B 30 36 2F 41 70 72 7.21 - - [06/Apr
> 040  2F 32 30 31 30 20 31 34 3A 33 30 3A 31 30 5D 20 /2010 14:30:10]
> 050  22 47 45 54 20 2F 74 72 61 63 2D 30 2D 31 31 62 "GET /trac-0-11b
> 060  20 48 54 54 50 2F 31 2E 31 22 20 35 30 30 20 2D  HTTP/1.1" 500 -
> 070  0A 74 72 75 6E 6B 2F 73 72 63 2F 75 69 2F 73 74 .trunk/src/ui/st
> 080  64 69 6E 20 48 54 54 50 2F 31 2E 31 22 20 35 30 din HTTP/1.1" 50
> 090  30 20 2D 0A 39 39 34 20 48 54 54 50 2F 31 2E 31 0 -.994 HTTP/1.1
> 0A0  22 20 35 30 30 20 2D 0A 2E 30 22 20 35 30 30 20 " 500 -..0" 500
> 0B0  2D 0A 20 35 30 30 20 2D 0A 54 50 2F 31 2E 30 22 -. 500 -.TP/1.0"
> 0C0  20 35 30 30 20 2D 0A 22 20 35 30 30 20 2D 0A 50  500 -." 500 -.P
> 0D0  2F 31 2E 31 22 20 35 30 30 20 2D 0A 31 2E 30 22 /1.1" 500 -.1.0"
> 0E0  20 35 30 30 20 2D 0A 30 30 39 2D 30 37 2D 32 34  500 -.009-07-24
> 0F0  54 32 30 25 33 41 35 30 25 33 41 35 35 5A 25 32 T20%3A50%3A55Z%2

All of that text in the header that looks like Apache logfile entires  
- none of that should be in the first page of the database file.  Nor  
does that text appear to be in a format that would appear anywhere in  
a valid SQLite database.  So I'm guessing that some other process has  
decided to open the SQLite database and overwrite it with log file  
information.

What does Trac call its database?  something.db?  Lots of programs use  
the ".db" suffix.  Perhaps one of these other programs mistook the  
SQLite database for a file in a different format and tried to  
overwrite it with new information that is in some other (non-SQLite)  
format.  Just a guess.

You are unlikely to be able to recover any useful content from a  
database that has been so thoroughly trashed.



>
> E.g. bytes 0x01c..0x01f should give the database size in 1K pages;  
> that
> should be 1408 / 1024 = 13750 = 0x35b6. It is 0x00. And
> other fields are as obviously broken.
>
> What can be the cause of this simultaneous corruption? Is there a  
> way to
> recover the database?
>
> Alas, we had a misunderstanding w/ our web host on backup policy; he
> turns out to keep multiple versions of the current files, but no  
> rollback...
>
> Thanks,
>
> Rutger Hofman
> VU Amsterdam
> http://www.rfidguardian.org
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

D. Richard Hipp
d...@hwaci.com



___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] 'DatabaseError: database disk image is malformed'

[Rutger Hofman]

[I first posted this on the Trac user list. There, I was referred to the 
sqlite mailing lists.]

I am running 3 trac instances on a FreeBSD server; trac 0.11b, sqlite3 
3.4.1, pysqlite-2.3.5. More or less simultaneously (at least within a 
few days) all three trac databases got corrupted. You can check for 
yourself at e.g. http://trac.rfidguardian.org:8000/trac-0-11b, it shows 
a python stack trace with at bottom a database disk image is malformed.

When I have a look at one of the sqlite3 databases, the header block is 
obviously corrupted:

000  53 51 4C 69 74 65 20 66 6F 72 6D 61 74 20 33 00 SQLite format 3.
010  04 00 01 01 00 40 20 20 00 01 D5 6D 00 00 00 00 .@  ...m
020  00 00 24 AA 00 00 04 9A 38 32 2E 39 35 2E 31 35 ..$.82.95.15
030  37 2E 32 31 20 2D 20 2D 20 5B 30 36 2F 41 70 72 7.21 - - [06/Apr
040  2F 32 30 31 30 20 31 34 3A 33 30 3A 31 30 5D 20 /2010 14:30:10]
050  22 47 45 54 20 2F 74 72 61 63 2D 30 2D 31 31 62 "GET /trac-0-11b
060  20 48 54 54 50 2F 31 2E 31 22 20 35 30 30 20 2D  HTTP/1.1" 500 -
070  0A 74 72 75 6E 6B 2F 73 72 63 2F 75 69 2F 73 74 .trunk/src/ui/st
080  64 69 6E 20 48 54 54 50 2F 31 2E 31 22 20 35 30 din HTTP/1.1" 50
090  30 20 2D 0A 39 39 34 20 48 54 54 50 2F 31 2E 31 0 -.994 HTTP/1.1
0A0  22 20 35 30 30 20 2D 0A 2E 30 22 20 35 30 30 20 " 500 -..0" 500
0B0  2D 0A 20 35 30 30 20 2D 0A 54 50 2F 31 2E 30 22 -. 500 -.TP/1.0"
0C0  20 35 30 30 20 2D 0A 22 20 35 30 30 20 2D 0A 50  500 -." 500 -.P
0D0  2F 31 2E 31 22 20 35 30 30 20 2D 0A 31 2E 30 22 /1.1" 500 -.1.0"
0E0  20 35 30 30 20 2D 0A 30 30 39 2D 30 37 2D 32 34  500 -.009-07-24
0F0  54 32 30 25 33 41 35 30 25 33 41 35 35 5A 25 32 T20%3A50%3A55Z%2

E.g. bytes 0x01c..0x01f should give the database size in 1K pages; that 
should be 1408 / 1024 = 13750 = 0x35b6. It is 0x00. And 
other fields are as obviously broken.

What can be the cause of this simultaneous corruption? Is there a way to 
recover the database?

Alas, we had a misunderstanding w/ our web host on backup policy; he 
turns out to keep multiple versions of the current files, but no rollback...

Thanks,

Rutger Hofman
VU Amsterdam
http://www.rfidguardian.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users