[sqlite] Database corruption using zipvfs on android 64 bits (x86_64)

[Dan Kennedy]

On 11/06/2015 11:57 PM, Alexandre Mainville wrote:
> Hi,
>
> I am experiencing database corruption using zipvfs on android 64
> bits (x86_64).
> The same program executing on 32 bits runs without problems.
>
> The program creates a new database on every run and does a series of
> inserts and updates (always the same series of operations).
>
> Both programs produce the same SQLITE_DEBUG trace up until the point the 64
> bit version outputs:
> zipvfs database is corrupt. Line 1100 of [9d6c1880fb]
> because of a zero size payload in a slot header.
>
> I am waiting on a device to test behavior with arm64-v8a.
>
> I am using sqlite 3.8.8.3 and zipvfs 3.8.8.
>
> If anyone has experienced the same or has some insight on this, it would be
> greatly appreciated.


Are you able to run this on a workstation? If so, try running the 
program under valgrind ( http://valgrind.org ).

One way to debug this might be to run the zipvfs integrity-check (i.e. 
ZIPVFS_CTRL_INTEGRITY_CHECK) after each transaction. If you can figure 
out which transaction is introducing the corruption and take a copy of 
the database before and after it is executed, we might be able to figure 
out what is going on based on that and the log below.

Another would be to capture the statements executed using an 
sqlite3_trace() callback, then try to use them to reproduce the 
corruption using the shell tool. Even if that doesn't produce the 
corruption, it will be worth running such a script under valgrind to 
search for uninitialized variable problems etc.

I can't see any bugfixes in the changelog since 3.8.8 to do with corrupt 
databases, but it might be worth upgrading to the latest zipvfs and 
sqlite to check if the problem is still present anyway.

Regards,
Dan.

[sqlite] Database corruption using zipvfs on android 64 bits (x86_64)

[Dominique Pellé]

Alexandre Mainville wrote:

> Hi,
>
> I am experiencing database corruption using zipvfs on android 64
> bits (x86_64).
> The same program executing on 32 bits runs without problems.
>
> The program creates a new database on every run and does a series of
> inserts and updates (always the same series of operations).
>
> Both programs produce the same SQLITE_DEBUG trace up until the point the 64
> bit version outputs:
> zipvfs database is corrupt. Line 1100 of [9d6c1880fb]
> because of a zero size payload in a slot header.
>
> I am waiting on a device to test behavior with arm64-v8a.
>
> I am using sqlite 3.8.8.3 and zipvfs 3.8.8.
>
> If anyone has experienced the same or has some insight on this, it would be
> greatly appreciated.
>
> Thanks
>
> Alex

I'm not sure whether it's the same problem, but on our
project we had memory corruptions on Android with zipvfs.
The bug happened only on Android. Valgrind and asan on
Linux was showing no bug.

The bug turned out not to be in SQLite, but in zlib.
Android was using zlib-1.2.7. After using zlib-1.2.8
instead, corruption did not happen anymore.

Reading the changes in zlib-1.2.8 at http://www.zlib.net I saw:

"Version 1.2.8 fixes a very rare bug in decompression.
All users are encouraged to upgrade immediately."

Can you check whether you use zlib-1.2.7 or zlib-1.2.8?

Dominique

[sqlite] Database corruption using zipvfs on android 64 bits (x86_64)

[Alexandre Mainville]

Hi,

I am experiencing database corruption using zipvfs on android 64
bits (x86_64).
The same program executing on 32 bits runs without problems.

The program creates a new database on every run and does a series of
inserts and updates (always the same series of operations).

Both programs produce the same SQLITE_DEBUG trace up until the point the 64
bit version outputs:
zipvfs database is corrupt. Line 1100 of [9d6c1880fb]
because of a zero size payload in a slot header.

I am waiting on a device to test behavior with arm64-v8a.

I am using sqlite 3.8.8.3 and zipvfs 3.8.8.

If anyone has experienced the same or has some insight on this, it would be
greatly appreciated.

Thanks

Alex

Here is the trace:

write page[1] to (456,127,0)
write page[2] to (589,20,17)
read page[1] from (24,127)
read page[1] from (24,127)
write page[1] to (632,126,0)
write page[2] to (764,33,4)
read page[1] from (24,126)
read page[1] from (24,126)
write page[1] to (807,509,0)
write page[3] to (1322,20,17)
write page[4] to (1365,20,17)
write page[5] to (1408,20,17)
write page[6] to (1451,20,17)
write page[7] to (1494,390,0)
write page[8] to (1890,383,0)
write page[9] to (2279,20,17)
write page[10] to (2322,20,17)
write page[11] to (2365,20,17)
write page[12] to (2408,20,17)
write page[13] to (2451,319,0)
write page[14] to (2776,20,17)
write page[15] to (2819,20,17)
write page[16] to (2862,20,17)
write page[17] to (2905,20,17)
write page[18] to (2948,379,0)
write page[19] to (,20,17)
write page[20] to (3376,20,17)
write page[21] to (3419,20,17)
write page[22] to (3462,20,17)
write page[23] to (3505,410,0)
write page[24] to (3921,20,17)
write page[25] to (3964,412,0)
write page[26] to (4382,20,17)
write page[27] to (4425,281,0)
write page[28] to (4712,20,17)
write page[29] to (4755,20,17)
write page[30] to (4798,384,0)
write page[31] to (5188,153,0)
write page[32] to (5347,20,17)
write page[33] to (5390,20,17)
write page[34] to (5433,20,17)
write page[35] to (5476,20,17)
write page[36] to (5519,427,0)
write page[37] to (5952,20,17)
write page[38] to (5995,20,17)
write page[39] to (6038,20,17)
write page[40] to (6081,20,17)
write page[41] to (6124,20,17)
write page[42] to (6167,363,0)
write page[43] to (6536,20,17)
write page[44] to (6579,20,17)
write page[45] to (6622,399,0)
write page[46] to (7027,20,17)
write page[47] to (7070,20,17)
write page[48] to (7113,20,17)
write page[49] to (7156,20,17)
write page[50] to (7199,370,0)
write page[51] to (7575,20,17)
write page[52] to (7618,20,17)
write page[53] to (7661,20,17)
write page[54] to (7704,385,0)
write page[55] to (8095,20,17)
write page[56] to (8138,20,17)
write page[57] to (8181,20,17)
write page[58] to (8224,20,17)
write page[59] to (8267,416,0)
write page[60] to (8689,20,17)
write page[61] to (8732,20,17)
write page[62] to (8775,20,17)
write page[63] to (8818,355,0)
write page[64] to (9179,20,17)
write page[65] to (9222,20,17)
write page[66] to (9265,20,17)
write page[67] to (9308,412,0)
write page[68] to (9726,20,17)
write page[69] to (9769,20,17)
write page[70] to (9812,432,0)
moving page[2] from (764,33,4) to (10250,33,0)
write page[71] to (10293,20,17)
write page[72] to (10336,20,17)
write page[73] to (10379,20,17)
write page[74] to (10422,20,17)
write page[75] to (10465,442,0)
moving page[1] from (807,509,0) to (10913,509,0)
write page[76] to (11428,20,17)
write page[77] to (11471,20,17)
write page[78] to (11514,275,0)
write page[79] to (11795,20,17)
write page[80] to (11838,20,17)
write page[81] to (11881,20,17)
read page[1] from (24,509)
read page[1] from (24,509)
read page[1] from (24,509)
write page[1] to (10913,509,0)
write page[11] to (11924,199,0)
write page[82] to (12129,806,0)
write page[83] to (12941,803,0)
write page[84] to (13750,804,0)
write page[85] to (14560,804,0)
read page[1] from (24,509)
read page[1] from (24,509)
read page[1] from (24,509)
write page[1] to (10913,509,0)
write page[11] to (15370,202,0)
write page[86] to (15578,804,0)
write page[87] to (16388,806,0)
write page[88] to (17200,808,0)
write page[89] to (18014,807,0)
write page[90] to (11924,199,0)
write page[91] to (18827,710,0)
read page[1] from (24,509)
read page[1] from (24,509)
read page[1] from (24,509)
write page[1] to (10913,509,0)
write page[11] to (19543,203,0)
write page[92] to (19752,808,0)
write page[93] to (20566,810,0)
write page[94] to (21382,808,0)
write page[95] to (22196,808,0)
write page[96] to (23010,633,0)
read page[1] from (24,509)
read page[1] from (24,509)
read page[1] from (24,509)
write page[1] to (10913,509,0)
write page[96] to (23649,751,0)
write page[97] to (24406,804,0)
write page[98] to (25216,808,0)
write page[99] to (26030,806,0)
write page[100] to (26842,805,0)
write page[101] to (27653,808,0)
read page[1] from (24,509)
read page[1] from (24,509)
read page[1] from (24,509)
write page[1] to (28467,510,0)
write page[11] to (19543,203,0)
write page[102] to (28983,806,0)
write page[103] to (29795,804,0)
write page[104] to (30605,807,0)
write page[105] to