[sqlite] SQLite version 3.8.11 coming soon...

[Dominique Pellé]

On Wed, Jul 15, 2015 at 8:43 PM, Richard Hipp  wrote:
> On 7/15/15, T?r?k Edwin  wrote:
>> On 07/15/2015 08:34 PM, T?r?k Edwin wrote:
>>
>> In fact such detection could be added to fuzzcheck.c too (I didn't know that
>> you can detect presence of address sanitizer at compile time until now):
>
> It's a clever idea.  I'm not so sure I want to add this to fuzzcheck.
> I think I'd like to have the ability to get ASAN to fail using
> fuzzcheck.


Not only asan can be detected at runtime, but asan also
provides an API to ?poison? memory regions:

https://code.google.com/p/address-sanitizer/wiki/ManualPoisoning

It could be useful for SQLite. For example, SQLite could use the
API to poison:
- memory not used in MEMSYS5 buffer provided by
  sqlite3_config(SQLITE_CONFIG_HEAP, ...).
- unused lookaside buffers
- unused scratch buffers

Regards
Dominique

[sqlite] SQLite version 3.8.11 coming soon...

[Török Edwin]

On 07/15/2015 08:34 PM, T?r?k Edwin wrote:
> On 07/15/2015 08:17 PM, Richard Hipp wrote:
>> On 7/15/15, T?r?k Edwin  wrote:
>>> This might be just the test runner and not sqlite itself, I'm not sure:
>>>
>>> Time: capi2.test 25 ms
>>> =
>>> ==2330==ERROR: AddressSanitizer: heap-use-after-free on address
>>> 0x6180003b58dc at pc 0x7f5bb4894d49 bp 0x7ffd1e988d20 sp 0x7ffd1e988d18
>>> READ of size 4 at 0x6180003b58dc thread T0
>>> #0 0x7f5bb4894d48 in sqlite3SafetyCheckSickOrOk
>>> /home/edwin/skylable/sqlite/sqlite3.c:25082
>>> #1 0x7f5bb49b1174 in sqlite3Close
>>
>> I'm not able to reproduce this problem.
>>
>> Under valgrind, there are some tests in capi3c.test that deliberately
>> do use-after-free in order to test some safety mechanisms built into
>> SQLite.  All such test cases have "misuse" in their names.
>>
>> The safety mechanisms implemented by sqlite3SafetyCheckSickOrOk() try
>> to validate that an "sqlite3*" pointer passed into various APIs is
>> really a valid "sqlite3*" pointer that has not been closed by looking
>> at the "magic" integer that is part of the sqlite3 object.  If the
>> magic number is not correct, an error is raised.  The magic number is
>> cleared whenever a database connection is closed.
>>
>> The tests in this case close a database connection, which causes the
>> sqlite3 object to be deallocated.  Then they pass a pointer to that
>> now-deallocated object into some other API and verify that the error
>> is immediately detected and that the API returns SQLITE_MISUSE.
>>
>> Obviously, such things should never occur in a debugged applications.
>> No application should ever do anything that returns SQLITE_MISUSE.
>> The SQLITE_MISUSE return code is purely to alert programmers to coding
>> errors during development.

Yes, SQLITE_MISUSE is very useful (along with SQLITE_CONFIG_LOG) at catching 
errors early.

>>
>> And, also obviously, since this mechanism violates memory safety rules
>> (by reading the "magic" field in a block of memory that has been
>> freed), it will raise alarms with memory debuggers.
>>
>> The test/releasetest.tcl script runs many test cases, including some
>> that make use of -fsanitize on clang and that use valgrind.  Whenever
>> the test/releasetest.tcl script uses memory debuggers, it is always
>> careful to omit "misuse" tests.
> 
> Thanks for the excellent explanation as always.
> Apparently GCC's and Clang's -fsanitize=address define different macros [1], 
> so
> lets enhance that detection to include GCC's -fsanitize=address too
>  (perhaps the function should be renamed too as its no longer clang specific):
> 
> Index: src/test1.c
> ==
> --- src/test1.c
> +++ src/test1.c
> @@ -271,10 +271,13 @@
>int res = 0;
>  #if defined(__has_feature)
>  # if __has_feature(address_sanitizer)
>res = 1;
>  # endif
> +#endif
> +#ifdef __SANITIZE_ADDRESS__
> +  res = 1;
>  #endif
>if( res==0 && getenv("OMIT_MISUSE")!=0 ) res = 1;
>Tcl_SetObjResult(interp, Tcl_NewIntObj(res));
>return TCL_OK;
>  }
> 
> 
> [1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=56454#c5

I have run with these patches and my original configure line with GCC's address 
sanitizer and 'make test' doesn't crash with use-after-free anymore:
1 errors out of 140291 tests
Failures on these tests: shell1-5.0

shell1-5.0...
Error: failed with byte E0 mismatch

In fact such detection could be added to fuzzcheck.c too (I didn't know that 
you can detect presence of address sanitizer at compile time until now):

Index: test/fuzzcheck.c
==
--- test/fuzzcheck.c
+++ test/fuzzcheck.c
@@ -737,10 +737,19 @@
   char *zDbName = "";  /* Appreviated name of a source database */
   const char *zFailCode = 0;   /* Value of the TEST_FAILURE environment 
variable */
   int cellSzCkFlag = 0;/* --cell-size-check */
   int sqlFuzz = 0; /* True for SQL fuzz testing. False for DB fuzz 
*/
   int iTimeout = 120;  /* Default 120-second timeout */
+
+#if defined(__has_feature)
+# if __has_feature(address_sanitizer)
+  cellSzCkFlag = 1;
+# endif
+#endif
+#ifdef __SANITIZE_ADDRESS__
+  cellSzCkFlag = 1;
+#endif

   iBegin = timeOfDay();
 #ifdef __unix__
   signal(SIGALRM, timeoutHandler);
 #endif

[sqlite] SQLite version 3.8.11 coming soon...

[Török Edwin]

On 07/15/2015 08:17 PM, Richard Hipp wrote:
> On 7/15/15, T?r?k Edwin  wrote:
>> This might be just the test runner and not sqlite itself, I'm not sure:
>>
>> Time: capi2.test 25 ms
>> =
>> ==2330==ERROR: AddressSanitizer: heap-use-after-free on address
>> 0x6180003b58dc at pc 0x7f5bb4894d49 bp 0x7ffd1e988d20 sp 0x7ffd1e988d18
>> READ of size 4 at 0x6180003b58dc thread T0
>> #0 0x7f5bb4894d48 in sqlite3SafetyCheckSickOrOk
>> /home/edwin/skylable/sqlite/sqlite3.c:25082
>> #1 0x7f5bb49b1174 in sqlite3Close
> 
> I'm not able to reproduce this problem.
> 
> Under valgrind, there are some tests in capi3c.test that deliberately
> do use-after-free in order to test some safety mechanisms built into
> SQLite.  All such test cases have "misuse" in their names.
> 
> The safety mechanisms implemented by sqlite3SafetyCheckSickOrOk() try
> to validate that an "sqlite3*" pointer passed into various APIs is
> really a valid "sqlite3*" pointer that has not been closed by looking
> at the "magic" integer that is part of the sqlite3 object.  If the
> magic number is not correct, an error is raised.  The magic number is
> cleared whenever a database connection is closed.
> 
> The tests in this case close a database connection, which causes the
> sqlite3 object to be deallocated.  Then they pass a pointer to that
> now-deallocated object into some other API and verify that the error
> is immediately detected and that the API returns SQLITE_MISUSE.
> 
> Obviously, such things should never occur in a debugged applications.
> No application should ever do anything that returns SQLITE_MISUSE.
> The SQLITE_MISUSE return code is purely to alert programmers to coding
> errors during development.
> 
> And, also obviously, since this mechanism violates memory safety rules
> (by reading the "magic" field in a block of memory that has been
> freed), it will raise alarms with memory debuggers.
> 
> The test/releasetest.tcl script runs many test cases, including some
> that make use of -fsanitize on clang and that use valgrind.  Whenever
> the test/releasetest.tcl script uses memory debuggers, it is always
> careful to omit "misuse" tests.

Thanks for the excellent explanation as always.
Apparently GCC's and Clang's -fsanitize=address define different macros [1], so
lets enhance that detection to include GCC's -fsanitize=address too
 (perhaps the function should be renamed too as its no longer clang specific):

Index: src/test1.c
==
--- src/test1.c
+++ src/test1.c
@@ -271,10 +271,13 @@
   int res = 0;
 #if defined(__has_feature)
 # if __has_feature(address_sanitizer)
   res = 1;
 # endif
+#endif
+#ifdef __SANITIZE_ADDRESS__
+  res = 1;
 #endif
   if( res==0 && getenv("OMIT_MISUSE")!=0 ) res = 1;
   Tcl_SetObjResult(interp, Tcl_NewIntObj(res));
   return TCL_OK;
 }


[1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=56454#c5

Best regards,
--Edwin

[sqlite] SQLite version 3.8.11 coming soon...

[Török Edwin]

On 07/15/2015 06:59 PM, Richard Hipp wrote:
> On 7/15/15, T?r?k Edwin  wrote:
>>>
>>> 1) unknown-crash (might be due to some alignment requirements in asan):
>>
>> Build with clang 3.4 shows a heap-use-after-free instead of unknown-crash,
>> and building a normal (just ./configure) executable with GCC and running
>> under valgrind shows an invalid read too, so this
>> does seem to be a real bug after all:
> 
> This is not a heap-use-after-free.  This is a read-past-end-of-buffer.
> (Probably clang is confused because the buffer we are reading off the
> end of is immediately followed by another allocation that was
> previously freed.)
> 
> The is a *read* off the end of a buffer only - not a write.  And it
> only comes up on certain types of very obscure database file
> corruption.  Basically, you have to engineer the database corruption
> to make this happen - it will never occur by accident.
> 
> SQLite can be made to detect the database corruption prior to the
> buffer over-read by setting:
> 
>  PRAGMA cell_size_check=ON;

Thanks, do I still need to turn this on if I run 'PRAGMA integrity_check'?

> 
> See https://www.sqlite.org/draft/pragma.html#pragma_cell_size_check
> for additional information on the cell_size_check pragma.  Turning
> cell_size_check on will always prevent the buffer overread, but it
> also involves a noticeable performance hit.  So it is off by default,
> since the vast majority of the billions of SQLite instances out there
> will never encounter a maliciously corrupted database file, and even
> if they do the worst that can happen is a buffer over-*read* not an
> over-write, and hence is not a security concern.
> 
> That said, applications that open and read SQLite database files
> received from untrusted sources, might want to set cell_size_check=ON
> out of an abundance of caution.
> 
> Give fuzzcheck the --cell-size-check command-line option to engage the
> cell_size_check pragma, in order to prevent problems being reported by
> -fsanitize or by valgrind.  You will notice that we already do the
> same in the "valgrindfuzz" target of the makefile:
> https://www.sqlite.org/src/artifact/6e8af213?ln=1050

Indeed fuzzcheck with --cell-size-check finishes successfully now, thanks for 
the detailed explanation.
Could you add a short comment in the Makefile explaining that cell-size-check 
is needed if you build sanitizers/memory debugging tools?

Thanks,
--Edwin

[sqlite] SQLite version 3.8.11 coming soon...

[Simon Slavin]

On 15 Jul 2015, at 5:28pm, T?r?k Edwin  wrote:

> Thanks, do I still need to turn this on if I run 'PRAGMA integrity_check'?

You don't need to turn it on at all.  It's there only to get rid of a warning 
from clang and a couple of other things.

Simon.

[sqlite] SQLite version 3.8.11 coming soon...

[Török Edwin]

On 07/15/2015 06:19 PM, Richard Hipp wrote:
> On 7/15/15, T?r?k Edwin  wrote:
>> On 07/15/2015 12:05 AM, Richard Hipp wrote:
>>
>> I've run the address and undefined behaviour sanitizer (+ usual hardening
>> and bug finding flags from Debian) from GCC 4.9.2 on Debian Jessie on this
>> fossil checkout: a73d7128fbca8dde5e90bd46ee915e39ae07dd1f 2015-07-14
>> 22:43:37 UTC
>> (the snapshots tarballs don't seem to include the tests).
>>
>> I found some issues, but they look more like bugs in the sanitizer or the
>> test runner than bugs in sqlite, but I'm posting it here just to
>> double-check:
>>
>> $ ./configure CFLAGS="-g -O2 -Werror=array-bounds -Werror=clobbered
>> -Werror=volatile-register-var -Werror=implicit-function-declaration -fPIE
>> -fstack-protector-strong -Wformat -Werror=format-security -fsanitize=address
>> -fsanitize=undefined -fno-omit-frame-pointer" CPPFLAGS="-D_FORTIFY_SOURCE=2"
>> LDFLAGS="-fPIE -pie -Wl,-z,relro -Wl,-z,now -fsanitize=address
>> -fsanitize=undefined -pthread" --enable-debug --enable-threadsafe
> 
> I guess you need GCC 4.9.x for this, because when I try the above
> using Ubuntu 14.04LTS (Gcc 4.8.4) I get:
> 
> checking whether the C compiler works... no
> 

Probably due to -fstack-protector-strong (which is not relevant for this bug), 
try this simplified cmdline instead:

$ ./configure CFLAGS="-g -O2 -fsanitize=address -fsanitize=undefined 
-fno-omit-frame-pointer" CPPFLAGS="-D_FORTIFY_SOURCE=2" 
LDFLAGS="-fsanitize=address -fsanitize=undefined -pthread" --enable-debug 
--enable-threadsafe
$ make fuzzcheck
$ ./fuzzcheck test/fuzzdata3.db

Should work with clang too (if you set CC=clang).

[sqlite] SQLite version 3.8.11 coming soon...

[Török Edwin]

On 07/15/2015 05:59 PM, T?r?k Edwin wrote:
> On 07/15/2015 12:05 AM, Richard Hipp wrote:
>> The plan is to release SQLite version 3.8.11 on or about the end of July.
>>
>> The current code is passing all tests that we have run against it.
>> Some soak tests are still running.  There are quite a few
>> cross-platform tests (running on PPC, Sparc, etc) that have yet to be
>> started, but which should not offer any trouble.  The current code is
>> stable and perfectly appropriate for beta testing.
>>
>> Please test the latest SQLite snapshot in your products and report any
>> problems to this list, or directly to me.
> 
> 
> I've run the address and undefined behaviour sanitizer (+ usual hardening and 
> bug finding flags from Debian) from GCC 4.9.2 on Debian Jessie on this fossil 
> checkout: a73d7128fbca8dde5e90bd46ee915e39ae07dd1f 2015-07-14 22:43:37 UTC
> (the snapshots tarballs don't seem to include the tests).
> 
> I found some issues, but they look more like bugs in the sanitizer or the 
> test runner than bugs in sqlite, but I'm posting it here just to double-check:
> 
> $ ./configure CFLAGS="-g -O2 -Werror=array-bounds -Werror=clobbered 
> -Werror=volatile-register-var -Werror=implicit-function-declaration -fPIE 
> -fstack-protector-strong -Wformat -Werror=format-security -fsanitize=address 
> -fsanitize=undefined -fno-omit-frame-pointer" CPPFLAGS="-D_FORTIFY_SOURCE=2" 
> LDFLAGS="-fPIE -pie -Wl,-z,relro -Wl,-z,now -fsanitize=address 
> -fsanitize=undefined -pthread" --enable-debug --enable-threadsafe
> $ make clean
> $ make -j10
> $ make test -j10
> 
> 1) unknown-crash (might be due to some alignment requirements in asan):

Build with clang 3.4 shows a heap-use-after-free instead of unknown-crash, and 
building a normal (just ./configure) executable with GCC and running under 
valgrind shows an invalid read too, so this
does seem to be a real bug after all:

==14598== Invalid read of size 1
==14598==at 0x4A0C9CE: memcpy@@GLIBC_2.14 (vg_replace_strmem.c:915)
==14598==by 0x426804: rebuildPage (sqlite3.c:60141)
==14598==by 0x4421BA: editPage (sqlite3.c:60370)
==14598==by 0x4421BA: balance_nonroot (sqlite3.c:61299)
==14598==by 0x442888: balance (sqlite3.c:61547)
==14598==by 0x445051: sqlite3BtreeInsert (sqlite3.c:61737)
==14598==by 0x45A57D: sqlite3VdbeExec (sqlite3.c:76236)
==14598==by 0x461986: sqlite3Step (sqlite3.c:70639)
==14598==by 0x461986: sqlite3_step (sqlite3.c:70700)
==14598==by 0x407A51: runSql (fuzzcheck.c:617)
==14598==by 0x406C92: main (fuzzcheck.c:975)
==14598==  Address 0x4ca3a10 is 0 bytes after a block of size 512 alloc'd
==14598==at 0x4A07C20: malloc (vg_replace_malloc.c:296)
==14598==by 0x426B70: sqlite3MemMalloc (sqlite3.c:17235)
==14598==by 0x40EDAC: mallocWithAlarm (sqlite3.c:20909)
==14598==by 0x40EDAC: sqlite3Malloc (sqlite3.c:20940)
==14598==by 0x40FA0A: pcache1Alloc (sqlite3.c:40705)
==14598==by 0x412507: sqlite3PageMalloc (sqlite3.c:40843)
==14598==by 0x412507: sqlite3PagerSetPagesize (sqlite3.c:45907)
==14598==by 0x44796B: sqlite3BtreeOpen (sqlite3.c:56012)
==14598==by 0x4792A6: openDatabase (sqlite3.c:132083)
==14598==by 0x406C4B: main (fuzzcheck.c:965)
==14598== 
==14598== Invalid read of size 1
==14598==at 0x4A0C9C0: memcpy@@GLIBC_2.14 (vg_replace_strmem.c:915)
==14598==by 0x426804: rebuildPage (sqlite3.c:60141)
==14598==by 0x4421BA: editPage (sqlite3.c:60370)
==14598==by 0x4421BA: balance_nonroot (sqlite3.c:61299)
==14598==by 0x442888: balance (sqlite3.c:61547)
==14598==by 0x445051: sqlite3BtreeInsert (sqlite3.c:61737)
==14598==by 0x45A57D: sqlite3VdbeExec (sqlite3.c:76236)
==14598==by 0x461986: sqlite3Step (sqlite3.c:70639)
==14598==by 0x461986: sqlite3_step (sqlite3.c:70700)
==14598==by 0x407A51: runSql (fuzzcheck.c:617)
==14598==by 0x406C92: main (fuzzcheck.c:975)
==14598==  Address 0x4ca3a12 is 2 bytes after a block of size 512 alloc'd
==14598==at 0x4A07C20: malloc (vg_replace_malloc.c:296)
==14598==by 0x426B70: sqlite3MemMalloc (sqlite3.c:17235)
==14598==by 0x40EDAC: mallocWithAlarm (sqlite3.c:20909)
==14598==by 0x40EDAC: sqlite3Malloc (sqlite3.c:20940)
==14598==by 0x40FA0A: pcache1Alloc (sqlite3.c:40705)
==14598==by 0x412507: sqlite3PageMalloc (sqlite3.c:40843)
==14598==by 0x412507: sqlite3PagerSetPagesize (sqlite3.c:45907)
==14598==by 0x44796B: sqlite3BtreeOpen (sqlite3.c:56012)
==14598==by 0x4792A6: openDatabase (sqlite3.c:132083)
==14598==by 0x406C4B: main (fuzzcheck.c:965)


./fuzzcheck /home/edwin/skylable/sqlite/test/fuzzdata3.db
fuzzdata3.db: Database fuzz as of 2015-06-24
fuzzdata3.db: 0% 10%/home/edwin/skylable/sqlite/sqlite3.c:24327: runtime error: 
value 9e+323 is outside the range of representable values of type 'double'
/home/edwin/skylable/sqlite/sqlite3.c:24327: runtime error: value 9e+323 is 
outside the range of representable values of type 'double'
=

[sqlite] SQLite version 3.8.11 coming soon...

[Török Edwin]

On 07/15/2015 12:05 AM, Richard Hipp wrote:
> The plan is to release SQLite version 3.8.11 on or about the end of July.
> 
> The current code is passing all tests that we have run against it.
> Some soak tests are still running.  There are quite a few
> cross-platform tests (running on PPC, Sparc, etc) that have yet to be
> started, but which should not offer any trouble.  The current code is
> stable and perfectly appropriate for beta testing.
> 
> Please test the latest SQLite snapshot in your products and report any
> problems to this list, or directly to me.


I've run the address and undefined behaviour sanitizer (+ usual hardening and 
bug finding flags from Debian) from GCC 4.9.2 on Debian Jessie on this fossil 
checkout: a73d7128fbca8dde5e90bd46ee915e39ae07dd1f 2015-07-14 22:43:37 UTC
(the snapshots tarballs don't seem to include the tests).

I found some issues, but they look more like bugs in the sanitizer or the test 
runner than bugs in sqlite, but I'm posting it here just to double-check:

$ ./configure CFLAGS="-g -O2 -Werror=array-bounds -Werror=clobbered 
-Werror=volatile-register-var -Werror=implicit-function-declaration -fPIE 
-fstack-protector-strong -Wformat -Werror=format-security -fsanitize=address 
-fsanitize=undefined -fno-omit-frame-pointer" CPPFLAGS="-D_FORTIFY_SOURCE=2" 
LDFLAGS="-fPIE -pie -Wl,-z,relro -Wl,-z,now -fsanitize=address 
-fsanitize=undefined -pthread" --enable-debug --enable-threadsafe
$ make clean
$ make -j10
$ make test -j10

1) unknown-crash (might be due to some alignment requirements in asan):

fuzzdata3.db: Database fuzz as of 2015-06-24
fuzzdata3.db: 0% 
10%=
==1050==ERROR: AddressSanitizer: unknown-crash on address 0x615abb41 at pc 
0x7fa3dd350ec9 bp 0x7ffd7b8ec180 sp 0x7ffd7b8ec178
READ of size 385 at 0x615abb41 thread T0
#0 0x7fa3dd350ec8 in memcpy /usr/include/x86_64-linux-gnu/bits/string3.h:51
#1 0x7fa3dd350ec8 in rebuildPage /home/edwin/skylable/sqlite/sqlite3.c:60141
#2 0x7fa3dd3f28b3 in editPage /home/edwin/skylable/sqlite/sqlite3.c:60370
#3 0x7fa3dd3f28b3 in balance_nonroot 
/home/edwin/skylable/sqlite/sqlite3.c:61299
#4 0x7fa3dd3f486e in balance /home/edwin/skylable/sqlite/sqlite3.c:61547
#5 0x7fa3dd40842f in sqlite3BtreeInsert 
/home/edwin/skylable/sqlite/sqlite3.c:61737
#6 0x7fa3dd48c765 in sqlite3VdbeExec 
/home/edwin/skylable/sqlite/sqlite3.c:76236
#7 0x7fa3dd4c4746 in sqlite3Step /home/edwin/skylable/sqlite/sqlite3.c:70639
#8 0x7fa3dd4c4746 in sqlite3_step 
/home/edwin/skylable/sqlite/sqlite3.c:70700
#9 0x7fa3dd2665f1 in runSql /home/edwin/skylable/sqlite/test/fuzzcheck.c:617
#10 0x7fa3dd262bb6 in main /home/edwin/skylable/sqlite/test/fuzzcheck.c:975
#11 0x7fa3da929b44 in __libc_start_main 
(/lib/x86_64-linux-gnu/libc.so.6+0x3452621b44)
#12 0x7fa3dd264343 (/home/edwin/skylable/sqlite/fuzzcheck+0x3bc343)

0x615abb80 is located 0 bytes to the right of 512-byte region 
[0x615ab980,0x615abb80)
allocated by thread T0 here:
#0 0x7fa3dbdfc73f in malloc 
(/usr/lib/x86_64-linux-gnu/libasan.so.1+0x3f2205473f)
#1 0x7fa3dd351631 in sqlite3MemMalloc 
/home/edwin/skylable/sqlite/sqlite3.c:17235
#2 0x7fa3dd2d4e98 in mallocWithAlarm 
/home/edwin/skylable/sqlite/sqlite3.c:20909
#3 0x7fa3dd2d4e98 in sqlite3Malloc 
/home/edwin/skylable/sqlite/sqlite3.c:20940
#4 0x7fa3dd2ea741 in pcache1Alloc 
/home/edwin/skylable/sqlite/sqlite3.c:40705
#5 0x7fa3dd2eab62 in sqlite3PageMalloc 
/home/edwin/skylable/sqlite/sqlite3.c:40843
#6 0x7fa3dd2eab62 in sqlite3PagerSetPagesize 
/home/edwin/skylable/sqlite/sqlite3.c:45907
#7 0x7fa3dd4196d2 in sqlite3BtreeOpen 
/home/edwin/skylable/sqlite/sqlite3.c:56012
#8 0x7fa3dd52fe42 in openDatabase 
/home/edwin/skylable/sqlite/sqlite3.c:132083
#9 0x7fa3dd262b64 in main /home/edwin/skylable/sqlite/test/fuzzcheck.c:965
#10 0x7fa3da929b44 in __libc_start_main 
(/lib/x86_64-linux-gnu/libc.so.6+0x3452621b44)

SUMMARY: AddressSanitizer: unknown-crash 
/usr/include/x86_64-linux-gnu/bits/string3.h:51 memcpy
Shadow bytes around the buggy address:
  0x0c2a8000d710: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2a8000d720: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2a8000d730: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2a8000d740: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2a8000d750: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c2a8000d760: 00 00 00 00 00 00 00 00[00]00 00 00 00 00 00 00
  0x0c2a8000d770: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2a8000d780: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2a8000d790: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2a8000d7a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2a8000d7b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:   00
  Partially addressab

[sqlite] SQLite version 3.8.11 coming soon...

[Richard Hipp]

On 7/15/15, T?r?k Edwin  wrote:
> On 07/15/2015 08:34 PM, T?r?k Edwin wrote:
>
> In fact such detection could be added to fuzzcheck.c too (I didn't know that
> you can detect presence of address sanitizer at compile time until now):

It's a clever idea.  I'm not so sure I want to add this to fuzzcheck.
I think I'd like to have the ability to get ASAN to fail using
fuzzcheck.

>
> Index: test/fuzzcheck.c
> ==
> --- test/fuzzcheck.c
> +++ test/fuzzcheck.c
> @@ -737,10 +737,19 @@
>char *zDbName = "";  /* Appreviated name of a source database */
>const char *zFailCode = 0;   /* Value of the TEST_FAILURE environment
> variable */
>int cellSzCkFlag = 0;/* --cell-size-check */
>int sqlFuzz = 0; /* True for SQL fuzz testing. False for DB
> fuzz */
>int iTimeout = 120;  /* Default 120-second timeout */
> +
> +#if defined(__has_feature)
> +# if __has_feature(address_sanitizer)
> +  cellSzCkFlag = 1;
> +# endif
> +#endif
> +#ifdef __SANITIZE_ADDRESS__
> +  cellSzCkFlag = 1;
> +#endif
>
>iBegin = timeOfDay();
>  #ifdef __unix__
>signal(SIGALRM, timeoutHandler);
>  #endif
>
>
>
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>


-- 
D. Richard Hipp
drh at sqlite.org

[sqlite] SQLite version 3.8.11 coming soon...

[Richard Hipp]

On 7/15/15, T?r?k Edwin  wrote:
>
> I've attached a patch that fixes these warnings from GCC
>

No attachments on this mailing list.  Can you send the patch via
direct email to me?

-- 
D. Richard Hipp
drh at sqlite.org

[sqlite] SQLite version 3.8.11 coming soon...

[Richard Hipp]

On 7/15/15, T?r?k Edwin  wrote:
> This might be just the test runner and not sqlite itself, I'm not sure:
>
> Time: capi2.test 25 ms
> =
> ==2330==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x6180003b58dc at pc 0x7f5bb4894d49 bp 0x7ffd1e988d20 sp 0x7ffd1e988d18
> READ of size 4 at 0x6180003b58dc thread T0
> #0 0x7f5bb4894d48 in sqlite3SafetyCheckSickOrOk
> /home/edwin/skylable/sqlite/sqlite3.c:25082
> #1 0x7f5bb49b1174 in sqlite3Close

I'm not able to reproduce this problem.

Under valgrind, there are some tests in capi3c.test that deliberately
do use-after-free in order to test some safety mechanisms built into
SQLite.  All such test cases have "misuse" in their names.

The safety mechanisms implemented by sqlite3SafetyCheckSickOrOk() try
to validate that an "sqlite3*" pointer passed into various APIs is
really a valid "sqlite3*" pointer that has not been closed by looking
at the "magic" integer that is part of the sqlite3 object.  If the
magic number is not correct, an error is raised.  The magic number is
cleared whenever a database connection is closed.

The tests in this case close a database connection, which causes the
sqlite3 object to be deallocated.  Then they pass a pointer to that
now-deallocated object into some other API and verify that the error
is immediately detected and that the API returns SQLITE_MISUSE.

Obviously, such things should never occur in a debugged applications.
No application should ever do anything that returns SQLITE_MISUSE.
The SQLITE_MISUSE return code is purely to alert programmers to coding
errors during development.

And, also obviously, since this mechanism violates memory safety rules
(by reading the "magic" field in a block of memory that has been
freed), it will raise alarms with memory debuggers.

The test/releasetest.tcl script runs many test cases, including some
that make use of -fsanitize on clang and that use valgrind.  Whenever
the test/releasetest.tcl script uses memory debuggers, it is always
careful to omit "misuse" tests.
-- 
D. Richard Hipp
drh at sqlite.org

[sqlite] SQLite version 3.8.11 coming soon...

[Richard Hipp]

On 7/15/15, T?r?k Edwin  wrote:
>>
>> 1) unknown-crash (might be due to some alignment requirements in asan):
>
> Build with clang 3.4 shows a heap-use-after-free instead of unknown-crash,
> and building a normal (just ./configure) executable with GCC and running
> under valgrind shows an invalid read too, so this
> does seem to be a real bug after all:

This is not a heap-use-after-free.  This is a read-past-end-of-buffer.
(Probably clang is confused because the buffer we are reading off the
end of is immediately followed by another allocation that was
previously freed.)

The is a *read* off the end of a buffer only - not a write.  And it
only comes up on certain types of very obscure database file
corruption.  Basically, you have to engineer the database corruption
to make this happen - it will never occur by accident.

SQLite can be made to detect the database corruption prior to the
buffer over-read by setting:

 PRAGMA cell_size_check=ON;

See https://www.sqlite.org/draft/pragma.html#pragma_cell_size_check
for additional information on the cell_size_check pragma.  Turning
cell_size_check on will always prevent the buffer overread, but it
also involves a noticeable performance hit.  So it is off by default,
since the vast majority of the billions of SQLite instances out there
will never encounter a maliciously corrupted database file, and even
if they do the worst that can happen is a buffer over-*read* not an
over-write, and hence is not a security concern.

That said, applications that open and read SQLite database files
received from untrusted sources, might want to set cell_size_check=ON
out of an abundance of caution.

Give fuzzcheck the --cell-size-check command-line option to engage the
cell_size_check pragma, in order to prevent problems being reported by
-fsanitize or by valgrind.  You will notice that we already do the
same in the "valgrindfuzz" target of the makefile:
https://www.sqlite.org/src/artifact/6e8af213?ln=1050

>
> ==14598== Invalid read of size 1
> ==14598==at 0x4A0C9CE: memcpy@@GLIBC_2.14 (vg_replace_strmem.c:915)
> ==14598==by 0x426804: rebuildPage (sqlite3.c:60141)
> ==14598==by 0x4421BA: editPage (sqlite3.c:60370)
> ==14598==by 0x4421BA: balance_nonroot (sqlite3.c:61299)
> ==14598==by 0x442888: balance (sqlite3.c:61547)
> ==14598==by 0x445051: sqlite3BtreeInsert (sqlite3.c:61737)
> ==14598==by 0x45A57D: sqlite3VdbeExec (sqlite3.c:76236)
> ==14598==by 0x461986: sqlite3Step (sqlite3.c:70639)
> ==14598==by 0x461986: sqlite3_step (sqlite3.c:70700)
> ==14598==by 0x407A51: runSql (fuzzcheck.c:617)
> ==14598==by 0x406C92: main (fuzzcheck.c:975)
> ==14598==  Address 0x4ca3a10 is 0 bytes after a block of size 512 alloc'd
> ==14598==at 0x4A07C20: malloc (vg_replace_malloc.c:296)
> ==14598==by 0x426B70: sqlite3MemMalloc (sqlite3.c:17235)
> ==14598==by 0x40EDAC: mallocWithAlarm (sqlite3.c:20909)
> ==14598==by 0x40EDAC: sqlite3Malloc (sqlite3.c:20940)
> ==14598==by 0x40FA0A: pcache1Alloc (sqlite3.c:40705)
> ==14598==by 0x412507: sqlite3PageMalloc (sqlite3.c:40843)
> ==14598==by 0x412507: sqlite3PagerSetPagesize (sqlite3.c:45907)
> ==14598==by 0x44796B: sqlite3BtreeOpen (sqlite3.c:56012)
> ==14598==by 0x4792A6: openDatabase (sqlite3.c:132083)
> ==14598==by 0x406C4B: main (fuzzcheck.c:965)
> ==14598==
> ==14598== Invalid read of size 1
> ==14598==at 0x4A0C9C0: memcpy@@GLIBC_2.14 (vg_replace_strmem.c:915)
> ==14598==by 0x426804: rebuildPage (sqlite3.c:60141)
> ==14598==by 0x4421BA: editPage (sqlite3.c:60370)
> ==14598==by 0x4421BA: balance_nonroot (sqlite3.c:61299)
> ==14598==by 0x442888: balance (sqlite3.c:61547)
> ==14598==by 0x445051: sqlite3BtreeInsert (sqlite3.c:61737)
> ==14598==by 0x45A57D: sqlite3VdbeExec (sqlite3.c:76236)
> ==14598==by 0x461986: sqlite3Step (sqlite3.c:70639)
> ==14598==by 0x461986: sqlite3_step (sqlite3.c:70700)
> ==14598==by 0x407A51: runSql (fuzzcheck.c:617)
> ==14598==by 0x406C92: main (fuzzcheck.c:975)
> ==14598==  Address 0x4ca3a12 is 2 bytes after a block of size 512 alloc'd
> ==14598==at 0x4A07C20: malloc (vg_replace_malloc.c:296)
> ==14598==by 0x426B70: sqlite3MemMalloc (sqlite3.c:17235)
> ==14598==by 0x40EDAC: mallocWithAlarm (sqlite3.c:20909)
> ==14598==by 0x40EDAC: sqlite3Malloc (sqlite3.c:20940)
> ==14598==by 0x40FA0A: pcache1Alloc (sqlite3.c:40705)
> ==14598==by 0x412507: sqlite3PageMalloc (sqlite3.c:40843)
> ==14598==by 0x412507: sqlite3PagerSetPagesize (sqlite3.c:45907)
> ==14598==by 0x44796B: sqlite3BtreeOpen (sqlite3.c:56012)
> ==14598==by 0x4792A6: openDatabase (sqlite3.c:132083)
> ==14598==by 0x406C4B: main (fuzzcheck.c:965)
>
>
> ./fuzzcheck /home/edwin/skylable/sqlite/test/fuzzdata3.db
> fuzzdata3.db: Database fuzz as of 2015-06-24
> fuzzdata3.db: 0% 10%/home/edwin/skylable/sqlite/sqlite3.c:24327: runtime
> error: value 9e+323 is outside the range of representable values of typ

[sqlite] SQLite version 3.8.11 coming soon...

[Richard Hipp]

On 7/15/15, T?r?k Edwin  wrote:
> On 07/15/2015 12:05 AM, Richard Hipp wrote:
>
> I've run the address and undefined behaviour sanitizer (+ usual hardening
> and bug finding flags from Debian) from GCC 4.9.2 on Debian Jessie on this
> fossil checkout: a73d7128fbca8dde5e90bd46ee915e39ae07dd1f 2015-07-14
> 22:43:37 UTC
> (the snapshots tarballs don't seem to include the tests).
>
> I found some issues, but they look more like bugs in the sanitizer or the
> test runner than bugs in sqlite, but I'm posting it here just to
> double-check:
>
> $ ./configure CFLAGS="-g -O2 -Werror=array-bounds -Werror=clobbered
> -Werror=volatile-register-var -Werror=implicit-function-declaration -fPIE
> -fstack-protector-strong -Wformat -Werror=format-security -fsanitize=address
> -fsanitize=undefined -fno-omit-frame-pointer" CPPFLAGS="-D_FORTIFY_SOURCE=2"
> LDFLAGS="-fPIE -pie -Wl,-z,relro -Wl,-z,now -fsanitize=address
> -fsanitize=undefined -pthread" --enable-debug --enable-threadsafe

I guess you need GCC 4.9.x for this, because when I try the above
using Ubuntu 14.04LTS (Gcc 4.8.4) I get:

checking whether the C compiler works... no

-- 
D. Richard Hipp
drh at sqlite.org

[sqlite] SQLite version 3.8.11 coming soon...

[Juan Pablo García Coello]

Will be one day support for storagefiles for Windows 10 to allow paths that are 
not inside the appfolder?

Sent from my Windows Phone

From: Richard Hipp<mailto:d...@sqlite.org>
Sent: ?14/?07/?2015 22:05
To: General Discussion of SQLite Database<mailto:sqlite-users at 
mailinglists.sqlite.org>; sqlite-dev<mailto:sqlite-dev at 
mailinglists.sqlite.org>
Subject: [sqlite] SQLite version 3.8.11 coming soon...

The plan is to release SQLite version 3.8.11 on or about the end of July.

The current code is passing all tests that we have run against it.
Some soak tests are still running.  There are quite a few
cross-platform tests (running on PPC, Sparc, etc) that have yet to be
started, but which should not offer any trouble.  The current code is
stable and perfectly appropriate for beta testing.

Please test the latest SQLite snapshot in your products and report any
problems to this list, or directly to me.

A summary of changes can be seen at
https://www.sqlite.org/draft/releaselog/3_8_11.html

A snapshot of the latest code can be downloaded from
https://www.sqlite.org/download.html

The release status board is online at
https://www.sqlite.org/checklists/3081100/index - when the status
board goes all-green, we will cut the release.  That will probably not
happen before 2015-07-31.

More testing and documentation work will continue over the next two
weeks, so periodically recheck the links above for updates.

--
D. Richard Hipp
drh at sqlite.org
___
sqlite-users mailing list
sqlite-users at mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] SQLite version 3.8.11 coming soon...

[Richard Hipp]

The plan is to release SQLite version 3.8.11 on or about the end of July.

The current code is passing all tests that we have run against it.
Some soak tests are still running.  There are quite a few
cross-platform tests (running on PPC, Sparc, etc) that have yet to be
started, but which should not offer any trouble.  The current code is
stable and perfectly appropriate for beta testing.

Please test the latest SQLite snapshot in your products and report any
problems to this list, or directly to me.

A summary of changes can be seen at
https://www.sqlite.org/draft/releaselog/3_8_11.html

A snapshot of the latest code can be downloaded from
https://www.sqlite.org/download.html

The release status board is online at
https://www.sqlite.org/checklists/3081100/index - when the status
board goes all-green, we will cut the release.  That will probably not
happen before 2015-07-31.

More testing and documentation work will continue over the next two
weeks, so periodically recheck the links above for updates.

-- 
D. Richard Hipp
drh at sqlite.org