Re: [sqlite] Getting occasional crashes with sqlite3_prepare_v2 and sqlite3_step in iOS app

[Rick Maddy]

Thanks. I'll hold off on checking the result of every sqlite3_bind_* for now.

It's unlikely I'm using a finalized statement otherwise I believe I would get a 
given crash much more often. The sqlite3_prepare_v2 statements that are failing 
only fail occasionally, not every time. But I'll double check.

Rick



On Jul 19, 2012, at 12:19 PM, Pavel Ivanov wrote:

> SQLite is written in such a way that no call to sqlite3_bind_* can
> cause memory corruption per se. It can return error if you are trying
> to bind wrong parameter and your app can corrupt memory if along with
> call to sqlite3_bind_* it changes some internal structures. But calls
> to sqlite3_* functions can cause corruption only if you try to work
> with already closed connection or with already finalized statement.
> 
> 
> Pavel

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Getting occasional crashes with sqlite3_prepare_v2 and sqlite3_step in iOS app

[Pavel Ivanov]

On Thu, Jul 19, 2012 at 2:07 PM, Rick Maddy  wrote:
> Didn't mean to imply that failing to check the return value resulted in 
> memory corruption. I was wondering if it was possible that one of the many 
> calls to sqlite3_bind_* in my code may actually be causing some memory 
> corruption. I can envision some possible buffer overflows associated with 
> those calls.

SQLite is written in such a way that no call to sqlite3_bind_* can
cause memory corruption per se. It can return error if you are trying
to bind wrong parameter and your app can corrupt memory if along with
call to sqlite3_bind_* it changes some internal structures. But calls
to sqlite3_* functions can cause corruption only if you try to work
with already closed connection or with already finalized statement.


Pavel
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Getting occasional crashes with sqlite3_prepare_v2 and sqlite3_step in iOS app

[Richard Hipp]

On Thu, Jul 19, 2012 at 2:07 PM, Rick Maddy  wrote:

> Didn't mean to imply that failing to check the return value resulted in
> memory corruption. I was wondering if it was possible that one of the many
> calls to sqlite3_bind_* in my code may actually be causing some memory
> corruption. I can envision some possible buffer overflows associated with
> those calls.
>
> None of the crashes are reproducible at all. I have personally never seen
> one of the crashes. I get one or two reports a week from my user base. This
> is what makes this such a pain. As we all know, it's very hard to diagnose
> a problem we can't replicate.
>

Suggest you enable error logging using
sqlite3_config(SQLITE_CONFIG_LOG,...) and see if that provides any clues.
I'm not hopeful that it will in this case, but it is at least worth the try.


>
> Rick
>
>
>
> On Jul 19, 2012, at 12:00 PM, Richard Hipp wrote:
>
> > On Thu, Jul 19, 2012 at 1:56 PM, Rick Maddy  wrote:
> >
> >> What about checking all the sqlite3_bind_* methods? Is it possible any
> of
> >> those could cause the problems I'm seeing?
> >>
> >
> > Being busing with other issues, I have not been following this thread
> > closely, so perhaps I have missed something, but
> >
> > Failure to check a return code from an SQLite API is not going to cause
> > memory corruption.  Memory corruption usually comes about when some other
> > subsystem in the application frees memory but continues to use it, then
> > SQLite allocates that memory and expects to have the memory to itself but
> > the other subsystem is still using too.
> >
> > The usual technique for finding that kind of problem is to run the
> > application in valgrind.  But I guess that isn't really an option for
> > you
> >
> > Is the problem reproducible for you in your lab at all?
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Getting occasional crashes with sqlite3_prepare_v2 and sqlite3_step in iOS app

[Rick Maddy]

Didn't mean to imply that failing to check the return value resulted in memory 
corruption. I was wondering if it was possible that one of the many calls to 
sqlite3_bind_* in my code may actually be causing some memory corruption. I can 
envision some possible buffer overflows associated with those calls.

None of the crashes are reproducible at all. I have personally never seen one 
of the crashes. I get one or two reports a week from my user base. This is what 
makes this such a pain. As we all know, it's very hard to diagnose a problem we 
can't replicate.

Rick



On Jul 19, 2012, at 12:00 PM, Richard Hipp wrote:

> On Thu, Jul 19, 2012 at 1:56 PM, Rick Maddy  wrote:
> 
>> What about checking all the sqlite3_bind_* methods? Is it possible any of
>> those could cause the problems I'm seeing?
>> 
> 
> Being busing with other issues, I have not been following this thread
> closely, so perhaps I have missed something, but
> 
> Failure to check a return code from an SQLite API is not going to cause
> memory corruption.  Memory corruption usually comes about when some other
> subsystem in the application frees memory but continues to use it, then
> SQLite allocates that memory and expects to have the memory to itself but
> the other subsystem is still using too.
> 
> The usual technique for finding that kind of problem is to run the
> application in valgrind.  But I guess that isn't really an option for
> you
> 
> Is the problem reproducible for you in your lab at all?

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Getting occasional crashes with sqlite3_prepare_v2 and sqlite3_step in iOS app

[Richard Hipp]

On Thu, Jul 19, 2012 at 1:56 PM, Rick Maddy  wrote:

> What about checking all the sqlite3_bind_* methods? Is it possible any of
> those could cause the problems I'm seeing?
>

Being busing with other issues, I have not been following this thread
closely, so perhaps I have missed something, but

Failure to check a return code from an SQLite API is not going to cause
memory corruption.  Memory corruption usually comes about when some other
subsystem in the application frees memory but continues to use it, then
SQLite allocates that memory and expects to have the memory to itself but
the other subsystem is still using too.

The usual technique for finding that kind of problem is to run the
application in valgrind.  But I guess that isn't really an option for
you

Is the problem reproducible for you in your lab at all?



>
> Rick
>
>
>
> On Jul 19, 2012, at 11:42 AM, Simon Slavin wrote:
>
> >
> > On 19 Jul 2012, at 6:03pm, Rick Maddy  wrote:
> >
> >> Thanks. Time to add checks to nearly 400 prepare and step statements.
> >
> > Sorry about that.  But you should be able to make a tiny test function
> which just tests the result and makes sure it's SQLITE_OK.
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Getting occasional crashes with sqlite3_prepare_v2 and sqlite3_step in iOS app

[Rick Maddy]

What about checking all the sqlite3_bind_* methods? Is it possible any of those 
could cause the problems I'm seeing?

Rick



On Jul 19, 2012, at 11:42 AM, Simon Slavin wrote:

> 
> On 19 Jul 2012, at 6:03pm, Rick Maddy  wrote:
> 
>> Thanks. Time to add checks to nearly 400 prepare and step statements.
> 
> Sorry about that.  But you should be able to make a tiny test function which 
> just tests the result and makes sure it's SQLITE_OK.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Getting occasional crashes with sqlite3_prepare_v2 and sqlite3_step in iOS app

[Rick Maddy]

I should know better. I've been coding in C-based languages for over 25 years. 
I guess I just never thought a prepare statement needed checking. I figured 
once the query worked all was good.

I already have a simple check method. I do check quite a few of the 'step' 
method result values, just not all, and none of the 'prepare' statements.

Thanks,
Rick



On Jul 19, 2012, at 11:42 AM, Simon Slavin wrote:

> 
> On 19 Jul 2012, at 6:03pm, Rick Maddy  wrote:
> 
>> Thanks. Time to add checks to nearly 400 prepare and step statements.
> 
> Sorry about that.  But you should be able to make a tiny test function which 
> just tests the result and makes sure it's SQLITE_OK.
> 
> You are nowhere near the only person who has had to add checking code long 
> after the coding was started.  One of the problems is that C allows you to 
> use a function which returns int as if it's a void function, without 
> generating a compiler warning or error.
> 
> Simon.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Getting occasional crashes with sqlite3_prepare_v2 and sqlite3_step in iOS app

[Simon Slavin]

On 19 Jul 2012, at 6:03pm, Rick Maddy  wrote:

> Thanks. Time to add checks to nearly 400 prepare and step statements.

Sorry about that.  But you should be able to make a tiny test function which 
just tests the result and makes sure it's SQLITE_OK.

You are nowhere near the only person who has had to add checking code long 
after the coding was started.  One of the problems is that C allows you to use 
a function which returns int as if it's a void function, without generating a 
compiler warning or error.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Getting occasional crashes with sqlite3_prepare_v2 and sqlite3_step in iOS app

[Rick Maddy]

Thanks but doesn't that code check to see if the database pointer has changed 
and not whether the memory it references has been corrupted? I guess that's a 
start though.

Rick



On Jul 19, 2012, at 11:02 AM, Black, Michael (IS) wrote:

> Buffer overflow issues can cause problems at seemingly random points in a 
> program.  I've worked on some really nasty ones before when no debugging was 
> available.
> 
> 
> 
> If dbRef is being corrupted then put a dbRef check in your program in every 
> function at beginning and end.
> 
> 
> 
> int dbRefCheck(sqlite3 *dbRef)
> 
> {
> 
>  static sqlite3 *mydbRef;
> 
>  if (dbRef == NULL) mydbRef = dbRef;
> 
>  if (dbRef != mydbRef) return true;
> 
>  return false;
> 
> }
> 
> 
> 
> if (dbRefCheck(dbRef)) {
> 
>  // error output -- with FILE and LINE macros if you have them.  Otherwise 
> some other unique id for each call so you know where it's happening.
> 
> }
> 

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Getting occasional crashes with sqlite3_prepare_v2 and sqlite3_step in iOS app

[Rick Maddy]

Thanks. Time to add checks to nearly 400 prepare and step statements.

Rick

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Getting occasional crashes with sqlite3_prepare_v2 and sqlite3_step in iOS app

[Black, Michael (IS)]

Buffer overflow issues can cause problems at seemingly random points in a 
program.  I've worked on some really nasty ones before when no debugging was 
available.



If dbRef is being corrupted then put a dbRef check in your program in every 
function at beginning and end.



int dbRefCheck(sqlite3 *dbRef)

{

  static sqlite3 *mydbRef;

  if (dbRef == NULL) mydbRef = dbRef;

  if (dbRef != mydbRef) return true;

  return false;

}



if (dbRefCheck(dbRef)) {

  // error output -- with FILE and LINE macros if you have them.  Otherwise 
some other unique id for each call so you know where it's happening.

}



Michael D. Black
Senior Scientist
Advanced Analytics Directorate
Advanced GEOINT Solutions Operating Unit
Northrop Grumman Information Systems


From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] on 
behalf of Rick Maddy [rma...@gmail.com]
Sent: Thursday, July 19, 2012 11:46 AM
To: General Discussion of SQLite Database
Subject: EXT :Re: [sqlite] Getting occasional crashes with sqlite3_prepare_v2 
and sqlite3_step in iOS app

Thanks. One of the users that reported one of the slqite3_prepare_v2 crashes 
also sent me their database. I just ran the 'pragma integrity_check' command on 
the database and it reported 'ok' though there was a journal file along with 
the database file. But there crash happened during a transaction so that should 
be OK.

I'll update my code to check the return value of all prepare statements.

With regard to memory issues, I don't see how this could be the case. One of 
the places the app crashes (on rare occasions) is:

sqlite3_stmt *load = nil;
sqlite3_prepare_v2(dbRef, "SELECT some_column FROM some_table WHERE some_id 
= ? ORDER BY display_order", -1, , nil);

How can memory issues creep into such a simple bit of code? Or could it be that 
the dbRef somehow became corrupted?

And come to think of it, how is checking the return value of the prepare 
statement going to help if the prepare statement crashes before returning?

Thanks,
Rick



On Jul 19, 2012, at 10:32 AM, Simon Slavin wrote:

>
> On 19 Jul 2012, at 5:09pm, Rick Maddy  wrote:
>
>> Exception Type:  SIGSEGV
>> Exception Codes: SEGV_ACCERR at 0x1a
>
> Those are segfaults.  They indicate an attempt by your app to access memory 
> which doesn't belong to it.  This often but not always happens because the 
> memory /did/ once belong to it but has now been released.
>
> First, run
>
> 
>
> to see if your database is corrupt.  Then put checks in your code for result 
> codes every time you call a function of the SQLite API.  For instance, 
> sqlite3_prepare_v2() returns a result code you should be checking.  Just 
> asserting that the int returned is SQLITE_OK will probably be enough.
>
> While putting the checks in, you'll probably notice whatever's causing your 
> bug.
>
> Simon.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Getting occasional crashes with sqlite3_prepare_v2 and sqlite3_step in iOS app

[Simon Slavin]

On 19 Jul 2012, at 5:46pm, Rick Maddy  wrote:

> With regard to memory issues, I don't see how this could be the case. One of 
> the places the app crashes (on rare occasions) is:
> 
>sqlite3_stmt *load = nil;
>sqlite3_prepare_v2(dbRef, "SELECT some_column FROM some_table WHERE 
> some_id = ? ORDER BY display_order", -1, , nil);
> 
> How can memory issues creep into such a simple bit of code? Or could it be 
> that the dbRef somehow became corrupted?

It is indeed the dbRef that I'm suspicious of.

> And come to think of it, how is checking the return value of the prepare 
> statement going to help if the prepare statement crashes before returning?

You need to do the checking on /every/ use of the API which returns a result 
code.  Including the one that opens the database in the first place.  I think 
it's one of the statements you execute /before/ the sqlite3_prepare_v2() that's 
going wrong.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Getting occasional crashes with sqlite3_prepare_v2 and sqlite3_step in iOS app

[Rick Maddy]

Thanks. One of the users that reported one of the slqite3_prepare_v2 crashes 
also sent me their database. I just ran the 'pragma integrity_check' command on 
the database and it reported 'ok' though there was a journal file along with 
the database file. But there crash happened during a transaction so that should 
be OK.

I'll update my code to check the return value of all prepare statements.

With regard to memory issues, I don't see how this could be the case. One of 
the places the app crashes (on rare occasions) is:

sqlite3_stmt *load = nil;
sqlite3_prepare_v2(dbRef, "SELECT some_column FROM some_table WHERE some_id 
= ? ORDER BY display_order", -1, , nil);

How can memory issues creep into such a simple bit of code? Or could it be that 
the dbRef somehow became corrupted?

And come to think of it, how is checking the return value of the prepare 
statement going to help if the prepare statement crashes before returning?

Thanks,
Rick



On Jul 19, 2012, at 10:32 AM, Simon Slavin wrote:

> 
> On 19 Jul 2012, at 5:09pm, Rick Maddy  wrote:
> 
>> Exception Type:  SIGSEGV
>> Exception Codes: SEGV_ACCERR at 0x1a
> 
> Those are segfaults.  They indicate an attempt by your app to access memory 
> which doesn't belong to it.  This often but not always happens because the 
> memory /did/ once belong to it but has now been released.
> 
> First, run
> 
> 
> 
> to see if your database is corrupt.  Then put checks in your code for result 
> codes every time you call a function of the SQLite API.  For instance, 
> sqlite3_prepare_v2() returns a result code you should be checking.  Just 
> asserting that the int returned is SQLITE_OK will probably be enough.
> 
> While putting the checks in, you'll probably notice whatever's causing your 
> bug.
> 
> Simon.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Getting occasional crashes with sqlite3_prepare_v2 and sqlite3_step in iOS app

[Simon Slavin]

On 19 Jul 2012, at 5:09pm, Rick Maddy  wrote:

> Exception Type:  SIGSEGV
> Exception Codes: SEGV_ACCERR at 0x1a

Those are segfaults.  They indicate an attempt by your app to access memory 
which doesn't belong to it.  This often but not always happens because the 
memory /did/ once belong to it but has now been released.

First, run



to see if your database is corrupt.  Then put checks in your code for result 
codes every time you call a function of the SQLite API.  For instance, 
sqlite3_prepare_v2() returns a result code you should be checking.  Just 
asserting that the int returned is SQLITE_OK will probably be enough.

While putting the checks in, you'll probably notice whatever's causing your bug.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Getting occasional crashes with sqlite3_prepare_v2 and sqlite3_step in iOS app

[Pavel Ivanov]

> For quite some time now I've been getting reports of crashes in my iOS app. 
> Specifically these are caused by crashes in sqlite3_prepare_v2 and 
> sqlite_step. The associated code works fine most of the time. So I'm looking 
> for thoughts on how to find and fix the problem since there seems to be no 
> pattern to when the rare crashes actually happen.

All the above means that your application corrupts memory somewhere
and it's not related to SQLite in any way. SQLite here is just an
unfortunate victim tripping over your memory corruption.

I don't know though if there are any tools on iOS for memory debugging.


Pavel


On Thu, Jul 19, 2012 at 12:09 PM, Rick Maddy  wrote:
> For quite some time now I've been getting reports of crashes in my iOS app. 
> Specifically these are caused by crashes in sqlite3_prepare_v2 and 
> sqlite_step. The associated code works fine most of the time. So I'm looking 
> for thoughts on how to find and fix the problem since there seems to be no 
> pattern to when the rare crashes actually happen.
>
> Here is some background information:
>
> All of the issues are from devices running iOS 5.1.1 which has sqlite version 
> 3.7.7. I open the database on app startup with the following call:
>
> if (sqlite3_open_v2([dbPath UTF8String], , 
> SQLITE_OPEN_READWRITE | SQLITE_OPEN_FULLMUTEX, NULL) == SQLITE_OK) {
> }
>
> I use the same 'dbRef' for every query performed in the app regardless of the 
> thread the query is used on.
>
> The output of 'sqlite3_threadsafe()' is 2 (SQLITE_CONFIG_SERIALIZED I 
> believe). So the database should be properly setup to work in a 
> multi-threaded environment.
>
> Every call to 'sqlite3_prepare_v2' that I use basically looks like this:
>
> sqlite3_stmt *query = nil;
> NSString *sql = @"some valid SQL";
> sqlite3_prepare_v2(dbRef, [sql UTF8String], -1, , nil);
>
>
> Here are a few examples of the stack traces from the crash reports:
>
> Sample 1:
>
> Exception Type:  SIGSEGV
> Exception Codes: SEGV_ACCERR at 0x1a
> Crashed Thread:  0
>
> Thread 0 Crashed:
> 0   libsqlite3.dylib0x34c783cc 0x34c43000 + 218060
> 1   libsqlite3.dylib0x34c5d3bd 0x34c43000 + 107453
> 2   libsqlite3.dylib0x34c4d5bb 0x34c43000 + 42427
> 3   libsqlite3.dylib0x34c4ab7b 0x34c43000 + 31611
> 4   libsqlite3.dylib0x34c4a367 0x34c43000 + 29543
> 5   libsqlite3.dylib0x34c49e95 0x34c43000 + 28309
> 6   libsqlite3.dylib0x34c49beb 0x34c43000 + 27627
> 7   libsqlite3.dylib0x34c80f97 sqlite3_prepare_v2 + 27
>
> ===
>
> Sample 2:
>
> Exception Type:  SIGSEGV
> Exception Codes: SEGV_ACCERR at 0x7
> Crashed Thread:  0
>
> Thread 0 Crashed:
> 0   libsqlite3.dylib0x34f38410 0x34f1e000 + 107536
> 1   libsqlite3.dylib0x34f2819f 0x34f1e000 + 41375
> 2   libsqlite3.dylib0x34f25b7b 0x34f1e000 + 31611
> 3   libsqlite3.dylib0x34f25367 0x34f1e000 + 29543
> 4   libsqlite3.dylib0x34f24e95 0x34f1e000 + 28309
> 5   libsqlite3.dylib0x34f24beb 0x34f1e000 + 27627
> 6   libsqlite3.dylib0x34f5bf97 sqlite3_prepare_v2 + 27
>
> ===
>
> Sample 3:
>
> Exception Type:  SIGSEGV
> Exception Codes: SEGV_ACCERR at 0x2
> Crashed Thread:  7
>
> Thread 7 Crashed:
> 0   libsqlite3.dylib0x34cf848c 0x34cc8000 + 197772
> 1   libsqlite3.dylib0x34cfd8f9 0x34cc8000 + 219385
> 2   libsqlite3.dylib0x34cf84ed 0x34cc8000 + 197869
> 3   libsqlite3.dylib0x34cf0bdd 0x34cc8000 + 166877
> 4   libsqlite3.dylib0x34cef6c9 sqlite3_step + 2105
>
> ===
>
> Thanks for any insights into these issues.
>
> Rick
>
>
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users