[sqlite] [Regression?] FTS function crashes since 3.7.7
Hi everybody, While testing some database upgrade function of mine, I ran into a segmentation fault. I started looking at the possible cause in my code, but I soon realized the crash would only happen using SQLite 3.7.7 or 3.7.8. If I use 3.7.6 or lower, things go safely. Since I am not familiar with SQLite's internals, I have run the thing under Valgrind and GDB to get some backtraces. Here is Valgrind's report: ==27928== Invalid read of size 4 ==27928==at 0x646251: sqlite3Fts3PendingTermsFlush (sqlite3.c:124996) ==27928==by 0x63A2A8: fts3SyncMethod (sqlite3.c:117025) ==27928==by 0x63A983: fts3SavepointMethod (sqlite3.c:117332) ==27928==by 0x6239C5: sqlite3VtabSavepoint (sqlite3.c:100182) ==27928==by 0x5E9F07: sqlite3VdbeExec (sqlite3.c:66431) ==27928==by 0x5E40D8: sqlite3Step (sqlite3.c:61954) ==27928==by 0x5E42B9: sqlite3_step (sqlite3.c:62027) ==27928==by 0x5B3BC3: SQLite::Query::exec() (Query.cc:179) ==27928==by 0x5B3D80: SQLite::Query::exec(QString const) (Query.cc:230) ==27928==by 0x590DF5: update8to9(SQLite::Query) (Database.cc:206) ==27928==by 0x59128A: Database::updateUserDB(int) (Database.cc:262) ==27928==by 0x5914CD: Database::checkUserDB() (Database.cc:297) ==27928== Address 0x150 is not stack'd, malloc'd or (recently) free'd ==27928== ==27928== ==27928== Process terminating with default action of signal 11 (SIGSEGV) ==27928== Access not within mapped region at address 0x150 ==27928==at 0x646251: sqlite3Fts3PendingTermsFlush (sqlite3.c:124996) ==27928==by 0x63A2A8: fts3SyncMethod (sqlite3.c:117025) ==27928==by 0x63A983: fts3SavepointMethod (sqlite3.c:117332) ==27928==by 0x6239C5: sqlite3VtabSavepoint (sqlite3.c:100182) ==27928==by 0x5E9F07: sqlite3VdbeExec (sqlite3.c:66431) ==27928==by 0x5E40D8: sqlite3Step (sqlite3.c:61954) ==27928==by 0x5E42B9: sqlite3_step (sqlite3.c:62027) ==27928==by 0x5B3BC3: SQLite::Query::exec() (Query.cc:179) ==27928==by 0x5B3D80: SQLite::Query::exec(QString const) (Query.cc:230) ==27928==by 0x590DF5: update8to9(SQLite::Query) (Database.cc:206) ==27928==by 0x59128A: Database::updateUserDB(int) (Database.cc:262) ==27928==by 0x5914CD: Database::checkUserDB() (Database.cc:297) And here is the GDB backtrace (using 3.7.8): #0 0x00646251 in sqlite3Fts3PendingTermsFlush (p=0x0) at sqlite3.c:124996 #1 0x0063a2a9 in fts3SyncMethod (pVtab=0x0) at sqlite3.c:117025 #2 0x0063a984 in fts3SavepointMethod (pVtab=0x0, iSavepoint=0) at sqlite3.c:117332 #3 0x006239c6 in sqlite3VtabSavepoint (db=0x9bf4a8, op=0, iSavepoint=0) at sqlite3.c:100182 #4 0x005e9f08 in sqlite3VdbeExec (p=0x9b9ac8) at sqlite3.c:66431 #5 0x005e40d9 in sqlite3Step (p=0x9b9ac8) at sqlite3.c:61954 #6 0x005e42ba in sqlite3_step (pStmt=0x9b9ac8) at sqlite3.c:62027 Here the faulty is pVtab which should not be NULL from fts3SavepointMethod, since it is unconditionally dereferenced. fts3SavepointMethod is called through a function pointer from line 100182 which reads the following: rc = xMethod(db-aVTrans[i]-pVtab, iSavepoint); That's about all the information I can get for now. Compiling with SQLITE_DEBUG=1 does not give anything more - it just crashes earlier in fts3SavepointMethod because of an assert that dereferences pVtab. I do not understand SQLite's code well, but it seems to me that a virtual table function should not be called with a NULL virtual table pointer anyway. I wish I could send a code sample that reproduces the issue, but unfortunately the case is rather big and complex. I will try to come with a minimal program though. If there is anything more I can do to help debugging, please let me know. Thanks, Alex. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] [Regression?] FTS function crashes since 3.7.7
On 10/29/2011 03:00 PM, Alexandre Courbot wrote: Hi everybody, While testing some database upgrade function of mine, I ran into a segmentation fault. I started looking at the possible cause in my code, but I soon realized the crash would only happen using SQLite 3.7.7 or 3.7.8. If I use 3.7.6 or lower, things go safely. Since I am not familiar with SQLite's internals, I have run the thing under Valgrind and GDB to get some backtraces. Here is Valgrind's report: And here is the GDB backtrace (using 3.7.8): #0 0x00646251 in sqlite3Fts3PendingTermsFlush (p=0x0) at sqlite3.c:124996 #1 0x0063a2a9 in fts3SyncMethod (pVtab=0x0) at sqlite3.c:117025 #2 0x0063a984 in fts3SavepointMethod (pVtab=0x0, iSavepoint=0) at sqlite3.c:117332 #3 0x006239c6 in sqlite3VtabSavepoint (db=0x9bf4a8, op=0, iSavepoint=0) at sqlite3.c:100182 #4 0x005e9f08 in sqlite3VdbeExec (p=0x9b9ac8) at sqlite3.c:66431 #5 0x005e40d9 in sqlite3Step (p=0x9b9ac8) at sqlite3.c:61954 #6 0x005e42ba in sqlite3_step (pStmt=0x9b9ac8) at sqlite3.c:62027 Thanks for reporting this. Found one problem: Ticket: http://www.sqlite.org/src/info/48f299634a Fix:http://www.sqlite.org/src/info/3565fcf898 Please follow up if you try this fix and you still get the crash. Dan. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Using SQLite on Windows 64bit
Yes, compiling to 32-bit is nice. It is like dancing at the tunes of Earth Wind and Fire, h, those were the days. Like the old good songs, the 32bit apps are a good memory, many people like it, but they are a thing of the past. We are developers and anyone of us who stays behind, is left behind. Not having the option of SQLite-64 is a major drawback and will leave us behind. So, come on folks, pull up your socks and let's get the SQLite apps into the 21st century, shall we? ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] [Regression?] FTS function crashes since 3.7.7
Dan, On Sat, Oct 29, 2011 at 8:48 PM, Dan Kennedy danielk1...@gmail.com wrote: Found one problem: Ticket: http://www.sqlite.org/src/info/48f299634a Fix: http://www.sqlite.org/src/info/3565fcf898 Please follow up if you try this fix and you still get the crash. Thanks for pointing me to this. Indeed, this solves my problem. Great to see this is already fixed - guess I will just have to blacklist SQLite 3.7.7 and 3.7.8. Thanks! Alex. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] [Regression?] FTS function crashes since 3.7.7
By the way, can we expect a point release (3.7.8.1?) that includes this fix soon? It seems rather serious to me, and the use case should not be so seldom. Alex. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Using SQLite on Windows 64bit
I have 64-bit binary package for my OS (kindly maintained by guy named Panu Matilainen, outside of the sqltie.org core team). I have installed this package after the command: yum install sqlite This currently installs 3.7.5 for me. If I want to test/use different version I type: fossill clone ..., fossil open version-3.7.8, ./configure, make, and I have 3.7.8 .so and shell. You are developer, what stops you to do something similar for your favorite OS? Then you can put this 64-bit build somewhere and announce this in favor to other users of the same OS in the list. Or you can ask your OS vendor (you are paying them If I remember well) to start maintain MSI for sqlite3. Why you are thinking that someone other is responsible for the lack of binary packages for your OS variant? On 29.10.2011 15:03, Arbol One wrote: Yes, compiling to 32-bit is nice. It is like dancing at the tunes of Earth Wind and Fire, h, those were the days. Like the old good songs, the 32bit apps are a good memory, many people like it, but they are a thing of the past. We are developers and anyone of us who stays behind, is left behind. Not having the option of SQLite-64 is a major drawback and will leave us behind. So, come on folks, pull up your socks and let's get the SQLite apps into the 21st century, shall we? ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] [Regression?] FTS function crashes since 3.7.7
On Sat, Oct 29, 2011 at 8:40 AM, Alexandre Courbot gnu...@gmail.com wrote: By the way, can we expect a point release (3.7.8.1?) that includes this fix soon? It seems rather serious to me, and the use case should not be so seldom. The fix will be in 3.7.9. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Using SQLite on Windows 64bit
As a programmer, I just build my own 64 bit version of Sqlite. You should try it, it's easy. Just a matter of selecting x64 when I compile. For my customers, 64 bit installs now outnumber 32 bit. Sqlite works just fine... C Saturday, October 29, 2011, 8:03:55 AM, you wrote: AO Yes, compiling to 32-bit is nice. It is like dancing at the tunes of Earth AO Wind and Fire, h, those were the days. Like the old good songs, the AO 32bit apps are a good memory, many people like it, but they are a thing of AO the past. We are developers and anyone of us who stays behind, is left AO behind. Not having the option of SQLite-64 is a major drawback and will AO leave us behind. So, come on folks, pull up your socks and let's get the AO SQLite apps into the 21st century, shall we? AO AO ___ AO sqlite-users mailing list AO sqlite-users@sqlite.org AO http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users -- Best regards, Tegmailto:t...@djii.com ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Slow inserts with UNIQUE
I have a table with one TEXT column. I insert 1 million rows of short strings, and then create an UNIQUE INDEX. The speed is very acceptable. Then I insert another 10.000 short strings, and the performance is very bad, it almosts take longer than inserting the initial million(!) rows to fill the table. Is this by design, or what can be causing this? I also observed that inserting the additional 10.000 rows goes progressively faster when nearing the end, I suppose because this is because at that time the index is completely in cache. But what is the reason for the bad performance of these inserts? I read things about b-tree page splitting, could it be related? ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Slow inserts with UNIQUE
On 29 Oct 2011, at 5:57pm, Fabian wrote: I have a table with one TEXT column. I insert 1 million rows of short strings, and then create an UNIQUE INDEX. The speed is very acceptable. Then I insert another 10.000 short strings, and the performance is very bad, it almosts take longer than inserting the initial million(!) rows to fill the table. When you insert the 10,000 strings are you doing it inside a transaction ? BEGIN TRANSACTION; INSERT ... INSERT ... INSERT ... COMMIT; Simon. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Slow inserts with UNIQUE
2011/10/29 Simon Slavin slav...@bigfraud.org When you insert the 10,000 strings are you doing it inside a transaction ? BEGIN TRANSACTION; INSERT ... INSERT ... INSERT ... COMMIT; Yes, I use transactions, prepared statements, cache_size, journal_mode and synchronous PRAGMA's, almost everything you can think of to make it faster. But most important: I use the exact same code to fill the initial million rows, so if there was anything wrong, the initial filling would be slow too I suppose? ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
