In our Mozilla debug builds we use sqlite3_trace to expose the SQL statements that are invoked via our PR_LOG logging subsystem. This usually works out well but the trace also catches FTS3's statements in its net. During a segment merge, a segment in a Thunderbird FTS3 table can easily be >100k.
Unfortunately, the calls to stringify the BLOB end up growing the string "%02x" character by "%02x" character, and sqlite3StrAccumAppend grows its allocation size linearly and without using realloc. This results in a tremendous number of calls to the allocator and a lot of memory traffic (both copying and memsetting). While this is definitely one of those "well, don't do that" cases, I think it would be beneficial if one or more of the following were to occur, if only because it would be nice to not have to turn sqlite3_trace off on any connection that uses FTS3. 1) Cause FTS3 statements to not generate OP_Trace codes or to ignore them during evaluation. 2) Use realloc in sqlite3StrAccumAppend so the memory allocator has a fighting chance. (With the large segment sizes at play, the mozilla jemalloc, for example, ends up unmapping things at least some of the time.) 3) Use a more efficient BLOB stringification routine. 4) Allow not stringifying BLOBs at all. 5) Place an upper bound or allow the sqlite3_trace invocation to place an upper bound on how large the SQL string can grow. It's very survivable if none of the above ever happen, too. Here's a snippet of an unhappy stack using the 3.7.4 amalgamation in the off-chance it helps provide better context: #7 sqlite3Malloc (n=264183) at sqlite3/src/sqlite3.c:17763 #8 sqlite3DbMallocRaw (db=ptr, n=264183) at sqlite3.c:18088 #9 sqlite3StrAccumAppend (p=ptr, z=ptr, N=2) at sqlite3.c:19002 #10 sqlite3VXPrintf (pAccum=ptr, useExtended=1, fmt=ptr "x", ap=0x7fcaba7fc8e0) at sqlite3.c:18952 #11 sqlite3XPrintf (p=ptr, zFormat=ptr "%02x") at sqlite3.c:19241 #12 sqlite3VdbeExpandSql (p=ptr, zRawSql=ptr ")") at sqlite3.c:59748 #13 sqlite3VdbeExec (p=0x7fcaac70c5d8) at sqlite3.c:66203 #14 sqlite3Step (p=ptr) at sqlite3.c:58704 #15 sqlite3_step (pStmt=ptr) at sqlite3.c:58768 #16 fts3WriteSegment (p=ptr, iBlock=ptr, z=ptr "", n=166139) at sqlite3.c:115871 Thanks, Andrew Sutherland _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
