Re: [sqlite] Assertion Failed In sqlite3
On 12/27/19, Richard Hipp wrote: > On 12/27/19, Richard Hipp wrote: >> >> This is the third such false-positive bug like that this month > > And now there is a fourth: https://sqlite.org/src/info/5fbc159eeb092130c6f2 For those of you still keeping score... A fifth example of this is ticket https://www.sqlite.org/src/info/de4b04149b9fdeae The problem there was another false-positive in the OP_SCopy misuse detection mechanism. The fix was to improve that mechanism to avoid the false positive. But since the OP_SCopy misuse detection is a debugging feature that is only present in the code when it is compiled with SQLITE_DEBUG, the "fix" makes no changes to deliverable code. That is to say, the compiled machine code for the SQLite library is byte-for-byte identical before and after the fix. Even so, the OP_SCopy misuse detection logic is an important testing component of SQLite, and we sincerely appreciate bug reports against that feature. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Assertion Failed In sqlite3
On 12/27/19, Richard Hipp wrote: > > This is the third such false-positive bug like that this month And now there is a fourth: https://sqlite.org/src/info/5fbc159eeb092130c6f2 CREATE TABLE t0(c0 NOT NULL DEFAULT 1, c1 AS(c0) UNIQUE); REPLACE INTO t0 VALUES(NULL); If you understood my prior email, then you should clearly see what is going on here. The t0.c0 starts out as NULL. That value is SCopy-ied into t1.c1. Then the REPLACE causes t0.c0 to be overwritten with a numeric 1, which invalidates t1.c1, causing a memIsValid() assert further along in the processing. But the overwrite of t0.c0 didn't really disturb the value in t1.c1 since NULLs are copied by value, not by reference. But notice how this reveals a deeper more subtle problem in the new (unreleased) generated columns feature. The values of generated columns are computed *before* the NOT NULL constraints run during the INSERT. But the NOT NULL constraints might cause (normal) columns to change values due to the REPLACE. If those normal columns where previously used by generated columns, it might cause incorrect generated column values to be stored (for a STORED generated column) or incorrect entries to be written in indexes based on VIRTUAL generated columns (as in the case above). This is a design problem in generated columns. It has never come up before with another database because (as far as I know) SQLite is the only database that supports both generated columns and the NOT NULL ON CONFLICT REPLACE behavior. I suppose the correct solution here is to recompute the values of all generated columns *after* all NOT NULL ON CONFLICT REPLACE constraints have been run, if those constrains caused any changes. The point of this email: The bug report complains about the memIsValid() assertion fault. That assertion fault is not really a problem. It is more like a compiler warning. But in this case, the compiler warning lead me to discover a different, subtle, and unrelated problem in the design. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Assertion Failed In sqlite3
On 12/27/19, Manuel Rigger wrote: > Hi Yongheng and Rui, > > This might be a duplicate to a bug that I reported, since both test cases > trigger the same assertion error. See > https://sqlite.org/src/tktview?name=37823501c6. That particular assert() is sort of like an ASAN fault except that it is for the SQLite byte-code rather than for machine code. The assert means that one of the registers in the virtual machine is being used when it is uninitialized or has been invalidated. This is usually harmless, but it is important to find and fix those problems nevertheless. Sometimes these warnings are false-positives. Let me explain: The byte-code engine has an instruction OP_SCopy that makes a "shallow copy" of the value in one register into some other register. The copy is "shallow" because if it is a string or a blob value, it only copies a pointer to the string or blob, not the string or blob itself. This is an important performance optimization when dealing with large strings and blobs. But using OP_SCopy carries risk. If you OP_SCopy register 1 into register 2, then you change the value of register 1, then the value in register 2 is invalid. When you compile with SQLITE_DEBUG, SQLite adds extra logic that looks for values that have been invalided after an SCopy. If you SCopy register 1 into register 2, then change the value of register 1, then try to access the value of register 2 in any way, you get the memIsValid() assertion. The value of register 2 only truly goes invalid if the SCopy-ed value was a string or a blob. But the memIsValid() mechanism marks the register as invalid regardless of what kind of value was copied, under the theory that the same SCopy opcode might copy a string or a blob on a different iteration, depending on the database content. The mrigger bug I'm working on right now (https://www.sqlite.org/src/info/37823501c68a09f9) is an example of false-positive in this SCopy error detection mechanism. In that ticket, the value of virtual column C1 is SCopy-ed from C0. If that value is a NULL, then the REPLACE conflict resolution logic causes the original value for C0 to be replaced by an empty string. But since C1 was SCopy-ed from C0, that invalidates the value inside of C1, which causes problems later. But this is a false positive, because C0 will only be overwritten in a NOT NULL ON CONFLICT REPLACE constraint if its original value was NULL. So the SCopy did not copy a string or a blob and so there really is no reason to invalidate the C1 register. So, the https://www.sqlite.org/src/info/37823501c68a09f9 ticket is really a false-positive in the SCopy misuse validation logic. It is still an important ticket and needs to be fixed. But the problem would never appear in practice. It is a bug in logic added by SQLITE_DEBUG and which does not appear in release builds. This is the third such false-positive bug like that this month. The previous two were: https://www.sqlite.org/src/info/5ad2aa6921faa1ee https://www.sqlite.org/src/info/c62c5e58524b204d None of these tickets would have resulted in actual problems in deployment. But it is important to fix them, just as it is important to get your C-code to compile without warnings even though the warnings in many cases are spurious. Warnings are often false-positive, but the warning mechanism does sometimes find real bugs, so it is helpful to keep it in place. And, of course, I could bypass all of this heartache by using OP_Copy (which does a deep copy of the entire value) everywhere instead of sometimes using OP_SCopy. Doing so would not make a noticeable difference for most applications, but it might cause some applications that deal with lots of large strings and blobs to run slower. In other words, these is really all about an optimization. That last sentence is true about most things we do. Almost without exception, bugs found in SQLite arise from our attempts to cut corners and make it run faster. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Assertion Failed In sqlite3
Hi Manuel, Hh, what a coincident. It might be so. The test case looks very different though. Let’s wait for Richard to find it out then. Yongheng & Rui > On Dec 27, 2019, at 2:03 PM, Manuel Rigger wrote: > > Hi Yongheng and Rui, > > This might be a duplicate to a bug that I reported, since both test cases > trigger the same assertion error. See > https://sqlite.org/src/tktview?name=37823501c6. > > Best, > Manuel > > On Fri, Dec 27, 2019 at 6:09 PM Yongheng Chen wrote: > >> Hi, >> >> We found an assertion violation bug in sqlite. Here’s the PoC: >> — >> CREATE TABLE v0 ( v1 INTEGER PRIMARY KEY ) ; INSERT INTO v0 VALUES ( 10 ) >> ; >> SELECT '29' , count () OVER( ORDER BY v1 ) AS m FROM v0 ORDER BY v1 > ( >> SELECT m ) ; >> — >> >> The bug exists in the latest development code and release code. >> >> Yongheng & Rui >> ___ >> sqlite-users mailing list >> sqlite-users@mailinglists.sqlite.org >> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users >> > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Assertion Failed In sqlite3
Hi Yongheng and Rui, This might be a duplicate to a bug that I reported, since both test cases trigger the same assertion error. See https://sqlite.org/src/tktview?name=37823501c6. Best, Manuel On Fri, Dec 27, 2019 at 6:09 PM Yongheng Chen wrote: > Hi, > > We found an assertion violation bug in sqlite. Here’s the PoC: > — > CREATE TABLE v0 ( v1 INTEGER PRIMARY KEY ) ; INSERT INTO v0 VALUES ( 10 ) > ; > SELECT '29' , count () OVER( ORDER BY v1 ) AS m FROM v0 ORDER BY v1 > ( > SELECT m ) ; > — > > The bug exists in the latest development code and release code. > > Yongheng & Rui > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Assertion Failed In sqlite3
Hi, We found an assertion violation bug in sqlite. Here’s the PoC: — CREATE TABLE v0 ( v1 INTEGER PRIMARY KEY ) ; INSERT INTO v0 VALUES ( 10 ) ; SELECT '29' , count () OVER( ORDER BY v1 ) AS m FROM v0 ORDER BY v1 > ( SELECT m ) ; — The bug exists in the latest development code and release code. Yongheng & Rui ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
