[sqlite] test
please ignore ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] test ignore
test ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] tesing please ignore
please ignore ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] just a test...ignore
just a test please ignore ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Multi-column index is not used with IN operator
Actually I'm already running ANALYZE with the SQLITE_ENABLE_STAT3 enabled. By the way I observe different behaviour with different SQLite versions. Version 3.6.19 (testing with Sqliteman 1.2.2) is selecting the multi-column index, whereas the v3.7 primary index. - Selen From: Hick GunterTo: 'Selen Schabenberger' ; 'General Discussion of SQLite Database' Sent: Friday, December 14, 2012 3:26 PM Subject: AW: [sqlite] Multi-column index is not used with IN operator From what I understand, SQLite performs query planning based on cost estimates. A full table scan costs less per record than an index lookup. 1) (scan cost) * (IN Lookup) is compared with 2) (IN size) * (key lookup) * (sort factor) IN Lookup cost is logarithmic; IN size is linear. The number 22 appears to be where a plot of the costs cross over. ANALYZE will improve SQLite's cost estimates. Gunter -Ursprüngliche Nachricht- Von: Selen Schabenberger [mailto:selen_oz...@yahoo.com] Gesendet: Donnerstag, 13. Dezember 2012 16:07 An: sqlite-users@sqlite.org Betreff: [sqlite] Multi-column index is not used with IN operator Hi All, I am observing some strange behaviour on my database when I execute a query with an IN operator having more than "22" expressions. My table structure looks basically as follows: CREATE TABLE "Messages" ("Id" INTEGER PRIMARY KEY NOT NULL, "Tag" INTEGER NOT NULL, "Flag" INTEGER ) I have a multi-column index on (Tag, Flag, Id) as well as a single column index on the Flag column. If I execute the following query on this table, the Messages table is scanned using the primary key and the results are returned in 20 seconds. > explain query plan Select Messages.Id from Messages where (Tag in ( 1146883, > 1146884, 1146886, 1146888, 1146892, 1146894, 1146896, 1146898, 1146920, > 1146922, 1147912, 1147914, 1147968, 1147970, 1147976, 1147978, 1148012, > 1148015, 1148016, 1148018, 1148020, 1148022, 1148040, 1148042, 1148079, > 1148136, 1148138, 1148191, 1148232, 1148234, 1167643, 1167659, 1167660, > 1167663, 1167667, 1167671, 1167675 ) and Flag=1) order by Messages.Id limit > 0, 100 >> Scan table Messages using integer primary key >> Execute list subquery1 However if I reduce the number of expressions in the IN operator, the index on the (Tag, Flag, Id) is used. > explain query plan Select Messages.MessageId from Messages where (AspTag in ( > 1146883, 1146888, 1146892, 1146894, 1146896, 1146898, 1146920, 1146922, > 1147912, 1147914, 1147968, 1147970, 1147976, 1147978, 1148012, 1148015, > 1148016, 1148018, 1148020, 1148022, 1148040, 1148042) and PduFlag=1) order > by MessageId limit 0, 100 >> Search table Messages using covering index IDX_TAG_FLAG_ID >> Execute list subquery1 >> Use temp b-tree for order by If I rewrite the first query in multiple IN operators connecting with OR then I can again use my multi-column index and it is much more efficient than the full table scan. > explain query plan Select Messages.Id from Messages where (Tag in ( > 1148138,1148191, 1148232, 1148234, 1167643, 1167659,1146883, 1146884, > 1146886, 1146888, 1146892, 1146894, 1146896, 1146898, 1146920, 1146922, > 1147912, 1147914, 1147968, 1147970, 1147976, 1147978 ) or Tag in (1148012, > 1148015, 1148016, 1148018, 1148020, 1148022, 1148040, 1148042, 1148079, > 1148136, 1167660, 1167663, 1167667, 1167671, 1167675) ) and Flag=1 order by > Messages.Id limit 0, 100 >> Execute list subquery1 >> Search table Messages using covering index IDX_TAG_FLAG_ID >> Execute list subquery1 >> Search table Messages using covering index IDX_TAG_FLAG_ID >> Use temp b-tree for order by Does anybody have any idea why it behaves so? Are there any restrictions with the IN operator? Regards, -Selen ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users -- Gunter Hick Software Engineer Scientific Games International GmbH Klitschgasse 2 – 4, A - 1130 Vienna, Austria FN 157284 a, HG Wien Tel: +43 1 80100 0 E-Mail: h...@scigames.at This e-mail is confidential and may well also be legally privileged. If you have received it in error, you are on notice as to its status and accordingly please notify us immediately by reply e-mail and then delete this message from your system. Please do not copy it or use it for any purposes, or disclose its contents to any person as to do so could be a breach of confidence. Thank you for your cooperation. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Multi-column index is not used with IN operator
>From what I understand, SQLite performs query planning based on cost estimates. A full table scan costs less per record than an index lookup. 1) (scan cost) * (IN Lookup) is compared with 2) (IN size) * (key lookup) * (sort factor) IN Lookup cost is logarithmic; IN size is linear. The number 22 appears to be where a plot of the costs cross over. ANALYZE will improve SQLite's cost estimates. Gunter -Ursprüngliche Nachricht- Von: Selen Schabenberger [mailto:selen_oz...@yahoo.com] Gesendet: Donnerstag, 13. Dezember 2012 16:07 An: sqlite-users@sqlite.org Betreff: [sqlite] Multi-column index is not used with IN operator Hi All, I am observing some strange behaviour on my database when I execute a query with an IN operator having more than "22" expressions. My table structure looks basically as follows: CREATE TABLE "Messages" ("Id" INTEGER PRIMARY KEY NOT NULL, "Tag" INTEGER NOT NULL, "Flag" INTEGER ) I have a multi-column index on (Tag, Flag, Id) as well as a single column index on the Flag column. If I execute the following query on this table, the Messages table is scanned using the primary key and the results are returned in 20 seconds. > explain query plan Select Messages.Id from Messages where (Tag in ( 1146883, > 1146884, 1146886, 1146888, 1146892, 1146894, 1146896, 1146898, 1146920, > 1146922, 1147912, 1147914, 1147968, 1147970, 1147976, 1147978, 1148012, > 1148015, 1148016, 1148018, 1148020, 1148022, 1148040, 1148042, 1148079, > 1148136, 1148138, 1148191, 1148232, 1148234, 1167643, 1167659, 1167660, > 1167663, 1167667, 1167671, 1167675 ) and Flag=1) order by Messages.Id limit > 0, 100 >> Scan table Messages using integer primary key >> Execute list subquery1 However if I reduce the number of expressions in the IN operator, the index on the (Tag, Flag, Id) is used. > explain query plan Select Messages.MessageId from Messages where (AspTag in ( > 1146883, 1146888, 1146892, 1146894, 1146896, 1146898, 1146920, 1146922, > 1147912, 1147914, 1147968, 1147970, 1147976, 1147978, 1148012, 1148015, > 1148016, 1148018, 1148020, 1148022, 1148040, 1148042) and PduFlag=1) order > by MessageId limit 0, 100 >> Search table Messages using covering index IDX_TAG_FLAG_ID >> Execute list subquery1 >> Use temp b-tree for order by If I rewrite the first query in multiple IN operators connecting with OR then I can again use my multi-column index and it is much more efficient than the full table scan. > explain query plan Select Messages.Id from Messages where (Tag in ( > 1148138,1148191, 1148232, 1148234, 1167643, 1167659,1146883, 1146884, > 1146886, 1146888, 1146892, 1146894, 1146896, 1146898, 1146920, 1146922, > 1147912, 1147914, 1147968, 1147970, 1147976, 1147978 ) or Tag in (1148012, > 1148015, 1148016, 1148018, 1148020, 1148022, 1148040, 1148042, 1148079, > 1148136, 1167660, 1167663, 1167667, 1167671, 1167675) ) and Flag=1 order by > Messages.Id limit 0, 100 >> Execute list subquery1 >> Search table Messages using covering index IDX_TAG_FLAG_ID >> Execute list subquery1 >> Search table Messages using covering index IDX_TAG_FLAG_ID >> Use temp b-tree for order by Does anybody have any idea why it behaves so? Are there any restrictions with the IN operator? Regards, -Selen ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users -- Gunter Hick Software Engineer Scientific Games International GmbH Klitschgasse 2 – 4, A - 1130 Vienna, Austria FN 157284 a, HG Wien Tel: +43 1 80100 0 E-Mail: h...@scigames.at This e-mail is confidential and may well also be legally privileged. If you have received it in error, you are on notice as to its status and accordingly please notify us immediately by reply e-mail and then delete this message from your system. Please do not copy it or use it for any purposes, or disclose its contents to any person as to do so could be a breach of confidence. Thank you for your cooperation. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Multi-column index is not used with IN operator
Reordering the index worked but actually I need the index in (Tag, Flag, Id) order. The Flag column has a very low cardinality, and my queries are mostly filtering Tag & Flag or the Tag alone. When I drop the index on the Flag, and leave the index on (Tag, Flag, Id), the query planner still chooses the primary key on the Id column. -Selen From: Richard HippTo: Selen Schabenberger ; General Discussion of SQLite Database Sent: Friday, December 14, 2012 3:09 PM Subject: Re: [sqlite] Multi-column index is not used with IN operator On Thu, Dec 13, 2012 at 10:06 AM, Selen Schabenberger wrote: Hi All, > >I am observing some strange behaviour on my database when I execute a query >with an IN operator having more than "22" expressions. My table structure >looks basically as follows: > >CREATE TABLE "Messages" ("Id" INTEGER PRIMARY KEY NOT NULL, "Tag" INTEGER NOT >NULL, "Flag" INTEGER ) > > >I have a multi-column index on (Tag, Flag, Id) as well as a single column >index on the Flag column. > My guess is that the single-column index on Flag is misleading the query optimizer. You can probably fix this by either (1) running ANALYZE or (2) adding a "+" in front of the "Flag" column name in the WHERE clause of your query, like this: "... +Flag=1 ..." -- D. Richard Hipp d...@sqlite.org From: Simon Slavin To: Selen Schabenberger ; General Discussion of SQLite Database Sent: Friday, December 14, 2012 2:11 PM Subject: Re: [sqlite] Multi-column index is not used with IN operator On 13 Dec 2012, at 3:06pm, Selen Schabenberger wrote: >> explain query plan Select Messages.Id from Messages where (Tag in ( 1146883, >> 1146884, 1146886, 1146888, 1146892, 1146894, 1146896, 1146898, 1146920, >> 1146922, 1147912, 1147914, 1147968, 1147970, 1147976, 1147978, 1148012, >> 1148015, 1148016, 1148018, 1148020, 1148022, 1148040, 1148042, 1148079, >> 1148136, 1148138, 1148191, 1148232, 1148234, 1167643, 1167659, 1167660, >> 1167663, 1167667, 1167671, 1167675 ) and Flag=1) order by Messages.Id limit >> 0, 100 Just out of interest, that form where you go ... WHERE (Tag in (a,b,c) AND Flag=1) ... Can you try a version where it uses instead ... WHERE Tag in (a,b,c) AND Flag=1 ... or ... WHERE (Tag in (a,b,c)) AND (Flag=1) ... I'm trying to predict how the parsing works here but I don't know whether I figured it out. Another thing to try is to reverse the order of your index > I have a multi-column index on (Tag, Flag, Id) as well as a single column > index on the Flag column. Can you add a new one, or change your existing one, to (Flag, Tag, Id) ? > Are there any restrictions with the IN operator? Nothing that you got anywhere close to. It's meant to be able to handle text and lists far longer than that. Simon. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] malloc failures on ubuntu
Of course that doesn't guarantee you're executing the same code. Only a static binary or a MD5Sum of all code in the libraries involved can ensure that. So the question is, what's different? Maybe if somebody could post a static binary for him to test that might help narrow things down to see if it's his hardware or software setup. I'm on RedHat but this is how to check what you're executing ldd -v sqlite3 linux-vdso.so.1 => (0x7fffb7d3e000) libreadline.so.5 => /usr/lib64/libreadline.so.5 (0x003d2ca0) libncurses.so.5 => /usr/lib64/libncurses.so.5 (0x003d3d40) libpthread.so.0 => /lib64/libpthread.so.0 (0x003d2ae0) libdl.so.2 => /lib64/libdl.so.2 (0x003d2aa0) libc.so.6 => /lib64/libc.so.6 (0x003d2a20) /lib64/ld-linux-x86-64.so.2 (0x003d29e0) -Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Richard Hipp Sent: Friday, December 14, 2012 7:13 AM To: General Discussion of SQLite Database Subject: Re: [sqlite] malloc failures on ubuntu On Fri, Dec 14, 2012 at 2:37 AM, bkkwrote: > Below are the steps i followed in my Ubuntu PC: > > 1. From the site : http://www.sqlite.org/download.html > 2. connected to http://www.sqlite.org/cgi/src (Dallas) > 3. clicked [52e755943f] Leaf ( which was on the top) > 4. clicked ZIP archive > 5. saved the file > 6. copied the zip file(SQLite-52e755943f87354f.zip) and extracted > 7. in the terminal navigated to the extracted directory > 8. Commanded for ./configure and then maketest > I get "0 errors out of 119415 test" when I follow the steps above on my Ubuntu desktop. > > Please correct, if something am doing wrong here. > > > > -- > View this message in context: > http://sqlite.1065341.n5.nabble.com/malloc-failures-on-ubuntu-tp65936p66136. html > Sent from the SQLite mailing list archive at Nabble.com. > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Multi-column index is not used with IN operator
On Thu, Dec 13, 2012 at 10:06 AM, Selen Schabenbergerwrote: > Hi All, > > I am observing some strange behaviour on my database when I execute a > query with an IN operator having more than "22" expressions. My table > structure looks basically as follows: > > CREATE TABLE "Messages" ("Id" INTEGER PRIMARY KEY NOT NULL, "Tag" INTEGER > NOT NULL, "Flag" INTEGER ) > > > I have a multi-column index on (Tag, Flag, Id) as well as a single column > index on the Flag column. > My guess is that the single-column index on Flag is misleading the query optimizer. You can probably fix this by either (1) running ANALYZE or (2) adding a "+" in front of the "Flag" column name in the WHERE clause of your query, like this: "... +Flag=1 ..." -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] malloc failures on ubuntu
On Fri, Dec 14, 2012 at 2:37 AM, bkkwrote: > Below are the steps i followed in my Ubuntu PC: > > 1. From the site : http://www.sqlite.org/download.html > 2. connected to http://www.sqlite.org/cgi/src (Dallas) > 3. clicked [52e755943f] Leaf ( which was on the top) > 4. clicked ZIP archive > 5. saved the file > 6. copied the zip file(SQLite-52e755943f87354f.zip) and extracted > 7. in the terminal navigated to the extracted directory > 8. Commanded for ./configure and then maketest > I get "0 errors out of 119415 test" when I follow the steps above on my Ubuntu desktop. > > Please correct, if something am doing wrong here. > > > > -- > View this message in context: > http://sqlite.1065341.n5.nabble.com/malloc-failures-on-ubuntu-tp65936p66136.html > Sent from the SQLite mailing list archive at Nabble.com. > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Multi-column index is not used with IN operator
On 13 Dec 2012, at 3:06pm, Selen Schabenbergerwrote: >> explain query plan Select Messages.Id from Messages where (Tag in ( 1146883, >> 1146884, 1146886, 1146888, 1146892, 1146894, 1146896, 1146898, 1146920, >> 1146922, 1147912, 1147914, 1147968, 1147970, 1147976, 1147978, 1148012, >> 1148015, 1148016, 1148018, 1148020, 1148022, 1148040, 1148042, 1148079, >> 1148136, 1148138, 1148191, 1148232, 1148234, 1167643, 1167659, 1167660, >> 1167663, 1167667, 1167671, 1167675 ) and Flag=1) order by Messages.Id limit >> 0, 100 Just out of interest, that form where you go ... WHERE (Tag in (a,b,c) AND Flag=1) ... Can you try a version where it uses instead ... WHERE Tag in (a,b,c) AND Flag=1 ... or ... WHERE (Tag in (a,b,c)) AND (Flag=1) ... I'm trying to predict how the parsing works here but I don't know whether I figured it out. Another thing to try is to reverse the order of your index > I have a multi-column index on (Tag, Flag, Id) as well as a single column > index on the Flag column. Can you add a new one, or change your existing one, to (Flag, Tag, Id) ? > Are there any restrictions with the IN operator? Nothing that you got anywhere close to. It's meant to be able to handle text and lists far longer than that. Simon. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] malloc failures on ubuntu
Below are the steps i followed in my Ubuntu PC: 1. From the site : http://www.sqlite.org/download.html 2. connected to http://www.sqlite.org/cgi/src (Dallas) 3. clicked [52e755943f] Leaf ( which was on the top) 4. clicked ZIP archive 5. saved the file 6. copied the zip file(SQLite-52e755943f87354f.zip) and extracted 7. in the terminal navigated to the extracted directory 8. Commanded for ./configure and then maketest Please correct, if something am doing wrong here. -- View this message in context: http://sqlite.1065341.n5.nabble.com/malloc-failures-on-ubuntu-tp65936p66136.html Sent from the SQLite mailing list archive at Nabble.com. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Multi-column index is not used with IN operator
Hi All, I am observing some strange behaviour on my database when I execute a query with an IN operator having more than "22" expressions. My table structure looks basically as follows: CREATE TABLE "Messages" ("Id" INTEGER PRIMARY KEY NOT NULL, "Tag" INTEGER NOT NULL, "Flag" INTEGER ) I have a multi-column index on (Tag, Flag, Id) as well as a single column index on the Flag column. If I execute the following query on this table, the Messages table is scanned using the primary key and the results are returned in 20 seconds. > explain query plan Select Messages.Id from Messages where (Tag in ( 1146883, > 1146884, 1146886, 1146888, 1146892, 1146894, 1146896, 1146898, 1146920, > 1146922, 1147912, 1147914, 1147968, 1147970, 1147976, 1147978, 1148012, > 1148015, 1148016, 1148018, 1148020, 1148022, 1148040, 1148042, 1148079, > 1148136, 1148138, 1148191, 1148232, 1148234, 1167643, 1167659, 1167660, > 1167663, 1167667, 1167671, 1167675 ) and Flag=1) order by Messages.Id limit > 0, 100 >> Scan table Messages using integer primary key >> Execute list subquery1 However if I reduce the number of expressions in the IN operator, the index on the (Tag, Flag, Id) is used. > explain query plan Select Messages.MessageId from Messages where (AspTag in ( > 1146883, 1146888, 1146892, 1146894, 1146896, 1146898, 1146920, 1146922, > 1147912, 1147914, 1147968, 1147970, 1147976, 1147978, 1148012, 1148015, > 1148016, 1148018, 1148020, 1148022, 1148040, 1148042) and PduFlag=1) order > by MessageId limit 0, 100 >> Search table Messages using covering index IDX_TAG_FLAG_ID >> Execute list subquery1 >> Use temp b-tree for order by If I rewrite the first query in multiple IN operators connecting with OR then I can again use my multi-column index and it is much more efficient than the full table scan. > explain query plan Select Messages.Id from Messages where (Tag in ( > 1148138,1148191, 1148232, 1148234, 1167643, 1167659,1146883, 1146884, > 1146886, 1146888, 1146892, 1146894, 1146896, 1146898, 1146920, 1146922, > 1147912, 1147914, 1147968, 1147970, 1147976, 1147978 ) or Tag in (1148012, > 1148015, 1148016, 1148018, 1148020, 1148022, 1148040, 1148042, 1148079, > 1148136, 1167660, 1167663, 1167667, 1167671, 1167675) ) and Flag=1 order by > Messages.Id limit 0, 100 >> Execute list subquery1 >> Search table Messages using covering index IDX_TAG_FLAG_ID >> Execute list subquery1 >> Search table Messages using covering index IDX_TAG_FLAG_ID >> Use temp b-tree for order by Does anybody have any idea why it behaves so? Are there any restrictions with the IN operator? Regards, -Selen ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users