Re: [sqlite] Why would batched write operations NOT be faster than individual ones
Hi, Von: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] > At 21:35 03/03/2014, you wrote: > ´¯¯¯ > >RAID3-4-5 was great when disks were expensive, in 80's an 90's. Now > >not. A minimal RAID5 needs 3 disks. A minimal RAID10 4. An enterprise > >disk SAS 15Krpm 146 GB 6G is $350, and a not enterprise grade cheaper > >and bigger. Now RAID1E and RAID10E give more flexibility and variable > >security, from "paranoid" to "i don't care" grades. > `--- > > The point being discussed was not on performance or cost, but on the > imaginary fact that RAID5-6 and variations have the inherent, by-design fatal > flaw that they break down because a parity block can be out of sync with > corresponding data blocks. This is bullshit, period. It is not. Period. > Nothing in RAID5-6 design mandates serialization of writes, by far. Yes. But I've yet to know a setup for harddisks which allows reliable transactional writes spanning several disks. (Kinda two-phase commit for disk writes). Of course, a dedicated hardware controller who issues the write requests to the disks absolutely synchronously lowers the risk by shrinking the time window. But it cannot totally eliminate it, for at least the following reasons: - The platters are usually not completely physically in sync, so the first disk may have written the block while the second disk still needs to wait for another 1/4 rotation for the block to be written. - One of the disks may have internally remapped a bad sector, needing a seek (and thus much longer time) to write the block. In reality, there are usually some more time variation, e. G. due to - Both disks may be connected through the same cable, thus the requests to the disks need to be serialized. - There may be other outstanding requests in the disk internal cache which the disk firmware might reorder in a different way. I admit that the remaining risk may be low, but it is not zero. Period. > It's only when cheap, unreliable hardware is put to work under below par > software that the issue can be a real-world problem. > > So the rant on the design against parity-enabled RAIDs is moot, if not plain > fallacious unless "software RAID without dedicated controller" is clearly > mentionned. I did mention using battery backed power as a way to mitigate the risk. And as far as I know, even the most expensive hardware RAID controllers and disks do not yet support multi-disk transactions, so your reference to not-yet existing hardware is moot. > About SAS disks: they have actual very high reliability and don't lie, > contrary to SATA disks (on both points). > > This is not a war about mine being bigger, but it's better to have facts > stated right. I fully agree there. > All high-end reliable machines and storage subsystems only run > parity-enabled RAID levels and this thechnology isn't going to disappear > tomorrow. I doubt that _all_ those machines exclusively run on parity-enabled RAID levels, but I'm strongly interested in a proof of your "fact" here. I remember reading that PostgreSQL and Oracle recommend using mirroring based levels instead of parity-enabled ones for performance reasons, so I'm really curious to read about how you back up your claim. Best regards Markus Schaber CODESYS® a trademark of 3S-Smart Software Solutions GmbH Inspiring Automation Solutions 3S-Smart Software Solutions GmbH Dipl.-Inf. Markus Schaber | Product Development Core Technology Memminger Str. 151 | 87439 Kempten | Germany Tel. +49-831-54031-979 | Fax +49-831-54031-50 E-Mail: m.scha...@codesys.com | Web: http://www.codesys.com | CODESYS store: http://store.codesys.com CODESYS forum: http://forum.codesys.com Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915 ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLITE_MAX_PAGE_COUNT
I will make the changes as per my knowledge and send it to all on this list to see if it helps in anyway. On Thu, Mar 6, 2014 at 5:25 AM, Roger Binnswrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 05/03/14 10:59, Raheel Gupta wrote: > > If you point out to me the changes required I will do it and have it > > reviewed. > > The changes required are to update the test suites (there are several) to > hit/cross the current limit, to modify all relevant code including any > code that calls that code, to audit for overflows, to provide a new api > and tests for that. > > And to ensure that stays maintained for the future lifetime of SQLite 3. > > Roger > > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.14 (GNU/Linux) > > iEYEARECAAYFAlMXuWsACgkQmOOfHg372QRWcgCgmLmuv+d7Mxu5sfn9XFxAwx8N > xagAoIpbTOEPWtPqXRsQSBuIjgV4NYhl > =RMyF > -END PGP SIGNATURE- > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Issues after upgrade to 3.8.2
Denis Gladkikh wrote: > > I still have this issue with SQLite 3.8.3.1, I could fix my case my > changing line > > "LastPlayed] = (select case when [UserPlaylist].[LastPlayed] > > s.[LastPlayed] then [UserPlaylist].[LastPlayed] else s.[LastPlayed] end > from [Song] as s where s.[SongId] = new.[SongId])," > Thanks for the detailed report. This issue has been fixed on trunk: https://www.sqlite.org/src/info/5bcd0b1ca5 -- Joe Mistachkin ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Troubleshooting query performance
> Don't put the PRIMARY KEY as the first column of your index. Ever. This > applies to all SQL database engines, not just SQLite. > > For that matter, don't put the PRIMARY KEY anywhere in your index. The > PRIMARY KEY will be added automatically at the end, where it belongs. > > If you remove the "id," from all of your indices, I think your performance > will probably improve dramatically. Richard, Simon, So much for testing at the console over a remote session, while I was used to waiting for a single row, the new query increased by so much it overwhelmed my display, heh. Nice and thanks for that tip! jlc ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Troubleshooting query performance
On 6 Mar 2014, at 12:20am, Joseph L. Casalewrote: > SELECT a.name, p.value > FROM p_attribute p > JOIN attribute a > ON a.id=p.aid > WHERE p.pid=? > > This returns all relevant rows I need, where table profile has ~6000 rows, > p_attribute has ~ 170k and attribute has ~60 rows. > > Analyze has been run, explain query plan shows: > recno selectedorder fromdetail > 0 0 0 SCAN TABLE p_attribute AS p > 0 1 1 SEARCH TABLE attribute AS a USING INTEGER PRIMARY KEY > (rowid=?) > > Any pointers as to what may not be optimal? You don't have ideal indexes. The main task of this SELECT is to look for certain values of p_attribute.pid. You don't have an index that makes this easy. Do something like CREATE INDEX p_attribute_idx_0 ON p_attribute (pid, id) Once you've done this do an ANALYZE just to make sure, then try the SELECT again. Simon. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Troubleshooting query performance
On Wed, Mar 5, 2014 at 7:20 PM, Joseph L. Casalewrote: > Hey guys, > > I have a query that's giving me abysmal performance and it's not > immediately > obvious to me as to what's wrong with the table structure to cause this. > > CREATE TABLE profile ( > id INTEGER PRIMARY KEY AUTOINCREMENT, > name TEXT UNIQUE NOT NULL > ); > CREATE INDEX profile_idx_0 ON profile ( > id, > name > ); > Don't put the PRIMARY KEY as the first column of your index. Ever. This applies to all SQL database engines, not just SQLite. For that matter, don't put the PRIMARY KEY anywhere in your index. The PRIMARY KEY will be added automatically at the end, where it belongs. If you remove the "id," from all of your indices, I think your performance will probably improve dramatically. > > CREATE TABLE p_attribute ( > id INTEGER PRIMARY KEY AUTOINCREMENT, > pid INTEGER NOT NULL > REFERENCES profile (id) > ON DELETE CASCADE, > aid INTEGER NOT NULL > REFERENCES attribute (id) > ON DELETE CASCADE, > valueTEXT > ); > CREATE INDEX p_attribute_idx_0 ON p_attribute ( > id, > pid, > aid > ); > > CREATE TABLE attribute ( > id INTEGER PRIMARY KEY AUTOINCREMENT, > name TEXT UNIQUE NOT NULL, > CHECK(UPPER(name) = name) > ); > CREATE INDEX attribute_idx_0 ON attribute ( > id, > name > ); > > > SELECT a.name, p.value > FROM p_attribute p > JOIN attribute a >ON a.id=p.aid > WHERE p.pid=? > > This returns all relevant rows I need, where table profile has ~6000 rows, > p_attribute has ~ 170k and attribute has ~60 rows. > > Analyze has been run, explain query plan shows: > recno selectedorder fromdetail > 0 0 0 SCAN TABLE p_attribute AS p > 0 1 1 SEARCH TABLE attribute AS a USING INTEGER PRIMARY > KEY (rowid=?) > > Any pointers as to what may not be optimal? > Thanks, > jlc > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Troubleshooting query performance
Hey guys, I have a query that's giving me abysmal performance and it's not immediately obvious to me as to what's wrong with the table structure to cause this. CREATE TABLE profile ( id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT UNIQUE NOT NULL ); CREATE INDEX profile_idx_0 ON profile ( id, name ); CREATE TABLE p_attribute ( id INTEGER PRIMARY KEY AUTOINCREMENT, pid INTEGER NOT NULL REFERENCES profile (id) ON DELETE CASCADE, aid INTEGER NOT NULL REFERENCES attribute (id) ON DELETE CASCADE, valueTEXT ); CREATE INDEX p_attribute_idx_0 ON p_attribute ( id, pid, aid ); CREATE TABLE attribute ( id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT UNIQUE NOT NULL, CHECK(UPPER(name) = name) ); CREATE INDEX attribute_idx_0 ON attribute ( id, name ); SELECT a.name, p.value FROM p_attribute p JOIN attribute a ON a.id=p.aid WHERE p.pid=? This returns all relevant rows I need, where table profile has ~6000 rows, p_attribute has ~ 170k and attribute has ~60 rows. Analyze has been run, explain query plan shows: recno selectedorder fromdetail 0 0 0 SCAN TABLE p_attribute AS p 0 1 1 SEARCH TABLE attribute AS a USING INTEGER PRIMARY KEY (rowid=?) Any pointers as to what may not be optimal? Thanks, jlc ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLITE_MAX_PAGE_COUNT
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/03/14 10:59, Raheel Gupta wrote: > If you point out to me the changes required I will do it and have it > reviewed. The changes required are to update the test suites (there are several) to hit/cross the current limit, to modify all relevant code including any code that calls that code, to audit for overflows, to provide a new api and tests for that. And to ensure that stays maintained for the future lifetime of SQLite 3. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.14 (GNU/Linux) iEYEARECAAYFAlMXuWsACgkQmOOfHg372QRWcgCgmLmuv+d7Mxu5sfn9XFxAwx8N xagAoIpbTOEPWtPqXRsQSBuIjgV4NYhl =RMyF -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] About "speed"
Hi *Elefterios, Simon,* *Wes McKinney gave us :- a fully detailed benchmark case (data + reproducible test),* *- where SQLite was : . abnormally less good than Postgresql (so could be better),* * . SQLdatabase in general were abnormally less good, . a hint "vertica"was given.* *Maybe it is a 'golden' benchmark, as it hints a possible 10x margin of improvement for SQLite. * ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite version 3.8.4 release schedule
I noticed the message: Enter SQL statements terminated with a ";" was removed from the shell. Is this supposed to be in the right direction when a red message is added to warn newbies to SQLite about working with in-memory database, yet they are assumed to know that SQL commands end with ";", when up to now everybody (even seasoned SQLiters) had to be reminded about this? My €0.02 ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Syntax diagram missing in sqlite3 docs
On 2/20/14, Zsbán Ambruswrote: > The page "http://sqlite.org/lang_transaction.html; should show the > syntax diagrams for commit-stmt and rollback-stmt. It seems that these bugs are now fixed in the draft documentation for sqlite 3.8.4. Thank you, sqlite maintainers. Ambrus ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] About "speed"
ok, Just updated with 3.8.4beta of 2014-03-05. I also re-did some previous measures as : - testing method improved a little, - I measured more carefully that SQLite has also a sort of caching benefit, when you run a query twice on windows7. Regards, ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Virtual table API performance
On 05.03.2014 11:02, RSmith wrote: On 2014/03/05 10:41, Dominique Devienne wrote: On Tue, Mar 4, 2014 at 9:41 PM, Elefterios Stamatogiannakis One thing that IMHO long term might improve the situation would be if SQLite's own "native" tables would use the same Virtual Table API,//... ...//Of course, the above is a "naive" abstract reflection which ignores the realities of VDBE, so it may sound rubbish to actual SQLite developers. Apologies for that. --DD I don't think it is rubbish at all, but maybe idealistic. The biggest problem I can see from making API's pov is that you can at any time alter, update, change the way SQLIte (or any other API) works with the base check that the input values produce the same (or maybe more-correct) results. Once you let the VT use the same API, any change is a potential change to how other people's programmed interfaces need to talk to - or get data from - the SQLite engine. This cannot simply change on a whim, so the levels of separation remain needed. That said, I'm all for making a more efficient VT API, but it would probably need to be "new" functionality since I cannot see how the existing interface could implement any of the mentioned enhancements without breaking existing behaviour. The OP's xNextRow suggestion seems a good idea, but opens up a whole can of what-ifs which other posters have alluded to, but something to that effect might be worthwhile if the efficiency bonus is significant. The whole thread so far is based on the OP observations in mixed C/PyPy and apsw/CPython environments (as being said already, we suffering the noise in both cases). To be helpful to the SQLite team, before proposing any changes, please let someone show some well designed, pure C vtable implementation demonstrating the possible vtable interface inefficiency. Let's remember that all xNextRow, xNextPage optimizations are applicable only for "select *" cases, not in the general "select f(x), y" case. Kind regards, Alek ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] About "speed"
On Wed, Mar 5, 2014 at 7:25 PM, Richard Hippwrote: > MySQL does very well on query 8 which is a repeat of query 6. This might > be because MySQL implements a query cache. It remembers the result of each > query and if that query occurs again, without an intervening INSERT, > DELETE, or UPDATE on one of the tables used by the query, just echos the > previous answer. > Counterpoint: MySQL's wire protocol does not support concurrent SELECTs to be running on the same connection, meaning client code has to do a lot more work to collect all row data in cases where sqlite3 can easily run queries while stepping over another. In my experience, writing code with their C API takes at least 3-5 times longer because it's just generally a pain to work with (it requires a great deal of the user). sqlite3 wins hands-down on sanity/usability of the C API and, consequently, development speed. That doesn't generally apply to script bindings (where using MySQL is also easy), but it does to the native C APIs. -- - stephan beal http://wanderinghorse.net/home/stephan/ http://gplus.to/sgbeal "Freedom is sloppy. But since tyranny's the only guaranteed byproduct of those who insist on a perfect world, freedom will have to do." -- Bigby Wolf ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLITE_MAX_PAGE_COUNT
Sir, will it require much change ? If you point out to me the changes required I will do it and have it reviewed. AT the moment I am checking code related to the following variables : mxPgno sqlite3PagerMaxPageCount (Function) pageSize On Thu, Mar 6, 2014 at 12:06 AM, Richard Hippwrote: > On Wed, Mar 5, 2014 at 1:24 PM, Raheel Gupta wrote: > > > SQLITE_MAX_PAGE_COUNT is 2147483646. > > After looking at the code, Pgno is a u32. > > So isnt SQLITE_MAX_PAGE_COUNT capable of actually being 4294967292 ? > > > > We have your request. But as it will require a lot of testing, and because > it is unlikely to actually be useful to anybody, it is a low priority. > > > -- > D. Richard Hipp > d...@sqlite.org > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLITE_MAX_PAGE_COUNT
On Wed, Mar 5, 2014 at 1:24 PM, Raheel Guptawrote: > SQLITE_MAX_PAGE_COUNT is 2147483646. > After looking at the code, Pgno is a u32. > So isnt SQLITE_MAX_PAGE_COUNT capable of actually being 4294967292 ? > We have your request. But as it will require a lot of testing, and because it is unlikely to actually be useful to anybody, it is a low priority. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] About "speed"
On Wed, Mar 5, 2014 at 9:29 AM, big stonewrote: > Timing updates with Mysql 5.6.16 > MySQL does very well on query 8 which is a repeat of query 6. This might be because MySQL implements a query cache. It remembers the result of each query and if that query occurs again, without an intervening INSERT, DELETE, or UPDATE on one of the tables used by the query, just echos the previous answer. I think it would be interesting to do a single simple INSERT, DELETE, or UPDATE on the FEC table after test 8, then repeat the same query again, to see if MySQL is able to maintain the spectacular performance of test 8. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] SQLITE_MAX_PAGE_COUNT
SQLITE_MAX_PAGE_COUNT is 2147483646. After looking at the code, Pgno is a u32. So isnt SQLITE_MAX_PAGE_COUNT capable of actually being 4294967292 ? Please correct me if I am wrong ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] sqlite issue when using AIX
On Tue, Mar 4, 2014 at 11:00 AM, Peter Simpsonwrote: > Hi, > > We are currently using SQLite 3.8.2. > > When using this on AIX, we encountered an issue with an "unsigned char" in > the sqliteProcessJoin procedure in sqlite3.c. > Can you please try the latest 3.8.4 beta on AIX for us, and verify that the problem is now fixed. We (the SQLite developers) do not ourselves have access to an AIX machine for testing. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite version 3.8.4 release schedule
On Wed, Mar 5, 2014 at 12:53 PM, big stonewrote: > > ==> Is it expected to have a big speed-up in index creation ? > Yes. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite version 3.8.4 release schedule
Hello, Comparing 3.8.3 to 3.8.4beta fo today : - both compiled in -o2 mode and running in ":memory:" , - on a windows pc. Changes : * 8% quicker on Recursive CTE (sudoku test of http://www.sqlite.org/lang_with.html) * 12% quicker on Index Creation over a 5M record table ( ftp://ftp.fec.gov/FEC/Presidential_Map/2012/P0001/P0001-ALL.zip) ==> Is it expected to have a big speed-up in index creation ? Or is it noise ? Regards, ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] sqlite issue when using AIX
On Tue, Mar 4, 2014 at 10:00 AM, Peter Simpsonwrote: > When using this on AIX, we encountered an issue with an "unsigned char" in > the sqliteProcessJoin procedure in sqlite3.c. > > For cases where "pSrc->nSrc" was 0, then "pSrc->nSrc-1" was being treated > as positive (since nSrc is defined as an unsigned char, u8), and the for > loop was inadvertently being entered. > Peter Simpson (Hopewiser)] > If you're using IBM's own compilers, specify "xlc" as your compiler (CC=xlc ./configure). The default behaviors of good old "cc" can be a bit dated. -- Ross ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] SQLite version 3.8.4 release schedule
A number of bugs have recently been revealed in SQLite. All are obscure. Nevertheless, we want to accelerate the release of version 3.8.4 in order to get the fixes in circulation. To this end, SQLite version 3.8.4 should now be considered "in beta". Amalgamation snapshots are now available on the download page ( http://www.sqlite.org/download.html) and will be updated regularly. Preliminary documentation for 3.8.4 is available at http://www.sqlite.org/draft with a change log at http://www.sqlite.org/draft/releaselog/3_8_4.html that will also be updated regularly. A status board (http://www.sqlite.org/checklists/3080400) is now up and when it goes all-green, the release will occur. The anticipated beta period is about 7 to 10 days. Please try out SQLite 3.8.4 beta in your applications and report any issues found to this mailing list, or directly to me, as soon as possible. If you are able to report on performance differences (hopefully performance *improvements*) then please do. There are no significant new features in SQLite 3.8.4. This will be primarily a performance-enhancement and bug-fix release. Thanks for your help. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] select where field in ($tcl_list) ?
On 2014/03/05 17:05, Chris wrote: Ok, fair enough. I thought that in the same way that sqlite looks for binary vs. string representations of referenced vars and has alternative ways of specifying variable to bind to ('@', ':'), it might also spot a list object and internally expand it to "elem_0,elem_1,elem_2". Would that be a useful feature, or does it introduce opportunities to draw the wrong conclusion? Most high-level languages have some sort of list expansion in list objects which obviates the need. You should for instance be bale to say something like this in most languages: mylist = New(List); mylist.add("Item1"); mylist.add("Item2"); ... mylist.add("Item6"); mylist.add("Item7"); etc.. mylist.separator=","; // Comma mylist.QuoteChar="'"; // Single Quote So the entire Query bit can become something simple like: qry = "SELECT a,b FROM t WHERE id IN ("+mylist.asString+");" or something to that effect Adding any SQLite function to try and break down your sent variable into lists would probably pay a much higher price in processing and added code - moreso than it really being difficult to do or in violation of some method or way of doing. If those lists get really really long or convoluted, it is best to break them into tables of their own via some query and then use the base query from that table as the list specifier. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] select where field in ($tcl_list) ?
> On Wed, Mar 5, 2014 at 4:59 AM, Chris wrote: > > > I'm a relative novice to sqlite (or sql in general), but I do understand > > the value of variable substitution when building queries: > > > > e.g. > > set someValue 23 > > db eval {SELECT something FROM myTable WHERE value=$someValue} > > > > It feels like there should be a comparable solution for IN, passing > > instead a Tcl list, but I've failed to find it documented or had any > > trial-and-errorr success. > > > > e.g. > > set someTags {1 23 45} > > db eval {SELECT something FROM myTable WHERE value IN ($someTags)} > > > > Can't do this. SQL has no "list" or "array" datatype. So $someTags cannot > be translated into an SQL value. > > One work-around: > > db transaction { >db eval {CREATE TEMP TABLE someTags(x INTEGER PRIMARY KEY);} >foreach x $someTags {db eval {INSERT INTO someTags VALUES($x)}} > } > db eval {SELECT something FROM myTable WHERE value IN someTags} > > -- > D. Richard Hipp > drh at sqlite.org Ok, fair enough. I thought that in the same way that sqlite looks for binary vs. string representations of referenced vars and has alternative ways of specifying variable to bind to ('@', ':'), it might also spot a list object and internally expand it to "elem_0,elem_1,elem_2". Would that be a useful feature, or does it introduce opportunities to draw the wrong conclusion? Chris ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] About "speed"
On Wed, Mar 5, 2014 at 9:29 AM, big stonewrote: > Timing updates with Mysql 5.6.16 > I wonder if you could update the timings for the current SQLite 3.8.4 beta? -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] select where field in ($tcl_list) ?
On Wed, Mar 5, 2014 at 4:59 AM, Chriswrote: > I'm a relative novice to sqlite (or sql in general), but I do understand > the value of variable substitution when building queries: > > e.g. > set someValue 23 > db eval {SELECT something FROM myTable WHERE value=$someValue} > > It feels like there should be a comparable solution for IN, passing > instead a Tcl list, but I've failed to find it documented or had any > trial-and-errorr success. > > e.g. > set someTags {1 23 45} > db eval {SELECT something FROM myTable WHERE value IN ($someTags)} > Can't do this. SQL has no "list" or "array" datatype. So $someTags cannot be translated into an SQL value. One work-around: db transaction { db eval {CREATE TEMP TABLE someTags(x INTEGER PRIMARY KEY);} foreach x $someTags {db eval {INSERT INTO someTags VALUES($x)}} } db eval {SELECT something FROM myTable WHERE value IN someTags} -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] select where field in ($tcl_list) ?
On Wed, Mar 5, 2014, at 09:59 AM, Chris wrote: > I'm a relative novice to sqlite (or sql in general), but I do understand > the value of variable substitution when building queries: > > e.g. > set someValue 23 > db eval {SELECT something FROM myTable WHERE value=$someValue} > > It feels like there should be a comparable solution for IN, passing > instead a Tcl list, but I've failed to find it documented or had any > trial-and-errorr success. > > e.g. > set someTags {1 23 45} > db eval {SELECT something FROM myTable WHERE value IN ($someTags)} > > If that's not a supported feature, what would be the recommended and > safe way of building the query? > > > As that's just a detail of a novice's attempt to implement a part of > something that most likely could more correctly be done in a different > way, here's more detail of the overall plan (a light-weight iTunes-like > library): > > Given a trio of tables created as: > > db eval { > CREATE TABLE IF NOT EXISTS media( > id INTEGER PRIMARY KEY AUTOINCREMENT, > filenameTEXT > -- further fields removed > ); > CREATE TABLE IF NOT EXISTS tags( > id INTEGER PRIMARY KEY AUTOINCREMENT, > tag TEXT, > unique(tag) > ); > CREATE TABLE IF NOT EXISTS tagUsage( > mediaId INTEGER, > tagId INTEGER, > unique(mediaId,tagId) > ); > CREATE TRIGGER IF NOT EXISTS tagCleanup AFTER DELETE ON tags > BEGIN > DELETE FROM tagUsage WHERE tagUsage.tagId=OLD.id; > END; > } > > 'media' describes an audio or video file. > 'tags' is just a list of words > 'tagUsage' associates tags to media (a media can have zero or more tags, > a tag can belong to zero or more media). > > At some point, I'm going to want to write queries that answer the > questions: > > 1. What media files have at least *one* of this set of tags? > 2. What media files have *all* of this set of tags? > > That feels like the sort of thing that it should be possible to write in > a single query, but I keep coming back to an initial query, followed by > some processing in code, with a follow up query. I'm sure there's a more > sql-ish way. > > Chris Having realised the above sounds very much like "please do my homework assignment for me", I should point out that that's not the case. I've been a C, C++ and Tcl developer for many years, but have only ever scratched the surface of sqlite. This is me trying to delve a little deeper while avoiding going down too many wrong turns at the beginning. Chris ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] About "speed"
Timing updates with Mysql 5.6.16 test = https://raw.github.com/stonebig/ztest_donotuse/master/benchmark_test01.txt results = https://github.com/stonebig/ztest_donotuse/blob/master/benchmark_test01_measures.GIF?raw=true ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] sqlite issue when using AIX
Peter Simpson wrote: > When using this on AIX, we encountered an issue with an "unsigned char". > > For cases where "nSrc" was 0, then "nSrc-1" was being treated as positive > (since nSrc is defined as an unsigned char, u8) This violates the C standard (any C standard). Does you compiler (whatever it is) claim conformance to a standard? Might this be just some optimization bug? Regards, Clemens ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] sqlite issue when using AIX
Hi, We are currently using SQLite 3.8.2. When using this on AIX, we encountered an issue with an "unsigned char" in the sqliteProcessJoin procedure in sqlite3.c. The issue occurred with the following line (99798) :- for(i=0; inSrc-1; i++, pRight++, pLeft++){ For cases where "pSrc->nSrc" was 0, then "pSrc->nSrc-1" was being treated as positive (since nSrc is defined as an unsigned char, u8), and the for loop was inadvertently being entered. To overcome this, we have resorted to casting the relevant code as follows :- for(i=0; i<((int)pSrc->nSrc)-1; i++, pRight++, pLeft++){ Apologies if this has already been reported. I did check through the current tickets searching for AIX, and didn't find anything so I'm assuming that it hasn't already been reported. Regards, Peter Simpson (Hopewiser) _ Hopewiser Ltd, Merlin Court, Atlantic Street, Altrincham, WA14 5NL Reg in England, number 1621544 Tel: 0161 924 2800, Fax: 0161 924 2809, Web: http://www.hopewiser.com/ The information contained in this email is intended only for the named recipient(s) and may be confidential and/or privileged. Unauthorised use or reproduction (including storage or re-distribution in any media) is prohibited. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Solved Weird out of memory problem a prepare
On Wed, 05 Mar 2014 11:12:45 +0200 RSmithwrote: > If that query fails in an SQLite tool too, then maybe there is a > problem, or if you use a custom altered version of the SQLite code. > Barring that, you need to hunt down the corrupting code - Good luck! I find it and solve. It's a macro expansion in one function extension that collides and redefines a #define in sqlite3.h Thanks to all. --- --- Eduardo Morras ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] How to write a query
On 2014/03/05 12:24, Igor Korot wrote: With UNION I will have 2 DB hits, correct? Meaning I execute the part on the left side of the UNION and then execute the right side of the UNION and then add the results together. Do I understand correctly? Thank you. Yes. Obviously you need to have the same number of field results for the part before the UNION than the part after the UNION to be able to be one query result list at the end without nulls and such. Also, when you don't want duplicates, just use "UNION", if you want all values to be listed, even if they are duplicates, use "UNION ALL" ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] How to write a query
On 2014/03/05 12:04, Igor Korot wrote: Hi, ALL, Let's say I have a table with following data: field1field2field3 field4 12 3 4 5 6 7 8 How do I write a query which will produce the output as: 1 2 5 6 3 4 7 8 Is it possible to write a single query which will produce the output like this? And not just in SQLite SELECT field1, field2 FROM tbl UNION ALL SELECT field3, field4 FROM tbl I can't imagine a use for this... but hey ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] How to write a query
On 5 March 2014 10:24, Igor Korotwrote: > Hi, Simon, > . . . >> >> Possibly: >> select v1, v2 from ( select 1 as ordr, f1 as v1, f2 as v2 from t union >> all select 2, f3, f4 from t ) order by ordr, v1; >> > > With UNION I will have 2 DB hits, correct? > Meaning I execute the part on the left side of the UNION and then execute > the right side of the UNION > and then add the results together. Do I understand correctly? I'm not sure. It is a single SQL statement > > Thank you. > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] How to write a query
Hi, Simon, On Wed, Mar 5, 2014 at 2:21 AM, Simon Davieswrote: > On 5 March 2014 10:04, Igor Korot wrote: > > Hi, ALL, > > Let's say I have a table with following data: > > > > field1field2field3 field4 > > 12 3 4 > > 5 6 7 8 > > > > How do I write a query which will produce the output as: > > > > 1 2 > > 5 6 > > 3 4 > > 7 8 > > > > Is it possible to write a single query which will produce the output like > > this? > > And not just in SQLite > > Possibly: > select v1, v2 from ( select 1 as ordr, f1 as v1, f2 as v2 from t union > all select 2, f3, f4 from t ) order by ordr, v1; > With UNION I will have 2 DB hits, correct? Meaning I execute the part on the left side of the UNION and then execute the right side of the UNION and then add the results together. Do I understand correctly? Thank you. > > > > > Thank you. > > Regards, > Simon > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] How to write a query
On 5 March 2014 10:04, Igor Korotwrote: > Hi, ALL, > Let's say I have a table with following data: > > field1field2field3 field4 > 12 3 4 > 5 6 7 8 > > How do I write a query which will produce the output as: > > 1 2 > 5 6 > 3 4 > 7 8 > > Is it possible to write a single query which will produce the output like > this? > And not just in SQLite Possibly: select v1, v2 from ( select 1 as ordr, f1 as v1, f2 as v2 from t union all select 2, f3, f4 from t ) order by ordr, v1; > > Thank you. Regards, Simon ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] How to write a query
Hi, ALL, Let's say I have a table with following data: field1field2field3 field4 12 3 4 5 6 7 8 How do I write a query which will produce the output as: 1 2 5 6 3 4 7 8 Is it possible to write a single query which will produce the output like this? And not just in SQLite Thank you. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] select where field in ($tcl_list) ?
I'm a relative novice to sqlite (or sql in general), but I do understand the value of variable substitution when building queries: e.g. set someValue 23 db eval {SELECT something FROM myTable WHERE value=$someValue} It feels like there should be a comparable solution for IN, passing instead a Tcl list, but I've failed to find it documented or had any trial-and-errorr success. e.g. set someTags {1 23 45} db eval {SELECT something FROM myTable WHERE value IN ($someTags)} If that's not a supported feature, what would be the recommended and safe way of building the query? As that's just a detail of a novice's attempt to implement a part of something that most likely could more correctly be done in a different way, here's more detail of the overall plan (a light-weight iTunes-like library): Given a trio of tables created as: db eval { CREATE TABLE IF NOT EXISTS media( id INTEGER PRIMARY KEY AUTOINCREMENT, filenameTEXT -- further fields removed ); CREATE TABLE IF NOT EXISTS tags( id INTEGER PRIMARY KEY AUTOINCREMENT, tag TEXT, unique(tag) ); CREATE TABLE IF NOT EXISTS tagUsage( mediaId INTEGER, tagId INTEGER, unique(mediaId,tagId) ); CREATE TRIGGER IF NOT EXISTS tagCleanup AFTER DELETE ON tags BEGIN DELETE FROM tagUsage WHERE tagUsage.tagId=OLD.id; END; } 'media' describes an audio or video file. 'tags' is just a list of words 'tagUsage' associates tags to media (a media can have zero or more tags, a tag can belong to zero or more media). At some point, I'm going to want to write queries that answer the questions: 1. What media files have at least *one* of this set of tags? 2. What media files have *all* of this set of tags? That feels like the sort of thing that it should be possible to write in a single query, but I keep coming back to an initial query, followed by some processing in code, with a follow up query. I'm sure there's a more sql-ish way. Chris ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Weird out of memory problem a prepare
On 2014/03/04 22:05, Eduardo Morras wrote: The tables have 4 rows each one, that's why I got suprised with the Out of Memory error. The biggest row has 12KB and with the join I do, shouldn't use more than 200KB. Changing the ',' with the join you propose, gives Out of Memory too. It happens on prepare phase, before binding the ? with my data. The query didn't reach the step call. Trying simple "SELECT r.name FROM resource AS r WHERE r.name = ?" gets "Out of memory" too calling preparev2. Surely something is rotten on my development platform... Yes, something is very rotten. I would start by looking for possible memory coruptors... an array being adjusted with an index out of the declared range, calling methods on an object via a reference that wasn't nulled after the actual object (via another ref) was disposed. The problem with these types of errors is the thing that breaks usually have nothing to do with the thing that causes the memory to go bad, so it is very hard to trace since we normally try to fix the thing that breaks and stares too long at the code of the thing that broke, with which there is rarely a problem. Do you use any memory profiling tools and checkers? It might give you some hints. If that query fails in an SQLite tool too, then maybe there is a problem, or if you use a custom altered version of the SQLite code. Barring that, you need to hunt down the corrupting code - Good luck! ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Weird out of memory problem a prepare
On 03/05/2014 03:05 AM, Eduardo Morras wrote: On Tue, 4 Mar 2014 15:19:24 + Simon Slavinwrote: On 4 Mar 2014, at 3:15pm, Simon Slavin wrote: On 4 Mar 2014, at 3:09pm, Eduardo Morras wrote: zSql= "SELECT r.name, s.content FROM resource AS r, static AS s WHERE (r.ids = s.ids AND r.name = ?);"; [snip] Error on query: out of memory I think this might require comparing every row in resource with every row in static. Which is a lot of temporary data to hold in memory. You might try something like SELECT r.name, s.content FROM resource AS r JOIN static AS s ON s.ids = r.ids WHERE r.name = ? and, of course, an index CREATE INDEX i1 on resource (name,ids) will make it run extremely quickly. I'll reply both answers here, if you don't mind. Thanks Simon, for the answers. The tables have 4 rows each one, that's why I got suprised with the Out of Memory error. The biggest row has 12KB and with the join I do, shouldn't use more than 200KB. Changing the ',' with the join you propose, gives Out of Memory too. It happens on prepare phase, before binding the ? with my data. The query didn't reach the step call. Trying simple "SELECT r.name FROM resource AS r WHERE r.name = ?" gets "Out of memory" too calling preparev2. Does the "out of memory" error come from sqlite3_prepare_v2() directly or from sqlite3_errmsg()? Is the sqlite3* pointer a NULL? Dan. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Virtual table API performance
On 2014/03/05 10:41, Dominique Devienne wrote: On Tue, Mar 4, 2014 at 9:41 PM, Elefterios Stamatogiannakis One thing that IMHO long term might improve the situation would be if SQLite's own "native" tables would use the same Virtual Table API,//... ...//Of course, the above is a "naive" abstract reflection which ignores the realities of VDBE, so it may sound rubbish to actual SQLite developers. Apologies for that. --DD I don't think it is rubbish at all, but maybe idealistic. The biggest problem I can see from making API's pov is that you can at any time alter, update, change the way SQLIte (or any other API) works with the base check that the input values produce the same (or maybe more-correct) results. Once you let the VT use the same API, any change is a potential change to how other people's programmed interfaces need to talk to - or get data from - the SQLite engine. This cannot simply change on a whim, so the levels of separation remain needed. That said, I'm all for making a more efficient VT API, but it would probably need to be "new" functionality since I cannot see how the existing interface could implement any of the mentioned enhancements without breaking existing behaviour. The OP's xNextRow suggestion seems a good idea, but opens up a whole can of what-ifs which other posters have alluded to, but something to that effect might be worthwhile if the efficiency bonus is significant. Cheers Ryan ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Virtual table API performance
On Tue, Mar 4, 2014 at 9:41 PM, Elefterios Stamatogiannakiswrote: > [...] Nevertheless, having people do crazy/clever hacks like that to avoid the > inefficiencies of an API, gives a hint that something might be problematic. > > In a previous email of mine, i had said half seriously/half joking about the > cost in Watts of SQLite's worldwide usage. For something that is so widely > used, even some % of efficiency improvement really makes a difference. It is > not an "SQLite destroys/saves civilization" kind of difference, but IMHO it > would be measurable in G/M Watts. The SQLite developers clearly care about performance, that much is clear. But they also care about the "lite"-ness aspects too (and so does part of its community, judging by this list). They try to find the right balance for them, with ease of implementation/testing/maintenance as the last variable of this equation. One thing that IMHO long term might improve the situation would be if SQLite's own "native" tables would use the same Virtual Table API, because then any bottleneck would apply to all "tables", not just the VT ones, and would be more likely to be removed. And as recently discussed, the VT API would be forced to acquire ways to communicate with Explain Query Plan (for example) to notice Full Scans or Covering Indexes not just for the "native" tables. VTs came after the "native" tables, so the current situation is normal, but if all table accesses, "native" or otherwise, could go thru the same API in the future (in SQLite3 or Sqlite4), it would further decouple the "front-end" from the "back-end", and ensure no performance differences between "native" and "virtual" tables. Of course, the above is a "naive" abstract reflection which ignores the realities of VDBE, so it may sound rubbish to actual SQLite developers. Apologies for that. --DD ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users