Re: [sqlite] Why would batched write operations NOT be faster than individual ones

2014-03-05 Thread Markus Schaber
Hi,

Von: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org]
> At 21:35 03/03/2014, you wrote:
> ´¯¯¯
> >RAID3-4-5 was great when disks were expensive, in 80's an 90's. Now
> >not. A minimal RAID5 needs 3 disks. A minimal RAID10 4. An enterprise
> >disk SAS 15Krpm 146 GB 6G is $350, and a not enterprise grade cheaper
> >and bigger. Now RAID1E and RAID10E give more flexibility and variable
> >security, from "paranoid" to "i don't care" grades.
> `---
> 
> The point being discussed was not on performance or cost, but on the
> imaginary fact that RAID5-6 and variations have the inherent, by-design fatal
> flaw that they break down because a parity block can be out of sync with
> corresponding data blocks. This is bullshit, period.

It is not. Period.

> Nothing in RAID5-6 design mandates serialization of writes, by far.

Yes. But I've yet to know a setup for harddisks which allows reliable
transactional writes spanning several disks. (Kinda two-phase commit 
for disk writes).

Of course, a dedicated hardware controller who issues the write requests
to the disks absolutely synchronously lowers the risk by shrinking the 
time window.

But it cannot totally eliminate it, for at least the following reasons:
- The platters are usually not completely physically in sync, so the 
  first disk may have written the block while the second disk still
  needs to wait for another 1/4 rotation for the block to be written.

- One of the disks may have internally remapped a bad sector, needing a
  seek (and thus much longer time) to write the block.

In reality, there are usually some more time variation, e. G. due to
- Both disks may be connected through the same cable, thus the requests
  to the disks need to be serialized.

- There may be other outstanding requests in the disk internal cache
  which the disk firmware might reorder in a different way.

I admit that the remaining risk may be low, but it is not zero. Period.

> It's only when cheap, unreliable hardware is put to work under below par
> software that the issue can be a real-world problem.
>
> So the rant on the design against parity-enabled RAIDs is moot, if not plain
> fallacious unless "software RAID without dedicated controller" is clearly
> mentionned.

I did mention using battery backed power as a way to mitigate the risk.

And as far as I know, even the most expensive hardware RAID controllers
and disks do not yet support multi-disk transactions, so your reference
to not-yet existing hardware is moot.

> About SAS disks: they have actual very high reliability and don't lie,
> contrary to SATA disks (on both points).
>
> This is not a war about mine being bigger, but it's better to have facts
> stated right.

I fully agree there.

> All high-end reliable machines and storage subsystems only run
> parity-enabled RAID levels and this thechnology isn't going to disappear
> tomorrow.

I doubt that _all_ those machines exclusively run on parity-enabled RAID 
levels, but I'm strongly interested in a proof of your "fact" here.

I remember reading that PostgreSQL and Oracle recommend using mirroring based
levels instead of parity-enabled ones for performance reasons, so I'm really
curious to read about how you back up your claim.


Best regards

Markus Schaber

CODESYS® a trademark of 3S-Smart Software Solutions GmbH

Inspiring Automation Solutions

3S-Smart Software Solutions GmbH
Dipl.-Inf. Markus Schaber | Product Development Core Technology
Memminger Str. 151 | 87439 Kempten | Germany
Tel. +49-831-54031-979 | Fax +49-831-54031-50

E-Mail: m.scha...@codesys.com | Web: http://www.codesys.com | CODESYS store: 
http://store.codesys.com
CODESYS forum: http://forum.codesys.com

Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade 
register: Kempten HRB 6186 | Tax ID No.: DE 167014915

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] SQLITE_MAX_PAGE_COUNT

2014-03-05 Thread Raheel Gupta
I will make the changes as per my knowledge and send it to all  on this
list to see if it helps in anyway.



On Thu, Mar 6, 2014 at 5:25 AM, Roger Binns  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> On 05/03/14 10:59, Raheel Gupta wrote:
> > If you point out to me the changes required I will do it and have it
> > reviewed.
>
> The changes required are to update the test suites (there are several) to
> hit/cross the current limit, to modify all relevant code including any
> code that calls that code, to audit for overflows, to provide a new api
> and tests for that.
>
> And to ensure that stays maintained for the future lifetime of SQLite 3.
>
> Roger
>
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.14 (GNU/Linux)
>
> iEYEARECAAYFAlMXuWsACgkQmOOfHg372QRWcgCgmLmuv+d7Mxu5sfn9XFxAwx8N
> xagAoIpbTOEPWtPqXRsQSBuIjgV4NYhl
> =RMyF
> -END PGP SIGNATURE-
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] Issues after upgrade to 3.8.2

2014-03-05 Thread Joe Mistachkin

Denis Gladkikh wrote:
> 
> I still have this issue with SQLite 3.8.3.1, I could fix my case my
> changing line
> 
> "LastPlayed] = (select case when [UserPlaylist].[LastPlayed] >
> s.[LastPlayed] then [UserPlaylist].[LastPlayed] else s.[LastPlayed] end
> from [Song] as s where s.[SongId] = new.[SongId]),"
> 

Thanks for the detailed report.  This issue has been fixed on trunk:

https://www.sqlite.org/src/info/5bcd0b1ca5

--
Joe Mistachkin

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] Troubleshooting query performance

2014-03-05 Thread Joseph L. Casale
> Don't put the PRIMARY KEY as the first column of your index.  Ever.  This
> applies to all SQL database engines, not just SQLite.
> 
> For that matter, don't put the PRIMARY KEY anywhere in your index.  The
> PRIMARY KEY will be added automatically at the end, where it belongs.
> 
> If you remove the "id," from all of your indices, I think your performance
> will probably improve dramatically.

Richard, Simon,
So much for testing at the console over a remote session, while I was used to
waiting for a single row, the new query increased by so much it overwhelmed
my display, heh.

Nice and thanks for that tip!
jlc
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] Troubleshooting query performance

2014-03-05 Thread Simon Slavin

On 6 Mar 2014, at 12:20am, Joseph L. Casale  wrote:

> SELECT a.name, p.value
>  FROM p_attribute p
>  JOIN attribute a
>   ON a.id=p.aid
> WHERE p.pid=?
> 
> This returns all relevant rows I need, where table profile has ~6000 rows,
> p_attribute has ~ 170k and attribute has ~60 rows.
> 
> Analyze has been run, explain query plan shows:
> recno selectedorder   fromdetail
> 0 0   0   SCAN TABLE p_attribute AS p
> 0 1   1   SEARCH TABLE attribute AS a USING INTEGER PRIMARY KEY 
> (rowid=?)
> 
> Any pointers as to what may not be optimal?

You don't have ideal indexes.  The main task of this SELECT is to look for 
certain values of p_attribute.pid.  You don't have an index that makes this 
easy.  Do something like

CREATE INDEX p_attribute_idx_0 ON p_attribute (pid, id)

Once you've done this do an ANALYZE just to make sure, then try the SELECT 
again.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] Troubleshooting query performance

2014-03-05 Thread Richard Hipp
On Wed, Mar 5, 2014 at 7:20 PM, Joseph L. Casale
wrote:

> Hey guys,
>
> I have a query that's giving me abysmal performance and it's not
> immediately
> obvious to me as to what's wrong with the table structure to cause this.
>
> CREATE TABLE profile (
> id   INTEGER PRIMARY KEY AUTOINCREMENT,
> name TEXT UNIQUE NOT NULL
> );
> CREATE INDEX profile_idx_0 ON profile (
> id,
> name
> );
>

Don't put the PRIMARY KEY as the first column of your index.  Ever.  This
applies to all SQL database engines, not just SQLite.

For that matter, don't put the PRIMARY KEY anywhere in your index.  The
PRIMARY KEY will be added automatically at the end, where it belongs.

If you remove the "id," from all of your indices, I think your performance
will probably improve dramatically.


>
> CREATE TABLE p_attribute (
> id   INTEGER PRIMARY KEY AUTOINCREMENT,
> pid  INTEGER NOT NULL
>  REFERENCES profile (id)
>  ON DELETE CASCADE,
> aid INTEGER NOT NULL
>  REFERENCES attribute (id)
>  ON DELETE CASCADE,
> valueTEXT
> );
> CREATE INDEX p_attribute_idx_0 ON p_attribute (
> id,
> pid,
> aid
> );
>
> CREATE TABLE attribute (
> id   INTEGER PRIMARY KEY AUTOINCREMENT,
> name TEXT UNIQUE NOT NULL,
> CHECK(UPPER(name) = name)
> );
> CREATE INDEX attribute_idx_0 ON attribute (
> id,
> name
> );
>
>
> SELECT a.name, p.value
>   FROM p_attribute p
>   JOIN attribute a
>ON a.id=p.aid
>  WHERE p.pid=?
>
> This returns all relevant rows I need, where table profile has ~6000 rows,
> p_attribute has ~ 170k and attribute has ~60 rows.
>
> Analyze has been run, explain query plan shows:
> recno   selectedorder   fromdetail
> 0   0   0   SCAN TABLE p_attribute AS p
> 0   1   1   SEARCH TABLE attribute AS a USING INTEGER PRIMARY
> KEY (rowid=?)
>
> Any pointers as to what may not be optimal?
> Thanks,
> jlc
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


[sqlite] Troubleshooting query performance

2014-03-05 Thread Joseph L. Casale
Hey guys,

I have a query that's giving me abysmal performance and it's not immediately
obvious to me as to what's wrong with the table structure to cause this.

CREATE TABLE profile (
id   INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT UNIQUE NOT NULL
);
CREATE INDEX profile_idx_0 ON profile (
id,
name
);

CREATE TABLE p_attribute (
id   INTEGER PRIMARY KEY AUTOINCREMENT,
pid  INTEGER NOT NULL
 REFERENCES profile (id)
 ON DELETE CASCADE,
aid INTEGER NOT NULL
 REFERENCES attribute (id)
 ON DELETE CASCADE,
valueTEXT
);
CREATE INDEX p_attribute_idx_0 ON p_attribute (
id,
pid,
aid
);

CREATE TABLE attribute (
id   INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT UNIQUE NOT NULL,
CHECK(UPPER(name) = name)
);
CREATE INDEX attribute_idx_0 ON attribute (
id,
name
);


SELECT a.name, p.value
  FROM p_attribute p
  JOIN attribute a
   ON a.id=p.aid
 WHERE p.pid=?

This returns all relevant rows I need, where table profile has ~6000 rows,
p_attribute has ~ 170k and attribute has ~60 rows.

Analyze has been run, explain query plan shows:
recno   selectedorder   fromdetail
0   0   0   SCAN TABLE p_attribute AS p
0   1   1   SEARCH TABLE attribute AS a USING INTEGER PRIMARY KEY 
(rowid=?)

Any pointers as to what may not be optimal?
Thanks,
jlc
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] SQLITE_MAX_PAGE_COUNT

2014-03-05 Thread Roger Binns
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 05/03/14 10:59, Raheel Gupta wrote:
> If you point out to me the changes required I will do it and have it 
> reviewed.

The changes required are to update the test suites (there are several) to
hit/cross the current limit, to modify all relevant code including any
code that calls that code, to audit for overflows, to provide a new api
and tests for that.

And to ensure that stays maintained for the future lifetime of SQLite 3.

Roger

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.14 (GNU/Linux)

iEYEARECAAYFAlMXuWsACgkQmOOfHg372QRWcgCgmLmuv+d7Mxu5sfn9XFxAwx8N
xagAoIpbTOEPWtPqXRsQSBuIjgV4NYhl
=RMyF
-END PGP SIGNATURE-
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] About "speed"

2014-03-05 Thread big stone
Hi

*Elefterios, Simon,*


*Wes McKinney gave us :- a fully detailed benchmark case (data +
reproducible test),*


*- where SQLite was :  . abnormally less good than Postgresql (so could be
better),*


*  . SQLdatabase in general were abnormally less good,   . a hint
"vertica"was given.*


*Maybe it is a 'golden' benchmark, as it hints a possible 10x margin of
improvement for SQLite. *
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] SQLite version 3.8.4 release schedule

2014-03-05 Thread tonyp

I noticed the message:

Enter SQL statements terminated with a ";"

was removed from the shell.  Is this supposed to be in the right direction 
when a red message is added to warn newbies to SQLite about working with 
in-memory database, yet they are assumed to know that SQL commands end with 
";", when up to now everybody (even seasoned SQLiters) had to be reminded 
about this?


My €0.02 


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] Syntax diagram missing in sqlite3 docs

2014-03-05 Thread Zsbán Ambrus
On 2/20/14, Zsbán Ambrus  wrote:
> The page "http://sqlite.org/lang_transaction.html; should show the
> syntax diagrams for commit-stmt and rollback-stmt.

It seems that these bugs are now fixed in the draft documentation for
sqlite 3.8.4.  Thank you, sqlite maintainers.

Ambrus
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] About "speed"

2014-03-05 Thread big stone
ok,

Just updated with 3.8.4beta of  2014-03-05.

I also re-did some previous measures as :
- testing method improved a little,
- I measured more carefully that SQLite has also a sort of caching benefit,
when you run a query twice on windows7.


Regards,
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] Virtual table API performance

2014-03-05 Thread Alek Paunov

On 05.03.2014 11:02, RSmith wrote:


On 2014/03/05 10:41, Dominique Devienne wrote:

On Tue, Mar 4, 2014 at 9:41 PM, Elefterios Stamatogiannakis

One thing that IMHO long term might improve the situation would be if
SQLite's own "native" tables would use the same Virtual Table API,//...

...//Of course, the above is a "naive" abstract reflection which ignores
the realities of VDBE, so it may sound rubbish to actual SQLite
developers. Apologies for that. --DD


I don't think it is rubbish at all, but maybe idealistic.  The biggest
problem I can see from making API's pov is that you can at any time
alter, update, change the way SQLIte (or any other API) works with the
base check that the input values produce the same (or maybe
more-correct) results.  Once you let the VT use the same API, any change
is a potential change to how other people's programmed interfaces need
to talk to - or get data from - the SQLite engine. This cannot simply
change on a whim, so the levels of separation remain needed.

That said, I'm all for making a more efficient VT API, but it would
probably need to be "new" functionality since I cannot see how the
existing interface could implement any of the mentioned enhancements
without breaking existing behaviour. The OP's xNextRow suggestion seems
a good idea, but opens up a whole can of what-ifs which other posters
have alluded to, but something to that effect might be worthwhile if the
efficiency bonus is significant.



The whole thread so far is based on the OP observations in mixed C/PyPy 
and apsw/CPython environments (as being said already, we suffering the 
noise in both cases).


To be helpful to the SQLite team, before proposing any changes, please 
let someone show some well designed, pure C vtable implementation 
demonstrating the possible vtable interface inefficiency.


Let's remember that all xNextRow, xNextPage optimizations are 
applicable only for "select *" cases, not in the general "select f(x), 
y" case.


Kind regards,
Alek

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] About "speed"

2014-03-05 Thread Stephan Beal
On Wed, Mar 5, 2014 at 7:25 PM, Richard Hipp  wrote:

> MySQL does very well on query 8 which is a repeat of query 6.  This might
> be because MySQL implements a query cache.  It remembers the result of each
> query and if that query occurs again, without an intervening INSERT,
> DELETE, or UPDATE on one of the tables used by the query, just echos the
> previous answer.
>

Counterpoint: MySQL's wire protocol does not support concurrent SELECTs to
be running on the same connection, meaning client code has to do a lot more
work to collect all row data in cases where sqlite3 can easily run queries
while stepping over another. In my experience, writing code with their C
API takes at least 3-5 times longer because it's just generally a pain to
work with (it requires a great deal of the user). sqlite3 wins hands-down
on sanity/usability of the C API and, consequently, development speed. That
doesn't generally apply to script bindings (where using MySQL is also
easy), but it does to the native C APIs.

-- 
- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] SQLITE_MAX_PAGE_COUNT

2014-03-05 Thread Raheel Gupta
Sir, will it require much change ?
If you point out to me the changes required I will do it and have it
reviewed.

AT the moment I am checking code related to the following variables :
mxPgno
sqlite3PagerMaxPageCount (Function)
pageSize


On Thu, Mar 6, 2014 at 12:06 AM, Richard Hipp  wrote:

> On Wed, Mar 5, 2014 at 1:24 PM, Raheel Gupta  wrote:
>
> > SQLITE_MAX_PAGE_COUNT is 2147483646.
> > After looking at the code, Pgno is a u32.
> > So isnt SQLITE_MAX_PAGE_COUNT capable of actually being 4294967292 ?
> >
>
> We have your request.  But as it will require a lot of testing, and because
> it is unlikely to actually be useful to anybody, it is a low priority.
>
>
> --
> D. Richard Hipp
> d...@sqlite.org
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] SQLITE_MAX_PAGE_COUNT

2014-03-05 Thread Richard Hipp
On Wed, Mar 5, 2014 at 1:24 PM, Raheel Gupta  wrote:

> SQLITE_MAX_PAGE_COUNT is 2147483646.
> After looking at the code, Pgno is a u32.
> So isnt SQLITE_MAX_PAGE_COUNT capable of actually being 4294967292 ?
>

We have your request.  But as it will require a lot of testing, and because
it is unlikely to actually be useful to anybody, it is a low priority.


-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] About "speed"

2014-03-05 Thread Richard Hipp
On Wed, Mar 5, 2014 at 9:29 AM, big stone  wrote:

> Timing updates with Mysql 5.6.16
>

MySQL does very well on query 8 which is a repeat of query 6.  This might
be because MySQL implements a query cache.  It remembers the result of each
query and if that query occurs again, without an intervening INSERT,
DELETE, or UPDATE on one of the tables used by the query, just echos the
previous answer.

I think it would be interesting to do a single simple INSERT, DELETE, or
UPDATE on the FEC table after test 8, then repeat the same query again, to
see if MySQL is able to maintain the spectacular performance of test 8.

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


[sqlite] SQLITE_MAX_PAGE_COUNT

2014-03-05 Thread Raheel Gupta
SQLITE_MAX_PAGE_COUNT is 2147483646.
After looking at the code, Pgno is a u32.
So isnt SQLITE_MAX_PAGE_COUNT capable of actually being 4294967292 ?

Please correct me if I am wrong
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] sqlite issue when using AIX

2014-03-05 Thread Richard Hipp
On Tue, Mar 4, 2014 at 11:00 AM, Peter Simpson
wrote:

> Hi,
>
> We are currently using SQLite 3.8.2.
>
> When using this on AIX, we encountered an issue with an "unsigned char" in
> the sqliteProcessJoin procedure in sqlite3.c.
>

Can you please try the latest 3.8.4 beta on AIX for us, and verify that the
problem is now fixed.  We (the SQLite developers) do not ourselves have
access to an AIX machine for testing.


-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] SQLite version 3.8.4 release schedule

2014-03-05 Thread Richard Hipp
On Wed, Mar 5, 2014 at 12:53 PM, big stone  wrote:

>
> ==> Is it expected to have a big speed-up in index creation ?
>

Yes.

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] SQLite version 3.8.4 release schedule

2014-03-05 Thread big stone
Hello,

Comparing 3.8.3 to 3.8.4beta fo today :
- both compiled in  -o2 mode and running in ":memory:" ,
- on a windows pc.

Changes :
* 8% quicker on Recursive CTE
   (sudoku test of  http://www.sqlite.org/lang_with.html)
* 12% quicker on Index Creation over a 5M record table
(
ftp://ftp.fec.gov/FEC/Presidential_Map/2012/P0001/P0001-ALL.zip)

==> Is it expected to have a big speed-up in index creation ? Or is it
noise ?


Regards,
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] sqlite issue when using AIX

2014-03-05 Thread Ross Hayden
On Tue, Mar 4, 2014 at 10:00 AM, Peter Simpson
wrote:


> When using this on AIX, we encountered an issue with an "unsigned char" in
> the sqliteProcessJoin procedure in sqlite3.c.
>
> For cases where "pSrc->nSrc" was 0, then "pSrc->nSrc-1" was being treated
> as positive (since nSrc is defined as an unsigned char, u8), and the for
> loop was inadvertently being entered.


> Peter Simpson (Hopewiser)]
>

If you're using IBM's own compilers, specify "xlc" as your compiler (CC=xlc
./configure). The default behaviors of good old "cc" can be a bit dated.

--
Ross
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


[sqlite] SQLite version 3.8.4 release schedule

2014-03-05 Thread Richard Hipp
A number of bugs have recently been revealed in SQLite.  All are obscure.
Nevertheless, we want to accelerate the release of version 3.8.4 in order
to get the fixes in circulation.

To this end, SQLite version 3.8.4 should now be considered "in beta".
Amalgamation snapshots are now available on the download page (
http://www.sqlite.org/download.html) and will be updated regularly.
Preliminary documentation for 3.8.4 is available at
http://www.sqlite.org/draft with a change log at
http://www.sqlite.org/draft/releaselog/3_8_4.html that will also be updated
regularly. A status board (http://www.sqlite.org/checklists/3080400) is now
up and when it goes all-green, the release will occur.  The anticipated
beta period is about 7 to 10 days.

Please try out SQLite 3.8.4 beta in your applications and report any issues
found to this mailing list, or directly to me, as soon as possible.  If you
are able to report on performance differences (hopefully performance
*improvements*) then please do.

There are no significant new features in SQLite 3.8.4.  This will be
primarily a performance-enhancement and bug-fix release.

Thanks for your help.
-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] select where field in ($tcl_list) ?

2014-03-05 Thread RSmith


On 2014/03/05 17:05, Chris wrote:


Ok, fair enough. I thought that in the same way that sqlite looks for
binary vs. string representations of referenced vars and has alternative
ways of specifying variable to bind to ('@', ':'), it might also spot a
list object and internally expand it to "elem_0,elem_1,elem_2". Would
that be a useful feature, or does it introduce opportunities to draw the
wrong conclusion?


Most high-level languages have some sort of list expansion in list objects 
which obviates the need.

You should for instance be bale to say something like this in most languages:

mylist = New(List);
mylist.add("Item1");
mylist.add("Item2");
...
mylist.add("Item6");
mylist.add("Item7");
etc..

mylist.separator=",";   // Comma
mylist.QuoteChar="'";  // Single Quote


So the entire Query bit can become something simple like:

qry = "SELECT a,b FROM t WHERE id IN ("+mylist.asString+");"

or something to that effect
Adding any SQLite function to try and break down your sent variable into lists would probably pay a much higher price in processing 
and added code - moreso than it really being difficult to do or in violation of some method or way of doing.


If those lists get really really long or convoluted, it is best to break them into tables of their own via some query and then use 
the base query from that table as the list specifier.


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] select where field in ($tcl_list) ?

2014-03-05 Thread Chris
> On Wed, Mar 5, 2014 at 4:59 AM, Chris  wrote:
> 
> > I'm a relative novice to sqlite (or sql in general), but I do understand
> > the value of variable substitution when building queries:
> >
> > e.g.
> > set someValue 23
> > db eval {SELECT something FROM myTable WHERE value=$someValue}
> >
> > It feels like there should be a comparable solution for IN, passing
> > instead a Tcl list, but I've failed to find it documented or had any
> > trial-and-errorr success.
> >
> > e.g.
> > set someTags {1 23 45}
> > db eval {SELECT something FROM myTable WHERE value IN ($someTags)}
> >
> 
> Can't do this.  SQL has no "list" or "array" datatype.  So $someTags cannot
> be translated into an SQL value.
> 
> One work-around:
> 
> db transaction {
>db eval {CREATE TEMP TABLE someTags(x INTEGER PRIMARY KEY);}
>foreach x $someTags {db eval {INSERT INTO someTags VALUES($x)}}
> }
> db eval {SELECT something FROM myTable WHERE value IN someTags}
> 
> -- 
> D. Richard Hipp
> drh at sqlite.org

Ok, fair enough. I thought that in the same way that sqlite looks for
binary vs. string representations of referenced vars and has alternative
ways of specifying variable to bind to ('@', ':'), it might also spot a
list object and internally expand it to "elem_0,elem_1,elem_2". Would
that be a useful feature, or does it introduce opportunities to draw the
wrong conclusion?

Chris
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] About "speed"

2014-03-05 Thread Richard Hipp
On Wed, Mar 5, 2014 at 9:29 AM, big stone  wrote:

> Timing updates with Mysql 5.6.16
>

I wonder if you could update the timings for the current SQLite 3.8.4 beta?

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] select where field in ($tcl_list) ?

2014-03-05 Thread Richard Hipp
On Wed, Mar 5, 2014 at 4:59 AM, Chris  wrote:

> I'm a relative novice to sqlite (or sql in general), but I do understand
> the value of variable substitution when building queries:
>
> e.g.
> set someValue 23
> db eval {SELECT something FROM myTable WHERE value=$someValue}
>
> It feels like there should be a comparable solution for IN, passing
> instead a Tcl list, but I've failed to find it documented or had any
> trial-and-errorr success.
>
> e.g.
> set someTags {1 23 45}
> db eval {SELECT something FROM myTable WHERE value IN ($someTags)}
>

Can't do this.  SQL has no "list" or "array" datatype.  So $someTags cannot
be translated into an SQL value.

One work-around:

db transaction {
   db eval {CREATE TEMP TABLE someTags(x INTEGER PRIMARY KEY);}
   foreach x $someTags {db eval {INSERT INTO someTags VALUES($x)}}
}
db eval {SELECT something FROM myTable WHERE value IN someTags}




-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] select where field in ($tcl_list) ?

2014-03-05 Thread Chris
On Wed, Mar 5, 2014, at 09:59 AM, Chris wrote:
> I'm a relative novice to sqlite (or sql in general), but I do understand
> the value of variable substitution when building queries:
> 
> e.g.
> set someValue 23
> db eval {SELECT something FROM myTable WHERE value=$someValue}
> 
> It feels like there should be a comparable solution for IN, passing
> instead a Tcl list, but I've failed to find it documented or had any
> trial-and-errorr success.
> 
> e.g.
> set someTags {1 23 45}
> db eval {SELECT something FROM myTable WHERE value IN ($someTags)}
> 
> If that's not a supported feature, what would be the recommended and
> safe way of building the query?
> 
> 
> As that's just a detail of a novice's attempt to implement a part of
> something that most likely could more correctly be done in a different
> way, here's more detail of the overall plan (a light-weight iTunes-like
> library):
> 
> Given a trio of tables created as:
> 
> db eval {
> CREATE TABLE IF NOT EXISTS media(
> id  INTEGER PRIMARY KEY AUTOINCREMENT,
> filenameTEXT
> -- further fields removed
> );
> CREATE TABLE IF NOT EXISTS tags(
> id  INTEGER PRIMARY KEY AUTOINCREMENT,
> tag TEXT,
> unique(tag)
> );
> CREATE TABLE IF NOT EXISTS tagUsage(
> mediaId INTEGER,
> tagId   INTEGER,
> unique(mediaId,tagId)
> );
> CREATE TRIGGER IF NOT EXISTS tagCleanup AFTER DELETE ON tags
> BEGIN
> DELETE FROM tagUsage WHERE tagUsage.tagId=OLD.id;
> END;
> }
> 
> 'media' describes an audio or video file.
> 'tags' is just a list of words
> 'tagUsage' associates tags to media (a media can have zero or more tags,
> a tag can belong to zero or more media).
> 
> At some point, I'm going to want to write queries that answer the
> questions:
> 
> 1. What media files have at least *one* of this set of  tags?
> 2. What media files have *all* of this set of tags?
> 
> That feels like the sort of thing that it should be possible to write in
> a single query, but I keep coming back to an initial query, followed by
> some processing in code, with a follow up query. I'm sure there's a more
> sql-ish way.
> 
> Chris

Having realised the above sounds very much like "please do my homework
assignment for me", I should point out that that's not the case. I've
been a C, C++ and Tcl developer for many years, but have only ever
scratched the surface of sqlite. This is me trying to delve a little
deeper while avoiding going down too many wrong turns at the beginning.

Chris
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] About "speed"

2014-03-05 Thread big stone
Timing updates with Mysql 5.6.16

test =
https://raw.github.com/stonebig/ztest_donotuse/master/benchmark_test01.txt

results =
https://github.com/stonebig/ztest_donotuse/blob/master/benchmark_test01_measures.GIF?raw=true
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] sqlite issue when using AIX

2014-03-05 Thread Clemens Ladisch
Peter Simpson wrote:
> When using this on AIX, we encountered an issue with an "unsigned char".
>
> For cases where "nSrc" was 0, then "nSrc-1" was being treated as positive
> (since nSrc is defined as an unsigned char, u8)

This violates the C standard (any C standard).  Does you compiler
(whatever it is) claim conformance to a standard?  Might this be just
some optimization bug?


Regards,
Clemens
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


[sqlite] sqlite issue when using AIX

2014-03-05 Thread Peter Simpson

Hi,

We are currently using SQLite 3.8.2.

When using this on AIX, we encountered an issue with an "unsigned char" 
in the sqliteProcessJoin procedure in sqlite3.c.


The issue occurred with the following line (99798) :-

for(i=0; inSrc-1; i++, pRight++, pLeft++){

For cases where "pSrc->nSrc" was 0, then "pSrc->nSrc-1" was being 
treated as positive (since nSrc is defined as an unsigned char, u8), and 
the for loop was inadvertently being entered. To overcome this, we have 
resorted to casting the relevant code as follows :-


for(i=0; i<((int)pSrc->nSrc)-1; i++, pRight++, pLeft++){

Apologies if this has already been reported. I did check through the 
current tickets searching for AIX, and didn't find anything so I'm 
assuming that it hasn't already been reported.


Regards,

Peter Simpson (Hopewiser)



_
Hopewiser Ltd, Merlin Court, Atlantic Street, Altrincham, WA14 5NL
Reg in England, number 1621544
Tel: 0161 924 2800, Fax: 0161 924 2809, Web: http://www.hopewiser.com/

The information contained in this email is intended only for the named
recipient(s) and may be confidential and/or privileged. Unauthorised
use or reproduction (including storage or re-distribution in any media)
is prohibited.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] Solved Weird out of memory problem a prepare

2014-03-05 Thread Eduardo Morras
On Wed, 05 Mar 2014 11:12:45 +0200
RSmith  wrote:

> If that query fails in an SQLite tool too, then maybe there is a
> problem, or if you use a custom altered version of the SQLite code.
> Barring that, you need to hunt down the corrupting code - Good luck!

I find it and solve. It's a macro expansion in one function extension that 
collides and redefines a #define in sqlite3.h

Thanks to all.

---   ---
Eduardo Morras 
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] How to write a query

2014-03-05 Thread RSmith


On 2014/03/05 12:24, Igor Korot wrote:
With UNION I will have 2 DB hits, correct? Meaning I execute the part on the left side of the UNION and then execute the right 
side of the UNION and then add the results together. Do I understand correctly? Thank you. 


Yes. Obviously you need to have the same number of field results for the part before the UNION than the part after the UNION to be 
able to be one query result list at the end without nulls and such.


Also, when you don't want duplicates, just use "UNION", if you want all values to be listed, even if they are duplicates, use "UNION 
ALL"



___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] How to write a query

2014-03-05 Thread RSmith


On 2014/03/05 12:04, Igor Korot wrote:

Hi, ALL,
Let's say I have a table with following data:

field1field2field3   field4
12  3 4
5   6   7 8

How do I write a query which will produce the output as:

1 2
5 6
3 4
7 8

Is it possible to write a single query which will produce the output like
this?
And not just in SQLite


SELECT field1, field2 FROM tbl
UNION ALL
SELECT field3, field4 FROM tbl

I can't imagine a use for this... but hey



___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] How to write a query

2014-03-05 Thread Simon Davies
On 5 March 2014 10:24, Igor Korot  wrote:
> Hi, Simon,
>
.
.
.
>>
>> Possibly:
>> select v1, v2 from ( select 1 as ordr, f1 as v1, f2 as v2 from t union
>> all select 2, f3, f4 from t ) order by ordr, v1;
>>
>
> With UNION I will have 2 DB hits, correct?
> Meaning I execute the part on the left side of the UNION and then execute
> the right side of the UNION
> and then add the results together. Do I understand correctly?

I'm not sure. It is a single SQL statement

>
> Thank you.
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] How to write a query

2014-03-05 Thread Igor Korot
Hi, Simon,


On Wed, Mar 5, 2014 at 2:21 AM, Simon Davies
wrote:

> On 5 March 2014 10:04, Igor Korot  wrote:
> > Hi, ALL,
> > Let's say I have a table with following data:
> >
> > field1field2field3   field4
> > 12  3 4
> > 5   6   7 8
> >
> > How do I write a query which will produce the output as:
> >
> > 1 2
> > 5 6
> > 3 4
> > 7 8
> >
> > Is it possible to write a single query which will produce the output like
> > this?
> > And not just in SQLite
>
> Possibly:
> select v1, v2 from ( select 1 as ordr, f1 as v1, f2 as v2 from t union
> all select 2, f3, f4 from t ) order by ordr, v1;
>

With UNION I will have 2 DB hits, correct?
Meaning I execute the part on the left side of the UNION and then execute
the right side of the UNION
and then add the results together. Do I understand correctly?

Thank you.


>
> >
> > Thank you.
>
> Regards,
> Simon
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] How to write a query

2014-03-05 Thread Simon Davies
On 5 March 2014 10:04, Igor Korot  wrote:
> Hi, ALL,
> Let's say I have a table with following data:
>
> field1field2field3   field4
> 12  3 4
> 5   6   7 8
>
> How do I write a query which will produce the output as:
>
> 1 2
> 5 6
> 3 4
> 7 8
>
> Is it possible to write a single query which will produce the output like
> this?
> And not just in SQLite

Possibly:
select v1, v2 from ( select 1 as ordr, f1 as v1, f2 as v2 from t union
all select 2, f3, f4 from t ) order by ordr, v1;

>
> Thank you.

Regards,
Simon
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


[sqlite] How to write a query

2014-03-05 Thread Igor Korot
Hi, ALL,
Let's say I have a table with following data:

field1field2field3   field4
12  3 4
5   6   7 8

How do I write a query which will produce the output as:

1 2
5 6
3 4
7 8

Is it possible to write a single query which will produce the output like
this?
And not just in SQLite

Thank you.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


[sqlite] select where field in ($tcl_list) ?

2014-03-05 Thread Chris
I'm a relative novice to sqlite (or sql in general), but I do understand
the value of variable substitution when building queries:

e.g.
set someValue 23
db eval {SELECT something FROM myTable WHERE value=$someValue}

It feels like there should be a comparable solution for IN, passing
instead a Tcl list, but I've failed to find it documented or had any
trial-and-errorr success.

e.g.
set someTags {1 23 45}
db eval {SELECT something FROM myTable WHERE value IN ($someTags)}

If that's not a supported feature, what would be the recommended and
safe way of building the query?


As that's just a detail of a novice's attempt to implement a part of
something that most likely could more correctly be done in a different
way, here's more detail of the overall plan (a light-weight iTunes-like
library):

Given a trio of tables created as:

db eval {
CREATE TABLE IF NOT EXISTS media(
id  INTEGER PRIMARY KEY AUTOINCREMENT,
filenameTEXT
-- further fields removed
);
CREATE TABLE IF NOT EXISTS tags(
id  INTEGER PRIMARY KEY AUTOINCREMENT,
tag TEXT,
unique(tag)
);
CREATE TABLE IF NOT EXISTS tagUsage(
mediaId INTEGER,
tagId   INTEGER,
unique(mediaId,tagId)
);
CREATE TRIGGER IF NOT EXISTS tagCleanup AFTER DELETE ON tags
BEGIN
DELETE FROM tagUsage WHERE tagUsage.tagId=OLD.id;
END;
}

'media' describes an audio or video file.
'tags' is just a list of words
'tagUsage' associates tags to media (a media can have zero or more tags,
a tag can belong to zero or more media).

At some point, I'm going to want to write queries that answer the
questions:

1. What media files have at least *one* of this set of  tags?
2. What media files have *all* of this set of tags?

That feels like the sort of thing that it should be possible to write in
a single query, but I keep coming back to an initial query, followed by
some processing in code, with a follow up query. I'm sure there's a more
sql-ish way.

Chris
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] Weird out of memory problem a prepare

2014-03-05 Thread RSmith


On 2014/03/04 22:05, Eduardo Morras wrote:


The tables have 4 rows each one, that's why I got suprised with the Out of 
Memory error. The biggest row has 12KB and with the join I do, shouldn't use 
more than 200KB.

Changing the ',' with the join you propose, gives Out of Memory too. It happens 
on prepare phase, before binding the ? with my data. The query didn't reach the 
step call.

Trying simple "SELECT r.name FROM resource AS r WHERE r.name = ?" gets "Out of 
memory" too calling preparev2.

Surely something is rotten on my development platform...


Yes, something is very rotten. I would start by looking for possible memory coruptors... an array being adjusted with an index out 
of the declared range, calling methods on an object via a reference that wasn't nulled after the actual object (via another ref) was 
disposed. The problem with these types of errors is the thing that breaks usually have nothing to do with the thing that causes the 
memory to go bad, so it is very hard to trace since we normally try to fix the thing that breaks and stares too long at the code of 
the thing that broke, with which there is rarely a problem.


Do you use any memory profiling tools and checkers? It might give you some 
hints.

If that query fails in an SQLite tool too, then maybe there is a problem, or if you use a custom altered version of the SQLite code. 
Barring that, you need to hunt down the corrupting code - Good luck!




___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] Weird out of memory problem a prepare

2014-03-05 Thread Dan Kennedy

On 03/05/2014 03:05 AM, Eduardo Morras wrote:

On Tue, 4 Mar 2014 15:19:24 +
Simon Slavin  wrote:


On 4 Mar 2014, at 3:15pm, Simon Slavin  wrote:


On 4 Mar 2014, at 3:09pm, Eduardo Morras  wrote:


zSql= "SELECT r.name, s.content FROM resource AS r, static AS s
WHERE (r.ids = s.ids AND r.name = ?);";

[snip]

Error on query: out of memory

I think this might require comparing every row in resource with
every row in static.  Which is a lot of temporary data to hold in
memory.

You might try something like

SELECT r.name, s.content FROM resource AS r JOIN static AS s ON
s.ids = r.ids  WHERE r.name = ?

and, of course, an index

CREATE INDEX i1 on resource (name,ids)

will make it run extremely quickly.

I'll reply both answers here, if you don't mind.

Thanks Simon, for the answers.

The tables have 4 rows each one, that's why I got suprised with the Out of 
Memory error. The biggest row has 12KB and with the join I do, shouldn't use 
more than 200KB.

Changing the ',' with the join you propose, gives Out of Memory too. It happens 
on prepare phase, before binding the ? with my data. The query didn't reach the 
step call.

Trying simple "SELECT r.name FROM resource AS r WHERE r.name = ?" gets "Out of 
memory" too calling preparev2.


Does the "out of memory" error come from sqlite3_prepare_v2()
directly or from sqlite3_errmsg()? Is the sqlite3* pointer
a NULL?

Dan.



___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] Virtual table API performance

2014-03-05 Thread RSmith


On 2014/03/05 10:41, Dominique Devienne wrote:

On Tue, Mar 4, 2014 at 9:41 PM, Elefterios Stamatogiannakis

One thing that IMHO long term might improve the situation would be if
SQLite's own "native" tables would use the same Virtual Table API,//...

...//Of course, the above is a "naive" abstract reflection which ignores
the realities of VDBE, so it may sound rubbish to actual SQLite
developers. Apologies for that. --DD


I don't think it is rubbish at all, but maybe idealistic.  The biggest problem I can see from making API's pov is that you can at 
any time alter, update, change the way SQLIte (or any other API) works with the base check that the input values produce the same 
(or maybe more-correct) results.  Once you let the VT use the same API, any change is a potential change to how other people's 
programmed interfaces need to talk to - or get data from - the SQLite engine. This cannot simply change on a whim, so the levels of 
separation remain needed.


That said, I'm all for making a more efficient VT API, but it would probably need to be "new" functionality since I cannot see how 
the existing interface could implement any of the mentioned enhancements without breaking existing behaviour. The OP's xNextRow 
suggestion seems a good idea, but opens up a whole can of what-ifs which other posters have alluded to, but something to that effect 
might be worthwhile if the efficiency bonus is significant.


Cheers
Ryan
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] Virtual table API performance

2014-03-05 Thread Dominique Devienne
On Tue, Mar 4, 2014 at 9:41 PM, Elefterios Stamatogiannakis
 wrote:
> [...] Nevertheless, having people do crazy/clever hacks like that to avoid the
> inefficiencies of an API, gives a hint that something might be problematic.
>
> In a previous email of mine, i had said half seriously/half joking about the
> cost in Watts of SQLite's worldwide usage. For something that is so widely
> used, even some % of efficiency improvement really makes a difference. It is
> not an "SQLite destroys/saves civilization" kind of difference, but IMHO it
> would be measurable in G/M Watts.

The SQLite developers clearly care about performance, that much is clear.

But they also care about the "lite"-ness aspects too (and so does part
of its community, judging by this list).

They try to find the right balance for them, with ease of
implementation/testing/maintenance as the last variable of this
equation.

One thing that IMHO long term might improve the situation would be if
SQLite's own "native" tables would use the same Virtual Table API,
because then any bottleneck would apply to all "tables", not just the
VT ones, and would be more likely to be removed. And as recently
discussed, the VT API would be forced to acquire ways to communicate
with Explain Query Plan (for example) to notice Full Scans or Covering
Indexes not just for the "native" tables. VTs came after the "native"
tables, so the current situation is normal, but if all table accesses,
"native" or otherwise, could go thru the same API in the future (in
SQLite3 or Sqlite4), it would further decouple the "front-end" from
the "back-end", and ensure no performance differences between "native"
and "virtual" tables.

Of course, the above is a "naive" abstract reflection which ignores
the realities of VDBE, so it may sound rubbish to actual SQLite
developers. Apologies for that. --DD
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users