Re: [sqlite] Re: Improving performance of SQLite. Anyone heard of DeviceSQL?

[John Stanton]

A good salesman taught me - "When you have made the sale, stop selling". 
 If you have identified "The Fox" (the hidden actual decision maker) 
and made your case with an erudite technical presentation all the rest 
is just noise, and can even be detrimental.  After all 'decent 
marketing" is measured by a simple objective test - did it sell?


Even if you sell a dud and manage to avoid breach of contract litigation 
you cannot avoid the bad publicity and rumors of "that dog don't hunt".


A. Pagaltzis wrote:

* John Stanton <[EMAIL PROTECTED]> [2007-12-15 22:55]:

Which is the better model?


False dilemma. Where there is a budget, there is no reason you
can’t have both a good product and at least decent marketing.

Even when the product isn’t good, it’s unlikely to be so useless
as to violate the terms of contract. Oracle seems to survive just
fine, say.

For the executive summary on the matter, read this short essay:

“Enterprise software” is a social, not technical, phenomenon
http://lists.canonical.org/pipermail/kragen-tol/2005-April/000772.html

Regards,



-
To unsubscribe, send email to [EMAIL PROTECTED]
-

[sqlite] Re: Improving performance of SQLite. Anyone heard of DeviceSQL?

[A. Pagaltzis]

* John Stanton <[EMAIL PROTECTED]> [2007-12-15 22:55]:
> Which is the better model?

False dilemma. Where there is a budget, there is no reason you
can’t have both a good product and at least decent marketing.

Even when the product isn’t good, it’s unlikely to be so useless
as to violate the terms of contract. Oracle seems to survive just
fine, say.

For the executive summary on the matter, read this short essay:

“Enterprise software” is a social, not technical, phenomenon
http://lists.canonical.org/pipermail/kragen-tol/2005-April/000772.html

Regards,
-- 
Aristotle Pagaltzis // 

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Default Encoding In Sqlite

[Daniel Önnerby]

I figure I'll keep it short since it's only for the FAQ.
English is my primary language, but here my suggestion:

Question: Does SQLite handle unicode?

Short answer: Yes!

Answer:
SQLite handles unicode very well. SQLite stores texts in either UTF-16 
or UTF-8 format depending on how the database is created (sqlite3_open 
or sqlite3_open16). SQLite will also seamlessly convert between  the 
different formats depending on how you retrieve the texts 
(sqlite3_column_text or sqlite_column_text16) regardless on what format 
it has been saved as.


There are some cases -like using case insensitive LIKE- where SQLite 
needs to be extended with the ICU extension to fully work with unicode 
strings.





[EMAIL PROTECTED] wrote:

=?ISO-8859-1?Q?Daniel_=D6nnerby?= <[EMAIL PROTECTED]> wrote:
  
Unicode questions seems to come up at least once a week on the 
mailinglist. Maybe there should be something about this in the FAQ or 
the features page?





I will happily accept suggested text for such entries.

--
D. Richard Hipp <[EMAIL PROTECTED]>


-
To unsubscribe, send email to [EMAIL PROTECTED]
-

  


-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] SQLite and Columnar Databases

[John Stanton]

[EMAIL PROTECTED] wrote:

Joe Wilson <[EMAIL PROTECTED]> wrote:

--- [EMAIL PROTECTED] wrote:

Joe Wilson <[EMAIL PROTECTED]> wrote:
The reason why I asked is that I haven't had much luck with sqlite3 
performance for databases larger than the size of RAM on my machine

regardless of PRAGMA settings.

This is probably do to the cache locality problem.  We know how
to fix this, Joe.  Would you like to have a go at it?
I think setting aside contiguous pages in the file for exclusive use 
by each btree would help improve locality of reference on disk.


For example, let A, B and C represent in-use pages of 3 btrees and 
a, b and c represent free pages corresponding to the same btrees:


 AAaAAaAaaaBBbbBBBbbbCCcCCC

Is this what you had in mind in your post?



No.

The problem is when inserting into large database that is
indexed, the values being indexed are randomly distributed.
So with each insert, SQLite has to seek to a new random
place in the file to insert the new index entry there.
It does not matter that pages of the index are not in
consecutive order.  What matters is that each insertion
is into a different place and that the places are randomly
distributed over a large file - larger than will fit in cache.
In this situation, each new entry probably needs to go on 
a page that is not in cache, and hence a real disk

seek and read is required (as opposed to simply reading
the value out of cache) when inserting each new entry.  Disk
seeks and reads are much, much slower than disk cache hits,
which really slows down insertion performance.

If you do many inserts such that the indexed values are
in sorted order, then most inserts will go on the same page
as the previous.  The previous page is already in cache
so it doesn't need to be read from disk.  It has also
already been journaled, so no excess writing is required.
Disk I/O is greatly reduced. Things go much, much faster.
The problem is that you really have the luxury of being
able to insert entries in sorted order.  And if you are
indexing multiple columns, it is impossible to sort
the entries on both columns at once.

The usual way to work around this problem is to only do random
inserts into indices which are small enough to fit into
your disk cache.  Suppose you start inserting into index A.
Once A gets too big to fit entirely in cache, stop inserting
into it and start inserting into B.  Once B gets to be the
cache size, merge A and B together into a new index C.
The merge operation requires reading A and B straight through
from beginning to end once.  This is a lot of disk I/O but
it is still much faster than jumping around within the file
reading bits here and there.  After creating C, reset A and
B back to being empty (since all records have been transfered
into C).  Start inserting into A again until it fills up.
Then fill up B again.  Merge A and B into D, then merge C and D
into E.  Reset A, B, C, and D.  Keep doing this, merging
smaller indices into larger indices, until you insert all
the records you need to insert.  Then make a single pass
through all of the smaller indices and merge them all
together into a single big index Z.  Z becomes the new
index used for search operations on the database.

The algorithm above should probably go into SQLite 
to construct an index as part of the CREATE INDEX

statement.  When populating a database will a large
amount of data, first put all of the data into an
unindexed table.  This will be very fast because each
new entry goes at the end of the table (and also at the
end of the file.)  After all data is in place, issue
the CREATE INDEX statements to index any fields you
think need indexing.  The CREATE INDEX statement has
to populate the index.  The current algorithm is to
scan through the original table and create and insert
index entries one by one as they are encountered.  I am
proposing that you substitute the algorithm outlined in
the previous paragraph in place of the current algorithm.

--
D. Richard Hipp <[EMAIL PROTECTED]>

I experimented with that problem of building B-Tree indices on tables 
and discovered that to first build the table then build a list of keys 
into a file, sort the file using an iterative combination of quicksorts 
and merges then build the index in an optimal manner since the space 
needed for non-leaf nodes can be calculated and reserved, produced good 
performance on large tables even in a restricted memory environment.


Using this approach the data table and temporary files are only ever 
accessed sequentially and the "locality of reference" situation 
sidestepped.  If there are multiple disk drives available the 
intermediate files can be on another drive to further limit head movement.


Ensuring that interior nodes were contiguous seemed to be a winning 
strategy.  Only filling nodes to say 80% also delivered an index which 
would accept quite a lot of inserts before fragmenting with node splits 
and implementing node merging if possible instead of a 

Re: [sqlite] Improving performance of SQLite. Anyone heard of DeviceSQL?

[John Stanton]

This also is an anecdote from some time back.  As we were signing a 
fairly significant software contract with a large organization their 
manager told us "You guys know nothing about marketing.  Your 
presentation was unprofessional, no glossy brochures, no audio visuals 
and we would not have bought except that you were the only ones who 
convinced us you could do the job".  We just smiled and watched the ink 
dry while we pondered "where did we go right?".


The simple truth is that if you hype a product and sell it into an area 
where it is inadequate your triumph is short lived and the scorn and 
litigation enduring.  On the other hand if you deliver a solution which 
works as well, or preferably better, than proposed you have generated 
raving fans who will buy again and endorse your product to all and 
sundry.  Which is the better model?


Fred Williams wrote:

This discussion reminds me of another long, long ago in a galaxy far,
far away. (When I worked on "Mainframes" with 32 K or less "core"
memory.)

Discussing the then lopsided world with my non-IBM salesman, in a local
watering hole, after a particularly trying day of dealing with
"management."  The topic was the state of the computer industry at that
time. (And yet today.)

I was complaining of managements' complete lack of ability to see the
superior to IBM technology, (IMHO) and cost effectiveness we had
installed.  That is when I learned of the non bits and bytes "real
world."  My late salesman friend said,  "Fred, don't you understand that
the computer industry is a Marketing industry based on technology, and
not a technology industry?"

Thirty years later nothing could be truer.  No matter how much things
change, they still stay the same...

Fred

Running Windoze  on a "PC".
I know, I know it should be Linux on a Mac.  But I live in the "real
world" today.
I rest my case.


-Original Message-
From: D. Richard Hipp [mailto:[EMAIL PROTECTED]
Sent: Saturday, December 15, 2007 7:04 AM
To: sqlite-users@sqlite.org
Subject: Re: [sqlite] Improving performance of SQLite. Anyone heard of
DeviceSQL?



On Dec 14, 2007, at 9:24 PM, Lynn Fredricks wrote:


That's true. A lot of those kinds of sales presentations

are correctly

targeted at decision makers that make financial decisions. I don't
consider
it a bad thing - it's really a necessity to be competitive.


My intent is  to provide complete detailed technical information
about SQLite, including its limitations and faults, and honest
comparisons and even recommendations of other products
(including, but not limited to DeviceSQL).  My intent is to avoid
sophistry, misrepresentation, exaggeration,  and hype.
This intent is sometimes imperfectly executed, but it is my goal.

If that means that SQLite is uncompetitive, then so be it.


D. Richard Hipp
[EMAIL PROTECTED]




--
---
To unsubscribe, send email to [EMAIL PROTECTED]
--
---




-
To unsubscribe, send email to [EMAIL PROTECTED]
-




-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] SQLite and Columnar Databases

[Joe Wilson]

--- [EMAIL PROTECTED] wrote:
> The problem is when inserting into large database that is
> indexed, the values being indexed are randomly distributed.
> So with each insert, SQLite has to seek to a new random
> place in the file to insert the new index entry there.
> It does not matter that pages of the index are not in
> consecutive order.

For bulk INSERTs I see your point.

But for SELECT performance on the resulting database in a cold OS file 
cache situation (assuming you do not have the luxury of running VACUUM 
beforehand) or when the database file is many times the size of RAM, it 
does improve SELECT performance if the pages of each index are contiguous.
It would be nice if contiguous index pages could be a fortunate "by-product" 
of the bulk INSERT optimization algorithm for this reason.

>  What matters is that each insertion
> is into a different place and that the places are randomly
> distributed over a large file - larger than will fit in cache.
> In this situation, each new entry probably needs to go on 
> a page that is not in cache, and hence a real disk
> seek and read is required (as opposed to simply reading
> the value out of cache) when inserting each new entry.  Disk
> seeks and reads are much, much slower than disk cache hits,
> which really slows down insertion performance.
> 
> If you do many inserts such that the indexed values are
> in sorted order, then most inserts will go on the same page
> as the previous.  The previous page is already in cache
> so it doesn't need to be read from disk.  It has also
> already been journaled, so no excess writing is required.
> Disk I/O is greatly reduced. Things go much, much faster.
> The problem is that you really have the luxury of being
> able to insert entries in sorted order.  And if you are
> indexing multiple columns, it is impossible to sort
> the entries on both columns at once.
> 
> The usual way to work around this problem is to only do random
> inserts into indices which are small enough to fit into
> your disk cache.  Suppose you start inserting into index A.
> Once A gets too big to fit entirely in cache, stop inserting
> into it and start inserting into B.  Once B gets to be the
> cache size, merge A and B together into a new index C.
> The merge operation requires reading A and B straight through
> from beginning to end once.  This is a lot of disk I/O but
> it is still much faster than jumping around within the file
> reading bits here and there.  After creating C, reset A and
> B back to being empty (since all records have been transfered
> into C).  Start inserting into A again until it fills up.
> Then fill up B again.  Merge A and B into D, then merge C and D
> into E.  Reset A, B, C, and D.  Keep doing this, merging
> smaller indices into larger indices, until you insert all
> the records you need to insert.  Then make a single pass
> through all of the smaller indices and merge them all
> together into a single big index Z.  Z becomes the new
> index used for search operations on the database.
> 
> The algorithm above should probably go into SQLite 
> to construct an index as part of the CREATE INDEX
> statement.  When populating a database will a large
> amount of data, first put all of the data into an
> unindexed table.  This will be very fast because each
> new entry goes at the end of the table (and also at the
> end of the file.)  After all data is in place, issue
> the CREATE INDEX statements to index any fields you
> think need indexing.  The CREATE INDEX statement has
> to populate the index.  The current algorithm is to
> scan through the original table and create and insert
> index entries one by one as they are encountered.  I am
> proposing that you substitute the algorithm outlined in
> the previous paragraph in place of the current algorithm.

Are you assuming that you do not have any indexes already in place on 
the table being populated?

It would be nice if the optimization could work in the general case
where the table already has data and indexes (implicit or explicit) 
as in:

  create foo (
a UNIQUE,
b UNIQUE,
c,
d,
primary key(d, b, c)
  );

Or are you suggesting that you first determine whether you're running 
INSERTs within a larger transaction and perform the new table/index population 
algorithm for each INSERT and only combine the various indexes at the
end of the transaction or whenever a page threshold is exceeded?

Guessing the end of INSERTs into the same table could be tricky within a 
larger transaction. End might be defined as when you're running an SQL 
command other than an insert into the same table, as in:

  BEGIN;
  INSERT INTO foo VALUES ...;
-- detect that we've started inserting into table foo
-- and begin the bulk insert optimization
  INSERT INTO foo VALUES ...;
  ...a million other inserts into foo...
  INSERT INTO foo VALUES ...;
  INSERT INTO foo VALUES ...;
  INSERT INTO foo VALUES ...;
-- detect that we've stopped inserting into foo 

Re: [sqlite] Default Encoding In Sqlite

[drh]

=?ISO-8859-1?Q?Daniel_=D6nnerby?= <[EMAIL PROTECTED]> wrote:
> Unicode questions seems to come up at least once a week on the 
> mailinglist. Maybe there should be something about this in the FAQ or 
> the features page?
> 

I will happily accept suggested text for such entries.

--
D. Richard Hipp <[EMAIL PROTECTED]>


-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Compiling Problem With SQLite 3.5.4

[Christian Werner]

Following patch applied to the CVS version compiled
w/o warnings on a CentOS 5 x86_64 box. But I didn't
run the test yet ... and autoconf logic regarding
availability of ptrdiff_t is missing, too.

Index: src/func.c
===
RCS file: /sqlite/sqlite/src/func.c,v
retrieving revision 1.181
diff -r1.181 func.c
878c878
<   int flags;/* 1: trimleft  2: trimright  3: trim */
---
>   ptrdiff_t flags;  /* 1: trimleft  2: trimright  3: trim */
919c919
< flags = (int)sqlite3_user_data(context);
---
> flags = (ptrdiff_t)sqlite3_user_data(context);
1464c1464
<   pArg = (void*)(int)argType;
---
>   pArg = (void*)(ptrdiff_t)argType;
1483c1483
< void *pArg = (void*)(int)aAggs[i].argType;
---
> void *pArg = (void*)(ptrdiff_t)aAggs[i].argType;
Index: src/table.c
===
RCS file: /sqlite/sqlite/src/table.c,v
retrieving revision 1.29
diff -r1.29 table.c
36c36
<   int nData;
---
>   ptrdiff_t nData;
197c197,198
< int i, n;
---
> int i;
> ptrdiff_t n;
200c201
< n = (int)azResult[0];
---
> n = (ptrdiff_t)azResult[0];
Index: src/vdbemem.c
===
RCS file: /sqlite/sqlite/src/vdbemem.c,v
retrieving revision 1.84
diff -r1.84 vdbemem.c
878c878
< if( (enc & SQLITE_UTF16_ALIGNED)!=0 && 1==(1&(int)pVal->z) ){
---
> if( (enc & SQLITE_UTF16_ALIGNED)!=0 && 1==(1&(ptrdiff_t)pVal->z) ){

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] SQLite and Columnar Databases

[drh]

Joe Wilson <[EMAIL PROTECTED]> wrote:
> --- [EMAIL PROTECTED] wrote:
> > Joe Wilson <[EMAIL PROTECTED]> wrote:
> > > The reason why I asked is that I haven't had much luck with sqlite3 
> > > performance for databases larger than the size of RAM on my machine
> > > regardless of PRAGMA settings.
> > 
> > This is probably do to the cache locality problem.  We know how
> > to fix this, Joe.  Would you like to have a go at it?
> 
> I think setting aside contiguous pages in the file for exclusive use 
> by each btree would help improve locality of reference on disk.
> 
> For example, let A, B and C represent in-use pages of 3 btrees and 
> a, b and c represent free pages corresponding to the same btrees:
> 
>  AAaAAaAaaaBBbbBBBbbbCCcCCC
> 
> Is this what you had in mind in your post?
> 

No.

The problem is when inserting into large database that is
indexed, the values being indexed are randomly distributed.
So with each insert, SQLite has to seek to a new random
place in the file to insert the new index entry there.
It does not matter that pages of the index are not in
consecutive order.  What matters is that each insertion
is into a different place and that the places are randomly
distributed over a large file - larger than will fit in cache.
In this situation, each new entry probably needs to go on 
a page that is not in cache, and hence a real disk
seek and read is required (as opposed to simply reading
the value out of cache) when inserting each new entry.  Disk
seeks and reads are much, much slower than disk cache hits,
which really slows down insertion performance.

If you do many inserts such that the indexed values are
in sorted order, then most inserts will go on the same page
as the previous.  The previous page is already in cache
so it doesn't need to be read from disk.  It has also
already been journaled, so no excess writing is required.
Disk I/O is greatly reduced. Things go much, much faster.
The problem is that you really have the luxury of being
able to insert entries in sorted order.  And if you are
indexing multiple columns, it is impossible to sort
the entries on both columns at once.

The usual way to work around this problem is to only do random
inserts into indices which are small enough to fit into
your disk cache.  Suppose you start inserting into index A.
Once A gets too big to fit entirely in cache, stop inserting
into it and start inserting into B.  Once B gets to be the
cache size, merge A and B together into a new index C.
The merge operation requires reading A and B straight through
from beginning to end once.  This is a lot of disk I/O but
it is still much faster than jumping around within the file
reading bits here and there.  After creating C, reset A and
B back to being empty (since all records have been transfered
into C).  Start inserting into A again until it fills up.
Then fill up B again.  Merge A and B into D, then merge C and D
into E.  Reset A, B, C, and D.  Keep doing this, merging
smaller indices into larger indices, until you insert all
the records you need to insert.  Then make a single pass
through all of the smaller indices and merge them all
together into a single big index Z.  Z becomes the new
index used for search operations on the database.

The algorithm above should probably go into SQLite 
to construct an index as part of the CREATE INDEX
statement.  When populating a database will a large
amount of data, first put all of the data into an
unindexed table.  This will be very fast because each
new entry goes at the end of the table (and also at the
end of the file.)  After all data is in place, issue
the CREATE INDEX statements to index any fields you
think need indexing.  The CREATE INDEX statement has
to populate the index.  The current algorithm is to
scan through the original table and create and insert
index entries one by one as they are encountered.  I am
proposing that you substitute the algorithm outlined in
the previous paragraph in place of the current algorithm.

--
D. Richard Hipp <[EMAIL PROTECTED]>



-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Compiling Problem With SQLite 3.5.4

[Robert L Cochran]

[EMAIL PROTECTED] wrote:

Robert L Cochran <[EMAIL PROTECTED]> wrote:
  
While compiling version 3.5.4 using gcc, I got these messages from 
'make' on my CentOS 5 host. It looks to me like the 'make' step failed. 
I can post the config.log if that would help.


How can I fix this to get a successful compile?




I don't see any errors in your make output, only warnings.
Did I overlook something, or are you concerned about the
warnings.

The warnings all have to do with the fact that you are
compiling on a machine with 64-bit pointers and 32-bit
integers.  The warnings are all harmless and the code
works as intended as long as

 sizeof(int) <= sizeof(void*)

Perhaps a reader can suggest ways of eliminating these
warnings.

--
D. Richard Hipp <[EMAIL PROTECTED]>


  
I did a poor job of explaining this issue. GCC builds of 3.5.4 seem to 
fail. I've sent Richard a bunch of log files comparing source code 
builds of versions 3.5.3 and 3.5.4 showing what I believe are symptoms 
of the alleged failure. If anyone wants copies of the same set of files, 
feel free to contact me.


Bob


-
To unsubscribe, send email to [EMAIL PROTECTED]
-



  



-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Compiling Problem With SQLite 3.5.4

[Christian Werner]

[EMAIL PROTECTED] wrote:
> 
> Christian Werner <[EMAIL PROTECTED]> wrote:
> >
> > What about using the ptrdiff_t type in those places instead.
> >
> 
> Is ptrdiff_t implemented on all C compiler environments that
> SQLite builds on?

I'm afraid not, thus some autoconf logic might be required to
detect its availability.

> Also, will it really eliminate the warnings.  In several places
> we are storing a (32-bit) integer in a pointer.
> 
>  p = (char*)i;
> 
> And then later we get the integer back out:
> 
>  i = (int)p;

When the context allows to change i's type to ptrdiff_t
then you get

  i = (ptrdift_t)p;

which eliminates the warning.

> If we change to ptrdiff_t, then we have:
> 
>  p = (char*)(ptrdiff_t)i;  /* Convoluted, but works */
>  i = (int)(ptrdiff_t)p;/* Still get a warning? */
> 
> In the conversion from pointer back to integer, don't we still
> get a warning about casting a 64-bit integer into a 32-bit
> integer?  (I don't know because I do not have a 64-bit machine
> easily at hand to test it on.)

See above, when it's irrelevant if i is a 32 or 64 bit data type,
the 32 vs. 64 bit warning blues is gone.

Regards,
Christian

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Improving performance of SQLite. Anyone heard of DeviceSQL?

[Scott Goodwin]

On Dec 14, 2007, at 9:24 PM, Lynn Fredricks wrote:


That's true. A lot of those kinds of sales presentations are correctly
targeted at decision makers that make financial decisions. I don't  
consider
it a bad thing - it's really a necessity to be competitive. The bear  
in the

woods isnt evil, he's just hungry like the other bears :-)


It is bad for the company you are selling to because the wrong people  
are making these product decisions within these companies and it is  
clear that these sales presentations take advantage of that. It is  
potentially bad for your company if your product is not suitable for  
their needs and you successfully sell it to them; you may endanger  
your company's reputation. While you are not responsible for the  
structure of any company you sell to, you are responsible for your own  
integrity. Making the claim that you're just doing what everyone else  
is does not absolve you or your salespeople of this responsibility.  
Your analogy about bears and competition tells me you and your  
salespeople might feel justified using tactics that give used car  
salesmen a bad name. A more appropriate analogy is to say that you  
would take candy from a child because you can, or that you would you  
sell a sub-prime mortgage to a borrower you know is not credit-worthy  
so you can earn a higher commission. If you knowingly convince  
decision makers who do not have the skills to make such decisions to  
buy products that are not appropriate to their needs, it is  
essentially the same thing. I would not trust any company that is  
afraid to sell to my engineering department.


I think what you are seeing is evolution of the software industry.  
It really
isnt necessary for there to be such an extreme split between  
engineering and
management  - and by evolution I mean that engineering has to adapt  
to a
tighter relationship with management, or they are destined to have  
their
roles outsourced. Noone should know the product than its own  
engineers, and
its those who can bridge that divide that will be running the  
engineering

and IT departments.


It is not the role of management to understand the engineering details  
or decisions, but to understand the impact that those decisions have  
in their own role of providing and managing the resources that their  
engineers and others need so they can build the best products possible  
at the lowest cost and sell them to their customers. Managers  
rightfully step in where there are resource issues or constraints, but  
they have no business making the technical decisions on component  
selection where the complexity of integration of those components is  
high and the resource constraints are not at issue. If a company's own  
engineers do not understand their own products I fail to see how  
managers can step in and make better engineering decisions.


It would be ideal if engineers could explain things in ways that  
managers could understand, but engineers are usually hired for their  
engineering skills, not their marketing or sales skills; to then say  
it is engineering's own fault that they cannot compete against  
professional salespeople and marketers is disingenuous. There are  
engineers who can bridge the gap fairly well and maybe they are or  
will be the ones running the engineering and IT departments, but the  
implicit assumption that it is management's role to make technical  
selection decisions still remains and it is that assumption that must  
be challenged. If it is true that management requires a tighter  
relationship with engineering in order to be successful, then it is  
beyond me how outsourcing those engineering roles could lead to that  
outcome.


The best feedback you will get to improve your product will be from  
the engineers in your customer's companies; the managers can only help  
you improve your ability to persuade other managers, but improving  
your sales pitch won't improve your product. It is your engineering  
department's responsibility to create a relationship with the  
engineering departments of the companies you sell to; doing this can  
create potential supporters within the engineering department of your  
customer's company and an understanding of where and how your product  
falls short and can be improved.


Any manager who overrules their engineers on technical selections is  
not making wise financial or product decisions and is not doing what  
is in the best interests of their company. That it happens all the  
time and that salespeople take advantage of this does not make it  
ethical or beneficial to anyone, even though it may look beneficial to  
the seller in the short term.


/s.

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Default Encoding In Sqlite

[Daniel Önnerby]

Unicode questions seems to come up at least once a week on the 
mailinglist. Maybe there should be something about this in the FAQ or 
the features page?


Trevor Talbot wrote:

I wrote:

  

The default storage encoding on disk is UTF-8, but it can be changed
to UTF-16 with a PRAGMA.



As Igor reminds me, if you create the database file using
sqlite3_open16() the default will be UTF-16 instead. You can still set
it explicitly via a PRAGMA if you wish.

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

  


-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] SQLite and Columnar Databases

[Joe Wilson]

--- [EMAIL PROTECTED] wrote:
> Joe Wilson <[EMAIL PROTECTED]> wrote:
> > The reason why I asked is that I haven't had much luck with sqlite3 
> > performance for databases larger than the size of RAM on my machine
> > regardless of PRAGMA settings.
> 
> This is probably do to the cache locality problem.  We know how
> to fix this, Joe.  Would you like to have a go at it?

I think setting aside contiguous pages in the file for exclusive use 
by each btree would help improve locality of reference on disk.

For example, let A, B and C represent in-use pages of 3 btrees and 
a, b and c represent free pages corresponding to the same btrees:

 AAaAAaAaaaBBbbBBBbbbCCcCCC

Is this what you had in mind in your post?

 http://www.mail-archive.com/sqlite-users@sqlite.org/msg27851.html

If not, could you elaborate?




  

Never miss a thing.  Make Yahoo your home page. 
http://www.yahoo.com/r/hs

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Compiling Problem With SQLite 3.5.4

[drh]

Christian Werner <[EMAIL PROTECTED]> wrote:
> [EMAIL PROTECTED] wrote:
> 
> > ...
> > The warnings all have to do with the fact that you are
> > compiling on a machine with 64-bit pointers and 32-bit
> > integers.  The warnings are all harmless and the code
> > works as intended as long as
> > 
> >  sizeof(int) <= sizeof(void*)
> > 
> > Perhaps a reader can suggest ways of eliminating these
> > warnings.
> 
> What about using the ptrdiff_t type in those places instead.
> 

Is ptrdiff_t implemented on all C compiler environments that
SQLite builds on?

Also, will it really eliminate the warnings.  In several places
we are storing a (32-bit) integer in a pointer.

 p = (char*)i;

And then later we get the integer back out:

 i = (int)p;

If we change to ptrdiff_t, then we have:

 p = (char*)(ptrdiff_t)i;  /* Convoluted, but works */
 i = (int)(ptrdiff_t)p;/* Still get a warning? */

In the conversion from pointer back to integer, don't we still
get a warning about casting a 64-bit integer into a 32-bit
integer?  (I don't know because I do not have a 64-bit machine
easily at hand to test it on.)

--
D. Richard Hipp <[EMAIL PROTECTED]>


-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Compiling Problem With SQLite 3.5.4

[Christian Werner]

[EMAIL PROTECTED] wrote:

> ...
> The warnings all have to do with the fact that you are
> compiling on a machine with 64-bit pointers and 32-bit
> integers.  The warnings are all harmless and the code
> works as intended as long as
> 
>  sizeof(int) <= sizeof(void*)
> 
> Perhaps a reader can suggest ways of eliminating these
> warnings.

What about using the ptrdiff_t type in those places instead.

Cheers,
Christian

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Compiling Problem With SQLite 3.5.4

[drh]

Robert L Cochran <[EMAIL PROTECTED]> wrote:
> While compiling version 3.5.4 using gcc, I got these messages from 
> 'make' on my CentOS 5 host. It looks to me like the 'make' step failed. 
> I can post the config.log if that would help.
> 
> How can I fix this to get a successful compile?
> 

I don't see any errors in your make output, only warnings.
Did I overlook something, or are you concerned about the
warnings.

The warnings all have to do with the fact that you are
compiling on a machine with 64-bit pointers and 32-bit
integers.  The warnings are all harmless and the code
works as intended as long as

 sizeof(int) <= sizeof(void*)

Perhaps a reader can suggest ways of eliminating these
warnings.

--
D. Richard Hipp <[EMAIL PROTECTED]>


-
To unsubscribe, send email to [EMAIL PROTECTED]
-

[sqlite] Compiling Problem With SQLite 3.5.4

[Robert L Cochran]

While compiling version 3.5.4 using gcc, I got these messages from 
'make' on my CentOS 5 host. It looks to me like the 'make' step failed. 
I can post the config.log if that would help.


How can I fix this to get a successful compile?

Bob Cochran





./libtool --mode=link gcc -g -O2 -I. -I../src -DNDEBUG   -DSQLITE_THREADSAFE=1 -DSQLITE_THREAD_OVERRIDE_LOCK=-1 -DSQLITE_OMIT_LOAD_EXTENSION=1  -o libsqlite3.la alter.lo analyze.lo attach.lo auth.lo btmutex.lo btree.lo build.lo callback.lo complete.lo date.lo delete.lo expr.lo func.lo hash.lo journal.lo insert.lo loadext.lo main.lo malloc.lo mem1.lo mem2.lo mem3.lo mem4.lo mutex.lo mutex_os2.lo mutex_unix.lo mutex_w32.lo opcodes.lo os.lo os_unix.lo os_win.lo os_os2.lo pager.lo parse.lo pragma.lo prepare.lo printf.lo random.lo select.lo table.lo tokenize.lo trigger.lo update.lo util.lo vacuum.lo vdbe.lo vdbeapi.lo vdbeaux.lo vdbeblob.lo vdbefifo.lo vdbemem.lo where.lo utf.lo legacy.lo vtab.lo -lpthread  \
 -rpath /usr/local/sqlite-3.5.4/lib -version-info "8:6:8"
gcc -shared  .libs/alter.o .libs/analyze.o .libs/attach.o .libs/auth.o .libs/btmutex.o .libs/btree.o .libs/build.o .libs/callback.o .libs/complete.o .libs/date.o .libs/delete.o .libs/expr.o .libs/func.o .libs/hash.o .libs/journal.o .libs/insert.o .libs/loadext.o .libs/main.o .libs/malloc.o .libs/mem1.o .libs/mem2.o .libs/mem3.o .libs/mem4.o .libs/mutex.o .libs/mutex_os2.o .libs/mutex_unix.o .libs/mutex_w32.o .libs/opcodes.o .libs/os.o .libs/os_unix.o .libs/os_win.o .libs/os_os2.o .libs/pager.o .libs/parse.o .libs/pragma.o .libs/prepare.o .libs/printf.o .libs/random.o .libs/select.o .libs/table.o .libs/tokenize.o .libs/trigger.o .libs/update.o .libs/util.o .libs/vacuum.o .libs/vdbe.o .libs/vdbeapi.o .libs/vdbeaux.o .libs/vdbeblob.o .libs/vdbefifo.o .libs/vdbemem.o .libs/where.o .libs/utf.o .libs/legacy.o .libs/vtab.o  -lpthread  -Wl,-soname -Wl,libsqlite3.so.0 -o .libs/libsqlite3.so.0.8.6
(cd .libs && rm -f libsqlite3.so.0 && ln -s libsqlite3.so.0.8.6 libsqlite3.so.0)
(cd .libs && rm -f libsqlite3.so && ln -s libsqlite3.so.0.8.6 libsqlite3.so)
ar cru .libs/libsqlite3.a  alter.o analyze.o attach.o auth.o btmutex.o btree.o build.o callback.o complete.o date.o delete.o expr.o func.o hash.o journal.o insert.o loadext.o main.o malloc.o mem1.o mem2.o mem3.o mem4.o mutex.o mutex_os2.o mutex_unix.o mutex_w32.o opcodes.o os.o os_unix.o os_win.o os_os2.o pager.o parse.o pragma.o prepare.o printf.o random.o select.o table.o tokenize.o trigger.o update.o util.o vacuum.o vdbe.o vdbeapi.o vdbeaux.o vdbeblob.o vdbefifo.o vdbemem.o where.o utf.o legacy.o vtab.o
ranlib .libs/libsqlite3.a
creating libsqlite3.la
(cd .libs && rm -f libsqlite3.la && ln -s ../libsqlite3.la libsqlite3.la)
rm -rf tsrc
mkdir -p tsrc
cp ../src/alter.c ../src/analyze.c ../src/attach.c ../src/auth.c ../src/btmutex.c ../src/btree.c ../src/btree.h ../src/build.c ../src/callback.c ../src/complete.c ../src/date.c ../src/delete.c ../src/expr.c ../src/func.c ../src/hash.c ../src/hash.h ../src/insert.c ../src/journal.c ../src/legacy.c ../src/loadext.c ../src/main.c ../src/malloc.c ../src/mem1.c ../src/mem2.c ../src/mem3.c ../src/mem4.c ../src/mutex.c ../src/mutex_os2.c ../src/mutex_unix.c ../src/mutex_w32.c ../src/os.c ../src/os_unix.c ../src/os_win.c ../src/os_os2.c ../src/pager.c ../src/pager.h ../src/parse.y ../src/pragma.c ../src/prepare.c ../src/printf.c ../src/random.c ../src/select.c ../src/shell.c ../src/sqlite.h.in ../src/sqliteInt.h ../src/table.c ../src/tclsqlite.c ../src/tokenize.c ../src/trigger.c ../src/utf.c ../src/update.c ../src/util.c ../src/vacuum.c ../src/vdbe.c ../src/vdbe.h ../src/vdbeapi.c ../src/vdbeaux.c ../src/vdbeblob.c ../src/vdbefifo.c ../src/vdbemem.c ../src/vdbeInt.h ../src/vtab.c ../src/where.c ../ext/fts1/fts1.c ../ext/fts1/fts1.h ../ext/fts1/fts1_hash.c ../ext/fts1/fts1_hash.h ../ext/fts1/fts1_porter.c ../ext/fts1/fts1_tokenizer.h ../ext/fts1/fts1_tokenizer1.c sqlite3.h ../src/btree.h ../src/btreeInt.h ../src/hash.h ../src/sqliteLimit.h ../src/mutex.h opcodes.h ../src/os.h ../src/os_common.h ../src/sqlite3ext.h ../src/sqliteInt.h ../src/vdbe.h parse.h ../ext/fts1/fts1.h ../ext/fts1/fts1_hash.h ../ext/fts1/fts1_tokenizer.h ../src/vdbeInt.h tsrc
cp: warning: source file `../src/btree.h' specified more than once
cp: warning: source file `../src/hash.h' specified more than once
cp: warning: source file `../src/sqliteInt.h' specified more than once
cp: warning: source file `../src/vdbe.h' specified more than once
cp: warning: source file `../ext/fts1/fts1.h' specified more than once
cp: warning: source file `../ext/fts1/fts1_hash.h' specified more than once
cp: warning: source file `../ext/fts1/fts1_tokenizer.h' specified more than once
cp: warning: source file `../src/vdbeInt.h' specified more than once
rm tsrc/sqlite.h.in tsrc/parse.y
cp parse.c opcodes.c keywordhash.h tsrc
tclsh ../tool/mksqlite3c.tcl
./libtool --mode=link gcc -g -O2 -I. -I../src -DNDEBUG   -DSQLITE_THREADSAFE=1 

RE: [sqlite] Improving performance of SQLite. Anyone heard of DeviceSQL?

[Fred Williams]

This discussion reminds me of another long, long ago in a galaxy far,
far away. (When I worked on "Mainframes" with 32 K or less "core"
memory.)

Discussing the then lopsided world with my non-IBM salesman, in a local
watering hole, after a particularly trying day of dealing with
"management."  The topic was the state of the computer industry at that
time. (And yet today.)

I was complaining of managements' complete lack of ability to see the
superior to IBM technology, (IMHO) and cost effectiveness we had
installed.  That is when I learned of the non bits and bytes "real
world."  My late salesman friend said,  "Fred, don't you understand that
the computer industry is a Marketing industry based on technology, and
not a technology industry?"

Thirty years later nothing could be truer.  No matter how much things
change, they still stay the same...

Fred

Running Windoze  on a "PC".
I know, I know it should be Linux on a Mac.  But I live in the "real
world" today.
I rest my case.

> -Original Message-
> From: D. Richard Hipp [mailto:[EMAIL PROTECTED]
> Sent: Saturday, December 15, 2007 7:04 AM
> To: sqlite-users@sqlite.org
> Subject: Re: [sqlite] Improving performance of SQLite. Anyone heard of
> DeviceSQL?
>
>
>
> On Dec 14, 2007, at 9:24 PM, Lynn Fredricks wrote:
>
> > That's true. A lot of those kinds of sales presentations
> are correctly
> > targeted at decision makers that make financial decisions. I don't
> > consider
> > it a bad thing - it's really a necessity to be competitive.
> >
>
> My intent is  to provide complete detailed technical information
> about SQLite, including its limitations and faults, and honest
> comparisons and even recommendations of other products
> (including, but not limited to DeviceSQL).  My intent is to avoid
> sophistry, misrepresentation, exaggeration,  and hype.
> This intent is sometimes imperfectly executed, but it is my goal.
>
> If that means that SQLite is uncompetitive, then so be it.
>
>
> D. Richard Hipp
> [EMAIL PROTECTED]
>
>
>
>
> --
> ---
> To unsubscribe, send email to [EMAIL PROTECTED]
> --
> ---
>


-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Improving performance of SQLite. Anyone heard of DeviceSQL?

[D. Richard Hipp]

On Dec 14, 2007, at 9:24 PM, Lynn Fredricks wrote:


That's true. A lot of those kinds of sales presentations are correctly
targeted at decision makers that make financial decisions. I don't  
consider

it a bad thing - it's really a necessity to be competitive.



My intent is  to provide complete detailed technical information
about SQLite, including its limitations and faults, and honest
comparisons and even recommendations of other products
(including, but not limited to DeviceSQL).  My intent is to avoid
sophistry, misrepresentation, exaggeration,  and hype.
This intent is sometimes imperfectly executed, but it is my goal.

If that means that SQLite is uncompetitive, then so be it.


D. Richard Hipp
[EMAIL PROTECTED]




-
To unsubscribe, send email to [EMAIL PROTECTED]
-