Re: [sqlite] suggestion on improving performance on UPDATE

2007-11-13 Thread Benilton Carvalho
That's what I'm testing right now... thanks a lot for the heads up... b On Nov 14, 2007 1:41 AM, Trevor Talbot <[EMAIL PROTECTED]> wrote: > On 11/13/07, Benilton Carvalho <[EMAIL PROTECTED]> wrote: > > > Then each column is added on the fly, using: > > > > ALTER TABLE table ADD COLUMN colunm REAL;

Re: [sqlite] suggestion on improving performance on UPDATE

2007-11-13 Thread Trevor Talbot
On 11/13/07, Benilton Carvalho <[EMAIL PROTECTED]> wrote: > Then each column is added on the fly, using: > > ALTER TABLE table ADD COLUMN colunm REAL; > > The columns in then filled used UPDATE as described previously. I haven't investigated the storage layout of this, but from what the docs impl

Re: [sqlite] Request for help with the SQLite Website

2007-11-13 Thread Trevor Talbot
On 11/13/07, Joe Wilson <[EMAIL PROTECTED]> wrote: > If you use the idiom whereby each thread solely takes its tasks > from a thread-safe work queue, you can have a clear separation of > responsibilities and minimal or preferably no shared-state between > threads. You get concurrency as a side eff

Re: [sqlite] Request for help with the SQLite Website

2007-11-13 Thread Joe Wilson
--- "D. Richard Hipp" <[EMAIL PROTECTED]> wrote: > > On Nov 13, 2007, at 10:55 PM, Joe Wilson wrote: > > > http://home.pacbell.net/ouster/threads.pdf > > JO and I reach a similar conclusion but by different > reasoning, I think. I like this line: Should You Abandon Threads? * No: important

Re: [sqlite] Request for help with the SQLite Website

2007-11-13 Thread D. Richard Hipp
On Nov 13, 2007, at 10:55 PM, Joe Wilson wrote: http://home.pacbell.net/ouster/threads.pdf JO and I reach a similar conclusion but by different reasoning, I think. --- Richard Klein <[EMAIL PROTECTED]> wrote: [EMAIL PROTECTED] wrote: What? And encourage people to write multitheaded prog

Re: [sqlite] Request for help with the SQLite Website

2007-11-13 Thread D. Richard Hipp
On Nov 13, 2007, at 10:37 PM, Richard Klein wrote: [EMAIL PROTECTED] wrote: What? And encourage people to write multitheaded programs? Not likely... I've been meaning to ask ... When you say that multiple threads are evil, do you mean "as opposed to multiple processes"? Or do you feel that

Re: [sqlite] Request for help with the SQLite Website

2007-11-13 Thread Joe Wilson
http://home.pacbell.net/ouster/threads.pdf --- Richard Klein <[EMAIL PROTECTED]> wrote: > [EMAIL PROTECTED] wrote: > > What? And encourage people to write multitheaded programs? > > Not likely... > > I've been meaning to ask ... When you say that multiple threads > are evil, do you mean "as oppo

Re: [sqlite] Request for help with the SQLite Website

2007-11-13 Thread Richard Klein
[EMAIL PROTECTED] wrote: What? And encourage people to write multitheaded programs? Not likely... I've been meaning to ask ... When you say that multiple threads are evil, do you mean "as opposed to multiple processes"? Or do you feel that multiprogramming in general is evil? - Richard Klein

Re: [sqlite] Request for help with the SQLite Website

2007-11-13 Thread Joe Wilson
--- [EMAIL PROTECTED] wrote: > Joe Wilson <[EMAIL PROTECTED]> wrote: > > You might mention the library is multi-thread safe in the Features > > section of http://www.sqlite.org/about.html > > What? And encourage people to write multitheaded programs? > Not likely... Good luck with that quest.

Re: [sqlite] suggestion on improving performance on UPDATE

2007-11-13 Thread Benilton Carvalho
It appears my storage bandwidth is awesome if I had only 10 columns (as my toy example had). My script creates the table and adds values to the primary key field. Then each column is added on the fly, using: ALTER TABLE table ADD COLUMN colunm REAL; The columns in then filled used UPDATE as

Re: [sqlite] Request for help with the SQLite Website

2007-11-13 Thread drh
Joe Wilson <[EMAIL PROTECTED]> wrote: > You might mention the library is multi-thread safe in the Features > section of http://www.sqlite.org/about.html What? And encourage people to write multitheaded programs? Not likely... > > Do you have a page that describes all the SQLITE_OMIT_* ifdefs

Re: [sqlite] Request for help with the SQLite Website

2007-11-13 Thread Joe Wilson
You might mention the library is multi-thread safe in the Features section of http://www.sqlite.org/about.html Do you have a page that describes all the SQLITE_OMIT_* ifdefs and compile options? Never

Re: [sqlite] Request for help with the SQLite Website

2007-11-13 Thread Joe Wilson
--- [EMAIL PROTECTED] wrote: > * Somebody please suggest a better tag line - > something better than "The World's Most Widely > Used SQL Database". New site looks good. The tag line is perfect. Some ideas: - Reduce the size of the logo and fonts everywhere - Allow clicking on the

Re: [sqlite] suggestion on improving performance on UPDATE

2007-11-13 Thread Trevor Talbot
On 11/13/07, Benilton Carvalho <[EMAIL PROTECTED]> wrote: > I have "fixed" my problem, working late night and not sleeping usually > causes that: I missed a WHERE on the UPDATE statement, so I could > modify multiple rows at a time. Basically what I did was create a > primary key (integer), which

RE: [sqlite] Request for help with the SQLite Website

2007-11-13 Thread Samuel R. Neff
I think the "about" text misses some of what, to me, are the most important parts of SQLite - in-process - zero maintenance Also as a .NET developer I would be put off by the "C-Library" reference. SQLite works very well in many languages regardless of the fact that it's written in C. I would p

Re: [sqlite] Request for help with the SQLite Website

2007-11-13 Thread Roger Binns
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 [EMAIL PROTECTED] wrote: > I am still looking for suggestions, comments, > and bug reports. There is no search on the front page or even anywhere else on the site that I can find. You don't even have to go to the hassle of adding your own - Google h

Re: [sqlite] Request for help with the SQLite Website

2007-11-13 Thread Rick Langschultz
This is a big improvement to the website. Keep up the great work on SQLite. Looking forward to 4.0 On Nov 13, 2007, at 7:40 PM, [EMAIL PROTECTED] wrote: The new look for the SQLite website is now in place, if you haven't already noticed: http://www.sqlite.org/ Even though the new look is

Re: [sqlite] suggestion on improving performance on UPDATE

2007-11-13 Thread Benilton Carvalho
Unfortunately the matrix is not sparse nor the direction of operations can be changed. The reason is that I have 2K samples, which are given in one file each (2K binary files). Each of the files has roughly 7M numbers, which I process by running some algorithms I have, reducing from 7M to 1M eleme

[sqlite] Request for help with the SQLite Website

2007-11-13 Thread drh
The new look for the SQLite website is now in place, if you haven't already noticed: http://www.sqlite.org/ Even though the new look is "in place" you should understand this as a work in progress, not a done deal. I am still looking for suggestions, comments, and bug reports. I am particula

RE: [sqlite] suggestion on improving performance on UPDATE

2007-11-13 Thread Griggs, Donald
Regarding:building a 100 row x 2000 column matrix If by chance these are sparse matrices (i.e., the majority of the values are empty) then conceivably you could store only the available values. The schema might then be something like: CREATE TABLE myTable ( matrixRow INTEGER,

Re: [sqlite] suggestion on improving performance on UPDATE

2007-11-13 Thread Trevor Talbot
On 11/13/07, John Stanton <[EMAIL PROTECTED]> wrote: > The point is that the more statements you have in each transaction the > better, up to a limit of perhaps a thousand or so. Actually, I get the impression that number of statements is not really the problem here. On 11/13/07, Benilton Carva

Re: [sqlite] One line batch file?

2007-11-13 Thread John
Owen Watson wrote: I'd like to have a one line batch file: sqlite3 test 'insert into testable values ('value1','value2')' have you tried something like: echo insert into testable values ('value1', 'value2'); | sqlite3 test note ";" at end of statement and "|" pipe char in command line. This s

Re: [sqlite] BLOB data performance?

2007-11-13 Thread John Stanton
You might find the method used by Squid to manage its cache would be worth emulating. Using TransmitFile on Windows or sendfile on Unix to despatch the file to the network is by far the most efficient way to pass on files from a cache. It avoids a number of levels of buffer shadowing. Andre

Re: [sqlite] suggestion on improving performance on UPDATE

2007-11-13 Thread John Stanton
The point is that the more statements you have in each transaction the better, up to a limit of perhaps a thousand or so. Benilton Carvalho wrote: I'm using RSQLite functions like: dbBeginTransaction(db2) dbGetPreparedQuery(db2, sql, bind.data=df) dbCommit(db2) where db2 is the connection to

Re: [sqlite] BLOB data performance?

2007-11-13 Thread Ken
I think your blob file performance may greatly depend upon the file system that it used and the workload. I found this article: http://oss.sgi.com/projects/xfs/papers/filesystem-perf-tm.pdf Andreas Volz <[EMAIL PROTECTED]> wrote: Am Tue, 13 Nov 2007 07:18:19 -0600 schrieb John Stanton: > In

RE: [sqlite] BLOB data performance?

2007-11-13 Thread Griggs, Donald
Regarding: The problem with many files in a directory: Another possible algorithm: Simply name the blob using the ROWID, e.g. Image0783.png In ancient days, a DOS directory of several hundred files might bog things down, but you can put thousands into a modern O.S.'s directory if ne

Re: [sqlite] BLOB data performance?

2007-11-13 Thread Andreas Volz
Am Tue, 13 Nov 2007 07:18:19 -0600 schrieb John Stanton: > In a cache situation I would expect that keeping the binary data in > files would be preferable because you can use far more efficient > mechanisms for loading them into your cache and in particular in > transmitting them downstream. Y

[sqlite] Foreign keys ?

2007-11-13 Thread Lothar Behrens
Hi, I am new to the list and to Sqlite. Reading over the archive didn't helped me much. It was only a 'flight' over the result searching for 'foreign'. What I have captured is the ability to define fireign keys in the tables and with the help of Cody Pisto creating the constraints as trigge

RE: [sqlite] Data encryption

2007-11-13 Thread Wilson, Ron
After some googling, Gunter has this domain: http://greschenz.de but it merely redirects to the dyndns url. It was active as late as 10/13/07 though. Maybe he shuts down his server at night? Ron Wilson, Senior Engineer, MPR Associates, 518.831.7546 -Original Message- From: A.J.Millan

Re: [sqlite] suggestion on improving performance on UPDATE

2007-11-13 Thread Benilton Carvalho
I'm using RSQLite functions like: dbBeginTransaction(db2) dbGetPreparedQuery(db2, sql, bind.data=df) dbCommit(db2) where db2 is the connection to the SQLite db, sql is either the INSERT or UPDATE statements I mentioned and df is the R object (data.frame) containing the data to be inserted/updated

Re: [sqlite] suggestion on improving performance on UPDATE

2007-11-13 Thread John Stanton
How are you using transactions? Benilton Carvalho wrote: Hi Everyone, I use R to create tools for analysis of microarrays (http://www.bioconductor.org). I'm in a situation where I need to handle 6 matrices with 1M rows and about 2K columns, ie each matrix takes roughly 15GB RAM. The procedure

[sqlite] suggestion on improving performance on UPDATE

2007-11-13 Thread Benilton Carvalho
Hi Everyone, I use R to create tools for analysis of microarrays (http://www.bioconductor.org). I'm in a situation where I need to handle 6 matrices with 1M rows and about 2K columns, ie each matrix takes roughly 15GB RAM. The procedure I'm working on can be divided in two parts: 1) I read an i

Re: [sqlite] BLOB data performance?

2007-11-13 Thread John Stanton
In a cache situation I would expect that keeping the binary data in files would be preferable because you can use far more efficient mechanisms for loading them into your cache and in particular in transmitting them downstream. Your DB only needs to store a pathname. Just be wary of directory

[sqlite] Version 3.52 text columns problem

2007-11-13 Thread Jacek Oleksy
Hi All, I have following problem: I'm using VC++6.0. I want to perform a select query on a single database table. What I do is: 1. prepare the statement using sqlite3_prepare_v2 2. step through the results using sqlite3_step No other process or thread is accessing the database at the time. Every