Re: [sqlite] Problem with EXPLAIN in Sqlite-3.3.5

2006-10-04 Thread drh
"Lowder, Brady M." <[EMAIL PROTECTED]> wrote: > Group, > > Compiling with -DSQLITE_DEBUG=1 has performance impacts as it adds = > other > debug capability that we may not need. I noticed that there is also a > define called SQLITE_OMIT_EXPLAIN in the source code. Is it possible to > just set that

RE: [sqlite] Optimize performance - reading from multiple databas e files, processing and writing to separate results database file?

2006-10-04 Thread Griggs, Donald
Regarding: "...I believe my only option is to attach databases one at a time,..." I can't say I know enough of your system to be sure, but keeping each subject's data in a separate database may not serve you well in the long run. Since I would suspect you will want to do various searches that

RE: [sqlite] Problem with EXPLAIN in Sqlite-3.3.5

2006-10-04 Thread Lowder, Brady M.
Group, Compiling with -DSQLITE_DEBUG=1 has performance impacts as it adds other debug capability that we may not need. I noticed that there is also a define called SQLITE_OMIT_EXPLAIN in the source code. Is it possible to just set that define properly (I assume to undefine it or perhaps set it to

Re: [sqlite] Optimize performance - reading from multiple databas e files, processing and writing to separate results database file?

2006-10-04 Thread John Stanton
Make several large DBs as your input in that case and do - select from DB1 select from DB2 ... select from DBn insert into output Serena Lien wrote: No, I am not copying anything to an in-memory database, this was just an alternative option I was suggesting, so please ignore the

Re: [sqlite] Optimize performance - reading from multiple databas e files, processing and writing to separate results database file?

2006-10-04 Thread Serena Lien
No, I am not copying anything to an in-memory database, this was just an alternative option I was suggesting, so please ignore the whole in-memory database thing from now on.. I have more databases to attach than the maximum number allowed, so no, I cannot attach my input databases to make one

[sqlite] "Cursor" struct in source code

2006-10-04 Thread Dave Dyer
It's only a structural problem, but the name "Cursor" is used in the mac environment, and so sqlite's "Cursor" struct causes conflicts. It would be handy if the name changed, say to SQCursor - To unsubscribe, send

Re: [sqlite] UTF-question

2006-10-04 Thread drh
Martin Pfeifle <[EMAIL PROTECTED]> wrote: > Hi, > is it possible to use UTF-16 access functionsa, e.g. sqlite3_bind_text16 or > sqlite3_column_text16, > together with a UTF-8 database? > > Do I get a byte identical UTF-8 database file if I write data to the database > using sqlite3_bind_text16

Re: [sqlite] Optimize performance - reading from multiple databas e files, processing and writing to separate results database file?

2006-10-04 Thread John Stanton
Serena Lien wrote: Thanks for the response. -- 1 Gbyte is not at all too big for Sqlite to handle in one file. True, but I think too big for an in-memory database - I was considering copying all my source data from separate databases into one table in an in-memory database so that I could

RE: [sqlite] Optimize performance - reading from multiple database files, processing and writing to separate results database file?

2006-10-04 Thread Tom Briggs
Hrm... I didn't realize that #define was limited to values less than 32. That being the case, you probably can't avoid starting multiple transactions. -Tom > -Original Message- > From: Serena Lien [mailto:[EMAIL PROTECTED] > Sent: Wednesday, October 04, 2006 9:57 AM > To:

RE: [sqlite] filename limit in sqlite_open

2006-10-04 Thread Shields, Daniel
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] > > "Shields, Daniel" <[EMAIL PROTECTED]> wrote: > > Does anyone know the maximum length of the filename passed to > > sqlite_open. We seem to see truncation on long file names (> 32 > > chars?) > > > > The SQLite core has a filename size

Re: [sqlite] filename limit in sqlite_open

2006-10-04 Thread Martin Jenkins
Shields, Daniel wrote: Does anyone know the maximum length of the filename passed to sqlite_open. We seem to see truncation on long file names (> 32 chars?) It's platform dependent and you didn't say which platform, so ... it'll be about whatever MAX_PATH or similar is on your platform. ;)

Re[2]: [sqlite] filename limit in sqlite_open

2006-10-04 Thread Teg
Hello drh, Wednesday, October 4, 2006, 10:00:38 AM, you wrote: dhc> "Shields, Daniel" <[EMAIL PROTECTED]> wrote: >> Does anyone know the maximum length of the filename passed >> to sqlite_open. We seem to see truncation on long file names >> (> 32 chars?) >> dhc> The SQLite core has a filename

Re: [sqlite] Optimize performance - reading from multiple databas e files, processing and writing to separate results database file?

2006-10-04 Thread Serena Lien
Thanks for the response. -- 1 Gbyte is not at all too big for Sqlite to handle in one file. True, but I think too big for an in-memory database - I was considering copying all my source data from separate databases into one table in an in-memory database so that I could access it easier later.

Re: [sqlite] filename limit in sqlite_open

2006-10-04 Thread drh
"Shields, Daniel" <[EMAIL PROTECTED]> wrote: > Does anyone know the maximum length of the filename passed > to sqlite_open. We seem to see truncation on long file names > (> 32 chars?) > The SQLite core has a filename size limit of around 500MB - not something you are likely to ever hit. The

Re: [sqlite] Optimize performance - reading from multiple database files, processing and writing to separate results database file?

2006-10-04 Thread Serena Lien
Yes, I could indeed determine which databases I want to attach to, but there is a limit to the number of databases you can attach (I think it's 32?) and I might have more than that. So I thought the safer route was to attach as I go.. Serena. On 10/4/06, Tom Briggs <[EMAIL PROTECTED]> wrote:

[sqlite] UTF-question

2006-10-04 Thread Martin Pfeifle
Hi, is it possible to use UTF-16 access functionsa, e.g. sqlite3_bind_text16 or sqlite3_column_text16, together with a UTF-8 database? Do I get a byte identical UTF-8 database file if I write data to the database using sqlite3_bind_text16 and if I use sqlite3_bind_text? Best Martin

RE: [sqlite] Optimize performance - reading from multiple database files, processing and writing to separate results database file?

2006-10-04 Thread Tom Briggs
Do you know all of the databases that you want to attach to at the start of processing? If so, there's no reason you can't simply attach them all beforehand, start your transaction, and complete all your processing. If you have to decide dynamically, based on the data, which databases you'll

RE: [sqlite] Optimize performance - reading from multiple databas e files, processing and writing to separate results database file?

2006-10-04 Thread Griggs, Donald
Hi Serena, I hope that more expert users will chime in here, but my initial thoughts were: -- 1 Gbyte is not at all too big for Sqlite to handle in one file. -- Some portion (large??) of the performance advantages of transactions will be lost in opening/closing files, even if transactions were

[sqlite] filename limit in sqlite_open

2006-10-04 Thread Shields, Daniel
Does anyone know the maximum length of the filename passed to sqlite_open. We seem to see truncation on long file names (> 32 chars?) Thanks, Daniel. == Please access the attached hyperlink for an important electronic

[sqlite] Optimize performance - reading from multiple database files, processing and writing to separate results database file?

2006-10-04 Thread Serena Lien
Hello, I have many databases, all in separate files. I want to choose a subset of them based on some query, and read the input data one at a time, process it somehow and write the results to another database file. The problems are caused by: - May have a large number of inputs (eg > 32