[sqlite] How to optimize a select that gets 17 rows from a 700k row DB (via perl using DBI)?

2007-01-12 Thread Jonathan . W . Crane
Hi all. I will try to post all data I can think of. The basic issue is that I have a Perl CGI script that shows the most recent load stats for a bunch of servers we use at our site for long, computing-intense programs. The environment is all RedHat Enterprise Linux 4 ( 2.6.9-42.26.ELsmp #1

[sqlite] Re: fts1 corruption debugging.

2007-01-12 Thread Scott Hess
http://www.sqlite.org/cvstrac/tktview?tn=2166 I'm probably not going to be back on this until Monday or Tuesday, unfortunately. -scott On 1/12/07, Scott Hess <[EMAIL PROTECTED]> wrote: [Find attached the file I'm using to debug this.] I think I've found the difference causing this, but I

[sqlite] fts1 corruption debugging.

2007-01-12 Thread Scott Hess
[Find attached the file I'm using to debug this.] I think I've found the difference causing this, but I don't understand why it matters. It all should apply to fts2, the code in question didn't change in a way likely to change this. When an insert is done against an fts1 table, index_insert()

Re: [sqlite] sqlite3_prepare_v2 and multi-connection schema changes

2007-01-12 Thread Ron Stevens
This should be fixed by checkin [3592]. Thanks for the quick fix! - To unsubscribe, send email to [EMAIL PROTECTED] -

Re: [sqlite] 3.3.10 data corruption on updating fts1 string table

2007-01-12 Thread Scott Hess
OK, there's definite meat, here. I have other reports of users seeing this problem. It's specifically related to doing UPDATE against an fts1 or fts2 table. INSERT and DELETE both work fine. As far as I can tell, UPDATE may have never worked, or may have worked only in specific circumstances.

Re: [sqlite] sqlite3_prepare_v2 and multi-connection schema changes

2007-01-12 Thread drh
"Ron Stevens" <[EMAIL PROTECTED]> wrote: > I just switched to 3.3.10 and I'm playing around with sqlite3_prepare_v2. I > have a database with two open connections, and each connection creates its > own table. After the second connection creates its table all statements > created with

Re: [sqlite] sqlite3_prepare_v2 and multi-connection schema changes

2007-01-12 Thread drh
"Ron Stevens" <[EMAIL PROTECTED]> wrote: > I'm wondering if sqlite3_prepare_v2 should try to reload the schema before > returning the SQLITE_SCHEMA error. Yes, it should. But I seem to have omitted this simple case from the test suite and I had a regression on the previous release. I suppose

[sqlite] sqlite3_prepare_v2 and multi-connection schema changes

2007-01-12 Thread Ron Stevens
I just switched to 3.3.10 and I'm playing around with sqlite3_prepare_v2. I have a database with two open connections, and each connection creates its own table. After the second connection creates its table all statements created with sqlite3_prepare_v2 on the first connection fail with

Re: [sqlite] Pager modification question

2007-01-12 Thread Ken
Peter, I'm in agreement with you, as you say as long as it doesn't loose its core features. John, Its not that I want different. Its that sqlite could be made capable of handling internal synchronization. I certainly don't wish to loose the embedded capability nor its simplicity. I do

Re: [sqlite] 3.3.10 data corruption on updating fts1 string table

2007-01-12 Thread ohadp
looks like the file came in without CRLF, here goes: -- #include "../sqlite-3_3_8/sqlite3.h" #include static sqlite3* db; void exec_dml(const TCHAR* cmd) { sqlite3_stmt* vm;

Re: [sqlite] 3.3.10 data corruption on updating fts1 string table

2007-01-12 Thread ohadp
here's a file, plain.cpp, which demonstrates the problem. if you're compiling on linux and not windows you'll have to do a few trivial changes. _T("string") is used to define a unicode string TCHAR is WORD/SHORT/etc Scott Hess wrote: > > There haven't been any changes to the fts1 code! > >

Re: [sqlite] Pager modification question

2007-01-12 Thread John Stanton
If you want to share a file you have to be able to synchronize access in some way. The POSIX type file locks permit you to do it quite well by giving read and write locks. If you want shared access to a file from multiple processes you either need some form of co-operative lock like a

Re: [sqlite] Pager modification question

2007-01-12 Thread Peter James
On 1/12/07, Ken <[EMAIL PROTECTED]> wrote: If it is a replacement for fopen, then why does it perform locking at all? Since sqlite has implemented threading and multiple connections, then the next logical step in its evoloution is to enable advanced locking techniques for even greater degrees

Re: [sqlite] Pager modification question

2007-01-12 Thread Ken
If it is a replacement for fopen, then why does it perform locking at all? Since sqlite has implemented threading and multiple connections, then the next logical step in its evoloution is to enable advanced locking techniques for even greater degrees of improved concurrency. Ken John

Re: [sqlite] 3.3.10 data corruption on updating fts1 string table

2007-01-12 Thread Scott Hess
There haven't been any changes to the fts1 code! Your best bet is to put together a .c file with a main() which causes the problem. Or a tcl script, or anything that would allow someone to see things locally. The problem right now is that there's no particular way for any of us to find the

Re: [sqlite] Pager modification question

2007-01-12 Thread John Stanton
If Sqlite were to implement its own locking then it would lose simplicity and versatility. A good way to regard Sqlite is a replacement for fopen. It is a way an application can access a single file. If you want more than that you are not looking for "'lite" and should go straight to

Re: [sqlite] A Build Question

2007-01-12 Thread John Stanton
When I build GNU C for example it works that way by default. Rich Shepard wrote: As 3.3.10 builds here I want to ask why it's done from a directory different from that of the source? I believe sqlite3 is the only application I've build using the standard suite of build tools (autoconf,

Re: [sqlite] A Build Question

2007-01-12 Thread Rich Shepard
On Fri, 12 Jan 2007, Martin Jenkins wrote: Housekeeping - it keeps the built objects separate from the source objects. If you're building more than one version it keeps the built objects separate from other built objects. Python & wxWidgets use the same "build elsewhere" approach. You can

Re: [sqlite] A Build Question

2007-01-12 Thread Martin Jenkins
Rich Shepard wrote: As 3.3.10 builds here I want to ask why it's done from a directory different from that of the source? Housekeeping - it keeps the built objects separate from the source objects. If you're building more than one version it keeps the built objects separate from other built

Re: [sqlite] Pager modification question

2007-01-12 Thread Ken
Regarding the locking: Yes certainly only within a single process architecture. I understand that SQLITE is usein g file locks to co-ordinate multiple process locks for unix is fcntl. (Fcntl is an ugly beast, imho sqlite would be better served managing locks internally). I guess there

Re: [sqlite] Pager modification question

2007-01-12 Thread Ken
Regarding the Journal: I was thinking that this would be useful in the context of a single process multiple threads and shared cache. All that would be required is an additional thread to handle the logging. Christian Smith <[EMAIL PROTECTED]> wrote: Ken uttered: > Would it be possible

[sqlite] A Build Question

2007-01-12 Thread Rich Shepard
As 3.3.10 builds here I want to ask why it's done from a directory different from that of the source? I believe sqlite3 is the only application I've build using the standard suite of build tools (autoconf, make, etc.) that is not run from the source directory. Just curious, Rich -- Richard

Re: [sqlite] 3.3.10 data corruption on updating fts1 string table

2007-01-12 Thread ohadp
it looks like fts1/2 return an ascii string in disguise of a unicode string after an update to the data. John Stanton wrote: > > In that case your problem will be in your code, specifically in the > function which gives you a pointer to the data. > > Show us your code. > > ohadp wrote: >>

Re: [sqlite] How to handle dual-field Primary Key - only one is Autoincrement

2007-01-12 Thread Ken
Besides the trigger issue with sqlite. I think you have a design issue with your tables. You are using a composite key. Why not have a master table of customer I'ds that you maintain, whith only the customer_id as the PK and autoincrement. The the table you refer to would then be a child

Re: [sqlite] Pager modification question

2007-01-12 Thread Christian Smith
Ken uttered: Would it be possible to implement a Page level locking system using sqlite? Deadlock detection/recovery and/or prevention is much more complex for page level locking. As there is no shared data between SQLite clients other than the database file itself, tracking locks in other

Re: [sqlite] 3.3.10 data corruption on updating fts1 string table

2007-01-12 Thread ohadp
It returns 3 (text) both before and after the update. I bet someone who's familiar with fts1 code would immediately spot the change that caused this encoding bug Dan Kennedy-4 wrote: > > On Thu, 2007-01-11 at 15:00 -0800, ohadp wrote: >> here's the detailed description of the apparent bug: >>