Re: [sqlite] Is there any API for counting number of rows in a particular table.

2012-01-31 Thread John Elrick
On Tue, Jan 31, 2012 at 6:01 AM, Fabio Spadaro wrote: > Hi > > 2012/1/30 Bart Smissaert > > > How do you make it open database files that have an extension other than > > .slt? > > > > RBS > > > Tonight I will see to create a new version with bug fixes. > Meanwhile you could give me some feedback

Re: [sqlite] Found it

2012-01-26 Thread John Elrick
On Thu, Jan 26, 2012 at 12:36 PM, Stephan Beal wrote: > On Thu, Jan 26, 2012 at 2:53 PM, Mohit Sindhwani wrote: > > > Absolutely! I come home from work and tune in to this thread, gripped :) > > > > +1 to Mohit and the others who's written similar responses. i rarely follow > threads which don't

Re: [sqlite] Found it

2012-01-26 Thread John Elrick
On Thu, Jan 26, 2012 at 3:29 AM, Jean-Denis MUYS wrote: > > On 25 janv. 2012, at 22:30, John Elrick wrote: > > > > > I have to say this has been a fascinating learning experience. It has > been > > enlightening reading the SQLite code and beginning to under

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
On Wed, Jan 25, 2012 at 4:21 PM, Nico Williams wrote: > On Wed, Jan 25, 2012 at 3:07 PM, Richard Hipp wrote: > > The SQLite byte-code engine was being too conservative and was reparsing > > the schema in places where it was not strictly necessary. The fix was to > > restrict the places where the

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
On Wed, Jan 25, 2012 at 4:07 PM, Richard Hipp wrote: > On Wed, Jan 25, 2012 at 4:01 PM, John Elrick >wrote: > > > Much better. I wasn't sure which version to incorporate the changes into > > (the link is to the source), so I put them into 3.7.9. > > > >

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
Thanks. On Wed, Jan 25, 2012 at 4:11 PM, Petite Abeille wrote: > > On Jan 25, 2012, at 10:09 PM, John Elrick wrote: > > > DML? > > http://en.wikipedia.org/wiki/Data_Manipulation_Language > > > ___ > sqlite-users maili

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
On Wed, Jan 25, 2012 at 4:07 PM, Richard Hipp wrote: > On Wed, Jan 25, 2012 at 4:01 PM, John Elrick >wrote: > > > Much better. I wasn't sure which version to incorporate the changes into > > (the link is to the source), so I put them into 3.7.9. > > > >

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
__ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > -- John Elrick Fenestra Technologies 540-868-1377 ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
Richard, I've found a problem that I don't understand. Looking at the case OP_VerifyCookie, I attempted to output pOp->opcode, which I expected to equal OP_VerifyCookie. However, it doesn't: Schema has changed current state = 'db=0025, pOp->opCode=66785672, db->aDb[pOp->p1].pSchema=02110AB

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
Richard, Is it reasonable to say that, assuming: 1. The schema has not changed 2. There is only one connection to the database that the value of iGeneration should always equal pOp->p3 in OP_VerifyCookie? ___ sqlite-users mailing list sqlite-users@sq

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
On Wed, Jan 25, 2012 at 1:13 PM, Richard Hipp wrote: > On Wed, Jan 25, 2012 at 10:18 AM, John Elrick >wrote: > > > > > I do have some more information for you. I tracking everything which > could > > call reset and discovered that all of these are stemming f

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
Addendum On Wed, Jan 25, 2012 at 1:12 PM, John Elrick wrote: > On Wed, Jan 25, 2012 at 9:50 AM, Richard Hipp wrote: > >> On Wed, Jan 25, 2012 at 9:02 AM, John Elrick > >wrote: >> >> > >> > current state = 'db=03FB1188, db->aDb[pOp-&g

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
On Wed, Jan 25, 2012 at 9:50 AM, Richard Hipp wrote: > On Wed, Jan 25, 2012 at 9:02 AM, John Elrick >wrote: > > > > > current state = 'db=03FB1188, db->aDb[pOp->p1].pSchema=02110AB0, > > db->aDb[pOp->p1].pSchema->schema_cookie=27, > > db

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
Ok. Possible scenario. I can't prove it, but somebody tell me if this is what could be going on: prepare query1 prepare query2 create something -> schema change query1.step - schema invalid so recreate - recreation of schema changes the generation query2.step - schema invalid so recreate

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
On Wed, Jan 25, 2012 at 10:32 AM, Simon Slavin wrote: > > On 25 Jan 2012, at 3:18pm, John Elrick wrote: > > > So, it appears that something is making SQLite think the schema has > changed. > > Can you read the following: > > <http://www.sqlite.org/pragma.html#

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
On Wed, Jan 25, 2012 at 9:50 AM, Richard Hipp wrote: > On Wed, Jan 25, 2012 at 9:02 AM, John Elrick >wrote: > > > > > current state = 'db=03FB1188, db->aDb[pOp->p1].pSchema=02110AB0, > > db->aDb[pOp->p1].pSchema->schema_cookie=27, > > db

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
invoked. Whatever is resetting the internal schema isn't coming from here. On Wed, Jan 25, 2012 at 9:02 AM, John Elrick wrote: > > On Tue, Jan 24, 2012 at 7:46 PM, Richard Hipp wrote: > >> On Tue, Jan 24, 2012 at 7:37 PM, John Elrick > >wrote: >> >> > >> &

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
On Tue, Jan 24, 2012 at 7:46 PM, Richard Hipp wrote: > On Tue, Jan 24, 2012 at 7:37 PM, John Elrick >wrote: > > > > > It appears that the value is set here: > > > > u.av.pBt = db->aDb[pOp->p1].pBt; > > if( u.av.pBt ){ > >sqlit

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
On Wed, Jan 25, 2012 at 8:02 AM, Richard Hipp wrote: > On Wed, Jan 25, 2012 at 7:53 AM, John Elrick >wrote: > > > I started with the place where iGeneration was being changed and found > > nothing out of the ordinary. I started an examination and noticed the > > foll

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
...especially given that the results of sqlite3SchemaClear are being passed as a parameter and from the declaration, sqlite3SchemaClear doesn't return anything. On Wed, Jan 25, 2012 at 7:53 AM, John Elrick wrote: > I started with the place where iGeneration was being changed and found &

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
Init(&p->tblHash); sqlite3HashInit(&p->idxHash); sqlite3HashInit(&p->trigHash); sqlite3HashInit(&p->fkeyHash); p->enc = SQLITE_UTF8; } return p; } On Tue, Jan 24, 2012 at 7:46 PM, Richard Hipp wrote: > On Tue, Jan 24, 2012 at 7:37 PM, John Elri

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
On Tue, Jan 24, 2012 at 7:26 PM, Richard Hipp wrote: > On Tue, Jan 24, 2012 at 7:18 PM, John Elrick >wrote: > SNIP > So the problem was introduced by > > http://www.sqlite.org/src/info/36c04dd1695f08 > > Which is the fix for this bug: > > http://www.sqli

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
I really found it this time. While I was looking over the code changes, I noticed a section of code dealing with case OP_VerifyCookie: Inside that section is the following in 3.7.6 if( u.av.iMeta!=pOp->p2 || u.av.iGen!=pOp->p3 ){ sqlite3DbFree(db, p->zErrMsg); p->zErrMsg = sqlite3DbStr

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
Let me rephrase that... I ran this test with a small case. See results below. On Tue, Jan 24, 2012 at 5:55 PM, John Elrick wrote: SNIP > > http://www.sqlite.org/fenstra/sqlite-201102040051.zip >> > > Low malloc count > 8,191 calls to malloc (consistent with 3

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
On Tue, Jan 24, 2012 at 5:36 PM, Richard Hipp wrote: > On Tue, Jan 24, 2012 at 5:21 PM, John Elrick >wrote: > > > On Tue, Jan 24, 2012 at 3:05 PM, Richard Hipp wrote: > > > > > > > > OK. Please try these: > > > > > > http://www.sqlit

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
On Tue, Jan 24, 2012 at 3:05 PM, Richard Hipp wrote: > On Tue, Jan 24, 2012 at 1:28 PM, John Elrick >wrote: > > > > > > > > http://www.sqlite.org/fenstra/sqlite-201103081639.zip > > > > > > This one gives the high number of mallocs just as doe

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
Nope and nope. They are happening apparently out of the blue. On Jan 24, 2012 2:40 PM, "Nico Williams" wrote: > Maybe a view is getting materialized, or a some other temp table's > creation under the hood is triggering this? > ___ > sqlite-users mailin

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
On Tue, Jan 24, 2012 at 12:53 PM, Richard Hipp wrote: > On Tue, Jan 24, 2012 at 12:15 PM, John Elrick >wrote: > > > > > > > Can you put a breakpoint on sqlite3ResetInternalSchema() and figure out > > > from whence it is being called so often? > > > &

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
On Tue, Jan 24, 2012 at 1:28 PM, John Elrick wrote: SNIP http://www.sqlite.org/fenstra/sqlite-201103241737.zip > > > This one is failing to compile. It needs _msize. That sounds like the > 3.7.10 issue reported recently. > > I opened the sqlite3.c file and it

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
On Tue, Jan 24, 2012 at 1:29 PM, Krystian Bigaj wrote: > You can compile sqlite3.dll with debug information, and then you can in VS > attach to your running .exe - you should be able to see sqlite > stacks/breakpoints/step-in/out/etc. should also work. > > Thanks very much for the suggestion. We

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
On Tue, Jan 24, 2012 at 12:53 PM, Richard Hipp wrote: > On Tue, Jan 24, 2012 at 12:15 PM, John Elrick >wrote: > > SNIP > http://www.sqlite.org/fenstra/sqlite-201103081639.zip This one gives the high number of mallocs just as does 3.7.6. > > http://www.sqli

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
On Tue, Jan 24, 2012 at 1:18 PM, Dan Kennedy wrote: > On 01/25/2012 12:15 AM, John Elrick wrote: > >> On Tue, Jan 24, 2012 at 12:06 PM, Richard Hipp wrote: >> >> On Tue, Jan 24, 2012 at 11:59 AM, John >> Elrick >>> >>>> wrote: >>>&g

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
On Tue, Jan 24, 2012 at 12:06 PM, Richard Hipp wrote: > On Tue, Jan 24, 2012 at 11:59 AM, John Elrick >wrote: > > > While reviewing our other thread, I noticed a piece of information > > concerning sqlite3ResetInternalSchema. Based on the number of times > that I

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
While reviewing our other thread, I noticed a piece of information concerning sqlite3ResetInternalSchema. Based on the number of times that I reported it had been called, I believe that something was changed between 3.7.5 and 3.7.6 which causes sqlite3ResetInternalSchema to be called when it wasn'

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
Addendum. After looking further it appears that the reparsing is happening independently of triggering. In this case, the reparsing occurs 30,502 times for every view and trigger I have examined. Only one trigger appears in the 3.7.5 listing and it is parsed exactly 1 time. 3.7.5 CREATE TRIGGE

[sqlite] Found it

2012-01-24 Thread John Elrick
Richard, Your last request for some detailed information about what queries were being processed did the trick. Here is a description of the problem (and yes, it appears to be in SQLite). In 3.7.5, the number one malloc consumer is INSERT INTO SEQUENCE_ELEMENTS ( SEQUENCE_ELEMENT_OID, SEQUENCE_

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-23 Thread John Elrick
On Mon, Jan 23, 2012 at 4:27 PM, Nico Williams wrote: > On Mon, Jan 23, 2012 at 3:02 PM, John Elrick > wrote: > > I think I can inject something to do some measurements. I seem to > recall, > > however, that there was no substantive difference in the number of > > tim

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-23 Thread John Elrick
On Mon, Jan 23, 2012 at 3:48 PM, Richard Hipp wrote: > On Mon, Jan 23, 2012 at 3:12 PM, John Elrick >wrote: > > > Brain hurts... > > > > Richard, more information for you. I rebuilt the call stack checking > > system into pure Delphi and confirmed that yy_re

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-23 Thread John Elrick
Brain hurts... Richard, more information for you. I rebuilt the call stack checking system into pure Delphi and confirmed that yy_reduce appears to be the malloc culprit. I further created a pair of procedures which I can use to track the yyruleno from yy_reduce as though it were a call. Below

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-22 Thread John Elrick
to SQLite and any > pure examination of SQLite's behaviour isn't going to spot anything, right > ? So a better place to query this would be a Delphi list ? > > Simon. > ___ > sqlite-users mailing list > sqlite-users@sqlite.org

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-21 Thread John Elrick
ifestation. As frustrating as it has been to narrow down the cause, I have nothing but respect and gratitude for everyone who has attempted to help. Thank you all. On Jan 20, 2012 3:47 PM, "Richard Hipp" wrote: > On Fri, Jan 20, 2012 at 3:40 PM, John Elrick >wrote: > > > I

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-20 Thread John Elrick
n Fri, Jan 20, 2012 at 2:55 PM, John Elrick wrote: > The problem is not in the queries. The problem is in a two order of > magnitude increase in _mallocs between the versions. The _mallocs are > coming from sqlite3Parser. > > > On Fri, Jan 20, 2012 at 2:37 PM, Max Vlasov wrote:

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-20 Thread John Elrick
The problem is not in the queries. The problem is in a two order of magnitude increase in _mallocs between the versions. The _mallocs are coming from sqlite3Parser. On Fri, Jan 20, 2012 at 2:37 PM, Max Vlasov wrote: > On Fri, Jan 20, 2012 at 10:05 PM, John Elrick >wrote: > >

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-20 Thread John Elrick
The change which results in a slow down occurred between 3.7.5.0 and 3.7.6.0. On Fri, Jan 20, 2012 at 8:34 AM, John Elrick wrote: > Thank you sir. I'll start looking. > > > On Fri, Jan 20, 2012 at 8:27 AM, Richard Hipp wrote: > >> On Fri, Jan 20, 2012 at 8:1

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-20 Thread John Elrick
Thank you sir. I'll start looking. On Fri, Jan 20, 2012 at 8:27 AM, Richard Hipp wrote: > On Fri, Jan 20, 2012 at 8:14 AM, John Elrick >wrote: > > > I asked in an earlier posting if the amalgamations were available > > somewhere. > > > > > > http:/

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-20 Thread John Elrick
h good and bad EXPLAIN QUERY output here. I'm sure this will be greek > for most of us :), but when Richard or Dan look at the difference in the > vdbe code, they could notice something > > Max > > On Fri, Jan 20, 2012 at 1:17 AM, John Elrick >wrote: > > > One more

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-19 Thread John Elrick
One more useful comparison. The following query is prepared and step is called with these results: step: UPDATE RESPONSES SET RESPONSE_NAME = :RESPONSE_NAME WHERE RESPONSE_OID = :RESPONSE_OID Cumulative Allocated Memory: 10,992 Count of _malloc Calls:18 Cumulati

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-19 Thread John Elrick
On Thu, Jan 19, 2012 at 11:49 AM, John Elrick wrote: > I've added more tracking to our libraries. Most queries result in minimal > (<200 calls) _malloc activity, But I have found several anomalies. I am > listing some representatives below. In the examples below, "st

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-19 Thread John Elrick
I've added more tracking to our libraries. Most queries result in minimal (<200 calls) _malloc activity, But I have found several anomalies. I am listing some representatives below. In the examples below, "step" is the operation. Note also that all queries should have been prepared with sqlite3

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-19 Thread John Elrick
On Thu, Jan 19, 2012 at 8:25 AM, John Elrick wrote: > > > On Thu, Jan 19, 2012 at 8:15 AM, Richard Hipp wrote: > >> On Thu, Jan 19, 2012 at 8:10 AM, John Elrick > >wrote: >> >> > >> > I've been reading through the code. Do I understand correc

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-19 Thread John Elrick
On Thu, Jan 19, 2012 at 8:15 AM, Richard Hipp wrote: > On Thu, Jan 19, 2012 at 8:10 AM, John Elrick >wrote: > > > > > I've been reading through the code. Do I understand correctly that if > one > > prepared statement binding is changed in such a way as may in

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-19 Thread John Elrick
On Thu, Jan 19, 2012 at 8:03 AM, Richard Hipp wrote: > On Thu, Jan 19, 2012 at 7:51 AM, John Elrick >wrote: > > > Interesting: > > > > 1. There are no database schema changes occurring after the system is > fully > > initialized > > 2. There are no A

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-19 Thread John Elrick
triggering the re-parse which did not do so in the earlier version. On Thu, Jan 19, 2012 at 7:45 AM, Richard Hipp wrote: > On Thu, Jan 19, 2012 at 7:41 AM, John Elrick >wrote: > > > Question: > > > > If a query has already been prepared with sqlite3_prepare_v2, why wou

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-19 Thread John Elrick
Question: If a query has already been prepared with sqlite3_prepare_v2, why would sqlite3_step need to call sqlite3Prepare, which in turn calls sqlite3RunParser? ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-18 Thread John Elrick
On Wed, Jan 18, 2012 at 4:32 PM, Richard Hipp wrote: > On Wed, Jan 18, 2012 at 4:30 PM, John Elrick >wrote: > > > I made one more stab and narrowed down the mallocs to this call: > > > > Count:119,133 = 'sqlite3Parser_lt_YYNSTATEpYYNRULE(12) > > sql

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-18 Thread John Elrick
NRULE); I started looking at yy_reduce, but it looks machine generated. If you give me some direction I'll see if I can narrow it down further. On Wed, Jan 18, 2012 at 4:02 PM, John Elrick wrote: > Whatever is happening, it appears I can only reproduce it under these > circumstances.

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-18 Thread John Elrick
Whatever is happening, it appears I can only reproduce it under these circumstances. I'll continue digging into the procedure tomorrow to see if I can narrow down where this is coming from. On Wed, Jan 18, 2012 at 3:54 PM, John Elrick wrote: > FYI, I traced the number of

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-18 Thread John Elrick
FYI, I traced the number of calls to sqlite3ResetInternalSchema() on a smaller set of data. It appears that it isn't the culprit. Total mallocs: 148,156 Total calls to sqlite3ResetInternalSchema(): 63 On Wed, Jan 18, 2012 at 3:41 PM, John Elrick wrote: > What else could trigger a

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-18 Thread John Elrick
What else could trigger a call to sqlite3ResetInternalSchema()? I'm getting a clear breakpoint tracing back to such innocuous calls as _sqlite3_step. On Wed, Jan 18, 2012 at 3:31 PM, Richard Hipp wrote: > On Wed, Jan 18, 2012 at 3:12 PM, John Elrick >wrote: > > > >

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-18 Thread John Elrick
I could not use any of the proffered tools for several reasons. After an inspiration yesterday, I dusted off my C programming skills and started hacking a pseudo-call stack system into SQLite3.c. I've put the statistics below for the top couple of call stacks. Total times _malloc called in test:

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-17 Thread John Elrick
On Mon, Jan 16, 2012 at 8:05 PM, John Elrick wrote: > > > On Mon, Jan 16, 2012 at 5:31 PM, Richard Hipp wrote: > >> On Mon, Jan 16, 2012 at 5:13 PM, John Elrick > >wrote: >> >> > I did this trace through the Delphi IDE and hand copied the call stack. >

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-17 Thread John Elrick
On Mon, Jan 16, 2012 at 5:36 PM, Richard Hipp wrote: > On Mon, Jan 16, 2012 at 5:31 PM, John Elrick >wrote: > > SNIP > This is very curious and perhaps a useful clue. SQLite does call > SNIP It is a clue that I need a break. While updating the counters to allow for nil p

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-17 Thread John Elrick
On Mon, Jan 16, 2012 at 11:47 PM, Max Vlasov wrote: > On Tue, Jan 17, 2012 at 5:05 AM, John Elrick >wrote: > > > > > > I can try that approach with BCC. I was concerned that the IDE would be > > getting inaccurate information from the .OBJ. Crossing into a >

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-16 Thread John Elrick
On Mon, Jan 16, 2012 at 5:31 PM, Richard Hipp wrote: > On Mon, Jan 16, 2012 at 5:13 PM, John Elrick >wrote: > > > I did this trace through the Delphi IDE and hand copied the call stack. > > Unfortunately, I can't give you line numbers, although I could go back >

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-16 Thread John Elrick
I will adjust the realloc tests tomorrow and let you know. The two routines are unique in the Delphi library unit. Thanks again for your time and efforts. On Mon, Jan 16, 2012 at 5:36 PM, Richard Hipp wrote: > On Mon, Jan 16, 2012 at 5:31 PM, John Elrick >wrote: > > > > >

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-16 Thread John Elrick
Richard, I don't know if the calls to _realloc will help also, but I've added tracing for those also. My updated output: Application: Surveyor.exe Sqlite version: 3.7.9 Date/Time:01/16/2012 17:20:58 Memory Used:

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-16 Thread John Elrick
On Mon, Jan 16, 2012 at 5:14 PM, Simon Slavin wrote: > > On 16 Jan 2012, at 9:37pm, John Elrick wrote: > > > Lacking the status interfaces for 3.6.17 I was forced to mark them as > > "interface not supported". Simon suggested I track the calls to _malloc, >

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-16 Thread John Elrick
I did this trace through the Delphi IDE and hand copied the call stack. Unfortunately, I can't give you line numbers, although I could go back and give you hex offsets from the start of the procedures. These are literally the first seven call stacks from _malloc after I initiate the trace. I'm n

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-16 Thread John Elrick
Addendum: This test was conducted using a full size test dataset. I had been using one that was an order of magnitude smaller for profiling before, but could use the full size for this test. On Mon, Jan 16, 2012 at 4:37 PM, John Elrick wrote: > Lacking the status interfaces for 3.6.17 I

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-16 Thread John Elrick
Lacking the status interfaces for 3.6.17 I was forced to mark them as "interface not supported". Simon suggested I track the calls to _malloc, so I've added extensive tracing code. Here is the final output from the two versions: Application: Surveyor.exe Sqlite version:

Re: [sqlite] sql/tcl script

2012-01-16 Thread John Elrick
_ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > -- John Elrick Fenestra Technologies 540-868-1377 ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-16 Thread John Elrick
On Fri, Jan 13, 2012 at 5:58 PM, Richard Hipp wrote: > On Fri, Jan 13, 2012 at 4:49 PM, John Elrick >wrote: > > > > > 3.6.17 > > > > Procedure % TimeTimeCalls > > _sqlite3_step 58.4% 1.795,052 >

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-16 Thread John Elrick
On Mon, Jan 16, 2012 at 10:33 AM, Richard Hipp wrote: > On Mon, Jan 16, 2012 at 10:29 AM, John Elrick >wrote: > > > > > > > > SQLite has lots of interfaces that can be used to determine performance > > and > > > status information. Some

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-16 Thread John Elrick
On Fri, Jan 13, 2012 at 5:58 PM, Richard Hipp wrote: > On Fri, Jan 13, 2012 at 4:49 PM, John Elrick >wrote: > SNIP > SQLite has lots of interfaces that can be used to determine performance and > status information. Some example code from the command-line shell that > ac

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-16 Thread John Elrick
On Mon, Jan 16, 2012 at 4:37 AM, Max Vlasov wrote: > On Sun, Jan 15, 2012 at 3:35 AM, John Elrick >wrote: > > > On Fri, Jan 13, 2012 at 5:13 PM, Max Vlasov > wrote: > > > > 5,008 calls to > > > > UPDATE RESPONSES SET > > RESPONSE_NAME = :RESPO

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-14 Thread John Elrick
On Fri, Jan 13, 2012 at 5:13 PM, Max Vlasov wrote: > John, > I read your previous post and unfortunately, your conversation with Richard > didn't reveal much details. I worked with scenario similar tou yours > (Delphi + statically linked sqlite) for several years and during this > period an advan

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-14 Thread John Elrick
On Sat, Jan 14, 2012 at 10:42 AM, Dan Kennedy wrote: > On 01/14/2012 04:49 AM, John Elrick wrote: > >> >> So, obviously the problem is that _malloc is being called a much larger >> number of times by 3.7.9 in this particular circumstance -- roughly 70 >> times a

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-14 Thread John Elrick
On Fri, Jan 13, 2012 at 5:58 PM, Richard Hipp wrote: > On Fri, Jan 13, 2012 at 4:49 PM, John Elrick >wrote: > > > > > 3.6.17 > > > > Procedure % TimeTimeCalls > > _sqlite3_step 58.4% 1.795,052 >

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-14 Thread John Elrick
On Fri, Jan 13, 2012 at 5:20 PM, Pavel Ivanov wrote: > > At this point I've exhausted my avenues of research. Does anyone have > any > > further ideas as to what may be going on which could cause this increase > in > > memory management activity? Or any suggestions of where to look next? > > II

[sqlite] Some pieces of the puzzle are coming together

2012-01-13 Thread John Elrick
e any further ideas as to what may be going on which could cause this increase in memory management activity? Or any suggestions of where to look next? -- John Elrick Fenestra Technologies 540-868-1377 ___ sqlite-users mailing list sqlite-users@sqlit

[sqlite] Is there some hidden way to create a log record of Sqlite commands?

2012-01-11 Thread John Elrick
ite, there doesn't appear to be a built in mechanism for creating this type of log. Is this correct or am I missing something? -- John Elrick Fenestra Technologies 540-868-1377 ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.

Re: [sqlite] Problems encountered while upgrading Sqlite from 3.6.7 to 3.7.9

2012-01-10 Thread John Elrick
otentially relevant information. I did add an assertion which runs during initialization to ensure that the version is as expected. I have confirmed that both the shell application and the full application concur on version numbers. On Tue, Jan 10, 2012 at 10:35 AM, John Elrick wrote: > O

Re: [sqlite] Problems encountered while upgrading Sqlite from 3.6.7 to 3.7.9

2012-01-10 Thread John Elrick
On Tue, Jan 10, 2012 at 10:14 AM, Richard Hipp wrote: > On Tue, Jan 10, 2012 at 10:08 AM, John Elrick wrote: > >> On Tue, Jan 10, 2012 at 8:31 AM, Richard Hipp wrote: >> > On Tue, Jan 10, 2012 at 8:14 AM, John Elrick > >wrote: >> > >> >> I will

Re: [sqlite] Problems encountered while upgrading Sqlite from 3.6.7 to 3.7.9

2012-01-10 Thread John Elrick
On Tue, Jan 10, 2012 at 8:31 AM, Richard Hipp wrote: > On Tue, Jan 10, 2012 at 8:14 AM, John Elrick wrote: > >> I will have to get back to you on this.  While running tests against >> isolated queries, I noticed an unusual circumstance.  When I isolate >> the queries into

Re: [sqlite] Problems encountered while upgrading Sqlite from 3.6.7 to 3.7.9

2012-01-10 Thread John Elrick
Mon, Jan 9, 2012 at 11:13 AM, Richard Hipp wrote: > >> Please send private email to d...@sqlite.org.  CC to d...@sqlite.org and >> j...@sqlite.org. >> >> >> On Mon, Jan 9, 2012 at 11:09 AM, John Elrick wrote: >> >>> Sorry "The combined files

Re: [sqlite] Problems encountered while upgrading Sqlite from 3.6.7 to 3.7.9

2012-01-09 Thread John Elrick
Sorry "The combined files are 2.4 MB zipped" On Mon, Jan 9, 2012 at 11:09 AM, John Elrick wrote: > Richard, > > I have the information available for you.  Unfortunately, it appears > that I cannot actually reduce the case, however, the issues appear to > be fairly confin

Re: [sqlite] Problems encountered while upgrading Sqlite from 3.6.7 to 3.7.9

2012-01-09 Thread John Elrick
. The combined files are 2.4 MB. How would you prefer I send this file to you? On Fri, Jan 6, 2012 at 4:51 PM, John Elrick wrote: > Yes.  Look for it on Monday.  I'm about to leave for the day. The > queries are automatically generated and will require some mining to > get specifi

Re: [sqlite] Problems encountered while upgrading Sqlite from 3.6.7 to 3.7.9

2012-01-06 Thread John Elrick
. On Fri, Jan 6, 2012 at 4:42 PM, Richard Hipp wrote: > Can you publish for us: > > (1) A sample database > (2) The specific queries that are causing your problems > > So that we can investigate further? > > On Fri, Jan 6, 2012 at 4:31 PM, John Elrick wrote: > >> B

[sqlite] Problems encountered while upgrading Sqlite from 3.6.7 to 3.7.9

2012-01-06 Thread John Elrick
pleased to hear them. For the moment we are reverting to 3.6.7 as a temporary solution. -- John Elrick Fenestra Technologies 540-868-1377 ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] parallelizing an update

2010-02-02 Thread John Elrick
Robert Citek wrote: > Are there some white papers or examples of how to do updates in > parallel using sqlite? > > I have a large dataset in sqlite that I need to process outside of > sqlite and then update the sqlite database. The process looks > something like this: > > sqlite3 -separator $'\t'

Re: [sqlite] BUG: the rowid column in view is autom a tically named as id

2009-12-16 Thread John Elrick
Alexey Pechnikov wrote: > Hello! > > On Monday 14 December 2009 11:33:32 Jens Miltner wrote: > >> ... >> > > You did not understand me. The manually named columns > may not be renamed automatically. But in the test the > column "id" was renamed to "id:1": > > sqlite> create table test (id

Re: [sqlite] SQL question

2009-12-15 Thread John Elrick
Florian Schricker wrote: > Hello Simon, > SNIP > Please excuse me for being so blunt: > You have no idea what I'm supposed to do here nor do you have any idea > for whatever historic reason the database or table is designed "as is" > here nor (and finally) do you have no idea about my education

Re: [sqlite] is ticket 4037 not a bug?

2009-11-19 Thread John Elrick
r; 1|hello sqlite> select myfoo.foo.* from myfoo.foo r; Error: near "*": syntax error sqlite> I cannot think of a good reason why: select myfoo.foo.* from myfoo.foo; would be expected to produce different results than: select r.* from myfoo.foo r; If the

Re: [sqlite] LAN and exclusive lock

2009-10-21 Thread John Elrick
Jan wrote: > Hi, > > Although I read in a recent post by drh that it is not recommended to > use sqlite in a local network I would like to ask if the following > approach would work: > > A database in the local network needs to be accessed by about 20 people. > I suppose the max. number at the s

Re: [sqlite] Question regarding BCC32

2009-10-13 Thread John Elrick
Cariotoglou Mike wrote: > Great. thanks for the info. however, this means that at least part of the > runtime library actually comes from the VCL, and not the BCC32 libraries, is > that not so ? > There are some OBJ files needed, we picked them up from somewhere, but I don't recall exactly w

Re: [sqlite] Question regarding BCC32

2009-10-12 Thread John Elrick
Cariotoglou Mike wrote: > Has anybody had experience using Borland's free compiler (bcc32) to > build the sqlite dll ? We use bcc32 to create an .obj which is linked into our Delphi application. No issues here and we've been using it for three years on a project deployed to several hundred thou

Re: [sqlite] Improving select distinct performance

2009-10-08 Thread John Elrick
Jason Freshwater wrote: > If this comes out garbled again the answer is java 1.6 on 1.8Ghz windows xp > > i get about 500,000 rows/second reading single character values via xerial > jdbc > Thanks, I did some additional experiments with only integers and got similar results. I had always bee

Re: [sqlite] Improving select distinct performance

2009-10-08 Thread John Elrick
Jason Freshwater wrote: > 2. Read every record into my application and use a hashtable to > do my own select distinct. This is pretty good, about 0.6s to > achieve the distinct list (I can count each distinct value > along the way which is a useful side effect). Would you mind my asking what langu

Re: [sqlite] Synchronising multiple copies of a database

2009-10-08 Thread John Elrick
Jean-Denis Muys wrote: > On 10/7/09 21:35 , "Adam DeVita" wrote: > > >> One can also get a mess if Mr. Red and Mr Black both get new customers, and >> enter them and they both get the same ID because the auto-generated int >> happens to be the same. Both copies get updated with the other guy's

Re: [sqlite] Synchronising multiple copies of a database

2009-10-07 Thread John Elrick
Simon Slavin wrote: > On 7 Oct 2009, at 8:33pm, John Elrick wrote: > > >> Isn't this a variation of the DVCS problem? In other words, would it >> be correct in saying that the underlying issue is treating this as a >> database problem, rather than it

  1   2   3   >