Re: [sqlite] Is there any API for counting number of rows in a particular table.

2012-01-31 Thread John Elrick
On Tue, Jan 31, 2012 at 6:01 AM, Fabio Spadaro fabiolinos...@gmail.comwrote: Hi 2012/1/30 Bart Smissaert bart.smissa...@gmail.com How do you make it open database files that have an extension other than .slt? RBS Tonight I will see to create a new version with bug fixes. Meanwhile

Re: [sqlite] Found it

2012-01-26 Thread John Elrick
On Thu, Jan 26, 2012 at 3:29 AM, Jean-Denis MUYS jdm...@kleegroup.comwrote: On 25 janv. 2012, at 22:30, John Elrick wrote: I have to say this has been a fascinating learning experience. It has been enlightening reading the SQLite code and beginning to understand its inner workings

Re: [sqlite] Found it

2012-01-26 Thread John Elrick
On Thu, Jan 26, 2012 at 12:36 PM, Stephan Beal sgb...@googlemail.comwrote: On Thu, Jan 26, 2012 at 2:53 PM, Mohit Sindhwani m...@onghu.com wrote: Absolutely! I come home from work and tune in to this thread, gripped :) +1 to Mohit and the others who's written similar responses. i rarely

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
); sqlite3HashInit(p-idxHash); sqlite3HashInit(p-trigHash); sqlite3HashInit(p-fkeyHash); p-enc = SQLITE_UTF8; } return p; } On Tue, Jan 24, 2012 at 7:46 PM, Richard Hipp d...@sqlite.org wrote: On Tue, Jan 24, 2012 at 7:37 PM, John Elrick john.elr...@fenestra.com wrote: It appears

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
...especially given that the results of sqlite3SchemaClear are being passed as a parameter and from the declaration, sqlite3SchemaClear doesn't return anything. On Wed, Jan 25, 2012 at 7:53 AM, John Elrick john.elr...@fenestra.comwrote: I started with the place where iGeneration was being

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
On Wed, Jan 25, 2012 at 8:02 AM, Richard Hipp d...@sqlite.org wrote: On Wed, Jan 25, 2012 at 7:53 AM, John Elrick john.elr...@fenestra.com wrote: I started with the place where iGeneration was being changed and found nothing out of the ordinary. I started an examination and noticed

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
On Tue, Jan 24, 2012 at 7:46 PM, Richard Hipp d...@sqlite.org wrote: On Tue, Jan 24, 2012 at 7:37 PM, John Elrick john.elr...@fenestra.com wrote: It appears that the value is set here: u.av.pBt = db-aDb[pOp-p1].pBt; if( u.av.pBt ){ sqlite3BtreeGetMeta(u.av.pBt

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
the internal schema isn't coming from here. On Wed, Jan 25, 2012 at 9:02 AM, John Elrick john.elr...@fenestra.comwrote: On Tue, Jan 24, 2012 at 7:46 PM, Richard Hipp d...@sqlite.org wrote: On Tue, Jan 24, 2012 at 7:37 PM, John Elrick john.elr...@fenestra.com wrote: It appears that the value

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
On Wed, Jan 25, 2012 at 9:50 AM, Richard Hipp d...@sqlite.org wrote: On Wed, Jan 25, 2012 at 9:02 AM, John Elrick john.elr...@fenestra.com wrote: current state = 'db=03FB1188, db-aDb[pOp-p1].pSchema=02110AB0, db-aDb[pOp-p1].pSchema-schema_cookie=27, db-aDb[pOp-p1].pSchema-iGeneration

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
On Wed, Jan 25, 2012 at 10:32 AM, Simon Slavin slav...@bigfraud.org wrote: On 25 Jan 2012, at 3:18pm, John Elrick wrote: So, it appears that something is making SQLite think the schema has changed. Can you read the following: http://www.sqlite.org/pragma.html#pragma_schema_version

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
Ok. Possible scenario. I can't prove it, but somebody tell me if this is what could be going on: prepare query1 prepare query2 create something - schema change query1.step - schema invalid so recreate - recreation of schema changes the generation query2.step - schema invalid so recreate

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
On Wed, Jan 25, 2012 at 9:50 AM, Richard Hipp d...@sqlite.org wrote: On Wed, Jan 25, 2012 at 9:02 AM, John Elrick john.elr...@fenestra.com wrote: current state = 'db=03FB1188, db-aDb[pOp-p1].pSchema=02110AB0, db-aDb[pOp-p1].pSchema-schema_cookie=27, db-aDb[pOp-p1].pSchema-iGeneration

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
Addendum On Wed, Jan 25, 2012 at 1:12 PM, John Elrick john.elr...@fenestra.comwrote: On Wed, Jan 25, 2012 at 9:50 AM, Richard Hipp d...@sqlite.org wrote: On Wed, Jan 25, 2012 at 9:02 AM, John Elrick john.elr...@fenestra.com wrote: current state = 'db=03FB1188, db-aDb[pOp-p1].pSchema

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
On Wed, Jan 25, 2012 at 1:13 PM, Richard Hipp d...@sqlite.org wrote: On Wed, Jan 25, 2012 at 10:18 AM, John Elrick john.elr...@fenestra.com wrote: I do have some more information for you. I tracking everything which could call reset and discovered that all of these are stemming from

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
Richard, Is it reasonable to say that, assuming: 1. The schema has not changed 2. There is only one connection to the database that the value of iGeneration should always equal pOp-p3 in OP_VerifyCookie? ___ sqlite-users mailing list

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
Richard, I've found a problem that I don't understand. Looking at the case OP_VerifyCookie, I attempted to output pOp-opcode, which I expected to equal OP_VerifyCookie. However, it doesn't: Schema has changed current state = 'db=0025, pOp-opCode=66785672, db-aDb[pOp-p1].pSchema=02110AB0,

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users -- John Elrick Fenestra Technologies 540-868-1377 ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
On Wed, Jan 25, 2012 at 4:07 PM, Richard Hipp d...@sqlite.org wrote: On Wed, Jan 25, 2012 at 4:01 PM, John Elrick john.elr...@fenestra.com wrote: Much better. I wasn't sure which version to incorporate the changes into (the link is to the source), so I put them into 3.7.9. The total

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
Thanks. On Wed, Jan 25, 2012 at 4:11 PM, Petite Abeille petite.abei...@gmail.comwrote: On Jan 25, 2012, at 10:09 PM, John Elrick wrote: DML? http://en.wikipedia.org/wiki/Data_Manipulation_Language ___ sqlite-users mailing list sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
On Wed, Jan 25, 2012 at 4:07 PM, Richard Hipp d...@sqlite.org wrote: On Wed, Jan 25, 2012 at 4:01 PM, John Elrick john.elr...@fenestra.com wrote: Much better. I wasn't sure which version to incorporate the changes into (the link is to the source), so I put them into 3.7.9. The total

Re: [sqlite] Found it

2012-01-25 Thread John Elrick
On Wed, Jan 25, 2012 at 4:21 PM, Nico Williams n...@cryptonector.comwrote: On Wed, Jan 25, 2012 at 3:07 PM, Richard Hipp d...@sqlite.org wrote: The SQLite byte-code engine was being too conservative and was reparsing the schema in places where it was not strictly necessary. The fix was to

[sqlite] Found it

2012-01-24 Thread John Elrick
Richard, Your last request for some detailed information about what queries were being processed did the trick. Here is a description of the problem (and yes, it appears to be in SQLite). In 3.7.5, the number one malloc consumer is INSERT INTO SEQUENCE_ELEMENTS ( SEQUENCE_ELEMENT_OID,

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
Addendum. After looking further it appears that the reparsing is happening independently of triggering. In this case, the reparsing occurs 30,502 times for every view and trigger I have examined. Only one trigger appears in the 3.7.5 listing and it is parsed exactly 1 time. 3.7.5 CREATE

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
While reviewing our other thread, I noticed a piece of information concerning sqlite3ResetInternalSchema. Based on the number of times that I reported it had been called, I believe that something was changed between 3.7.5 and 3.7.6 which causes sqlite3ResetInternalSchema to be called when it

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
On Tue, Jan 24, 2012 at 12:06 PM, Richard Hipp d...@sqlite.org wrote: On Tue, Jan 24, 2012 at 11:59 AM, John Elrick john.elr...@fenestra.com wrote: While reviewing our other thread, I noticed a piece of information concerning sqlite3ResetInternalSchema. Based on the number of times

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
On Tue, Jan 24, 2012 at 1:18 PM, Dan Kennedy danielk1...@gmail.com wrote: On 01/25/2012 12:15 AM, John Elrick wrote: On Tue, Jan 24, 2012 at 12:06 PM, Richard Hippd...@sqlite.org wrote: On Tue, Jan 24, 2012 at 11:59 AM, John Elrickjohn.elrick@fenestra.**comjohn.elr...@fenestra.com

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
On Tue, Jan 24, 2012 at 12:53 PM, Richard Hipp d...@sqlite.org wrote: On Tue, Jan 24, 2012 at 12:15 PM, John Elrick john.elr...@fenestra.com wrote: SNIP http://www.sqlite.org/fenstra/sqlite-201103081639.zip This one gives the high number of mallocs just as does 3.7.6. http

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
On Tue, Jan 24, 2012 at 1:29 PM, Krystian Bigaj krystian.bi...@gmail.comwrote: You can compile sqlite3.dll with debug information, and then you can in VS attach to your running .exe - you should be able to see sqlite stacks/breakpoints/step-in/out/etc. should also work. Thanks very much for

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
On Tue, Jan 24, 2012 at 1:28 PM, John Elrick john.elr...@fenestra.comwrote: SNIP http://www.sqlite.org/fenstra/sqlite-201103241737.zip This one is failing to compile. It needs _msize. That sounds like the 3.7.10 issue reported recently. I opened the sqlite3.c file and it is version

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
On Tue, Jan 24, 2012 at 12:53 PM, Richard Hipp d...@sqlite.org wrote: On Tue, Jan 24, 2012 at 12:15 PM, John Elrick john.elr...@fenestra.com wrote: Can you put a breakpoint on sqlite3ResetInternalSchema() and figure out from whence it is being called so often? Just did

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
Nope and nope. They are happening apparently out of the blue. On Jan 24, 2012 2:40 PM, Nico Williams n...@cryptonector.com wrote: Maybe a view is getting materialized, or a some other temp table's creation under the hood is triggering this? ___

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
On Tue, Jan 24, 2012 at 3:05 PM, Richard Hipp d...@sqlite.org wrote: On Tue, Jan 24, 2012 at 1:28 PM, John Elrick john.elr...@fenestra.com wrote: http://www.sqlite.org/fenstra/sqlite-201103081639.zip This one gives the high number of mallocs just as does 3.7.6. OK. Please try

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
On Tue, Jan 24, 2012 at 5:36 PM, Richard Hipp d...@sqlite.org wrote: On Tue, Jan 24, 2012 at 5:21 PM, John Elrick john.elr...@fenestra.com wrote: On Tue, Jan 24, 2012 at 3:05 PM, Richard Hipp d...@sqlite.org wrote: OK. Please try these: http://www.sqlite.org/fenstra/sqlite

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
Let me rephrase that... I ran this test with a small case. See results below. On Tue, Jan 24, 2012 at 5:55 PM, John Elrick john.elr...@fenestra.comwrote: SNIP http://www.sqlite.org/fenstra/sqlite-201102040051.zip Low malloc count 8,191 calls to malloc (consistent with 3.7.5

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
I really found it this time. While I was looking over the code changes, I noticed a section of code dealing with case OP_VerifyCookie: Inside that section is the following in 3.7.6 if( u.av.iMeta!=pOp-p2 || u.av.iGen!=pOp-p3 ){ sqlite3DbFree(db, p-zErrMsg); p-zErrMsg =

Re: [sqlite] Found it

2012-01-24 Thread John Elrick
On Tue, Jan 24, 2012 at 7:26 PM, Richard Hipp d...@sqlite.org wrote: On Tue, Jan 24, 2012 at 7:18 PM, John Elrick john.elr...@fenestra.com wrote: SNIP So the problem was introduced by http://www.sqlite.org/src/info/36c04dd1695f08 Which is the fix for this bug: http

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-23 Thread John Elrick
Brain hurts... Richard, more information for you. I rebuilt the call stack checking system into pure Delphi and confirmed that yy_reduce appears to be the malloc culprit. I further created a pair of procedures which I can use to track the yyruleno from yy_reduce as though it were a call. Below

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-23 Thread John Elrick
On Mon, Jan 23, 2012 at 3:48 PM, Richard Hipp d...@sqlite.org wrote: On Mon, Jan 23, 2012 at 3:12 PM, John Elrick john.elr...@fenestra.com wrote: Brain hurts... Richard, more information for you. I rebuilt the call stack checking system into pure Delphi and confirmed that yy_reduce

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-23 Thread John Elrick
On Mon, Jan 23, 2012 at 4:27 PM, Nico Williams n...@cryptonector.comwrote: On Mon, Jan 23, 2012 at 3:02 PM, John Elrick john.elr...@fenestra.com wrote: I think I can inject something to do some measurements. I seem to recall, however, that there was no substantive difference in the number

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-22 Thread John Elrick
a better place to query this would be a Delphi list ? Simon. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users -- John Elrick Fenestra Technologies 540-868-1377

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-21 Thread John Elrick
. As frustrating as it has been to narrow down the cause, I have nothing but respect and gratitude for everyone who has attempted to help. Thank you all. On Jan 20, 2012 3:47 PM, Richard Hipp d...@sqlite.org wrote: On Fri, Jan 20, 2012 at 3:40 PM, John Elrick john.elr...@fenestra.com wrote: I

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-20 Thread John Elrick
output here. I'm sure this will be greek for most of us :), but when Richard or Dan look at the difference in the vdbe code, they could notice something Max On Fri, Jan 20, 2012 at 1:17 AM, John Elrick john.elr...@fenestra.com wrote: One more useful comparison. The following query

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-20 Thread John Elrick
Thank you sir. I'll start looking. On Fri, Jan 20, 2012 at 8:27 AM, Richard Hipp d...@sqlite.org wrote: On Fri, Jan 20, 2012 at 8:14 AM, John Elrick john.elr...@fenestra.com wrote: I asked in an earlier posting if the amalgamations were available somewhere. http://www.sqlite.org

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-20 Thread John Elrick
The change which results in a slow down occurred between 3.7.5.0 and 3.7.6.0. On Fri, Jan 20, 2012 at 8:34 AM, John Elrick john.elr...@fenestra.comwrote: Thank you sir. I'll start looking. On Fri, Jan 20, 2012 at 8:27 AM, Richard Hipp d...@sqlite.org wrote: On Fri, Jan 20, 2012 at 8:14 AM

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-20 Thread John Elrick
The problem is not in the queries. The problem is in a two order of magnitude increase in _mallocs between the versions. The _mallocs are coming from sqlite3Parser. On Fri, Jan 20, 2012 at 2:37 PM, Max Vlasov max.vla...@gmail.com wrote: On Fri, Jan 20, 2012 at 10:05 PM, John Elrick john.elr

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-20 Thread John Elrick
20, 2012 at 2:55 PM, John Elrick john.elr...@fenestra.comwrote: The problem is not in the queries. The problem is in a two order of magnitude increase in _mallocs between the versions. The _mallocs are coming from sqlite3Parser. On Fri, Jan 20, 2012 at 2:37 PM, Max Vlasov max.vla

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-19 Thread John Elrick
Question: If a query has already been prepared with sqlite3_prepare_v2, why would sqlite3_step need to call sqlite3Prepare, which in turn calls sqlite3RunParser? ___ sqlite-users mailing list sqlite-users@sqlite.org

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-19 Thread John Elrick
triggering the re-parse which did not do so in the earlier version. On Thu, Jan 19, 2012 at 7:45 AM, Richard Hipp d...@sqlite.org wrote: On Thu, Jan 19, 2012 at 7:41 AM, John Elrick john.elr...@fenestra.com wrote: Question: If a query has already been prepared with sqlite3_prepare_v2, why

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-19 Thread John Elrick
On Thu, Jan 19, 2012 at 8:03 AM, Richard Hipp d...@sqlite.org wrote: On Thu, Jan 19, 2012 at 7:51 AM, John Elrick john.elr...@fenestra.com wrote: Interesting: 1. There are no database schema changes occurring after the system is fully initialized 2. There are no ATTACHed databases

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-19 Thread John Elrick
On Thu, Jan 19, 2012 at 8:15 AM, Richard Hipp d...@sqlite.org wrote: On Thu, Jan 19, 2012 at 8:10 AM, John Elrick john.elr...@fenestra.com wrote: I've been reading through the code. Do I understand correctly that if one prepared statement binding is changed in such a way as may

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-19 Thread John Elrick
On Thu, Jan 19, 2012 at 8:25 AM, John Elrick john.elr...@fenestra.comwrote: On Thu, Jan 19, 2012 at 8:15 AM, Richard Hipp d...@sqlite.org wrote: On Thu, Jan 19, 2012 at 8:10 AM, John Elrick john.elr...@fenestra.com wrote: I've been reading through the code. Do I understand correctly

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-19 Thread John Elrick
I've added more tracking to our libraries. Most queries result in minimal (200 calls) _malloc activity, But I have found several anomalies. I am listing some representatives below. In the examples below, step is the operation. Note also that all queries should have been prepared with

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-19 Thread John Elrick
On Thu, Jan 19, 2012 at 11:49 AM, John Elrick john.elr...@fenestra.comwrote: I've added more tracking to our libraries. Most queries result in minimal (200 calls) _malloc activity, But I have found several anomalies. I am listing some representatives below. In the examples below, step

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-19 Thread John Elrick
One more useful comparison. The following query is prepared and step is called with these results: step: UPDATE RESPONSES SET RESPONSE_NAME = :RESPONSE_NAME WHERE RESPONSE_OID = :RESPONSE_OID Cumulative Allocated Memory: 10,992 Count of _malloc Calls:18

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-18 Thread John Elrick
I could not use any of the proffered tools for several reasons. After an inspiration yesterday, I dusted off my C programming skills and started hacking a pseudo-call stack system into SQLite3.c. I've put the statistics below for the top couple of call stacks. Total times _malloc called in

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-18 Thread John Elrick
What else could trigger a call to sqlite3ResetInternalSchema()? I'm getting a clear breakpoint tracing back to such innocuous calls as _sqlite3_step. On Wed, Jan 18, 2012 at 3:31 PM, Richard Hipp d...@sqlite.org wrote: On Wed, Jan 18, 2012 at 3:12 PM, John Elrick john.elr...@fenestra.com

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-18 Thread John Elrick
FYI, I traced the number of calls to sqlite3ResetInternalSchema() on a smaller set of data. It appears that it isn't the culprit. Total mallocs: 148,156 Total calls to sqlite3ResetInternalSchema(): 63 On Wed, Jan 18, 2012 at 3:41 PM, John Elrick john.elr...@fenestra.comwrote: What else could

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-18 Thread John Elrick
Whatever is happening, it appears I can only reproduce it under these circumstances. I'll continue digging into the procedure tomorrow to see if I can narrow down where this is coming from. On Wed, Jan 18, 2012 at 3:54 PM, John Elrick john.elr...@fenestra.comwrote: FYI, I traced the number

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-18 Thread John Elrick
started looking at yy_reduce, but it looks machine generated. If you give me some direction I'll see if I can narrow it down further. On Wed, Jan 18, 2012 at 4:02 PM, John Elrick john.elr...@fenestra.comwrote: Whatever is happening, it appears I can only reproduce it under these circumstances

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-18 Thread John Elrick
On Wed, Jan 18, 2012 at 4:32 PM, Richard Hipp d...@sqlite.org wrote: On Wed, Jan 18, 2012 at 4:30 PM, John Elrick john.elr...@fenestra.com wrote: I made one more stab and narrowed down the mallocs to this call: Count:119,133 = 'sqlite3Parser_lt_YYNSTATEpYYNRULE(12) sqlite3Parser(11

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-17 Thread John Elrick
On Mon, Jan 16, 2012 at 11:47 PM, Max Vlasov max.vla...@gmail.com wrote: On Tue, Jan 17, 2012 at 5:05 AM, John Elrick john.elr...@fenestra.com wrote: I can try that approach with BCC. I was concerned that the IDE would be getting inaccurate information from the .OBJ. Crossing

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-17 Thread John Elrick
On Mon, Jan 16, 2012 at 5:36 PM, Richard Hipp d...@sqlite.org wrote: On Mon, Jan 16, 2012 at 5:31 PM, John Elrick john.elr...@fenestra.com wrote: SNIP This is very curious and perhaps a useful clue. SQLite does call SNIP It is a clue that I need a break. While updating the counters

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-17 Thread John Elrick
On Mon, Jan 16, 2012 at 8:05 PM, John Elrick john.elr...@fenestra.comwrote: On Mon, Jan 16, 2012 at 5:31 PM, Richard Hipp d...@sqlite.org wrote: On Mon, Jan 16, 2012 at 5:13 PM, John Elrick john.elr...@fenestra.com wrote: I did this trace through the Delphi IDE and hand copied the call

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-16 Thread John Elrick
On Mon, Jan 16, 2012 at 4:37 AM, Max Vlasov max.vla...@gmail.com wrote: On Sun, Jan 15, 2012 at 3:35 AM, John Elrick john.elr...@fenestra.com wrote: On Fri, Jan 13, 2012 at 5:13 PM, Max Vlasov max.vla...@gmail.com wrote: 5,008 calls to UPDATE RESPONSES SET RESPONSE_NAME

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-16 Thread John Elrick
On Fri, Jan 13, 2012 at 5:58 PM, Richard Hipp d...@sqlite.org wrote: On Fri, Jan 13, 2012 at 4:49 PM, John Elrick john.elr...@fenestra.com wrote: SNIP SQLite has lots of interfaces that can be used to determine performance and status information. Some example code from the command-line

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-16 Thread John Elrick
On Mon, Jan 16, 2012 at 10:33 AM, Richard Hipp d...@sqlite.org wrote: On Mon, Jan 16, 2012 at 10:29 AM, John Elrick john.elr...@fenestra.com wrote: SQLite has lots of interfaces that can be used to determine performance and status information. Some example code from the command

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-16 Thread John Elrick
On Fri, Jan 13, 2012 at 5:58 PM, Richard Hipp d...@sqlite.org wrote: On Fri, Jan 13, 2012 at 4:49 PM, John Elrick john.elr...@fenestra.com wrote: 3.6.17 Procedure % TimeTimeCalls _sqlite3_step 58.4% 1.795,052 _memcpy

Re: [sqlite] sql/tcl script

2012-01-16 Thread John Elrick
list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users -- John Elrick Fenestra Technologies 540-868-1377 ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-16 Thread John Elrick
Lacking the status interfaces for 3.6.17 I was forced to mark them as interface not supported. Simon suggested I track the calls to _malloc, so I've added extensive tracing code. Here is the final output from the two versions: Application: Surveyor.exe Sqlite version:

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-16 Thread John Elrick
Addendum: This test was conducted using a full size test dataset. I had been using one that was an order of magnitude smaller for profiling before, but could use the full size for this test. On Mon, Jan 16, 2012 at 4:37 PM, John Elrick john.elr...@fenestra.comwrote: Lacking the status

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-16 Thread John Elrick
I did this trace through the Delphi IDE and hand copied the call stack. Unfortunately, I can't give you line numbers, although I could go back and give you hex offsets from the start of the procedures. These are literally the first seven call stacks from _malloc after I initiate the trace. I'm

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-16 Thread John Elrick
On Mon, Jan 16, 2012 at 5:14 PM, Simon Slavin slav...@bigfraud.org wrote: On 16 Jan 2012, at 9:37pm, John Elrick wrote: Lacking the status interfaces for 3.6.17 I was forced to mark them as interface not supported. Simon suggested I track the calls to _malloc, so I've added extensive

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-16 Thread John Elrick
Richard, I don't know if the calls to _realloc will help also, but I've added tracing for those also. My updated output: Application: Surveyor.exe Sqlite version: 3.7.9 Date/Time:01/16/2012 17:20:58 Memory Used:

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-16 Thread John Elrick
I will adjust the realloc tests tomorrow and let you know. The two routines are unique in the Delphi library unit. Thanks again for your time and efforts. On Mon, Jan 16, 2012 at 5:36 PM, Richard Hipp d...@sqlite.org wrote: On Mon, Jan 16, 2012 at 5:31 PM, John Elrick john.elr...@fenestra.com

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-16 Thread John Elrick
On Mon, Jan 16, 2012 at 5:31 PM, Richard Hipp d...@sqlite.org wrote: On Mon, Jan 16, 2012 at 5:13 PM, John Elrick john.elr...@fenestra.com wrote: I did this trace through the Delphi IDE and hand copied the call stack. Unfortunately, I can't give you line numbers, although I could go back

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-14 Thread John Elrick
On Fri, Jan 13, 2012 at 5:20 PM, Pavel Ivanov paiva...@gmail.com wrote: At this point I've exhausted my avenues of research. Does anyone have any further ideas as to what may be going on which could cause this increase in memory management activity? Or any suggestions of where to look

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-14 Thread John Elrick
On Fri, Jan 13, 2012 at 5:58 PM, Richard Hipp d...@sqlite.org wrote: On Fri, Jan 13, 2012 at 4:49 PM, John Elrick john.elr...@fenestra.com wrote: 3.6.17 Procedure % TimeTimeCalls _sqlite3_step 58.4% 1.795,052 _memcpy

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-14 Thread John Elrick
On Sat, Jan 14, 2012 at 10:42 AM, Dan Kennedy danielk1...@gmail.com wrote: On 01/14/2012 04:49 AM, John Elrick wrote: So, obviously the problem is that _malloc is being called a much larger number of times by 3.7.9 in this particular circumstance -- roughly 70 times as often. _memcpy

Re: [sqlite] Some pieces of the puzzle are coming together

2012-01-14 Thread John Elrick
On Fri, Jan 13, 2012 at 5:13 PM, Max Vlasov max.vla...@gmail.com wrote: John, I read your previous post and unfortunately, your conversation with Richard didn't reveal much details. I worked with scenario similar tou yours (Delphi + statically linked sqlite) for several years and during this

[sqlite] Some pieces of the puzzle are coming together

2012-01-13 Thread John Elrick
further ideas as to what may be going on which could cause this increase in memory management activity? Or any suggestions of where to look next? -- John Elrick Fenestra Technologies 540-868-1377 ___ sqlite-users mailing list sqlite-users@sqlite.org

[sqlite] Is there some hidden way to create a log record of Sqlite commands?

2012-01-11 Thread John Elrick
to be a built in mechanism for creating this type of log. Is this correct or am I missing something? -- John Elrick Fenestra Technologies 540-868-1377 ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo

Re: [sqlite] Problems encountered while upgrading Sqlite from 3.6.7 to 3.7.9

2012-01-10 Thread John Elrick
:13 AM, Richard Hipp d...@sqlite.org wrote: Please send private email to d...@sqlite.org.  CC to d...@sqlite.org and j...@sqlite.org. On Mon, Jan 9, 2012 at 11:09 AM, John Elrick john.elr...@fenestra.comwrote: Sorry The combined files are 2.4 MB zipped On Mon, Jan 9, 2012 at 11:09 AM, John

Re: [sqlite] Problems encountered while upgrading Sqlite from 3.6.7 to 3.7.9

2012-01-10 Thread John Elrick
On Tue, Jan 10, 2012 at 8:31 AM, Richard Hipp d...@sqlite.org wrote: On Tue, Jan 10, 2012 at 8:14 AM, John Elrick john.elr...@fenestra.comwrote: I will have to get back to you on this.  While running tests against isolated queries, I noticed an unusual circumstance.  When I isolate

Re: [sqlite] Problems encountered while upgrading Sqlite from 3.6.7 to 3.7.9

2012-01-10 Thread John Elrick
On Tue, Jan 10, 2012 at 10:14 AM, Richard Hipp d...@sqlite.org wrote: On Tue, Jan 10, 2012 at 10:08 AM, John Elrick john.elr...@fenestra.comwrote: On Tue, Jan 10, 2012 at 8:31 AM, Richard Hipp d...@sqlite.org wrote: On Tue, Jan 10, 2012 at 8:14 AM, John Elrick john.elr...@fenestra.com wrote

Re: [sqlite] Problems encountered while upgrading Sqlite from 3.6.7 to 3.7.9

2012-01-09 Thread John Elrick
. The combined files are 2.4 MB. How would you prefer I send this file to you? On Fri, Jan 6, 2012 at 4:51 PM, John Elrick john.elr...@fenestra.com wrote: Yes.  Look for it on Monday.  I'm about to leave for the day. The queries are automatically generated and will require some mining to get

Re: [sqlite] Problems encountered while upgrading Sqlite from 3.6.7 to 3.7.9

2012-01-09 Thread John Elrick
Sorry The combined files are 2.4 MB zipped On Mon, Jan 9, 2012 at 11:09 AM, John Elrick john.elr...@fenestra.com wrote: Richard, I have the information available for you.  Unfortunately, it appears that I cannot actually reduce the case, however, the issues appear to be fairly confined

[sqlite] Problems encountered while upgrading Sqlite from 3.6.7 to 3.7.9

2012-01-06 Thread John Elrick
be pleased to hear them. For the moment we are reverting to 3.6.7 as a temporary solution. -- John Elrick Fenestra Technologies 540-868-1377 ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Problems encountered while upgrading Sqlite from 3.6.7 to 3.7.9

2012-01-06 Thread John Elrick
. On Fri, Jan 6, 2012 at 4:42 PM, Richard Hipp d...@sqlite.org wrote: Can you publish for us: (1) A sample database (2) The specific queries that are causing your problems So that we can investigate further? On Fri, Jan 6, 2012 at 4:31 PM, John Elrick john.elr...@fenestra.comwrote: Background

Re: [sqlite] parallelizing an update

2010-02-02 Thread John Elrick
Robert Citek wrote: Are there some white papers or examples of how to do updates in parallel using sqlite? I have a large dataset in sqlite that I need to process outside of sqlite and then update the sqlite database. The process looks something like this: sqlite3 -separator $'\t'

Re: [sqlite] BUG: the rowid column in view is autom a tically named as id

2009-12-16 Thread John Elrick
Alexey Pechnikov wrote: Hello! On Monday 14 December 2009 11:33:32 Jens Miltner wrote: ... You did not understand me. The manually named columns may not be renamed automatically. But in the test the column id was renamed to id:1: sqlite create table test (id INTEGER PRIMARY

Re: [sqlite] SQL question

2009-12-15 Thread John Elrick
Florian Schricker wrote: Hello Simon, SNIP Please excuse me for being so blunt: You have no idea what I'm supposed to do here nor do you have any idea for whatever historic reason the database or table is designed as is here nor (and finally) do you have no idea about my education on

Re: [sqlite] is ticket 4037 not a bug?

2009-11-19 Thread John Elrick
: select myfoo.foo.* from myfoo.foo; would be expected to produce different results than: select r.* from myfoo.foo r; If the former is invalid, then the latter should be also and vis versa. My own opinion, but there it is. FWIW, John Elrick cid:part1.02020305.03070606@fenestra.com

Re: [sqlite] LAN and exclusive lock

2009-10-21 Thread John Elrick
Jan wrote: Hi, Although I read in a recent post by drh that it is not recommended to use sqlite in a local network I would like to ask if the following approach would work: A database in the local network needs to be accessed by about 20 people. I suppose the max. number at the same

Re: [sqlite] Question regarding BCC32

2009-10-13 Thread John Elrick
Cariotoglou Mike wrote: Great. thanks for the info. however, this means that at least part of the runtime library actually comes from the VCL, and not the BCC32 libraries, is that not so ? There are some OBJ files needed, we picked them up from somewhere, but I don't recall exactly

Re: [sqlite] Synchronising multiple copies of a database

2009-10-08 Thread John Elrick
Jean-Denis Muys wrote: On 10/7/09 21:35 , Adam DeVita adev...@verifeye.com wrote: One can also get a mess if Mr. Red and Mr Black both get new customers, and enter them and they both get the same ID because the auto-generated int happens to be the same. Both copies get updated with the

Re: [sqlite] Improving select distinct performance

2009-10-08 Thread John Elrick
Jason Freshwater wrote: 2. Read every record into my application and use a hashtable to do my own select distinct. This is pretty good, about 0.6s to achieve the distinct list (I can count each distinct value along the way which is a useful side effect). Would you mind my asking what language

Re: [sqlite] Improving select distinct performance

2009-10-08 Thread John Elrick
Jason Freshwater wrote: If this comes out garbled again the answer is java 1.6 on 1.8Ghz windows xp i get about 500,000 rows/second reading single character values via xerial jdbc Thanks, I did some additional experiments with only integers and got similar results. I had always been

Re: [sqlite] Synchronising multiple copies of a database

2009-10-07 Thread John Elrick
Simon Slavin wrote: On 7 Oct 2009, at 7:20pm, Adam DeVita wrote: regarding this The fault is that almost nobody does it right: they neglect to keep an 'unaltered central copy' and think they can cross-apply journals each time two databases talk to one-another. That does not work for

Re: [sqlite] Synchronising multiple copies of a database

2009-10-07 Thread John Elrick
Simon Slavin wrote: On 7 Oct 2009, at 8:33pm, John Elrick wrote: Isn't this a variation of the DVCS problem? In other words, would it be correct in saying that the underlying issue is treating this as a database problem, rather than it being a versioning problem which happens

Re: [sqlite] Binding parameters to prepared statements

2009-08-25 Thread John Elrick
(m_insert, 2, idItem); sqlite3_bind_blob(m_insert, 1, p, DATA_KEY_RECORD_LENGTH, FreeBlob); Assuming m_insert is your statement. HTH John Elrick Fenestra Technologies ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin

  1   2   3   >