Re: [sqlite] Memory leak in csv.c virtual table module

[Richard Hipp]

Fixed now on trunk.  Thanks for the bug report.

On 4/24/18, Ralf Junker  wrote:
> This SQL causes a memory leak in the csv.c virtual table module:
>
>CREATE VIRTUAL TABLE if not exists t1
>  USING csv(filename='FileDoesNotExists.csv');
>
> Cause is that the zIn buffer which is allocated here:
>
>http://localhost:8081/artifact?name=1a009b93650732e2&ln=128
>
> is only freed in csv_reader_reset() if the file was opened successfully
> and the file handle is assigned:
>
>http://localhost:8081/artifact?name=1a009b93650732e2&ln=102-106
>
> Hence no file open, no file handle, no buffer freed, memory leak.
>
> Ralf
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>


-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Memory leak in csv.c virtual table module

[Ralf Junker]

This SQL causes a memory leak in the csv.c virtual table module:

  CREATE VIRTUAL TABLE if not exists t1
USING csv(filename='FileDoesNotExists.csv');

Cause is that the zIn buffer which is allocated here:

  http://localhost:8081/artifact?name=1a009b93650732e2&ln=128

is only freed in csv_reader_reset() if the file was opened successfully 
and the file handle is assigned:


  http://localhost:8081/artifact?name=1a009b93650732e2&ln=102-106

Hence no file open, no file handle, no buffer freed, memory leak.

Ralf
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] memory leak?

[Peter Da Silva]

It is normal in all modern operating systems for unused memory to be allocated 
to buffer cache, so over time the "free" memory Is expected to go down unless 
disk activity is near quiescent.  Some operating systems will combine "cache" 
and "free" memory to hide this from the casual user, but I consider this 
deceptive.

On 4/12/18, 9:34 PM, "sqlite-users on behalf of king3306" 
 
wrote:

first thanks you reply 

After running for some time  i found sqlite3_memory_used return value keep
constant about 2M,but linux free memory is fewer and fewer,i make sure no
other places are leaked,why?

this whether or not a normal behavior?

if not ,how can i to analysis this problem?  

Looking forward to your answer



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] memory leak?

[king3306]

first thanks you reply 

After running for some time  i found sqlite3_memory_used return value keep
constant about 2M,but linux free memory is fewer and fewer,i make sure no
other places are leaked,why?

this whether or not a normal behavior?

if not ,how can i to analysis this problem?  

Looking forward to your answer



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] memory leak?

[king3306]

my platform is armv5 not support valgrind and lint,

can you help me ?



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] memory leak?

[king3306]

thans your reply , when find the problem,i try to use valgrind tool,but i
find this tools is not support armv5,i use mcu is AT91SAM9G25



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] memory leak?

[Tim Streater]

On 12 Apr 2018, at 21:34, Simon Slavin  wrote:

> On 12 Apr 2018, at 8:00pm, Warren Young  wrote:
>
>> Also: http://valgrind.org/docs/manual/faq.html#faq.pronounce
>
> I didn't know that.  So they want it to be pronounced like 'lint'.
> Interesting.

That they want it pronounced that way is a matter of indifference to me.


-- 
Cheers  --  Tim
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] memory leak?

[Simon Slavin]

On 12 Apr 2018, at 8:00pm, Warren Young  wrote:

> Also: http://valgrind.org/docs/manual/faq.html#faq.pronounce

I didn't know that.  So they want it to be pronounced like 'lint'.  Interesting.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] memory leak?

[Warren Young]

On Apr 12, 2018, at 11:14 AM, Jens Alfke  wrote:
> 
> I'm a Mac/iOS developer so I use the 'leaks' tool and Instruments app; I 
> don't know how this is done on Linux.

Valgrind: http://valgrind.org/

Also: http://valgrind.org/docs/manual/faq.html#faq.pronounce

:)
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] memory leak?

[Jens Alfke]

Trying to find memory leaks by looking at the OS memory statistics is 
pointless. It's like looking at satellite photos to find a dropped contact 
lens. :) The kernel's virtual memory system (on any modern OS) is very complex, 
and the behavior of malloc/free in a process is also complex.

If you want to look for memory leaks, the best way is to use 
(platform-specific) instrumentation tools for that purpose. I'm a Mac/iOS 
developer so I use the 'leaks' tool and Instruments app; I don't know how this 
is done on Linux.

You can also look at the process's overall heap size (again, the name of this 
varies by platform, it's called RPRVT on Mac OS) and see if it's growing 
monotonically over time. But this is only a rough guide, as malloc/free don't 
always free up address space when heap blocks are freed.

—Jens
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] memory leak?

[Richard Hipp]

On 4/12/18, king3306 <1809860...@qq.com> wrote:
>
> 1:
> before i use insert cmd ,the linux memory and sqlite3_memory_used is
> linux: tota=29126656 used=16998400 free=12128256
> sqlite3_memory_used:1372576
>
> after i use inset cmd ,the linux memory and sqlite3_memory_used is
> linux: tota=29126656 used=17608704 free=11517952
> sqlite3_memory_used:1710016
>
> why 1710016-1372576=337440   !=   12128256-11517952=610304

When SQLite calls free() to release memory back to the native memory
allocation, free() does not necessarily release that memory back to
the operating system.  Usually free() will keep that memory around to
be reused for the next malloc() request.  I think you are seeing the
memory that free() has kept back for reuse.

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] memory leak?

[king3306]

sorry i made a mistake  SQLite version 3.3.6  is linux built-in sqlite3


i use is SQLITE_VERSION"3.22.0" on arm 




--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] memory leak?

[Richard Hipp]

On 4/12/18, king3306 <1809860...@qq.com> wrote:
>
> I am use SQLite version 3.3.6

Version 3.3.6 dates from 2006-06-06.  Version 3.23.1 is current.
Please upgrade.  Report back if you are still having problems.

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] memory leak?

[king3306]

HI

I am use SQLite version 3.3.6

default page_size is 4096  , i set 1024

i have four question ,i don't understand 


1:
before i use insert cmd ,the linux memory and sqlite3_memory_used is 
linux: tota=29126656 used=16998400 free=12128256
sqlite3_memory_used:1372576

after i use inset cmd ,the linux memory and sqlite3_memory_used is 
linux: tota=29126656 used=17608704 free=11517952
sqlite3_memory_used:1710016

why 1710016-1372576=337440   !=   12128256-11517952=610304


2: 
i keep the sqlite3 run two days,i found when i use sqlite select cmd or
insert cmd  i found sqlite3_memory_used = 2315024 is not change but linux
free memory is reduce(use free cmd)

if i  comment out the select cmd and insert cmd ,linux free memory is not
change

this whether or not  a Normal behavior ?


3:
sqlite3_memory_used  return value whether or not include the memory leak?

4:
i found anther situation,when linux free memory reduce to about 1M,free
memory immediately increase to about 8M




my code:

void insert_table_data(mtype_group_str table,uint8_t class_type,char
time_str[])
{
char sql_cmd[MAX_BUFF_LEN];
char meter_addr_str[CJT188_ADDR_LEN*2+100];
int32_t result;
sqlite3_stmt *stmt=NULL;
uint8_t id_index;
uint16_t meter_index;
uint32_t di_offset=0;
printf_mem_infor("insert_table_data1");
begin_transaction();
printf_mem_infor("insert_table_data2");
make_insert_cmd(table,class_type,sql_cmd);
printf_mem_infor("insert_table_data3");
result = sqlite3_prepare_v2(sqlite_mdb.mdb,sql_cmd,-1,&stmt,NULL);
if(result != SQLITE_OK)
{
elog_d(SYS_LOG_MDB,"insert_table_data:result=%d",result);
sqlite3_finalize(stmt);
rollback_transaction();
return ;
}
printf_mem_infor("insert_table_data4");
for(meter_index = 0; meter_index < table.m_num; meter_index++)
{
result = sqlite3_bind_text(stmt,1,xxx,-1,SQLITE_STATIC);
if(result != SQLITE_OK)
{
elog_d(SYS_LOG_MDB,"insert_table_data:result1=%d",result);
}
di_offset = 0;
if(class_type == REAL_DATA)
{
result = sqlite3_bind_text(stmt,2,time_str,-1,SQLITE_STATIC);
if(result != SQLITE_OK)
{
elog_d(SYS_LOG_MDB,"insert_table_data:result2=%d",result);
}
}
else if(class_type == DAY_DATA)
{
result = sqlite3_bind_text(stmt,2,time_str,-1,SQLITE_STATIC);
if(result != SQLITE_OK)
{
elog_d(SYS_LOG_MDB,"insert_table_data:result3=%d",result);
}
}
else if(class_type == MONTH_DATA)
{
result = sqlite3_bind_text(stmt,2,time_str,-1,SQLITE_STATIC);
if(result != SQLITE_OK)
{
elog_d(SYS_LOG_MDB,"insert_table_data:result4=%d",result);
}
}
for(id_index = 0; id_index < table.mtype_infor[class_type].di_sum;
id_index++)
{
di_offset += ID_INDEX_SIZE;
result =
sqlite3_bind_blob(stmt,id_index+3,xxx,SQLITE_STATIC);
if(result != SQLITE_OK)
{
elog_d(SYS_LOG_MDB,"insert_table_data:result5=%d",result);
}
di_offset += DATA_TIME_SIZE +
table.mtype_infor[class_type].mtype_di[id_index].data_len;
}
printf_mem_infor("insert_table_data5");
result = sqlite3_step(stmt);
printf_mem_infor("insert_table_data6");
if(result != SQLITE_DONE)
{
elog_d(SYS_LOG_MDB,"result = sqlite3_step(stmt)=%d",result);
rollback_transaction();
}
printf_mem_infor("insert_table_data7");
result = sqlite3_reset(stmt);
printf_mem_infor("insert_table_data8");
if(result != SQLITE_OK)
{
elog_d(SYS_LOG_MDB,"result = sqlite3_reset(stmt)=%d",result);
}
printf_mem_infor("insert_table_data9");
result = sqlite3_clear_bindings(stmt);
printf_mem_infor("insert_table_data10");
if(result != SQLITE_OK)
{
elog_d(SYS_LOG_MDB,"result =
sqlite3_clear_bindings(stmt)=%d",result);
}
}
printf_mem_infor("insert_table_data11");
sqlite3_finalize(stmt);
printf_mem_infor("insert_table_data12");
commit_transaction();
printf_mem_infor("insert_table_data13");
}



uint8_t get_fre_infor(mtype_group_str table,uint8_t class_type,char
time_str[])
{
char sql_cmd[MAX_BUFF_LEN];
int32_t result;
char *errmsg=NULL,**dbresult=NULL;
int16_t amr_time_index = 0;
int32_t nrow = 0,ncol=0;

sprintf(sql_cmd,"select distinct amr_time from tab_%d_%d_%c order by
amr_time asc",table.big_type,table.sml_type,data_type_mark[class_type]);
result =
sqlite3_get_table(sqlite_mdb.mdb,sql_cmd,&dbresult,&nrow,&ncol,&errmsg);
elog_d(SYS_LOG_MDB,"get_fre_infor:ncol=%d nrow=%d",ncol,nrow);
if(result == SQLITE_OK)
{
for(amr_time_index = 0; amr_

Re: [sqlite] memory leak

[Yuriy M. Kaminskiy]

On 11/05/17 03:28 , Lev wrote:
> I'm fighting with some memory leak. From time to time the vsize of the
> process goes up with 100k. I have several function like this, and I
> call them in each second. My memory growth happens every 10 minutes or
> so.
>
> int dbSqliteSelectSensorsToReport(sdmd_t *sdmd, sensor_t *sensor, int
> deviceType, int target) { char sqlQuery[256];
>   char *tableName;
>   char *targetName;
>   int haveWork = 1;
>   int ret, retVal;
>   sqlite3_stmt *sq3Stmt;

+   sq3Stmt = NULL;

>   /*Reset the state whatever it is*/
>
>   sensor->stat = SENSOR_DATA_INVALID;
>
>   do {
>
>   switch (deviceType) {
>   case SENSOR_TYPE_AN:
>   tableName = "analog";
>   break;
>   case SENSOR_TYPE_DIG:
>   tableName = "digital";
>   break;
>   case SENSOR_TYPE_UNKNOWN:
>   default:
>   tableName = NULL;
>   break;
>   }
>
>
>   switch (target) {
>   case TARGET_MQTT:
>   targetName = "read_mqtt";
>   break;
>   case TARGET_MODBUS: /*This makes no sence,
>   hence it is done in an other process... but whatever*/
>   targetName = NULL; break;
>   default:
>   targetName = NULL;
>   break;
>   }
>
>   if (targetName == NULL || tableName == NULL) {
>   retVal = DB_SQLITE_ERR;
>   break;
>   }
>
>   sprintf(sqlQuery, "SELECT rowid, sensor_id, value, time
>   FROM %s WHERE %s=0 LIMIT 1;", tableName,
>   targetName); /*Read one by one. I don't want to block
>   DB access.*/
>
>   sqlite3_prepare_v2(sdmd->db, sqlQuery, -1, &sq3Stmt,
>   NULL);
>
>   while (haveWork == 1) {
>   ret = sqlite3_step(sq3Stmt);
>   switch (ret) {
>   case SQLITE_ROW:
>   /*Read the data*/
>   sensor->id =
>   sqlite3_column_int64(sq3Stmt, 0); sensor->sensorId =
>   sqlite3_column_int64(sq3Stmt, 1); sensor->value =
>   sqlite3_column_double(sq3Stmt, 2); sensor->time =
>   sqlite3_column_int64(sq3Stmt, 3); sensor->stat =
>   SENSOR_DATA_VALID; sprintf(sensor->name, "%s_%d",
>   tableName, sensor->sensorId); break;
>   case SQLITE_DONE:
>   haveWork = 0;
>   retVal = DB_SQLITE_OK;
>   break;
>   case SQLITE_BUSY:
>   debug(sdmd->dbg, DEBUG_ERR, "%s():
>   sqlite (BUSY): %s\n", __func__,
>   sqlite3_errmsg(sdmd->db)); haveWork = 0; retVal =
>   DB_SQLITE_ERR; break;
>   default:
>   haveWork = 0;
>   debug(sdmd->dbg, DEBUG_ERR, "%s():
>   sqlite (%d): %s\n", __func__, ret,
>   sqlite3_errmsg(sdmd->db)); debug(sdmd->dbg, DEBUG_ERR,
>   "%s(): sqlite query: %s\n", __func__, sqlQuery); retVal
>   = DB_SQLITE_ERR; break; }
>   }
>   } while(0);
>
>   sqlite3_finalize(sq3Stmt); /*Release the dB*/
>
>   return retVal;
> }
>
>
> What do I do wrong?

I don't see anything that can trigger memory leak here.

You seems don't initialize sq3Stmt (see above +), so that you will call
sqlite3_finalize on random junk from stack on some error paths; but that
should not trigger any leaks (more likely result would be heap
corruption and crash).

And while current uses are probably safe, I'd replace sprintf with
snprintf to avoid any nasty surprises.

Maybe sqlite just use some memory for page cache (IIRC, by default,
cache size is 2MB).

Also allocated vsize can increase due to memory fragmentation (depending
on chosen malloc implementation and its fine-tuning).

You can also try running with valgrind or other memory debugger.

> Any help is appreciated.

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] memory leak

[Lev]

I'm fighting with some memory leak. From time to time the vsize of the
process goes up with 100k. I have several function like this, and I
call them in each second. My memory growth happens every 10 minutes or
so.



int dbSqliteSelectSensorsToReport(sdmd_t *sdmd, sensor_t *sensor, int
deviceType, int target) { char sqlQuery[256];
char *tableName;
char *targetName;
int haveWork = 1;
int ret, retVal;
sqlite3_stmt *sq3Stmt;


/*Reset the state whatever it is*/

sensor->stat = SENSOR_DATA_INVALID;

do {

switch (deviceType) {
case SENSOR_TYPE_AN:
tableName = "analog";
break;
case SENSOR_TYPE_DIG:
tableName = "digital";
break;
case SENSOR_TYPE_UNKNOWN:
default:
tableName = NULL;
break;
}


switch (target) {
case TARGET_MQTT:
targetName = "read_mqtt";
break;
case TARGET_MODBUS: /*This makes no sence,
hence it is done in an other process... but whatever*/
targetName = NULL; break;
default:
targetName = NULL;
break;
}

if (targetName == NULL || tableName == NULL) {
retVal = DB_SQLITE_ERR;
break;
}

sprintf(sqlQuery, "SELECT rowid, sensor_id, value, time
FROM %s WHERE %s=0 LIMIT 1;", tableName,
targetName); /*Read one by one. I don't want to block
DB access.*/

sqlite3_prepare_v2(sdmd->db, sqlQuery, -1, &sq3Stmt,
NULL);

while (haveWork == 1) {
ret = sqlite3_step(sq3Stmt);
switch (ret) {
case SQLITE_ROW:
/*Read the data*/
sensor->id =
sqlite3_column_int64(sq3Stmt, 0); sensor->sensorId =
sqlite3_column_int64(sq3Stmt, 1); sensor->value =
sqlite3_column_double(sq3Stmt, 2); sensor->time =
sqlite3_column_int64(sq3Stmt, 3); sensor->stat =
SENSOR_DATA_VALID; sprintf(sensor->name, "%s_%d",
tableName, sensor->sensorId); break;
case SQLITE_DONE:
haveWork = 0;
retVal = DB_SQLITE_OK;
break;
case SQLITE_BUSY:
debug(sdmd->dbg, DEBUG_ERR, "%s():
sqlite (BUSY): %s\n", __func__,
sqlite3_errmsg(sdmd->db)); haveWork = 0; retVal =
DB_SQLITE_ERR; break;
default:
haveWork = 0;
debug(sdmd->dbg, DEBUG_ERR, "%s():
sqlite (%d): %s\n", __func__, ret,
sqlite3_errmsg(sdmd->db)); debug(sdmd->dbg, DEBUG_ERR,
"%s(): sqlite query: %s\n", __func__, sqlQuery); retVal
= DB_SQLITE_ERR; break; }
}
} while(0);

sqlite3_finalize(sq3Stmt); /*Release the dB*/

return retVal;
}


What do I do wrong?

Any help is appreciated.
Levente
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Memory leak?

[Matthias Schmitt]

Hello,

sorry for the late answer. My weekend was full of events.

> On 13 Mar 2015, at 23:32, Gregory Moore  wrote:
> 
> Are you interacting directly with sqlite?  Or are you using a 3rd party API 
> such as FMDB?  I take it you are not using Core Data (which optionally can 
> use a sqlite database as a datastore.)

I am using FMDB. BTW, this was visible in the call stack I sent in my first 
mail.

> Are you using ARC?  

No. The first versions of my programs are more than 3 years old. At this time I 
made bad experiences with ARC, so I do not like ARC. 

> Have you implemented the -(void)dealloc method for all your Objective-C (or 
> Swift?) objects?

Yes.

> Are you willing to share some of your application code where it interacts 
> with the sqlite database?

No problem with that. Here is a minimum example, which leaks memory.

int numberOfRestos;

FMResultSet *rs = [db executeQuery:@"SELECT count() FROM restaurant"];
while ([rs next]) {
  numberOfRestos = [rs intForColumnIndex:0];
}
[rs close];

I have chosen this example, because it does not involve any complex parameter 
passing or result handling.
This code was working perfectly without memory leaks for years. Now this 
example is leaking two memory segments of ~4.5 KB each.

Interesting fact: I am getting the same memory leaks when I am compiling Sqlite 
from source or if I am using the precompiled library, which comes with the 
Xcode development environment.

Best regards

Matthias Schmitt

magic moving pixel s.a.
23, Avenue Grande-Duchesse Charlotte
L-3441 Dudelange
Luxembourg
Phone: +352 54 75 75
http://www.mmp.lu

[sqlite] Memory leak?

[Gregory Moore]

Hi, Matt

I also have an iPhone app which directly interacts with sqlite using its C 
language API.  So, a few questions for you, to get a better idea of what?s 
happening:

Are you interacting directly with sqlite?  Or are you using a 3rd party API 
such as FMDB?  I take it you are not using Core Data (which optionally can use 
a sqlite database as a datastore.)

Are you using ARC?  For the uninitiated, ARC stands for Automatic Reference 
Counting.  It is an option which can be enabled on a per project basis.  Files 
within the project can opt out of using it.  ARC, reduces the amount of memory 
management code the developer has to write, but it does not license the 
developer to design unwisely.  For instance, it is still the developer?s 
responsibility to design in such a way that such things as retain cycles are 
avoided.

Have you implemented the -(void)dealloc method for all your Objective-C (or 
Swift?) objects?  I have had some success getting memory released by 
implementing -(void)dealloc and within that implementation setting specific 
objects to nil.  Note that under ARC you are not allowed to call [super 
dealloc] directly but you still can implement the -(void)dealloc method.

Are you willing to share some of your application code where it interacts with 
the sqlite database?



> On Mar 13, 2015, at 5:13 AM, Matthias Schmitt  wrote:
> 
> Hello,
> 
> thank you for your fast response.
> 
>> On 12 Mar 2015, at 17:35, Richard Hipp  wrote:
>> 
>> Let's start with the basics:  How do you know that the memory was in
>> fact leaked and is not instead simply being held for reuse?
> 
> The Xcode development environment comes with a debugging tool named 
> ?Instruments? which is able to detect memory leaks quite reliable. It traces 
> all references to code segments. If a reference to a code segment is 
> overwritten with a new value, but the previously addressed memory segment was 
> not released, then the software detects this as a leak. The same thing is 
> true when a variable created inside a code block references allocated memory. 
> When the program leaves the code block without releasing the previously 
> allocated memory, the reference get inaccessible and the allocated memory 
> cannot be released any more.
> 
>> On 12 Mar 2015, at 17:31, Simon Slavin  wrote:
>> Is it, by any chance, always the first SQL command executed after doing 
>> sqlite3_open() ?
> 
> No, there are multiple memory leaks which add over time.
> 
>> Is there any chance you can look at this memory and see what appears there ?
> 
> Here is an example. But different leaked memory segments look different.
> 
> <0x7fc2938b5600>
> 
> 5600: 0072 2b9d271d
> 5608: c9b753a9 bfbc7557
> 5610: dadb57e5 cffoeefb
> 5618: db68dd48 09aa68da
> 5620: aad489da b44f4329
> 5628: dd3ba2ff 00f0ff00
> 5630: 3a653236 a9b72d6a
> 5638: 9d8ceaae 7571942b
> 5640: 51f251fe ed6bcc73
> 5648: 3a2d224a 72d43bbd
> 5650: e9fb96a8 c496a4a8
> 5658: 57a53a82 96aac489
> 5660: da9ebd2a 1fbb5657
> 5668: 6ff1536e e6713d4f
> 5670: e0ff00c2 ed73e2ff
> 5678: 008f74df 046883ca
> 
>> Do the extra 4344 bytes get included in the number shown by 
>> "sqlite3_memory_used()" ?  You could try using that function after 
>> sqlite_close() and see what it shows.
> 
> I am closing and reopening the database now after every transaction. 
> sqlite3_memory_used() shows always 0. I guess that sqlite is telling me by it 
> that it is not aware of any lost memory, correct?
> 
>> Is the memory released at any later time ?  When you use _close() on the 
>> connection, for example ?  Or, if you can track it, when SQLite calls 
>> "sqlite3_db_release_memory()" internally ?
> 
> 
> The closing and re-opening the database does not change anything in the 
> display of ?Instruments?. The program still reports memory leaks. In another 
> attempt I tried to compile the code with SQLITE_ENABLE_MEMORY_MANAGEMENT 
> compile-time option. Then I tried to call sqlite3_release_memory() after 
> every database command. Same result.
> 
> Best regards
> 
> Matthias Schmitt
> 
> magic moving pixel s.a.
> 23, Avenue Grande-Duchesse Charlotte
> L-3441 Dudelange
> Luxembourg
> Phone: +352 54 75 75
> http://www.mmp.lu
> 
> 
> 
> 
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Gregory Moore
thewatchfulone at gmail.com

[sqlite] Memory leak?

[Simon Slavin]

Okay.  The hex you showed us doesn't represent ASCII characters, so the bug is 
not obviously leaking memory which contains the sort of data you'd be storing.  
Apart from a few sequences of 'ff00' I see no obvious patterns.

How are you getting your SQLite library ?  Are you calling a library built into 
the development environment or are you including your own copies of sqlite.c 
and sqlite.h ?

> I am closing and reopening the database now after every transaction. 
> sqlite3_memory_used() shows always 0. I guess that sqlite is telling me by it 
> that it is not aware of any lost memory, correct?

Yes, I believe so.

> Then I tried to call sqlite3_release_memory() after every database command. 
> Same result.

This indicates the same thing: the memory leak is not part of SQLite's own 
memory management strategy, but something which is happening outside of it.  
I'm out of ideas but I hope someone else can help.

Simon.

[sqlite] Memory leak?

[Matthias Schmitt]

Hello,

thank you for your fast response.

> On 12 Mar 2015, at 17:35, Richard Hipp  wrote:
> 
> Let's start with the basics:  How do you know that the memory was in
> fact leaked and is not instead simply being held for reuse?

The Xcode development environment comes with a debugging tool named 
?Instruments? which is able to detect memory leaks quite reliable. It traces 
all references to code segments. If a reference to a code segment is 
overwritten with a new value, but the previously addressed memory segment was 
not released, then the software detects this as a leak. The same thing is true 
when a variable created inside a code block references allocated memory. When 
the program leaves the code block without releasing the previously allocated 
memory, the reference get inaccessible and the allocated memory cannot be 
released any more.

> On 12 Mar 2015, at 17:31, Simon Slavin  wrote:
> Is it, by any chance, always the first SQL command executed after doing 
> sqlite3_open() ?

No, there are multiple memory leaks which add over time.

> Is there any chance you can look at this memory and see what appears there ?

Here is an example. But different leaked memory segments look different.

<0x7fc2938b5600>

5600: 0072 2b9d271d
5608: c9b753a9 bfbc7557
5610: dadb57e5 cffoeefb
5618: db68dd48 09aa68da
5620: aad489da b44f4329
5628: dd3ba2ff 00f0ff00
5630: 3a653236 a9b72d6a
5638: 9d8ceaae 7571942b
5640: 51f251fe ed6bcc73
5648: 3a2d224a 72d43bbd
5650: e9fb96a8 c496a4a8
5658: 57a53a82 96aac489
5660: da9ebd2a 1fbb5657
5668: 6ff1536e e6713d4f
5670: e0ff00c2 ed73e2ff
5678: 008f74df 046883ca

> Do the extra 4344 bytes get included in the number shown by 
> "sqlite3_memory_used()" ?  You could try using that function after 
> sqlite_close() and see what it shows.

I am closing and reopening the database now after every transaction. 
sqlite3_memory_used() shows always 0. I guess that sqlite is telling me by it 
that it is not aware of any lost memory, correct?

> Is the memory released at any later time ?  When you use _close() on the 
> connection, for example ?  Or, if you can track it, when SQLite calls 
> "sqlite3_db_release_memory()" internally ?


The closing and re-opening the database does not change anything in the display 
of ?Instruments?. The program still reports memory leaks. In another attempt I 
tried to compile the code with SQLITE_ENABLE_MEMORY_MANAGEMENT compile-time 
option. Then I tried to call sqlite3_release_memory() after every database 
command. Same result.

Best regards

Matthias Schmitt

magic moving pixel s.a.
23, Avenue Grande-Duchesse Charlotte
L-3441 Dudelange
Luxembourg
Phone: +352 54 75 75
http://www.mmp.lu

[sqlite] Memory leak?

[Scott Hess]

On Fri, Mar 13, 2015 at 3:13 AM, Matthias Schmitt  wrote:
>> On 12 Mar 2015, at 17:35, Richard Hipp  wrote:
>>
>> Let's start with the basics:  How do you know that the memory was in
>> fact leaked and is not instead simply being held for reuse?
>
> The Xcode development environment comes with a debugging tool named 
> ?Instruments? which is able to detect memory leaks quite reliable. It traces 
> all references to code segments. If a reference to a code segment is 
> overwritten with a new value, but the previously addressed memory segment was 
> not released, then the software detects this as a leak. The same thing is 
> true when a variable created inside a code block references allocated memory. 
> When the program leaves the code block without releasing the previously 
> allocated memory, the reference get inaccessible and the allocated memory 
> cannot be released any more.

By default SQLite stores database pages by allocating space for the
page plus space for metadata after the page, and then passes around
references to that metadata.  This can mislead tools of this sort,
because the reference is to a point well within the allocation rather
than to the actual allocation.  You might experiment with
SQLITE_PCACHE_SEPARATE_HEADER to see if that changes the results.

-scott

[sqlite] Memory leak?

[R.Smith]

On 2015-03-12 06:21 PM, Matthias Schmitt wrote:
> Hello,
>
> I know, I am touching a hot iron, when claiming that sqlite might have a 
> memory leak. But I am trying to debug memory leaks related to sqlite now 
> since 4 days and I am running against walls.

Hi Matthias,

Firstly, there are no hot irons here, it either works as designed, or it 
doesn't. If it does work correctly, we will gladly show you the way. if 
it doesn't, then we will all be glad it came to light and await the fix 
eagerly - you are very welcome to post the concern.
It is a little suspect that millions of applications around the world 
gets compiled and memory-leak-checked daily with sqlite and your's is 
the only one finding a leak - BUT, it is not impossible at all.

> Here is my problem:
>
> - I am writing an iPhone app using iOS 8.2, Xcode 4.2, sqlite 3.8.8.3.
>
> - The memory leak appears in the iPhone simulator more often, than on an 
> original iPhone device, but it appears always.
>
> - The leak appears with totally different type of queries. It might be, that 
> a leak appears in a specific query in one debug session and in another debug 
> session, it doesn?t (which makes it difficult to debug and drives me totally 
> nuts). For me it seems to be related to a timing/threading issue. This is why 
> it makes not much sense to offer some code here.
>
> - When debugging inside sqlite the leak is always related to the same code 
> line, independent from the type of database command (select, insert, delete).
>
> - It appears always with the memory allocation in sqlite.c line 16872 in 
> function sqlite3MemMalloc
>   Here is the line: void *p = SQLITE_MALLOC( nByte );
>
> - The leaked memory has always a size of nByte = 4344 bytes.
>
> - Here is a typical call stack up to the leaked code:

This is a bit weird, how do you know the memory is leaked when you are 
still in the functions dealing with it?  The definition of a memory leak 
(simplified for our purposes) is where some memory is acquired into 
processes repeatedly with the needed memory already existing and not 
released so that perpetual memory-adding will eventually eat up all the 
available resources. To be more succinct: It's an accumulation of 
allocations over time.

Lots of times memory is allocated once and kept, or cached, or used by 
shared resources or many other things that  might speed up processes. If 
the memory is only ever released when the object or only program 
structure that can use it is released/freed/disposed, then it is still 
ok and not a leak.  If memory is retained even after the destruction of 
it's keeper, then it may also be regarded as a "leak", even without 
accumulation.

I cannot see from anything you said that the memory survives the death 
of the object, nor that it accumulates in any way. It may well still be 
an unnecessary allocation, but it can't possibly be a "Leak" (unless I 
am misreading your information).

All that said, you may well ask "What is this allocation, and should it 
happen?"  (Which a dev might answer) - but is it hurting your program or 
the use thereof? (That would be a real cause for concern).

[sqlite] Memory leak?

[Matthias Schmitt]

Hello,

I know, I am touching a hot iron, when claiming that sqlite might have a memory 
leak. But I am trying to debug memory leaks related to sqlite now since 4 days 
and I am running against walls.
Here is my problem:

- I am writing an iPhone app using iOS 8.2, Xcode 4.2, sqlite 3.8.8.3.

- The memory leak appears in the iPhone simulator more often, than on an 
original iPhone device, but it appears always.

- The leak appears with totally different type of queries. It might be, that a 
leak appears in a specific query in one debug session and in another debug 
session, it doesn?t (which makes it difficult to debug and drives me totally 
nuts). For me it seems to be related to a timing/threading issue. This is why 
it makes not much sense to offer some code here.

- When debugging inside sqlite the leak is always related to the same code 
line, independent from the type of database command (select, insert, delete).

- It appears always with the memory allocation in sqlite.c line 16872 in 
function sqlite3MemMalloc
 Here is the line: void *p = SQLITE_MALLOC( nByte );

- The leaked memory has always a size of nByte = 4344 bytes.

- Here is a typical call stack up to the leaked code:

  0 libsystem_malloc.dylib malloc_zone_malloc
  1 MYAPP sqlite3MemMalloc sqlite3.c:16872
  2 MYAPP mallocWithAlarm sqlite3.c:20510
  3 MYAPP sqlite3Malloc sqlite3.c:20541
  4 MYAPP pcache1Alloc sqlite3.c:39749
  5 MYAPP pcache1AllocPage sqlite3.c:39833
  6 MYAPP pcache1FetchStage2 sqlite3.c:40266
  7 MYAPP pcache1Fetch sqlite3.c:40367
  8 MYAPP sqlite3PcacheFetch sqlite3.c:39119
  9 MYAPP sqlite3PagerAcquire sqlite3.c:46573
 10 MYAPP btreeGetPage sqlite3.c:54380
 11 MYAPP getAndInitPage sqlite3.c:54435
 12 MYAPP moveToRoot sqlite3.c:57233
 13 MYAPP sqlite3BtreeMovetoUnpacked sqlite3.c:57465
 14 MYAPP sqlite3VdbeExec sqlite3.c:74055
 15 MYAPP sqlite3Step sqlite3.c:68839
 16 MYAPP sqlite3_step sqlite3.c:68905
 17 MYAPP -[FMResultSet nextWithError:] FMResultSet.m:155
 18 MYAPP -[FMResultSet next] FMResultSet.m:150
 19 MYAPP __40-[MYAPPAppDelegate loadPreferences]_block_invoke 
MYAPPAppDelegate.m:323
 20 MYAPP __30-[FMDatabaseQueue inDatabase:]_block_invoke FMDatabaseQueue.m:150
 21 libdispatch.dylib _dispatch_client_callout
 22 libdispatch.dylib _dispatch_barrier_sync_f_invoke
 23 MYAPP -[FMDatabaseQueue inDatabase:] FMDatabaseQueue.m:147
 24 MYAPP -[MYAPPAppDelegate loadPreferences] MYAPPAppDelegate.m:321
 25 MYAPP -[MYAPPAppDelegate application:didFinishLaunchingWithOptions:] 
MYAPPAppDelegate.m:157
 26 UIKit -[UIApplication 
_handleDelegateCallbacksWithOptions:isSuspended:restoreState:]
 27 UIKit -[UIApplication 
_callInitializationDelegatesForMainScene:transitionContext:]
 28 UIKit -[UIApplication _runWithMainScene:transitionContext:completion:]
 29 UIKit -[UIApplication workspaceDidEndTransaction:]
 30 FrontBoardServices __31-[FBSSerialQueue performAsync:]_block_invoke
 31 CoreFoundation __CFRUNLOOP_IS_CALLING_OUT_TO_A_BLOCK__
 32 CoreFoundation __CFRunLoopDoBlocks
 33 CoreFoundation __CFRunLoopRun
 34 CoreFoundation CFRunLoopRunSpecific
 35 UIKit -[UIApplication _run]
 36 UIKit UIApplicationMain
 37 MYAPP main main.m:16
 38 libdyld.dylib start

Does anybody have an idea how to debug something like this? Any help would be 
highly appreciated.

Best regards

Matthias Schmitt

magic moving pixel s.a.
23, Avenue Grande-Duchesse Charlotte
L-3441 Dudelange
Luxembourg
Phone: +352 54 75 75
http://www.mmp.lu

[sqlite] Memory leak?

[Simon Slavin]

On 12 Mar 2015, at 4:21pm, Matthias Schmitt  wrote:

> - The memory leak appears in the iPhone simulator more often, than on an 
> original iPhone device, but it appears always.
> 
> - The leak appears with totally different type of queries. It might be, that 
> a leak appears in a specific query in one debug session and in another debug 
> session, it doesn?t (which makes it difficult to debug and drives me totally 
> nuts). For me it seems to be related to a timing/threading issue. This is why 
> it makes not much sense to offer some code here.
> 
> - When debugging inside sqlite the leak is always related to the same code 
> line, independent from the type of database command (select, insert, delete).

Is it, by any chance, always the first SQL command executed after doing 
sqlite3_open() ?

> - It appears always with the memory allocation in sqlite.c line 16872 in 
> function sqlite3MemMalloc
> Here is the line: void *p = SQLITE_MALLOC( nByte );
> 
> - The leaked memory has always a size of nByte = 4344 bytes.

Is there any chance you can look at this memory and see what appears there ?

Do the extra 4344 bytes get included in the number shown by 
"sqlite3_memory_used()" ?  You could try using that function after 
sqlite_close() and see what it shows.

Is the memory released at any later time ?  When you use _close() on the 
connection, for example ?  Or, if you can track it, when SQLite calls 
"sqlite3_db_release_memory()" internally ?

Simon.

[sqlite] Memory leak?

[Richard Hipp]

On 3/12/15, Matthias Schmitt  wrote:
> Hello,
>
> I know, I am touching a hot iron, when claiming that sqlite might have a
> memory leak. But I am trying to debug memory leaks related to sqlite now
> since 4 days and I am running against walls.
> Here is my problem:
>
> - I am writing an iPhone app using iOS 8.2, Xcode 4.2, sqlite 3.8.8.3.
>
> - The memory leak appears in the iPhone simulator more often, than on an
> original iPhone device, but it appears always.

Let's start with the basics:  How do you know that the memory was in
fact leaked and is not instead simply being held for reuse?

>
> - The leak appears with totally different type of queries. It might be, that
> a leak appears in a specific query in one debug session and in another debug
> session, it doesn?t (which makes it difficult to debug and drives me totally
> nuts). For me it seems to be related to a timing/threading issue. This is
> why it makes not much sense to offer some code here.
>
> - When debugging inside sqlite the leak is always related to the same code
> line, independent from the type of database command (select, insert,
> delete).
>
> - It appears always with the memory allocation in sqlite.c line 16872 in
> function sqlite3MemMalloc
>  Here is the line: void *p = SQLITE_MALLOC( nByte );
>
> - The leaked memory has always a size of nByte = 4344 bytes.
>
> - Here is a typical call stack up to the leaked code:
>
>   0 libsystem_malloc.dylib malloc_zone_malloc
>   1 MYAPP sqlite3MemMalloc sqlite3.c:16872
>   2 MYAPP mallocWithAlarm sqlite3.c:20510
>   3 MYAPP sqlite3Malloc sqlite3.c:20541
>   4 MYAPP pcache1Alloc sqlite3.c:39749
>   5 MYAPP pcache1AllocPage sqlite3.c:39833
>   6 MYAPP pcache1FetchStage2 sqlite3.c:40266
>   7 MYAPP pcache1Fetch sqlite3.c:40367
>   8 MYAPP sqlite3PcacheFetch sqlite3.c:39119
>   9 MYAPP sqlite3PagerAcquire sqlite3.c:46573
>  10 MYAPP btreeGetPage sqlite3.c:54380
>  11 MYAPP getAndInitPage sqlite3.c:54435
>  12 MYAPP moveToRoot sqlite3.c:57233
>  13 MYAPP sqlite3BtreeMovetoUnpacked sqlite3.c:57465
>  14 MYAPP sqlite3VdbeExec sqlite3.c:74055
>  15 MYAPP sqlite3Step sqlite3.c:68839
>  16 MYAPP sqlite3_step sqlite3.c:68905
>  17 MYAPP -[FMResultSet nextWithError:] FMResultSet.m:155
>  18 MYAPP -[FMResultSet next] FMResultSet.m:150
>  19 MYAPP __40-[MYAPPAppDelegate loadPreferences]_block_invoke
> MYAPPAppDelegate.m:323
>  20 MYAPP __30-[FMDatabaseQueue inDatabase:]_block_invoke
> FMDatabaseQueue.m:150
>  21 libdispatch.dylib _dispatch_client_callout
>  22 libdispatch.dylib _dispatch_barrier_sync_f_invoke
>  23 MYAPP -[FMDatabaseQueue inDatabase:] FMDatabaseQueue.m:147
>  24 MYAPP -[MYAPPAppDelegate loadPreferences] MYAPPAppDelegate.m:321
>  25 MYAPP -[MYAPPAppDelegate application:didFinishLaunchingWithOptions:]
> MYAPPAppDelegate.m:157
>  26 UIKit -[UIApplication
> _handleDelegateCallbacksWithOptions:isSuspended:restoreState:]
>  27 UIKit -[UIApplication
> _callInitializationDelegatesForMainScene:transitionContext:]
>  28 UIKit -[UIApplication _runWithMainScene:transitionContext:completion:]
>  29 UIKit -[UIApplication workspaceDidEndTransaction:]
>  30 FrontBoardServices __31-[FBSSerialQueue performAsync:]_block_invoke
>  31 CoreFoundation __CFRUNLOOP_IS_CALLING_OUT_TO_A_BLOCK__
>  32 CoreFoundation __CFRunLoopDoBlocks
>  33 CoreFoundation __CFRunLoopRun
>  34 CoreFoundation CFRunLoopRunSpecific
>  35 UIKit -[UIApplication _run]
>  36 UIKit UIApplicationMain
>  37 MYAPP main main.m:16
>  38 libdyld.dylib start
>
> Does anybody have an idea how to debug something like this? Any help would
> be highly appreciated.
>
> Best regards
>
> Matthias Schmitt
>
> magic moving pixel s.a.
> 23, Avenue Grande-Duchesse Charlotte
> L-3441 Dudelange
> Luxembourg
> Phone: +352 54 75 75
> http://www.mmp.lu
>
>
>
>
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>


-- 
D. Richard Hipp
drh at sqlite.org

[sqlite] Memory Leak in ext/misc/compress.c

[Keith Medcalf]

static void uncompressFunc(
  sqlite3_context *context,
  int argc,
  sqlite3_value **argv
){
  const unsigned char *pIn;
  unsigned char *pOut;
  unsigned int nIn;
  unsigned long int nOut;
  int rc;
  int i;

  pIn = sqlite3_value_blob(argv[0]);
  nIn = sqlite3_value_bytes(argv[0]);
  nOut = 0;
  for(i=0; i

Re: [sqlite] Memory leak in online backup api

[Richard Hipp]

On Mon, Mar 10, 2014 at 10:06 AM, techi eth  wrote:

> Any hint will help me to identify root cause of leak.
>

Statically link against SQLite.  Compile with -g.  Use the
--leak-check=full option to valgrind.
-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak in online backup api

[techi eth]

Any hint will help me to identify root cause of leak.

Thanks


On Mon, Mar 10, 2014 at 2:58 PM, techi eth  wrote:

> Hi,
>
> I come across memory leak with sqlite3 online backup api (Leak detected by
> valgrind tool). I have used reference code from below link for test.
> (Online backup for running database)
>
> http://www.sqlite.org/backup.html
>
> Please find below memory leak snapshot for reference.
>
> ==6084== 78,688 (880 direct, 77,808 indirect) bytes in 1 blocks are
> definitely lost in loss record 52 of 52
> ==6084==at 0x4C2B6CD: malloc (in
> /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==6084==by 0x571C73E: ??? (in
> /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
> ==6084==by 0x56FA238: ??? (in
> /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
> ==6084==by 0x5701B17: ??? (in
> /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
> ==6084==by 0x5701B44: ??? (in
> /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
> ==6084==by 0x576FB91: ??? (in
> /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
> ==6084==by 0x4E36001: db_sqlite3_backup (db_sqlite3.c:2291)
> ==6084==by 0x4026A7:  Test_backup (db_sqlite3_test.c:994)
> ==6084==by 0x40283D: main (db_sqlite3_test.c:1170)
> ==6084==
> ==6084== LEAK SUMMARY:
> ==6084==definitely lost: 880 bytes in 1 blocks
>
> Thanks
>
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Memory leak in online backup api

[techi eth]

Hi,

I come across memory leak with sqlite3 online backup api (Leak detected by
valgrind tool). I have used reference code from below link for test.
(Online backup for running database)

http://www.sqlite.org/backup.html

Please find below memory leak snapshot for reference.

==6084== 78,688 (880 direct, 77,808 indirect) bytes in 1 blocks are
definitely lost in loss record 52 of 52
==6084==at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==6084==by 0x571C73E: ??? (in
/usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
==6084==by 0x56FA238: ??? (in
/usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
==6084==by 0x5701B17: ??? (in
/usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
==6084==by 0x5701B44: ??? (in
/usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
==6084==by 0x576FB91: ??? (in
/usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
==6084==by 0x4E36001: db_sqlite3_backup (db_sqlite3.c:2291)
==6084==by 0x4026A7:  Test_backup (db_sqlite3_test.c:994)
==6084==by 0x40283D: main (db_sqlite3_test.c:1170)
==6084==
==6084== LEAK SUMMARY:
==6084==definitely lost: 880 bytes in 1 blocks

Thanks
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] memory leak in sqlite3_load_extension

[a . furieri]

Hi,

I'm Sandro Furieri, developer and maintainer of libspatialite:
https://www.gaia-gis.it/fossil/libspatialite/index

While testing my own library under Valgrind I've casually noticed
a very small memory leak caused by the sqlite3_load_extension and
sqlite3_reset_auto_extension logic:

32 bytes in 1 blocks are still reachable in loss record 1 of 1
   at 0x4C29DB4: calloc (in 
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)

   by 0x696C59F: _dlerror_run (dlerror.c:142)
   by 0x696BFC0: dlopen@@GLIBC_2.2.5 (dlopen.c:88)
   by 0x57C5B26: sqlite3_load_extension (in 
/usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)

   by 0x57C5D81: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
   by 0x57E58F5: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
   by 0x57EB450: sqlite3_step (in 
/usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
   by 0x57EEE4B: sqlite3_exec (in 
/usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)

   by 0x4012DE: main (check_extension.c:87)

the check_extension.c basic skeleton is:

char *sql = "SELECT load_extension('libspatialite.so')");
sqlite3_exec (handle, sql, NULL, NULL, &err_msg);
...
sqlite3_reset_auto_extension ();

I've got a quick glance to the sqlite3.c code, and I've noticed that
sqlite3_reset_auto_extension omits at all calling dlclose

once the extension library has been successfully loaded there is no
apparent way allowing to effectively finalize the library handle
before exiting.
absolutely not a big issue; anyway a little bit annoying, because it
tends to pollute Valgrind reports with many "false positive" issues.

best regards,
Sandro

--
Il messaggio e' stato analizzato alla ricerca di virus o
contenuti pericolosi da MailScanner, ed e'
risultato non infetto.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] memory leak in transactions

[rui]

Thanks for your reply.
I am using version 1.0.79.0 of System.Data.sqlite
Here is how i am using it, i only use executescalar and
executenonquery, let me know if you want to see how i pass the
parameters etc.


RunBigtransaction()
{

 using (IDbTransaction tran = Connection.BeginTransaction())
 {
foreach (Message msg in messages)
{
   CreateMessage(msg);
}
tran.Commit();
 }
}


 void CreateMessage()
 {
  string tableName = gettablename();

  // here i check if a table exists using a select query
  string query = select "name" from "sqlite_master" where
"type=table", name=tablename";
  string tablename = ExecuteQuery(query)

  if (tablename == null)

  {
// here i do a query on another table in schema to read 
a script to
create a new table, which uses the ExecuteGetTableQuery.
string script = getscript();
Exceute(script);
  }

  // here i execute an insert and update

  // insert uses the same pattern like ExecuteGetTableQuery but 
with
query changed to insert into with 15 parameters.
RawInsertMessage(tableName, msg);

// insert or update where update does just Execute 
function.
SetMessageFolderId(msg.StoreId, msg.Id, msg.ParentId);  
 }

 public void  Execute(string script)
 {
using (IDbCommand cmd = Connection.CreateCommand())
{
cmd.CommandText = script;
   cmd.ExecuteNonQuery();
}
 }  

 public string ExecuteGetTableQuery(string b)
 {
 using (IDbCommand cmd = Connection.CreateCommand())
 {
cmd.CommandText = b.ToString();

object r = cmd.ExecuteScalar();

if (r == null || r is DBNull) return default(T);
if (!(r is T)) throw new 
InvalidCastException("cannot cast " +
r.GetType() + " to " + typeof (T));
return (T) r;
 }
 }




Regards,
Raj


On Thu, Nov 15, 2012 at 8:52 PM, Joe Mistachkin  wrote:
>
> rui wrote:
>>
>> I am seeing explosive memory growth when i am using transactions using
>> System.Data.SQLite.
>>
>
> The best way to address this issue is to utilize "using" blocks for any 
> SQLiteCommand, SQLiteDataReader, and SQLiteTransaction objects used.  That 
> way, you won't have to wait until they are garbage collected later on by the 
> CLR.
>
>>
>> All the object are properly disposed from trasnsaction to command etc.
>>
>
> Could you provide some example C# code that demonstrates the behavior you are 
> seeing?
>
>>
>> The SQLiteConnection is kept open for the life time of the session,
>> which could span hours.
>>
>
> This means the memory associated with the connection will be kept around; 
> however, this should be OK and should not result in the memory usage numbers 
> you are seeing.
>
> --
> Joe Mistachkin 
>
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] memory leak in transactions

[Joe Mistachkin]

rui wrote:
> 
> I am seeing explosive memory growth when i am using transactions using
> System.Data.SQLite.
> 

The best way to address this issue is to utilize "using" blocks for any 
SQLiteCommand, SQLiteDataReader, and SQLiteTransaction objects used.  That way, 
you won't have to wait until they are garbage collected later on by the CLR.

>
> All the object are properly disposed from trasnsaction to command etc.
>

Could you provide some example C# code that demonstrates the behavior you are 
seeing?

> 
> The SQLiteConnection is kept open for the life time of the session,
> which could span hours.
> 

This means the memory associated with the connection will be kept around; 
however, this should be OK and should not result in the memory usage numbers 
you are seeing.

-- 
Joe Mistachkin 


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] memory leak in transactions

[Rob Richardson]

Do you have your inserts wrapped in a single transaction?  It used to be that I 
wasn't worrying about transactions in my projects, but I noticed things were 
very slow.  I realized that it was creating and committing one transaction for 
each insert I was doing.  When I wrapped all inserts into a single transaction, 
speed jumped dramatically.

But, of course, this doesn't address your memory issue.  Unless the increased 
memory is somewhere in the transaction management logic.

RobR

-Original Message-
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] 
On Behalf Of rui
Sent: Thursday, November 15, 2012 12:21 PM
To: sqlite-users@sqlite.org
Subject: [sqlite] memory leak in transactions

Hi,

I am seeing explosive memory growth when i am using transactions using 
System.Data.SQLite.

I see this when the transactions which were before taking fraction of second, 
taking 15 sec for 50 row insert into a table.

I have tried to use sqlite3_db_release_memory but that only frees upto 2mb 
after every transaction.

After some time say half an hour - i do see memory usage drop but that's not 
acceptable.

All the object are properly disposed from trasnsaction to command etc.

The SQLiteConnection is kept open for the life time of the session, which could 
span hours.

I would really appreciate if somebody can help me in getting the reason for 
such excessive memory usage, i have seen working set(memory)go up from 70 mb to 
400 mb in 1 minute where three transactions only doing 50 insert in a table.

Regards,
Raj
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] memory leak in transactions

[rui]

Hi,

I am seeing explosive memory growth when i am using transactions using
System.Data.SQLite.

I see this when the transactions which were before taking fraction of
second, taking 15 sec for 50 row insert into a table.

I have tried to use sqlite3_db_release_memory but that only frees upto
2mb after every transaction.

After some time say half an hour - i do see memory usage drop but
that's not acceptable.

All the object are properly disposed from trasnsaction to command etc.

The SQLiteConnection is kept open for the life time of the session,
which could span hours.

I would really appreciate if somebody can help me in getting the
reason for such excessive memory usage, i have seen working
set(memory)go up from 70 mb to 400 mb in 1 minute where three
transactions only doing 50 insert in a table.

Regards,
Raj
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak?

[Anze Rozman]

Some mistakes...
* I use also PRAGMA cache_size=100 but the some problem

On Wed, Jul 13, 2011 at 3:12 PM, Anze Rozman  wrote:

> Hi!
>
> I have an application which inserts into database most of the time. Problem
> is that memory is decreasing. Aplication is in java and I use Werner's
> sqlite wrapper - http://www.ch-werner.de/javasqlite/ (because of J2ME). I
> use method 
> (*exec
> *(String  sql,
> Callback  cb) )
> from werner's wrapper. I use also page_cache=100, and I use transactions
> with BEGIN and COMMIT, but the problem remains. After few hours application
> eat almost all free memory. I find out that problem is because of INSERT
> statements. If I run application without writing into database memory is not
> increasing. OS is linux and after 2 days system reboots because there is not
> any more free memory for other application. I have read many forums and
> websites but I havent found any solution to this. I also close and open
> database after 1000 tranactions, but there is no improvance. Can you help
> me?
>
> BR, Rose.
>



-- 
lp,
Anže
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Memory leak?

[Anze Rozman]

Hi!

I have an application which inserts into database most of the time. Problem
is that memory is decreasing. Aplication is in java and I use Werner's
sqlite wrapper - http://www.ch-werner.de/javasqlite/ (because of J2ME). I
use method 
(*exec
*(String  sql,
Callback  cb) )
from werner's wrapper. I use also page_cache=100, and I use transactions
with BEGIN and COMMIT, but the problem remains. After few hours application
eat almost all free memory. I find out that problem is because of INSERT
statements. If I run application without writing into database memory is not
increasing. OS is linux and after 2 days system reboots because there is not
any more free memory for other application. I have read many forums and
websites but I havent found any solution to this. I also close and open
database after 1000 tranactions, but there is no improvance. Can you help
me?

BR, Rose.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak in SQlite

[Black, Michael (IS)]

#1 I don't see where you're freeing m_szErrorString (not real sure if it gets 
malloc'd on success) -- but you do need to free it on errors for sure.



And where's your Callback function?  Why are you calling SaveResultSet (which 
you also don't show)?  That should probably be done inside the Callback because 
I believe the data is destroyed coming out of the Callback.





Michael D. Black

Senior Scientist

NG Information Systems

Advanced Analytics Directorate




From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] on 
behalf of Ian Hardingham [i...@omroth.com]
Sent: Tuesday, May 10, 2011 7:17 AM
To: General Discussion of SQLite Database
Subject: EXT :[sqlite] Memory leak in SQlite

Hey guys.

I'm sure that this is to do with the way I am using SQLite. I do not
have time to radically change my methodology at this point, but I do
need to fix a rather severe memory leak I'm having.

(My apologies for the code)

Any help is much appreciated.

I query like this:


int SQLiteObject::ExecuteSQL(const char* sql, int hack)
{
int iResult;
sqlite_resultset* pResultSet;

// create a new resultset
pResultSet = new sqlite_resultset;

if (pResultSet)
{
   pResultSet->bValid = false;
   pResultSet->iCurrentColumn = 0;
   pResultSet->iCurrentRow = 0;
   pResultSet->iNumCols = 0;
   pResultSet->iNumRows = 0;
   pResultSet->iResultSet = m_iNextResultSet;
   pResultSet->vRows.clear();
   m_iLastResultSet = m_iNextResultSet;
   m_iNextResultSet++;
}
else
   return 0;


 iResult = sqlite3_exec(m_pDatabase, sql, Callback,
(void*)pResultSet, &m_szErrorString);

 if (iResult == 0)
 {
 //SQLITE_OK


 SaveResultSet(pResultSet);
 //Con::executef(this, 1, "onQueryFinished()");
 return pResultSet->iResultSet;

 }
 else
 {
 // error occured
 Con::executef(this, 2, "onQueryFailed", m_szErrorString);
 delete pResultSet;
 return 0;
 }

return 0;
}


And I "clear a result" like this:


void SQLiteObject::ClearResultSet(int index)
{



sqlite_resultset* resultSet;
sqlite_resultrow* resultRow;
S32 rows, cols, iResultSet;

// Get the result set specified by index
resultSet = GetResultSet(index);
iResultSet = GetResultSetIndex(index);
if ((!resultSet) || (!resultSet->bValid))
{
   Con::warnf("Warning SQLiteObject::ClearResultSet(%i) failed to
retrieve specified result set.  Result set was NOT cleared.", index);
   return;
}
// Now we have the specific result set to be cleared.
// What we need to do now is iterate through each "Column" in each "Row"
// and free the strings, then delete the entries.
VectorPtr::iterator iRow;
VectorPtr::iterator iColumnName;
VectorPtr::iterator iColumnValue;

for (iRow = resultSet->vRows.begin(); iRow !=
resultSet->vRows.end(); iRow++)
{
   // Iterate through rows
   // for each row iterate through all the column values and names
   for (iColumnName = (*iRow)->vColumnNames.begin(); iColumnName !=
(*iRow)->vColumnNames.end(); iColumnName++)
   {
  // Iterate through column names.  Free the memory.
  delete[] (*iColumnName);
   }
   for (iColumnValue = (*iRow)->vColumnValues.begin(); iColumnValue
!= (*iRow)->vColumnValues.end(); iColumnValue++)
   {
  // Iterate through column values.  Free the memory.
  delete[] (*iColumnValue);
   }
   // free memory used by the row
   delete (*iRow);
}
// empty the resultset
resultSet->vRows.clear();
resultSet->bValid = false;
delete resultSet;
m_vResultSets.erase_fast(iResultSet);
}

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak in SQlite

[Stephan Beal]

On Tue, May 10, 2011 at 2:17 PM, Ian Hardingham  wrote:

> I'm sure that this is to do with the way I am using SQLite. I do not
> have time to radically change my methodology at this point, but I do
> need to fix a rather severe memory leak I'm having.
>
>
i don't see any immediate leaks in what you posted, but what you posted
relies on tons of code we can't see. Pointer-managing classes like
VectorPtr can certainly be a source of leaks.

i highly recommend running your program through valgrind and see what it
says - it will point you directly to the source of the leak. Valgrind can be
used like this:

valgrind --leak-check=full -v --show-reachable=yes --track-origins=yes
./MyProgram


-- 
- stephan beal
http://wanderinghorse.net/home/stephan/
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Memory leak in SQlite

[Ian Hardingham]

Hey guys.

I'm sure that this is to do with the way I am using SQLite. I do not 
have time to radically change my methodology at this point, but I do 
need to fix a rather severe memory leak I'm having.

(My apologies for the code)

Any help is much appreciated.

I query like this:


int SQLiteObject::ExecuteSQL(const char* sql, int hack)
{
int iResult;
sqlite_resultset* pResultSet;

// create a new resultset
pResultSet = new sqlite_resultset;

if (pResultSet)
{
   pResultSet->bValid = false;
   pResultSet->iCurrentColumn = 0;
   pResultSet->iCurrentRow = 0;
   pResultSet->iNumCols = 0;
   pResultSet->iNumRows = 0;
   pResultSet->iResultSet = m_iNextResultSet;
   pResultSet->vRows.clear();
   m_iLastResultSet = m_iNextResultSet;
   m_iNextResultSet++;
}
else
   return 0;


 iResult = sqlite3_exec(m_pDatabase, sql, Callback, 
(void*)pResultSet, &m_szErrorString);

 if (iResult == 0)
 {
 //SQLITE_OK


 SaveResultSet(pResultSet);
 //Con::executef(this, 1, "onQueryFinished()");
 return pResultSet->iResultSet;

 }
 else
 {
 // error occured
 Con::executef(this, 2, "onQueryFailed", m_szErrorString);
 delete pResultSet;
 return 0;
 }

return 0;
}


And I "clear a result" like this:


void SQLiteObject::ClearResultSet(int index)
{



sqlite_resultset* resultSet;
sqlite_resultrow* resultRow;
S32 rows, cols, iResultSet;

// Get the result set specified by index
resultSet = GetResultSet(index);
iResultSet = GetResultSetIndex(index);
if ((!resultSet) || (!resultSet->bValid))
{
   Con::warnf("Warning SQLiteObject::ClearResultSet(%i) failed to 
retrieve specified result set.  Result set was NOT cleared.", index);
   return;
}
// Now we have the specific result set to be cleared.
// What we need to do now is iterate through each "Column" in each "Row"
// and free the strings, then delete the entries.
VectorPtr::iterator iRow;
VectorPtr::iterator iColumnName;
VectorPtr::iterator iColumnValue;

for (iRow = resultSet->vRows.begin(); iRow != 
resultSet->vRows.end(); iRow++)
{
   // Iterate through rows
   // for each row iterate through all the column values and names
   for (iColumnName = (*iRow)->vColumnNames.begin(); iColumnName != 
(*iRow)->vColumnNames.end(); iColumnName++)
   {
  // Iterate through column names.  Free the memory.
  delete[] (*iColumnName);
   }
   for (iColumnValue = (*iRow)->vColumnValues.begin(); iColumnValue 
!= (*iRow)->vColumnValues.end(); iColumnValue++)
   {
  // Iterate through column values.  Free the memory.
  delete[] (*iColumnValue);
   }
   // free memory used by the row
   delete (*iRow);
}
// empty the resultset
resultSet->vRows.clear();
resultSet->bValid = false;
delete resultSet;
m_vResultSets.erase_fast(iResultSet);
}

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak in sqlite 3.7.0

[Andy Gibbs]

On Tuesday, July 27, 2010 6:43 PM, Richard Hipp wrote:

>> [ ... ]
>>
>> However, I have also attached journaltest2.log which I think does
>> demonstrate the memory leak.  Having trawled through the full log file, 
>> it
>> seems that the memory leak is coming from the FTS3 tests, so the
>> journaltest2.log file is the output from:
>>
>> ./testfixture test/permutations.test journaltest fts3cov.test
>> fts3malloc.test memsubsys2.test
>>
>> I tried a similar run with inmemory_journal instead of journaltest, but
>> this gave me a "database disk image is malformed" error (see
>> inmemory_journal2.log), but this is not an error I received when running 
>> the
>> full test suite, so it is probably because I'm running a test sequence 
>> that
>> I shouldn't.
>
>
> Correct.  fts3cov.test is not compatible with inmemory_journal.
>

So does this mean that the memory leak that comes out when running "make 
fulltest" or through "./testfixture test/permutations.test journaltest 
fts3cov.test fts3malloc.test memsubsys2.test" is not really a memory leak, 
just an incompatible test that has made it into the test suite?

Oh, and thanks for merging my suggested change for "configure" into your 
repository - I spotted it on my daily browse through.

Regards
Andy

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak in sqlite 3.7.0

[Richard Hipp]

On Mon, Jul 26, 2010 at 11:16 AM, Andy Gibbs wrote:

> On Monday, July 26, 2010 4:44 PM, Richard Hipp wrote:
>
>
>  What do you get when you run:
>>
>>   ./testfixture test/permutations.test journaltest test/memsubsys2.test
>>   ./testfixture test/permutations.test inmemory_journal
>> test/memsubsys2.test
>>
>>
> Please find the log files attached as journaltest.log and
> inmemory_journal.log respectively.  Both passed fine with no errors.
>
> However, I have also attached journaltest2.log which I think does
> demonstrate the memory leak.  Having trawled through the full log file, it
> seems that the memory leak is coming from the FTS3 tests, so the
> journaltest2.log file is the output from:
>
> ./testfixture test/permutations.test journaltest fts3cov.test
> fts3malloc.test memsubsys2.test
>
> I tried a similar run with inmemory_journal instead of journaltest, but
> this gave me a "database disk image is malformed" error (see
> inmemory_journal2.log), but this is not an error I received when running the
> full test suite, so it is probably because I'm running a test sequence that
> I shouldn't.


Correct.  fts3cov.test is not compatible with inmemory_journal.


>
>
> Here also is the configuration used for building sqlite:
>
> configure CFLAGS="-DSQLITE_DEFAULT_FILE_FORMAT=4
> -DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS
> -DSQLITE_ENABLE_STAT2 -DSQLITE_ENABLE_UPDATE_DELETE_LIMIT"
> --enable-threadsafe --enable-debug --with-tcl=/usr/lib/tcl8.5
>
> (The only change between the source I have here and the "official" source
> is the fix mentioned in another of my emails today regarding 'configure'
> finding the wrong tclsh).
>
> I hope this helps!  Let me know if you'd like me to run further tests for
> you.
>
> Andy
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
>


-- 
-
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak in sqlite 3.7.0

[Andy Gibbs]

On Monday, July 26, 2010 4:44 PM, Richard Hipp wrote:



What do you get when you run:

   ./testfixture test/permutations.test journaltest test/memsubsys2.test
   ./testfixture test/permutations.test inmemory_journal
test/memsubsys2.test



Please find the log files attached as journaltest.log and 
inmemory_journal.log respectively.  Both passed fine with no errors.


However, I have also attached journaltest2.log which I think does 
demonstrate the memory leak.  Having trawled through the full log file, it 
seems that the memory leak is coming from the FTS3 tests, so the 
journaltest2.log file is the output from:


./testfixture test/permutations.test journaltest fts3cov.test 
fts3malloc.test memsubsys2.test


I tried a similar run with inmemory_journal instead of journaltest, but this 
gave me a "database disk image is malformed" error (see 
inmemory_journal2.log), but this is not an error I received when running the 
full test suite, so it is probably because I'm running a test sequence that 
I shouldn't.


Here also is the configuration used for building sqlite:

configure CFLAGS="-DSQLITE_DEFAULT_FILE_FORMAT=4
-DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS
-DSQLITE_ENABLE_STAT2 -DSQLITE_ENABLE_UPDATE_DELETE_LIMIT"
--enable-threadsafe --enable-debug --with-tcl=/usr/lib/tcl8.5

(The only change between the source I have here and the "official" source is 
the fix mentioned in another of my emails today regarding 'configure' 
finding the wrong tclsh).


I hope this helps!  Let me know if you'd like me to run further tests for 
you.


Andy
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak in sqlite 3.7.0

[Richard Hipp]

What do you get when you run:

./testfixture test/permutations.test journaltest test/memsubsys2.test
./testfixture test/permutations.test inmemory_journal
test/memsubsys2.test

On Mon, Jul 26, 2010 at 9:47 AM, Andy Gibbs  wrote:

> Hi,
>
> I'm afraid I believe I have observed a memory leak when running the full
> test suite that is part of sqlite 3.7.0.
>
> I have a log file generated from running the test suite which I can send if
> it is of interest to the developers - it is 5Mb compressed so I didn't
> think
> it fair to just post it here!
>
> The bottom of the log is as follows:
>
> 18 errors out of 1760926 tests
> Failures on these tests: journaltest.memsubsys2-3.1
> journaltest.memsubsys2-3.3 journaltest.memsubsys2-3.4
> journaltest.memsubsys2-3.6 journaltest.memsubsys2-3.7
> journaltest.memsubsys2-4.1 journaltest.memsubsys2-4.3
> journaltest.memsubsys2-4.4 journaltest.memsubsys2-4.11
> inmemory_journal.memsubsys2-3.1 inmemory_journal.memsubsys2-3.3
> inmemory_journal.memsubsys2-3.4 inmemory_journal.memsubsys2-3.6
> inmemory_journal.memsubsys2-3.7 inmemory_journal.memsubsys2-4.1
> inmemory_journal.memsubsys2-4.3 inmemory_journal.memsubsys2-4.4
> inmemory_journal.memsubsys2-4.11
> Unfreed memory: 29664 bytes
> Writing unfreed memory log to "./memleak.txt"
> Memory used:  now  29664  max3467232  max-size102
> Page-cache used:  now  0  max 13  max-size   4096
> Page-cache overflow:  now  0  max2781888
> Scratch memory used:  now  0  max  0
> Scratch overflow: now  0  max  33080  max-size  33080
> Maximum memory usage: 3467232 bytes
> Current memory usage: 29664 bytes
> Number of malloc()  : -1 calls
> make: *** [fulltest] Error 1
>
> Drop me an email if you would like the log file sent, and/or details on how
> I compiled sqlite and if there is anything else you may wish to know  ;o)
>
> Regards
> Andy
>
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
-
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Memory leak in sqlite 3.7.0

[Andy Gibbs]

Hi,

I'm afraid I believe I have observed a memory leak when running the full 
test suite that is part of sqlite 3.7.0.

I have a log file generated from running the test suite which I can send if 
it is of interest to the developers - it is 5Mb compressed so I didn't think 
it fair to just post it here!

The bottom of the log is as follows:

18 errors out of 1760926 tests
Failures on these tests: journaltest.memsubsys2-3.1 
journaltest.memsubsys2-3.3 journaltest.memsubsys2-3.4 
journaltest.memsubsys2-3.6 journaltest.memsubsys2-3.7 
journaltest.memsubsys2-4.1 journaltest.memsubsys2-4.3 
journaltest.memsubsys2-4.4 journaltest.memsubsys2-4.11 
inmemory_journal.memsubsys2-3.1 inmemory_journal.memsubsys2-3.3 
inmemory_journal.memsubsys2-3.4 inmemory_journal.memsubsys2-3.6 
inmemory_journal.memsubsys2-3.7 inmemory_journal.memsubsys2-4.1 
inmemory_journal.memsubsys2-4.3 inmemory_journal.memsubsys2-4.4 
inmemory_journal.memsubsys2-4.11
Unfreed memory: 29664 bytes
Writing unfreed memory log to "./memleak.txt"
Memory used:  now  29664  max3467232  max-size102
Page-cache used:  now  0  max 13  max-size   4096
Page-cache overflow:  now  0  max2781888
Scratch memory used:  now  0  max  0
Scratch overflow: now  0  max  33080  max-size  33080
Maximum memory usage: 3467232 bytes
Current memory usage: 29664 bytes
Number of malloc()  : -1 calls
make: *** [fulltest] Error 1

Drop me an email if you would like the log file sent, and/or details on how 
I compiled sqlite and if there is anything else you may wish to know  ;o)

Regards
Andy


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak on inserting from sqlite c/c++ interfaces

[Igor Tandetnik]

Zachary Turner 
wrote:
> I still don't understand the transactions.  For example, I issue a
> single BEGIN at the start of my application and then insert about
> 500MB of data through many small inserts (about 4KB each).  During
> this whole time I never issue a commit.  But the main db file grows
> very large, and the journal file remains small.  Is sqlite manually
> forcing commits for me at some fixed threshold?

SQLite uses an undo journal: changes are written to the database file, 
and the journal keeps the original pages. Committing a transaction 
simply means discarding the journal; rolling back means copying saved 
pages from journal back to database file.

Your massive insert operation doesn't modify many pages in the database 
file - it mostly creates new ones by extending the file. For this case, 
the only thing one needs to store in the journal is the original size of 
the database, so that the file could be truncated on rollback. That's 
why the journal file remains the same size even as the main file grows.

Igor Tandetnik 



___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak on inserting from sqlite c/c++ interfaces

[Zachary Turner]

On Wed, Jul 22, 2009 at 1:42 PM, Pavel Ivanov wrote:
>
> Again, try to call sqlite3_memory_used() several times during work of
> your application. What does it say to you?
It says it's using about 3MB.  Which suggests maybe it's my application.

I believe I've identified the problem in my code but it takes some
work to fix so I can't report with 100% certainty how much of a
difference it will make.

> If you don't do commits (and "begins") of transactions by yourself
> then SQLite does that automatically after each executed statement. So
> when sqlite3_step returns you can be sure that everything is committed
> and everything is on disk. SQLite doesn't do any write-through
> caching. And as your transactions are small in volume then journal
> size is also always small.

I still don't understand the transactions.  For example, I issue a
single BEGIN at the start of my application and then insert about
500MB of data through many small inserts (about 4KB each).  During
this whole time I never issue a commit.  But the main db file grows
very large, and the journal file remains small.  Is sqlite manually
forcing commits for me at some fixed threshold?
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak on inserting from sqlite c/c++ interfaces

[Pavel Ivanov]

> Assuming I don't do any manual commits, what does sqlite do with the
> data that has yet to be committed?

If you don't do commits (and "begins") of transactions by yourself
then SQLite does that automatically after each executed statement. So
when sqlite3_step returns you can be sure that everything is committed
and everything is on disk. SQLite doesn't do any write-through
caching. And as your transactions are small in volume then journal
size is also always small.

> Unfortunately I don't have access
> to memory leak detection tools, otherwise that would obviously be
> ideal.

Again, try to call sqlite3_memory_used() several times during work of
your application. What does it say to you?

Pavel

On Wed, Jul 22, 2009 at 2:30 PM, Zachary Turner wrote:
> On Wed, Jul 22, 2009 at 10:47 AM, Pavel Ivanov wrote:
>> SQLite synchronizes with disk during every commit (either issued by
>> yourself or automatic) at least 2 times (I don't know exact number).
>> So it's quite natural that it spends most of the time in winSync().
>> But I still didn't understand from your explanation how exactly your
>> application works and whether it's SQLite uses memory or your
>> application does.
>> BTW, how do you measure memory usage and how do you see leakage? What
>> does sqlite3_memory_used() returns for you?
>>
>> Pavel
>
> I was measuring memory usage by just looking at windows task manager.
> If I watch it for about 20 seconds, it goes up indefinitely until I
> stop reading more data from the file (and thus stop issuing insert
> statements), at which point it steadly declines for a while.
>
> Assuming I don't do any manual commits, what does sqlite do with the
> data that has yet to be committed?  I thought it would store it in the
> journal file, but the journal file always remains consistently very
> small (around 8K max), and data gets written to the actual database
> file even when I'm not doing commits.
>
> I have some ideas about the memory consumption problem that turns out
> to be related to my own code (I agree it's amazingly complicated, but
> it has to be for reasons outside of what we're doing with sqlite).  I
> will investigate that further and post back if I am able to pinpoint
> the issue to sqlite more closely.  Unfortunately I don't have access
> to memory leak detection tools, otherwise that would obviously be
> ideal.
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak on inserting from sqlite c/c++ interfaces

[Zachary Turner]

On Wed, Jul 22, 2009 at 10:47 AM, Pavel Ivanov wrote:
> SQLite synchronizes with disk during every commit (either issued by
> yourself or automatic) at least 2 times (I don't know exact number).
> So it's quite natural that it spends most of the time in winSync().
> But I still didn't understand from your explanation how exactly your
> application works and whether it's SQLite uses memory or your
> application does.
> BTW, how do you measure memory usage and how do you see leakage? What
> does sqlite3_memory_used() returns for you?
>
> Pavel

I was measuring memory usage by just looking at windows task manager.
If I watch it for about 20 seconds, it goes up indefinitely until I
stop reading more data from the file (and thus stop issuing insert
statements), at which point it steadly declines for a while.

Assuming I don't do any manual commits, what does sqlite do with the
data that has yet to be committed?  I thought it would store it in the
journal file, but the journal file always remains consistently very
small (around 8K max), and data gets written to the actual database
file even when I'm not doing commits.

I have some ideas about the memory consumption problem that turns out
to be related to my own code (I agree it's amazingly complicated, but
it has to be for reasons outside of what we're doing with sqlite).  I
will investigate that further and post back if I am able to pinpoint
the issue to sqlite more closely.  Unfortunately I don't have access
to memory leak detection tools, otherwise that would obviously be
ideal.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak on inserting from sqlite c/c++ interfaces

[Marcus Grimm]

just a point:
In your pseudocode you precreate statements for the
begin and commit and trying to reuse them later on.
If your real code is like this I would suggest to
take a look at this. I think it is not possible
to "re-step" a begin or commit, you may try to
recreate these statements completely.
Maybe that is a reason you are seeing a lot of file-syncs
because the begin/commit mechanism doesn't actually
work as intended by you ?

Marcus

Zachary Turner wrote:
> On Tue, Jul 21, 2009 at 6:45 PM, Pavel Ivanov wrote:
>>> If I remove or comment out the
>>> three lines with //* in the code above, I get no memory leaks.
>> So basically you're saying that if you don't insert any data into your
>> database and thus effectively don't do with your database anything and
>> thus SQLite don't have to cache anything from database then you
>> observe no memory leaks, right? Is it by any chance a growing database
>> cache is what you see? What if you execute at the very beginning of
>> your application this:
>> pragma cache_size = 1;
>>
>> Pavel
> 
> Thanks for your suggestion.  I didn't have too much luck with it
> unfortunately, but I did learn a few more things about the problem.
> Just to make sure I understood you though, after creating the database
> with sqlite3_open_v2 I ran the command
> 
> sqlite3_exec(db_, "pragma cache_size=1;");
> 
> The return value was successful so I assume it worked.
> 
> This doesn't appear to change the allocation behavior.  I feel like it
> might be related to me issuing INSERT queries too fast for sqlite to
> deal with.  This is asynchronous (not using the experimental Sqlite
> asynchronous vfs but a completely different async model I wrote
> myself) and the thread that issues inserts to sqlite is different than
> the thread that reads data from the file.
> 
> I changed the code a little so that it permanently stops reading more
> data from the file after some fixed amount (I chose about 500MB) but
> continues writing until all data has been committed to the database.
> when I do this, I notice that after I stop my reads, it takes a *long
> time* before all the writes are finished in sqlite.  99% of this time
> is spent in the winSync() function in the sqlite source code
> constantly calling FlushFileBuffers().
> 
> Upon reflection it makes sense that this would happen in an
> asynchronous environment, but I thought that after a certain threshold
> running an insert statement on a non-in-memory table would generate a
> sync and simply block my sqlite background thread until there was no
> more buffered data.
> 
> In my async model I post messages between threads, so for example the
> main thread posts a message to the sqlite thread saying "write this
> data", which runs the pseudocode I posted in the OP, and after that's
> done it posts a message to the main thread saying "I'm done writing
> that data".  the main thread only issues new read requests as a result
> of getting that message, so if sqlite were syncing every so often then
> I could guarantee that I wouldn't be in this situation because the
> main thread wouldn't be able to issue any more reads until the sync
> was done.
> 
> 
> Am I misunderstanding sqlite's filesystem sync'ing algorithm?
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
> 
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak on inserting from sqlite c/c++ interfaces

[Pavel Ivanov]

SQLite synchronizes with disk during every commit (either issued by
yourself or automatic) at least 2 times (I don't know exact number).
So it's quite natural that it spends most of the time in winSync().
But I still didn't understand from your explanation how exactly your
application works and whether it's SQLite uses memory or your
application does.
BTW, how do you measure memory usage and how do you see leakage? What
does sqlite3_memory_used() returns for you?

Pavel

On Wed, Jul 22, 2009 at 11:26 AM, Zachary Turner wrote:
> On Tue, Jul 21, 2009 at 6:45 PM, Pavel Ivanov wrote:
>>> If I remove or comment out the
>>> three lines with //* in the code above, I get no memory leaks.
>>
>> So basically you're saying that if you don't insert any data into your
>> database and thus effectively don't do with your database anything and
>> thus SQLite don't have to cache anything from database then you
>> observe no memory leaks, right? Is it by any chance a growing database
>> cache is what you see? What if you execute at the very beginning of
>> your application this:
>> pragma cache_size = 1;
>>
>> Pavel
>
> Thanks for your suggestion.  I didn't have too much luck with it
> unfortunately, but I did learn a few more things about the problem.
> Just to make sure I understood you though, after creating the database
> with sqlite3_open_v2 I ran the command
>
> sqlite3_exec(db_, "pragma cache_size=1;");
>
> The return value was successful so I assume it worked.
>
> This doesn't appear to change the allocation behavior.  I feel like it
> might be related to me issuing INSERT queries too fast for sqlite to
> deal with.  This is asynchronous (not using the experimental Sqlite
> asynchronous vfs but a completely different async model I wrote
> myself) and the thread that issues inserts to sqlite is different than
> the thread that reads data from the file.
>
> I changed the code a little so that it permanently stops reading more
> data from the file after some fixed amount (I chose about 500MB) but
> continues writing until all data has been committed to the database.
> when I do this, I notice that after I stop my reads, it takes a *long
> time* before all the writes are finished in sqlite.  99% of this time
> is spent in the winSync() function in the sqlite source code
> constantly calling FlushFileBuffers().
>
> Upon reflection it makes sense that this would happen in an
> asynchronous environment, but I thought that after a certain threshold
> running an insert statement on a non-in-memory table would generate a
> sync and simply block my sqlite background thread until there was no
> more buffered data.
>
> In my async model I post messages between threads, so for example the
> main thread posts a message to the sqlite thread saying "write this
> data", which runs the pseudocode I posted in the OP, and after that's
> done it posts a message to the main thread saying "I'm done writing
> that data".  the main thread only issues new read requests as a result
> of getting that message, so if sqlite were syncing every so often then
> I could guarantee that I wouldn't be in this situation because the
> main thread wouldn't be able to issue any more reads until the sync
> was done.
>
>
> Am I misunderstanding sqlite's filesystem sync'ing algorithm?
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak on inserting from sqlite c/c++ interfaces

[Zachary Turner]

On Tue, Jul 21, 2009 at 6:45 PM, Pavel Ivanov wrote:
>> If I remove or comment out the
>> three lines with //* in the code above, I get no memory leaks.
>
> So basically you're saying that if you don't insert any data into your
> database and thus effectively don't do with your database anything and
> thus SQLite don't have to cache anything from database then you
> observe no memory leaks, right? Is it by any chance a growing database
> cache is what you see? What if you execute at the very beginning of
> your application this:
> pragma cache_size = 1;
>
> Pavel

Thanks for your suggestion.  I didn't have too much luck with it
unfortunately, but I did learn a few more things about the problem.
Just to make sure I understood you though, after creating the database
with sqlite3_open_v2 I ran the command

sqlite3_exec(db_, "pragma cache_size=1;");

The return value was successful so I assume it worked.

This doesn't appear to change the allocation behavior.  I feel like it
might be related to me issuing INSERT queries too fast for sqlite to
deal with.  This is asynchronous (not using the experimental Sqlite
asynchronous vfs but a completely different async model I wrote
myself) and the thread that issues inserts to sqlite is different than
the thread that reads data from the file.

I changed the code a little so that it permanently stops reading more
data from the file after some fixed amount (I chose about 500MB) but
continues writing until all data has been committed to the database.
when I do this, I notice that after I stop my reads, it takes a *long
time* before all the writes are finished in sqlite.  99% of this time
is spent in the winSync() function in the sqlite source code
constantly calling FlushFileBuffers().

Upon reflection it makes sense that this would happen in an
asynchronous environment, but I thought that after a certain threshold
running an insert statement on a non-in-memory table would generate a
sync and simply block my sqlite background thread until there was no
more buffered data.

In my async model I post messages between threads, so for example the
main thread posts a message to the sqlite thread saying "write this
data", which runs the pseudocode I posted in the OP, and after that's
done it posts a message to the main thread saying "I'm done writing
that data".  the main thread only issues new read requests as a result
of getting that message, so if sqlite were syncing every so often then
I could guarantee that I wouldn't be in this situation because the
main thread wouldn't be able to issue any more reads until the sync
was done.


Am I misunderstanding sqlite's filesystem sync'ing algorithm?
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak on inserting from sqlite c/c++ interfaces

[Pavel Ivanov]

> If I remove or comment out the
> three lines with //* in the code above, I get no memory leaks.

So basically you're saying that if you don't insert any data into your
database and thus effectively don't do with your database anything and
thus SQLite don't have to cache anything from database then you
observe no memory leaks, right? Is it by any chance a growing database
cache is what you see? What if you execute at the very beginning of
your application this:
pragma cache_size = 1;

Pavel

On Tue, Jul 21, 2009 at 7:01 PM, Zachary Turner wrote:
> Hello, I'm a bit new to sqlite, I wonder if someone can advise me here.
>
> I'm using the Sqlite C/C++ interfaces, and I'm trying to do some very
> basic things.  Currently I'm just creating a database with 1 table,
> and this table has 1 column of type blob.  I then read some data out
> of a file and insert it into the database, where each row will contain
> some number N of bytes from the file, where N may not necessarily be
> the same in each row.
>
> I'm doing this using the following pseudocode:
>
> ---Initialization---
> 1) db = sqlite3_open_v2("C:\\foo.db")
> 2) sqlite3_exec(db, "CREATE TABLE DummyTable (DummyColumn BLOB)")
> 3) insert_query = sqlite3_prepare_v2(db, "INSERT INTO DummyTable
> (DummyColumn) VALUES (?1)")
> 4) commit_query = sqlite3_prepare_v2(db, "commit")
> 5) begin_query = sqlite3_prepare_v2(db, "begin")
>
>
> ---When I want to write a chunk of the file into the database---
> if (!active_transaction_)
> {
>   //begin a new transaction
>
>   sqlite3_step(begin_query)
>   active_transaction_ = true;
> }
>
> //bind the data to the query and execute the query
> sqlite3_bind_blob(insert_query, data, length)
> sqlite3_step(insert_query)                       // *
> sqlite3_clear_bindings(insert_query)        // *
> sqlite3_reset(insert_query)                      // *
>
> //128 is a made up number, just for the sake of illustration
> if (++count >= 128)
> {
>   //commit the transaction
>
>   sqlite3_step(commit_query)
>   active_transaction_ = false;
> }
>
>
> When I run this code for a while my memory usage grows extremely
> quickly, and I don't understand why.  If I remove or comment out the
> three lines with //* in the code above, I get no memory leaks.  If it
> makes a difference when I call sqlite3_bind_blob I'm using
> SQLITE_TRANSIENT for the final parameter, but my understanding is that
> this is supposed to automatically free the memory when it's no longer
> needed.  Furthermore, the bind itself isn't what's causing the
> problem, because if I leave the bind in and only comment out the
> insert, I don't get the leak anymore.
>
>
> Am I using the interfaces incorrectly or is perhaps something else
> going on that I need to be aware of?
>
> Thanks
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Memory leak on inserting from sqlite c/c++ interfaces

[Zachary Turner]

Hello, I'm a bit new to sqlite, I wonder if someone can advise me here.

I'm using the Sqlite C/C++ interfaces, and I'm trying to do some very
basic things.  Currently I'm just creating a database with 1 table,
and this table has 1 column of type blob.  I then read some data out
of a file and insert it into the database, where each row will contain
some number N of bytes from the file, where N may not necessarily be
the same in each row.

I'm doing this using the following pseudocode:

---Initialization---
1) db = sqlite3_open_v2("C:\\foo.db")
2) sqlite3_exec(db, "CREATE TABLE DummyTable (DummyColumn BLOB)")
3) insert_query = sqlite3_prepare_v2(db, "INSERT INTO DummyTable
(DummyColumn) VALUES (?1)")
4) commit_query = sqlite3_prepare_v2(db, "commit")
5) begin_query = sqlite3_prepare_v2(db, "begin")


---When I want to write a chunk of the file into the database---
if (!active_transaction_)
{
   //begin a new transaction

   sqlite3_step(begin_query)
   active_transaction_ = true;
}

//bind the data to the query and execute the query
sqlite3_bind_blob(insert_query, data, length)
sqlite3_step(insert_query)   // *
sqlite3_clear_bindings(insert_query)// *
sqlite3_reset(insert_query)  // *

//128 is a made up number, just for the sake of illustration
if (++count >= 128)
{
   //commit the transaction

   sqlite3_step(commit_query)
   active_transaction_ = false;
}


When I run this code for a while my memory usage grows extremely
quickly, and I don't understand why.  If I remove or comment out the
three lines with //* in the code above, I get no memory leaks.  If it
makes a difference when I call sqlite3_bind_blob I'm using
SQLITE_TRANSIENT for the final parameter, but my understanding is that
this is supposed to automatically free the memory when it's no longer
needed.  Furthermore, the bind itself isn't what's causing the
problem, because if I leave the bind in and only comment out the
insert, I don't get the leak anymore.


Am I using the interfaces incorrectly or is perhaps something else
going on that I need to be aware of?

Thanks
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1

[Greg Morehead]

Thanks Eric, you nailed it on the head!

With this parameter I am able to configure the page usage and stop the memory 
growth where I want it!  


-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org]on Behalf Of Eric Minbiole
Sent: Monday, June 29, 2009 12:06 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1


> So it seems by best bet is to close and open the connection once every 5 
> minutes or so?

While this might "work", I don't think you should resort to that.  As 
Kees noted earlier, there will be performance drawbacks if you 
close/re-open the database.

As others have indicated, the heap growth is likely due to SQLite 
intentionally caching frequently used disk pages in order to improve 
performance.  Rather than closing/re-opening, I suggest that you set the 
page cache size to a limit that is reasonable for your application:

http://www.sqlite.org/pragma.html#pragma_cache_size

The default is around 2MB.  You can decrease to as little as 10KB.  Note 
that there are some other buffers used by SQLite (Scratch, Lookaside, 
etc).  However, these are typically small compared to the page cache. 
Additional info here:

http://www.sqlite.org/malloc.html

~Eric
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1

[Simon Slavin]

On 29 Jun 2009, at 1:11pm, Greg Morehead wrote:

> So it seems by best bet is to close and open the connection once  
> every 5 minutes or so?

No.  The library is unlikely to have a leak of that kind in.  Are you  
certain that what you're seeing is a real leak ?  If so, the leak may  
be in your own code.  Otherwise, check to see that SQLite is not  
staying entirely within the expected cache memory usage:

http://www.sqlite.org/pragma.html#pragma_default_cache_size

It's possible to manage the cache yourself

http://www.sqlite.org/c3ref/pcache_methods.html

but frankly this is above my head and I wouldn't know where to start.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1

[Eric Minbiole]

> So it seems by best bet is to close and open the connection once every 5 
> minutes or so?

While this might "work", I don't think you should resort to that.  As 
Kees noted earlier, there will be performance drawbacks if you 
close/re-open the database.

As others have indicated, the heap growth is likely due to SQLite 
intentionally caching frequently used disk pages in order to improve 
performance.  Rather than closing/re-opening, I suggest that you set the 
page cache size to a limit that is reasonable for your application:

http://www.sqlite.org/pragma.html#pragma_cache_size

The default is around 2MB.  You can decrease to as little as 10KB.  Note 
that there are some other buffers used by SQLite (Scratch, Lookaside, 
etc).  However, these are typically small compared to the page cache. 
Additional info here:

http://www.sqlite.org/malloc.html

~Eric
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1

[Jay A. Kreibich]

On Mon, Jun 29, 2009 at 08:10:20AM -0400, Greg Morehead scratched on the wall:
> I'm looking specifically at the heap for the process I wrote.

  Well, that's where it would be

   -j

> -Original Message-
> From: sqlite-users-boun...@sqlite.org
> [mailto:sqlite-users-boun...@sqlite.org]on Behalf Of Jay A. Kreibich
> Sent: Friday, June 26, 2009 5:32 PM
> To: General Discussion of SQLite Database
> Subject: Re: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1
> 
> 
> On Fri, Jun 26, 2009 at 05:07:16PM -0400, Greg Morehead scratched on the wall:
> > 
> > If I close then reopen the database all my memory is recovered.  
> > 
> > Is this by design???  I was intending on keeping a connection open most of 
> > time.
> 
>   Are you sure you're not looking at the page cache?
> 
>-j

-- 
Jay A. Kreibich < J A Y  @  K R E I B I.C H >

"Our opponent is an alien starship packed with atomic bombs.  We have
 a protractor."   "I'll go home and see if I can scrounge up a ruler
 and a piece of string."  --from Anathem by Neal Stephenson
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1

[Greg Morehead]

I'm looking specifically at the heap for the process I wrote.

-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org]on Behalf Of Jay A. Kreibich
Sent: Friday, June 26, 2009 5:32 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1


On Fri, Jun 26, 2009 at 05:07:16PM -0400, Greg Morehead scratched on the wall:
> 
> If I close then reopen the database all my memory is recovered.  
> 
> Is this by design???  I was intending on keeping a connection open most of 
> time.

  Are you sure you're not looking at the page cache?

   -j

-- 
Jay A. Kreibich < J A Y  @  K R E I B I.C H >

"Our opponent is an alien starship packed with atomic bombs.  We have
 a protractor."   "I'll go home and see if I can scrounge up a ruler
 and a piece of string."  --from Anathem by Neal Stephenson
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1

[Greg Morehead]

So it seems by best bet is to close and open the connection once every 5 
minutes or so?


-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org]on Behalf Of Kees Nuyt
Sent: Friday, June 26, 2009 5:31 PM
To: sqlite-users@sqlite.org
Subject: Re: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1


On Fri, 26 Jun 2009 17:07:16 -0400, "Greg Morehead"
 wrote:

>
>If I close then reopen the database all my memory is recovered.  
>
>Is this by design???  

Yes, what you see is probably the page cache.


>I was intending on keeping a connection open most of time.

That's a good idea, for at least two reasons:

- opening a connection has to parse the schema, 
  and though it's fast code, it should be avoided.

- the contents of the page cache aren't wasted, 
  it may be re-used by subsequent statements.

-- 
  (  Kees Nuyt
  )
c[_]
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1

[Jay A. Kreibich]

On Fri, Jun 26, 2009 at 05:07:16PM -0400, Greg Morehead scratched on the wall:
> 
> If I close then reopen the database all my memory is recovered.  
> 
> Is this by design???  I was intending on keeping a connection open most of 
> time.

  Are you sure you're not looking at the page cache?

   -j

-- 
Jay A. Kreibich < J A Y  @  K R E I B I.C H >

"Our opponent is an alien starship packed with atomic bombs.  We have
 a protractor."   "I'll go home and see if I can scrounge up a ruler
 and a piece of string."  --from Anathem by Neal Stephenson
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1

[Kees Nuyt]

On Fri, 26 Jun 2009 17:07:16 -0400, "Greg Morehead"
 wrote:

>
>If I close then reopen the database all my memory is recovered.  
>
>Is this by design???  

Yes, what you see is probably the page cache.


>I was intending on keeping a connection open most of time.

That's a good idea, for at least two reasons:

- opening a connection has to parse the schema, 
  and though it's fast code, it should be avoided.

- the contents of the page cache aren't wasted, 
  it may be re-used by subsequent statements.

-- 
  (  Kees Nuyt
  )
c[_]
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1

[Greg Morehead]

If I close then reopen the database all my memory is recovered.  

Is this by design???  I was intending on keeping a connection open most of time.


-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org]on Behalf Of Greg Morehead
Sent: Friday, June 26, 2009 4:41 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1


FYI.  I replaced the sqlite3_exec call with sqlite3_prepare_v2, sqlite3_step, 
sqlite3_finalize.

Same results.

-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org]on Behalf Of Greg Morehead
Sent: Friday, June 26, 2009 4:21 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1


Based on the documentation in the link you sent I should only need to call free 
if there was an error message.

But, since there is no harm in calling sqlite3_free on a null pointer I moved 
it out of the if statement.  It had no impact, still leaking like faucet.

-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org]on Behalf Of Ribeiro, Glauber
Sent: Friday, June 26, 2009 4:17 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1


I took only a quick look, but it seems to me that sqlite3_free is only
being called if there is an error.
See http://sqlite.org/c3ref/exec.html

g

-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Greg Morehead
Sent: Friday, June 26, 2009 2:49 PM
To: sqlite-users@sqlite.org
Subject: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1

The extremely simple app below leaks.  What am I doing wrong??  Please
help.


#include 
#include 
#include "stdio.h"
#include "sqlite3.h"
#include 

#define TFQ_SQL_DB_NAME "/powerblock/datalog/TFQ-test.db"
#define _TABLE_NAME "sqltest"

int main(int argc, char *argv[]) 
{
char sql[1024];
char * errmsg;
sqlite3* db_;
int flags = SQLITE_OPEN_FULLMUTEX | SQLITE_OPEN_READWRITE |
SQLITE_OPEN_CREATE;

if (sqlite3_open_v2(TFQ_SQL_DB_NAME
, &db_
, flags
, NULL) != SQLITE_OK)
{
return EXIT_FAILURE;
}

sprintf(sql,
"CREATE TABLE %s_info (col_idx, col_name,
data_type, meta_type, description, grouping); "
, _TABLE_NAME);

int retval = sqlite3_exec(db_, sql, 0, 0, &errmsg);
int idx = 1;
while(retval==SQLITE_OK)
{
sprintf(sql,
"INSERT INTO %s_info VALUES(%d,
\"rowid\", \"INTEGER\", \"%s\", \"Unique Row ID\", \"_SQL\");"
, _TABLE_NAME, idx++, "INTEGER" );

retval = sqlite3_exec(db_, sql, 0, 0, &errmsg); 

if(retval != SQLITE_OK) {
printf("Recieved an sqlite failure. : Failed
msg: %s", errmsg);
sqlite3_free(errmsg);
}
else
printf("Successfully inserted row idx %d\r\n",
idx);

usleep(5);
}
return EXIT_SUCCESS;
}

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1

[Greg Morehead]

FYI.  I replaced the sqlite3_exec call with sqlite3_prepare_v2, sqlite3_step, 
sqlite3_finalize.

Same results.

-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org]on Behalf Of Greg Morehead
Sent: Friday, June 26, 2009 4:21 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1


Based on the documentation in the link you sent I should only need to call free 
if there was an error message.

But, since there is no harm in calling sqlite3_free on a null pointer I moved 
it out of the if statement.  It had no impact, still leaking like faucet.

-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org]on Behalf Of Ribeiro, Glauber
Sent: Friday, June 26, 2009 4:17 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1


I took only a quick look, but it seems to me that sqlite3_free is only
being called if there is an error.
See http://sqlite.org/c3ref/exec.html

g

-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Greg Morehead
Sent: Friday, June 26, 2009 2:49 PM
To: sqlite-users@sqlite.org
Subject: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1

The extremely simple app below leaks.  What am I doing wrong??  Please
help.


#include 
#include 
#include "stdio.h"
#include "sqlite3.h"
#include 

#define TFQ_SQL_DB_NAME "/powerblock/datalog/TFQ-test.db"
#define _TABLE_NAME "sqltest"

int main(int argc, char *argv[]) 
{
char sql[1024];
char * errmsg;
sqlite3* db_;
int flags = SQLITE_OPEN_FULLMUTEX | SQLITE_OPEN_READWRITE |
SQLITE_OPEN_CREATE;

if (sqlite3_open_v2(TFQ_SQL_DB_NAME
, &db_
, flags
, NULL) != SQLITE_OK)
{
return EXIT_FAILURE;
}

sprintf(sql,
"CREATE TABLE %s_info (col_idx, col_name,
data_type, meta_type, description, grouping); "
, _TABLE_NAME);

int retval = sqlite3_exec(db_, sql, 0, 0, &errmsg);
int idx = 1;
while(retval==SQLITE_OK)
{
sprintf(sql,
"INSERT INTO %s_info VALUES(%d,
\"rowid\", \"INTEGER\", \"%s\", \"Unique Row ID\", \"_SQL\");"
, _TABLE_NAME, idx++, "INTEGER" );

retval = sqlite3_exec(db_, sql, 0, 0, &errmsg); 

if(retval != SQLITE_OK) {
printf("Recieved an sqlite failure. : Failed
msg: %s", errmsg);
sqlite3_free(errmsg);
}
else
printf("Successfully inserted row idx %d\r\n",
idx);

usleep(5);
}
return EXIT_SUCCESS;
}

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1

[Greg Morehead]

Based on the documentation in the link you sent I should only need to call free 
if there was an error message.

But, since there is no harm in calling sqlite3_free on a null pointer I moved 
it out of the if statement.  It had no impact, still leaking like faucet.

-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org]on Behalf Of Ribeiro, Glauber
Sent: Friday, June 26, 2009 4:17 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1


I took only a quick look, but it seems to me that sqlite3_free is only
being called if there is an error.
See http://sqlite.org/c3ref/exec.html

g

-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Greg Morehead
Sent: Friday, June 26, 2009 2:49 PM
To: sqlite-users@sqlite.org
Subject: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1

The extremely simple app below leaks.  What am I doing wrong??  Please
help.


#include 
#include 
#include "stdio.h"
#include "sqlite3.h"
#include 

#define TFQ_SQL_DB_NAME "/powerblock/datalog/TFQ-test.db"
#define _TABLE_NAME "sqltest"

int main(int argc, char *argv[]) 
{
char sql[1024];
char * errmsg;
sqlite3* db_;
int flags = SQLITE_OPEN_FULLMUTEX | SQLITE_OPEN_READWRITE |
SQLITE_OPEN_CREATE;

if (sqlite3_open_v2(TFQ_SQL_DB_NAME
, &db_
, flags
, NULL) != SQLITE_OK)
{
return EXIT_FAILURE;
}

sprintf(sql,
"CREATE TABLE %s_info (col_idx, col_name,
data_type, meta_type, description, grouping); "
, _TABLE_NAME);

int retval = sqlite3_exec(db_, sql, 0, 0, &errmsg);
int idx = 1;
while(retval==SQLITE_OK)
{
sprintf(sql,
"INSERT INTO %s_info VALUES(%d,
\"rowid\", \"INTEGER\", \"%s\", \"Unique Row ID\", \"_SQL\");"
, _TABLE_NAME, idx++, "INTEGER" );

retval = sqlite3_exec(db_, sql, 0, 0, &errmsg); 

if(retval != SQLITE_OK) {
printf("Recieved an sqlite failure. : Failed
msg: %s", errmsg);
sqlite3_free(errmsg);
}
else
printf("Successfully inserted row idx %d\r\n",
idx);

usleep(5);
}
return EXIT_SUCCESS;
}

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1

[Greg Morehead]

I'm watching the heap from the eclipse IDE which connects to the remote 
debugging qconn service on the target.

I've included a screen shot from the last run of the code I included.  This was 
over 1 minute period where onlky 2782 records were written!

The problem is much worse on the much larger real db's I am planning on using.


-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org]on Behalf Of D. Richard Hipp
Sent: Friday, June 26, 2009 3:58 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1



On Jun 26, 2009, at 3:49 PM, Greg Morehead wrote:

> The extremely simple app below leaks.  What am I doing wrong??   
> Please help.

How do you know it is leaking memory?  How are you measuring?

>
>
> #include 
> #include 
> #include "stdio.h"
> #include "sqlite3.h"
> #include 
>
> #define TFQ_SQL_DB_NAME "/powerblock/datalog/TFQ-test.db"
> #define _TABLE_NAME "sqltest"
>
> int main(int argc, char *argv[])
> {
>   char sql[1024];
>   char * errmsg;
>   sqlite3* db_;
>   int flags = SQLITE_OPEN_FULLMUTEX | SQLITE_OPEN_READWRITE |  
> SQLITE_OPEN_CREATE;
>   
>   if (sqlite3_open_v2(TFQ_SQL_DB_NAME
>   , &db_
>   , flags
>   , NULL) != SQLITE_OK)   {
>   return EXIT_FAILURE;
>   }
>   
>   sprintf(sql,
>   "CREATE TABLE %s_info (col_idx, col_name, data_type, 
> meta_type,  
> description, grouping); "
>   , _TABLE_NAME);
>
>   int retval = sqlite3_exec(db_, sql, 0, 0, &errmsg);
>   int idx = 1;
>   while(retval==SQLITE_OK)
>   {
>   sprintf(sql,
>   "INSERT INTO %s_info VALUES(%d, \"rowid\", 
> \"INTEGER\", \"%s\",  
> \"Unique Row ID\", \"_SQL\");"
>   , _TABLE_NAME, idx++, "INTEGER" );
>   
>   retval = sqlite3_exec(db_, sql, 0, 0, &errmsg); 
>   
>   if(retval != SQLITE_OK) {
>   printf("Recieved an sqlite failure. : Failed msg: %s", 
> errmsg);
>   sqlite3_free(errmsg);
>   }
>   else
>   printf("Successfully inserted row idx %d\r\n", idx);
>   
>   usleep(5);
>   }
>   return EXIT_SUCCESS;
> }
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

D. Richard Hipp
d...@hwaci.com



___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1

[Ribeiro, Glauber]

I took only a quick look, but it seems to me that sqlite3_free is only
being called if there is an error.
See http://sqlite.org/c3ref/exec.html

g

-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Greg Morehead
Sent: Friday, June 26, 2009 2:49 PM
To: sqlite-users@sqlite.org
Subject: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1

The extremely simple app below leaks.  What am I doing wrong??  Please
help.


#include 
#include 
#include "stdio.h"
#include "sqlite3.h"
#include 

#define TFQ_SQL_DB_NAME "/powerblock/datalog/TFQ-test.db"
#define _TABLE_NAME "sqltest"

int main(int argc, char *argv[]) 
{
char sql[1024];
char * errmsg;
sqlite3* db_;
int flags = SQLITE_OPEN_FULLMUTEX | SQLITE_OPEN_READWRITE |
SQLITE_OPEN_CREATE;

if (sqlite3_open_v2(TFQ_SQL_DB_NAME
, &db_
, flags
, NULL) != SQLITE_OK)
{
return EXIT_FAILURE;
}

sprintf(sql,
"CREATE TABLE %s_info (col_idx, col_name,
data_type, meta_type, description, grouping); "
, _TABLE_NAME);

int retval = sqlite3_exec(db_, sql, 0, 0, &errmsg);
int idx = 1;
while(retval==SQLITE_OK)
{
sprintf(sql,
"INSERT INTO %s_info VALUES(%d,
\"rowid\", \"INTEGER\", \"%s\", \"Unique Row ID\", \"_SQL\");"
, _TABLE_NAME, idx++, "INTEGER" );

retval = sqlite3_exec(db_, sql, 0, 0, &errmsg); 

if(retval != SQLITE_OK) {
printf("Recieved an sqlite failure. : Failed
msg: %s", errmsg);
sqlite3_free(errmsg);
}
else
printf("Successfully inserted row idx %d\r\n",
idx);

usleep(5);
}
return EXIT_SUCCESS;
}

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory leak with sqlite3_exec on qnx 6.4.1

[D. Richard Hipp]

On Jun 26, 2009, at 3:49 PM, Greg Morehead wrote:

> The extremely simple app below leaks.  What am I doing wrong??   
> Please help.

How do you know it is leaking memory?  How are you measuring?

>
>
> #include 
> #include 
> #include "stdio.h"
> #include "sqlite3.h"
> #include 
>
> #define TFQ_SQL_DB_NAME "/powerblock/datalog/TFQ-test.db"
> #define _TABLE_NAME "sqltest"
>
> int main(int argc, char *argv[])
> {
>   char sql[1024];
>   char * errmsg;
>   sqlite3* db_;
>   int flags = SQLITE_OPEN_FULLMUTEX | SQLITE_OPEN_READWRITE |  
> SQLITE_OPEN_CREATE;
>   
>   if (sqlite3_open_v2(TFQ_SQL_DB_NAME
>   , &db_
>   , flags
>   , NULL) != SQLITE_OK)   {
>   return EXIT_FAILURE;
>   }
>   
>   sprintf(sql,
>   "CREATE TABLE %s_info (col_idx, col_name, data_type, 
> meta_type,  
> description, grouping); "
>   , _TABLE_NAME);
>
>   int retval = sqlite3_exec(db_, sql, 0, 0, &errmsg);
>   int idx = 1;
>   while(retval==SQLITE_OK)
>   {
>   sprintf(sql,
>   "INSERT INTO %s_info VALUES(%d, \"rowid\", 
> \"INTEGER\", \"%s\",  
> \"Unique Row ID\", \"_SQL\");"
>   , _TABLE_NAME, idx++, "INTEGER" );
>   
>   retval = sqlite3_exec(db_, sql, 0, 0, &errmsg); 
>   
>   if(retval != SQLITE_OK) {
>   printf("Recieved an sqlite failure. : Failed msg: %s", 
> errmsg);
>   sqlite3_free(errmsg);
>   }
>   else
>   printf("Successfully inserted row idx %d\r\n", idx);
>   
>   usleep(5);
>   }
>   return EXIT_SUCCESS;
> }
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

D. Richard Hipp
d...@hwaci.com



___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Memory leak with sqlite3_exec on qnx 6.4.1

[Greg Morehead]

The extremely simple app below leaks.  What am I doing wrong??  Please help.


#include 
#include 
#include "stdio.h"
#include "sqlite3.h"
#include 

#define TFQ_SQL_DB_NAME "/powerblock/datalog/TFQ-test.db"
#define _TABLE_NAME "sqltest"

int main(int argc, char *argv[]) 
{
char sql[1024];
char * errmsg;
sqlite3* db_;
int flags = SQLITE_OPEN_FULLMUTEX | SQLITE_OPEN_READWRITE | 
SQLITE_OPEN_CREATE;

if (sqlite3_open_v2(TFQ_SQL_DB_NAME
, &db_
, flags
, NULL) != SQLITE_OK)   {
return EXIT_FAILURE;
}

sprintf(sql,
"CREATE TABLE %s_info (col_idx, col_name, data_type, 
meta_type, description, grouping); "
, _TABLE_NAME);

int retval = sqlite3_exec(db_, sql, 0, 0, &errmsg);
int idx = 1;
while(retval==SQLITE_OK)
{
sprintf(sql,
"INSERT INTO %s_info VALUES(%d, \"rowid\", 
\"INTEGER\", \"%s\", \"Unique Row ID\", \"_SQL\");"
, _TABLE_NAME, idx++, "INTEGER" );

retval = sqlite3_exec(db_, sql, 0, 0, &errmsg); 

if(retval != SQLITE_OK) {
printf("Recieved an sqlite failure. : Failed msg: %s", 
errmsg);
sqlite3_free(errmsg);
}
else
printf("Successfully inserted row idx %d\r\n", idx);

usleep(5);
}
return EXIT_SUCCESS;
}

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Memory Leak on select ?

[Igor Tandetnik]

<[EMAIL PROTECTED]> wrote in message
news:[EMAIL PROTECTED]
> I'm using sqlite3 on embedded system.
> I've got a very very large
> memory usage.
>
> zSql =
> sqlite3_mprintf("SELECT PortNumber, RangeMin, RangeMax, Offset, MinL,
> MinLL, MaxH, MaxHH, Delay, Hysteresis, " \
>"Alarm,
> TimeAlarm FROM input_ai WHERE Address=%d and Monitor='Y';", 7);

sqlite3_mprintf allocates memory on the heap and returns pointer to it. 
You should deallocate it with sqlite3_free. Better still, use 
parameterized query - see sqlite3_bind_*

Igor Tandetnik



___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Memory Leak on select ?

[[EMAIL PROTECTED]]

I'm using sqlite3 on embedded system. 
I've got a very very large 
memory usage.

My program must loop to show some values catched from a 
table using SELECT statement.
In a while memory usage grow over 
phisical RAM, so my device lock.
If I run the same code on a PC after 
some minutes the process use a very large amount of memory (512M+).

I 
need to use some function to force memory release or I make some 
mistake ?

Thanks in advance
Pierluigi 

Follow sample code:

#include 

#include 
#include 
#include 

int main (int argc, char *argv[])
{

// Define variables
sqlite3 
*db;
int rc,i;
char *zErrMsg = 0;
char *zSql;
char *prevStatus;

sqlite3_stmt *stmt;

struct data
{
int iPortNumber;
float 
fRangeMin;
float fRangeMax;
float fOffset;
float fMinL;

float fMinLL;
float fMaxH;
float fMaxHH;
int iDelay;
int 
iHyster;
int iAlarm;
char *sTimeAlarm;
};  
struct data riga
[16];

// Open Database
rc = sqlite3_open("/tmp/test.db", &db);
if( 
rc )
{
   fprintf(stderr, "Can't open database: %s\n", sqlite3_errmsg
(db));
   sqlite3_close(db);
   exit(-1);
}

for(rc=0; rc < 
1; rc++)
{
// Init variables
i=0;

zSql = 
sqlite3_mprintf("SELECT PortNumber, RangeMin, RangeMax, Offset, MinL, 
MinLL, MaxH, MaxHH, Delay, Hysteresis, " \
   "Alarm, 
TimeAlarm FROM input_ai WHERE Address=%d and Monitor='Y';", 7);

if 
(sqlite3_prepare(db, zSql, -1, &stmt, 0) != SQLITE_OK)
return(-1);


   
// Save at least 16 rows
while ((sqlite3_step(stmt)
==SQLITE_ROW) & (i < 16))
{
  
  riga[i].iPortNumber = 
sqlite3_column_int(stmt,0);
  riga[i].fRangeMin = 
sqlite3_column_double(stmt,1);
  riga[i].fRangeMax = 
sqlite3_column_double(stmt,2);
  riga[i].fOffset = 
sqlite3_column_double(stmt,3);
  riga[i].fMinL = 
sqlite3_column_double(stmt,4);
  riga[i].fMinLL = 
sqlite3_column_double(stmt,5);
  riga[i].fMaxH = 
sqlite3_column_double(stmt,6);
  riga[i].fMaxHH = 
sqlite3_column_double(stmt,7);
  riga[i].iDelay = 
sqlite3_column_int(stmt,8);
  riga[i].iHyster = sqlite3_column_int
(stmt,9);
  riga[i].iAlarm = sqlite3_column_int(stmt,10);
  
riga[i].sTimeAlarm = (char *) sqlite3_column_text(stmt,11);

  
i++;
}

sqlite3_finalize(stmt);

}
sqlite3_close(db);

}






___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] memory leak on sqlite3_open

[drh]

arbalest06 <[EMAIL PROTECTED]> wrote:
> 
> Is this a valid memory leak?
> 

No, it is not a real leak.  But we used to get complaints about
it about ever week so I reworked the code to make Valgrind stop
complaining.  That happened about 6 months ago.  If you run with
the latest version of SQLite Valgrind will no longer complain.

--
D. Richard Hipp <[EMAIL PROTECTED]>


-
To unsubscribe, send email to [EMAIL PROTECTED]
-

[sqlite] memory leak on sqlite3_open

[arbalest06]

Good day!

I have a C function that would open a database. But when i ran it in
valgrind, it gave me the following leak:

==7230== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 16 from 1)
==7230== malloc/free: in use at exit: 272 bytes in 2 blocks.
==7230== malloc/free: 289 allocs, 287 frees, 42,704 bytes allocated.
==7230== For counts of detected errors, rerun with: -v
==7230== searching for pointers to 2 not-freed blocks.
==7230== checked 21,050,692 bytes.
==7230==
==7230== 272 bytes in 2 blocks are possibly lost in loss record 1 of 1
==7230==at 0x40056BF: calloc (vg_replace_malloc.c:279)
==7230==by 0x35771A: _dl_allocate_tls (in /lib/ld-2.3.4.so)
==7230==by 0x5B591E: pthread_create@@GLIBC_2.1 (in
/lib/tls/libpthread-2.3.4.so)
==7230==by 0x4030E52: findLockInfo (os_unix.c:504)
==7230==by 0x4031ABD: unixOpen (os_unix.c:2268)
==7230==by 0x403080C: sqlite3OsOpen (os.c:134)
==7230==by 0x40330A7: sqlite3PagerOpen (pager.c:2160)
==7230==by 0x401707E: sqlite3BtreeOpen (btree.c:1226)
==7230==by 0x402F03C: sqlite3BtreeFactory (main.c:767)
==7230==by 0x402F68D: openDatabase (main.c:1016)
==7230==by 0x8049482: OpenDB (in
/home/sample/testing/sqlite3/v0.08/driver)
==7230==by 0x80489F5: main (DbDriver.c:60)
==7230==
==7230== LEAK SUMMARY:
==7230==definitely lost: 0 bytes in 0 blocks.
==7230==  possibly lost: 272 bytes in 2 blocks.
==7230==still reachable: 0 bytes in 0 blocks.
==7230== suppressed: 0 bytes in 0 blocks.
==7230== Reachable blocks (those to which a pointer was found) are not
shown.
==7230== To see them, rerun with: --show-reachable=yes

In DbDriver.c, my line number 60 is:

[60] if( ( res_i = OpenDB( "this.db" ) ) == SQLITE_OK )
[61] {
  ...
   }

and my OpenDB looks like this:

int OpenDB( char *dbName_cp )
{
int retVal_i = CDBM_FAIL; /*< error code */

/* checks if the database name is not equal to NULL */
if( dbName_cp != NULL )
{
/* opens the database */
retVal_i = sqlite3_open( dbName_cp, &GDBM_Db_p );
}

return( retVal_i );
}

Is this a valid memory leak?..where did i go wrong in my codes?..

Thank you and God bless!
-- 
View this message in context: 
http://www.nabble.com/memory-leak-on-sqlite3_open-tp14629415p14629415.html
Sent from the SQLite mailing list archive at Nabble.com.


-
To unsubscribe, send email to [EMAIL PROTECTED]
-

[sqlite] Memory leak in 3.3.14 & 3.3.15

[Juri Wichanow]

With Windows we found a memory leak in Exclusive access mode. Doing just 
selects sqlite takes more and more memory. With the same code in Normal access 
mode we do not have any memory leaks. Did anyone else find this, too?

We haven't had similar problems with older versions, yet.

Thanks
Juri

Re: [sqlite] memory leak

[Martin Jenkins]

Hi,

I think you're going to have to be a bit more rigorous about what you're
testing if you're hoping for a fix.

In your root message you posted a bit of code and said you'd found a
leak in v2.8.17 of the library.

Then you said the leak was present in v2.8.17 and v3.3.6 (the valgrind
of 'bash' post) and showed the header from the sqlite shell v3.3.6

Then you said the leak was present in 3.3.6 and posted a valgrind log of
the sqlite shell v3.2.2 run from a directory .../3.3.6.

You didn't say what hardware you're running and in the latter cases
where you're testing the shell it would be useful if you said what you
DID in the shell.

I just compiled valgrind-3.2.0 (vs your valgrind-3.1.1) on a Pentium 1
machine and ran it against a locally built 3.3.6 shell. All I did in the
shell was issue a ".q" to quit. My log is rather different to yours, as
shown below:

deb:~/valgrind-3.2.0# uname -a
Linux deb 2.6.8-3-386 #1 Sat Jul 15 09:26:40 UTC 2006 i586 GNU/Linux
deb:~/valgrind-3.2.0#
deb:~/valgrind-3.2.0# valgrind --leak-check=full sqlite3
==19856== Memcheck, a memory error detector.
==19856== Copyright (C) 2002-2006, and GNU GPL'd, by Julian Seward et al.
==19856== Using LibVEX rev 1606, a library for dynamic binary translation.
==19856== Copyright (C) 2004-2006, and GNU GPL'd, by OpenWorks LLP.
==19856== Using valgrind-3.2.0, a dynamic binary instrumentation framework.
==19856== Copyright (C) 2000-2006, and GNU GPL'd, by Julian Seward et al.
==19856== For more details, rerun with: -v
==19856==
--19856-- DWARF2 CFI reader: unhandled CFI instruction 0:50
--19856-- DWARF2 CFI reader: unhandled CFI instruction 0:50
SQLite version 3.3.6
Enter ".help" for instructions
sqlite> .q
==19856==
==19856== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 13 from 1)
==19856== malloc/free: in use at exit: 31 bytes in 2 blocks.
==19856== malloc/free: 8 allocs, 6 frees, 1,536 bytes allocated.
==19856== For counts of detected errors, rerun with: -v
==19856== searching for pointers to 2 not-freed blocks.
==19856== checked 86,964 bytes.
==19856==
==19856== 6 bytes in 1 blocks are definitely lost in loss record 1 of 2
==19856==at 0x401B6E7: malloc (vg_replace_malloc.c:149)
==19856==by 0x804CC44: find_home_dir (shell.c:1580)
==19856==by 0x804D169: main (shell.c:1806)
==19856==
==19856==
==19856== 25 bytes in 1 blocks are definitely lost in loss record 2 of 2
==19856==at 0x401B6E7: malloc (vg_replace_malloc.c:149)
==19856==by 0x804D182: main (shell.c:1807)
==19856==
==19856== LEAK SUMMARY:
==19856==definitely lost: 31 bytes in 2 blocks.
==19856==  possibly lost: 0 bytes in 0 blocks.
==19856==still reachable: 0 bytes in 0 blocks.
==19856== suppressed: 0 bytes in 0 blocks.
==19856== Reachable blocks (those to which a pointer was found) are not
shown.
==19856== To see them, rerun with: --show-reachable=yes

Looking at the code in shell.c these are valid leak reports but they're
in the shell, not the library, and I suspect there will be plenty of
things with a higher priority than adding code to explicitly free a
couple of small buffers which will be freed anyway when the shell is
closed.

Since all I did was issue a ".q" it's not a valid test of the library,
so I did another run on the following session and the valgrind report
was EXACTLY the same as the simple ".q" test above - two trivial leaks
reported at lines 1806/7 in shell.c.

SQLite version 3.3.6
Enter ".help" for instructions
sqlite> create table fred(a,b,c);
sqlite> insert into fred values(1,2,3);
sqlite> select * from fred;
1|2|3
sqlite> .q

As you don't say what you did in the shell to get those leak reports
it's hard to duplicate your results. If you post what you did I'll
re-run the valgrind test.

Martin

Re: [sqlite] memory leak

[mycmos]

--- [EMAIL PROTECTED] wrote:


> It appears that you are in fact running /bin/bash,
> not
> SQLite, and finding memory leaks there.  Perhaps
> "sqlite3"
> is really a shell script on your system.  Try
> running
> valgrind on a real sqlite3 executable and I believe
> you
> will find that there are no memory leaks.
> --
> D. Richard Hipp   <[EMAIL PROTECTED]>
> 
> 

 Execuse me for my wrong with wrong sqlite3, Here
is the real binary sqlite3 in valgrind, and I has NOT
defined SQLITE_ENABLE_MEMORY_MANAGEMENT :

[EMAIL PROTECTED] /data/lfs/sqlite-3.3.6/.libs]# valgrind
--leak-check=full ./sqlite3
==28654== Memcheck, a memory error detector.
==28654== Copyright (C) 2002-2005, and GNU GPL'd, by
Julian Seward et al.
==28654== Using LibVEX rev 1575, a library for dynamic
binary translation.
==28654== Copyright (C) 2004-2005, and GNU GPL'd, by
OpenWorks LLP.
==28654== Using valgrind-3.1.1, a dynamic binary
instrumentation framework.
==28654== Copyright (C) 2000-2005, and GNU GPL'd, by
Julian Seward et al.
==28654== For more details, rerun with: -v
==28654==
SQLite version 3.2.2
Enter ".help" for instructions
sqlite> .exit
==28654==
==28654== ERROR SUMMARY: 0 errors from 0 contexts
(suppressed: 19 from 2)
==28654== malloc/free: in use at exit: 53,786 bytes in
322 blocks.
==28654== malloc/free: 487 allocs, 165 frees, 67,138
bytes allocated.
==28654== For counts of detected errors, rerun with:
-v
==28654== searching for pointers to 322 not-freed
blocks.
==28654== checked 142,284 bytes.
==28654==
==28654== 6 bytes in 1 blocks are possibly lost in
loss record 2 of 14
==28654==at 0x401B46D: malloc
(vg_replace_malloc.c:149)
==28654==by 0x804C4DB: find_home_dir
(shell.c:1553)
==28654==by 0x804C9EB: main (shell.c:1779)
==28654==
==28654==
==28654== 25 bytes in 1 blocks are definitely lost in
loss record 6 of 14
==28654==at 0x401B46D: malloc
(vg_replace_malloc.c:149)
==28654==by 0x804CA10: main (shell.c:1780)
==28654==
==28654==
==28654== 40 bytes in 2 blocks are definitely lost in
loss record 7 of 14
==28654==at 0x401B46D: malloc
(vg_replace_malloc.c:149)
==28654==by 0x4153FB9: xmalloc (in
/usr/lib/libreadline.so.4.3)
==28654==by 0x41575EC: sh_set_lines_and_columns
(in /usr/lib/libreadline.so.4.3)
==28654==by 0x4150AAE: _rl_get_screen_size (in
/usr/lib/libreadline.so.4.3)
==28654==by 0x415114C: _rl_init_terminal_io (in
/usr/lib/libreadline.so.4.3)
==28654==by 0x413FA77: rl_initialize (in
/usr/lib/libreadline.so.4.3)
==28654==by 0x413FC42: readline (in
/usr/lib/libreadline.so.4.3)
==28654==by 0x804A988: process_input (shell.c:198)
==28654==by 0x804CA3E: main (shell.c:1786)
==28654==
==28654== LEAK SUMMARY:
==28654==definitely lost: 65 bytes in 3 blocks.
==28654==  possibly lost: 6 bytes in 1 blocks.
==28654==still reachable: 53,715 bytes in 318
blocks.
==28654== suppressed: 0 bytes in 0 blocks.
==28654== Reachable blocks (those to which a pointer
was found) are not shown.
==28654== To see them, rerun with:
--show-reachable=yes
[EMAIL PROTECTED] /data/lfs/sqlite-3.3.6/.libs]#








___ 
Mp3疯狂搜-新歌热歌高速下
http://music.yahoo.com.cn/?source=mail_mailbox_footer

RE: [sqlite] memory leak

[Brannon King]

One quick question, is that with memory management enabled in the sqlite
compile or not? I've done some extensive memory checking with memory
management disabled and never found a leak. However, with memory management
enabled, all bets are off.

> I believe the memory leak exists. The following is excerpt 
> from sqlite-3.3.6 in valgrind (In fact, the
> sqlite-2.8.17 in valgrind is similar):

Re: [sqlite] memory leak

[drh]

mycmos <[EMAIL PROTECTED]> wrote:
> 
> I believe the memory leak exists. The following is
> excerpt from sqlite-3.3.6 in valgrind (In fact, the
> sqlite-2.8.17 in valgrind is similar):
> 
> [EMAIL PROTECTED] /data/lfs/sqlite-3.3.6]# valgrind
> --leak-check=full ./sqlite3
[...]
> ==11765==
> ==11765== 27 bytes in 1 blocks are definitely lost in
> loss record 1 of 4
> ==11765==at 0x401B46D: malloc
> (vg_replace_malloc.c:149)
> ==11765==by 0x80958E1: xmalloc (in /bin/bash)
> ==11765==by 0x8069D65: execute_command_internal
> (in /bin/bash)
> ==11765==by 0x8099EBB: parse_and_execute (in
> /bin/bash)
> ==11765==by 0x807E28A: command_substitute (in
> /bin/bash)
> ==11765==by 0x8081F74: (within /bin/bash)
> ==11765==by 0x8081C03: (within /bin/bash)
> ==11765==by 0x80835A1: (within /bin/bash)
> ==11765==by 0x806A901: execute_command_internal
> (in /bin/bash)
> ==11765==by 0x806C67F: execute_command (in
> /bin/bash)
> ==11765==by 0x806D7C3: (within /bin/bash)
> ==11765==by 0x806A4F6: execute_command_internal
> (in /bin/bash)

It appears that you are in fact running /bin/bash, not
SQLite, and finding memory leaks there.  Perhaps "sqlite3"
is really a shell script on your system.  Try running
valgrind on a real sqlite3 executable and I believe you
will find that there are no memory leaks.
--
D. Richard Hipp   <[EMAIL PROTECTED]>

Re: [sqlite] memory leak

[mycmos]

--- Martin Jenkins <[EMAIL PROTECTED]>写道:

> mycmos wrote:
> > When I use cat /proc/meminfo, I found the MemFree
> is 
> > decreasing. Does that means some degree memory
> leak to
> > Sqlite2.8.17?
> 
> No. ;)
> 
> If you run "ps -ef" or "top" you'll see a list of
> all of the processes
> running on your machine. How could you associate
> changes in MemFree with
> any one of those processes? Short answer - you
> can't. ;)
> 
> I ran sqlite's test suite last week and (IIRC) it
> has about 20,000 test
> cases. I didn't watch the whole test but I'd be
> surprised if there
> wasn't a memory leak test in there somewhere and
> given the number of
> people using sqlite I'm sure a leak in 2.8.17 would
> have been found and
> fixed by now.
> 
> Martin
> 
> PS Usual stuff about 2.8.17 being old and 3.3.6
> being new goes here.
> 

I believe the memory leak exists. The following is
excerpt from sqlite-3.3.6 in valgrind (In fact, the
sqlite-2.8.17 in valgrind is similar):

[EMAIL PROTECTED] /data/lfs/sqlite-3.3.6]# valgrind
--leak-check=full ./sqlite3
==11762== Memcheck, a memory error detector.
==11762== Copyright (C) 2002-2005, and GNU GPL'd, by
Julian Seward et al.
==11762== Using LibVEX rev 1575, a library for dynamic
binary translation.
==11762== Copyright (C) 2004-2005, and GNU GPL'd, by
OpenWorks LLP.
==11762== Using valgrind-3.1.1, a dynamic binary
instrumentation framework.
==11762== Copyright (C) 2000-2005, and GNU GPL'd, by
Julian Seward et al.
==11762== For more details, rerun with: -v
==11762==
==11765==
==11765== ERROR SUMMARY: 0 errors from 0 contexts
(suppressed: 16 from 1)
==11765== malloc/free: in use at exit: 24,808 bytes in
1,109 blocks.
==11765== malloc/free: 2,831 allocs, 1,722 frees,
107,119 bytes allocated.
==11765== For counts of detected errors, rerun with:
-v
==11765== searching for pointers to 1,109 not-freed
blocks.
==11765== checked 129,076 bytes.
==11765==
==11765== 27 bytes in 1 blocks are definitely lost in
loss record 1 of 4
==11765==at 0x401B46D: malloc
(vg_replace_malloc.c:149)
==11765==by 0x80958E1: xmalloc (in /bin/bash)
==11765==by 0x8069D65: execute_command_internal
(in /bin/bash)
==11765==by 0x8099EBB: parse_and_execute (in
/bin/bash)
==11765==by 0x807E28A: command_substitute (in
/bin/bash)
==11765==by 0x8081F74: (within /bin/bash)
==11765==by 0x8081C03: (within /bin/bash)
==11765==by 0x80835A1: (within /bin/bash)
==11765==by 0x806A901: execute_command_internal
(in /bin/bash)
==11765==by 0x806C67F: execute_command (in
/bin/bash)
==11765==by 0x806D7C3: (within /bin/bash)
==11765==by 0x806A4F6: execute_command_internal
(in /bin/bash)
==11765==
==11765== LEAK SUMMARY:
==11765==definitely lost: 27 bytes in 1 blocks.
==11765==  possibly lost: 0 bytes in 0 blocks.
==11765==still reachable: 24,781 bytes in 1,108
blocks.
==11765== suppressed: 0 bytes in 0 blocks.
==11765== Reachable blocks (those to which a pointer
was found) are not shown.
==11765== To see them, rerun with:
--show-reachable=yes
==11764==
==11764== ERROR SUMMARY: 0 errors from 0 contexts
(suppressed: 16 from 1)
==11764== malloc/free: in use at exit: 24,274 bytes in
1,085 blocks.
==11764== malloc/free: 2,792 allocs, 1,707 frees,
105,672 bytes allocated.
==11764== For counts of detected errors, rerun with:
-v
==11764== searching for pointers to 1,085 not-freed
blocks.
==11764== checked 128,544 bytes.
==11764==
==11764== LEAK SUMMARY:
==11764==definitely lost: 0 bytes in 0 blocks.
==11764==  possibly lost: 0 bytes in 0 blocks.
==11764==still reachable: 24,274 bytes in 1,085
blocks.
==11764== suppressed: 0 bytes in 0 blocks.
==11764== Reachable blocks (those to which a pointer
was found) are not shown.
==11764== To see them, rerun with:
--show-reachable=yes
==11767==
==11767== ERROR SUMMARY: 0 errors from 0 contexts
(suppressed: 16 from 1)
==11767== malloc/free: in use at exit: 24,716 bytes in
1,104 blocks.
==11767== malloc/free: 2,868 allocs, 1,764 frees,
107,574 bytes allocated.
==11767== For counts of detected errors, rerun with:
-v
==11767== searching for pointers to 1,104 not-freed
blocks.
==11767== checked 129,208 bytes.
==11767==
==11767== 15 bytes in 1 blocks are definitely lost in
loss record 1 of 4
==11767==at 0x401B46D: malloc
(vg_replace_malloc.c:149)
==11767==by 0x80958E1: xmalloc (in /bin/bash)
==11767==by 0x806AF94: execute_command_internal
(in /bin/bash)
==11767==by 0x806DC83: (within /bin/bash)
==11767==by 0x806A1F4: execute_command_internal
(in /bin/bash)
==11767==by 0x8099EBB: parse_and_execute (in
/bin/bash)
==11767==by 0x807E28A: command_substitute (in
/bin/bash)
==11767==by 0x8081F74: (within /bin/bash)
==11767==by 0x8082634: (within /bin/bash)
==11767==by 0x8082EF0: (within /bin/bash)
==11767==by 0x8082F3C: expand_string_unsplit (in
/bin/bash)
==11767==by 0x807A974: (within /bin/bash)
==11767==
==11767== LEAK SUMMARY:
==11767==definitely lost: 15 bytes in 1 bloc

Re: [sqlite] memory leak

[Peter Weilbacher]

mycmos wrote:

> I am using sqlite2.8.17 for linux2.6. My program
> is as following:
>
> int main()
> {
>sqlite *db;
> 
>while(1){
>   db = sqlite_open("./example.db",0777,0);
>   if(db == 0){
>  printf("Could not open database.");
>  exit(1);
>   }
>   sleep(2);
>   sqlite_close(db);
>}
>  
>return 1;
> }
> 
> When I use cat /proc/meminfo, I found the MemFree is 
> decreasing. Does that means some degree memory leak to
> Sqlite2.8.17?

That can mean a number of things, a leak not necessarily being the most
likely. If there is a leak then you should be able to verify that by
running under the Valgrind tool (http://valgrind.org/).
   Peter.

Re: [sqlite] memory leak

[Martin Jenkins]

mycmos wrote:
> When I use cat /proc/meminfo, I found the MemFree is 
> decreasing. Does that means some degree memory leak to
> Sqlite2.8.17?

No. ;)

If you run "ps -ef" or "top" you'll see a list of all of the processes
running on your machine. How could you associate changes in MemFree with
any one of those processes? Short answer - you can't. ;)

I ran sqlite's test suite last week and (IIRC) it has about 20,000 test
cases. I didn't watch the whole test but I'd be surprised if there
wasn't a memory leak test in there somewhere and given the number of
people using sqlite I'm sure a leak in 2.8.17 would have been found and
fixed by now.

Martin

PS Usual stuff about 2.8.17 being old and 3.3.6 being new goes here.

[sqlite] memory leak

[mycmos]

Hi all,
I am using sqlite2.8.17 for linux2.6. My program
is as following:
   
int main()
{
   sqlite *db;

   while(1){
  db = sqlite_open("./example.db",0777,0);
  if(db == 0){
 printf("Could not open database.");
 exit(1);
  }
  sleep(2);
  sqlite_close(db);
   }
 
   return 1;
}

When I use cat /proc/meminfo, I found the MemFree is 
decreasing. Does that means some degree memory leak to
Sqlite2.8.17?

Best regards
mycmos






___ 
Mp3疯狂搜-新歌热歌高速下
http://music.yahoo.com.cn/?source=mail_mailbox_footer

[sqlite] RE:Re: [sqlite] RE:Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE

[[EMAIL PROTECTED]]

Hi Robert,
I have never used SQLite with sqlite3_prepare(), 
sqlite3_step(), sqlite3_reset() and sqlite3_finalize().
Do you have any code example that i can use to avoid the use of 
sqlite_get_table() ?
Thank you,
Eduardo

---Mensaje original---I don't use the sqlite_get_table() function, and don't 
recommend it to 
others to use. As I understand it, it's there for legacy application 
support. New programs written for SQLite should use sqlite3_prepare(), 
sqlite3_step(), sqlite3_reset() and sqlite3_finalize() instead.

Robert


- Original Message - 
From: 
To: ; 
Sent: Friday, March 17, 2006 7:08 AM
Subject: [sqlite] RE:Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on 
Windows CE



Hi Robert,
I was talking about 3 selects satements using the same connections.
Anyway, thank you very much for your advice of using "PRAGMA cache 
size=8", that solved all the problems related to sqlite3_exec memory 
problems with a select statement, baut the memory problems are not solved at 
all, stilll i get a memory leak when using sqlite3_get_table, even if i use 
sqlite3_free_table not all the memory is freed. Someone pointed that this 
might be an error on the source code becuse the ARM processor architecture 
is quite different from the X86 processor. I have been taking a look at the 
table.c file but i am not that good programer, to see if anything fails, Do 
you see anything on the table.c that might be generating the problem?.
I have attached to this email a memory graphic of sqlite3_get_table problem




El día del padre está cerca... ¿Ya tienes el regalo? 
http://buscawanadoo.es/search?type=pref&destino=web&origen=homespot&buscar=Regalos%20Día%20del%20Padre

Re: [sqlite] RE:Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE

[Robert Simpson]

I don't use the sqlite_get_table() function, and don't recommend it to 
others to use.  As I understand it, it's there for legacy application 
support.  New programs written for SQLite should use sqlite3_prepare(), 
sqlite3_step(), sqlite3_reset() and sqlite3_finalize() instead.


Robert


- Original Message - 
From: <[EMAIL PROTECTED]>

To: ; <[EMAIL PROTECTED]>
Sent: Friday, March 17, 2006 7:08 AM
Subject: [sqlite] RE:Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on 
Windows CE




Hi Robert,
I was talking about 3 selects satements using the same connections.
Anyway, thank you very much for your advice of using "PRAGMA cache 
size=8", that solved all the problems related to sqlite3_exec memory 
problems with a select statement, baut the memory problems are not solved at 
all, stilll i get a memory leak when using sqlite3_get_table, even if i use 
sqlite3_free_table not all the memory is freed. Someone pointed that this 
might be an error on the source code becuse the ARM processor architecture 
is quite different from the X86 processor. I have been taking a look at the 
table.c file but i am not that good programer, to see if anything fails, Do 
you see anything on the table.c that might be generating the problem?.

I have attached to this email a memory graphic of sqlite3_get_table problem

[sqlite] RE:Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE

[[EMAIL PROTECTED]]

Hi Robert,
I was talking about 3 selects satements using the same connections.
Anyway, thank you very much for your advice of using "PRAGMA cache 
size=8", that solved all the problems related to sqlite3_exec memory 
problems with a select statement, baut the memory problems are not solved at 
all, stilll i get a memory leak when using sqlite3_get_table, even if i use 
sqlite3_free_table not all the memory is freed. Someone pointed that this might 
be an error on the source code becuse the ARM processor architecture is quite 
different from the X86 processor. I have been taking a look at the table.c file 
but i am not that good programer, to see if anything fails, Do you see anything 
on the table.c that might be generating the problem?.
I have attached to this email a memory graphic of sqlite3_get_table problem 

---Mensaje original Original Message - 
From: 

> I have run your program on the CE emulator (Pocket PC 2003)
> and i got the same memory leak.
> I have inserted 2 buttons on a MFC dialog application.
> The first button executes your code and the second button
> closes the application.
> If you examine the memory you will discover that the program
> only free the memory once you exit from the apllication,
> meanwhile it reserves memory as its needed (on demand, but
> see details below).
> the memory behaviour of SQLite is quite strange, an example:
> lets say that a select sentence reserves 1000kb of memory,
> once this local process has finished memory keeps reserved
> for the program (it should be freed), if another process
> executes a select sentence that needs 200kb SQLite will not
> reserve 200k more, it will use 200k of the previous 1000k
> reserved. if a 3rd process executes a select sentence that
> needs 1300k SQlite will reserve 300kb more and those 1300kb
> will not be freed until the main dialog application closes
> (even if the 3 process where local methods or functions).

Ok this is where you lost me. 3 processes? Is your program running 3 times 
on the CE platform? If CE is running 3 instances of your program, then they 
definitely won't be sharing any memory and yes you'll definitely run out. 
Also if I recall correctly, CE 5.0 will not let you run multiple instances 
of the same program.

If you're talking about 3 SELECT statements in the same program using the 
same connection instance, then that's another story.

> The problem is that if a select sentence consume most of the
> memory it will not be freed and the program will execute very
> slow until you exit from the application because there will
> be so little memory left for other not SQLite process that
> the program might be unusable.

SQLite's default cache size is about 3mb. After opening a connection, try 
executing "PRAGMA cache_size=8" or some really low number and tell me 
if its 
still "leaking".

Robert




El día del padre está cerca... ¿Ya tienes el regalo? 
http://buscawanadoo.es/search?type=pref&destino=web&origen=homespot&buscar=Regalos%20Día%20del%20Padre

Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE

[John Stanton]

Doug Nebeker wrote:

It sounds to me that he isn't saying it is leaking--it simply isn't
releasing memory after a SELECT statement finishes (is finalized).  This
might be by design.  I would expect the data to be released if the
database connection is closed, but not necessarily after each SELECT.
It would be nice to have an API to force any cached pages to be released
(maybe it already exists and I don't know about it?). 


-Original Message-
From: Robert Simpson [mailto:[EMAIL PROTECTED] 
Sent: Thursday, March 16, 2006 12:13 PM

To: sqlite-users@sqlite.org
Subject: Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows
CE

- Original Message -
From: <[EMAIL PROTECTED]>

I have run your program on the CE emulator (Pocket PC 2003) and i got 
the same memory leak.

I have inserted 2 buttons on a MFC dialog application.
The first button executes your code and the second button closes the 
application.

If you examine the memory you will discover that the program only free



the memory once you exit from the apllication, meanwhile it reserves 
memory as its needed (on demand, but see details below).

the memory behaviour of SQLite is quite strange, an example:
lets say that a select sentence reserves 1000kb of memory, once this 
local process has finished memory keeps reserved for the program (it 
should be freed), if another process executes a select sentence that 
needs 200kb SQLite will not reserve 200k more, it will use 200k of the



previous 1000k reserved. if a 3rd process executes a select sentence 
that needs 1300k SQlite will reserve 300kb more and those 1300kb will 
not be freed until the main dialog application closes (even if the 3 
process where local methods or functions).



Ok this is where you lost me.  3 processes?  Is your program running 3
times on the CE platform?  If CE is running 3 instances of your program,
then they definitely won't be sharing any memory and yes you'll
definitely run out. 
Also if I recall correctly, CE 5.0 will not let you run multiple

instances of the same program.

If you're talking about 3 SELECT statements in the same program using
the same connection instance, then that's another story.



The problem is that if a select sentence consume most of the memory it



will not be freed and the program will execute very slow until you 
exit from the application because there will be so little memory left 
for other not SQLite process that the program might be unusable.



SQLite's default cache size is about 3mb.  After opening a connection,
try executing "PRAGMA cache_size=8" or some really low number and tell
me if its still "leaking".

Robert




To find out more about Reuters visit www.about.reuters.com

Any views expressed in this message are those of the individual sender, except 
where the sender specifically states them to be the views of Reuters Ltd.

He uses the table API call to get the output.  By necessity this must 
allocate dynamic memory to accommodate the result set.  We can guess 
that he then runs through the table and looks at the result rows.


If he were to use an sqlite3_step and process each row as it is 
retrieved that dynamic memory requirement would be eliminated.  Not only 
would memory usage be better controlled, but the application would run 
faster by eliminating the delay while te table is created.


Some thought on how malloc and free work would help in conceptualising 
the application.  The best that can usually be achieved is that over a 
period of time the application hits a memory "high water mark" and does 
not go beyond that.  Minimizing dynamic memory allocation limits that 
high water level.  Free cannot be guaranteed to ensure that malloc'd 
memory is totally reused.  That requires garbage collection.

[sqlite] RE:Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE

[[EMAIL PROTECTED]]

Hi,
Thank you very much for your help, i have modify the line, it compiles, it 
doesn´t fire any exception, but memory is not freed at all (works the same way)
Anyone has another idea?, i tought that SQLite was working since a long time 
ago on Windows CE devices, it seems quite strange that no one detected this 
memory problem before?.

Thank you,
Eduardo
---Mensaje original---On March 16, 2006 12:28 pm, Jose Da Silva wrote:

I forgot a line:
char x[12] = "Hello world\0";
char *ptr;
ptr = x;
while (*ptr) {printf("%c",*ptr);   ptr;};

> If you figure out the right sizeof(???) value to use, then I think it
> could be submitted as a bug-fix for \src\table.c

You want to move the pointer from azResult[1] to azResult[0] but you 
want to say it without doing it like this "azResult--;" on line 192, 
but saying it like this looks sort of ugly although it might work:
azResult -= ( &azResult[1] - &azResult[0] );

..so if someone has a simpler solution, thanks :-)


El día del padre está cerca... ¿Ya tienes el regalo? 
http://buscawanadoo.es/search?type=pref&destino=web&origen=homespot&buscar=Regalos%20Día%20del%20Padre

[sqlite] RE:Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE

[[EMAIL PROTECTED]]

Hi Robert,
I was talking about 3 selects satements using the same connections.
Anyway, thank you very much for your advice of using "PRAGMA cache 
size=8", that solved all the problems related to sqlite3_exec memory 
problems with a select statement, baut the memory problems are not solved at 
all, stilll i get a memory leak when using sqlite3_get_table, even if i use 
sqlite3_free_table not all the memory is freed. Someone pointed that this might 
be an error on the source code becuse the ARM processor architecture is quite 
different from the X86 processor. I have been takin a look at the table.c file 
but i am not that good programer, to see if anything fails, Do you see anything 
on the table.c that might be generating the problem?.
Thank you very much,
Eduardo

---Mensaje original Original Message - 
From: 

> I have run your program on the CE emulator (Pocket PC 2003)
> and i got the same memory leak.
> I have inserted 2 buttons on a MFC dialog application.
> The first button executes your code and the second button
> closes the application.
> If you examine the memory you will discover that the program
> only free the memory once you exit from the apllication,
> meanwhile it reserves memory as its needed (on demand, but
> see details below).
> the memory behaviour of SQLite is quite strange, an example:
> lets say that a select sentence reserves 1000kb of memory,
> once this local process has finished memory keeps reserved
> for the program (it should be freed), if another process
> executes a select sentence that needs 200kb SQLite will not
> reserve 200k more, it will use 200k of the previous 1000k
> reserved. if a 3rd process executes a select sentence that
> needs 1300k SQlite will reserve 300kb more and those 1300kb
> will not be freed until the main dialog application closes
> (even if the 3 process where local methods or functions).

Ok this is where you lost me. 3 processes? Is your program running 3 times 
on the CE platform? If CE is running 3 instances of your program, then they 
definitely won't be sharing any memory and yes you'll definitely run out. 
Also if I recall correctly, CE 5.0 will not let you run multiple instances 
of the same program.

If you're talking about 3 SELECT statements in the same program using the 
same connection instance, then that's another story.

> The problem is that if a select sentence consume most of the
> memory it will not be freed and the program will execute very
> slow until you exit from the application because there will
> be so little memory left for other not SQLite process that
> the program might be unusable.

SQLite's default cache size is about 3mb. After opening a connection, try 
executing "PRAGMA cache_size=8" or some really low number and tell me 
if its 
still "leaking".

Robert




El día del padre está cerca... ¿Ya tienes el regalo? 
http://buscawanadoo.es/search?type=pref&destino=web&origen=homespot&buscar=Regalos%20Día%20del%20Padre

[sqlite] RE:Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE

[[EMAIL PROTECTED]]

Hi Robert,
I was talking about 3 selects satements using the same connections.
Anyway, thank you very much for your advice of using "PRAGMA cache 
size=8", that solved all the problems related to sqlite3_exec memory 
problems with a select statement, baut the memory problems are not solved at 
all, stilll i get a memory leak when using sqlite3_get_table, even if i use 
sqlite3_free_table not all the memory is freed. Someone pointed that this might 
be an error on the source code becuse the ARM processor architecture is quite 
different from the X86 processor. I have been takin a look at the table.c file 
but i am not that good programer, to see if anything fails, Do you see anything 
on the table.c that might be generating the problem?.
Thank you very much,
Eduardo

---Mensaje original Original Message - 
From: 

> I have run your program on the CE emulator (Pocket PC 2003)
> and i got the same memory leak.
> I have inserted 2 buttons on a MFC dialog application.
> The first button executes your code and the second button
> closes the application.
> If you examine the memory you will discover that the program
> only free the memory once you exit from the apllication,
> meanwhile it reserves memory as its needed (on demand, but
> see details below).
> the memory behaviour of SQLite is quite strange, an example:
> lets say that a select sentence reserves 1000kb of memory,
> once this local process has finished memory keeps reserved
> for the program (it should be freed), if another process
> executes a select sentence that needs 200kb SQLite will not
> reserve 200k more, it will use 200k of the previous 1000k
> reserved. if a 3rd process executes a select sentence that
> needs 1300k SQlite will reserve 300kb more and those 1300kb
> will not be freed until the main dialog application closes
> (even if the 3 process where local methods or functions).

Ok this is where you lost me. 3 processes? Is your program running 3 times 
on the CE platform? If CE is running 3 instances of your program, then they 
definitely won't be sharing any memory and yes you'll definitely run out. 
Also if I recall correctly, CE 5.0 will not let you run multiple instances 
of the same program.

If you're talking about 3 SELECT statements in the same program using the 
same connection instance, then that's another story.

> The problem is that if a select sentence consume most of the
> memory it will not be freed and the program will execute very
> slow until you exit from the application because there will
> be so little memory left for other not SQLite process that
> the program might be unusable.

SQLite's default cache size is about 3mb. After opening a connection, try 
executing "PRAGMA cache_size=8" or some really low number and tell me 
if its 
still "leaking".

Robert




El día del padre está cerca... ¿Ya tienes el regalo? 
http://buscawanadoo.es/search?type=pref&destino=web&origen=homespot&buscar=Regalos%20Día%20del%20Padre

[sqlite] RE:Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE

[[EMAIL PROTECTED]]

Hi,
I guess that you are right. Robert gave me the hint to change chae size using 
the PRAGMA command, and that did solve all the problems i hadt with the 
sqlite3_exec command, but it did not solved the problems related to 
sqlite3_get_table.
I have been taking a look at table.c code and i am not a great programmer too 
and i reviwed the code buti do not see anything strange that might be 
generating this problem, i hope someone with more experience could solve it?.

Thank you,
Eduardo
---Mensaje original---On March 16, 2006 08:49 am, [EMAIL PROTECTED] wrote:
> Hi Again Robert,
> I have run your program on the CE emulator (Pocket PC 2003) and i got
> the same memory leak. I have inserted 2 buttons on a MFC dialog
> application.
> The first button executes your code and the second button closes the
> application. If you examine the memory you will discover that the
> program only free the memory once you exit from the apllication,
> meanwhile it reserves memory as its needed (on demand, but see
> details below). the memory behaviour of SQLite is quite strange, an
> example: lets say that a select sentence reserves 1000kb of memory,
> once this local process has finished memory keeps reserved for the
> program (it should be freed), if another process executes a select
> sentence that needs 200kb SQLite will not reserve 200k more, it will
> use 200k of the previous 1000k reserved. if a 3rd process executes a
> select sentence that needs 1300k SQlite will reserve 300kb more and
> those 1300kb will not be freed until the main dialog application
> closes (even if the 3 process where local methods or functions). The
> problem is that if a select sentence consume most of the memory it
> will not be freed and the program will execute very slow until you
> exit from the application because there will be so little memory left
> for other not SQLite process that the program might be unusable.
>
> Any ideas,

Hi again, I think the problem lies with the type of processor being used 
and the way that sqlite (or other programs for that matter) are 
written. I think your problem is also related to another thread:
"[sqlite] SQLITE_CORRUPT problem in Mac OS X"
The reason I suggest that is because the Mac uses a power PC I believe, 
and in your case it uses an ARM processor.
The Intel, x86, and several processors tend to be able to pack bytes, 
words, longs, etc, next to each other. The ARM, the power PC, Sun, and 
other risc processors tend to align them on boundaries.
So while, it is easy to point at a string and then use a basic increment 
function to look up-n-down a string on an x86, it tends to cause 
problems with processors that pack things differently. example:

Please realize I did not test this code, so it may not run as typed.
char x[12] = "Hello world\0";
char *ptr;
while (*ptr) {printf("%c",*ptr);   ptr;};
The code above should work okay on an x86 or other CISC processor, but 
may have problems on some RISC processors due to alignment issues, 
especially considering the fact the string was defined using [] but it 
is getting accessed using a *ptr and being incremented usingor --.
With a RISC processor, you need to take alignment into consideration, 
and therefore the distance may not necessarily beor --, but some 
other distance.

When I looked at src\table.c I noticed that malloc uses a formula like 
(sizeof(char*)*stringsize 1) which is why I suggested trying to modify 
line 192 using "result -=(char*);" instead of "result--;"

I hope this makes sense, so although I guessed that -=sizeof(char*); 
perhaps it should be another value inserted there, but I'm not that 
great a programmer to have guessed the correct value;
Going back to table.c you have this:
--
void sqlite3_free_table(char **azResult){
if( azResult ){ <--this is pointing to azResult[1] if you look at 
the previous "sqlite3_get_table()" routines.
int i, n;
azResult--; <--I think this is wrong for ARM processor.
and should be a distance of azResult[1] to azResult[0] which I think is 
another value than --
if( azResult==0 ) return; <--probably pointing to wrong spot now!
n = (int)azResult[0];
for(i=1; i(the line above is accessing info correctly for an ARM but can be 
coded 
quicker using *azResult if an x86 type processor was used, but 
unfortunately wouldn't work then)
free(azResult);
}
}
--

I hope the above makes sense, so although I though it was 
-=sizeof(char*);
It probably has to be another value, hopefully someone else know.

If you figure out the right sizeof(???) value to use, then I think it 
could be submitted as a bug-fix for \src\table.c



El día del padre está cerca... ¿Ya tienes el regalo? 
http://buscawanadoo.es/search?type=pref&destino=web&origen=homespot&buscar=Regalos%20Día%20del%20Padre

Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE

[Jose Da Silva]

On March 16, 2006 12:28 pm, Jose Da Silva wrote:

I forgot a line:
char x[12] = "Hello world\0";
char *ptr;
ptr = x;
while (*ptr) {printf("%c",*ptr); ++ptr;};

> If you figure out the right sizeof(???) value to use, then I think it
> could be submitted as a bug-fix for \src\table.c

You want to move the pointer from azResult[1] to azResult[0] but you 
want to say it without doing it like this "azResult--;" on line 192, 
but saying it like this looks sort of ugly although it might work:
azResult -= ( &azResult[1] - &azResult[0] );

...so if someone has a simpler solution, thanks  :-)

Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE

[Jose Da Silva]

On March 16, 2006 08:49 am, [EMAIL PROTECTED]  wrote:
> Hi Again Robert,
> I have run your program on the CE emulator (Pocket PC 2003) and i got
> the same memory leak. I have inserted 2 buttons on a MFC dialog
> application.
> The first button executes your code and the second button closes the
> application. If you examine the memory you will discover that the
> program only free the memory once you exit from the apllication,
> meanwhile it reserves memory as its needed (on demand, but see
> details below). the memory behaviour of SQLite is quite strange, an
> example: lets say that a select sentence reserves 1000kb of memory,
> once this local process has finished memory keeps reserved for the
> program (it should be freed), if another process executes a select
> sentence that needs 200kb SQLite will not reserve 200k more, it will
> use 200k of the previous 1000k reserved. if a 3rd process executes a
> select sentence that needs 1300k SQlite will reserve 300kb more and
> those 1300kb will not be freed until the main dialog application
> closes (even if the 3 process where local methods or functions). The
> problem is that if a select sentence consume most of the memory it
> will not be freed and the program will execute very slow until you
> exit from the application because there will be so little memory left
> for other not SQLite process that the program might be unusable.
>
> Any ideas,

Hi again, I think the problem lies with the type of processor being used 
and the way that sqlite (or other programs for that matter) are 
written. I think your problem is also related to another thread:
"[sqlite] SQLITE_CORRUPT problem in Mac OS X"
The reason I suggest that is because the Mac uses a power PC I believe, 
and in your case it uses an ARM processor.
The Intel, x86, and several processors tend to be able to pack bytes, 
words, longs, etc, next to each other. The ARM, the power PC, Sun, and 
other risc processors tend to align them on boundaries.
So while, it is easy to point at a string and then use a basic increment 
function to look up-n-down a string on an x86, it tends to cause 
problems with processors that pack things differently. example:

Please realize I did not test this code, so it may not run as typed.
char x[12] = "Hello world\0";
char *ptr;
while (*ptr) {printf("%c",*ptr); ++ptr;};
The code above should work okay on an x86 or other CISC processor, but 
may have problems on some RISC processors due to alignment issues, 
especially considering the fact the string was defined using [] but it 
is getting accessed using a *ptr and being incremented using ++ or --.
With a RISC processor, you need to take alignment into consideration, 
and therefore the distance may not necessarily be ++ or --, but some 
other distance.

When I looked at src\table.c I noticed that malloc uses a formula like 
(sizeof(char*)*stringsize+1) which is why I suggested trying to modify 
line 192 using "result -=(char*);" instead of "result--;"

I hope this makes sense, so although I guessed that -=sizeof(char*); 
perhaps it should be another value inserted there, but I'm not that 
great a programmer to have guessed the correct value;
Going back to table.c you have this:
--
void sqlite3_free_table(char **azResult){
  if( azResult ){ <--this is pointing to azResult[1] if you look at 
the previous "sqlite3_get_table()" routines.
int i, n;
azResult--;  <--I think this is wrong for ARM processor.
and should be a distance of azResult[1] to azResult[0] which I think is 
another value than --
if( azResult==0 ) return; <--probably pointing to wrong spot now!
n = (int)azResult[0];
for(i=1; i

RE: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE

[Doug Nebeker]

It sounds to me that he isn't saying it is leaking--it simply isn't
releasing memory after a SELECT statement finishes (is finalized).  This
might be by design.  I would expect the data to be released if the
database connection is closed, but not necessarily after each SELECT.
It would be nice to have an API to force any cached pages to be released
(maybe it already exists and I don't know about it?). 

-Original Message-
From: Robert Simpson [mailto:[EMAIL PROTECTED] 
Sent: Thursday, March 16, 2006 12:13 PM
To: sqlite-users@sqlite.org
Subject: Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows
CE

- Original Message -
From: <[EMAIL PROTECTED]>

> I have run your program on the CE emulator (Pocket PC 2003) and i got 
> the same memory leak.
> I have inserted 2 buttons on a MFC dialog application.
> The first button executes your code and the second button closes the 
> application.
> If you examine the memory you will discover that the program only free

> the memory once you exit from the apllication, meanwhile it reserves 
> memory as its needed (on demand, but see details below).
> the memory behaviour of SQLite is quite strange, an example:
> lets say that a select sentence reserves 1000kb of memory, once this 
> local process has finished memory keeps reserved for the program (it 
> should be freed), if another process executes a select sentence that 
> needs 200kb SQLite will not reserve 200k more, it will use 200k of the

> previous 1000k reserved. if a 3rd process executes a select sentence 
> that needs 1300k SQlite will reserve 300kb more and those 1300kb will 
> not be freed until the main dialog application closes (even if the 3 
> process where local methods or functions).

Ok this is where you lost me.  3 processes?  Is your program running 3
times on the CE platform?  If CE is running 3 instances of your program,
then they definitely won't be sharing any memory and yes you'll
definitely run out. 
Also if I recall correctly, CE 5.0 will not let you run multiple
instances of the same program.

If you're talking about 3 SELECT statements in the same program using
the same connection instance, then that's another story.

> The problem is that if a select sentence consume most of the memory it

> will not be freed and the program will execute very slow until you 
> exit from the application because there will be so little memory left 
> for other not SQLite process that the program might be unusable.

SQLite's default cache size is about 3mb.  After opening a connection,
try executing "PRAGMA cache_size=8" or some really low number and tell
me if its still "leaking".

Robert




To find out more about Reuters visit www.about.reuters.com

Any views expressed in this message are those of the individual sender, except 
where the sender specifically states them to be the views of Reuters Ltd.

Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE

[Robert Simpson]

- Original Message - 
From: <[EMAIL PROTECTED]>



I have run your program on the CE emulator (Pocket PC 2003)
and i got the same memory leak.
I have inserted 2 buttons on a MFC dialog application.
The first button executes your code and the second button
closes the application.
If you examine the memory you will discover that the program
only free the memory once you exit from the apllication,
meanwhile it reserves memory as its needed (on demand, but
see details below).
the memory behaviour of SQLite is quite strange, an example:
lets say that a select sentence reserves 1000kb of memory,
once this local process has finished memory keeps reserved
for the program (it should be freed), if another process
executes a select sentence that needs 200kb SQLite will not
reserve 200k more, it will use 200k of the previous 1000k
reserved. if a 3rd process executes a select sentence that
needs 1300k SQlite will reserve 300kb more and those 1300kb
will not be freed until the main dialog application closes
(even if the 3 process where local methods or functions).


Ok this is where you lost me.  3 processes?  Is your program running 3 times 
on the CE platform?  If CE is running 3 instances of your program, then they 
definitely won't be sharing any memory and yes you'll definitely run out. 
Also if I recall correctly, CE 5.0 will not let you run multiple instances 
of the same program.


If you're talking about 3 SELECT statements in the same program using the 
same connection instance, then that's another story.



The problem is that if a select sentence consume most of the
memory it will not be freed and the program will execute very
slow until you exit from the application because there will
be so little memory left for other not SQLite process that
the program might be unusable.


SQLite's default cache size is about 3mb.  After opening a connection, try 
executing "PRAGMA cache_size=8" or some really low number and tell me if its 
still "leaking".


Robert

[sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE

[[EMAIL PROTECTED]]

Hi Again Robert,
I have run your program on the CE emulator (Pocket PC 2003) and i got the same 
memory leak.
I have inserted 2 buttons on a MFC dialog application.
The first button executes your code and the second button closes the 
application.
If you examine the memory you will discover that the program only free the 
memory once you exit from the apllication, meanwhile it reserves memory as its 
needed (on demand, but see details below).
the memory behaviour of SQLite is quite strange, an example:
lets say that a select sentence reserves 1000kb of memory, once this local 
process has finished memory keeps reserved for the program (it should be 
freed), if another process executes a select sentence that needs 200kb SQLite 
will not reserve 200k more, it will use 200k of the previous 1000k reserved. if 
a 3rd process executes a select sentence that needs 1300k SQlite will reserve 
300kb more and those 1300kb will not be freed until the main dialog application 
closes (even if the 3 process where local methods or functions).
The problem is that if a select sentence consume most of the memory it will not 
be freed and the program will execute very slow until you exit from the 
application because there will be so little memory left for other not SQLite 
process that the program might be unusable.

Any ideas,
Thank you in advance
Eduardo
---Mensaje original---Ok, here's what I did ...

On the desktop I created a SQLite database with one table and inserted 
120,000 rows into it. I then copied it over to the CE emulator.

Then I ran the following code on the CE (Pocket PC 2003 SE) emulator:

#include 
#include 
#include "sqlite3.h"

int WINAPI _tWinMain(HINSTANCE hInst, HINSTANCE hPrev, LPTSTR pszCmdLine, 
int nCmdShow)
{
sqlite3 *pdb;
int rc;

for (int n = 0; n < 1; n  )
{
rc = sqlite3_open("\\test.db3", &pdb);
if (rc) break;
rc = sqlite3_exec(pdb, "select * from testcase", 0, 0, 0);
if (rc) break;
rc = sqlite3_close(pdb);
if (rc) break;
}
return 0;
}

On the first call to sqlite3_exec(), available program memory dropped from 
an initial 10.5mb to 8.45mb. However, once that was done, available memory 
remained rock solid for the duration of the loop at a constant 8.45mb and no 
non-zero error codes were ever returned.

Robert




El día del padre está cerca... ¿Ya tienes el regalo? 
http://buscawanadoo.es/search?type=pref&destino=web&origen=homespot&buscar=Regalos%20Día%20del%20Padre

[sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE

[[EMAIL PROTECTED]]

Hi Robert,
Thank you for your test. I have not test it on the emulator but in 4 different 
Windows CE devices i have at work (with different Windows CE OS versions) and 
it always give me the same memory leak result.
I will run your test tomorrow at work using the emulator and i will let you 
know the results.
Thank you again,
Eduardo

---Mensaje original---Ok, here's what I did ...

On the desktop I created a SQLite database with one table and inserted 
120,000 rows into it. I then copied it over to the CE emulator.

Then I ran the following code on the CE (Pocket PC 2003 SE) emulator:

#include 
#include 
#include "sqlite3.h"

int WINAPI _tWinMain(HINSTANCE hInst, HINSTANCE hPrev, LPTSTR pszCmdLine, 
int nCmdShow)
{
sqlite3 *pdb;
int rc;

for (int n = 0; n < 1; n  )
{
rc = sqlite3_open("\\test.db3", &pdb);
if (rc) break;
rc = sqlite3_exec(pdb, "select * from testcase", 0, 0, 0);
if (rc) break;
rc = sqlite3_close(pdb);
if (rc) break;
}
return 0;
}

On the first call to sqlite3_exec(), available program memory dropped from 
an initial 10.5mb to 8.45mb. However, once that was done, available memory 
remained rock solid for the duration of the loop at a constant 8.45mb and no 
non-zero error codes were ever returned.

Robert




El día del padre está cerca... ¿Ya tienes el regalo? 
http://buscawanadoo.es/search?type=pref&destino=web&origen=homespot&buscar=Regalos%20Día%20del%20Padre

[sqlite] RE:[sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE

[[EMAIL PROTECTED]]

Hi again Robert,
I think i forgot to tell you that to detect the memory leak you must not close 
the application testing program, the easiest way to detect the memory leak is:
create a simple MFC dialog project. Add 2 buttons, one for executing your 
testing program an another one for close the application from it.
If you execute your testing program and you monitorize the memory usage with 
the Remote performance monitor (from the tools menu at the EVC   4.0 
IDE->tools->Remote Performance Monitor->Select your device->press 
the   sign (add)->change combo Box selection to Memory->add a graphic 
line for memory load). You will notice that memory is only recovered once you 
execute the button to close the application.
I hope this might help you.
Thank you,
Eduardo

---Mensaje original---
Hi Robert,
Thank you for your test. I have not test it on the emulator but in 4 di
fferent Windows CE devices i have at work (with different Windows CE OS
versions) and it always give me the same memory leak result.
I will run your test tomorrow at work using the emulator and i will l
et you know the results.
Thank you again,
Eduardo
---Mensaje original---Ok, here's what I did ...

On the desktop I created a SQLite database with one table and inserted 

120,000 rows into it. I then copied it over to the CE emulator.

Then I ran the following code on the CE (Pocket PC 2003 SE) emulator:

#include 
#include 
#include "sqlite3.h"

int WINAPI _tWinMain(HINSTANCE hInst, HINSTANCE hPrev, LPTSTR pszCmdLin
e, 
int nCmdShow)
{
sqlite3 *pdb;
int rc;

for (int n = 0; n < 1; n )
{
rc = sqlite3_open("\\test.db3", &pdb);
if (rc) break;
rc = sqlite3_exec(pdb, "select * from testcase", 0, 0, 0);
if (rc) break;
rc = sqlite3_close(pdb);
if (rc) break;
}
return 0;
}

On the first call to sqlite3_exec(), available program memory dropped f
rom 
an initial 10.5mb to 8.45mb. However, once that was done, available mem
ory 
remained rock solid for the duration of the loop at a constant 8.45mb a
nd no 
non-zero error codes were ever returned.

Robert




El día del padre está cerca... ¿Ya tienes el regalo? http://busca
wanadoo.es/search?type=pref&destino=web&origen=homespot&buscar=
Regalos%20Día%20del%20Padre

El día del padre está cerca... ¿Ya tienes el regalo? 
http://buscawanadoo.es/search?type=pref&destino=web&origen=homespot&buscar=Regalos%20Día%20del%20Padre

[sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE

[[EMAIL PROTECTED]]

Hi Robert,
Thank you for your test. I have not test it on the emulator but in 4 different 
Windows CE devices i have at work (with different Windows CE OS versions) and 
it always give me the same memory leak result.
I will run your test tomorrow at work using the emulator and i will let you 
know the results.
Thank you again,
Eduardo
---Mensaje original---Ok, here's what I did ...

On the desktop I created a SQLite database with one table and inserted 
120,000 rows into it. I then copied it over to the CE emulator.

Then I ran the following code on the CE (Pocket PC 2003 SE) emulator:

#include 
#include 
#include "sqlite3.h"

int WINAPI _tWinMain(HINSTANCE hInst, HINSTANCE hPrev, LPTSTR pszCmdLine, 
int nCmdShow)
{
sqlite3 *pdb;
int rc;

for (int n = 0; n < 1; n  )
{
rc = sqlite3_open("\\test.db3", &pdb);
if (rc) break;
rc = sqlite3_exec(pdb, "select * from testcase", 0, 0, 0);
if (rc) break;
rc = sqlite3_close(pdb);
if (rc) break;
}
return 0;
}

On the first call to sqlite3_exec(), available program memory dropped from 
an initial 10.5mb to 8.45mb. However, once that was done, available memory 
remained rock solid for the duration of the loop at a constant 8.45mb and no 
non-zero error codes were ever returned.

Robert




El día del padre está cerca... ¿Ya tienes el regalo? 
http://buscawanadoo.es/search?type=pref&destino=web&origen=homespot&buscar=Regalos%20Día%20del%20Padre

Re: [sqlite] RE: SQLite memory leak on Windows CE

[Robert Simpson]

Ok, here's what I did ...

On the desktop I created a SQLite database with one table and inserted 
120,000 rows into it.  I then copied it over to the CE emulator.


Then I ran the following code on the CE (Pocket PC 2003 SE) emulator:

#include 
#include 
#include "sqlite3.h"

int WINAPI _tWinMain(HINSTANCE hInst, HINSTANCE hPrev, LPTSTR pszCmdLine, 
int nCmdShow)

{
 sqlite3 *pdb;
 int rc;

 for (int n = 0; n < 1; n++)
 {
   rc = sqlite3_open("\\test.db3", &pdb);
   if (rc) break;
   rc = sqlite3_exec(pdb, "select * from testcase", 0, 0, 0);
   if (rc) break;
   rc = sqlite3_close(pdb);
   if (rc) break;
 }
 return 0;
}

On the first call to sqlite3_exec(), available program memory dropped from 
an initial 10.5mb to 8.45mb.  However, once that was done, available memory 
remained rock solid for the duration of the loop at a constant 8.45mb and no 
non-zero error codes were ever returned.


Robert

Re: [sqlite] RE: SQLite memory leak on Windows CE

[Robert Simpson]

What method(s) are you using to detect the memory leak?

I'll try and test this out today and see if I can come up with anything.

Robert


- Original Message - 
From: <[EMAIL PROTECTED]>

To: 
Sent: Wednesday, March 15, 2006 6:18 AM
Subject: [sqlite] RE: SQLite memory leak on Windows CE



Hi again, i have tried your advice of changing line 192 at table.c and it 
didn´t work (it fires an exception). I do not think that the problem is 
related to sqlite3_get_table or sqlite3_free_table source code.
To make sure, i have test it with another simple program (see the source 
code below), that doesn´t make use of the sqlite_x_table stuff and Windows 
CE doesn´t free memory properly (memory only gets freed once the main 
program has exited). Please see the testing code i am using (it is quite 
simple)

[sqlite] RE: SQLite memory leak on Windows CE

[[EMAIL PROTECTED]]

Hi again, i have tried your advice of changing line 192 at table.c and it 
didn´t work (it fires an exception). I do not think that the problem is related 
to sqlite3_get_table or sqlite3_free_table source code.
To make sure, i have test it with another simple program (see the source code 
below), that doesn´t make use of the sqlite_x_table stuff and Windows CE 
doesn´t free memory properly (memory only gets freed once the main program has 
exited). Please see the testing code i am using (it is quite simple) 
void CMemoryMDlg::OnButton1() 
{   
   sqlite3 *db;
   char *zErrMsg = 0;
   int rc;
   rc = sqlite3_open("\\articulos.db", &db);
   if( rc ){
   MessageBox(_T("NO"),_T(""),MB_OK);
   sqlite3_free(zErrMsg);
   sqlite3_close(db);
   CDialog::OnCancel();
   }
   rc = sqlite3_exec(db, "select * from ARTIC ;", 0, 0, &zErrMsg);
   if( rc!=SQLITE_OK ){
   sqlite3_free(zErrMsg);
   MessageBox(_T("NO2"),_T(""),MB_OK);
   }
   sqlite3_free(zErrMsg);
   sqlite3_close(db);
}
Thank you in advance,
Eduardo Esteban
---Mensaje original---On March 14, 2006 01:49 pm, [EMAIL PROTECTED] wrote:
> Hello again,
> Thank for your answers:
> first of all thank you who pointed that i should use a close(db) at
> line 17 and call sqlite3_free_table(result) even if there was an
> error, but these solutions didn´t solve my problem (the execution
> code never reached that point because ret value was always
> SQLITE_OK). Do anyone of you work with SQLite on Windows CE?

No, I don't have win ce either.
However, I looked at the sqlite code and there may be a bug.

In SQLite3.3.4 go to:
"src\table.c" line 192 and change "azResult--;"
changed_from-
190 if( azResult ){
191 int i, n;
192 azResult--;
193 if( azResult==0 ) return;
194 n = (int)azResult[0];
changed_to-
190 if( azResult ){
191 int i, n;
192 azResult -= sizeof(char*);