Re: [sqlite] How to how many memory is used by sqlite
pierr wrote: > > > pierr wrote: >> >> >> D. Richard Hipp wrote: >>> >>> >>> On Jun 15, 2009, at 9:04 AM, pierr wrote: >>> Hi all, I am using an memory type sqlite database in our embedded application. After the applicate run a while ,the system will run ourt of out of memory. So, is there a way to know how many memory is used by sqlite? That would include the memory for page cache , any other other internal memory used the sqlite ,as well as the memory for database itself. Thanks for your help. >>> >>> http://www.sqlite.org/c3ref/memory_highwater.html >>> http://www.sqlite.org/malloc.html >>> >>> >> >> By using the memory_highwater api above , I found that most of the memory >> is used for page cache. By limiting the cache size , I can have a good >> control of how many heap memory will be used. >> >> INSERT Record NumbersPAGE SiZE CACHE SIZE HIGH WATER >> MARK >> (110bytes per record) >> >> 10001024 20 >> 98,256 >> 10001024 2000 >> 271,040 >> 80001024 2000 >> 1,562,144 >> 80001024 20 >> 99,200 >> >> And as Simon suggest ,when I close the connection, the memory used by >> sqlite will be Zero. >> >> I am more clear about how sqlite use memory now , however ,as I am using >> memory database ,I also cares about the database size. That is what is >> proportion of database size to the raw data size. >> >> I insert a record of 112 bytes 10,1000,and 1 times repectively and >> here list the output database size. For The last case ,it has a propotion >> about 1.45. What cost the extra 0.45 here beside the btree? >> >> 10240 2009-06-17 14:38 /etc/eit.flash.sqlite.10 >> 171008 2009-06-17 14:32 /etc/eit.flash.sqlite.1000 >> 1634304 2009-06-17 14:42 /etc/eit.flash.sqlite.1 >> >> (I am using the Flash database instead of the memory data here as I think >> the result should be the same.) >> >> > > > I was wrong here. > > In my application (Not the test enviroment mentiond above), after the > sqlite eating up more than 8M bytes in the memory , I dumpded the database > to the Flash but it is only 360K which is very reasonable for my > application. The database was configured as cache_szie = 20 > ,page_size=1024 ,so the page cache would not take too much memory ,it > should be less than 100K according to above measurement. > > So, for the non-memory based data base ,the total storage needed is : > generated database size + memory used by sqlite3 internal. > > But for the memory database, the total storage needed will be much bigger > than that. What will eat the extra buck of memory here? I can think of > the journey file ,but it should be K level memory.. > > Thanks for help me out. > I made a mistake . Before insering to the database , I did a query but fogot to finalize the statement, which ate lots of memory. By add the finalize statement, the memory comsumption is reasonable now. -- View this message in context: http://www.nabble.com/How-to-know-how-many-memory-is-used-by-sqlite-tp24034261p24091514.html Sent from the SQLite mailing list archive at Nabble.com. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] How to how many memory is used by sqlite
pierr wrote: > > > D. Richard Hipp wrote: >> >> >> On Jun 15, 2009, at 9:04 AM, pierr wrote: >> >>> >>> Hi all, >>> I am using an memory type sqlite database in our embedded >>> application. After the applicate run a while ,the system will run >>> ourt of >>> out of memory. So, is there a way to know how many memory is used by >>> sqlite? >>> That would include the memory for page cache , any other other >>> internal >>> memory used the sqlite ,as well as the memory for database itself. >>> Thanks >>> for your help. >> >> http://www.sqlite.org/c3ref/memory_highwater.html >> http://www.sqlite.org/malloc.html >> >> > > By using the memory_highwater api above , I found that most of the memory > is used for page cache. By limiting the cache size , I can have a good > control of how many heap memory will be used. > > INSERT Record NumbersPAGE SiZE CACHE SIZE HIGH WATER > MARK > (110bytes per record) > > 10001024 20 > 98,256 > 10001024 2000 > 271,040 > 80001024 2000 > 1,562,144 > 80001024 20 > 99,200 > > And as Simon suggest ,when I close the connection, the memory used by > sqlite will be Zero. > > I am more clear about how sqlite use memory now , however ,as I am using > memory database ,I also cares about the database size. That is what is > proportion of database size to the raw data size. > > I insert a record of 112 bytes 10,1000,and 1 times repectively and > here list the output database size. For The last case ,it has a propotion > about 1.45. What cost the extra 0.45 here beside the btree? > > 10240 2009-06-17 14:38 /etc/eit.flash.sqlite.10 > 171008 2009-06-17 14:32 /etc/eit.flash.sqlite.1000 > 1634304 2009-06-17 14:42 /etc/eit.flash.sqlite.1 > > (I am using the Flash database instead of the memory data here as I think > the result should be the same.) > > I was wrong here. In my application (Not the test enviroment mentiond above), after the sqlite eating up more than 8M bytes in the memory , I dumpded the database to the Flash but it is only 360K which is very reasonable for my application. The database was configured as cache_szie = 20 ,page_size=1024 ,so the page cache would not take too much memory ,it should be less than 100K according to above measurement. So, for the non-memory based data base ,the total storage needed is : generated database size + memory used by sqlite3 internal. But for the memory database, the total storage needed will be much bigger than that. What will eat the extra buck of memory here? I can think of the journey file ,but it should be K level memory.. Thanks for help me out. -- View this message in context: http://www.nabble.com/How-to-how-many-memory-is-used-by-sqlite-tp24034261p24073060.html Sent from the SQLite mailing list archive at Nabble.com. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] How to how many memory is used by sqlite
D. Richard Hipp wrote: > > > On Jun 15, 2009, at 9:04 AM, pierr wrote: > >> >> Hi all, >> I am using an memory type sqlite database in our embedded >> application. After the applicate run a while ,the system will run >> ourt of >> out of memory. So, is there a way to know how many memory is used by >> sqlite? >> That would include the memory for page cache , any other other >> internal >> memory used the sqlite ,as well as the memory for database itself. >> Thanks >> for your help. > > http://www.sqlite.org/c3ref/memory_highwater.html > http://www.sqlite.org/malloc.html > > By using the memory_highwater api above , I found that most of the memory is used for page cache. By limiting the cache size , I can have a good control of how many heap memory will be used. INSERT Record NumbersPAGE SiZE CACHE SIZE HIGH WATER MARK (110bytes per record) 10001024 20 98,256 10001024 2000 271,040 80001024 2000 1,562,144 80001024 20 99,200 And as Simon suggest ,when I close the connection, the memory used by sqlite will be Zero. I am more clear about how sqlite use memory now , however ,as I am using memory database ,I also cares about the database size. That is what is proportion of database size to the raw data size. I insert a record of 112 bytes 10,1000,and 1 times repectively and here list the output database size. For The last case ,it has a propotion about 1.45. What cost the extra 0.45 here beside the btree? 10240 2009-06-17 14:38 /etc/eit.flash.sqlite.10 171008 2009-06-17 14:32 /etc/eit.flash.sqlite.1000 1634304 2009-06-17 14:42 /etc/eit.flash.sqlite.1 (I am using the Flash database instead of the memory data here as I think the result should be the same.) Here are the table structure. For the 112 bytes in each record , 100 bytes goes to the event_name field. CREATE TABLE IF NOT EXISTS tblEvent_basic( sguid INT, service_id INT, event_id INT, start_time VARCHAR(5), duration VARCHAR(3), running_status INT, free_ca_mode INT, event_name VARCHAR(256), text VARCHAR(256)) Thanks for your comment. -- View this message in context: http://www.nabble.com/How-to-how-many-memory-is-used-by-sqlite-tp24034261p24068533.html Sent from the SQLite mailing list archive at Nabble.com. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] How to how many memory is used by sqlite
On Jun 15, 2009, at 9:04 AM, pierr wrote: > > Hi all, > I am using an memory type sqlite database in our embedded > application. After the applicate run a while ,the system will run > ourt of > out of memory. So, is there a way to know how many memory is used by > sqlite? > That would include the memory for page cache , any other other > internal > memory used the sqlite ,as well as the memory for database itself. > Thanks > for your help. http://www.sqlite.org/c3ref/memory_highwater.html http://www.sqlite.org/malloc.html D. Richard Hipp d...@hwaci.com ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] How to how many memory is used by sqlite
On 15 Jun 2009, at 2:04pm, pierr wrote: > I am using an memory type sqlite database in our embedded > application. After the applicate run a while ,the system will run > ourt of > out of memory. Does the memory usage grow slowly all the time, or does it get bigger very suddenly ? Test this: at some point when the memory usage is big, close the database and reopen it again. Do you suddenly recover lots of memory ? Simon. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
