Kalyani Tummala uttered:
I am planning to use sqlite as a database for storing and retrieving media data of about 5-10k records in a device whose main memory is extremely small. A sequence of insert statements increasing the heap usage to nearly 70K(almost saturating point) which is crashing my application. I want to restrict this to 30K.I tried closing database and reopen after some inserts but of no use. I have observed that, when I open the database with about 1K to 2K records in it, inserts and updates take more heap and also gradually increase than a a database with less than 1k records in it.
When updating the database, SQLite will keep a bitmap representing modified pages in memory, so as to manage the rollback journal. Therefore, making your minimum pages size smaller will now require more bits to track all the potentially modified pages in the database file. Instead, using the stock SQLite parameters, increase the page size and reduce the number of buffers. Increasing the page size will reduce the number of pages being tracked, as well as increasing the number of rows in each page. But, depending on how big your database is, this may not be a significant amount of memory. How big is a typical database?
My objective is to reduce the peak heap usage during inserts, updates and also deletes with little or no performance degradation.
You'll get a certain amount of slowdown when reducing the number of available buffers, as you'll be spilling dirty buffers to disk more often.
Please suggest me if I can do anything to do so.
You could try profiling memory usage before randomly changing parameters. In the source, perhaps on a test machine rather than the target platform, replace sqliteMalloc with a macro to log memory allocation, along with source file and line number information, something like what is done now with memory debugging turned on (see src/malloc.c and src/sqliteInt.h).
Thank you in advance Kalyani -----Original Message----- From: John Stanton [mailto:[EMAIL PROTECTED] Sent: Tuesday, May 29, 2007 6:51 PM To: sqlite-users@sqlite.org Subject: Re: [sqlite] How to restrict the peak heap usage during multiple inserts and updates? Since you are only using part of Sqlite have you considered using a much smaller footprint storage system which only implements the functions you are using? Kalyani Tummala wrote:Hi joe, Thanks for your response. In order to reduce the footprint size, I have bypassed parsercompletelyand using byte codes directly as my schema and queries are almost compile time fixed. Hence I am not using sqlite3_prepare(). The following is the schema and inserts I am using. CREATE TABLE OBJECT( PUOI INTEGER PRIMARY KEY, Storage_Id INTEGER, Object_Format INTEGER, Protection_Status INTEGER, Object_Size INTEGER, Parent_Object INTEGER, Non_Consumable INTEGER, Object_file_name TEXT, Name TEXT, File_Path TEXT ); CREATE TABLE AUDIO( PUOI INTEGER PRIMARY KEY, Use_Count INTEGER, Audio_Bit_Rate INTEGER, Sample_Rate INTEGER, Audio_Codec_Type INTEGER, Number_of_Channels INTEGER, Track INTEGER, Artist TEXT, Title TEXT, Genre TEXT, Album_Name TEXT, File_Path TEXT ); INSERT INTO OBJECT VALUES ( 7, 65537, 12297, 0, 475805, 6, 0, 'ANJANEYASTOTRAM.mp3', NULL, 'C:\\MTPSim\\Store0\\Music\\Artist\\Album\\ANJANEYASTOTRAM.mp3' ); INSERT INTO AUDIO VALUES ( 7, 6, 144100, 0, 0, 0, 6, NULL, NULL, NULL, NULL, 'C:\\MTPSim\\Store0\\Music\\Artist\\Album\\ANJANEYASTOTRAM.mp3' ); INSERT INTO OBJECT VALUES ( 8, 65537, 12297, 0, 387406, 6, 0, 'BHADRAM.mp3', NULL, 'C:\\MTPSim\\Store0\\Music\\Artist\\Album\\BHADRAM.mp3' ); INSERT INTO AUDIO VALUES ( 8, 6, 144100, 0, 0, 0, 6, NULL, NULL, NULL, NULL, 'C:\\MTPSim\\Store0\\Music\\Artist\\Album\\BHADRAM.mp3' ); Warm regards Kalyani -----Original Message----- From: Joe Wilson [mailto:[EMAIL PROTECTED] Sent: Tuesday, May 29, 2007 9:42 AM To: sqlite-users@sqlite.org Subject: Re: [sqlite] How to restrict the peak heap usage during multiple inserts and updates?I am working at porting sqlite ( ver 3.3.8 ) on an embedded devicewithextremely low main memory. I tried running select queries on the tables( with about 2k recordseachhaving about 5 strings) and they do well within 20kB of runtime heap usage. But, when I try new insertions, the heap usage grows tremendously(about70 kB at peak).Perhaps preparing the statements (sqlite3_prepare) might decrease RAM use somewhat. Can you post an example of your schema and these insert statements?____________________________________________________________________________________Choose the right car based on your needs. Check outYahoo!Autos new Car Finder tool. http://autos.yahoo.com/carfinder/----------------------------------------------------------------------------- To unsubscribe, send email to [EMAIL PROTECTED]----------------------------------------------------------------------------- ********************************************************************** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify [EMAIL PROTECTED] **********************************************************************------------------------------------------------------------------------ -----To unsubscribe, send email to [EMAIL PROTECTED]------------------------------------------------------------------------ ----------------------------------------------------------------------------- ----- To unsubscribe, send email to [EMAIL PROTECTED] ------------------------------------------------------------------------ ----- ----------------------------------------------------------------------------- To unsubscribe, send email to [EMAIL PROTECTED] -----------------------------------------------------------------------------
--
/"\
\ / ASCII RIBBON CAMPAIGN - AGAINST HTML MAIL
X - AGAINST MS ATTACHMENTS
/ \
-----------------------------------------------------------------------------
To unsubscribe, send email to [EMAIL PROTECTED]
-----------------------------------------------------------------------------
