Just in case what I was asking wasn't clear, I'd like to give an example. If you look at the definition of "LSM_CONFIG_MMAP", if it is true, the entire database file is memory mapped. If it's false, data is accessed using ordinary OS file read and write primitives.
But if you look at flag such as "LSM_CONFIG_AUTOCHECKPOINT", it's about how often the data is to be synced to disk. Well, if you set MMAP to true, does "sync'ing to disk" even an issue? It's always on disk because that's how memory-mapped files work. So does that mean that, when you LSM_CONFIG_MMAP, then the value of LSM_CONFIG_AUTOCHECKPOINT is ignored? Can you say the same for LSM_CONFIG_AUTOFLUSH and LSM_CONFIG_AUTOWORK and LSM_CONFIG_SAFETY and LSM_CONFIG_USE_LOG. All of these seem moot since they have to do with controlling how often things are written to disk, which seems unnecessary when LSM_CONFIG_MMAP is used. What would happen if I say LSM_SAFETY_OFF and LSM_CONFIG_MMAP? Note that, the database is not used after the process terminates. The database only exists while the process is running. If the main process starts, and it finds a database file, it will delete it and create a new one. The only reason the database exists is to manage more data during the execution of the process than can fit in physical memory. -----Original Message----- From: sqlite-users-bounces at mailinglists.sqlite.org [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On Behalf Of Kasajian, Kenneth Sent: Thursday, May 19, 2016 8:13 PM To: sqlite-users at mailinglists.sqlite.org Subject: [sqlite] Using LSM and Memory Mapped files -- Question about LSM_CONFIG_MMAP I am interested in using the LSM (https://www.sqlite.org/src4/doc/trunk/www/lsmusr.wiki) The characteristics of my application is that I would like to enable the memory-mapped file feature so that I can offload data in memory to disk temporarily. I don't care about data-recover after my application shuts down or it crashes. In fact, the application will create a new database each time it starts up. The reason to do this is to be able to address hundreds of gigs of data in a key/value store method. We would it put it in RAM but that would require a lot of RAM. By putting the data on disk, we can work with very large sets of data. I care a lot about performance. If I had the RAM, all of this would be in RAM (non-persistent and transien) My question is, if I have LSM_CONFIG_MMAP enabled, and on a 64-bit system I believe it's enabled by default, what happens if I also turn off LSM_CONFIG_SAFETY and LSM_CONFIG_USE_LOG. Would that not make things a lot faster? I don't need data to be written to disk right away.. In fact, I would be okay if data were only written to disk when there the memory that is occupied by the data has to be reused - because persistence is not a factor. Anyone use LSM for this type of a use case? ________________________________ *** Confidentiality Notice: This e-mail, including any associated or attached files, is intended solely for the individual or entity to which it is addressed. This e-mail is confidential and may well also be legally privileged. If you have received it in error, you are on notice of its status. Please notify the sender immediately by reply e-mail and then delete this message from your system. Please do not copy it or use it for any purposes, or disclose its contents to any other person. _______________________________________________ sqlite-users mailing list sqlite-users at mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. ______________________________________________________________________ *** Confidentiality Notice: This e-mail, including any associated or attached files, is intended solely for the individual or entity to which it is addressed. This e-mail is confidential and may well also be legally privileged. If you have received it in error, you are on notice of its status. Please notify the sender immediately by reply e-mail and then delete this message from your system. Please do not copy it or use it for any purposes, or disclose its contents to any other person.
