Re: Re: lmdb memory usage when writing lots of data with memorymap option

2014-01-16 Thread Luc Vlaming
Hi, First of all, sorry for not being clear. I meant resident memory usage when mentioning the 5GB memory usage. For clarity here are the pmap outputs pmap with memorymap: Address Kbytes RSS Dirty Mode Mapping . total kB52687820 5153964 16992 pmap without

Re: lmdb memory usage when writing lots of data with memorymap option

2014-01-16 Thread Howard Chu
Luc Vlaming wrote: Hi, If it was simply writing to the memory map, shouldn't memory usage decrease as soon as everything is written? Why? Sounds to me like you don't understand how demand paging works in a virtual memory system. The key rule here - if there is no demand, then there is no

Re: Antw: Re: lmdb memory usage when writing lots of data with memorymap option

2014-01-16 Thread Howard Chu
Ulrich Windl wrote: Off-topic: I can remember a statement of the late 80ies where a programmer claimed the 32-bit address space is so large that one does not have to care about garbage collection in virtual address space; just use new addresses. I think even with 64 bit one should always try not

Re: Antw: Re: lmdb memory usage when writing lots of data with memorymap option

2014-01-16 Thread Ulrich Windl
Howard Chu h...@symas.com schrieb am 16.01.2014 um 11:03 in Nachricht 52d7ae78.7090...@symas.com: Ulrich Windl wrote: Off-topic: I can remember a statement of the late 80ies where a programmer claimed the 32-bit address space is so large that one does not have to care about garbage collection

Re: Antw: Re: lmdb memory usage when writing lots of data with memorymap option

2014-01-16 Thread Luc Vlaming
Hi, Thanks for the explanations Regards, Luc On Thu, Jan 16, 2014 at 12:24 PM, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: Howard Chu h...@symas.com schrieb am 16.01.2014 um 11:03 in Nachricht 52d7ae78.7090...@symas.com: Ulrich Windl wrote: Off-topic: I can remember a

lmdb memory usage when writing lots of data with memorymap option

2014-01-15 Thread Luc Vlaming
Hi, Currently I am creating support for using LMDB as a new storage backend for one of our products. At the moment I am testing import bulk data into lmdb using transactions that span a single record of 10MB. The total db size afterwards is 5GB. I also tested with records of 1MB. I noticed a

Re: lmdb memory usage when writing lots of data with memorymap option

2014-01-15 Thread Howard Chu
Luc Vlaming wrote: Hi, Currently I am creating support for using LMDB as a new storage backend for one of our products. At the moment I am testing import bulk data into lmdb using transactions that span a single record of 10MB. The total db size afterwards is 5GB. I also tested with records of

Re: lmdb memory usage when writing lots of data with memorymap option

2014-01-15 Thread Luc Vlaming
Hi, If it was simply writing to the memory map, shouldn't memory usage decrease as soon as everything is written? The memory usage continuous to be high for as long as the database is open, even if the program just waits afterwards. Is that to be expected as well? Because that would mean that the

Antw: Re: lmdb memory usage when writing lots of data with memorymap option

2014-01-15 Thread Ulrich Windl
Howard Chu h...@symas.com schrieb am 15.01.2014 um 23:10 in Nachricht 52d7076b.3020...@symas.com: Luc Vlaming wrote: Hi, Currently I am creating support for using LMDB as a new storage backend for one of our products. At the moment I am testing import bulk data into lmdb using transactions