On 11/25/2015 02:27 PM, Howard Chu wrote: > Dominik Taborsky wrote: >> Hello, >> >> I've tried stress-testing LMDB in our use case and I've discovered something >> I consider a bug. >> >> In our use case we need to keep inserting to the DB until it's full, flush >> it and repeat. We've designed the insert function to fail if there are not >> enough headroom pages for safe delete / DB drop. During the tests the DB >> works correctly for some amount of repeats, but then it suddenly refuses to >> insert anything. It seems LMDB doesn't clear or reuse all of its pages. I >> would like to get anyone's opinion on this. >> >> I have attached the source file containing the test. The DB size is set to >> 20MB, the inserted values have by default 25kB. In the first run the DB >> accepts 365 inserts, which decreases over time and stabilizes at 280. After >> 18 repeats the DB does not accept any inserts at all: >> ok 1 - pass #1 fillup run (365 inserts) >> ok 2 - pass #2 fillup run (300 inserts) >> ok 3 - pass #3 fillup run (286 inserts) >> ok 4 - pass #4 fillup run (283 inserts) >> ok 5 - pass #5 fillup run (281 inserts) >> ok 6 - pass #6 fillup run (280 inserts) >> ok 7 - pass #7 fillup run (280 inserts) >> ok 8 - pass #8 fillup run (280 inserts) >> ok 9 - pass #9 fillup run (280 inserts) >> ok 10 - pass #10 fillup run (280 inserts) >> ok 11 - pass #11 fillup run (280 inserts) >> ok 12 - pass #12 fillup run (280 inserts) >> ok 13 - pass #13 fillup run (280 inserts) >> ok 14 - pass #14 fillup run (280 inserts) >> ok 15 - pass #15 fillup run (280 inserts) >> ok 16 - pass #16 fillup run (280 inserts) >> ok 17 - pass #17 fillup run (280 inserts) >> ok 18 - pass #18 fillup run (108 inserts) >> not ok 19 - pass #19 fillup run (0 inserts) >> not ok 20 - pass #20 fillup run (0 inserts) >> >> The test is run 4 times with different approaches that we thought could have >> had some impact. These approaches combine opening the DB per insert or per >> single fillup, and reopening the DB just for the DB drop. In this case these >> options have no effect and the results are the same, but originally a unit >> test that this originates from had some differences so I kept them in. >> >> Note that I also tried calling mdb_drop with "1" as the last parameter, but >> it had no effect on the result. >> >> I am linking against the latest LMDB from sources. >> >> I also tried adding a constant number of pages to the reserve before the >> insert function fails. This only changes the amount of inserted items but >> doesn't change the functionality. >> >> If you compile the program you can supply it with an argument to change the >> default size of the item value size. Lower values break the DB sooner (20kB >> requires only 12 repeated fillups and drops). >> >> If anyone has any idea what might be the issue, please let me know. I know >> deleting the DB from filesystem and creating a new one would work, but >> that's a hack, not a fix. > > Your calculation of max_pages vs used_pages is totally bogus. There's nothing > wrong with LMDB. >
Thanks for checking it out, Howard, but can you please advise on how to compute a safe buffer then?
