>From: Ideriha, Takeshi [mailto:ideriha.take...@jp.fujitsu.com] >Does this result show that hard-limit size option with memory accounting >doesn't harm >to usual users who disable hard limit size option?
Hi, I've implemented relation cache size limitation with LRU list and built-in memory context size account. And I'll share some results coupled with a quick recap of catcache so that we can resume discussion if needed though relation cache bloat was also discussed in this thread but right now it's pending and catcache feature is not fixed. But a variety of information could be good I believe. Regarding catcache it seems to me recent Horiguchi san posts shows a pretty detailed stats including comparison LRU overhead and full scan of hash table. According to the result, lru overhead seems small but for simplicity this thread go without LRU. https://www.postgresql.org/message-id/20190404.215255.09756748.horiguchi.kyotaro%40lab.ntt.co.jp When there was hard limit of catcach, there was built-in memory context size accounting machinery. I checked the overhead of memory accounting, and when repeating palloc and pfree of 800 byte area many times it was 4% down on the other hand in case of 32768 byte there seems no overhead. https://www.postgresql.org/message-id/4E72940DA2BF16479384A86D54D0988A6F44564E%40G01JPEXMBKW04 Regarding relcache hard limit (relation_cache_max_size), most of the architecture was similar as catcache one with LRU list except memory accounting. Relcaches are managed by LRU list. To prune LRU cache, we need to know overall relcache sizes including objects pointed by relcache like 'index info'. So in this patch relcache objects are allocated under RelCacheMemoryContext, which is child of CacheMemoryContext. Objects pointed by relcache is allocated under child context of RelCacheMemoryContext. In built-in size accounting, if memoryContext is set to collect "group(family) size", you can get context size including child easily. I ran two experiments: A) One is pgbench using Tomas's script he posted while ago, which is randomly select 1 from many tables. https://www.postgresql.org/message-id/4E72940DA2BF16479384A86D54D0988A6F426207%40G01JPEXMBKW04 B) The other is to check memory context account overhead using the same method. https://www.postgresql.org/message-id/4E72940DA2BF16479384A86D54D0988A6F44564E%40G01JPEXMBKW04 A) randomly select 1 from many tables Results are average of 5 times each. number of tables | 100 |1000 |10000 ----------------------------------------------------------- TPS (master) |11105 |10815 |8915 TPS (patch; limit feature off) |11254 (+1%) |11176 (+3%) |9242 (+4%) TPS (patch: limit on with 1MB) |11317 (+2%) |10491 (-3%) |7380 (-17%) The results are noisy but it seems overhead of LRU and memory accounting is small when turning off the relcache limit feature. When turning on the limit feature, after exceeding the limit it drops 17%, which is no surprise. B) Repeat palloc/pfree "With group accounting" means that account test context and its child context with built-in accounting using "palloc_bench_family()". The other one is that using palloc_bench(). Please see palloc_bench.gz. [Size=32768, iter=1,000,000] Master | 59.97 ms Master with group account | 59.57 ms patched |67.23 ms patched with family |68.81 ms It seems that overhead seems large in this patch. So it needs more inspection this area. regards, Takeshi Ideriha
palloc_bench.gz
Description: palloc_bench.gz
v15-0001-4-Add-dlist_move_tail.patch
Description: v15-0001-4-Add-dlist_move_tail.patch
binj9L3vPCahv.bin
Description: v15-0002-4-ideriha-Memory-consumption-report-reature-of-memorycontext.patch
v15-0004-4-ideriha-Remove-RelCache-Entries.patch
Description: v15-0004-4-ideriha-Remove-RelCache-Entries.patch
v15-0003-4-ideriha-Remove-CatCache-Entries.patch
Description: v15-0003-4-ideriha-Remove-CatCache-Entries.patch
