On Thu, Jul 9, 2020 at 12:03 AM Chun-Yu Shei <cs...@google.com> wrote:
> Awesome! Here's a patch that adds @lru_cache to use_reduce, vercmp, and > catpkgsplit. use_reduce was split into 2 functions, with the outer one > converting lists/sets to tuples so they can be hashed and creating a > copy of the returned list (since the caller seems to modify it > sometimes). I tried to select cache sizes that minimized memory use > increase, > while still providing about the same speedup compared to a cache with > unbounded size. "emerge -uDvpU --with-bdeps=y @world" runtime decreases > from 44.32s -> 29.94s -- a 48% speedup, while the maximum value of the > RES column in htop increases from 280 MB -> 290 MB. > > "emerge -ep @world" time slightly decreases from 18.77s -> 17.93, while > max observed RES value actually decreases from 228 MB -> 214 MB (similar > values observed across a few before/after runs). > > Here are the cache hit stats, max observed RES memory, and runtime in > seconds for various sizes in the update case. Caching for each > function was tested independently (only 1 function with caching enabled > at a time): > > catpkgsplit: > CacheInfo(hits=1222233, misses=21419, maxsize=None, currsize=21419) > 270 MB > 39.217 > > CacheInfo(hits=1218900, misses=24905, maxsize=10000, currsize=10000) > 271 MB > 39.112 > > CacheInfo(hits=1212675, misses=31022, maxsize=5000, currsize=5000) > 271 MB > 39.217 > > CacheInfo(hits=1207879, misses=35878, maxsize=2500, currsize=2500) > 269 MB > 39.438 > > CacheInfo(hits=1199402, misses=44250, maxsize=1000, currsize=1000) > 271 MB > 39.348 > > CacheInfo(hits=1149150, misses=94610, maxsize=100, currsize=100) > 271 MB > 39.487 > > > use_reduce: > CacheInfo(hits=45326, misses=18660, maxsize=None, currsize=18561) > 407 MB > 35.77 > > CacheInfo(hits=45186, misses=18800, maxsize=10000, currsize=10000) > 353 MB > 35.52 > > CacheInfo(hits=44977, misses=19009, maxsize=5000, currsize=5000) > 335 MB > 35.31 > > CacheInfo(hits=44691, misses=19295, maxsize=2500, currsize=2500) > 318 MB > 35.85 > > CacheInfo(hits=44178, misses=19808, maxsize=1000, currsize=1000) > 301 MB > 36.39 > > CacheInfo(hits=41211, misses=22775, maxsize=100, currsize=100) > 299 MB > 37.175 > > > I didn't bother collecting detailed stats for vercmp, since the > inputs/outputs are quite small and don't cause much memory increase. > Please let me know if there are any other suggestions/improvements (and > thanks Sid for the lru_cache suggestion!). > I don't see a patch attached; can you link to it? -A > > Thanks, > Chun-Yu > > > >
