[ https://issues.apache.org/jira/browse/CARBONDATA-484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ben Manes updated CARBONDATA-484: --------------------------------- Comment: was deleted (was: You might consider using [TinyLFU|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html] instead of LRU. This can provide significantly higher hit rate by approximating the historic frequencies.) > Implement LRU cache for B-Tree > ------------------------------- > > Key: CARBONDATA-484 > URL: https://issues.apache.org/jira/browse/CARBONDATA-484 > Project: CarbonData > Issue Type: New Feature > Reporter: Mohammad Shahid Khan > Assignee: Mohammad Shahid Khan > Attachments: B-Tree LRU Cache.pdf > > Time Spent: 20m > Remaining Estimate: 0h > > LRU Cache for B-Tree is proposed to ensure to avoid out memory, when too > many number of tables exits and all are not frequently used. > Problem: > CarbonData is maintaining two level of B-Tree cache, one at the driver level > and another at executor level. Currently CarbonData has the mechanism to > invalidate the segments and blocks cache for the invalid table segments, but > there is no eviction policy for the unused cached object. So the instance at > which complete memory is utilized then the system will not be able to process > any new requests. > Solution: > In the cache maintained at the driver level and at the executor there must be > objects in cache currently not in use. Therefore system should have the > mechanism to below mechanism. > 1. Set the max memory limit till which objects could be hold in the > memory. > 2. When configured memory limit reached then identify the cached > objects currently not in use so that the required memory could be freed > without impacting the existing process. > 3. Eviction should be done only till the required memory is not meet. > For details please refer to attachments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)