Hi all,
Currently carbon supports caching mechanism for Blocks/Blocklets. Even
though it allows end user to set the Cache size, it is still very
limited in functionality, and user arbitrarily chooses the carbon
property *carbon.max.driver.lru.cache.size* where before launching the
carbon session, he/she has no idea of how much cache should be set for
his/her requirement.
For this problem, I propose the following imporvements in carbon caching
mechanism.
1. Support DDL for showing current cache used per table.
2. Support DDL for showing current cache used for a particular table.
For these two points, QiangCai has already has a PR:
https://github.com/apache/carbondata/pull/3078
3. Support DDL for clearing all the entries cache.
This will look like:
CLEAN CACHE
4. Support DDL for clearing cache for a particular table.
This will clear all the entries in the cache which belong to a
particular table. This will look like
CLEAN CACHE FOR TABLE tablename
5. Support DDL to estimate required cache for a particular table.
As explained above, the user does not know beforehand how much cache
will be required for his/her current work. So this DDL will let the
user estimate how much cache will be required for a particular
table. For this we will launch a job and estimate the memory
required for all the blocks, and sum it up.
6. Dynamic "max cache size" configration
Suppose now the user knows required cache size he needs, but the
current system requires the user to set the
*carbon.max.driver.lru.cache.size* and restart the JDBC server for
it to take effect. For this I am suggesting to make the carbon
property *carbon.max.driver.lru.cache.size* dynamically configurable
which allows the user to change the max LRU cache size on the fly.
Any suggestion from the community is greatly appreciated.
Thanks
Regards
Naman Rastogi
Technical Lead - BigData Kernel
Huawei Technologies India Pvt. Ltd.
