Re: [Discussion] DDLs to operate on CarbonLRUCache

2019-02-20 Thread litao
Hi Naman I think in the current version capabilities, it is necessary to do some ddl support. +1 for point 1,2 In Point 1 or 2, is it possible output a list of information for all tables,to use limit as a filter.In this way, user can see the whole pciture of the cache and guide subsequent operatio

Re: [Discussion] DDLs to operate on CarbonLRUCache

2019-02-20 Thread Naman Rastogi
@manish I looked at the code of CarbonLRUCache, and I dont think we have actual implementation of LRU for cache. I am planning to make it actual Least-Recently-Used caching mechanism. If we make it actual LRU cache, the problem of stale elements in the cache should be resolved, because, they wont

Re: [Discussion] DDLs to operate on CarbonLRUCache

2019-02-20 Thread Naman Rastogi
@akash 1. No sync issues will be there are the operation will be atomic, only one operation is allowed on lruCache at a particular point of time (will be taken care in code). 2. It will act as it is expected to, concurrent operation on cache will be blocked. 3 & 4. We will do it the same way we

Re: [Discussion] DDLs to operate on CarbonLRUCache

2019-02-20 Thread Naman Rastogi
@dhatchayani If we are to support estimation of cache size required for a particular table, we have to make max_cache_size dynamic, and change the cache size dynamically, otherwise user will have to restart the JDBC server all over again, which does not seem like a good idea from the perspective o

Re: [Discussion] DDLs to operate on CarbonLRUCache

2019-02-19 Thread akashrn5
Hi naman, Thanks for proposing the feature. Looks really helpful from user and developer perspective. Basically needed the design document, so that all the doubts would be cleared. 1. Basicaly how you are going to handle the sync issues like, multiple queries with drop and show cache. are you g

Re: [Discussion] DDLs to operate on CarbonLRUCache

2019-02-19 Thread sujith chacko
Hi Naman, Thanks for proposing this feature, seems to be pretty interesting feature, few points i want to bring up here 1) I think we shall require a detailed design for this feature where all the DDL's you are going to expose shall be clearly mentioned as frequent updation of DDL's are not rec

Re: [Discussion] DDLs to operate on CarbonLRUCache

2019-02-19 Thread David CaiQiang
+1 for 5,6, after point 5 estimated the cache size, the point 6 can modify the configuration dynamically. +1 for 3,4: maybe need to add a lock to sync the concurrent operations. If it wants to release cache, it will not need to restart the driver Maybe we also need to check how to use these oper

Re: [Discussion] DDLs to operate on CarbonLRUCache

2019-02-18 Thread xuchuanyin
+1 for advices from manish -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: [Discussion] DDLs to operate on CarbonLRUCache

2019-02-18 Thread manish gupta
Hi Naman +1 for point 1, 2 and 6. -1 for point 3, 4 and 5. 1. For point 1, 2 --> Add a design doc to mention all those things that will be considered for caching while displaying the caching size. 2. For point 3, 4 --> I feel that cleaning of cache should be an internal thing and not exposed to

Re: [Discussion] DDLs to operate on CarbonLRUCache

2019-02-18 Thread dhatchayani
Hi Naman, This will be very much useful for the users to control the cache_size and the utilization of cache. Please clarify me the below point. Dynamic "max cache size" configuration should be supported? "carbon.max.driver.lru.cache.size" is a system level configuration whereas dynamic property

[Discussion] DDLs to operate on CarbonLRUCache

2019-02-18 Thread Naman Rastogi
Hi all, Currently carbon supports caching mechanism for Blocks/Blocklets. Even though it allows end user to set the Cache size, it is still very limited in functionality, and user arbitrarily chooses the carbon property *carbon.max.driver.lru.cache.size* where before launching the carbon session,