@manish I looked at the code of CarbonLRUCache, and I dont think we have actual implementation of LRU for cache. I am planning to make it actual Least-Recently-Used caching mechanism. If we make it actual LRU cache, the problem of stale elements in the cache should be resolved, because, they wont be accessed for some time and untimately removed from the cache when some other entries are being entered in the cache and cache is full.
Other thing I could think of is Counter-Based caching mechanism. What should be it, the LRU based caching or counter based caching? Ok, if DDLs does not look a good idea for this, we can also use CarbonCLI also. What do you think on this? @Sujith 1. Yes, as soon as the discussion reaches a conslusion, what DDLs to support and what to omit, I will share a design document. 3. Yes, drop table automatically clears the cache for the table. 4. Yes, it may happen, but the estimation is only to give the user a rough idea of how much memory the table will acquire in cache. User will accordingly configure the cache size, with some slack. Regards Naman Rastogi Technical Lead - BigData Kernel Huawei Technologies India Pvt. Ltd. On Tue, Feb 19, 2019 at 11:15 AM manish gupta <tomanishgupt...@gmail.com> wrote: > Hi Naman > > +1 for point 1, 2 and 6. > -1 for point 3, 4 and 5. > > 1. For point 1, 2 --> Add a design doc to mention all those things that > will be considered for caching while displaying the caching size. > 2. For point 3, 4 --> I feel that cleaning of cache should be an internal > thing and not exposed to the user. This might also suppress any bugs that > are there while cleaning the cache at the time of dropping the table. You > can think of stale cache clean up through a separate thread which checks > for stale cache clean up at intervals or you can try to integrate the > functionality with Clean DDL command. > 3. For point 5 --> We should think of introducing a command to collect the > System statistics something like Spark and from there we should calculate > the memory requirements instead of exposing a DDL specifically for cache > calculations. > > Regards > Manish Gupta > > On Tue, Feb 19, 2019 at 7:28 AM dhatchayani <dhatcha.offic...@gmail.com> > wrote: > > > Hi Naman, > > > > This will be very much useful for the users to control the cache_size and > > the utilization of cache. > > > > Please clarify me the below point. > > > > Dynamic "max cache size" configuration should be supported? > > "carbon.max.driver.lru.cache.size" is a system level configuration > whereas > > dynamic property is the session level property. We can support the > > dynamically SET for which the purpose of the property still holds good to > > the system. I think in this case, it does not hold well to the system. > > > > Thanks, > > Dhatchayani > > > > > > > > -- > > Sent from: > > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/ > > >
