@akash 1. No sync issues will be there are the operation will be atomic, only one operation is allowed on lruCache at a particular point of time (will be taken care in code).
2. It will act as it is expected to, concurrent operation on cache will be blocked. 3 & 4. We will do it the same way we do now. Summing up memory size for all the BlockDataMap -> summing up for all the DataMapRow. Just that we will distribute that among different executors. Regards Naman Rastogi Technical Lead - BigData Kernel Huawei Technologies India Pvt. Ltd. On Wed, Feb 20, 2019 at 7:51 PM Naman Rastogi <naman.rastogi...@gmail.com> wrote: > @dhatchayani > > If we are to support estimation of cache size required for a particular > table, we have to make max_cache_size dynamic, and change the cache size > dynamically, otherwise user will have to restart the JDBC server all > over again, which does not seem like a good idea from the perspective of > end user. So, it will be like one default system property will be there > (CARBON.MAX.DRIVER.LRU.CACHE.SIZE) which will be considered while > starting the server, and then user, later, can change the size of cache > according to his need on the fly using session property > CARBON.MAX.DRIVER.LRU.CACHE.SIZE.DYNAMIC (or something else) using set > command > set CARBON.MAX.DRIVER.LRU.CACHE.SIZE.DYNAMIC=100 > here 100 is in MB, which is similar to CARBON.MAX.DRIVER.LRU.CACHE.SIZE. > > One restriction we can add over this is that user, while changing the > size of cache, can only increase the cache size, and not decrease it, so > that we dont have to the the remove some entries which are already > present in the cache. This is also open to discussion, that we should > add this restriction or not. > > So, this way, user will be able to start the server with some value for > max cache size, can be -1 as well, then change the max cache size on the > fly. > Regards > > Naman Rastogi > Technical Lead - BigData Kernel > Huawei Technologies India Pvt. Ltd. > > > On Tue, Feb 19, 2019 at 6:51 PM akashrn5 <akashnilu...@gmail.com> wrote: > >> Hi naman, >> >> Thanks for proposing the feature. Looks really helpful from user and >> developer perspective. >> >> Basically needed the design document, so that all the doubts would be >> cleared. >> >> 1. Basicaly how you are going to handle the sync issues like, multiple >> queries with drop and show cache. are you going to introduce any locking >> mechanism? >> >> 2. if user clears cache during query? how it is noing to behave? is it >> allowed or concurrent operaion is blocked? >> >> 3. How it os gonna work with Distributed index server, and types of it >> like >> embedded, presto and other and local server? basically what is the impact >> with that? >> >> 4. You said you will launch a job to get the size from all the blocks >> present. Currently we create the block or blocklet datamap, and calcute >> each >> datamap size and then we add to cache based on the lru cach esize >> configured. So wanted to know how you will be calculating the size in your >> case >> >> Regards, >> Akash >> >> >> >> -- >> Sent from: >> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/ >> >
