Hi Akash, I think better to PrePrime only after each load(Async).
As mentioned in design doc, when index servers is started, if the table or db is configured, until and unless all the configured things are loaded into cache, Index server won't be available for query. So query cannot get benefit of pre-prime untill all the metadata is loaded to cache. In order to avoid this user can run count(*) after startup to pre-prime only required tables. Any extra ddl is not required as count(*) can be used as a DDL to load the cache. -Regards Kumar Vishal On Wed, Aug 21, 2019 at 6:57 PM Akash Nilugal <[email protected]> wrote: > Hi chetan, > > As mentioned in design , loading to cache will be an asyc operation, and > we will load only the corresponding segment to cache, so there wont be any > hit. > Logs will be added > > On 2019/08/21 13:18:05, chetan bhat <[email protected]> wrote: > > Hi Akash, > > > > 1. Will the performance of end to end dataload operation be impacted if > the segment datamap is loaded to cache once the load is finished. > > 2. Will there be a notification in logs stating that the loading of > datamap cache is completed. > > > > Regards > > > > On 2019/08/15 12:03:09, Akash Nilugal <[email protected]> wrote: > > > Hi Community, > > > > > > Currently, we have an index server which basically helps in distributed > > > caching of the datamaps in a separate spark application. > > > > > > The caching of the datamaps in index server will start once the query > is > > > fired on the table for the first time, all the datamaps will be loaded > > > > > > if the count(*) is fired and only required will be loaded for any > filter > > > query. > > > > > > > > > Here the problem or the bottleneck is, until and unless the query is > fired > > > on table, the caching won’t be done for the table datamaps. > > > > > > So consider a scenario where we are just loading the data to table for > > > whole day and then next day we query, > > > > > > so all the segments will start loading into cache. So first time the > query > > > will be slow. > > > > > > > > > What if we load the datamaps into cache or preprime the cache without > > > waititng for any query on the table? > > > > > > Yes, what if we load the cache after every load is done, what if we > load > > > the cache for all the segments at once, > > > > > > so that first time query need not do all this job, which makes it > faster. > > > > > > > > > Here i have attached the design document for the pre-priming of cache > into > > > index server. Please have a look at it > > > > > > and any suggestions or inputs on this are most welcomed. > > > > > > > > > > https://drive.google.com/file/d/1YUpDUv7ZPUyZQQYwQYcQK2t2aBQH18PB/view?usp=sharing > > > > > > > > > > > > Regards, > > > > > > Akash R Nilugal > > > > > >
