+1
Thanks,
Ajantha
On Fri, 6 Dec, 2019, 8:38 pm Akash Nilugal, wrote:
> +1
>
> Regards,
> Akash R Nilugal
>
> On 2019/11/26 13:47:11, vikramahuja1001
> wrote:
> > Hi Community!
> > The support for prepriming in the case of Bloom and Lucene have to be
> > removed from the design document as tho
+1
Regards,
Akash R Nilugal
On 2019/11/26 13:47:11, vikramahuja1001 wrote:
> Hi Community!
> The support for prepriming in the case of Bloom and Lucene have to be
> removed from the design document as those datamaps are only created during
> query time and no the load time. Since they are not c
+1
Regards
Kumar Vishal
On Tue, Nov 26, 2019 at 7:02 PM vikramahuja1001
wrote:
> Hi Community!
> The support for prepriming in the case of Bloom and Lucene have to be
> removed from the design document as those datamaps are only created during
> query time and no the load time. Since they are no
Hi Community!
The support for prepriming in the case of Bloom and Lucene have to be
removed from the design document as those datamaps are only created during
query time and no the load time. Since they are not created during the load
time, they cannot be preprimed. PFA the updated design document.
Hi David,
Thanks for the input.
Here anyway at one point of time query is gonna happen on table, if we giev one
more table property, simply it will be like complex, like handle property,
compatibility, set and unset support. let's not make this more cumbersome.
Anyway LRU will take care to evi
+1 for me
When the table takes care of the performance of the first query, this feature
will help to improve it.
For other tables which don't take care of it, maybe no need to pre-cache index.
So it maybe has a table-level property to enable Cache Pre Priming.
Regards
David QiangCai
On 2019/08/19 09:53:10, Manhua wrote:
> Hi Akash,
>
> 1. cache will be full when loading is still running all the time. the reason
> I mention the invalidation is to avoid case, specially, when cache is full
> before all targeted index is loaded.
>
> When server just starting, keeping pre-
lease get back for any clarifications or inputs.
Thanks and Regards
Akash R Nilugal
From: xuchuanyin
> Date: Sat, 17 Aug, 2019, 11:55 AM
> Subject: Re: [DISCUSSION] Cache Pre Priming
> To:
>
>
> Hi, I've two questions about the current index server implementation:
&g
Hi Akash,
+1 for Vishal suggestion.Better focus on load data cache sync.
Regards,
Ravindra.
On Fri, 23 Aug 2019 at 16:35, Akash Nilugal wrote:
> Hi vishal,
>
> Your point is correct, we can focus on just loading to cache after data
> load is finished (Async Operation).
> for DDL support, count
Hi vishal,
Your point is correct, we can focus on just loading to cache after data load is
finished (Async Operation).
for DDL support, count(*) can be used for all the required tables to load into
cache.
Regards,
Akash
On 2019/08/22 14:44:34, Kumar Vishal wrote:
> Hi Akash,
>
> I think bet
Hi Akash,
I think better to PrePrime only after each load(Async).
As mentioned in design doc, when index servers is started, if the table or
db is configured, until and unless all the configured things are loaded
into cache, Index server won't be available for query. So query cannot get
benefit o
Hi chetan,
As mentioned in design , loading to cache will be an asyc operation, and we
will load only the corresponding segment to cache, so there wont be any hit.
Logs will be added
On 2019/08/21 13:18:05, chetan bhat wrote:
> Hi Akash,
>
> 1. Will the performance of end to end dataload oper
Hi Akash,
1. Will the performance of end to end dataload operation be impacted if the
segment datamap is loaded to cache once the load is finished.
2. Will there be a notification in logs stating that the loading of datamap
cache is completed.
Regards
On 2019/08/15 12:03:09, Akash Nilugal wro
Hi litao,
Basically
If total first time query takes x amount of time and in that y time takes for
to connect to index server , cache and return, then with pre prime we can save
this y time may if all the segments are not loaded then we can save less than Y
time, so we will get the benefit, ben
hi akash
count(*) can only load one table,if the table is very more, it is better we
can have a command to trigger the cache load.
On 2019/08/21 09:42:10, Akash Nilugal wrote:
> Hi Litao,
>
> Initially with first time count(*) , it used to take around 32seconds as it
> used to load into
hi, akash
How much of the performance difference between the first and second querys
is affected by caching index and how much is affected by Hadoop caching.
We should open it up and take a look at the time-consuming analysis on the
driver side.
On 2019/08/21 09:42:10, Akash Nilugal wro
Hi Litao,
Initially with first time count(*) , it used to take around 32seconds as it
used to load into cache, and second time query takes 1.5sec to 2 i think, so
with pre-prime we can achieve more improvement in first time query.
Regards,
Akash
On 2019/08/21 03:03:55, tao li wrote:
> hi Aka
On 2019/08/21 02:39:45, tao li wrote:
> hi Akash,
> I have a few questions.
> 1. About the ways to Pre-Prime: there are 2 ways. one is cache when data
> loading， another is when the cace server started.
> i think the latter is not desirable，because of load cache may take
hi Akash,
Before development, we need to know how much improvement can be made to
queries by caching part of the index in advance.
We need to compare the first and second query and analyze them. We need
to find time differences for several important steps.
It can analyze the per
hi Akash,
I have a few questions.
1. About the ways to Pre-Prime: there are 2 ways. one is cache when data
loading， another is when the cace server started.
i think the latter is not desirable，because of load cache may take more
it can course the cache server long time no res
Hi Manhua,
1. You are right that size will be full at one point, and according to you if
we stop pre-priming, then query will go and try to load cache and if it does
not get the size,
it will evict and do, so even pre-prime does the same thing LRU will handle
that for us. I will still think on
Hi Akash,
1. cache will be full when loading is still running all the time. the reason I
mention the invalidation is to avoid case, specially, when cache is full before
all targeted index is loaded.
When server just starting, keeping pre-prime and swap out the earliest loaded
index is not good
Hi xuchianyin,
Thanks for the question
1. Currently implementation is no need to load all the segments, only
required will be loaded during filter and all segments will be loaded
during query like count *.
2. Cache loading is fired during pruning phase in query, it will go to
index server prune
Hi manhua,
Thanks for the inputs.
1. No need to take care separately to invalidate the cache, i agree that it
will have limit. Since we already have eviction policy, when next query
comes, whenever required, it will evict and load the segments required, so
better not to have a separate mechanism
Hi, I come up with following ideas:
1. Although index server can provide more memory to hold the cache for index
data, its space still has a limit.
So cache managment(especially cache invalid) should be paid attention if we
Pre-Prime during data load or start of index server which easily fill
Hi, I've two questions about the current index server implementation:
1. Currently do we need to load all the index data of all segments to cache
server while doing filter query OR only load the segments required by this
query?
2. When do we trigger the cache loading action during the query?
As
Hi All,
I have raised a jira and attached the design doc there .please refer
CARBONDATA - 3492
Regards,
Akash
On Thu, Aug 15, 2019, 5:33 PM Akash Nilugal wrote:
> Hi Community,
>
> Currently, we have an index server which basically helps in distributed
> caching of the datamaps in a separate
Hi Akash,
Could you please raise a JIRA and attach the design doc? I cannot access
Thanks
---Original---
From: "Akash Nilugal"
Date: Thu, Aug 15, 2019 20:03 PM
To: "dev";
Subject: [DISCUSSION] Cache Pre Priming
Hi Community,
Currently, we have an index server whic
Hi Community,
Currently, we have an index server which basically helps in distributed
caching of the datamaps in a separate spark application.
The caching of the datamaps in index server will start once the query is
fired on the table for the first time, all the datamaps will be loaded
if the co
29 matches
Mail list logo
