[
https://issues.apache.org/jira/browse/ATLAS-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249689#comment-15249689
]
venkata madugundu edited comment on ATLAS-683 at 4/20/16 11:47 AM:
-------------------------------------------------------------------
Thanks [~yhemanth] for taking a look.
I have been following few threads where you are working on fixing issues
related to ZK throwing exceptions while running a large number of queries
concurrently. I will keep a tab on ATLAS-561.
Agree with you on tradeoffs with caching strategies, though I do not have much
design experience dealing with caches. My initial experiment started with
trying to retain the local in-process cache with each Atlas instance, but sync
it whenever type updates happen (using ZK notification mechanism). Few of our
my co-workers had reservations of such a sync from scale perspective, in the
sense if there are too many nodes, all would be talking whenever type updates
happen. The advantage of in-process cache is obviously the performance benefit
if the type-cache is looked up pretty frequently. The type updates (excluding
traits), I am pretty sure they happen only in service upgrades.
Thanks for noting the TABs in the patch. I remember configuring Eclipse to set
TABs as spaces probably in the middle of the work. I will reupload the patch
fixing the TABs.
Dave suggested to submit the patch for internal review before I submit for
review with you, as this will be first patch to Atlas. I will add you as
reviewer once I address Dave's comments.
was (Author: vmadugun):
Thanks [~yhemanth] for taking a look.
I have been following few threads where you are working on fixing issues
related to ZK throwing exceptions while running a large number of queries
concurrently. I will keep a tab on ATLAS-561.
Agree with you on tradeoffs with caching strategies, though I do not have much
design experience dealing with caches. My initial experiment started with
trying to retain the local in-process cache with each Atlas instance, but sync
it whenever type updates happen (using ZK notification mechanism). Few of our
my co-workers had reservations of such a sync from scale perspective, in the
sense if there are too many nodes, all would be talking whenever type updates
happen. The advantage of in-process cache is obviously the performance benefit
if the type-cache is looked up pretty frequently. The type updates (excluding
traits), I am pretty sure they happen only in service upgrades.
Thanks for noting the TABs in the patch. I remember configuring Eclipse to set
TABs as spaces probably in the middle of the work. I will reupload the patch
fixing the TABs.
> Refactor local type-system cache with cache provider interface
> --------------------------------------------------------------
>
> Key: ATLAS-683
> URL: https://issues.apache.org/jira/browse/ATLAS-683
> Project: Atlas
> Issue Type: Sub-task
> Affects Versions: 0.7-incubating
> Reporter: venkata madugundu
> Assignee: venkata madugundu
> Priority: Critical
> Labels: high-availability, performance, scalability
> Fix For: 0.7-incubating
>
>
> As noted in ATLAS-488, local type-system cache makes Atlas runtime stateful
> and prevents multiple Atlas instances to be active in a cluster. Either the
> type-cache should be synched across Atlas instances (on all type
> create/update requests) or the type-cache should be moved out of Atlas to
> something like a distributed cache.
> 1. As a first step, the local type-cache code in TypeSystem.java can be
> refactored to be carved out as an interface like TypeCacheProvider (whose
> default implementation for a standalone Atlas server would just use
> in-process local cache). The cache provider implementation itself could be
> specified as an optional configuration property. Expert users of Atlas can
> choose to inject a custom cache provider which can likely hit a distributed
> cache. We are evaluating the use of a distributed cache.
> 2. As a second step, some more refactoring can be done to minimize/optimize
> the calls made to TypeSystem for type lookup queries. Essentially, in a given
> transaction/request, once a type lookup is done, it should not be requeried
> again. A request scoped variable (guice would probably help with that
> scoping) can hold all the lookups made in a request. This might sound like a
> cache of a cache, but I think it should help in reducing the hits to cache
> provider if the provider is hitting a remote cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
