[
https://issues.apache.org/jira/browse/HAWQ-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16558775#comment-16558775
]
Lav Jain commented on HAWQ-1622:
--------------------------------
We can limit the number of UGIs we create by caching them for the duration of
the session
1. LRU/timed expiration isn't appropriate because someone might be holding a
reference to the UGI object when it's evicted from the cache. Existing java
caching libraries (Guava,JCS,EHCache) dont' allow us to implement a policy
which leverages both reference and expiry.
2. A WeakReference-based cache wouldn't work because we need fine-grained
control over when UGIs are removed from the cache.
> Cache PXF proxy UGI so that cleanup of FileSystem cache doesn't have to be
> done on each request
> -----------------------------------------------------------------------------------------------
>
> Key: HAWQ-1622
> URL: https://issues.apache.org/jira/browse/HAWQ-1622
> Project: Apache HAWQ
> Issue Type: Improvement
> Components: PXF
> Reporter: Alexander Denissov
> Assignee: Lav Jain
> Priority: Major
>
> Closing PXF proxy UGIs on each request (implemented in HAWQ-1621) slows down
> PXF request response time significantly when several threads work
> concurrently as it locks FileSystem cache and holds the lock while the
> cleanup of DFSClients is completed.
> This can be avoided by caching the proxy UGI for a given proxy user between
> requests. Care must be taken to remove the cached entry after some
> pre-defined TTL if and only if there are no current threads using any
> FileSystem entries held by the cache. A combination of TTL-based cache with
> ref-counting might be utilized to achieve this.
>
> For some example of this, see:
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/service/UserGroupInformationService.java
> Caching UGIs might be tricky when Kerberos support is implemented later, see:
> https://issues.apache.org/jira/browse/HIVE-3098?focusedCommentId=13398979&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13398979
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
