[
https://issues.apache.org/jira/browse/RANGER-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17422965#comment-17422965
]
Pavi Subenderan commented on RANGER-3442:
-----------------------------------------
[~brahmareddy] We recently created ~500 keys in our cluster which is when this
problem became very evident. Before that we only had 10-15 keys in our cluster
for 2+ years. (FYI we are using ranger 2.0.0)
I have been running this fix for almost a week and the memory issue seems to be
completely resolved. I can now easily add hundreds of keys with no noticeable
bump in memory usage. We also haven't detected any other regressions from
clearing the EntityManager.
I have attached the patch to this ticket:
[^RANGER-3442-entity-manager-clear.patch]
Not familiar with the contribution process, should I open a reviewboard request
for this?
> Ranger KMS DAO memory issues when many new keys are created
> -----------------------------------------------------------
>
> Key: RANGER-3442
> URL: https://issues.apache.org/jira/browse/RANGER-3442
> Project: Ranger
> Issue Type: Bug
> Components: kms
> Affects Versions: 2.0.0
> Reporter: Pavi Subenderan
> Priority: Major
> Attachments: RANGER-3442-entity-manager-clear.patch
>
>
> We have many keys created in our KMS keystore and recently we found that when
> we create new keys, the KMS instances easily hit against the memory limit.
> We can reproduce this with a script to call KMS createKey and then
> getMetadata for new keys in a loop. Basically we restart our instances and
> memory usage is approximately 1.5GB out of 8GB, but after running this script
> for a bit (1-5 minutes), we hit close to the 8GB limit and the memory usage
> does not go back down after that.
> I did a heap dump and saw that most of the memory was being retained by
> XXRangerKeystore and eclipse EntityManagerImpl.
> * org.eclipse.persistence.internal.jpa.EntityManagerImpl
> * org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork
> And the largest shallow size object was char[] with 4GB+...
>
> *My fix*
> I was ultimately able to solve this issue by adding an
> getEntityManager().clear() call in BaseDao.java getAllKeys().
> After I added this fix, we can now run as many KMS CreateKey / getMetadata
> calls as we want without any increase in memory usage whatsoever. Memory
> usage now stays constant at <1.7GB.
> My understanding is that Ranger KMS has a many instances of ThreadLocal
> EntityManager (160+ according to my heap dump) which each held a cache of the
> results for getAllKeys. Since we have so many keys in our KMS, this would
> easily put as at the memory limit.
> Not sure if there are any drawbacks to clearing EntityManager in
> BaseDao.getAllKeys() but we are seeing greatly improved performance in our
> case since we aren't constantly hitting the memory limit anymore.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
