[
https://issues.apache.org/jira/browse/RANGER-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516740#comment-17516740
]
Bhavik Patel commented on RANGER-3442:
--------------------------------------
[~vel] [~madhan] [~rmani] [~abhayk] Please add [~pavitheran] as a
contributor.
> Ranger KMS DAO memory issues when many new keys are created
> -----------------------------------------------------------
>
> Key: RANGER-3442
> URL: https://issues.apache.org/jira/browse/RANGER-3442
> Project: Ranger
> Issue Type: Bug
> Components: kms
> Affects Versions: 2.0.0
> Reporter: Pavi Subenderan
> Priority: Major
> Attachments: RANGER-3442-entity-manager-clear.patch, kms-key.py
>
>
> We have many keys created in our KMS keystore and recently we found that when
> we create new keys, the KMS instances easily hit against the memory limit.
> We can reproduce this with a script to call KMS createKey and then
> getMetadata for new keys in a loop. Basically we restart our instances and
> memory usage is approximately 1.5GB out of 8GB, but after running this script
> for a bit (1-5 minutes), we hit close to the 8GB limit and the memory usage
> does not go back down after that.
> I did a heap dump and saw that most of the memory was being retained by
> XXRangerKeystore and eclipse EntityManagerImpl.
> * org.eclipse.persistence.internal.jpa.EntityManagerImpl
> * org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork
> And the largest shallow size object was char[] with 4GB+...
>
> *My fix*
> I was ultimately able to solve this issue by adding an
> getEntityManager().clear() call in BaseDao.java getAllKeys().
> After I added this fix, we can now run as many KMS CreateKey / getMetadata
> calls as we want without any increase in memory usage whatsoever. Memory
> usage now stays constant at <1.7GB.
> My understanding is that Ranger KMS has a many instances of ThreadLocal
> EntityManager (160+ according to my heap dump) which each held a cache of the
> results for getAllKeys. Since we have so many keys in our KMS, this would
> easily put as at the memory limit.
> Not sure if there are any drawbacks to clearing EntityManager in
> BaseDao.getAllKeys() but we are seeing greatly improved performance in our
> case since we aren't constantly hitting the memory limit anymore.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
