[ https://issues.apache.org/jira/browse/RANGER-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17515328#comment-17515328 ]
Bhavik Patel commented on RANGER-3442: -------------------------------------- Thanks [~pavitheran] for the patch. Can you sent an email to the dev list with your apache id? [~dhavalshah9131] did you got chance to check the issue and solution? > Ranger KMS DAO memory issues when many new keys are created > ----------------------------------------------------------- > > Key: RANGER-3442 > URL: https://issues.apache.org/jira/browse/RANGER-3442 > Project: Ranger > Issue Type: Bug > Components: kms > Affects Versions: 2.0.0 > Reporter: Pavi Subenderan > Priority: Major > Attachments: RANGER-3442-entity-manager-clear.patch, kms-key.py > > > We have many keys created in our KMS keystore and recently we found that when > we create new keys, the KMS instances easily hit against the memory limit. > We can reproduce this with a script to call KMS createKey and then > getMetadata for new keys in a loop. Basically we restart our instances and > memory usage is approximately 1.5GB out of 8GB, but after running this script > for a bit (1-5 minutes), we hit close to the 8GB limit and the memory usage > does not go back down after that. > I did a heap dump and saw that most of the memory was being retained by > XXRangerKeystore and eclipse EntityManagerImpl. > * org.eclipse.persistence.internal.jpa.EntityManagerImpl > * org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork > And the largest shallow size object was char[] with 4GB+... > > *My fix* > I was ultimately able to solve this issue by adding an > getEntityManager().clear() call in BaseDao.java getAllKeys(). > After I added this fix, we can now run as many KMS CreateKey / getMetadata > calls as we want without any increase in memory usage whatsoever. Memory > usage now stays constant at <1.7GB. > My understanding is that Ranger KMS has a many instances of ThreadLocal > EntityManager (160+ according to my heap dump) which each held a cache of the > results for getAllKeys. Since we have so many keys in our KMS, this would > easily put as at the memory limit. > Not sure if there are any drawbacks to clearing EntityManager in > BaseDao.getAllKeys() but we are seeing greatly improved performance in our > case since we aren't constantly hitting the memory limit anymore. -- This message was sent by Atlassian Jira (v8.20.1#820001)