[jira] [Commented] (RANGER-3442) Ranger KMS DAO memory issues when many new keys are created
[ https://issues.apache.org/jira/browse/RANGER-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17517315#comment-17517315 ] Dhaval Shah commented on RANGER-3442: - Hi, Apache master commit : https://github.com/apache/ranger/commit/7214d6d7ba90bbe377614682f6972479a7722dc5 Apache ranger-2.3 commit: https://github.com/apache/ranger/commit/3dcccf22049f3442915c7cb6f9d1f2d3304637a4 Thanks. [~pavitheran], Please close the review request and Apache Jira. Thanks > Ranger KMS DAO memory issues when many new keys are created > --- > > Key: RANGER-3442 > URL: https://issues.apache.org/jira/browse/RANGER-3442 > Project: Ranger > Issue Type: Bug > Components: kms >Affects Versions: 2.0.0 >Reporter: Pavi Subenderan >Assignee: Pavi Subenderan >Priority: Major > Attachments: RANGER-3442-entity-manager-clear.patch, kms-key.py > > > We have many keys created in our KMS keystore and recently we found that when > we create new keys, the KMS instances easily hit against the memory limit. > We can reproduce this with a script to call KMS createKey and then > getMetadata for new keys in a loop. Basically we restart our instances and > memory usage is approximately 1.5GB out of 8GB, but after running this script > for a bit (1-5 minutes), we hit close to the 8GB limit and the memory usage > does not go back down after that. > I did a heap dump and saw that most of the memory was being retained by > XXRangerKeystore and eclipse EntityManagerImpl. > * org.eclipse.persistence.internal.jpa.EntityManagerImpl > * org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork > And the largest shallow size object was char[] with 4GB+... > > *My fix* > I was ultimately able to solve this issue by adding an > getEntityManager().clear() call in BaseDao.java getAllKeys(). > After I added this fix, we can now run as many KMS CreateKey / getMetadata > calls as we want without any increase in memory usage whatsoever. Memory > usage now stays constant at <1.7GB. > My understanding is that Ranger KMS has a many instances of ThreadLocal > EntityManager (160+ according to my heap dump) which each held a cache of the > results for getAllKeys. Since we have so many keys in our KMS, this would > easily put as at the memory limit. > Not sure if there are any drawbacks to clearing EntityManager in > BaseDao.getAllKeys() but we are seeing greatly improved performance in our > case since we aren't constantly hitting the memory limit anymore. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (RANGER-3442) Ranger KMS DAO memory issues when many new keys are created
[ https://issues.apache.org/jira/browse/RANGER-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516869#comment-17516869 ] Velmurugan Periasamy commented on RANGER-3442: -- [~pavitheran] is added as contributor. Thanks. CC [~dhavalshah9131] / [~spolavarapu] > Ranger KMS DAO memory issues when many new keys are created > --- > > Key: RANGER-3442 > URL: https://issues.apache.org/jira/browse/RANGER-3442 > Project: Ranger > Issue Type: Bug > Components: kms >Affects Versions: 2.0.0 >Reporter: Pavi Subenderan >Assignee: Pavi Subenderan >Priority: Major > Attachments: RANGER-3442-entity-manager-clear.patch, kms-key.py > > > We have many keys created in our KMS keystore and recently we found that when > we create new keys, the KMS instances easily hit against the memory limit. > We can reproduce this with a script to call KMS createKey and then > getMetadata for new keys in a loop. Basically we restart our instances and > memory usage is approximately 1.5GB out of 8GB, but after running this script > for a bit (1-5 minutes), we hit close to the 8GB limit and the memory usage > does not go back down after that. > I did a heap dump and saw that most of the memory was being retained by > XXRangerKeystore and eclipse EntityManagerImpl. > * org.eclipse.persistence.internal.jpa.EntityManagerImpl > * org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork > And the largest shallow size object was char[] with 4GB+... > > *My fix* > I was ultimately able to solve this issue by adding an > getEntityManager().clear() call in BaseDao.java getAllKeys(). > After I added this fix, we can now run as many KMS CreateKey / getMetadata > calls as we want without any increase in memory usage whatsoever. Memory > usage now stays constant at <1.7GB. > My understanding is that Ranger KMS has a many instances of ThreadLocal > EntityManager (160+ according to my heap dump) which each held a cache of the > results for getAllKeys. Since we have so many keys in our KMS, this would > easily put as at the memory limit. > Not sure if there are any drawbacks to clearing EntityManager in > BaseDao.getAllKeys() but we are seeing greatly improved performance in our > case since we aren't constantly hitting the memory limit anymore. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (RANGER-3442) Ranger KMS DAO memory issues when many new keys are created
[ https://issues.apache.org/jira/browse/RANGER-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516740#comment-17516740 ] Bhavik Patel commented on RANGER-3442: -- [~vel] [~madhan] [~rmani] [~abhayk] Please add [~pavitheran] as a contributor. > Ranger KMS DAO memory issues when many new keys are created > --- > > Key: RANGER-3442 > URL: https://issues.apache.org/jira/browse/RANGER-3442 > Project: Ranger > Issue Type: Bug > Components: kms >Affects Versions: 2.0.0 >Reporter: Pavi Subenderan >Priority: Major > Attachments: RANGER-3442-entity-manager-clear.patch, kms-key.py > > > We have many keys created in our KMS keystore and recently we found that when > we create new keys, the KMS instances easily hit against the memory limit. > We can reproduce this with a script to call KMS createKey and then > getMetadata for new keys in a loop. Basically we restart our instances and > memory usage is approximately 1.5GB out of 8GB, but after running this script > for a bit (1-5 minutes), we hit close to the 8GB limit and the memory usage > does not go back down after that. > I did a heap dump and saw that most of the memory was being retained by > XXRangerKeystore and eclipse EntityManagerImpl. > * org.eclipse.persistence.internal.jpa.EntityManagerImpl > * org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork > And the largest shallow size object was char[] with 4GB+... > > *My fix* > I was ultimately able to solve this issue by adding an > getEntityManager().clear() call in BaseDao.java getAllKeys(). > After I added this fix, we can now run as many KMS CreateKey / getMetadata > calls as we want without any increase in memory usage whatsoever. Memory > usage now stays constant at <1.7GB. > My understanding is that Ranger KMS has a many instances of ThreadLocal > EntityManager (160+ according to my heap dump) which each held a cache of the > results for getAllKeys. Since we have so many keys in our KMS, this would > easily put as at the memory limit. > Not sure if there are any drawbacks to clearing EntityManager in > BaseDao.getAllKeys() but we are seeing greatly improved performance in our > case since we aren't constantly hitting the memory limit anymore. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (RANGER-3442) Ranger KMS DAO memory issues when many new keys are created
[ https://issues.apache.org/jira/browse/RANGER-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516207#comment-17516207 ] Pavi Subenderan commented on RANGER-3442: - Thanks [~bpatel] I previously sent my apache id to the dev list on Sept 30 2021. I believe my apache id is: pavitheran Let me know if you need anything else. > Ranger KMS DAO memory issues when many new keys are created > --- > > Key: RANGER-3442 > URL: https://issues.apache.org/jira/browse/RANGER-3442 > Project: Ranger > Issue Type: Bug > Components: kms >Affects Versions: 2.0.0 >Reporter: Pavi Subenderan >Priority: Major > Attachments: RANGER-3442-entity-manager-clear.patch, kms-key.py > > > We have many keys created in our KMS keystore and recently we found that when > we create new keys, the KMS instances easily hit against the memory limit. > We can reproduce this with a script to call KMS createKey and then > getMetadata for new keys in a loop. Basically we restart our instances and > memory usage is approximately 1.5GB out of 8GB, but after running this script > for a bit (1-5 minutes), we hit close to the 8GB limit and the memory usage > does not go back down after that. > I did a heap dump and saw that most of the memory was being retained by > XXRangerKeystore and eclipse EntityManagerImpl. > * org.eclipse.persistence.internal.jpa.EntityManagerImpl > * org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork > And the largest shallow size object was char[] with 4GB+... > > *My fix* > I was ultimately able to solve this issue by adding an > getEntityManager().clear() call in BaseDao.java getAllKeys(). > After I added this fix, we can now run as many KMS CreateKey / getMetadata > calls as we want without any increase in memory usage whatsoever. Memory > usage now stays constant at <1.7GB. > My understanding is that Ranger KMS has a many instances of ThreadLocal > EntityManager (160+ according to my heap dump) which each held a cache of the > results for getAllKeys. Since we have so many keys in our KMS, this would > easily put as at the memory limit. > Not sure if there are any drawbacks to clearing EntityManager in > BaseDao.getAllKeys() but we are seeing greatly improved performance in our > case since we aren't constantly hitting the memory limit anymore. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (RANGER-3442) Ranger KMS DAO memory issues when many new keys are created
[ https://issues.apache.org/jira/browse/RANGER-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17515328#comment-17515328 ] Bhavik Patel commented on RANGER-3442: -- Thanks [~pavitheran] for the patch. Can you sent an email to the dev list with your apache id? [~dhavalshah9131] did you got chance to check the issue and solution? > Ranger KMS DAO memory issues when many new keys are created > --- > > Key: RANGER-3442 > URL: https://issues.apache.org/jira/browse/RANGER-3442 > Project: Ranger > Issue Type: Bug > Components: kms >Affects Versions: 2.0.0 >Reporter: Pavi Subenderan >Priority: Major > Attachments: RANGER-3442-entity-manager-clear.patch, kms-key.py > > > We have many keys created in our KMS keystore and recently we found that when > we create new keys, the KMS instances easily hit against the memory limit. > We can reproduce this with a script to call KMS createKey and then > getMetadata for new keys in a loop. Basically we restart our instances and > memory usage is approximately 1.5GB out of 8GB, but after running this script > for a bit (1-5 minutes), we hit close to the 8GB limit and the memory usage > does not go back down after that. > I did a heap dump and saw that most of the memory was being retained by > XXRangerKeystore and eclipse EntityManagerImpl. > * org.eclipse.persistence.internal.jpa.EntityManagerImpl > * org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork > And the largest shallow size object was char[] with 4GB+... > > *My fix* > I was ultimately able to solve this issue by adding an > getEntityManager().clear() call in BaseDao.java getAllKeys(). > After I added this fix, we can now run as many KMS CreateKey / getMetadata > calls as we want without any increase in memory usage whatsoever. Memory > usage now stays constant at <1.7GB. > My understanding is that Ranger KMS has a many instances of ThreadLocal > EntityManager (160+ according to my heap dump) which each held a cache of the > results for getAllKeys. Since we have so many keys in our KMS, this would > easily put as at the memory limit. > Not sure if there are any drawbacks to clearing EntityManager in > BaseDao.getAllKeys() but we are seeing greatly improved performance in our > case since we aren't constantly hitting the memory limit anymore. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (RANGER-3442) Ranger KMS DAO memory issues when many new keys are created
[
https://issues.apache.org/jira/browse/RANGER-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17448815#comment-17448815
]
Pavi Subenderan commented on RANGER-3442:
-
[~dhavalshah9131] Here is the script used to create a key and then get the
metadata of a key: [^kms-key.py]
Make sure you update the variables for "KMS_PROTOCOL", "KMS_PORT", and
KMS_HOST" in the above python script to match the protocol/host/port used by
your environment.
And then we combined the above script with this loop to generate many keys in a
loop:
{code:java}
for year in {2010..2020}; do for month in {01..12}; do cat
src_categorization_level_out | xargs -I {} ./kms-key.py -c v2_{}_$year$month -d
"data categorization key" --commit; done; done {code}
"cat src_categorization_level_out" in the above loop is just a file with a key
name on each line like:
{noformat}
testkey1
privatekey
usernamekey
securitykey
laptopsec
category5
category6
cat7{noformat}
Running this should create hundreds of keys very quickly.
> Ranger KMS DAO memory issues when many new keys are created
> ---
>
> Key: RANGER-3442
> URL: https://issues.apache.org/jira/browse/RANGER-3442
> Project: Ranger
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.0.0
>Reporter: Pavi Subenderan
>Priority: Major
> Attachments: RANGER-3442-entity-manager-clear.patch, kms-key.py
>
>
> We have many keys created in our KMS keystore and recently we found that when
> we create new keys, the KMS instances easily hit against the memory limit.
> We can reproduce this with a script to call KMS createKey and then
> getMetadata for new keys in a loop. Basically we restart our instances and
> memory usage is approximately 1.5GB out of 8GB, but after running this script
> for a bit (1-5 minutes), we hit close to the 8GB limit and the memory usage
> does not go back down after that.
> I did a heap dump and saw that most of the memory was being retained by
> XXRangerKeystore and eclipse EntityManagerImpl.
> * org.eclipse.persistence.internal.jpa.EntityManagerImpl
> * org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork
> And the largest shallow size object was char[] with 4GB+...
>
> *My fix*
> I was ultimately able to solve this issue by adding an
> getEntityManager().clear() call in BaseDao.java getAllKeys().
> After I added this fix, we can now run as many KMS CreateKey / getMetadata
> calls as we want without any increase in memory usage whatsoever. Memory
> usage now stays constant at <1.7GB.
> My understanding is that Ranger KMS has a many instances of ThreadLocal
> EntityManager (160+ according to my heap dump) which each held a cache of the
> results for getAllKeys. Since we have so many keys in our KMS, this would
> easily put as at the memory limit.
> Not sure if there are any drawbacks to clearing EntityManager in
> BaseDao.getAllKeys() but we are seeing greatly improved performance in our
> case since we aren't constantly hitting the memory limit anymore.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (RANGER-3442) Ranger KMS DAO memory issues when many new keys are created
[ https://issues.apache.org/jira/browse/RANGER-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442257#comment-17442257 ] Dhaval Shah commented on RANGER-3442: - Hi [~pavitheran], Thanks for the improvement. Can you please share the scripts which you have mentioned in description for creating keys and getting metadata. Thanks > Ranger KMS DAO memory issues when many new keys are created > --- > > Key: RANGER-3442 > URL: https://issues.apache.org/jira/browse/RANGER-3442 > Project: Ranger > Issue Type: Bug > Components: kms >Affects Versions: 2.0.0 >Reporter: Pavi Subenderan >Priority: Major > Attachments: RANGER-3442-entity-manager-clear.patch > > > We have many keys created in our KMS keystore and recently we found that when > we create new keys, the KMS instances easily hit against the memory limit. > We can reproduce this with a script to call KMS createKey and then > getMetadata for new keys in a loop. Basically we restart our instances and > memory usage is approximately 1.5GB out of 8GB, but after running this script > for a bit (1-5 minutes), we hit close to the 8GB limit and the memory usage > does not go back down after that. > I did a heap dump and saw that most of the memory was being retained by > XXRangerKeystore and eclipse EntityManagerImpl. > * org.eclipse.persistence.internal.jpa.EntityManagerImpl > * org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork > And the largest shallow size object was char[] with 4GB+... > > *My fix* > I was ultimately able to solve this issue by adding an > getEntityManager().clear() call in BaseDao.java getAllKeys(). > After I added this fix, we can now run as many KMS CreateKey / getMetadata > calls as we want without any increase in memory usage whatsoever. Memory > usage now stays constant at <1.7GB. > My understanding is that Ranger KMS has a many instances of ThreadLocal > EntityManager (160+ according to my heap dump) which each held a cache of the > results for getAllKeys. Since we have so many keys in our KMS, this would > easily put as at the memory limit. > Not sure if there are any drawbacks to clearing EntityManager in > BaseDao.getAllKeys() but we are seeing greatly improved performance in our > case since we aren't constantly hitting the memory limit anymore. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (RANGER-3442) Ranger KMS DAO memory issues when many new keys are created
[ https://issues.apache.org/jira/browse/RANGER-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17423009#comment-17423009 ] Velmurugan Periasamy commented on RANGER-3442: -- Please send an email to the dev list with your apache id. I can add you as a contributor. > Ranger KMS DAO memory issues when many new keys are created > --- > > Key: RANGER-3442 > URL: https://issues.apache.org/jira/browse/RANGER-3442 > Project: Ranger > Issue Type: Bug > Components: kms >Affects Versions: 2.0.0 >Reporter: Pavi Subenderan >Priority: Major > Attachments: RANGER-3442-entity-manager-clear.patch > > > We have many keys created in our KMS keystore and recently we found that when > we create new keys, the KMS instances easily hit against the memory limit. > We can reproduce this with a script to call KMS createKey and then > getMetadata for new keys in a loop. Basically we restart our instances and > memory usage is approximately 1.5GB out of 8GB, but after running this script > for a bit (1-5 minutes), we hit close to the 8GB limit and the memory usage > does not go back down after that. > I did a heap dump and saw that most of the memory was being retained by > XXRangerKeystore and eclipse EntityManagerImpl. > * org.eclipse.persistence.internal.jpa.EntityManagerImpl > * org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork > And the largest shallow size object was char[] with 4GB+... > > *My fix* > I was ultimately able to solve this issue by adding an > getEntityManager().clear() call in BaseDao.java getAllKeys(). > After I added this fix, we can now run as many KMS CreateKey / getMetadata > calls as we want without any increase in memory usage whatsoever. Memory > usage now stays constant at <1.7GB. > My understanding is that Ranger KMS has a many instances of ThreadLocal > EntityManager (160+ according to my heap dump) which each held a cache of the > results for getAllKeys. Since we have so many keys in our KMS, this would > easily put as at the memory limit. > Not sure if there are any drawbacks to clearing EntityManager in > BaseDao.getAllKeys() but we are seeing greatly improved performance in our > case since we aren't constantly hitting the memory limit anymore. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-3442) Ranger KMS DAO memory issues when many new keys are created
[ https://issues.apache.org/jira/browse/RANGER-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17423008#comment-17423008 ] Pavi Subenderan commented on RANGER-3442: - Opened review board request: [https://reviews.apache.org/r/73621/] By the way, how can I become the assignee on this ticket? > Ranger KMS DAO memory issues when many new keys are created > --- > > Key: RANGER-3442 > URL: https://issues.apache.org/jira/browse/RANGER-3442 > Project: Ranger > Issue Type: Bug > Components: kms >Affects Versions: 2.0.0 >Reporter: Pavi Subenderan >Priority: Major > Attachments: RANGER-3442-entity-manager-clear.patch > > > We have many keys created in our KMS keystore and recently we found that when > we create new keys, the KMS instances easily hit against the memory limit. > We can reproduce this with a script to call KMS createKey and then > getMetadata for new keys in a loop. Basically we restart our instances and > memory usage is approximately 1.5GB out of 8GB, but after running this script > for a bit (1-5 minutes), we hit close to the 8GB limit and the memory usage > does not go back down after that. > I did a heap dump and saw that most of the memory was being retained by > XXRangerKeystore and eclipse EntityManagerImpl. > * org.eclipse.persistence.internal.jpa.EntityManagerImpl > * org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork > And the largest shallow size object was char[] with 4GB+... > > *My fix* > I was ultimately able to solve this issue by adding an > getEntityManager().clear() call in BaseDao.java getAllKeys(). > After I added this fix, we can now run as many KMS CreateKey / getMetadata > calls as we want without any increase in memory usage whatsoever. Memory > usage now stays constant at <1.7GB. > My understanding is that Ranger KMS has a many instances of ThreadLocal > EntityManager (160+ according to my heap dump) which each held a cache of the > results for getAllKeys. Since we have so many keys in our KMS, this would > easily put as at the memory limit. > Not sure if there are any drawbacks to clearing EntityManager in > BaseDao.getAllKeys() but we are seeing greatly improved performance in our > case since we aren't constantly hitting the memory limit anymore. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-3442) Ranger KMS DAO memory issues when many new keys are created
[ https://issues.apache.org/jira/browse/RANGER-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17422991#comment-17422991 ] Velmurugan Periasamy commented on RANGER-3442: -- Thanks [~pavitheran] for the contribution. Please create a review board request. > Ranger KMS DAO memory issues when many new keys are created > --- > > Key: RANGER-3442 > URL: https://issues.apache.org/jira/browse/RANGER-3442 > Project: Ranger > Issue Type: Bug > Components: kms >Affects Versions: 2.0.0 >Reporter: Pavi Subenderan >Priority: Major > Attachments: RANGER-3442-entity-manager-clear.patch > > > We have many keys created in our KMS keystore and recently we found that when > we create new keys, the KMS instances easily hit against the memory limit. > We can reproduce this with a script to call KMS createKey and then > getMetadata for new keys in a loop. Basically we restart our instances and > memory usage is approximately 1.5GB out of 8GB, but after running this script > for a bit (1-5 minutes), we hit close to the 8GB limit and the memory usage > does not go back down after that. > I did a heap dump and saw that most of the memory was being retained by > XXRangerKeystore and eclipse EntityManagerImpl. > * org.eclipse.persistence.internal.jpa.EntityManagerImpl > * org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork > And the largest shallow size object was char[] with 4GB+... > > *My fix* > I was ultimately able to solve this issue by adding an > getEntityManager().clear() call in BaseDao.java getAllKeys(). > After I added this fix, we can now run as many KMS CreateKey / getMetadata > calls as we want without any increase in memory usage whatsoever. Memory > usage now stays constant at <1.7GB. > My understanding is that Ranger KMS has a many instances of ThreadLocal > EntityManager (160+ according to my heap dump) which each held a cache of the > results for getAllKeys. Since we have so many keys in our KMS, this would > easily put as at the memory limit. > Not sure if there are any drawbacks to clearing EntityManager in > BaseDao.getAllKeys() but we are seeing greatly improved performance in our > case since we aren't constantly hitting the memory limit anymore. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-3442) Ranger KMS DAO memory issues when many new keys are created
[ https://issues.apache.org/jira/browse/RANGER-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17422965#comment-17422965 ] Pavi Subenderan commented on RANGER-3442: - [~brahmareddy] We recently created ~500 keys in our cluster which is when this problem became very evident. Before that we only had 10-15 keys in our cluster for 2+ years. (FYI we are using ranger 2.0.0) I have been running this fix for almost a week and the memory issue seems to be completely resolved. I can now easily add hundreds of keys with no noticeable bump in memory usage. We also haven't detected any other regressions from clearing the EntityManager. I have attached the patch to this ticket: [^RANGER-3442-entity-manager-clear.patch] Not familiar with the contribution process, should I open a reviewboard request for this? > Ranger KMS DAO memory issues when many new keys are created > --- > > Key: RANGER-3442 > URL: https://issues.apache.org/jira/browse/RANGER-3442 > Project: Ranger > Issue Type: Bug > Components: kms >Affects Versions: 2.0.0 >Reporter: Pavi Subenderan >Priority: Major > Attachments: RANGER-3442-entity-manager-clear.patch > > > We have many keys created in our KMS keystore and recently we found that when > we create new keys, the KMS instances easily hit against the memory limit. > We can reproduce this with a script to call KMS createKey and then > getMetadata for new keys in a loop. Basically we restart our instances and > memory usage is approximately 1.5GB out of 8GB, but after running this script > for a bit (1-5 minutes), we hit close to the 8GB limit and the memory usage > does not go back down after that. > I did a heap dump and saw that most of the memory was being retained by > XXRangerKeystore and eclipse EntityManagerImpl. > * org.eclipse.persistence.internal.jpa.EntityManagerImpl > * org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork > And the largest shallow size object was char[] with 4GB+... > > *My fix* > I was ultimately able to solve this issue by adding an > getEntityManager().clear() call in BaseDao.java getAllKeys(). > After I added this fix, we can now run as many KMS CreateKey / getMetadata > calls as we want without any increase in memory usage whatsoever. Memory > usage now stays constant at <1.7GB. > My understanding is that Ranger KMS has a many instances of ThreadLocal > EntityManager (160+ according to my heap dump) which each held a cache of the > results for getAllKeys. Since we have so many keys in our KMS, this would > easily put as at the memory limit. > Not sure if there are any drawbacks to clearing EntityManager in > BaseDao.getAllKeys() but we are seeing greatly improved performance in our > case since we aren't constantly hitting the memory limit anymore. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-3442) Ranger KMS DAO memory issues when many new keys are created
[
https://issues.apache.org/jira/browse/RANGER-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17421273#comment-17421273
]
Brahma Reddy Battula commented on RANGER-3442:
--
[~pavitheran] thanks for reporting and neat analysis. I do see that ranger_KMS
using more memory and need to do profiling (pasting the heapdump here) , not
sure it might to similar you reported but your analysis make sense to me. (
using ranger-0.7.0 with some patches.)
{noformat}
num #instances #bytes class name
--
1: 12514991 1125037584 [C
2: 3194026 629310896 [Ljava.util.HashMap$Node;
3: 4547927 421361800 [B
4: 12492061 399745952 java.lang.String
5: 7398577 355131696 java.util.HashMap$Node
6: 3189692 280692896 org.apache.zookeeper.data.Stat
7: 3197676 204651264 java.util.HashMap
8: 3207962 153982176 java.util.concurrent.ConcurrentHashMap$Node
9: 3189693 127587720
org.apache.curator.framework.recipes.cache.ChildData
10: 3190260 76566240 java.util.HashSet
11: 3189714 76553136 java.util.concurrent.atomic.AtomicReference
12: 1349 67582872
[Ljava.util.concurrent.ConcurrentHashMap$Node;
13: 1263345 40427040 org.apache.hadoop.io.Text
14: 23853 35543088 [Ljava.lang.Object;
15: 421113 33689040
org.apache.hadoop.security.token.delegation.web.DelegationTokenIdentifier
16: 421113 16844520
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager$DelegationTokenInformation
num #instances #bytes class name
--
1: 12494150 1126339552 [C
2: 3189356 561492704 [Ljava.util.HashMap$Node;
3: 4627687 429962248 [B
4: 12470326 399050432 java.lang.String
5: 6811252 326940096 java.util.HashMap$Node
6: 3186259 280390792 org.apache.zookeeper.data.Stat
7: 3194119 204423616 java.util.HashMap
8: 3204325 153807600 java.util.concurrent.ConcurrentHashMap$Node
9: 3186257 127450280
org.apache.curator.framework.recipes.cache.ChildData
10: 3186819 76483656 java.util.HashSet
11: 3186279 76470696 java.util.concurrent.atomic.AtomicReference
12: 1420 67593664
[Ljava.util.concurrent.ConcurrentHashMap$Node;
13: 96175 43101976 [Ljava.lang.Object;
14: 1253046 40097472 org.apache.hadoop.io.Text
15: 417680 33414400
org.apache.hadoop.security.token.delegation.web.DelegationTokenIdentifier
16: 417680 16707200
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager$DelegationTokenInformation
17: 79595 14008720 java.net.SocksSocketImpl
18: 112616 7207424 java.lang.ref.Finalizer
19: 79591 7004008 java.net.SocketInputStream
20: 79591 7004008 java.net.SocketOutputStream
21: 414599 6633584 java.lang.Object
22: 79101 5695272 org.apache.tomcat.util.buf.ByteChunk
23: 67540 4322560 org.apache.tomcat.util.buf.CharChunk
24: 61120 3911680 org.apache.tomcat.util.buf.MessageBytes
25: 39328 3885784 [Ljava.lang.String;
26: 84221 3368840 java.util.ArrayList
27: 79610 3184400 java.io.FileDescriptor
{noformat}
> Ranger KMS DAO memory issues when many new keys are created
> ---
>
> Key: RANGER-3442
> URL: https://issues.apache.org/jira/browse/RANGER-3442
> Project: Ranger
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.0.0
>Reporter: Pavi Subenderan
>Priority: Major
>
> We have many keys created in our KMS keystore and recently we found that when
> we create new keys, the KMS instances easily hit against the memory limit.
> We can reproduce this with a script to call KMS createKey and then
> getMetadata for new keys in a loop. Basically we restart our instances and
> memory usage is approximately 1.5GB out of 8GB, but after running this script
> for a bit (1-5 minutes), we hit close to the 8GB limit and the memory usage
> does not go back down after that.
> I did a heap dump and saw that most of the memory was being retained by
> XXRangerKeystore and eclipse EntityManagerImpl.
> * org.eclipse.persistence.internal.jpa.EntityManagerImpl
> * org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork
> And the largest shallow size object was char[] with 4GB+...
>
> *My fix*
