[
https://issues.apache.org/jira/browse/RANGER-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17421273#comment-17421273
]
Brahma Reddy Battula edited comment on RANGER-3442 at 9/28/21, 9:28 AM:
------------------------------------------------------------------------
[~pavitheran] thanks for reporting and neat analysis. I do see that ranger_KMS
using more memory and need to do profiling (pasting the heapdump here) , not
sure it might to similar you reported but your analysis make sense to me. (
using ranger-0.7.0 with some patches.). In our cluster we've 16 keys, our heap
usage is 60GB+..
How many keys present in your cluster..?
{noformat}
num #instances #bytes class name
----------------------------------------------
1: 12514991 1125037584 [C
2: 3194026 629310896 [Ljava.util.HashMap$Node;
3: 4547927 421361800 [B
4: 12492061 399745952 java.lang.String
5: 7398577 355131696 java.util.HashMap$Node
6: 3189692 280692896 org.apache.zookeeper.data.Stat
7: 3197676 204651264 java.util.HashMap
8: 3207962 153982176 java.util.concurrent.ConcurrentHashMap$Node
9: 3189693 127587720
org.apache.curator.framework.recipes.cache.ChildData
10: 3190260 76566240 java.util.HashSet
11: 3189714 76553136 java.util.concurrent.atomic.AtomicReference
12: 1349 67582872
[Ljava.util.concurrent.ConcurrentHashMap$Node;
13: 1263345 40427040 org.apache.hadoop.io.Text
14: 23853 35543088 [Ljava.lang.Object;
15: 421113 33689040
org.apache.hadoop.security.token.delegation.web.DelegationTokenIdentifier
16: 421113 16844520
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager$DelegationTokenInformation
num #instances #bytes class name
----------------------------------------------
1: 12494150 1126339552 [C
2: 3189356 561492704 [Ljava.util.HashMap$Node;
3: 4627687 429962248 [B
4: 12470326 399050432 java.lang.String
5: 6811252 326940096 java.util.HashMap$Node
6: 3186259 280390792 org.apache.zookeeper.data.Stat
7: 3194119 204423616 java.util.HashMap
8: 3204325 153807600 java.util.concurrent.ConcurrentHashMap$Node
9: 3186257 127450280
org.apache.curator.framework.recipes.cache.ChildData
10: 3186819 76483656 java.util.HashSet
11: 3186279 76470696 java.util.concurrent.atomic.AtomicReference
12: 1420 67593664
[Ljava.util.concurrent.ConcurrentHashMap$Node;
13: 96175 43101976 [Ljava.lang.Object;
14: 1253046 40097472 org.apache.hadoop.io.Text
15: 417680 33414400
org.apache.hadoop.security.token.delegation.web.DelegationTokenIdentifier
16: 417680 16707200
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager$DelegationTokenInformation
17: 79595 14008720 java.net.SocksSocketImpl
18: 112616 7207424 java.lang.ref.Finalizer
19: 79591 7004008 java.net.SocketInputStream
20: 79591 7004008 java.net.SocketOutputStream
21: 414599 6633584 java.lang.Object
22: 79101 5695272 org.apache.tomcat.util.buf.ByteChunk
23: 67540 4322560 org.apache.tomcat.util.buf.CharChunk
24: 61120 3911680 org.apache.tomcat.util.buf.MessageBytes
25: 39328 3885784 [Ljava.lang.String;
26: 84221 3368840 java.util.ArrayList
27: 79610 3184400 java.io.FileDescriptor
{noformat}
was (Author: brahmareddy):
[~pavitheran] thanks for reporting and neat analysis. I do see that ranger_KMS
using more memory and need to do profiling (pasting the heapdump here) , not
sure it might to similar you reported but your analysis make sense to me. (
using ranger-0.7.0 with some patches.)
{noformat}
num #instances #bytes class name
----------------------------------------------
1: 12514991 1125037584 [C
2: 3194026 629310896 [Ljava.util.HashMap$Node;
3: 4547927 421361800 [B
4: 12492061 399745952 java.lang.String
5: 7398577 355131696 java.util.HashMap$Node
6: 3189692 280692896 org.apache.zookeeper.data.Stat
7: 3197676 204651264 java.util.HashMap
8: 3207962 153982176 java.util.concurrent.ConcurrentHashMap$Node
9: 3189693 127587720
org.apache.curator.framework.recipes.cache.ChildData
10: 3190260 76566240 java.util.HashSet
11: 3189714 76553136 java.util.concurrent.atomic.AtomicReference
12: 1349 67582872
[Ljava.util.concurrent.ConcurrentHashMap$Node;
13: 1263345 40427040 org.apache.hadoop.io.Text
14: 23853 35543088 [Ljava.lang.Object;
15: 421113 33689040
org.apache.hadoop.security.token.delegation.web.DelegationTokenIdentifier
16: 421113 16844520
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager$DelegationTokenInformation
num #instances #bytes class name
----------------------------------------------
1: 12494150 1126339552 [C
2: 3189356 561492704 [Ljava.util.HashMap$Node;
3: 4627687 429962248 [B
4: 12470326 399050432 java.lang.String
5: 6811252 326940096 java.util.HashMap$Node
6: 3186259 280390792 org.apache.zookeeper.data.Stat
7: 3194119 204423616 java.util.HashMap
8: 3204325 153807600 java.util.concurrent.ConcurrentHashMap$Node
9: 3186257 127450280
org.apache.curator.framework.recipes.cache.ChildData
10: 3186819 76483656 java.util.HashSet
11: 3186279 76470696 java.util.concurrent.atomic.AtomicReference
12: 1420 67593664
[Ljava.util.concurrent.ConcurrentHashMap$Node;
13: 96175 43101976 [Ljava.lang.Object;
14: 1253046 40097472 org.apache.hadoop.io.Text
15: 417680 33414400
org.apache.hadoop.security.token.delegation.web.DelegationTokenIdentifier
16: 417680 16707200
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager$DelegationTokenInformation
17: 79595 14008720 java.net.SocksSocketImpl
18: 112616 7207424 java.lang.ref.Finalizer
19: 79591 7004008 java.net.SocketInputStream
20: 79591 7004008 java.net.SocketOutputStream
21: 414599 6633584 java.lang.Object
22: 79101 5695272 org.apache.tomcat.util.buf.ByteChunk
23: 67540 4322560 org.apache.tomcat.util.buf.CharChunk
24: 61120 3911680 org.apache.tomcat.util.buf.MessageBytes
25: 39328 3885784 [Ljava.lang.String;
26: 84221 3368840 java.util.ArrayList
27: 79610 3184400 java.io.FileDescriptor
{noformat}
> Ranger KMS DAO memory issues when many new keys are created
> -----------------------------------------------------------
>
> Key: RANGER-3442
> URL: https://issues.apache.org/jira/browse/RANGER-3442
> Project: Ranger
> Issue Type: Bug
> Components: kms
> Affects Versions: 2.0.0
> Reporter: Pavi Subenderan
> Priority: Major
>
> We have many keys created in our KMS keystore and recently we found that when
> we create new keys, the KMS instances easily hit against the memory limit.
> We can reproduce this with a script to call KMS createKey and then
> getMetadata for new keys in a loop. Basically we restart our instances and
> memory usage is approximately 1.5GB out of 8GB, but after running this script
> for a bit (1-5 minutes), we hit close to the 8GB limit and the memory usage
> does not go back down after that.
> I did a heap dump and saw that most of the memory was being retained by
> XXRangerKeystore and eclipse EntityManagerImpl.
> * org.eclipse.persistence.internal.jpa.EntityManagerImpl
> * org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork
> And the largest shallow size object was char[] with 4GB+...
>
> *My fix*
> I was ultimately able to solve this issue by adding an
> getEntityManager().clear() call in BaseDao.java getAllKeys().
> After I added this fix, we can now run as many KMS CreateKey / getMetadata
> calls as we want without any increase in memory usage whatsoever. Memory
> usage now stays constant at <1.7GB.
> My understanding is that Ranger KMS has a many instances of ThreadLocal
> EntityManager (160+ according to my heap dump) which each held a cache of the
> results for getAllKeys. Since we have so many keys in our KMS, this would
> easily put as at the memory limit.
> Not sure if there are any drawbacks to clearing EntityManager in
> BaseDao.getAllKeys() but we are seeing greatly improved performance in our
> case since we aren't constantly hitting the memory limit anymore.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
