[
https://issues.apache.org/jira/browse/RANGER-4147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17707800#comment-17707800
]
Vikas Kumar commented on RANGER-4147:
-------------------------------------
Hi [~madhan] , [~bpatel] ,
Just thought to share my observation, please see if it is useful as we also
faced similar scenario( not exactly same ) in one of the environment, where
client was getting connection timeout and KMS service was getting restarted
abruptly.
I do agree that CPU usage of 50 % is normal, but I seriously doubt the memory
part. In our case also, 8GB was not sufficient and we were getting OOM and we
increased to higher memory. But it didn't solve the problem. Further we found
that culprit was not kms service logic but ZK delegation tokens ( kms depends
on hadoop-commmon lib for that) that were being stored in-memory in a map and
it was consuming around 90% of Heap's Tenure GEN.
Second observation was the container threads, all request first goes to
authentication and all threads were BLOCKED at one point where it either reads
from token map to verify token or read from ZK. In one of the thread dump,
around 80% container threads were blocked at that point. In such cases also,
connection will not be available and client may get timeout.
We were not able to get heap dump on OOM ( even this is set
-XX:+HeapDumpOnOutOfMemoryError).
[~bpatel] , if environment is in your control, can you please share following:
# have you checked if KMS service is getting rebooted when client got timeout?
# Are you getting heap dump when KMS is crashing, due to
-XX:+HeapDumpOnOutOfMemoryError
# Thread dumps when load is high.
# CPU usage, how you are finding CPU load. Is it "top" command ? Because top
command shows cpu percentage per core, like if it shows 200%, that means
current process is utilising two cpu cores to their 100% limit, not all cores.
> Ranger KMS consume 50% of CPU memory
> -------------------------------------
>
> Key: RANGER-4147
> URL: https://issues.apache.org/jira/browse/RANGER-4147
> Project: Ranger
> Issue Type: Bug
> Components: kms
> Affects Versions: 2.3.0
> Environment: secured
> Reporter: Bhavik Patel
> Priority: Critical
>
> Ranger KMS consume 50% of CPU memory and many time it consume even 150%
> cc: [~dhavalshah9131] [~kirbyzhou]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
