I suspect what you are seeing is a cache, perhaps try dropping cache ' echo 3 > /proc/sys/vm/drop_caches' would help.
If Knox process was leaking it would have OOMed but that does not mean that it is strictly staying within the -Xms and -Xmx bounds [1] as you rightly pointed this could be because of the thread stacks and native handles. I would suggest keeping Xms and Xms sizes same. [1] https://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/geninfo/diagnos/garbage_collect.html On Tue, Feb 6, 2018 at 9:27 AM, Vaibhav Kulkarni <[email protected]> wrote: > Hi Sandeep, > > Yes, actually, I did try that. > > APP_MEM_OPTS > = "-XX:ParallelGCThreads=8 -XX:NewSize=512m -XX:MaxNewSize=1024m > -XX:+UseConcMarkSweepGC -Xloggc:$APP_HOME_DIR/logs/gc.log-`date > +'%Y%m%d%H%M'` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps > -XX:+PrintGCDateStamps -Xms1024m -Xmx2048m -Xss512k > -XX:HeapDumpPath=$APP_HOME_DIR/logs -XX:+HeapDumpOnOutOfMemoryError -XX: > CMSInitiatingOccupancyFraction=60 -XX:+UseCMSInitiatingOccupancyOnly > -XX:MaxDirectMemorySize=1024m" > > > But, despite that the memory used by knox process keeps increasing and > goes well beyond the Java heap size. So, I definitely think this is a > memory leak. But, not sure what is the best way to figure out where the > leak is happening. It is possible that the leak might be happening in one > of the native libraries (pam etc) . Any pointers will help > > Thanks, > > > > > On Tue, Feb 6, 2018 at 7:34 PM, Sandeep Moré <[email protected]> > wrote: > >> Hello Vaibhav, >> >> Did you try setting memory options for Knox ? >> i.e. setting APP_MEM_OPTS in gateway.sh. >> >> Sometimes, in cases of large memory GC does not run as frequently as it >> should. >> >> Best >> Sandeep >> >> On Tue, Feb 6, 2018 at 6:49 AM, Vaibhav Kulkarni <[email protected]> >> wrote: >> >>> Hi, >>> >>> >>> I am using Knox server gateway (version 0.12.0) with Hortonworks >>> HDP-2.6.2.0 . There seems to be a memory leak with the knox process >>> because of which it keeps acquiring more and more memory. My test workload >>> does very frequent webhdfs rest apis from hundreds of connections. When the >>> workload is executing, I notice that the knox gateway process memory keeps >>> increasing. >>> >>> >>> >>> PID VSZ SZ RSS %CPU TIME CMD >>> >>> 24703 44988224 11247056 2195740 78.8 01:37:02 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/ga >>> >>> teway.jar >>> >>> 24703 44988224 11247056 2195740 78.8 01:37:03 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/ga >>> >>> teway.jar >>> >>> 24703 44989420 11247355 2335764 84.3 01:44:13 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/ga >>> >>> teway.jar >>> >>> 24703 44989576 11247394 2326344 89.5 01:51:04 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/ga >>> >>> teway.jar >>> >>> 24703 44989744 11247436 2345940 94.8 01:58:06 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/ga >>> >>> teway.jar >>> >>> 24703 44989908 11247477 2515100 100 02:05:13 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990072 11247518 2653252 105 02:12:18 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990244 11247561 2777268 110 02:19:26 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990324 11247581 2878924 113 02:23:04 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990324 11247581 2878924 112 02:23:04 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990324 11247581 2878872 112 02:23:04 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990324 11247581 2878872 111 02:23:04 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990324 11247581 2878792 111 02:23:05 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990324 11247581 2878792 110 02:23:05 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990324 11247581 2878748 110 02:23:05 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990324 11247581 2878748 110 02:23:05 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990324 11247581 2878660 109 02:23:05 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990324 11247581 2878660 109 02:23:05 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990324 11247581 2878580 108 02:23:05 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990324 11247581 2878580 108 02:23:05 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990324 11247581 2878500 107 02:23:05 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990324 11247581 2878500 107 02:23:05 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990324 11247581 2878392 107 02:23:05 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990324 11247581 2878392 106 02:23:05 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990324 11247581 2878348 106 02:23:05 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990324 11247581 2878348 105 02:23:05 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990324 11247581 2878268 105 02:23:06 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990324 11247581 2878268 105 02:23:06 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990324 11247581 2878124 104 02:23:06 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990324 11247581 2878124 104 02:23:06 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990324 11247581 2878016 104 02:23:06 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> 24703 44990324 11247581 2878016 103 02:23:06 >>> /usr/jdk64/jdk1.8.0_112/bin/java >>> -Djava.library.path=/usr/hdp/current/knox-server/ext/native >>> -jar /usr/hdp/current/knox-server/bin/gat >>> >>> eway.jar >>> >>> >>> >>> Has anyone seen this behaviour with knox ? >>> >>> >>> Following is my Knox topology, I am using LDAP authentication >>> >>> >>> >>> <topology> >>> >>> >>> <gateway> >>> >>> >>> <provider> >>> >>> <role>authentication</role> >>> >>> <name>ShiroProvider</name> >>> >>> <enabled>true</enabled> >>> >>> <param> >>> >>> <name>sessionTimeout</name> >>> >>> <value>30</value> >>> >>> </param> >>> >>> <param> >>> >>> <name>main.ldapRealm</name> >>> >>> <value>org.apache.hadoop.gatew >>> ay.shirorealm.KnoxLdapRealm</value> >>> >>> </param> >>> >>> <param> >>> >>> <name>main.ldapRealm.userDnTemplate</name> >>> >>> <value>uid={0},ou=people,dc=ha >>> doop,dc=apache,dc=org</value> >>> >>> </param> >>> >>> <param> >>> >>> <name>main.ldapRealm.contextFactory.url</name> >>> >>> <value>ldap://{{knox_host_name}}:33389</value> >>> >>> </param> >>> >>> <param> >>> >>> <name>main.ldapRealm.contextFa >>> ctory.authenticationMechanism</name> >>> >>> <value>simple</value> >>> >>> </param> >>> >>> <param> >>> >>> <name>urls./**</name> >>> >>> <value>authcBasic</value> >>> >>> </param> >>> >>> </provider> >>> >>> >>> <provider> >>> >>> <role>authorization</role> >>> >>> <name>AclsAuthz</name> >>> >>> <enabled>true</enabled> >>> >>> <param> >>> >>> <name>knox.acl</name> >>> >>> <value>admin;*;*</value> >>> >>> </param> >>> >>> </provider> >>> >>> >>> <provider> >>> >>> <role>identity-assertion</role> >>> >>> <name>Default</name> >>> >>> <enabled>true</enabled> >>> >>> </provider> >>> >>> >>> </gateway> >>> >>> >>> <service> >>> >>> <role>KNOX</role> >>> >>> </service> >>> >>> >>> </topology> >>> >>> >>> Thanks, >>> >>> Vaibhav Kulkarni >>> >>> >> >
