Dear Silvio Thank you for your reply.
I didn't think about kernel logs. I will check more logs (kernel output and others). Thanks, Toshimichi Naito ________________________________________ 差出人: [email protected] <[email protected]> が の代理で送信Silvio Bierman <[email protected]> 送信日時: 2017年7月25日 18:08 宛先: [email protected] 件名: Re: [jetty-users] Need help with finding logs when Jetty killed Dear Toshimichi, I would start with checking /var/log/syslog and look for a kernel panic. Under extreme situations the kernel may decide it needs to kill one or more resource hungry processes. I have seen many Java processes being killed because of out of memory situations, especially if the maximum heap size (-Xmx...) is too large considering the available physical memory and possible memory demands from other applications. Lacking that the only situations where a Java server process would suddenly die on me was when I employed a third party library that decided to call System#exit under some error conditions. That was quite a puzzler... Good luck, Silvio On 25-07-17 10:47, [email protected] wrote: > Hi, > > We are using Jetty 9.3.9.v20160517. > Sometimes, Jetty's process killed. > > We checked following logs, but couldn't find the cause. > - /var/log/messages > -- can't find oom killer or signal messages. > - /var/jetty/logs/*.stderrout.log > -- no exception messages. > - /var/jetty/logs/*.request.log > -- everytime final request didn't make response, but no relation on each > endpoints. > - application custom logs > -- when hundle uncaught exception, put stacktrace on log file. but no logs. > > I tried some JAVA_OPTIONS settings and finally Jetty looks running stably. > Probably "CompressedClassSpaceSize" or "MetaspaceSize" was too small. > Because after increase two parameter's value, Jetty isn't killed. > > I have a question. > - Is there no log when Jetty killed by "CompressedClassSpaceSize" and/or > "MetaspaceSize" shortage? > - How can I find evidence that "CompressedClassSpaceSize" and/or > "MetaspaceSize" is wrong? > - Is there a possibility that these guesses are wrong and this is Jetty's > issue? > > > Environment is follows: > - aws ec2 t2.2xlarge(4vCPU, 16GiB), Amazon linux 2017.03 > - jetty-distribution-9.3.9.v20160517 > - OpenJDK Runtime Environment (build 1.8.0_131-b11) > - hosts 3 web application(all apps are using Spring FW 4.1.7) > > seting files are follows: > ---------------- > # cat /etc/default/jetty > JETTY_HOME=/opt/jetty > JETTY_BASE=/var/jetty > JETTY_RUN=/var/run > > JAVA_OPTIONS="\ > -Xms16384m \ > -Xmx16384m \ > -XX:CompressedClassSpaceSize=80m \ > -XX:MetaspaceSize=128m \ > -XX:MaxMetaspaceSize=192m \ > -verbose:gc \ > -Xloggc:/tmp/jetty.gc.`date +%Y%m%d-%H%M%S`.log \ > -XX:+UseGCLogFileRotation \ > -XX:GCLogFileSize=10M \ > -XX:NumberOfGCLogFiles=5 \ > -XX:+PrintGC \ > -XX:+PrintGCDetails \ > -XX:+PrintGCDateStamps \ > -XX:+PrintGCTimeStamps \ > -XX:+PrintClassHistogram \ > -XX:+HeapDumpOnOutOfMemoryError \ > -XX:HeapDumpPath=/tmp \ > -XX:+ExitOnOutOfMemoryError \ > -XX:ErrorFile=/tmp/jetty.hs_err_pid%p.log \ > -XX:+UseConcMarkSweepGC \ > -XX:CMSInitiatingOccupancyFraction=70 \ > -XX:+UseCMSInitiatingOccupancyOnly \ > -Dspring.profiles.active=stg \ > -Dfile.encoding=UTF-8 \ > " > ---------------- > new CompressedClassSpaceSize is 200m > new MaxMetaspaceSize is 512m > > ---------------- > # cat /var/jetty/start.ini > jetty.server.dumpBeforeStop=true > --module=home-base-warning > --module=ext > --module=resources > --module=server > jetty.threadPool.minThreads=10 > jetty.threadPool.maxThreads=100 > jetty.httpConfig.sendServerVersion=false > --module=http > jetty.http.host=0.0.0.0 > --module=deploy > jetty.deploy.monitoredDir=webapps > jetty.deploy.scanInterval=0 > jetty.deploy.extractWars=true > --module=jsp > --module=requestlog > jetty.logging.dir=${jetty.base}/logs > jetty.requestlog.dir=/logs > jetty.requestlog.filePath=${jetty.requestlog.dir}/yyyy_mm_dd.request.log > jetty.requestlog.filenameDateFormat=yyyy_MM_dd > jetty.requestlog.timezone=JST > --module=debug > jetty.debug.timezone=JST > --module=setuid > ---------------- > > > Thanks, > Toshimichi Naito > _______________________________________________ > jetty-users mailing list > [email protected] > To change your delivery options, retrieve your password, or unsubscribe from > this list, visit > https://dev.eclipse.org/mailman/listinfo/jetty-users _______________________________________________ jetty-users mailing list [email protected] To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/jetty-users _______________________________________________ jetty-users mailing list [email protected] To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/jetty-users
