High CPU Consumption with Tomcat 8.5.4 NIO and SSL

[Chen Levy]

Hello All

Since Tomcat 8.0.x development is about to be stopped, we’ve decided to switch 
to 8.5.4, but since are facing the following issue:
A few hours after startup, the Java process starts consuming 100% CPU, and 
while Tomcat is still responsive, it is very slow. This is consistent, been 
going on for a couple of weeks and is only resolved each time by restarting 
Tomcat.

I’ve tried a few things:
Reverted back to Tomcat 8.0.33 -> problem solved
Switched from NIO to NIO2 -> problem solved, but other issues appeared with 
connections ending prematurely, so not a valid option
Switched to the new connector configuration scheme -> problem persists

Setup: 
Current Amazon Linux, Tomcat 8.5.4, Java 1.8.0_92-b14, no APR

      <Connector port="443" 
protocol="org.apache.coyote.http11.Http11NioProtocol"
                   maxThreads="500" minSpareThreads="25"
                   enableLookups="false" disableUploadTimeout="true"
                   connectionTimeout="10000"
                   compression="on"
                   compressionMinSize="2048"
                   
compressableMimeType="text/html,text/xml,text/json,text/javascript,text/css,text/plain,application/javascript,application/x-javascript,application/xml"
                   SSLEnabled="true" scheme="https" secure="true">
            <SSLHostConfig protocols="TLSv1,TLSv1.1,TLSv1.2">
                <Certificate certificateKeystoreFile="conf/tomcat.keystore"
                             certificateKeyAlias="tomcat"
                             certificateKeystorePassword="tomcat"
                             certificateKeystoreType="PKCS12"/>
            </SSLHostConfig>
        </Connector>


Analyzing the threads, there’s one which takes all of the CPU time:

"https-jsse-nio-8443-ClientPoller-0" daemon prio=5 RUNNABLE
        sun.misc.Unsafe.unpark(Native Method)
        java.util.concurrent.locks.LockSupport.unpark(LockSupport.java:141)
        
java.util.concurrent.locks.AbstractQueuedSynchronizer.unparkSuccessor(AbstractQueuedSynchronizer.java:662)
        
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1264)
        java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)
        
java.util.concurrent.LinkedBlockingQueue.signalNotEmpty(LinkedBlockingQueue.java:176)
        
java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:430)
        org.apache.tomcat.util.threads.TaskQueue.offer(TaskQueue.java:74)
        org.apache.tomcat.util.threads.TaskQueue.offer(TaskQueue.java:31)
        
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1361)
        
org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:163)
        
org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:141)
        
org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:828)
        
org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(NioEndpoint.java:850)
        org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:826)
        java.lang.Thread.run(Thread.java:745)


This is similar to an issue on a Windows machine reported here:
http://marc.info/?l=tomcat-user&m=147082413626214&w=2

I’ll appreciate any insight or advice
Thanks in advance
Chen