-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Mike,
On 8/10/16 6:15 AM, Mike Noordermeer wrote: > Hi, > > After an upgrade to Tomcat 8.5, we are experiencing an issue where > Tomcat starts generating a high CPU load (100%), probably after an > HTTP network scan. The bug seems to be related to Windows, NIO and > possibly SSL. I have a Yourkit dump and several thread dumps that > show the issue, and was wondering if anyone is interested in this, > and if we can gather any extra information to help debug this > issue. > > Setup: Windows 2k8r2, Tomcat 8.5.4, Java 8u102, NIO HTTP and NIO > JSSE HTTPS connector. > > Out of nothing, Tomcat starts using 100% CPU. I made some thread > dumps, available here: > > https://gist.github.com/MikeN123/f4a85f09231cfda7a9e632b64f27dcdc > https://gist.github.com/MikeN123/7dfe17ae95b8d516d86e0d7126cbaa02 > https://gist.github.com/MikeN123/750da8580e04e0498f70b81dbd1a5c52 > https://gist.github.com/MikeN123/2e83307b7c1216339d4fa73b30c02f1a > https://gist.github.com/MikeN123/8850ef2a60d39a4dc140b2d8fef18c3f > > I also have some Yourkit stats available, but as these may contain > confidential information, I won't share them in public. Basically, > what we see is that the thread https-jsse-nio-443-ClientPoller-0 > is continuously runnable and using CPU on > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(), and various > other https-jsse-nio-443-exec threads are waiting (parked) or > running. These threads together take up all the CPU. A Yourkit > thread view showing the issue starting around 11:02: > https://dl.eveoh.nl/yc_fal.png > > We _suspect_ the issue is triggered by an HTTP scan, which > generates the following requests in the access log, but we are > still trying to confirm this: > https://gist.github.com/MikeN123/581d1f17aae100f06b8c65b86870a64a > > Also, we are trying to confirm whether or not NIO2 shows the same > behaviour. > > The behaviour seems to be the same as in this tomcat-users thread: > https://mail-archives.apache.org/mod_mbox/tomcat-users/201604.mbox/%3C CAE-ydNF84pnoX2tP8BJ4vQisabgycP0y2vpnmjNhddz9+BKp=w...@mail.gmail.com%3E > > A similar issue is mentioned for some other products, but I'm not > sure if there's a relation: > > https://bugs.eclipse.org/bugs/show_bug.cgi?id=357240 > https://developer.jboss.org/thread/240618?start=0&tstart=0 > https://github.com/netty/netty/issues/3857 > > Our next steps are: > > - Switching the production site to NIO2, to see if that fixes the > issue - Checking if we can reproduce the issue by triggering the > HTTP vulnerability scan manually > > Any ideas or requests for more information are more than welcome. Are you fronting with a web server/reverse proxy? Those "-" requests looks suspiciously like the kinds of requests that Apache httpd makes to itself to verify that worker threads are still available for certain things. Maybe that's a way that HTTP scanners are trying to avoid detection: by looking like "normal" stuff in the logs. I'm curious... why are the requests coming from "10.xxx"... isn't that within your own network? Shouldn't you KNOW what that stuff is? - -chris -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJXq0xpAAoJEBzwKT+lPKRYu/sP/RdIRuBnSq36wHr8eYj6IAc9 xJmlcjdfBB4CL7DYwDvH+fddDb/W8k48ksQIk/nhvrqf0mP7V7g21L9pS3Cl/ki4 i/pKR6WhxPEkcKGZh7uxlD6bPUpJCVcvDX0W8jfl+yXXX5eJKVWwrfJHxZ7i4hEL c8Vz/JUyXfAATWbfkiBGRhrcSqA3QG9WD1D6XZxAl6saJgqUD2tn7CbXJ+XG6GZB 563qD/fBg20FG5xq5atXhnjAl4icmhIEcaszjsgNb7cwzwPKV2PBcgF+V6ObGvHA +5v0by0jGpaX6I5wozjVMIOpjdcQLFApimdDS4yBaqP400jKrFsbwVVjYn1E08oK n9lxW2yfzC1l9AmbnHa2OcpRSyFVV9XyZdgkElVETz8XQbJu+E18gK9eTKufweYA PDUVmNwXg/WaCNpZtuG8kvgsP93v4mfTzsru9I5ktMSdSti5cJPtrkHGnZicBZNB 5oWQzgtc38exUyJg9WKalAZMNiAR1WlH4mxCCDmS8TAUd0JzKexRRsQYRDKYQCJY SU3iW5zvj5n1dcloMdyx1BbP6pGIK/FvP9Jxm4oKhDfN4WnrFHa/eYWam1B7lpO/ sgu7KJe815zgGOZS8H5cstegY2SAsi48zG1JLQut+WZE/2fBkozZVqdj/D/ybd4n WPi5Lf4v5RcbL5w1f6OB =19NA -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
