Hi Mark (et al),
We're seeing really high CPU utilization with just a 1000 active websockets
that are sending/receiving 100 byte binary messages. The profiling snippet
below was with 1000 connections. If we start to increase the number of
connections, the time spent in the NIO logic starts to consume the majority of
the processing. As a result, I'm trying to verify:
* Is this something that can be tuned/improved using Tomcat's NIO
settings?
* Is this a bug in Tomcat 7 (websocket/NIO)?
* Are we hitting some issue with 1000+ highly active NIO sockets?
The nature of this application is that it's primary functionality is to process
binary messages on (ideally 10 - 20K websockets), forward them to another
server for processing, then send the responses back. As a result, the
performance of the websocket communication is a critical factor for success.
Please let me know if you'd like to discuss this - I'm available any time.
Thanks,
Bob
We have been profiling this in YourKit and as we increase the number of
websockets, naturally we see the CPU utilization increase. The concern is that
we're seeing this at what seems to be a fairly low number of websockets. The
YourKit profiling showed 41% of the time being spent in the following code. My
apologies for the formatting. I can provide an attachment out-of-band if it
would be helpful.
| +---org.apache.tomcat.util.net.NioEndpoint$Poller.run()
| 38,594 41 % | 0 |
| | |
| | |
| | +---sun.nio.ch.SelectorImpl.select(long)
| 25,615 27 % | 0 |
| | | |
| | |
| | | +---sun.nio.ch.SelectorImpl.lockAndDoSelect(long)
| 25,615 27 % | 0 |
| | | |
| | |
| | | +---sun.nio.ch.WindowsSelectorImpl.doSelect(long)
| 25,615 27 % | 46 |
| | | |
| | |
| | | +---sun.nio.ch.WindowsSelectorImpl.resetWakeupSocket()
| 13,681 14 % | 1,107 |
| | | |
| | |
| | |
+---sun.nio.ch.WindowsSelectorImpl$StartLock.access$300(WindowsSelectorImpl$StartLock)
| 10,826 11 % | 0 |
| | | |
| | |
| | | +---sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys()
| 1,029 1 % | 0 |
| | | |
| | |
| | | +---sun.nio.ch.SelectorImpl.processDeregisterQueue()
| 31 0 % | 0 |
| | |
| | |
| |
+---org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(SelectionKey,
NioEndpoint$KeyAttachment) | 11,996 13 % | 31 |
| | | |
| | |
| | | +---org.apache.tomcat.util.net.NioEndpoint.processSocket(NioChannel,
SocketStatus, boolean) | 11,778 12 % | 0 |
| | | | |
| | |
| | | |
+---org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(Runnable)
| 11,746 12 % | 0 |
| | | | | |
| | |
| | | | |
+---org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(Runnable, long,
TimeUnit) | 11,746 12 % | 0 |
| | | | | |
| | |
| | | | | +---java.util.concurrent.ThreadPoolExecutor.execute(Runnable)
| 11,746 12 % | 15 |
| | | | | |
| | |
| | | | | +---org.apache.tomcat.util.threads.TaskQueue.offer(Object)
| 11,731 12 % | 0 |
| | | | | |
| | |
| | | | |
+---org.apache.tomcat.util.threads.TaskQueue.offer(Runnable)
| 11,731 12 % | 46 |
| | | | | |
| | |
| | | | |
+---java.util.concurrent.LinkedBlockingQueue.offer(Object)
| 11,668 12 % | 15 |
Bob DeRemer
Senior Director, Architecture and Development
[Description: Description: Description: Description:
cid:image001.png@01CBE3DE.51A12030]
http://www.thingworx.com<http://www.thingworx.com/>
Skype: bob.deremer.thingworx
O: 610.594.6200 x812
M: 717.881.3986
