> -----Original Message----- > From: Mark Thomas [mailto:ma...@apache.org] > Sent: Friday, November 01, 2013 7:16 PM > To: Tomcat Users List > Subject: Re: attempting to achieve 100K concurrent websocket connections on > Tomcat 7.0.48 NIO > > On 01/11/2013 22:00, Bob DeRemer wrote: > > QUESTION: I'm looking for some advice on what Tomcat NIO connector > > settings to use to support 100K concurrent websocket connections. > > Hopefully I can reach this goal through a combination of Tomcat NIO > > Connector settings, and Server 2008 R2 configuration [if needed]. > > > > BACKGROUND: We're scale testing our websocket application and looking > > to see how many concurrent websocket connections we can get on a > > single Tomcat instance - with the goal being 100K. I've provided the > > test landscape details at the bottom - all VERY BIG EC2 instances over > > 10 GB network, so memory, CPU and network do not appear to be the > > problem when monitoring. > > > > PROBLEM: The problem we are running into is that we can't seem to > > establish even 50K connections into Tomcat. At some point, we start > > getting connect failures, similar to the following: > > OK. Win 2k8 R2, 16GB RAM, bunch of other stuff running, client and server on > same machine, no tuning. I get 16,313 connections before it falls over. That > is > consistent with the defaults which should allow a maximum of 16384 (given I > have other stuff running). > > (I'll commit the test to trunk shortly). > > After > netsh int ipv4 set dynamicport tcp start=10000 num=55536 > > I got as far as 25121 before I hit GC issues. > > -Xmx12G -Xms12G fixed the GC problems (well, I say fixed allowed them to be > ignored would be closer) > > The next run got to 55464 connections which looks to be about the limit of the > ephemeral ports. > > > I'm hoping that someone may be able to advise what changes we might > > make to the following Tomcat NIO connector setting that will allow > > upwards of 100K websocket connections: > > > > <Connector port="80" > > protocol="org.apache.coyote.http11.Http11NioProtocol" > > connectionTimeout="20000" maxConnections="100000" > > maxThreads="100000" redirectPort="8443" /> > > As Chris said the threads are probably too high. > > You can set maxThreads="-1" for unlimited connections (worth doing for > testing). > > > TEST LANDSCAPE: > > > > The test landscape is all Windows Server 2008 R2 boxes running in EC2 > > and the Test Client environment: > > > > Our test client is a multi-threaded java client that makes use of the > > JSR356 ClientEndpoint functionality. We're creating 40K+ websocket > > connections from a single test client machine. The test client is > > Server 2008 R2 and we have configured it to allow 50K ephemeral ports, > > so we should be able to establish 40K+ outbound websocket connections. > > > > Server environment: > > > > * EC2 instance (cc2.8xlarge) (60 GB, 10 GB network, 16 > > vCPUs) > > > > * Server 2008 R2 > > > > * Tomcat 7.0.48 (trunk) > > > > * Java 1.7.0_45 > > Look at your memory usage. My guess based on my simple test results is that > the JVM for both Tomcat and the client isn't grabbing as much memory as it > really needs. If that is the problem then -Xms20G -Xmx30G should do the trick. > > Keep in mind that Tomcat is allocating at several 8k buffers for each > connection (client and server - I'd need to check the code to be sure how many > buffers are allocated per connection) so you are going to need a fair amount > of > RAM. The TCP buffers will need quite a lot of space too but that should be > outside of the Java Object Heap.
In monitoring the memory, it's not nearing the max available we configured for the JVM, but perhaps there's something going on GC wise. I don't have GC logging enabled and probably should. I haven't done much with that aspect of tuning yet. >From a Tomcat connector standpoint, should we consider increasing any of the >acceptThreadCount and acceptCount values - given it's such a large machine? > > Mark > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
