I give more details when error occurs:
1) nginx suddently report lots of " Connection timed out" like this
2010/07/09 19:26:37 [error] 11558#0: *3067544729 connect() failed (110:
Connection timed out) while connecting to upstream, ...
2) there are still a few requests can connect tomcat and well processed, but
most requests from nginx is connection timeout. the cpu and net traffic of
tomcat server decreased a lot
3) tomcat acceptor and poller thread is RUNNABLE, but most worker thread is
"TIMED_WAITING"
"http-10.132.23.74-8090-ClientPoller" daemon prio=10 tid=0x00002aab355e2c00
nid=0x5afd runnable [0x0000000044169000..0x0000000044169e20]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00002aaaba1d8c90> (a sun.nio.ch.Util$1)
- locked <0x00002aaaba1d8c78> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaaba1d8a98> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
at java.lang.Thread.run(Thread.java:619)
"http-10.132.23.74-8090-Acceptor-0" daemon prio=10 tid=0x00002aab355df800
nid=0x5afc runnable [0x0000000044068000..0x0000000044068ca0]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
- locked <0x00002aaab8936bd8> (a java.lang.Object)
at
org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
at java.lang.Thread.run(Thread.java:619)
4) there is no "java.lang.OutOfMemoryError" in catalina.out
5) I tried JIO instead of NIO, but error still here
6) my NIO configuration is below:
<Connector port="8090" enableLookups="false" redirectPort="8443"
protocol="org.apache.coyote.http11.Http11NioProtocol"
connectionTimeout="30000" maxThreads="1000" acceptCount="800"
URIEncoding="UTF-8" />
any other reason cause tomcat can't accept requests?
On 10/07/2010 06:32, jikai wrote:
> Hi all:
>
> Out web site use Nginx infront of tomcat6.0.28, nio connector, in
heavy
> load running about 24 hours, tomcat can't accept request, we must restart
> tomcat .
>
> check nginx error log, there are lots of "Connection timed out", print
> tomcat jstack, found most tomcat threads is "TIMED_WAITING", like this:
>
>
>
> "http-10.132.23.74-8090-exec-54" daemon prio=10 tid=0x00002aab36ecec00
> nid=0x3175 waiting on condition [0x00000000498c0000..0x00000000498c0e20]
>
> java.lang.Thread.State: TIMED_WAITING (parking)
>
> at sun.misc.Unsafe.park(Native Method)
>
> - parking to wait for <0x00002aaaba3cf768> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>
> at
>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
>
> at
>
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>
> at
>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>
> at
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
>
> at java.lang.Thread.run(Thread.java:619)
>
>
>
> print jmap, we found we have lots of memory leak:
>
>
>
> num #instances #bytes class name
>
> ----------------------------------------------
>
> 1: 7772927 336536416 [C
>
> 2: 7774657 310986280 java.lang.String
>
> 3: 398536 162602688 com.manu.dynasty.base.domain.User
>
> 4: 3460146 138405840 java.sql.Timestamp
>
> 5: 2334971 112078608 java.util.HashMap$Entry
>
> 6: 792095 101388160
com.manu.dynasty.function.domain.Friend
>
> .
>
> 108: 437 80408 java.net.SocksSocketImpl
>
>
>
> Objects num in memory like "com.manu.dynasty.base.domain.User",
> "com.manu.dynasty.function.domain.Friend" is too big than normal
situation,
>
> And the num of "java.net.SocksSocketImpl" is too small.
>
>
>
> So my question is the memory leak cause tomcat can't create Socket to
accept
> nginx's request?
A memory leak will mean that eventually the JVM is unable to allocate
memory for the creation of any new objects, and yes that will include
network connections.
Work out what's holding onto those objects. Are you storing them in the
user session perhaps?
p
> Our app is running in product envionment now , and every day we have to
> restart tomcat, so any help is very appreciated
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
