Ant
This is the same error seen by Indika on Windows.. and I think my
analysis is correct. If you run the test for the first time or after a
few minutes of running the test last, you should be able to go to
around 1000 iterations. After you start to hit this issue, even 200
iterations would give you the error. At this time, doing a netstat -na
should show you that most of the tcp ports are in TIME_WAIT state.
Usually it could take at least one minute till a port is cleared up by
the OS. The tuning parameters I specified for Linux tells the OS to use
the full port range for applications, and to set the tcp fin timeout to
30 secs - to clear up the ports as quickly as possible. Without *any*
OS tuning and on a Windows XP system - you definitely will encounter
this issue.
asankha
ant elder wrote:
I've tried again with the latest Synapse and HTTP
components code and several JVMs. The results feel slightly different
than before but the end result is still always the root exception
included below. Sometime it doesn't occur till around 1000 requests,
but sometimes it happens after not many requests at all.
...ant
java.io.IOException: Unable to establish loopback connection
at sun.nio.ch.PipeImpl$Initializer.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.nio.ch.PipeImpl.<init>(Unknown Source)
at sun.nio.ch.SelectorProviderImpl.openPipe(Unknown Source)
at java.nio.channels.Pipe.open(Unknown Source)
at
org.apache.axis2.transport.nhttp.ServerHandler.requestReceived
(ServerHandler.java:108)
at
org.apache.axis2.transport.nhttp.LoggingNHttpServiceHandler.requestReceived(LoggingNHttpServiceHandler.java:83)
at
org.apache.http.impl.nio.DefaultNHttpServerConnection.consumeInput
(DefaultNHttpServerConnection.java:96)
at
org.apache.axis2.transport.nhttp.PlainServerIOEventDispatch.inputReady(PlainServerIOEventDispatch.java:67)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable
(BaseIOReactor.java:68)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:160)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java
:145)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:127)
at
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java
:153)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.BindException: Address already in use: connect
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect
(Unknown Source)
at java.nio.channels.SocketChannel.open(Unknown Source)
On 3/23/07, Asankha C. Perera <
[EMAIL PROTECTED]> wrote:
Ant
I am quite sure that the problem seen by Indika now was related to the
ports being exhausted - see the following articles and esp. the
"MaxUserPort" and "TcpTimedWaitDelay" parameters that could tweaked -
to be consistent with what I am using before running a load test on
Linux. I will ask Indika to check these on Monday - but you may try
this in the meantime if you get a chance
http://www.microsoft.com/technet/network/deploy/depovg/tcpip2k.mspx
http://www.microsoft.com/technet/community/columns/cableguy/cg1205.mspx
http://www.psc.edu/networking/projects/tcptune/OStune/winxp/winxp_stepbystep.html
asankha
Asankha C. Perera wrote:
Hi Ant
I fixed this for Linux and JDK 1.5 - I am confident of this fix as I
was able to first recreate the issue consistently and then see the fix
in action using 5 concurrent users sending a total of 5000 messages
multiple times. However Indika is still seeing a 'similar' issue in
Windows using JDK 1.4. We will try to see if its related to JDK 1.4 or
Windows. If you get the latest nhttp code and build the nhttp JAR you
could verify this fix - and let me know.
I am listing some of the linux commands that came in handy for the
resolution incase someone wants to check this.
lsof -p 7426 => lists the open files for the pid given after the -p
option
ls -l /proc/9976/fd | wc -l => for each process the /proc filesystem
lists the files used and thus you could count the open files with this
command
asankha
Asankha C. Perera wrote:
Ant / Oleg
I can recreate this issue on both Windows and Linux and think its
caused by my code related to use of Pipes.. and I am actively looking
into this right now.. will get back to you on what I find.
asankha
ant elder wrote:
I've tried on several JDKs now and _always_ get similar
intermittent I/O related errors. I can use JMeter directly against
Axis2-1.1.1 without any problems at all, so this does look like some
issue with the NIO transport. Be really good to hear from other Windows
users to see if this is just my specific environment or a more general
problem problem.
To recreate:
1) build Synapse server sample by running 'ant' in the
samples\axis2Server\src\SimpleStockQuoteService directory
2) start the sample service by running
samples\axis2Server\axis2server.bat
3) get the Synapse config (either 8 or 501) from http://people.apache.org/~antelder/temp/,
put in repository\conf\sample and start syanps: bin\synapse.bat
-sample=8
4) get the JMeter config test1.jmx from http://people.apache.org/~antelder/temp/,
start Jmeter and File -> Open and point to the test1.jmx file
5) JMeter Run -> Start and after not to long IO errors should appear
in the Syanpse console
...ant
---------- Forwarded message ----------
From: Asankha
C. Perera <[EMAIL PROTECTED]>
Date: Mar 22, 2007 4:58 PM
Subject: Re: [jira] Resolved: (HTTPCORE-60) Transport appears to be
hanging because an unchecked exception caused the I/O dispatch thread
to terminate
To: HttpComponents Project <
[email protected]>
Oleg/Ant
I am guessing this is something to do with Windows or the JDK you use..
But I am unable to test this week, so will try to my best to try this
sometime next week. As I said, on Linux I have run the system through
thousands of messages and multiple threads concurrently and have fixed
all the issues I came across.
So Oleg, I do not see this as a blocker for the HttpCore
release - but
I will use your latest snapshots in Synapse to check on this in future
if it occurs again
thanks
asankha
Oleg Kalnichevski (JIRA) wrote:
[ https://issues.apache.org/jira/browse/HTTPCORE-60?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Kalnichevski resolved HTTPCORE-60.
---------------------------------------
Resolution: Fixed
Anthony
It turned out ClosedChannelException is a checked I/O exception so it cannot kill the I/O dispatch thread. So, apparently I was wrong in my initial assertion about the cause of the Synapse I/O transport lockup. I tweaked HttpCore code a little and changed the IOSessionImpl to catch all ChannelClosedException-s thrown by the underlying byte channel just in case.
Please review the changes and let me know if it is okay to proceed with the release
Oleg
Transport appears to be hanging because an unchecked exception caused the I/O dispatch thread to terminate
----------------------------------------------------------------------------------------------------------
Key: HTTPCORE-60
URL: https://issues.apache.org/jira/browse/HTTPCORE-60
Project: HttpComponents Core
Issue Type: Bug
Affects Versions: 4.0-alpha4
Reporter: ant elder
Assigned To: Oleg Kalnichevski
Fix For: 4.0-alpha4
See discussion on synapse-dev mailing list: http://www.nabble.com/Intermittent-IO-Errors-using-Synapse-tf3439957.html
The transport appears to be hanging because an unchecked exception
caused the I/O dispatch thread to terminate. I believe there are several
different types of problems (at least two) that we are seeing here.
[I/O reactor worker thread 5] ERROR ServerHandler - I/O Error : null
java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:112)
at
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java
:139)
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
|
