We have a service set up running in tomcat 7.0.26 with cxf version 2.5.2.
currently it is set up on two hosts behind a hardware load balancer.
it is working flawlessly when hit by a php client using the standard
php soap library and when each server is hit directly with a java cxf
client.
when a java cxf client hits the service via the load balancer, we
intermittently see very long lags and the many times, the following
exception:
java.lang.RuntimeException: Couldn't parse stream.
at
org.apache.cxf.staxutils.StaxUtils.createXMLStreamReader(StaxUtils.java:1181)
at
org.apache.cxf.interceptor.StaxInInterceptor.handleMessage(StaxInInterceptor.java:104)
at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:263)
at
org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:123)
at
org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:207)
at
org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:213)
at
org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:193)
at
org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:126)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:185)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:108)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:641)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:164)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:197)
at
net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:171)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:579)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:307)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: com.ctc.wstx.exc.WstxIOException: Read timed out
at
com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:536)
at com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:585)
at com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:610)
at
com.ctc.wstx.stax.WstxInputFactory.createXMLStreamReader(WstxInputFactory.java:316)
at
org.apache.cxf.staxutils.StaxUtils.createXMLStreamReader(StaxUtils.java:1179)
... 31 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:146)
at
org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:532)
at
org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501)
at
org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:563)
at
org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:118)
at
org.apache.coyote.http11.AbstractInputBuffer.doRead(AbstractInputBuffer.java:326)
at org.apache.coyote.Request.doRead(Request.java:422)
at
org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:290)
at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:431)
at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:315)
at
org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:200)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at com.ctc.wstx.io.BaseReader.readBytes(BaseReader.java:155)
at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:368)
at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:111)
at
com.ctc.wstx.io.ReaderBootstrapper.initialLoad(ReaderBootstrapper.java:250)
at
com.ctc.wstx.io.ReaderBootstrapper.bootstrapInput(ReaderBootstrapper.java:133)
at
com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:531)
... 35 more
we have checked the contents of the request by comparing a tcp dump
that the client sends vs what is received on the server on the other
side of the load balancer and they match.
again, the cxf client works perfectly when hitting each machine
directly. it is only when going through the hardware load balancer
that we see the issue, yet we've verified that the actual contents of
the request passes through the load balancer unchanged.
any advice on what may be going on here and how to resolve it would be
very helpful.
thanks
