Re: Errors with NIO processor

2011-11-16 Thread Mark Thomas 
On 15/11/2011 17:00, Matthew Tyson wrote:
> Hey Guys,
> 
> We are seeing the following errors (in production of course, testing didn't
> reveal this) after switching to NIO protocol.
> 
> This is Tomcat 7.0.22 on CentOS 6.  There is a load balancer sending only
> comet traffic to port 8080, where the NIO protocol is used.
> 
> Nov 15, 2011 8:39:29 AM org.apache.tomcat.util.net.NioEndpoint processSocket
> SEVERE: Error allocating socket processor
> java.lang.NullPointerException
> Nov 15, 2011 8:39:51 AM org.apache.tomcat.util.net.NioEndpoint processSocket
> SEVERE: Error allocating socket processor
> java.lang.NullPointerException
> at
> org.apache.tomcat.util.net.NioEndpoint.processSocket(NioEndpoint.java:712)
> at
> org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(NioEndpoint.java:1200)
> at
> org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1136)
> at java.lang.Thread.run(Thread.java:662)
> Nov 15, 2011 8:39:52 AM
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler process
> SEVERE: null
> java.lang.IllegalStateException: Calling [asyncPostProcess()] is not valid
> for a request with Async state [STARTED]
> at
> org.apache.coyote.AsyncStateMachine.asyncPostProcess(AsyncStateMachine.java:202)
> at
> org.apache.coyote.AbstractProcessor.asyncPostProcess(AbstractProcessor.java:104)
> at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:519)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1550)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Exception in declaration()
> 
> I see more of the "Calling [asyncPostProcess()] is not valid for a request
> with Async state [STARTED]" error by itself also.  Here is the connector
> setup:
> 
> connectionTimeout="2"
>redirectPort="8443" />
> 
> Any direction on where to look for the cause?

It could be a bug somewhere in the NIO connector. There has been a lot
of refactoring to reduce duplication between the connectors. In the long
term that should reduce the bugs and makes those that remain easier to
fix. In the short term, there have been a couple of regressions.

What we really need is a reproducible test case. The simpler, the better.

It could also be an application bug. A test case would help identify
that too.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Errors with NIO processor

2011-11-15 Thread Matthew Tyson 
Thanks Bob.  It doesn't seem to be a load problem.  It happens consistently
even for just 1 user.

If I switch the connector back to HTTP/1.1, instead of NIO, the problem
goes away.

Sometimes, there doesn't appear to be an error in catalina.out, but there
is a response with no body, just headers like this:

DateWed, 16 Nov 2011 00:43:58 GMT
ServerApache-Coyote/1.1
Content-Typetext/html;charset=ISO-8859-1
Cache-Controlmax-age=2
ExpiresWed, 16 Nov 2011 00:44:00 GMT
Set-Cookiexgh=gnweb10; path=/; BIGipCookie=00
000
000
VaryUser-Agent,Accept-Encoding
P3P
policyref=http://www.company.net/w3c/p3p.xml<http://www.gaggle.net/w3c/p3p.xml>,
CP=ALL
Content-Encodinggzip
Content-Length20
Connectionclose



On Tue, Nov 15, 2011 at 4:51 PM, Bob Hall  wrote:

> Matt,
>
> Did the testing include load testing?  Have you checked the "open file
> limit" values?
>
> If not, you may be running into an "open file limit" for the OS and/or
> user that is running Tomcat.
>
> - Bob
>
>
> 
> From: Matthew Tyson 
> To: Tomcat Users List 
> Sent: Tuesday, November 15, 2011 4:18 PM
> Subject: Re: Errors with NIO processor
>
> Is there more info I can provide to help diagnose this error?  It is
> killing us.
>
> Thanks,
>
> Matt
>
>
>
> On Tue, Nov 15, 2011 at 9:00 AM, Matthew Tyson
> wrote:
>
> > Hey Guys,
> >
> > We are seeing the following errors (in production of course, testing
> > didn't reveal this) after switching to NIO protocol.
> >
> > This is Tomcat 7.0.22 on CentOS 6.  There is a load balancer sending only
> > comet traffic to port 8080, where the NIO protocol is used.
> >
> > Nov 15, 2011 8:39:29 AM org.apache.tomcat.util.net.NioEndpoint
> > processSocket
> > SEVERE: Error allocating socket processor
> > java.lang.NullPointerException
> > Nov 15, 2011 8:39:51 AM org.apache.tomcat.util.net.NioEndpoint
> > processSocket
> > SEVERE: Error allocating socket processor
> > java.lang.NullPointerException
> > at
> >
> org.apache.tomcat.util.net.NioEndpoint.processSocket(NioEndpoint.java:712)
> > at
> >
> org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(NioEndpoint.java:1200)
> > at
> > org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1136)
> > at java.lang.Thread.run(Thread.java:662)
> > Nov 15, 2011 8:39:52 AM
> > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler process
> > SEVERE: null
> > java.lang.IllegalStateException: Calling [asyncPostProcess()] is not
> valid
> > for a request with Async state [STARTED]
> > at
> >
> org.apache.coyote.AsyncStateMachine.asyncPostProcess(AsyncStateMachine.java:202)
> > at
> >
> org.apache.coyote.AbstractProcessor.asyncPostProcess(AbstractProcessor.java:104)
> > at
> >
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:519)
> > at
> >
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1550)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> > Exception in declaration()
> >
> > I see more of the "Calling [asyncPostProcess()] is not valid for a
> request
> > with Async state [STARTED]" error by itself also.  Here is the connector
> > setup:
> >
> >  > protocol="org.apache.coyote.http11.Http11NioProtocol"
> >connectionTimeout="2"
> >redirectPort="8443" />
> >
> > Any direction on where to look for the cause?
> >
> > Thanks,
> >
> > Matt
> >
>

Re: Errors with NIO processor

2011-11-15 Thread Bob Hall 
Matt,

Did the testing include load testing?  Have you checked the "open file limit" 
values?

If not, you may be running into an "open file limit" for the OS and/or user 
that is running Tomcat.

- Bob



From: Matthew Tyson 
To: Tomcat Users List 
Sent: Tuesday, November 15, 2011 4:18 PM
Subject: Re: Errors with NIO processor

Is there more info I can provide to help diagnose this error?  It is
killing us.

Thanks,

Matt



On Tue, Nov 15, 2011 at 9:00 AM, Matthew Tyson
wrote:

> Hey Guys,
>
> We are seeing the following errors (in production of course, testing
> didn't reveal this) after switching to NIO protocol.
>
> This is Tomcat 7.0.22 on CentOS 6.  There is a load balancer sending only
> comet traffic to port 8080, where the NIO protocol is used.
>
> Nov 15, 2011 8:39:29 AM org.apache.tomcat.util.net.NioEndpoint
> processSocket
> SEVERE: Error allocating socket processor
> java.lang.NullPointerException
> Nov 15, 2011 8:39:51 AM org.apache.tomcat.util.net.NioEndpoint
> processSocket
> SEVERE: Error allocating socket processor
> java.lang.NullPointerException
>         at
> org.apache.tomcat.util.net.NioEndpoint.processSocket(NioEndpoint.java:712)
>         at
> org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(NioEndpoint.java:1200)
>         at
> org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1136)
>         at java.lang.Thread.run(Thread.java:662)
> Nov 15, 2011 8:39:52 AM
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler process
> SEVERE: null
> java.lang.IllegalStateException: Calling [asyncPostProcess()] is not valid
> for a request with Async state [STARTED]
>         at
> org.apache.coyote.AsyncStateMachine.asyncPostProcess(AsyncStateMachine.java:202)
>         at
> org.apache.coyote.AbstractProcessor.asyncPostProcess(AbstractProcessor.java:104)
>         at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:519)
>         at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1550)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> Exception in declaration()
>
> I see more of the "Calling [asyncPostProcess()] is not valid for a request
> with Async state [STARTED]" error by itself also.  Here is the connector
> setup:
>
>  protocol="org.apache.coyote.http11.Http11NioProtocol"
>                connectionTimeout="2"
>                redirectPort="8443" />
>
> Any direction on where to look for the cause?
>
> Thanks,
>
> Matt
>

Re: Errors with NIO processor

2011-11-15 Thread Matthew Tyson 
Is there more info I can provide to help diagnose this error?  It is
killing us.

Thanks,

Matt



On Tue, Nov 15, 2011 at 9:00 AM, Matthew Tyson
wrote:

> Hey Guys,
>
> We are seeing the following errors (in production of course, testing
> didn't reveal this) after switching to NIO protocol.
>
> This is Tomcat 7.0.22 on CentOS 6.  There is a load balancer sending only
> comet traffic to port 8080, where the NIO protocol is used.
>
> Nov 15, 2011 8:39:29 AM org.apache.tomcat.util.net.NioEndpoint
> processSocket
> SEVERE: Error allocating socket processor
> java.lang.NullPointerException
> Nov 15, 2011 8:39:51 AM org.apache.tomcat.util.net.NioEndpoint
> processSocket
> SEVERE: Error allocating socket processor
> java.lang.NullPointerException
> at
> org.apache.tomcat.util.net.NioEndpoint.processSocket(NioEndpoint.java:712)
> at
> org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(NioEndpoint.java:1200)
> at
> org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1136)
> at java.lang.Thread.run(Thread.java:662)
> Nov 15, 2011 8:39:52 AM
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler process
> SEVERE: null
> java.lang.IllegalStateException: Calling [asyncPostProcess()] is not valid
> for a request with Async state [STARTED]
> at
> org.apache.coyote.AsyncStateMachine.asyncPostProcess(AsyncStateMachine.java:202)
> at
> org.apache.coyote.AbstractProcessor.asyncPostProcess(AbstractProcessor.java:104)
> at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:519)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1550)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Exception in declaration()
>
> I see more of the "Calling [asyncPostProcess()] is not valid for a request
> with Async state [STARTED]" error by itself also.  Here is the connector
> setup:
>
>  protocol="org.apache.coyote.http11.Http11NioProtocol"
>connectionTimeout="2"
>redirectPort="8443" />
>
> Any direction on where to look for the cause?
>
> Thanks,
>
> Matt
>

Errors with NIO processor

2011-11-15 Thread Matthew Tyson 
Hey Guys,

We are seeing the following errors (in production of course, testing didn't
reveal this) after switching to NIO protocol.

This is Tomcat 7.0.22 on CentOS 6.  There is a load balancer sending only
comet traffic to port 8080, where the NIO protocol is used.

Nov 15, 2011 8:39:29 AM org.apache.tomcat.util.net.NioEndpoint processSocket
SEVERE: Error allocating socket processor
java.lang.NullPointerException
Nov 15, 2011 8:39:51 AM org.apache.tomcat.util.net.NioEndpoint processSocket
SEVERE: Error allocating socket processor
java.lang.NullPointerException
at
org.apache.tomcat.util.net.NioEndpoint.processSocket(NioEndpoint.java:712)
at
org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(NioEndpoint.java:1200)
at
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1136)
at java.lang.Thread.run(Thread.java:662)
Nov 15, 2011 8:39:52 AM
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler process
SEVERE: null
java.lang.IllegalStateException: Calling [asyncPostProcess()] is not valid
for a request with Async state [STARTED]
at
org.apache.coyote.AsyncStateMachine.asyncPostProcess(AsyncStateMachine.java:202)
at
org.apache.coyote.AbstractProcessor.asyncPostProcess(AbstractProcessor.java:104)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:519)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1550)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Exception in declaration()

I see more of the "Calling [asyncPostProcess()] is not valid for a request
with Async state [STARTED]" error by itself also.  Here is the connector
setup:



Any direction on where to look for the cause?

Thanks,

Matt