executor filter chain behavior
All - We are having some strange behavior with our MINA 1.0.9-based server during load test. I may be misinterpreting the purpose of an executor in the filter chain - please let me know if I have an incorrect understanding of its usage. Our server is a pretty simple asynchronous request/response server where multiple requests may come in all at once and be answered in any order. We have an Executor filter set up at the end of our filter chain to process requests concurrently. However, when under load, our server is processing all requests in order, in the same thread. For example, please see the log below: [SocketAcceptorIoProcessor-0.0] INFO - [/127.0.0.1:2824] RECEIVED: n0002..omitted for readability [SocketAcceptorIoProcessor-0.0] INFO - [/127.0.0.1:2824] RECEIVED: n0003..omitted [SocketAcceptorIoProcessor-0.0] INFO - [/127.0.0.1:2824] RECEIVED: n0004..omitted [SocketAcceptorIoProcessor-0.0] INFO - [/127.0.0.1:2824] RECEIVED: n0005..omitted [pool-4-thread-1] INFO - [/127.0.0.1:2824] WRITE: 0001E24 ...SENT's omitted for readability [pool-4-thread-1] INFO - [/127.0.0.1:2824] WRITE: 000232nA [pool-4-thread-1] INFO - [/127.0.0.1:2824] WRITE: 000332nA [pool-4-thread-1] INFO - [/127.0.0.1:2824] WRITE: 000432nA [pool-4-thread-1] INFO - [/127.0.0.1:2824] WRITE: 000532nA It was my understanding that simultaneous requests like this would spawn five different threads (currently all executed in order, in pool-4-thread-1). Am I wrong? If so, is there a way to have these processed concurrently, as we are facing some performance issues? Our filter chain is set up as follows: filterChainBuilder = acceptorConfig.getFilterChain(); filterChainBuilder.addLast(codec, new ProtocolCodecFilter(new TcCodecFactory())); filterChainBuilder.addLast(logger, new LoggingFilter()); filterChainBuilder.addLast(threadPool, new ExecutorFilter( Executors.newCachedThreadPool())); Thanks in advance! I greatly appreciate everyone's help. -Matt
Re: executor filter chain behavior
I see now that this is the expected behavior in Mina 1.x. We can't use Mina 2.0/UnorderedExecutorFilter as we cannot use Java 1.5 in our environment. Has anyone published a custom version of an unordered ExecutorFilter for Java 1.4? Otherwise I guess we'll just write our own. Thanks, Matt On Tue, Mar 11, 2008 at 2:11 PM, Matt Mehalso [EMAIL PROTECTED] wrote: All - We are having some strange behavior with our MINA 1.0.9-based server during load test. I may be misinterpreting the purpose of an executor in the filter chain - please let me know if I have an incorrect understanding of its usage. Our server is a pretty simple asynchronous request/response server where multiple requests may come in all at once and be answered in any order. We have an Executor filter set up at the end of our filter chain to process requests concurrently. However, when under load, our server is processing all requests in order, in the same thread. For example, please see the log below: [SocketAcceptorIoProcessor-0.0] INFO - [/127.0.0.1:2824] RECEIVED: n0002..omitted for readability [SocketAcceptorIoProcessor-0.0] INFO - [/127.0.0.1:2824] RECEIVED: n0003..omitted [SocketAcceptorIoProcessor-0.0] INFO - [/127.0.0.1:2824] RECEIVED: n0004..omitted [SocketAcceptorIoProcessor-0.0] INFO - [/127.0.0.1:2824] RECEIVED: n0005..omitted [pool-4-thread-1] INFO - [/127.0.0.1:2824] WRITE: 0001E24 ...SENT's omitted for readability [pool-4-thread-1] INFO - [/127.0.0.1:2824] WRITE: 000232nA [pool-4-thread-1] INFO - [/127.0.0.1:2824] WRITE: 000332nA [pool-4-thread-1] INFO - [/127.0.0.1:2824] WRITE: 000432nA [pool-4-thread-1] INFO - [/127.0.0.1:2824] WRITE: 000532nA It was my understanding that simultaneous requests like this would spawn five different threads (currently all executed in order, in pool-4-thread-1). Am I wrong? If so, is there a way to have these processed concurrently, as we are facing some performance issues? Our filter chain is set up as follows: filterChainBuilder = acceptorConfig.getFilterChain(); filterChainBuilder.addLast(codec, new ProtocolCodecFilter(new TcCodecFactory())); filterChainBuilder.addLast(logger, new LoggingFilter()); filterChainBuilder.addLast(threadPool, new ExecutorFilter( Executors.newCachedThreadPool())); Thanks in advance! I greatly appreciate everyone's help. -Matt
Re: Environment switch causing strange SocketExceptions
Thanks, Trustin. I took a look at the thread pool stuff and think i was just a little confused when i wrote up the question. I debugged the server and found my threads were returning to the pool as they should. We are seeing this exception a lot, but will now assume it is an issue w/ the network and/or client. Thanks, Matt On 9/28/07, Trustin Lee [EMAIL PROTECTED] wrote: Hi Matt, You can simply ignore 'Connection reset by peer' error. It's a little bit different from normal close, it anyways means the connection has been closed. And ExecutorFilter doesn't do anything related with choosing a thread. It just calls execute() method on the Executor you specified. If you didn't specify any executor, it uses a new ThreadPoolExecutorFilter instance. Therefore, if you want to control how threads are pulled from a pool, you need to provide an Executor instance that behaves as you expect. HTH, Trustin On 9/26/07, Matt Mehalso [EMAIL PROTECTED] wrote: Also noticed during testing in my own dev environment that if I wait for a few minutes between sending messages to the server, a new thread is always used by the executor filter. For example, if I connect and send three requests one after the other, the executor filter uses thread-1 for each request. If i wait five minutes and send the same request again on the same connection, the log shows thread-2 being used. Looking at my task manager, I can confirm that a new thread is indeed being launched after every increment. Thanks, Matt On 9/26/07, Matt Mehalso [EMAIL PROTECTED] wrote: All - We've just recently deployed a Mina-based server (using mina-core-1.0.3) to production that had been in testing for some time. We are now getting a strange error that we had never seen during testing. The server is a fairly simple request / response multi-threaded server that uses our own custom protocol codec for encoding and decoding messages. The codec uses a line length to separate initialization messages, and then a delimiter for all subsequent messages. We use a edu.emory.mathcs.backport.java.util.concurrent.Executors.newCachedThreadPoolexecutorwith an ExecutorFilter to handle requests in different threads. The max # of threads is set using the # of processors + 1, as in the mina tutorial. We are getting a strange socket connection that is always paired with a strange java.lang.Integer / java.lang.String mismatch? Additionally, we've noticed that a new thread is being spawned for each incoming request and that these threads are no longer returning to the thread pool as they were during the testing phase. The server runs as a windows service using JavaService and we are continuously restarting as the memory usage climbs above 500 MB or so. We are now getting the following exceptions, sometimes after a single request, sometimes after 10-15, as extracted from our log files: 2007-09-25 10:28:07,821 [SocketAcceptorIoProcessor-0.3] INFO package.handler.ReceiptTaskHandler - [/56.116.73.225:4630] EXCEPTION: java.net.SocketException: Connection reset by peer: Read failed at sun.nio.ch.SocketDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:42) at sun.nio.ch.IOUtil.readIntoNativeBuffer (IOUtil.java:265) at sun.nio.ch.IOUtil.read(IOUtil.java:238) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:223) at org.apache.mina.transport.socket.nio.SocketIoProcessor.read( SocketIoProcessor.java :267) at org.apache.mina.transport.socket.nio.SocketIoProcessor.process( SocketIoProcessor.java:241) at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$500( SocketIoProcessor.java:44) at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run ( SocketIoProcessor.java:563) at org.apache.mina.util.NamePreservingRunnable.run( NamePreservingRunnable.java:43) at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java :987) at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:528) at java.lang.Thread.run(Thread.java:832) 2007-09-25 10:28:07,821 [SocketAcceptorIoProcessor-0.3 ] INFO package.handler.ReceiptTaskHandler - [/56.116.73.225:4630] CLOSED 2007-09-25 10:28:07,821 [pool-1-thread-32] ERROR package.handler.ReceiptTaskHandler - [/56.116.73.225:4630] Connection with client closed abnormally. Exception: Connection reset by peer: Read failed 2007-09-25 10:28:07,836 [pool-1-thread-32] ERROR package.handler.ReceiptTaskHandler - [/56.116.73.225:4630] Connection with client closed abnormally. Exception: java/lang/Integer incompatible with java/lang/String As you can see, this is on the 32nd thread spawned and is the 32nd
Environment switch causing strange SocketExceptions
All - We've just recently deployed a Mina-based server (using mina-core-1.0.3) to production that had been in testing for some time. We are now getting a strange error that we had never seen during testing. The server is a fairly simple request / response multi-threaded server that uses our own custom protocol codec for encoding and decoding messages. The codec uses a line length to separate initialization messages, and then a delimiter for all subsequent messages. We use a edu.emory.mathcs.backport.java.util.concurrent.Executors.newCachedThreadPoolexecutor with an ExecutorFilter to handle requests in different threads. The max # of threads is set using the # of processors + 1, as in the mina tutorial. We are getting a strange socket connection that is always paired with a strange java.lang.Integer / java.lang.String mismatch? Additionally, we've noticed that a new thread is being spawned for each incoming request and that these threads are no longer returning to the thread pool as they were during the testing phase. The server runs as a windows service using JavaService and we are continuously restarting as the memory usage climbs above 500 MB or so. We are now getting the following exceptions, sometimes after a single request, sometimes after 10-15, as extracted from our log files: 2007-09-25 10:28:07,821 [SocketAcceptorIoProcessor-0.3] INFO package.handler.ReceiptTaskHandler - [/56.116.73.225:4630] EXCEPTION: java.net.SocketException: Connection reset by peer: Read failed at sun.nio.ch.SocketDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:42) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:265) at sun.nio.ch.IOUtil.read(IOUtil.java:238) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:223) at org.apache.mina.transport.socket.nio.SocketIoProcessor.read( SocketIoProcessor.java:267) at org.apache.mina.transport.socket.nio.SocketIoProcessor.process( SocketIoProcessor.java:241) at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$500( SocketIoProcessor.java:44) at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run( SocketIoProcessor.java:563) at org.apache.mina.util.NamePreservingRunnable.run( NamePreservingRunnable.java:43) at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:987) at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:528) at java.lang.Thread.run(Thread.java:832) 2007-09-25 10:28:07,821 [SocketAcceptorIoProcessor-0.3] INFO package.handler.ReceiptTaskHandler - [/56.116.73.225:4630] CLOSED 2007-09-25 10:28:07,821 [pool-1-thread-32] ERROR package.handler.ReceiptTaskHandler - [/56.116.73.225:4630] Connection with client closed abnormally. Exception: Connection reset by peer: Read failed 2007-09-25 10:28:07,836 [pool-1-thread-32] ERROR package.handler.ReceiptTaskHandler - [/56.116.73.225:4630] Connection with client closed abnormally. Exception: java/lang/Integer incompatible with java/lang/String As you can see, this is on the 32nd thread spawned and is the 32nd message received since the server started. We currently only have a single client connected to the server. Has anyone ever seen anything like this before? Any ideas? Thanks in advance for your help! Best, Matt
Re: Environment switch causing strange SocketExceptions
Also noticed during testing in my own dev environment that if I wait for a few minutes between sending messages to the server, a new thread is always used by the executor filter. For example, if I connect and send three requests one after the other, the executor filter uses thread-1 for each request. If i wait five minutes and send the same request again on the same connection, the log shows thread-2 being used. Looking at my task manager, I can confirm that a new thread is indeed being launched after every increment. Thanks, Matt On 9/26/07, Matt Mehalso [EMAIL PROTECTED] wrote: All - We've just recently deployed a Mina-based server (using mina-core-1.0.3) to production that had been in testing for some time. We are now getting a strange error that we had never seen during testing. The server is a fairly simple request / response multi-threaded server that uses our own custom protocol codec for encoding and decoding messages. The codec uses a line length to separate initialization messages, and then a delimiter for all subsequent messages. We use a edu.emory.mathcs.backport.java.util.concurrent.Executors.newCachedThreadPoolexecutor with an ExecutorFilter to handle requests in different threads. The max # of threads is set using the # of processors + 1, as in the mina tutorial. We are getting a strange socket connection that is always paired with a strange java.lang.Integer / java.lang.String mismatch? Additionally, we've noticed that a new thread is being spawned for each incoming request and that these threads are no longer returning to the thread pool as they were during the testing phase. The server runs as a windows service using JavaService and we are continuously restarting as the memory usage climbs above 500 MB or so. We are now getting the following exceptions, sometimes after a single request, sometimes after 10-15, as extracted from our log files: 2007-09-25 10:28:07,821 [SocketAcceptorIoProcessor-0.3] INFO package.handler.ReceiptTaskHandler - [/56.116.73.225:4630] EXCEPTION: java.net.SocketException: Connection reset by peer: Read failed at sun.nio.ch.SocketDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:42) at sun.nio.ch.IOUtil.readIntoNativeBuffer (IOUtil.java:265) at sun.nio.ch.IOUtil.read(IOUtil.java:238) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:223) at org.apache.mina.transport.socket.nio.SocketIoProcessor.read( SocketIoProcessor.java :267) at org.apache.mina.transport.socket.nio.SocketIoProcessor.process( SocketIoProcessor.java:241) at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$500( SocketIoProcessor.java:44) at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run ( SocketIoProcessor.java:563) at org.apache.mina.util.NamePreservingRunnable.run( NamePreservingRunnable.java:43) at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java :987) at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:528) at java.lang.Thread.run(Thread.java:832) 2007-09-25 10:28:07,821 [SocketAcceptorIoProcessor-0.3 ] INFO package.handler.ReceiptTaskHandler - [/56.116.73.225:4630] CLOSED 2007-09-25 10:28:07,821 [pool-1-thread-32] ERROR package.handler.ReceiptTaskHandler - [/56.116.73.225:4630] Connection with client closed abnormally. Exception: Connection reset by peer: Read failed 2007-09-25 10:28:07,836 [pool-1-thread-32] ERROR package.handler.ReceiptTaskHandler - [/56.116.73.225:4630] Connection with client closed abnormally. Exception: java/lang/Integer incompatible with java/lang/String As you can see, this is on the 32nd thread spawned and is the 32nd message received since the server started. We currently only have a single client connected to the server. Has anyone ever seen anything like this before? Any ideas? Thanks in advance for your help! Best, Matt
MINA Web Containers
Has anyone had any success using Mina within a WebContainer, such as WebSphere? Is there any way to configure Mina to run in a single thread, or to integrate the framework with a WorkManager (container-managed thread pool)? The reason that I ask is that we have a requirement to answer some web requests with information from a remote system. We must conform to existing specifications regarding the connection with this system, and thus are confined to tcp/ip socket request/response messaging. Thanks, Matt
Re: MINA Web Containers
Hi Maarten, Thank you for the information. I thought the servlet spec stated that the ability to spawn threads in a J2EE container could not be relied upon. I'm not sure how WebSphere would behave in this situation. Our communication spec guarantees a synchronous request/response protocol, so we may just go with java sockets on this one to avoid any threading issues. We used Mina for a server implementation of a similar, but asynchronous, protocol, and did this outside of the web container, and it works perfectly. We were hoping to use it again for the client side, though no big deal if not possible. Thanks for your help, Matt On 5/21/07, Maarten Bosteels [EMAIL PROTECTED] wrote: Hi Matt, I am not sure I understand/problem your question eniterly. We are running a MINA server within tomcat. The application is a regular web app (with spring controllers etc for admin functions) that happens to also start a MINA SocketAccepor (on another port obviously) Our application is not single-threaded though, the servlet spec does not forbid you to start new threads. IIRC, entreprise beans (Entity beans and session beans) are not allowed to start new threads. Maarten On 5/21/07, Matt Mehalso [EMAIL PROTECTED] wrote: Has anyone had any success using Mina within a WebContainer, such as WebSphere? Is there any way to configure Mina to run in a single thread, or to integrate the framework with a WorkManager (container-managed thread pool)? The reason that I ask is that we have a requirement to answer some web requests with information from a remote system. We must conform to existing specifications regarding the connection with this system, and thus are confined to tcp/ip socket request/response messaging. Thanks, Matt
Simple Threaded Server - Problem Running on Windows Server 2003
Hi All - We have developed a simple request - response server using Mina 1.0.3. The server uses a logging filter, a custom codec, and an ExecutorFilter to intercept incoming events/messages before hitting our custom HandlerAdapter. The server has been fully tested on our development machines. However, we have ported over to the production box this morning (running Windows Server 2k3), and are running into an odd problem. I think I've pinned it down to the ExecutorFilter. Basically, what happens on the new box is this: Upon a new connection, the ExecutorFilter fires off a new event. However, the thread handling this event never exits (i.e. normally, the ExecutorFilter logs Exiting since queue is empty for /127.0.0.1:). We never get this message when running on the new box. When receiving a message, the protocol codec handles everything correctly (we get .RECEIVED: 'msg'), but the ExecutorFilter never fires off a new thread and therefore, the message is not handled. Basically, all we have now is a log of incoming messages with no handling logic being performed and nothing being sent back. Again, we've tested the server on several of our development machines and everything works fine. Any ideas? Are there some security settings we need to enable on our Win2K3 box? Thanks for your help! Our experience with MINA thus far has been fantastic! -Matt
Re: Simple Threaded Server - Problem Running on Windows Server 2003
Actually, I turned off the ExecutorFilter and I'm still having the same problem, so scratch the last message. It looks like the handler is stalling in the sessionOpened event method when trying to load a new ClassPathXmlApplicationContext. I wonder if this is some sort of security issue with Server 2K3 and accessing the filesystem. On 5/15/07, Matt Mehalso [EMAIL PROTECTED] wrote: Hi All - We have developed a simple request - response server using Mina 1.0.3. The server uses a logging filter, a custom codec, and an ExecutorFilter to intercept incoming events/messages before hitting our custom HandlerAdapter. The server has been fully tested on our development machines. However, we have ported over to the production box this morning (running Windows Server 2k3), and are running into an odd problem. I think I've pinned it down to the ExecutorFilter. Basically, what happens on the new box is this: Upon a new connection, the ExecutorFilter fires off a new event. However, the thread handling this event never exits (i.e. normally, the ExecutorFilter logs Exiting since queue is empty for /127.0.0.1:). We never get this message when running on the new box. When receiving a message, the protocol codec handles everything correctly (we get .RECEIVED: 'msg'), but the ExecutorFilter never fires off a new thread and therefore, the message is not handled. Basically, all we have now is a log of incoming messages with no handling logic being performed and nothing being sent back. Again, we've tested the server on several of our development machines and everything works fine. Any ideas? Are there some security settings we need to enable on our Win2K3 box? Thanks for your help! Our experience with MINA thus far has been fantastic! -Matt
