[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
Finally I have been able to tune Jboss AS to work as per the expectation of our company.I have made few configuration changes and I think combination of those helped. Those are underlined below 1>I have disabled XA recovery coniguration outlined in chapter 8 of JBoss messaging installation. 2>Found this https://jira.jboss.org/jira/browse/JBAS-5801. We were getting some transaction exceptions after 2-3 weeks. So added this attribute to jms-ds.xml 3> As Howard suggested we updated to newer version of Jboss remoting which had completely eliminated the issue of messages getting stuck in queues. 4>Removed jmx-console war from deployment which I think was the main reason behind slow jboss shutdown. 5>We had earlier removed 'FILE' logger from jboss log4j configuration knowing that logging does have performance impact. Now we have removed all jboss specific log items and allowed only application logs. Hope this is going to be helpful for all developers working with Jboss messaging. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4242460#4242460 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4242460 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
Sure will try that in our development environment. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4234766#4234766 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4234766 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
Howard, Nice to see your reply again. As you suggested earlier we are using the jboss-remoting version 2.2.2.SP11 Thanks, Pratim View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4234547#4234547 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4234547 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
Tim, The number of client session created by remote clients are constant at 9 as they use a connection pool. However we have some local components on the jboss server which opens and closes connections while they put messages to the queues.We are using the java:/JmsXA connection factory. Should we be using the connection pooling mecahnism for local components as well? View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4234546#4234546 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4234546 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
Our JBoss server in production became very slow so we had to bounce it. But we were able to get some important information. We found this kind of messages in our log file May 29, 01:25:41 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id ad7d147:95eb:4a1f69f5:7f4 invoked while multiple threads active within it. May 29, 01:25:41 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action ad7d147:95eb:4a1f69f5:7f4 aborting with 1 threads active! Also we found out that "JMS SessionPool " thread went very high along with the Worker Thread. Please help. Thanks, Pratim View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4234042#4234042 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4234042 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
Clebert, We did the stack trace and found out that the number of WorkerThread for the clients beyond a firewall are steadily increasing by 1 or 2 everyday.It's shwoing up on jconsole as WorkerThread#17[10.**.2*.1**:34128] or similar.When it's reaching about 30-35 it's bringing down the server on it's knees.I was trying to find out if we close the connections properly after each 30 mins refresh period that I mentioned in the first post. We found that the code is correct. Just for curiosity I tried to find out what happens with standalone clients. So we made a simple standalone client(non-firewall) and connected to jboss and did a stack trace through jconsole while doing load testing. We found out that the worker threads remain even after the client disconnects. Is it normal ? View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4233966#4233966 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4233966 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
Thanks for replying. Let us analyze more on the stack trace and the histogram and will update you. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4232358#4232358 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4232358 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
Clebert, I tried with the jmap -histo option which gives me the below results while the production server is running on a normal day. It doesn't look like anything consuming a huge memory compared to the allocated 2GB memory. But our server has become slow already. It takes almost 10 minutes to shut down,however starting the server again is normal. We have found out that after few days if we bring down one of the jboss in cluster and try to start again it cannot join the cluster. or the running one doesn't recognize the one being started. SizeCount Class description --- 16301352159950 char[] 1423349615189 int[] 1167578492177 java.lang.Object[] 1155016018668 byte[] 1135240890303 * ConstMethodKlass 6506832 90303 * MethodKlass 5362616 128417 * SymbolKlass 4816200 8454* ConstantPoolKlass 4815816 200659 java.lang.String 3883456 121358 java.util.TreeMap$Entry 3808368 49192 short[] 3568064 8454* InstanceKlassKlass 2897120 7281* ConstantPoolCacheKlass 1658216 17595 java.util.HashMap$Entry[] 1488640 18608 java.lang.reflect.Method 1215312 50638 java.util.HashMap$Entry 1094040 27351 java.util.TreeMap View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4232321#4232321 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4232321 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
thanks for the suggestion,let me try that. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4232076#4232076 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4232076 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
forgot to mention that we use two Jboss servers in cluster and JBoss messaging database in Oracle 10g. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4232071#4232071 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4232071 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - JBoss slowing down
We are using JBM 1.4.2 GA SP1 with JBoss 4.2 GA. We are hosting our jboss servers on Solaris 10 (4 processors /8 GB memory). We do not persist messages more than 30-40 at a time. We also use JMS transactions for message rollback within session beans. Some of the message consumers are local MDBs and some of them are remote custom JMS clients. The remote clients are beyond a firewall and we have to refresh the connections between the client and JBM every 30 minutes. We are noticing that the JBoss server is gradually slowing down as more messages are processed and after about 3-4 weeks it tries to consume more than 100% of the cpu resources and crashes. We have checked with JProbe that there are no memory leak and also tuned the JVM with these parameters -Xms2048m -Xmx2048m -server -Dsun.rmi.dgc.client.gcInterval=180 -Dsun.rmi.dgc.server.gcInterval=180 -XX:MaxPermSize=768m -XX:+UseParallelGC -X X:ParallelGCThreads=4 -XX:NewRatio=1 -XX:SurvivorRatio=12 -XX:+UseLargePages -verb ose:gc -XX:+PrintGCDetails" We also see this kind of messages before crash javax.jms.JMSException: Could not create a session: org.jboss.resource.JBossResourceExcept ion: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (java.lang.IllegalStateException: Transaction TransactionImple < ac, BasicAction: ad7d148:b3f9:49ff9638:c7b9 status: ActionStatus.ABORTED > is not active STATUS_ROLLEDBACK) 2009-05-13 05:08:51,229 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] Periodic recovery - second pass 2009-05-13 05:08:51,367 ERROR [org.jboss.ejb.plugins.LogInterceptor] TransactionRolledbackLocalException in method: public abstract java.lang.String ebond.co mmon.XMLCommunicationHandlerLocal.putToWaitingQueue(java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String), causedBy: java.lang.IllegalStateException: [com.arjuna.ats.internal.jta.transaction.arjunacore.inactive] [com.arjuna.ats.internal.jta.transaction.arjunacore.inactive] The transaction is not active! Any help would be appreciated. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4232069#4232069 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4232069 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: Memory leak in JBoss Messaging 1.4.0SP3-CP02
We are running jboss-messaging-1.4.2.GA-SP1 on Jboss AS 4.2.2GA. We also have a similar situation where a transacted stateless session EJB is having a non transacted JMS connection . Our application is quite big for testing with Jboss profiler. Any idea whether it can cause a memory leak ? Thanks in advance. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4228465#4228465 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4228465 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [EJB/JBoss] - Re: MDB calling a session bean, session bean not getting reu
I am coming across this problem in EJB 2. Any idea how to fix this issue with EJB 2 on Jboss 4.2.2 GA? View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4228458#4228458 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4228458 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss messaging client timeout
my bad, JBM 1.4.2 SP1 works fine. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4213961#4213961 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4213961 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss messaging client timeout
Messaging 4.2.2 SP1 also works fine. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4213960#4213960 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4213960 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss messaging client timeout
Thanks a lot Howard. I tried with 1.4.0 SP03-CP07 and it works. No timeout no messages stuck in the queue!! However during building the messaging part I found etc/ folder missing under src/ so I copied the whole etc folder from output and it did the trick. Another team is trying with 1.4.2 GA SP1,I'll update you on that also. Thanks once again. Pratim View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4213814#4213814 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4213814 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss messaging client timeout
Howard, I am little bit curious if JBM 1.4.2-GA-SP1 fixes the issue as well.We are using 1.4.2 GA,so would like to upgrade to the next higher version. Once again thanks for your help. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4213229#4213229 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4213229 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss messaging client timeout
Howard, while we are testing failover I am seeing this in the logs WARN org.jboss.remoting.Client - unable to remove remote callback handler: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://***:4457/?JBM_clientMaxPoolSize=500&clientLeasePeriod=5&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&connectionWait=6&dataType=jms&marshaller=org.jboss.jms.wireformat.JMSWireFormat&numberOfCallRetries=1&numberOfRetries=30&pingFrequency=214748364&pingWindowFactor=10&socket.check_connection=false&stopLeaseOnFailure=true&timeout=0&unmarshaller=org.jboss.jms.wireformat.JMSWireFormat&validatorPingPeriod=1&validatorPingTimeout=5] As you suggested I had increased clientLeasePeriod to 5 and validatorPingTimeout to 5. Those seems to be passed to the client now which we see in the logs. But still we are seeing this socket connection timeout for a few messages after a long idle time. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4213211#4213211 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4213211 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss messaging client timeout
I downloaded jboss-remoting SP11 from http://repository.jboss.com/jboss/remoting/2.2.2.SP11-brew/lib/ and made changes suggested by Howard. We had given an idle time of 14 hours for our listener. Now some of the messages sent to the JBM queue are getting stuck as the listener is not able to pick up the message.I am seeing the following exception in the JBoss server log for those failed message deliveries. We are using JMS based listener,please help. We never had this problem when we used JBM without clustering. ERROR [ServerInvokerCallbackHandler] Error handling callback org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://10.***.205.***:634256266/callback?callbackServerHost=10.***.205.***&callbackServerPort=634256266&callbackServerProtocol=bisocket&clientMaxPoolSize=1&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&datatype=jms&guid=a5e5p4y-ksiwrb-frkxz15x-1-frlpylal-2lx&isCallbackServer=true&onewayThreadPool=org.jboss.jms.server.remoting.DirectThreadPool&serverSocketClass=org.jboss.jms.server.remoting.ServerSocketWrapper] at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:579) at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:422) at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:133) at org.jboss.remoting.Client.invoke(Client.java:1640) at org.jboss.remoting.Client.invoke(Client.java:554) at org.jboss.remoting.Client.invokeOneway(Client.java:604) at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:826) at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:697) at org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1452) at org.jboss.jms.server.endpoint.ServerSessionEndpoint.replicateDeliveryResponseReceived(ServerSessionEndpoint.java:1066) at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice$3.run(MessagingPostOffice.java:1346) at org.jboss.messaging.util.OrderedExecutorFactory$ChildExecutor.run(OrderedExecutorFactory.java:120) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) at java.lang.Thread.run(Thread.java:595) Caused by: java.io.IOException: Timed out trying to create socket at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.createSocket(BisocketClientInvoker.java:508) at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:827) at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:569) ... 14 more Thanks, Pratim View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4213103#4213103 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4213103 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss messaging client timeout
I should add one more thing here. Earlier we migrated from JBoss MQ to JBoss Messaging. There was no cluster for JBoss Messaging and the client side failover error handling was through the JMS ExceptionListener. We refreshed the connection every half an hour to get over the firewall freeze problem and it worked perfectly without any problem. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4212771#4212771 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4212771 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss messaging client timeout
Thanks Howard for the suggestion. We downloaded SP11 and increased validatorPingTimeout and clientLeasePeriod to 5 in remoting bisocket file. We need to keep our dev server running overnight to see if the error goes away. Meanwhile on the client side as I mentioned before that we have a idle client timeout of 1 hour in the firewall setup. We refresh the client listner connections every half an hour for that. Now on the client side log on every hour(but not on th 1/2th hour) we are seeing this message for every client connection refreshed [Finalizer] WARN org.jboss.remoting.LeasePinger - LeasePinger[SocketClientInvoker[c6bb6c, bisocket://cawcremdd6:4457](a5e5p4y-ksiwrb-frkxz15x-1-frkyc7jf-7t)] tried to remove lease for client with session ID a5e5p4y-ksiwrb-frkxz15x-1-frl29p2s-kq, but no such lease was found But messages are picked up by the listener after that for the time being.We do not see anything on the server logs.Do you think there could be any other problem for the messages getting stuck in JBM queues? View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4212749#4212749 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4212749 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - JBoss messaging client timeout
Hi, I am using JBoss 4.2.2GA and Messaging 1.4.2 with remoting version 2.2.2.SP10. We have setup JBM cluster with oracle as database. The jboss server is accessed by a remote client located beyond a firewall with a one hour timeout for any idle client. We have a JMS listener which listnes for messages on JBM through the firewall with remoting bisocket configured for 4458 secondary port.To get over the idle timeout issue we are refreshing the message listener connection every half an hour.But even after doing that, after 12 hours of idle time we are seeing that the messages coming on JBM queues are getting stuck for a few messages. Do I need to change any paramater in remoting bisocket to get over this issue ? Any help would be appreciated. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4212435#4212435 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4212435 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
