[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
Finally I have been able to tune Jboss AS to work as per the expectation of our company.I have made few configuration changes and I think combination of those helped. Those are underlined below 1>I have disabled XA recovery coniguration outlined in chapter 8 of JBoss messaging installation. 2>Found this https://jira.jboss.org/jira/browse/JBAS-5801. We were getting some transaction exceptions after 2-3 weeks. So added this attribute to jms-ds.xml 3> As Howard suggested we updated to newer version of Jboss remoting which had completely eliminated the issue of messages getting stuck in queues. 4>Removed jmx-console war from deployment which I think was the main reason behind slow jboss shutdown. 5>We had earlier removed 'FILE' logger from jboss log4j configuration knowing that logging does have performance impact. Now we have removed all jboss specific log items and allowed only application logs. Hope this is going to be helpful for all developers working with Jboss messaging. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4242460#4242460 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4242460 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
Sure will try that in our development environment. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4234766#4234766 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4234766 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
Hi Pratim, jboss remoting has fixed some ServerThread issue in 2.2.3. Can you try it to see if the fix is good for you too? Howard View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4234597#4234597 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4234597 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
Howard, Nice to see your reply again. As you suggested earlier we are using the jboss-remoting version 2.2.2.SP11 Thanks, Pratim View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4234547#4234547 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4234547 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
Tim, The number of client session created by remote clients are constant at 9 as they use a connection pool. However we have some local components on the jboss server which opens and closes connections while they put messages to the queues.We are using the java:/JmsXA connection factory. Should we be using the connection pooling mecahnism for local components as well? View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4234546#4234546 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4234546 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
Can I ask what Rmoting version are you using? I ask this because I saw you mentioned the ServerThreads were leaking, which might be a remoting version issue. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4234301#4234301 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4234301 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
Perhaps you are creating a huge number of sessions in your code? View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4234068#4234068 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4234068 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
Our JBoss server in production became very slow so we had to bounce it. But we were able to get some important information. We found this kind of messages in our log file May 29, 01:25:41 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id ad7d147:95eb:4a1f69f5:7f4 invoked while multiple threads active within it. May 29, 01:25:41 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action ad7d147:95eb:4a1f69f5:7f4 aborting with 1 threads active! Also we found out that "JMS SessionPool " thread went very high along with the Worker Thread. Please help. Thanks, Pratim View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4234042#4234042 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4234042 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
Clebert, We did the stack trace and found out that the number of WorkerThread for the clients beyond a firewall are steadily increasing by 1 or 2 everyday.It's shwoing up on jconsole as WorkerThread#17[10.**.2*.1**:34128] or similar.When it's reaching about 30-35 it's bringing down the server on it's knees.I was trying to find out if we close the connections properly after each 30 mins refresh period that I mentioned in the first post. We found that the code is correct. Just for curiosity I tried to find out what happens with standalone clients. So we made a simple standalone client(non-firewall) and connected to jboss and did a stack trace through jconsole while doing load testing. We found out that the worker threads remain even after the client disconnects. Is it normal ? View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4233966#4233966 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4233966 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
Thanks for replying. Let us analyze more on the stack trace and the histogram and will update you. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4232358#4232358 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4232358 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
Anything.. I mean... *anything*. you can't even say it is a JBM until you fully analyze the histogram. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4232337#4232337 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4232337 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
At this point It's impossible to say what it is. You need to keep looking down the list.. for JBoss/JGroups related objects. It could be anything.. from JGroups retransmissions because of the network to real leakages. Also.. look at the stack traces on the dump itself. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4232336#4232336 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4232336 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
Clebert, I tried with the jmap -histo option which gives me the below results while the production server is running on a normal day. It doesn't look like anything consuming a huge memory compared to the allocated 2GB memory. But our server has become slow already. It takes almost 10 minutes to shut down,however starting the server again is normal. We have found out that after few days if we bring down one of the jboss in cluster and try to start again it cannot join the cluster. or the running one doesn't recognize the one being started. SizeCount Class description --- 16301352159950 char[] 1423349615189 int[] 1167578492177 java.lang.Object[] 1155016018668 byte[] 1135240890303 * ConstMethodKlass 6506832 90303 * MethodKlass 5362616 128417 * SymbolKlass 4816200 8454* ConstantPoolKlass 4815816 200659 java.lang.String 3883456 121358 java.util.TreeMap$Entry 3808368 49192 short[] 3568064 8454* InstanceKlassKlass 2897120 7281* ConstantPoolCacheKlass 1658216 17595 java.util.HashMap$Entry[] 1488640 18608 java.lang.reflect.Method 1215312 50638 java.util.HashMap$Entry 1094040 27351 java.util.TreeMap View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4232321#4232321 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4232321 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
thanks for the suggestion,let me try that. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4232076#4232076 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4232076 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
"pratimkm" wrote : forgot to mention that we use two Jboss servers in cluster and JBoss messaging database in Oracle 10g. Right... The ClassHistogram would be able to tell you where your memory is going to. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4232073#4232073 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4232073 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
You should produce a Stack trace of your server when things are slowing down. Add -XX:+PrintClassHistogram to your VM Options. and call kill -3 when the process is getting slow (before it crashed). -XX:+PrintClassHistogram or kill -3 wouldn't hurt your production system. But if you can replicate this on a test environment it's better. This link will provide you some extra information: http://www.jboss.org/community/wiki/StackTrace You are probably running out of memory. I used to work with Solaris years ago, and running out of memory would certainly crash the VM on Solaris. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4232072#4232072 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4232072 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBoss slowing down
forgot to mention that we use two Jboss servers in cluster and JBoss messaging database in Oracle 10g. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4232071#4232071 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4232071 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
