[jboss-user] [JBoss Messaging] - connection failure message should report client ip
In JBM 2.0.0.BETA3, if a client shuts down without closing its connections (e.g. it crashes) you see the following in the server log file: anonymous wrote : 09:37:31,347 WARNING [org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has been detected Did not receive ping from client. It is likely a client has exited or crashed without closing its connection, or the network between the server and client has failed. The connection will now be closed.:3 | 09:37:31,348 WARNING [org.jboss.messaging.core.server.impl.ServerSessionImpl] Client connection failed, clearing up resources for session 0687b9f5-7d74-11de-9648-0019b9ebba6b | 09:37:31,348 WARNING [org.jboss.messaging.core.server.impl.ServerSessionImpl] Cleared up resources for session 0687b9f5-7d74-11de-9648-0019b9ebba6b If possible it would be great if this message could output the IP address of the client that caused this. This would help to determine the location of a faulty client. View the original post : http://www.jboss.org/index.html?module=bbop=viewtopicp=4247200#4247200 Reply to the post : http://www.jboss.org/index.html?module=bbop=postingmode=replyp=4247200 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBM 2.0.0b3 CPU and memory usage compared to ActiveMQ
As promised here is the CPU usage for the first week: http://massdosage.co.za/temp/jbm-perf1/jbm-cpu-first_week.png and here is the memory usage for the first week: http://massdosage.co.za/temp/jbm-perf1/jbm-memory-first_week.png After the spikes at the start and the subsequent restart on the Wednesday, both CPU and memory usage appear to have stabilised and are even at slightly lower levels to ActiveMQ when I look back at the graphs for the past month. Good news! View the original post : http://www.jboss.org/index.html?module=bbop=viewtopicp=4246357#4246357 Reply to the post : http://www.jboss.org/index.html?module=bbop=postingmode=replyp=4246357 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: JBM 2.0.0b3 CPU and memory usage compared to ActiveMQ
Quick update: I've been watching this over the past few days and the CPU and memory usage seems to have returned to normal. So I'm guessing those spikes are related to the ping issue we saw at around the same time which brought the server down. I will post updated graphs once the server has been running for a week. View the original post : http://www.jboss.org/index.html?module=bbop=viewtopicp=4245767#4245767 Reply to the post : http://www.jboss.org/index.html?module=bbop=postingmode=replyp=4245767 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - JBM 2.0.0b3 CPU and memory usage compared to ActiveMQ
On Monday morning I started migrating a set of applications over to using a JBM topic for sending and receiving messages, this was previously done by ActiveMQ. I thought I would share the results of monitoring the JBM server over this time period. I realise you guys haven't done much work on the memory and cpu usage aspects of JBM yet, but this at least proves that there is definitely a case for some work to be put into this area. We have a graph of memory usage on the server for the last week here: http://massdosage.co.za/temp/jbm-perf1/jbm-memory-first_2days.png and CPU on the same server for the same time period: http://massdosage.co.za/temp/jbm-perf1/jbm-cpu-first_2days.png On Monday we switched about 6 clients over to use JBM instead of AMQ. You notice a slight increase in memory and CPU compared with ActiveMQ up until that point. On Tuesday we then switched another 10 clients over, increasing the load from tens of messages a minute to hundreds (which is still not extreme). You can then see the memory go through the roof and the swap even gets called in (the purple, something we dread in terms of performance). The CPU also goes through the roof at around the same time. The sharp drop-offs on the right of the graphs are where I restarted the server today. The CPU shoots back up very quickly, the memory seems to be taking a bit longer. I will keep an eye on this, just thought I'd report it here so you can keep this in mind for the future. If there is any more information I can provide you with I'm more than happy to. View the original post : http://www.jboss.org/index.html?module=bbop=viewtopicp=4245335#4245335 Reply to the post : http://www.jboss.org/index.html?module=bbop=postingmode=replyp=4245335 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - JBM server failure on abrupt client shutdown
Hey guys, We discussed this issue on irc this morning but as you suggested here is a forum post with all the details I could gather. Hopefully this is all due to the ping issue that has now been fixed in trunk, but this might be useful to review and double check. In a nutshell - we have 16 client applications which are publishing messages to and receiving messages from a single topic (over JMS). The jbm-jms.xml config file is here: http://massdosage.co.za/temp/jbm-fail1/jbm-jms.xml (these forums really could do with an attach file feature). The other configuration files aren't really interesting - no persistence, no security, some JMX and standard netty transport. We are running JBM 2.0.0.beta3. At some point early this morning one of the client machines crashed, we can't tell how but the machine required a restart so I think we can assume the JVM would have shutdown abruptly. I don't have logs for the exact time at which this happened as they rolled over but from the earliest I can find the jbm log file was flooded with these log messages (10 a second or more): 09:07:56,796 WARNING [org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has been detected Did not receive ping from client. It is likely a client has exited or crashed without closing its connection, or the network between the server and client has failed. The connection will now be closed.:3 This went on for hours, and these messages were interspersed with two exception stacks, this one occurred the most: 09:07:57,110 SEVERE [org.jboss.messaging.core.server.impl.ServerSessionPacketHandler] Caught unexpected exception | java.lang.NullPointerException | at org.jboss.messaging.core.server.impl.ServerSessionImpl.handleCloseConsumer(ServerSessionImpl.java:921) | and there were a few of these: 09:07:59,789 SEVERE [org.jboss.messaging.core.server.impl.MessagingServerPacketHandler] Failed to reattach session | java.lang.IllegalStateException: 1571589352 Can't find packet to clear: last received command id 321535 first stored command id 320494 | at org.jboss.messaging.core.remoting.impl.ChannelImpl.clearUpTo(ChannelImpl.java:741) | at org.jboss.messaging.core.remoting.impl.ChannelImpl.replayCommands(ChannelImpl.java:503) | I have attached some fragments of the log file with the full stack traces here: http://massdosage.co.za/temp/jbm-fail1/messaging.log.3.crash-exceptions.txt I restarted the JBM server and then expected the clients onException() handler to be triggered so they would reconnect (this is usually what happens if I restart the JBM server and after around 30s all the clients are reconnected and sending/receiving messages as normal). However the clients did not seem to pick up the fact that the server was back. I left one of the clients for around 5 hours and it still hadn't reconnected. I then tried to shutdown the clients and they refused to exit gracefully. I ran jstack on the clients and this reported a deadlock which I think was preventing the shutdown. I have put the jstack output from two of our clients which show the deadlock occuring in the JBM client code here: http://massdosage.co.za/temp/jbm-fail1/jstack.crash.t2.txt http://massdosage.co.za/temp/jbm-fail1/jstack-s1.crash.txt The deadlocks seem similar but different. Maybe this part of our experience indicates a new bug? Hope this is useful. If you need any more information just let me know. View the original post : http://www.jboss.org/index.html?module=bbop=viewtopicp=4245325#4245325 Reply to the post : http://www.jboss.org/index.html?module=bbop=postingmode=replyp=4245325 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - session.createTextMessage() returns null?
I've noticed some strange behaviour in JBM 2.0.0.BETA3. We have a client application which queues up messages and sends them periodically to a topic. Sometimes the client gets into a strange state where: message = session.createTextMessage(); returns null. I've never seen this happen before. The strange thing is that it doesn't happen on every call to createTextMessage. It will work for a few calls, then return null, then work again for a few more etc. It also doesn't happen all the time, our other client application running the same code at the same time does not exhibit this behaviour. In what cases would this method return null? The spec says nothing about this other than that a JMSException should be throw if an error occurs creating the message. View the original post : http://www.jboss.org/index.html?module=bbop=viewtopicp=4243790#4243790 Reply to the post : http://www.jboss.org/index.html?module=bbop=postingmode=replyp=4243790 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: session.createTextMessage() returns null?
Yeah, you are right, that code can't return null. I will have another look at our code, my guess is that due to the number of deploys we have done recently with different versions there is a mismatch in line numbers which is confusing the issue here. Sorry about that. View the original post : http://www.jboss.org/index.html?module=bbop=viewtopicp=4243809#4243809 Reply to the post : http://www.jboss.org/index.html?module=bbop=postingmode=replyp=4243809 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: using log4j in JBoss Messaging client
Thanks guys, that worked perfectly! View the original post : http://www.jboss.org/index.html?module=bbop=viewtopicp=4242365#4242365 Reply to the post : http://www.jboss.org/index.html?module=bbop=postingmode=replyp=4242365 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: deleting messages rather than blocking if server down fo
The threads are coming from my client application - there is a thread per http request and at the end of the request it calls send(). Hundreds of these threads are created every minute so if the call to send() blocks then it doesn't take long for the blocking to overwhelm the VM. I guess I could re-architect my application to put the messages in a queue and have one thread pull the messages off and send them, this just feels a bit like I'm implementing something which could (should?) be in the messaging client. As for the ExceptionListener solution, I thought about this too. I have JBM setup to auto-reconnect so I don't actually get an exception when the server goes down (I just see some warning messages in the logs). From what I can tell the sender queues up the messages in the background. When the server comes back, only then does my ExceptionListener callback get called, at which point I redo the JNDI lookup and recreate the connections and sessions etc. This works well. I think if I set JBM not to auto-connect then the ExceptionListener callback is called right away when the server goes down, but then I have to do my own reconnect logic. I'm happy with this ExceptionListener logic, it would just be nice to be able to throw messages away instead of blocking, but maybe I should code this up myself. View the original post : http://www.jboss.org/index.html?module=bbop=viewtopicp=4242366#4242366 Reply to the post : http://www.jboss.org/index.html?module=bbop=postingmode=replyp=4242366 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: deleting messages rather than blocking if server down fo
The clients are calling send() on the same MessageProducer instance, and this isn't synchronized. This hasn't caused any noticeable problems in the past (when the MessageProducer was using ActiveMQ) or since we started evaluating JBM (although the loads on those 2 servers are still very low). This makes me think I really should queue up the messages to be sent myself and have one thread call send(), which will also get rid of the blocking problem. As for the exception listener stuff - yes, I understand how that works with the different re-connect values and it makes sense. View the original post : http://www.jboss.org/index.html?module=bbop=viewtopicp=4242374#4242374 Reply to the post : http://www.jboss.org/index.html?module=bbop=postingmode=replyp=4242374 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: deleting messages rather than blocking if server down fo
We have a thread pool - the problem is, we really *do* need to be able to handle thousands of simultaneous request threads. The application in this case is an audio streaming server which ideally should only be limited by IO reading files from disk. Currently we have this scaling up to a few thousand simultaneous users but the idea is for this to go up into the tens of thousands. I will code up a message queue so message sending happens in one thread. Thanks for the input. View the original post : http://www.jboss.org/index.html?module=bbop=viewtopicp=4242380#4242380 Reply to the post : http://www.jboss.org/index.html?module=bbop=postingmode=replyp=4242380 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: Could not find reference with id (JBM 2.0.0.BETA3)
I can confirm that the stack trace is related to the pre-ack mode. I just changed this to DUPS_OK in the client and set pre-ack to false in the server and restarted the apps and server and instead of the stack trace I now see messages like this in the log: Message has expired. No expiry queue configured for queue 1433214e-35df-4691-b4e9-0e165a81a364 so dropping it View the original post : http://www.jboss.org/index.html?module=bbop=viewtopicp=4242391#4242391 Reply to the post : http://www.jboss.org/index.html?module=bbop=postingmode=replyp=4242391 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Could not find reference with id (JBM 2.0.0.BETA3)
I am evaluating JBoss Messaging as a replacement for ActiveMQ at our company. I have installed 2.0.0.BETA3 and am very impressed with the documentation and examples - top stuff. I have the server running on one machine with a jms setup and pre-acknowledgetrue/pre-acknowledge in the config. I have two client machines which send messages to a topic and also receive messages from the same topic. They are configured with the same pre-acknowledge mode (I don't really care if messages go missing, I'm more interested in overall throughput and stability). When run, one of the clients successfully sends messages to the topic and receives messages back (including its own messages). However, the other client (the exact same code) sends messages fine but only receives its own messages back (*not* the messages from the other client). There are no error messages in the client log files. However on the server I see this error over and over again: Failed to acknowledge java.lang.IllegalStateException: Could not find reference with id 878 backup false closed false Does anybody have any idea what's going wrong here? Here is a full stack trace for reference: 15:42:04,894 SEVERE [org.jboss.messaging.core.server.impl.ServerSessionImpl] Failed to acknowledge java.lang.IllegalStateException: Could not find reference with id 893 backup false closed false at org.jboss.messaging.core.server.impl.ServerConsumerImpl.getExpired(ServerConsumerImpl.java:421) at org.jboss.messaging.core.server.impl.ServerSessionImpl.doHandleExpired(ServerSessionImpl.java:1623) at org.jboss.messaging.core.server.impl.ServerSessionImpl.handleExpired(ServerSessionImpl.java:478) at org.jboss.messaging.core.server.impl.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:152) at org.jboss.messaging.core.remoting.impl.ChannelImpl.handlePacket(ChannelImpl.java:621) at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:399) at org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:483) at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.messageReceived(MessagingChannelHandler.java:75) at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:567) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:803) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:387) at org.jboss.netty.handler.codec.frame.FrameDecoder.fireMessageReceived(FrameDecoder.java:284) at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:261) at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:184) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:87) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:567) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:344) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:331) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:303) at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:176) at org.jboss.netty.util.internal.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72) at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:49) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:619) View the original post : http://www.jboss.org/index.html?module=bbop=viewtopicp=4242241#4242241 Reply to the post : http://www.jboss.org/index.html?module=bbop=postingmode=replyp=4242241 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - Re: Could not find reference with id (JBM 2.0.0.BETA3)
Thanks for the quick reply. I created a Jira issue here: https://jira.jboss.org/jira/browse/JBMESSAGING-1679 Let me know if you need any more info or need me to test a fix. I noticed that in the create issue screen there is a link READ THIS BEFORE ENTERING NEW REPORTS. which links to a 404 (http://wiki.jboss.org/wiki/Wiki.jsp?page=HelpBugReport). I have some other feedback regarding JBM, but I'll create separate forum threads for that... View the original post : http://www.jboss.org/index.html?module=bbop=viewtopicp=4242252#4242252 Reply to the post : http://www.jboss.org/index.html?module=bbop=postingmode=replyp=4242252 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - using log4j in JBoss Messaging client
I have a JBM client application (2.0.0.BETA3) and when I restart the JBM server a number of error messages appear on sys err in the client. I would like to direct these using log4j. In the documentation I read that JBM uses Java logging and that adding this line handlers=org.jboss.messaging.integration.logging.Log4jLoggerHandler to a logging.properties file on the classpath would cause JBM to use log4j. However I have the feeling this may only apply to the server as I tried this on the client and the messages still go to sys err. Is there a way to use log4j on the client? View the original post : http://www.jboss.org/index.html?module=bbop=viewtopicp=4242253#4242253 Reply to the post : http://www.jboss.org/index.html?module=bbop=postingmode=replyp=4242253 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBoss Messaging] - deleting messages rather than blocking if server down for lo
I am in the process of evaluating replacing ActiveMQ with JBoss Messaging. Our production systems generate loads of up to around 1 thousand messages a second at peack times. According to the JBM documentation if the server goes down then the client producer caches outgoing messages until the cache size is reached, after this the calls to send will block. ActiveMQ does something similar and this has led to problems in the past where the server has gone down for an extended period of time and the clients end up with so many blocked threads that they take down the client VM. In this situation we would much rather lose messages and have the clients continue running. Can JBM be configured in such a way that if the server is down that messages are dropped by the client instead of the client threads blocking? (they could be dropped FIFO via some sort of Bounded Queue or based on TTL expiry). View the original post : http://www.jboss.org/index.html?module=bbop=viewtopicp=4242255#4242255 Reply to the post : http://www.jboss.org/index.html?module=bbop=postingmode=replyp=4242255 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
