Re: mKahaDB - no clean-up because of ACKs
Hello Tim. Thanks for the hint about the XA transactions. There were indeed 2 XA transactions hanging in the kahaDB files. After I committed them via JConsole the store was cleaned-up. Joachim -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
mKahaDB - no clean-up because of ACKs
Hello. Recently we had a problem on a ActiveMQ 5.10 version (with manually applied patch for AMQ-5542). The mKahaDB data store increased to ~30 GB and couldn´t clean up the data files anymore. The log showed always something like this: /not removing data file: 317633 as contained ack(s) refer to referenced file: [317632, 317633]/ I´m aware that the data files can´t be cleaned up if there is a not consumed message in a queue. But that´s not the case here. I have started a ActiveMQ with the copied storage on my local machine and checked every queue and topic via JConsole if there is any message in it - but every queue/topic shows a size of 0. So it seems to me that the messages are processed but just the ACK is somewhat stuck in the store. Is there a way to (manually) get rid of the ACKs? Or is there a way to have a deeper analysis of the kahaDB storage files to find the reason for the stucked ACKs? I can provide the whole log with the KahaDB recovering if this is of any help. Thanks. Joachim -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Question about rollbackOnlyOnAsyncException (AMQ-3166)
Yes, we use XA transactions. I don´t think that the NIO mixes this up somehow. We can run our system with a standalone ActiveMQ using tcp or nio but also in an embedded mode inside Wildfly using the VM protocol. I just run our tests again with the enableAckCompaction set to false for the kahaDB - it looked a lot better as we now only have a few message sends taking ~10ms or ~20ms (which is also a long time...) but no send took longer than that. I´ll have further tests in that direction and will keep this post updated. Thanks -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Question about rollbackOnlyOnAsyncException (AMQ-3166)
Hello Gary. Thanks for your reply. I already tried the rollbackOnlyOnAsyncException. But this one only works if the transaction still exists when the error in the async send happens. But if the TX is already finished, than there is no such action because the code which looks up the transaction returns null (naturally) and therfore can´t do a rollback. That´s why I asked if there is some way to add a synchronization on the TX commit on the asynch outcome... Some time ago I also tested out the preallocation strategies but found no real performance differences between those. I think this has to do with the Windows file system which we are using; on Unix based systems this seems to have more of an effect. I haven´t tried the ackCompaction yet. But if I understand it correctly I don´t think this will have much influence either because we aren´t doing much bulk processing. In each transaction there are only a few messages received / send. But I will have a look at this and also to Artemis. -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Question about rollbackOnlyOnAsyncException (AMQ-3166)
Hello Tim. Thank you for your suggestion but I already checked the GC (forgot to mention it). The times of the slow JMS sends doesn´t correlate with the garbage collector events. Btw: we use CMS as collector. Any other ideas? Thanks. Joachim -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Question about rollbackOnlyOnAsyncException (AMQ-3166)
Hello. I have a question about the AMQ-3166 issue and the introduced 'rollbackOnlyOnAsyncException' flag. In the comments of the AMQ-3166 task Gary Tully says async exceptions on transactional ops - message send and message ack will result in the transaction being marked rollback-only. Commit will fail with an exception. But if I send the message in async mode than it´s likely that the client side transaction is already committed before(!) an exception occurs on the AMQ side. In that case the TransportConnection class will no longer find a transaction for the message (as it´s already committed) and of course it can´t be rolled-back anymore. Or do I misunderstand the behaviour? My current problem is that we switched from async to sync sends because we´ve seen some messages getting lost without "seeing" it on the client side. For our use-case it´s not acceptable to have a message loss. So we switched to the sync send. But this comes with a high performance impact. We are seeing a lot of messages to be send in 0-3ms but we see occassional spikes in the send times (in the ActiveMQMessageProducer) of 30-500ms. The 'producerFlowControl' is already disabled in the configuration; the storage is mKahaDB. And we already tested a lot of configuration combinations with no real (positive) effect on the performance. Is there a way to avoid such spikes in the sync send? (Tests has shown that the spikes are most likely caused due to I/O issues. Because when using a RAM disk as location for the kahaDB I don´t see any such spikes.) Or in async mode: is there a way to have a synchronization on transaction commit which verifies that the message is processed? This way the message processing could be triggered in async mode and the client could do further work and only synchronizes on the transaction commit. Thanks. Joachim btw: we are running AMQ 5.14.5 (unfortunatelly on Windows) -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Question about rollbackOnlyOnAsyncException (AMQ-3166)
Hello. I have a question about the AMQ-3166 issue and the introduced 'rollbackOnlyOnAsyncException' flag. In the comments of the AMQ-3166 task Gary Tully says /async exceptions on transactional ops - message send and message ack will result in the transaction being marked rollback-only. Commit will fail with an exception./ But if I send the message in async mode than it´s likely that the client side transaction is already committed before(!) an exception occurs on the AMQ side. In that case the TransportConnection class will no longer find a transaction for the message (as it´s already committed) and of course it can´t be rolled-back anymore. Or do I misunderstand the behaviour? My current problem is that we switched from async to sync sends because we´ve seen some messages getting lost without "seeing" it on the client side. For our use-case it´s not acceptable to have a message loss. So we switched to the sync send. But this comes with a high performance impact. We are seeing a lot of messages to be send in 0-3ms but we see occassional spikes in the send times (in the ActiveMQMessageProducer) of 30-500ms. The 'producerFlowControl' is already disabled in the configuration; the storage is mKahaDB. And we already tested a lot of configuration combinations with no real (positive) effect on the performance. Is there a way to avoid such spikes in the sync send? (Tests has shown that the spikes are most likely caused due to I/O issues. Because when using a RAM disk as location for the kahaDB I don´t see any such spikes.) Or in async mode: is there a way to have a synchronization on transaction commit which verifies that the message is processed? This way the message processing could be triggered in async mode and the client could do further work and only synchronizes on the transaction commit. Thanks. btw: we are running AMQ 5.14.5 (unfortunatelly on Windows) -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: UseDedicatedTaskRunner configuration
Thanks for the answer. I´ve created the JIRA issue https://issues.apache.org/jira/browse/AMQ-6883 -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
UseDedicatedTaskRunner configuration
Hello. The documentation (http://activemq.apache.org/how-do-i-configure-10s-of-1000s-of-queues-in-a-single-broker-.html) states that the DedicatedTaskRunner is disabled by default from 5.6 onwards. This is true if the ActiveMQ is started directly, e.g. with "activemq.bat start". But if the ActiveMQ is started as a windows service from the wrapper.exe the DedicatedTaskRunner is still activated because the wrapper.conf has following entry: wrapper.java.additional.8=-Dorg.apache.activemq.UseDedicatedTaskRunner=true Question: is there a reason why the DedicatedTaskRunner is configured differently for a standard AMQ start vs. service? If so, can someone please explain it to me. If the DedicatedTaskRunner should also be disabled for the service wrapper than the default configuration should be changed accordingly. -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Clarification of temp usage
Ok. Thank you. That explains the behaviour I´v seen... -- View this message in context: http://activemq.2283324.n4.nabble.com/Clarification-of-temp-usage-tp4711305p4711422.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Clarification of temp usage
Hello. I have a question regarding the ActiveMQ temp usage. (The ActiveMQ version is 5.10.0 but also checked with latest 5.13.2; as storage I´ve tested with KahaDb and mKahadb) In our system we have some queues and topics. The topics has persistent messages; the queues has combinations of persistent and non-persistent messages (also mixed in one queue). The producer flow control is turned off due to performance considerations on the producer side. My understanding is, that persistent messages are stored via fileCursor in the files. And now to the observations in a running system: - the topic (ServerPushTopic) gets "a lot" of messages in a short amount of time (approx. 5000/minute at peak times) -- the consumer is relatively slow -- the producers are typically demon jobs which produces the vast amount of messages; but also UI interactions - the other queues are randomely accessed by producers and consumers; both sides fast - the messages are a mix of text and object messages; but typically really small The "enqueued" count of the ServerPushTopic increases fast; the dequeued count only slow because of the slow consumers. After some time (dependend on the system usage configuration) we get following error messages: http://activemq.apache.org/producer-flow-control.html for more info> If I look in the temp files of the KahaDb (./data/tmp_storage) I see mostly topic messages of ServerPushTopic inside. But: why are these messages stored in the temp files and not in the persistent memory of KahaDb? (Remember: messages to ServerPushTopic are persistent.) I´ve already tried a lot of configuration options for the ServerPushTopic, for example: - pendingSubscriberPolicy - pendingDurableSubscriberPolicy - slowConsumerStrategy - and a lot of combinations thereof The system usage part of the broker.xml (values are trimmed down to reproduce the 100% temp store error faster; on production side, these values are of course much higher.) The only way I could get around this problem is to enable the producerFlowControl; but as mentioned earlier I´d really like to have it turned off because of performance reasons on the producer side. Could someone please explain, why the ServerPushTopic messages are stored into the tempUsage and causes the mentioned error? Best regards Joachim -- View this message in context: http://activemq.2283324.n4.nabble.com/Clarification-of-temp-usage-tp4711305.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
