Hi Quinn I just tested the POM changes you posted and the second run failed (without failover-URL). I then tested with the failover-URL and the third attempt failed.
The latter is no big surprise since I discovered the problem during failover tests in a master-slave-config. I then reduced the setup to a single broker environment and it was still there. My test broker is apache-activemq-5.11.0.redhat-620133, a patched Redhat version of AMQ 5.11. As you, I also don't change the AMQ version number in the POM, I just use a newer broker than the library version. My broker runs on another machine than the test. Perhaps this increases the probability of losing a message? Regards Stephan On Thu, Feb 4, 2016 at 7:06 PM, Quinn Stevenson <qu...@pronoia-solutions.com > wrote: > I tested this with a 5.9.0 broker and I am seeing messages dropped with > the TxText, but I still have to use the failover URL or the test just stops > after the broker is restarted. > > I don’t have a 5.9.1 broker to test with, so I don’t know if that would > help, but the next oldest broker I have is 5.10.1, and it seems to be > working with that broker. > > NOTE: I’m not changing the activemq-version in the POM when I change the > broker version - I’m just starting a different broker (locally) on the same > port. > > > > On Feb 4, 2016, at 10:41 AM, Quinn Stevenson < > qu...@pronoia-solutions.com> wrote: > > > > I still can’t make either test drop messages between the input and the > output queue with the POM changes I sent, but I did find one difference > between what you’ve done and what I normally do that changes the output I’m > seeing - I always use a failover URL > > > > <property name="brokerURL" > value="failover:(tcp://localhost:61616?wireFormat.tightEncodingEnabled=false > <tcp://localhost:61616?wireFormat.tightEncodingEnabled=false>)"/> > > > > My test broker is v 5.10.1 as well - I’ll see if it makes any difference > with 5.9.0 > > > > > > > >> On Feb 4, 2016, at 9:52 AM, Quinn Stevenson < > qu...@pronoia-solutions.com <mailto:qu...@pronoia-solutions.com>> wrote: > >> > >> It is strange - I’m trying to compare what you have in the “standard” > version to what I did before. We tested our configs pretty heavily under > all sorts of strange conditions to verify we weren’t looking messages, but > we were using newer versions of Camel and ActiveMQ. > >> > >> So we’re on the same page - can you try your tests again with POM > dependencies that look something like this? > >> > >> <properties> > >> <camel-version>2.12.5</camel-version> > >> <activemq-version>5.9.0</activemq-version> > >> </properties> > >> > >> <dependencies> > >> <dependency> > >> <groupId>org.apache.activemq</groupId> > >> <artifactId>activemq-all</artifactId> > >> <version>${activemq-version}</version> > >> </dependency> > >> <dependency> > >> <groupId>org.apache.activemq</groupId> > >> <artifactId>activemq-pool</artifactId> > >> <version>${activemq-version}</version> > >> </dependency> > >> > >> <dependency> > >> <groupId>org.apache.camel</groupId> > >> <artifactId>camel-spring</artifactId> > >> <version>${camel-version}</version> > >> </dependency> > >> <dependency> > >> <groupId>org.apache.camel</groupId> > >> <artifactId>camel-jms</artifactId> > >> <version>${camel-version}</version> > >> </dependency> > >> > >> <dependency> > >> <groupId>org.apache.camel</groupId> > >> <artifactId>camel-test-spring</artifactId> > >> <version>${camel-version}</version> > >> <scope>test</scope> > >> </dependency> > >> > >> <dependency> > >> <groupId>commons-collections</groupId> > >> <artifactId>commons-collections</artifactId> > >> <version>3.2.1</version> > >> <scope>test</scope> > >> </dependency> > >> <dependency> > >> <groupId>org.hamcrest</groupId> > >> <artifactId>hamcrest-integration</artifactId> > >> <version>1.3</version> > >> <scope>test</scope> > >> </dependency> > >> > >> </dependencies> > >> > >> > >> > >>> On Feb 4, 2016, at 9:49 AM, Stephan Burkard <sburk...@gmail.com > <mailto:sburk...@gmail.com>> wrote: > >>> > >>> Hi Quinn > >>> > >>> The "standard" version is the big mystery. As I stated in my first > post, a > >>> Redhat engineer analysed a similar project (with less book-keeping and > >>> logging stuff) and his conclusion was that as soon as a transaction > manager > >>> is explicitly defined, Spring JMS Template (that is used by Camel > under the > >>> hood) creates two of them by bug, by accident or just by strange > behaviour. > >>> > >>> This conclusion was quite suprising since that meant that all our > Camel-JMS > >>> project are theoretically suffering from message loss. > >>> > >>> The "no-tx" version should definitely be OK, see also CAMEL-5055 for > the " > >>> lazyCreateTransactionManager" flag. The JMS transaction manager may > not be > >>> defined but it creates one implicitly because of "transacted = true". > >>> > >>> The two "flaws" you mentioned are perhaps an issue. It would be somehow > >>> calming if it is my project who has a flaw. > >>> > >>> Regards > >>> Stephan > >>> > >>> > >>> > >>> On Thu, Feb 4, 2016 at 4:44 PM, Quinn Stevenson < > qu...@pronoia-solutions.com <mailto:qu...@pronoia-solutions.com> > >>>> wrote: > >>> > >>>> I’m still going through the project, but the first couple of things > that > >>>> jump out at me are you have two Spring versions - the one you > explicitly > >>>> put in your POM (3.2.8.RELEASE) and the one pulled in by camel-spring > >>>> (3.2.11.RELEASE). Also, camel-spring should be included in the POM > since > >>>> you’re using Spring routes. I’m not sure if that’s enough to cause > issues > >>>> or not. > >>>> > >>>> I believe what’s going on with the “no-tx” version is you’re actually > >>>> using JMS transactions since you still have transacted set to true in > the > >>>> JmsConfiguration. > >>>> > >>>> I’m not sure what’s going in with the “standard” version - it looks > >>>> similar to some XA stuff I’ve setup before (because I had multiple > brokers > >>>> involved) except I had to use XA Connection Factories. > >>>> > >>>> > >>>> > >>>>> On Feb 3, 2016, at 3:12 PM, Stephan Burkard <sburk...@gmail.com > <mailto:sburk...@gmail.com>> wrote: > >>>>> > >>>>> Yes, same broker. There is only one ActiveMQ connection config in the > >>>>> project. > >>>>> > >>>>> > >>>>> > >>>>> On Wed, Feb 3, 2016 at 8:00 PM, Quinn Stevenson < > >>>> qu...@pronoia-solutions.com <mailto:qu...@pronoia-solutions.com> > >>>>>> wrote: > >>>>> > >>>>>> Are both the source and destination queues hosted by the same > ActiveMQ > >>>>>> broker? > >>>>>> > >>>>>> > >>>>>> > >>>>>>> On Feb 3, 2016, at 8:21 AM, Stephan Burkard <sburk...@gmail.com > <mailto:sburk...@gmail.com>> > >>>> wrote: > >>>>>>> > >>>>>>> Hi > >>>>>>> > >>>>>>> I have built a small Maven project (attached) to demonstrate a JMS > >>>>>> transaction problem in Camel routes under certain load conditions. > In > >>>> fact > >>>>>> I am losing messages between two queues. > >>>>>>> > >>>>>>> The project contains two different flavours of the same test. One > of > >>>>>> them suffers from the problem, the other (due to my tests) not. > >>>>>>> > >>>>>>> > >>>>>>> *** What does the testcase? > >>>>>>> 1. Produces 1000 messages (100/s) and sends them to an "input" > queue. > >>>>>>> 2. Sends the messages from the "input" queue to an "output" queue. > >>>>>>> 3. Finally consumes the messages from the "output" queue to count > them. > >>>>>>> > >>>>>>> > >>>>>>> *** What is the difference between the two test flavours? > >>>>>>> - There is a "standard" flavour that suffers from the problem > >>>>>>> - And there is a "noTxManager" flavour that seems to not have the > >>>> problem > >>>>>>> - The "standard" flavour is kind of a well known Camel/ActiveMQ > >>>>>> configuration > >>>>>>> - with a Spring transaction manager > >>>>>>> - with a Spring transaction policy > >>>>>>> - With a "transacted" flag in Camel routes > >>>>>>> - The "noTxManager" flavour is a "simple" configuration > >>>>>>> - no Spring transaction manager > >>>>>>> - no Spring transaction policy > >>>>>>> - no "transacted" flag in Camel routes > >>>>>>> - BUT: "lazyCreateTransactionManager" = false (so routes are > >>>>>> transacted too) > >>>>>>> > >>>>>>> > >>>>>>> *** How to run the testcases? > >>>>>>> 1. Replace "[yourBrokerHost]" with the hostname of your ActiveMQ > broker > >>>>>>> 2. Run the testcase as JUnit test > >>>>>>> 3. When you see lots of console messages that messages are sent, > stop > >>>>>> your ActiveMQ broker (do not kill-9 it, just shut it down normally) > >>>>>>> 4. Exceptions are thrown on the console output > >>>>>>> 5. After some seconds start your broker again > >>>>>>> 6. The test finish normally and after some seconds dumps a book > keeping > >>>>>> on the console > >>>>>>> > >>>>>>> > >>>>>>> *** How to interpret the results? > >>>>>>> - When the test is successful, no message is lost. You can run the > test > >>>>>> without broker shutdown/startup and it will obviously always be > >>>> successful. > >>>>>>> - When the test fails, one or more messages are lost between queue > >>>>>> "input" and "output". In my tests I was not able to run the > "standard" > >>>>>> flavour three times in a row successfully. About every second run > >>>> failed. > >>>>>> In contrast, the "noTxManager" flavour never failed in my tests. > >>>>>>> > >>>>>>> The book keeping for a failed test looks like the following. In > this > >>>>>> example Message number 281 is arrived at the input queue but not at > the > >>>>>> output queue. So it is lost. > >>>>>>> > >>>>>>> Messages created by Client: 1000 > >>>>>>> Client Exceptions during send: 0 [] > >>>>>>> > >>>>>>> Messages received at input queue: 993 > >>>>>>> Missing Messages at input queue: 7 > [282,283,284,285,286,287,288] > >>>>>>> Duplicate Messages at input queue: 0 [] > >>>>>>> > >>>>>>> Messages received at output queue: 992 > >>>>>>> Missing Messages at output queue: 8 > >>>> [281,282,283,284,285,286,287,288] > >>>>>>> Duplicate Messages at output queue: 0 [] > >>>>>>> > >>>>>>> Lost Messages between Queues: 1 [281] > >>>>>>> > >>>>>>> > >>>>>>> *** What is the problem? > >>>>>>> A Redhat engineer tracked the problem down to a Spring JMS template > >>>>>> behaviour that is kind of strange. If a Spring transaction manager > is > >>>>>> defined in the config, it will end up with two of them. Therefore > the > >>>> small > >>>>>> time range where messages can get lost that arises only when you > have a > >>>>>> certain load. > >>>>>>> > >>>>>>> > >>>>>>> *** So, what is my question? > >>>>>>> - Does this really mean that it is unsafe to use the "standard" > flavour > >>>>>> of configuration? > >>>>>>> - Is there another config with TxManager etc that works correctly? > >>>>>>> - What are limits of the "noTxManager" config? When is it not > >>>> sufficent? > >>>>>>> > >>>>>>> Regards > >>>>>>> Stephan > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> <CamelAmqTxTest.zip> > >>>>>> > >>>>>> > >>>> > >>>> > >> > > > >
