stress tests not showing all messages being processed as predicted
------------------------------------------------------------------
Key: AMQNET-252
URL: https://issues.apache.org/activemq/browse/AMQNET-252
Project: ActiveMQ .Net
Issue Type: Bug
Components: NMS
Affects Versions: 1.3.0
Environment: ActiveMQ 5.2, Windows Server 2008 hosting brokers and
consumers/subscribers in windows services. producers sending messages from
Windows XP
Reporter: Mark Gellings
Assignee: Jim Gomes
Fix For: 1.3.0
Should say right away the below is the stress test model we used when
thoroughly testing NMS v1.1 (of which we use quite extensively in our
production environment). The tests pass consistently with NMS v1.1. These
tests are done using a framework we wrote that rests on top of Apache NMS
ActiveMQ provider. We can't distribute that code for proprietary reasons. I
did however include a console application to be used of which I have replicated
the problem on at least one occasion, albeit the stress test is not near as
sophisticated.
Stress Test model
// *note first character "P" is a producer
// *note first character "C" is a topic consumer
// *note first character "v" is a virtual topic
// *note character "t" is a topic
// *note all numbers denote seperate topics, producers, consumers, and brokers
// *note (durable) means a durable subscription
| |
P1 --- vt1 ----> | | ---- t1 (durable), vt1 ----> C1 ---- t2 ---->
BROKER 1
P2 ---- t1 ----> | |
P3 ---- t1 ----> | BROKER 1 | -- t1 (durable), t5 (durable) --> C2 -- t3 -->
P4 ---- t2 ----> | |
BROKER 2 --- t3 (durable), t4 ---> C4
P5 ---- t2 ----> | | -------- t1, t2 ------> C3 -------- t4 ------>
P6 ---- t5 ----> | |
| | ------- vt1, t5 (durable) -----> C5 ---- t2
----> BROKER 1
Stress Test #1
* First producer sends 10,000 msgs to VirtualTopic.t1
* Two producers send 10,000 msgs to t1.
* Two producers send 10,000 msgs to t2.
* One producer sends 10,000 msgs to t5.
With the above test all brokers, producers, and consumers were not restarted.
Test passes, results:
ProcessorName msg_cnt
QuadNMSConsumer/vt1 10000
QuadNMSConsumer/t11 20000
QuadNMSConsumer/t12 20000
QuadNMSConsumer/t13 20000
QuadNMSConsumer/t23 60000
QuadNMSConsumer/t34 30000
QuadNMSConsumer/t44 80000
QuadNMSConsumer/t52 10000
QuadNMSConsumer/t55 10000
Stress Test #2
* First producer sends 1,000 msgs to VirtualTopic.t1
* Two producers send 1,000 msgs to t1.
* Two producers send 1,000 msgs to t2.
* One producer sends 1,000 msgs to t5.
# Consumers use client acknowledgement
# During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least
once to ensure they receive all messages as they both have durable subscriptions
# We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running
jdbc master/slave setup) at least once to ensure that we do not receive
duplicate messages
The end results show (from what I can tell) a failed test according to vt1,
t12, t52, and t55. The other topic subscriptions are non-durable so the
restarts throw off the counts.
ProcessorName msg_cnt note
QuadNMSConsumer/vt1 990 should be 1000 messages
QuadNMSConsumer/t11 860 should be 2000 messages
QuadNMSConsumer/t12 822 should be 2000 messages
QuadNMSConsumer/t13 185
QuadNMSConsumer/t23 1802
QuadNMSConsumer/t34 1270
QuadNMSConsumer/t44 1987
QuadNMSConsumer/t52 448 should be 1000 messages
QuadNMSConsumer/t55 479 should be 1000 messages
Stress Test #3
* First producer sends 1,000 msgs to VirtualTopic.t1
* Two producers send 1,000 msgs to t1.
* Two producers send 1,000 msgs to t2.
* One producer sends 1,000 msgs to t5.
# Consumers use individual acknowledgement
# During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least
once to ensure they receive all messages as they both have durable subscriptions
# We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running
jdbc master/slave setup) at least once to ensure that we do not receive
duplicate messages, as the consumers use our home grown idempotent consumer
logic
The end results show another failed test.
ProcessorName msg_cnt note
QuadNMSConsumer/vt1 995 should be 1000 messages
QuadNMSConsumer/t11 2000 should be 2000 messages
QuadNMSConsumer/t12 1995 should be 2000 messages
QuadNMSConsumer/t13 329
QuadNMSConsumer/t23 3028
QuadNMSConsumer/t34 2988
QuadNMSConsumer/t44 3356
QuadNMSConsumer/t52 995 should be 1000 messages
QuadNMSConsumer/t55 995 should be 1000 messages
Not sure what the problem exactly is here but with NMS v1.1 these tests pass
consistently. I can see the message in the database sitting in the activemq
messages table with the failed stress tests.
These stress tests are rather large so I put together a console application to
replicate the problem with a virtual topic.
Putting this sort of stress test in a unit test probably wouldn't make sense
which is why I've attached the console with test case. We'll want to figure
out the problem and then write a small targeted unit test, ensure it fails,
make the fix and then ensure the unit test passes.
I should also note I can't replicate this problem easily without our framework
on top, but have replicated the problem with the test below at least once out
of about 5 tests.
Only thing I really have to go by at this point is the above stress tests pass
with NMS v1.1.
To run the test:
1) start a broker somewhere
2) run three instances of the attached console.
3) start two consumers processing a virtual topic
4) start one producer publishing 20,000 messages to the virtual topic
5) Press enter on the console consumers to restart them and restart the broker
a number of times
7) When all messages are processed by the consoles, look at the queue in the
web console. There will be at least one pending message on the virtual topic
but when you drill down the message doesn't exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
