unacked persistent messages don't get to messageStore of newbie cluster broker. -------------------------------------------------------------------------------
Key: QPID-1618 URL: https://issues.apache.org/jira/browse/QPID-1618 Project: Qpid Issue Type: Bug Components: C++ Broker Affects Versions: M4 Reporter: michael j. goulish Priority: Critical Attachments: unack_newbie.diff When a new broker is added to a cluster, persistent messages that have not yet been ACKed do not get stored in the new brokers messageStore. How to reproduce: ( this is overview -- actual scripts follow ) ======================================================== 1. kill any MY_CLUSTER brokers from previous test 2. remove data dirs from previous test, and recreate 3. start node 1 as first member of MY_CLUSTER 4 declare the MY_CLUSTER queue -- durable 5. connect to it with receiver -- use ack frequency of 10; 6. connect with "publish" send only 5 persistent msgs, receiver will get them but not ack. 7. start second node 8. when the state transfer completes kill both nodes. (receiver should also perish) 9. start up the second node only, as new first member of MY_CLUSTER. ( I.e. use 2nd broker's data dir path. ) 10. start and attach a new receiver PREVIOUS RESULT -- nothing. messages were never stored in broker2's messageStore. RESULT -- new receiver now gets 5 messages. durable queue says that "sender confirmed point moved to (5+0)" ============================================== Scripts for reproducing problem. ============================================== ################################################################### ############# step 1 ############## ################################################################### #! /bin/bash echo "------------------------------------------------" echo "Step 1: Kill any brokers already running in the MY_CLUSTER cluster" echo "------------------------------------------------" kill -9 `pgrep -f MY_STORE` echo "There should be no remaining brokers. Here they are: " ps -aef | grep qpidd | grep -v grep ################################################################### ############# step 2 ############## ################################################################### #! /bin/bash echo "------------------------------------------------" echo "Step 2: remove and rebuild the store data dirs." echo "------------------------------------------------" rm -rf ./data mkdir -p ./data/1 ./data/2 ################################################################### ############# step 3 ############## ################################################################### #! /bin/bash echo "------------------------------------------------" echo "Step 3: Start the first broker in a cluster, using Store in ./data/1 ." echo "------------------------------------------------" rm broker_1.log $QPID_ROOT/cpp/src/qpidd --no-module-dir \ --load-module $QPID_ROOT/cpp/src/.libs/cluster.so \ --load-module $STORE_ROOT/cpp/lib/.libs/msgstore.so \ --cluster-name MY_CLUSTER -p 5813 \ --auth=no --mgmt-enable=no \ --log-enable debug --log-to-file ./broker_1.log \ --data-dir ./data/1 ################################################################### ############# step 4 ############## ################################################################### #! /bin/bash echo "Step 4: Declaring queue." $QPID_ROOT/cpp/examples/direct/declare_queues # !! NOTE !! # edit declare_queues.cpp to do port 5813 # in call to session.queueDeclare use arg::durable=true # and arg::queue="MY_QUEUE", # edit exchangeBind call to use: # arg::queue = "MY_QUEUE", ################################################################### ############# step 5 ############## ################################################################### #! /bin/bash echo "Step 5: Starting receiver..." $QPID_ROOT/cpp/src/tests/receiver \ -p 5813 \ --queue MY_QUEUE \ --messages 10 \ --ack-frequency 10 ################################################################### ############# step 6 ############## ################################################################### #! /bin/bash echo "Publish only 5 messages, so the receiver will not yet ack." $QPID_ROOT/cpp/src/tests/publish \ -p 5813 \ --count 5 \ --durable yes \ --destination amq.direct \ --routing-key routing_key \ --log-enable debug #! /bin/bash ################################################################### ############# step 7 ############## ################################################################### rm ./broker_2.log echo "------------------------------------------------" echo "Step 7: Start second node, using Store in ./data/2 ." echo " note! you should see correct store dirs replicated to this broker." echo "------------------------------------------------" $QPID_ROOT/cpp/src/qpidd --no-module-dir \ --load-module $QPID_ROOT/cpp/src/.libs/cluster.so \ --load-module $STORE_ROOT/cpp/lib/.libs/msgstore.so \ --cluster-name MY_CLUSTER -p 5814 \ --auth=no --mgmt-enable=no \ --log-enable debug --log-to-file ./broker_2.log \ --data-dir ./data/2 ################################################################### ############# step 8 ############## ################################################################### #! /bin/bash echo "When the second node is up and synced, kill both nodes." echo " note: receiver should now die also." kill -9 `pgrep -f MY_CLUSTER` ps -aef | grep qpidd | grep -v grep ################################################################### ############# step 9 ############## ################################################################### #! /bin/bash echo "------------------------------------------------" echo "Step 9: Start second node, using Store in ./data/2 ." echo "------------------------------------------------" $QPID_ROOT/cpp/src/qpidd --no-module-dir \ --load-module $QPID_ROOT/cpp/src/.libs/cluster.so \ --load-module $STORE_ROOT/cpp/lib/.libs/msgstore.so \ --cluster-name MY_CLUSTER -p 5813 \ --auth=no --mgmt-enable=yes \ --log-enable debug --log-to-file ./broker_3.log \ --data-dir ./data/2 ################################################################### ############# step 10 ############## ################################################################### #! /bin/bash echo " " echo "Start final receiver -- it should get unacked messages." echo " " $QPID_ROOT/cpp/src/tests/receiver \ -p 5813 \ --queue MY_QUEUE \ --messages 10 \ --ack-frequency 5 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org