[jira] Updated: (QPID-1618) unacked persistent messages don't get to messageStore of newbie cluster broker.
[ https://issues.apache.org/jira/browse/QPID-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] michael j. goulish updated QPID-1618: - Attachment: qpid_1618.diff > unacked persistent messages don't get to messageStore of newbie cluster > broker. > --- > > Key: QPID-1618 > URL: https://issues.apache.org/jira/browse/QPID-1618 > Project: Qpid > Issue Type: Bug > Components: C++ Broker >Affects Versions: M4 >Reporter: michael j. goulish >Priority: Critical > Attachments: qpid_1618.diff > > > When a new broker is added to a cluster, persistent messages that have not > yet been ACKed do not get stored in the new brokers messageStore. > How to reproduce: ( this is overview -- actual scripts follow ) > > 1. kill any MY_CLUSTER brokers from previous test > 2. remove data dirs from previous test, and recreate > 3. start node 1 as first member of MY_CLUSTER > 4 declare the MY_CLUSTER queue -- durable > 5. connect to it with receiver -- use ack frequency of 10; > 6. connect with "publish" send only 5 persistent msgs, receiver >will get them but not ack. > 7. start second node > 8. when the state transfer completes kill both nodes. >(receiver should also perish) > 9. start up the second node only, as new first member of MY_CLUSTER. >( I.e. use 2nd broker's data dir path. ) > 10. start and attach a new receiver > PREVIOUS RESULT -- nothing. messages were never stored in broker2's >messageStore. > RESULT -- new receiver now gets 5 messages. > durable queue says that "sender confirmed point moved to (5+0)" > == > Scripts for reproducing problem. > == > ### > #step 1 ## > ### > #! /bin/bash > echo "" > echo "Step 1: Kill any brokers already running in the MY_CLUSTER cluster" > echo "" > kill -9 `pgrep -f MY_STORE` > echo "There should be no remaining brokers. Here they are: " > ps -aef | grep qpidd | grep -v grep > ### > #step 2 ## > ### > #! /bin/bash > echo "" > echo "Step 2: remove and rebuild the store data dirs." > echo "" > rm -rf ./data > mkdir -p ./data/1 ./data/2 > ### > #step 3 ## > ### > #! /bin/bash > echo "" > echo "Step 3: Start the first broker in a cluster, using Store in ./data/1 ." > echo "" > rm broker_1.log > $QPID_ROOT/cpp/src/qpidd --no-module-dir\ > --load-module $QPID_ROOT/cpp/src/.libs/cluster.so \ > --load-module $STORE_ROOT/cpp/lib/.libs/msgstore.so \ > --cluster-name MY_CLUSTER -p 5813 \ > --auth=no --mgmt-enable=no\ > --log-enable debug --log-to-file ./broker_1.log \ > --data-dir ./data/1 > ### > #step 4 ## > ### > #! /bin/bash > echo "Step 4: Declaring queue." > $QPID_ROOT/cpp/examples/direct/declare_queues > # !! NOTE !! > # edit declare_queues.cpp to do port 5813 > # in call to session.queueDeclare use arg::durable=true > # and arg::queue="MY_QUEUE", > # edit exchangeBind call to use: > # arg::queue = "MY_QUEUE", > ### > #step 5 ## > ### > #! /bin/bash > echo "Step 5: Starting receiver..." > $QPID_ROOT/cpp/src/tests/receiver \ > -p 5813 \ > --queue MY_QUEUE \ > --messages 10 \ > --ack-frequency 10 > ### > #step 6 ## >
[jira] Updated: (QPID-1618) unacked persistent messages don't get to messageStore of newbie cluster broker.
[ https://issues.apache.org/jira/browse/QPID-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] michael j. goulish updated QPID-1618: - Attachment: (was: qpid_1618.diff) > unacked persistent messages don't get to messageStore of newbie cluster > broker. > --- > > Key: QPID-1618 > URL: https://issues.apache.org/jira/browse/QPID-1618 > Project: Qpid > Issue Type: Bug > Components: C++ Broker >Affects Versions: M4 >Reporter: michael j. goulish >Priority: Critical > Attachments: qpid_1618.diff > > > When a new broker is added to a cluster, persistent messages that have not > yet been ACKed do not get stored in the new brokers messageStore. > How to reproduce: ( this is overview -- actual scripts follow ) > > 1. kill any MY_CLUSTER brokers from previous test > 2. remove data dirs from previous test, and recreate > 3. start node 1 as first member of MY_CLUSTER > 4 declare the MY_CLUSTER queue -- durable > 5. connect to it with receiver -- use ack frequency of 10; > 6. connect with "publish" send only 5 persistent msgs, receiver >will get them but not ack. > 7. start second node > 8. when the state transfer completes kill both nodes. >(receiver should also perish) > 9. start up the second node only, as new first member of MY_CLUSTER. >( I.e. use 2nd broker's data dir path. ) > 10. start and attach a new receiver > PREVIOUS RESULT -- nothing. messages were never stored in broker2's >messageStore. > RESULT -- new receiver now gets 5 messages. > durable queue says that "sender confirmed point moved to (5+0)" > == > Scripts for reproducing problem. > == > ### > #step 1 ## > ### > #! /bin/bash > echo "" > echo "Step 1: Kill any brokers already running in the MY_CLUSTER cluster" > echo "" > kill -9 `pgrep -f MY_STORE` > echo "There should be no remaining brokers. Here they are: " > ps -aef | grep qpidd | grep -v grep > ### > #step 2 ## > ### > #! /bin/bash > echo "" > echo "Step 2: remove and rebuild the store data dirs." > echo "" > rm -rf ./data > mkdir -p ./data/1 ./data/2 > ### > #step 3 ## > ### > #! /bin/bash > echo "" > echo "Step 3: Start the first broker in a cluster, using Store in ./data/1 ." > echo "" > rm broker_1.log > $QPID_ROOT/cpp/src/qpidd --no-module-dir\ > --load-module $QPID_ROOT/cpp/src/.libs/cluster.so \ > --load-module $STORE_ROOT/cpp/lib/.libs/msgstore.so \ > --cluster-name MY_CLUSTER -p 5813 \ > --auth=no --mgmt-enable=no\ > --log-enable debug --log-to-file ./broker_1.log \ > --data-dir ./data/1 > ### > #step 4 ## > ### > #! /bin/bash > echo "Step 4: Declaring queue." > $QPID_ROOT/cpp/examples/direct/declare_queues > # !! NOTE !! > # edit declare_queues.cpp to do port 5813 > # in call to session.queueDeclare use arg::durable=true > # and arg::queue="MY_QUEUE", > # edit exchangeBind call to use: > # arg::queue = "MY_QUEUE", > ### > #step 5 ## > ### > #! /bin/bash > echo "Step 5: Starting receiver..." > $QPID_ROOT/cpp/src/tests/receiver \ > -p 5813 \ > --queue MY_QUEUE \ > --messages 10 \ > --ack-frequency 10 > ### > #step 6 ## > #
[jira] Updated: (QPID-1618) unacked persistent messages don't get to messageStore of newbie cluster broker.
[ https://issues.apache.org/jira/browse/QPID-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] michael j. goulish updated QPID-1618: - Attachment: qpid_1618.diff almost the same change as before -- force message enqueue-to-store during cluster newbie braindump if the message has not been acked. except now there's a unit test. Unit test does this: 1. derive new class from NullMessageStore 2. make a plugin out of it. 3. use ForkedBroker to make a 2-cluster where both brokers use these specialized NullMessageStores 4. Make sure that the second one enqueues its unacked messages. (In this case "enqueue" means "put in the store" (!) ) > unacked persistent messages don't get to messageStore of newbie cluster > broker. > --- > > Key: QPID-1618 > URL: https://issues.apache.org/jira/browse/QPID-1618 > Project: Qpid > Issue Type: Bug > Components: C++ Broker >Affects Versions: M4 >Reporter: michael j. goulish >Priority: Critical > Attachments: qpid_1618.diff > > > When a new broker is added to a cluster, persistent messages that have not > yet been ACKed do not get stored in the new brokers messageStore. > How to reproduce: ( this is overview -- actual scripts follow ) > > 1. kill any MY_CLUSTER brokers from previous test > 2. remove data dirs from previous test, and recreate > 3. start node 1 as first member of MY_CLUSTER > 4 declare the MY_CLUSTER queue -- durable > 5. connect to it with receiver -- use ack frequency of 10; > 6. connect with "publish" send only 5 persistent msgs, receiver >will get them but not ack. > 7. start second node > 8. when the state transfer completes kill both nodes. >(receiver should also perish) > 9. start up the second node only, as new first member of MY_CLUSTER. >( I.e. use 2nd broker's data dir path. ) > 10. start and attach a new receiver > PREVIOUS RESULT -- nothing. messages were never stored in broker2's >messageStore. > RESULT -- new receiver now gets 5 messages. > durable queue says that "sender confirmed point moved to (5+0)" > == > Scripts for reproducing problem. > == > ### > #step 1 ## > ### > #! /bin/bash > echo "" > echo "Step 1: Kill any brokers already running in the MY_CLUSTER cluster" > echo "" > kill -9 `pgrep -f MY_STORE` > echo "There should be no remaining brokers. Here they are: " > ps -aef | grep qpidd | grep -v grep > ### > #step 2 ## > ### > #! /bin/bash > echo "" > echo "Step 2: remove and rebuild the store data dirs." > echo "" > rm -rf ./data > mkdir -p ./data/1 ./data/2 > ### > #step 3 ## > ### > #! /bin/bash > echo "" > echo "Step 3: Start the first broker in a cluster, using Store in ./data/1 ." > echo "" > rm broker_1.log > $QPID_ROOT/cpp/src/qpidd --no-module-dir\ > --load-module $QPID_ROOT/cpp/src/.libs/cluster.so \ > --load-module $STORE_ROOT/cpp/lib/.libs/msgstore.so \ > --cluster-name MY_CLUSTER -p 5813 \ > --auth=no --mgmt-enable=no\ > --log-enable debug --log-to-file ./broker_1.log \ > --data-dir ./data/1 > ### > #step 4 ## > ### > #! /bin/bash > echo "Step 4: Declaring queue." > $QPID_ROOT/cpp/examples/direct/declare_queues > # !! NOTE !! > # edit declare_queues.cpp to do port 5813 > # in call to session.queueDeclare use arg::durable=true > # and arg::queue="MY_QUEUE", > # edit exchangeBind call to use: > # arg::queue = "MY_QUEUE", > ### > #step 5
[jira] Updated: (QPID-1618) unacked persistent messages don't get to messageStore of newbie cluster broker.
[ https://issues.apache.org/jira/browse/QPID-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] michael j. goulish updated QPID-1618: - Attachment: (was: persist.diff) > unacked persistent messages don't get to messageStore of newbie cluster > broker. > --- > > Key: QPID-1618 > URL: https://issues.apache.org/jira/browse/QPID-1618 > Project: Qpid > Issue Type: Bug > Components: C++ Broker >Affects Versions: M4 >Reporter: michael j. goulish >Priority: Critical > > When a new broker is added to a cluster, persistent messages that have not > yet been ACKed do not get stored in the new brokers messageStore. > How to reproduce: ( this is overview -- actual scripts follow ) > > 1. kill any MY_CLUSTER brokers from previous test > 2. remove data dirs from previous test, and recreate > 3. start node 1 as first member of MY_CLUSTER > 4 declare the MY_CLUSTER queue -- durable > 5. connect to it with receiver -- use ack frequency of 10; > 6. connect with "publish" send only 5 persistent msgs, receiver >will get them but not ack. > 7. start second node > 8. when the state transfer completes kill both nodes. >(receiver should also perish) > 9. start up the second node only, as new first member of MY_CLUSTER. >( I.e. use 2nd broker's data dir path. ) > 10. start and attach a new receiver > PREVIOUS RESULT -- nothing. messages were never stored in broker2's >messageStore. > RESULT -- new receiver now gets 5 messages. > durable queue says that "sender confirmed point moved to (5+0)" > == > Scripts for reproducing problem. > == > ### > #step 1 ## > ### > #! /bin/bash > echo "" > echo "Step 1: Kill any brokers already running in the MY_CLUSTER cluster" > echo "" > kill -9 `pgrep -f MY_STORE` > echo "There should be no remaining brokers. Here they are: " > ps -aef | grep qpidd | grep -v grep > ### > #step 2 ## > ### > #! /bin/bash > echo "" > echo "Step 2: remove and rebuild the store data dirs." > echo "" > rm -rf ./data > mkdir -p ./data/1 ./data/2 > ### > #step 3 ## > ### > #! /bin/bash > echo "" > echo "Step 3: Start the first broker in a cluster, using Store in ./data/1 ." > echo "" > rm broker_1.log > $QPID_ROOT/cpp/src/qpidd --no-module-dir\ > --load-module $QPID_ROOT/cpp/src/.libs/cluster.so \ > --load-module $STORE_ROOT/cpp/lib/.libs/msgstore.so \ > --cluster-name MY_CLUSTER -p 5813 \ > --auth=no --mgmt-enable=no\ > --log-enable debug --log-to-file ./broker_1.log \ > --data-dir ./data/1 > ### > #step 4 ## > ### > #! /bin/bash > echo "Step 4: Declaring queue." > $QPID_ROOT/cpp/examples/direct/declare_queues > # !! NOTE !! > # edit declare_queues.cpp to do port 5813 > # in call to session.queueDeclare use arg::durable=true > # and arg::queue="MY_QUEUE", > # edit exchangeBind call to use: > # arg::queue = "MY_QUEUE", > ### > #step 5 ## > ### > #! /bin/bash > echo "Step 5: Starting receiver..." > $QPID_ROOT/cpp/src/tests/receiver \ > -p 5813 \ > --queue MY_QUEUE \ > --messages 10 \ > --ack-frequency 10 > ### > #step 6 ## > ###
[jira] Updated: (QPID-1618) unacked persistent messages don't get to messageStore of newbie cluster broker.
[ https://issues.apache.org/jira/browse/QPID-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] michael j. goulish updated QPID-1618: - Attachment: persist.diff New fix for same problem. > unacked persistent messages don't get to messageStore of newbie cluster > broker. > --- > > Key: QPID-1618 > URL: https://issues.apache.org/jira/browse/QPID-1618 > Project: Qpid > Issue Type: Bug > Components: C++ Broker >Affects Versions: M4 >Reporter: michael j. goulish >Priority: Critical > Attachments: persist.diff > > > When a new broker is added to a cluster, persistent messages that have not > yet been ACKed do not get stored in the new brokers messageStore. > How to reproduce: ( this is overview -- actual scripts follow ) > > 1. kill any MY_CLUSTER brokers from previous test > 2. remove data dirs from previous test, and recreate > 3. start node 1 as first member of MY_CLUSTER > 4 declare the MY_CLUSTER queue -- durable > 5. connect to it with receiver -- use ack frequency of 10; > 6. connect with "publish" send only 5 persistent msgs, receiver >will get them but not ack. > 7. start second node > 8. when the state transfer completes kill both nodes. >(receiver should also perish) > 9. start up the second node only, as new first member of MY_CLUSTER. >( I.e. use 2nd broker's data dir path. ) > 10. start and attach a new receiver > PREVIOUS RESULT -- nothing. messages were never stored in broker2's >messageStore. > RESULT -- new receiver now gets 5 messages. > durable queue says that "sender confirmed point moved to (5+0)" > == > Scripts for reproducing problem. > == > ### > #step 1 ## > ### > #! /bin/bash > echo "" > echo "Step 1: Kill any brokers already running in the MY_CLUSTER cluster" > echo "" > kill -9 `pgrep -f MY_STORE` > echo "There should be no remaining brokers. Here they are: " > ps -aef | grep qpidd | grep -v grep > ### > #step 2 ## > ### > #! /bin/bash > echo "" > echo "Step 2: remove and rebuild the store data dirs." > echo "" > rm -rf ./data > mkdir -p ./data/1 ./data/2 > ### > #step 3 ## > ### > #! /bin/bash > echo "" > echo "Step 3: Start the first broker in a cluster, using Store in ./data/1 ." > echo "" > rm broker_1.log > $QPID_ROOT/cpp/src/qpidd --no-module-dir\ > --load-module $QPID_ROOT/cpp/src/.libs/cluster.so \ > --load-module $STORE_ROOT/cpp/lib/.libs/msgstore.so \ > --cluster-name MY_CLUSTER -p 5813 \ > --auth=no --mgmt-enable=no\ > --log-enable debug --log-to-file ./broker_1.log \ > --data-dir ./data/1 > ### > #step 4 ## > ### > #! /bin/bash > echo "Step 4: Declaring queue." > $QPID_ROOT/cpp/examples/direct/declare_queues > # !! NOTE !! > # edit declare_queues.cpp to do port 5813 > # in call to session.queueDeclare use arg::durable=true > # and arg::queue="MY_QUEUE", > # edit exchangeBind call to use: > # arg::queue = "MY_QUEUE", > ### > #step 5 ## > ### > #! /bin/bash > echo "Step 5: Starting receiver..." > $QPID_ROOT/cpp/src/tests/receiver \ > -p 5813 \ > --queue MY_QUEUE \ > --messages 10 \ > --ack-frequency 10 > ### > #step 6 ## >
[jira] Updated: (QPID-1618) unacked persistent messages don't get to messageStore of newbie cluster broker.
[ https://issues.apache.org/jira/browse/QPID-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] michael j. goulish updated QPID-1618: - Attachment: (was: unack_newbie.diff) > unacked persistent messages don't get to messageStore of newbie cluster > broker. > --- > > Key: QPID-1618 > URL: https://issues.apache.org/jira/browse/QPID-1618 > Project: Qpid > Issue Type: Bug > Components: C++ Broker >Affects Versions: M4 >Reporter: michael j. goulish >Priority: Critical > > When a new broker is added to a cluster, persistent messages that have not > yet been ACKed do not get stored in the new brokers messageStore. > How to reproduce: ( this is overview -- actual scripts follow ) > > 1. kill any MY_CLUSTER brokers from previous test > 2. remove data dirs from previous test, and recreate > 3. start node 1 as first member of MY_CLUSTER > 4 declare the MY_CLUSTER queue -- durable > 5. connect to it with receiver -- use ack frequency of 10; > 6. connect with "publish" send only 5 persistent msgs, receiver >will get them but not ack. > 7. start second node > 8. when the state transfer completes kill both nodes. >(receiver should also perish) > 9. start up the second node only, as new first member of MY_CLUSTER. >( I.e. use 2nd broker's data dir path. ) > 10. start and attach a new receiver > PREVIOUS RESULT -- nothing. messages were never stored in broker2's >messageStore. > RESULT -- new receiver now gets 5 messages. > durable queue says that "sender confirmed point moved to (5+0)" > == > Scripts for reproducing problem. > == > ### > #step 1 ## > ### > #! /bin/bash > echo "" > echo "Step 1: Kill any brokers already running in the MY_CLUSTER cluster" > echo "" > kill -9 `pgrep -f MY_STORE` > echo "There should be no remaining brokers. Here they are: " > ps -aef | grep qpidd | grep -v grep > ### > #step 2 ## > ### > #! /bin/bash > echo "" > echo "Step 2: remove and rebuild the store data dirs." > echo "" > rm -rf ./data > mkdir -p ./data/1 ./data/2 > ### > #step 3 ## > ### > #! /bin/bash > echo "" > echo "Step 3: Start the first broker in a cluster, using Store in ./data/1 ." > echo "" > rm broker_1.log > $QPID_ROOT/cpp/src/qpidd --no-module-dir\ > --load-module $QPID_ROOT/cpp/src/.libs/cluster.so \ > --load-module $STORE_ROOT/cpp/lib/.libs/msgstore.so \ > --cluster-name MY_CLUSTER -p 5813 \ > --auth=no --mgmt-enable=no\ > --log-enable debug --log-to-file ./broker_1.log \ > --data-dir ./data/1 > ### > #step 4 ## > ### > #! /bin/bash > echo "Step 4: Declaring queue." > $QPID_ROOT/cpp/examples/direct/declare_queues > # !! NOTE !! > # edit declare_queues.cpp to do port 5813 > # in call to session.queueDeclare use arg::durable=true > # and arg::queue="MY_QUEUE", > # edit exchangeBind call to use: > # arg::queue = "MY_QUEUE", > ### > #step 5 ## > ### > #! /bin/bash > echo "Step 5: Starting receiver..." > $QPID_ROOT/cpp/src/tests/receiver \ > -p 5813 \ > --queue MY_QUEUE \ > --messages 10 \ > --ack-frequency 10 > ### > #step 6 ## > ##
[jira] Updated: (QPID-1618) unacked persistent messages don't get to messageStore of newbie cluster broker.
[ https://issues.apache.org/jira/browse/QPID-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] michael j. goulish updated QPID-1618: - Attachment: unack_newbie.diff > unacked persistent messages don't get to messageStore of newbie cluster > broker. > --- > > Key: QPID-1618 > URL: https://issues.apache.org/jira/browse/QPID-1618 > Project: Qpid > Issue Type: Bug > Components: C++ Broker >Affects Versions: M4 >Reporter: michael j. goulish >Priority: Critical > Attachments: unack_newbie.diff > > > When a new broker is added to a cluster, persistent messages that have not > yet been ACKed do not get stored in the new brokers messageStore. > How to reproduce: ( this is overview -- actual scripts follow ) > > 1. kill any MY_CLUSTER brokers from previous test > 2. remove data dirs from previous test, and recreate > 3. start node 1 as first member of MY_CLUSTER > 4 declare the MY_CLUSTER queue -- durable > 5. connect to it with receiver -- use ack frequency of 10; > 6. connect with "publish" send only 5 persistent msgs, receiver >will get them but not ack. > 7. start second node > 8. when the state transfer completes kill both nodes. >(receiver should also perish) > 9. start up the second node only, as new first member of MY_CLUSTER. >( I.e. use 2nd broker's data dir path. ) > 10. start and attach a new receiver > PREVIOUS RESULT -- nothing. messages were never stored in broker2's >messageStore. > RESULT -- new receiver now gets 5 messages. > durable queue says that "sender confirmed point moved to (5+0)" > == > Scripts for reproducing problem. > == > ### > #step 1 ## > ### > #! /bin/bash > echo "" > echo "Step 1: Kill any brokers already running in the MY_CLUSTER cluster" > echo "" > kill -9 `pgrep -f MY_STORE` > echo "There should be no remaining brokers. Here they are: " > ps -aef | grep qpidd | grep -v grep > ### > #step 2 ## > ### > #! /bin/bash > echo "" > echo "Step 2: remove and rebuild the store data dirs." > echo "" > rm -rf ./data > mkdir -p ./data/1 ./data/2 > ### > #step 3 ## > ### > #! /bin/bash > echo "" > echo "Step 3: Start the first broker in a cluster, using Store in ./data/1 ." > echo "" > rm broker_1.log > $QPID_ROOT/cpp/src/qpidd --no-module-dir\ > --load-module $QPID_ROOT/cpp/src/.libs/cluster.so \ > --load-module $STORE_ROOT/cpp/lib/.libs/msgstore.so \ > --cluster-name MY_CLUSTER -p 5813 \ > --auth=no --mgmt-enable=no\ > --log-enable debug --log-to-file ./broker_1.log \ > --data-dir ./data/1 > ### > #step 4 ## > ### > #! /bin/bash > echo "Step 4: Declaring queue." > $QPID_ROOT/cpp/examples/direct/declare_queues > # !! NOTE !! > # edit declare_queues.cpp to do port 5813 > # in call to session.queueDeclare use arg::durable=true > # and arg::queue="MY_QUEUE", > # edit exchangeBind call to use: > # arg::queue = "MY_QUEUE", > ### > #step 5 ## > ### > #! /bin/bash > echo "Step 5: Starting receiver..." > $QPID_ROOT/cpp/src/tests/receiver \ > -p 5813 \ > --queue MY_QUEUE \ > --messages 10 \ > --ack-frequency 10 > ### > #step 6 ## > ##
