[jira] [Commented] (QPID-7149) [HA] active HA broker memory leak when ring queue discards overflow messages
[
https://issues.apache.org/jira/browse/QPID-7149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227195#comment-15227195
]
ASF subversion and git services commented on QPID-7149:
---
Commit 1737887 from [~aconway] in branch 'qpid/trunk'
[ https://svn.apache.org/r1737887 ]
QPID-7149: Fixed messaging client shutdown.
Fix previous commit that did not respect the plugin boundaries for
messaging::shutdown.
> [HA] active HA broker memory leak when ring queue discards overflow messages
>
>
> Key: QPID-7149
> URL: https://issues.apache.org/jira/browse/QPID-7149
> Project: Qpid
> Issue Type: Bug
> Components: C++ Broker
> Environment: RHEL6
> qpid trunk svn rev. 1735384
> - issue seen in very old releases (since active-passive HA cluster initial
> implementation, most probably)
> libstdc++-devel-4.4.7-4.el6.x86_64
> gcc-c++-4.4.7-4.el6.x86_64
> libgcc-4.4.7-4.el6.x86_64
> libstdc++-4.4.7-4.el6.x86_64
> gcc-4.4.7-4.el6.x86_64
>Reporter: Pavel Moravec
>Assignee: Alan Conway
>
> There is a memory leak on active HA broker, triggered most probably by
> purging overflow message from a ring queue. Basic scenario is to setup HA
> cluster, promote to primary and feed forever a ring queue with messages.
> Detailed scenario:
> 1) Start brokers and promote one to primary:
> {noformat}
> start_broker() {
> port=$1
> shift
> rm -rf _${port}
> mkdir _${port}
> nohup qpidd --load-module=ha.so --port=$port
> --log-to-file=qpidd.$port.log --data-dir=_${port} --auth=no
> --log-to-stderr=no --ha-cluster=yes
> --ha-brokers-url="$(hostname):5672,$(hostname):5673,$(hostname):5674"
> --ha-replicate=all --acl-file=/root/qpidd.acl "$@" > /dev/null 2>&1 &
> sleep 1
> }
> killall qpidd qpid-receive 2> /dev/null
> rm -f qpidd.*.log
> start_broker 5672
> sleep 1
> qpid-ha promote -b $(hostname):5672 --cluster-manager
> sleep 1
> start_broker 5673
> sleep 1
> start_broker 5674
> {noformat}
> 2) Create ring queues and send there messages (it is enough to have 1 queue,
> having more should show the leak faster):
> {noformat}
> for i in $(seq 0 9); do
> qpid-config add queue FromKeyServer_$i --max-queue-size=1
> --max-queue-count=10 --limit-policy=ring --argument=x-qpid-priorities=10
> done
> while true; do
> for j in $(seq 1 10); do
> for i in $(seq 1 10); do
> for k in $(seq 0 9); do
> qpid-send -a FromKeyServer_$k -m 100
> --send-rate=50 -- priority=$(($((RANDOM))%10)) &
> done
> done
> wait
> while [ $(qpid-stat -q | grep broker-replicator | sed "s/Y//g"
> | awk '{ print $2 }' | sort -n | tail -n1) != "0" ]; do
> sleep 1
> done
> done
> date
> ps aux | grep qpidd | grep "port=5672" | awk -F "--store-dir" '{ print
> $1 }'
> done
> {noformat}
> (the "while [ $(qpid-stat -q | .." cycle is there just to slow down the
> message enqueues to ensure replication federation queues dont have big
> backlog - that would interfere with memory consumpiton observation)
> 3) Run those scripts and monitor memory consumption.
> - without using priority queues and sending messages without priorities, leak
> is evident as well - sometimes smaller, sometimes the same
> - valgrind (on some older versions I tested before more thoroughly) detects
> nothing (neither leaked memory or reachable at shutdown)
> - same leak is evident even with --ha-replicate=none
> - number of backup brokers does not affect the memory leak
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7149) [HA] active HA broker memory leak when ring queue discards overflow messages
[
https://issues.apache.org/jira/browse/QPID-7149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226551#comment-15226551
]
ASF subversion and git services commented on QPID-7149:
---
Commit 1737852 from [~aconway] in branch 'qpid/trunk'
[ https://svn.apache.org/r1737852 ]
QPID-7149: Active HA broker memory leak
The leak was caused by the sys::Poller. When two pollers run in the same process
and one is idles, PollerHandles accumulate on the idle poller threads and are
never released. The HA broker uses the qpid::messaging API to get initial
status from
other brokers. The messaging API creates a separate Poller from the broker.
The fix is to shut down the qpid::messaging poller as soon as initial status
checks are complete so it does not interfere with the broker's Poller.
> [HA] active HA broker memory leak when ring queue discards overflow messages
>
>
> Key: QPID-7149
> URL: https://issues.apache.org/jira/browse/QPID-7149
> Project: Qpid
> Issue Type: Bug
> Components: C++ Broker
> Environment: RHEL6
> qpid trunk svn rev. 1735384
> - issue seen in very old releases (since active-passive HA cluster initial
> implementation, most probably)
> libstdc++-devel-4.4.7-4.el6.x86_64
> gcc-c++-4.4.7-4.el6.x86_64
> libgcc-4.4.7-4.el6.x86_64
> libstdc++-4.4.7-4.el6.x86_64
> gcc-4.4.7-4.el6.x86_64
>Reporter: Pavel Moravec
>Assignee: Alan Conway
>
> There is a memory leak on active HA broker, triggered most probably by
> purging overflow message from a ring queue. Basic scenario is to setup HA
> cluster, promote to primary and feed forever a ring queue with messages.
> Detailed scenario:
> 1) Start brokers and promote one to primary:
> {noformat}
> start_broker() {
> port=$1
> shift
> rm -rf _${port}
> mkdir _${port}
> nohup qpidd --load-module=ha.so --port=$port
> --log-to-file=qpidd.$port.log --data-dir=_${port} --auth=no
> --log-to-stderr=no --ha-cluster=yes
> --ha-brokers-url="$(hostname):5672,$(hostname):5673,$(hostname):5674"
> --ha-replicate=all --acl-file=/root/qpidd.acl "$@" > /dev/null 2>&1 &
> sleep 1
> }
> killall qpidd qpid-receive 2> /dev/null
> rm -f qpidd.*.log
> start_broker 5672
> sleep 1
> qpid-ha promote -b $(hostname):5672 --cluster-manager
> sleep 1
> start_broker 5673
> sleep 1
> start_broker 5674
> {noformat}
> 2) Create ring queues and send there messages (it is enough to have 1 queue,
> having more should show the leak faster):
> {noformat}
> for i in $(seq 0 9); do
> qpid-config add queue FromKeyServer_$i --max-queue-size=1
> --max-queue-count=10 --limit-policy=ring --argument=x-qpid-priorities=10
> done
> while true; do
> for j in $(seq 1 10); do
> for i in $(seq 1 10); do
> for k in $(seq 0 9); do
> qpid-send -a FromKeyServer_$k -m 100
> --send-rate=50 -- priority=$(($((RANDOM))%10)) &
> done
> done
> wait
> while [ $(qpid-stat -q | grep broker-replicator | sed "s/Y//g"
> | awk '{ print $2 }' | sort -n | tail -n1) != "0" ]; do
> sleep 1
> done
> done
> date
> ps aux | grep qpidd | grep "port=5672" | awk -F "--store-dir" '{ print
> $1 }'
> done
> {noformat}
> (the "while [ $(qpid-stat -q | .." cycle is there just to slow down the
> message enqueues to ensure replication federation queues dont have big
> backlog - that would interfere with memory consumpiton observation)
> 3) Run those scripts and monitor memory consumption.
> - without using priority queues and sending messages without priorities, leak
> is evident as well - sometimes smaller, sometimes the same
> - valgrind (on some older versions I tested before more thoroughly) detects
> nothing (neither leaked memory or reachable at shutdown)
> - same leak is evident even with --ha-replicate=none
> - number of backup brokers does not affect the memory leak
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7149) [HA] active HA broker memory leak when ring queue discards overflow messages
[
https://issues.apache.org/jira/browse/QPID-7149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15220151#comment-15220151
]
Alan Conway commented on QPID-7149:
---
I have a fix under test, see
https://bugzilla.redhat.com/show_bug.cgi?id=1318180. Will commit to trunk as
soon as verified.
> [HA] active HA broker memory leak when ring queue discards overflow messages
>
>
> Key: QPID-7149
> URL: https://issues.apache.org/jira/browse/QPID-7149
> Project: Qpid
> Issue Type: Bug
> Components: C++ Broker
> Environment: RHEL6
> qpid trunk svn rev. 1735384
> - issue seen in very old releases (since active-passive HA cluster initial
> implementation, most probably)
> libstdc++-devel-4.4.7-4.el6.x86_64
> gcc-c++-4.4.7-4.el6.x86_64
> libgcc-4.4.7-4.el6.x86_64
> libstdc++-4.4.7-4.el6.x86_64
> gcc-4.4.7-4.el6.x86_64
>Reporter: Pavel Moravec
>Assignee: Alan Conway
>
> There is a memory leak on active HA broker, triggered most probably by
> purging overflow message from a ring queue. Basic scenario is to setup HA
> cluster, promote to primary and feed forever a ring queue with messages.
> Detailed scenario:
> 1) Start brokers and promote one to primary:
> {noformat}
> start_broker() {
> port=$1
> shift
> rm -rf _${port}
> mkdir _${port}
> nohup qpidd --load-module=ha.so --port=$port
> --log-to-file=qpidd.$port.log --data-dir=_${port} --auth=no
> --log-to-stderr=no --ha-cluster=yes
> --ha-brokers-url="$(hostname):5672,$(hostname):5673,$(hostname):5674"
> --ha-replicate=all --acl-file=/root/qpidd.acl "$@" > /dev/null 2>&1 &
> sleep 1
> }
> killall qpidd qpid-receive 2> /dev/null
> rm -f qpidd.*.log
> start_broker 5672
> sleep 1
> qpid-ha promote -b $(hostname):5672 --cluster-manager
> sleep 1
> start_broker 5673
> sleep 1
> start_broker 5674
> {noformat}
> 2) Create ring queues and send there messages (it is enough to have 1 queue,
> having more should show the leak faster):
> {noformat}
> for i in $(seq 0 9); do
> qpid-config add queue FromKeyServer_$i --max-queue-size=1
> --max-queue-count=10 --limit-policy=ring --argument=x-qpid-priorities=10
> done
> while true; do
> for j in $(seq 1 10); do
> for i in $(seq 1 10); do
> for k in $(seq 0 9); do
> qpid-send -a FromKeyServer_$k -m 100
> --send-rate=50 -- priority=$(($((RANDOM))%10)) &
> done
> done
> wait
> while [ $(qpid-stat -q | grep broker-replicator | sed "s/Y//g"
> | awk '{ print $2 }' | sort -n | tail -n1) != "0" ]; do
> sleep 1
> done
> done
> date
> ps aux | grep qpidd | grep "port=5672" | awk -F "--store-dir" '{ print
> $1 }'
> done
> {noformat}
> (the "while [ $(qpid-stat -q | .." cycle is there just to slow down the
> message enqueues to ensure replication federation queues dont have big
> backlog - that would interfere with memory consumpiton observation)
> 3) Run those scripts and monitor memory consumption.
> - without using priority queues and sending messages without priorities, leak
> is evident as well - sometimes smaller, sometimes the same
> - valgrind (on some older versions I tested before more thoroughly) detects
> nothing (neither leaked memory or reachable at shutdown)
> - same leak is evident even with --ha-replicate=none
> - number of backup brokers does not affect the memory leak
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7149) [HA] active HA broker memory leak when ring queue discards overflow messages
[
https://issues.apache.org/jira/browse/QPID-7149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15215607#comment-15215607
]
Pavel Moravec commented on QPID-7149:
-
It allows everything:
acl allow all all
I explicitly set that to overwrite the default used ACL file with a rule
disabling federation links (what would _prevent_ the mem.leak).
> [HA] active HA broker memory leak when ring queue discards overflow messages
>
>
> Key: QPID-7149
> URL: https://issues.apache.org/jira/browse/QPID-7149
> Project: Qpid
> Issue Type: Bug
> Components: C++ Broker
> Environment: RHEL6
> qpid trunk svn rev. 1735384
> - issue seen in very old releases (since active-passive HA cluster initial
> implementation, most probably)
> libstdc++-devel-4.4.7-4.el6.x86_64
> gcc-c++-4.4.7-4.el6.x86_64
> libgcc-4.4.7-4.el6.x86_64
> libstdc++-4.4.7-4.el6.x86_64
> gcc-4.4.7-4.el6.x86_64
>Reporter: Pavel Moravec
>Assignee: Alan Conway
>
> There is a memory leak on active HA broker, triggered most probably by
> purging overflow message from a ring queue. Basic scenario is to setup HA
> cluster, promote to primary and feed forever a ring queue with messages.
> Detailed scenario:
> 1) Start brokers and promote one to primary:
> {noformat}
> start_broker() {
> port=$1
> shift
> rm -rf _${port}
> mkdir _${port}
> nohup qpidd --load-module=ha.so --port=$port
> --log-to-file=qpidd.$port.log --data-dir=_${port} --auth=no
> --log-to-stderr=no --ha-cluster=yes
> --ha-brokers-url="$(hostname):5672,$(hostname):5673,$(hostname):5674"
> --ha-replicate=all --acl-file=/root/qpidd.acl "$@" > /dev/null 2>&1 &
> sleep 1
> }
> killall qpidd qpid-receive 2> /dev/null
> rm -f qpidd.*.log
> start_broker 5672
> sleep 1
> qpid-ha promote -b $(hostname):5672 --cluster-manager
> sleep 1
> start_broker 5673
> sleep 1
> start_broker 5674
> {noformat}
> 2) Create ring queues and send there messages (it is enough to have 1 queue,
> having more should show the leak faster):
> {noformat}
> for i in $(seq 0 9); do
> qpid-config add queue FromKeyServer_$i --max-queue-size=1
> --max-queue-count=10 --limit-policy=ring --argument=x-qpid-priorities=10
> done
> while true; do
> for j in $(seq 1 10); do
> for i in $(seq 1 10); do
> for k in $(seq 0 9); do
> qpid-send -a FromKeyServer_$k -m 100
> --send-rate=50 -- priority=$(($((RANDOM))%10)) &
> done
> done
> wait
> while [ $(qpid-stat -q | grep broker-replicator | sed "s/Y//g"
> | awk '{ print $2 }' | sort -n | tail -n1) != "0" ]; do
> sleep 1
> done
> done
> date
> ps aux | grep qpidd | grep "port=5672" | awk -F "--store-dir" '{ print
> $1 }'
> done
> {noformat}
> (the "while [ $(qpid-stat -q | .." cycle is there just to slow down the
> message enqueues to ensure replication federation queues dont have big
> backlog - that would interfere with memory consumpiton observation)
> 3) Run those scripts and monitor memory consumption.
> - without using priority queues and sending messages without priorities, leak
> is evident as well - sometimes smaller, sometimes the same
> - valgrind (on some older versions I tested before more thoroughly) detects
> nothing (neither leaked memory or reachable at shutdown)
> - same leak is evident even with --ha-replicate=none
> - number of backup brokers does not affect the memory leak
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7149) [HA] active HA broker memory leak when ring queue discards overflow messages
[
https://issues.apache.org/jira/browse/QPID-7149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211737#comment-15211737
]
Alan Conway commented on QPID-7149:
---
I just noticed your reproducer has --acl-file=/root/qpidd.acl. Is that
necessary? If so can you attach the contents of the ACL file.
> [HA] active HA broker memory leak when ring queue discards overflow messages
>
>
> Key: QPID-7149
> URL: https://issues.apache.org/jira/browse/QPID-7149
> Project: Qpid
> Issue Type: Bug
> Components: C++ Broker
> Environment: RHEL6
> qpid trunk svn rev. 1735384
> - issue seen in very old releases (since active-passive HA cluster initial
> implementation, most probably)
> libstdc++-devel-4.4.7-4.el6.x86_64
> gcc-c++-4.4.7-4.el6.x86_64
> libgcc-4.4.7-4.el6.x86_64
> libstdc++-4.4.7-4.el6.x86_64
> gcc-4.4.7-4.el6.x86_64
>Reporter: Pavel Moravec
>Assignee: Alan Conway
>
> There is a memory leak on active HA broker, triggered most probably by
> purging overflow message from a ring queue. Basic scenario is to setup HA
> cluster, promote to primary and feed forever a ring queue with messages.
> Detailed scenario:
> 1) Start brokers and promote one to primary:
> {noformat}
> start_broker() {
> port=$1
> shift
> rm -rf _${port}
> mkdir _${port}
> nohup qpidd --load-module=ha.so --port=$port
> --log-to-file=qpidd.$port.log --data-dir=_${port} --auth=no
> --log-to-stderr=no --ha-cluster=yes
> --ha-brokers-url="$(hostname):5672,$(hostname):5673,$(hostname):5674"
> --ha-replicate=all --acl-file=/root/qpidd.acl "$@" > /dev/null 2>&1 &
> sleep 1
> }
> killall qpidd qpid-receive 2> /dev/null
> rm -f qpidd.*.log
> start_broker 5672
> sleep 1
> qpid-ha promote -b $(hostname):5672 --cluster-manager
> sleep 1
> start_broker 5673
> sleep 1
> start_broker 5674
> {noformat}
> 2) Create ring queues and send there messages (it is enough to have 1 queue,
> having more should show the leak faster):
> {noformat}
> for i in $(seq 0 9); do
> qpid-config add queue FromKeyServer_$i --max-queue-size=1
> --max-queue-count=10 --limit-policy=ring --argument=x-qpid-priorities=10
> done
> while true; do
> for j in $(seq 1 10); do
> for i in $(seq 1 10); do
> for k in $(seq 0 9); do
> qpid-send -a FromKeyServer_$k -m 100
> --send-rate=50 -- priority=$(($((RANDOM))%10)) &
> done
> done
> wait
> while [ $(qpid-stat -q | grep broker-replicator | sed "s/Y//g"
> | awk '{ print $2 }' | sort -n | tail -n1) != "0" ]; do
> sleep 1
> done
> done
> date
> ps aux | grep qpidd | grep "port=5672" | awk -F "--store-dir" '{ print
> $1 }'
> done
> {noformat}
> (the "while [ $(qpid-stat -q | .." cycle is there just to slow down the
> message enqueues to ensure replication federation queues dont have big
> backlog - that would interfere with memory consumpiton observation)
> 3) Run those scripts and monitor memory consumption.
> - without using priority queues and sending messages without priorities, leak
> is evident as well - sometimes smaller, sometimes the same
> - valgrind (on some older versions I tested before more thoroughly) detects
> nothing (neither leaked memory or reachable at shutdown)
> - same leak is evident even with --ha-replicate=none
> - number of backup brokers does not affect the memory leak
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7149) [HA] active HA broker memory leak when ring queue discards overflow messages
[
https://issues.apache.org/jira/browse/QPID-7149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15204363#comment-15204363
]
Alan Conway commented on QPID-7149:
---
That narrows it down considerably to the code that runs on the primary even
when no backups are present. The primary does do some tracking of queue state
even when there are no backups, in order to be ready if a backup does connect.
That is probably where the trouble lies.
> [HA] active HA broker memory leak when ring queue discards overflow messages
>
>
> Key: QPID-7149
> URL: https://issues.apache.org/jira/browse/QPID-7149
> Project: Qpid
> Issue Type: Bug
> Components: C++ Broker
> Environment: RHEL6
> qpid trunk svn rev. 1735384
> - issue seen in very old releases (since active-passive HA cluster initial
> implementation, most probably)
> libstdc++-devel-4.4.7-4.el6.x86_64
> gcc-c++-4.4.7-4.el6.x86_64
> libgcc-4.4.7-4.el6.x86_64
> libstdc++-4.4.7-4.el6.x86_64
> gcc-4.4.7-4.el6.x86_64
>Reporter: Pavel Moravec
>
> There is a memory leak on active HA broker, triggered most probably by
> purging overflow message from a ring queue. Basic scenario is to setup HA
> cluster, promote to primary and feed forever a ring queue with messages.
> Detailed scenario:
> 1) Start brokers and promote one to primary:
> {noformat}
> start_broker() {
> port=$1
> shift
> rm -rf _${port}
> mkdir _${port}
> nohup qpidd --load-module=ha.so --port=$port
> --log-to-file=qpidd.$port.log --data-dir=_${port} --auth=no
> --log-to-stderr=no --ha-cluster=yes
> --ha-brokers-url="$(hostname):5672,$(hostname):5673,$(hostname):5674"
> --ha-replicate=all --acl-file=/root/qpidd.acl "$@" > /dev/null 2>&1 &
> sleep 1
> }
> killall qpidd qpid-receive 2> /dev/null
> rm -f qpidd.*.log
> start_broker 5672
> sleep 1
> qpid-ha promote -b $(hostname):5672 --cluster-manager
> sleep 1
> start_broker 5673
> sleep 1
> start_broker 5674
> {noformat}
> 2) Create ring queues and send there messages (it is enough to have 1 queue,
> having more should show the leak faster):
> {noformat}
> for i in $(seq 0 9); do
> qpid-config add queue FromKeyServer_$i --max-queue-size=1
> --max-queue-count=10 --limit-policy=ring --argument=x-qpid-priorities=10
> done
> while true; do
> for j in $(seq 1 10); do
> for i in $(seq 1 10); do
> for k in $(seq 0 9); do
> qpid-send -a FromKeyServer_$k -m 100
> --send-rate=50 -- priority=$(($((RANDOM))%10)) &
> done
> done
> wait
> while [ $(qpid-stat -q | grep broker-replicator | sed "s/Y//g"
> | awk '{ print $2 }' | sort -n | tail -n1) != "0" ]; do
> sleep 1
> done
> done
> date
> ps aux | grep qpidd | grep "port=5672" | awk -F "--store-dir" '{ print
> $1 }'
> done
> {noformat}
> (the "while [ $(qpid-stat -q | .." cycle is there just to slow down the
> message enqueues to ensure replication federation queues dont have big
> backlog - that would interfere with memory consumpiton observation)
> 3) Run those scripts and monitor memory consumption.
> - without using priority queues and sending messages without priorities, leak
> is evident as well - sometimes smaller, sometimes the same
> - valgrind (on some older versions I tested before more thoroughly) detects
> nothing (neither leaked memory or reachable at shutdown)
> - same leak is evident even with --ha-replicate=none
> - number of backup brokers does not affect the memory leak
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7149) [HA] active HA broker memory leak when ring queue discards overflow messages
[
https://issues.apache.org/jira/browse/QPID-7149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15200045#comment-15200045
]
Pavel Moravec commented on QPID-7149:
-
The leak is present even if
{noformat}
--ha-cluster=yes
--ha-brokers-url="$(hostname):5672,$(hostname):5673,$(hostname):5674"
--ha-replicate=none
{noformat}
is used in the reproducer and _no_ backup broker is running.
Removing the above options causes no leak is present.
> [HA] active HA broker memory leak when ring queue discards overflow messages
>
>
> Key: QPID-7149
> URL: https://issues.apache.org/jira/browse/QPID-7149
> Project: Qpid
> Issue Type: Bug
> Components: C++ Broker
> Environment: RHEL6
> qpid trunk svn rev. 1735384
> - issue seen in very old releases (since active-passive HA cluster initial
> implementation, most probably)
> libstdc++-devel-4.4.7-4.el6.x86_64
> gcc-c++-4.4.7-4.el6.x86_64
> libgcc-4.4.7-4.el6.x86_64
> libstdc++-4.4.7-4.el6.x86_64
> gcc-4.4.7-4.el6.x86_64
>Reporter: Pavel Moravec
>
> There is a memory leak on active HA broker, triggered most probably by
> purging overflow message from a ring queue. Basic scenario is to setup HA
> cluster, promote to primary and feed forever a ring queue with messages.
> Detailed scenario:
> 1) Start brokers and promote one to primary:
> {noformat}
> start_broker() {
> port=$1
> shift
> rm -rf _${port}
> mkdir _${port}
> nohup qpidd --load-module=ha.so --port=$port
> --log-to-file=qpidd.$port.log --data-dir=_${port} --auth=no
> --log-to-stderr=no --ha-cluster=yes
> --ha-brokers-url="$(hostname):5672,$(hostname):5673,$(hostname):5674"
> --ha-replicate=all --acl-file=/root/qpidd.acl "$@" > /dev/null 2>&1 &
> sleep 1
> }
> killall qpidd qpid-receive 2> /dev/null
> rm -f qpidd.*.log
> start_broker 5672
> sleep 1
> qpid-ha promote -b $(hostname):5672 --cluster-manager
> sleep 1
> start_broker 5673
> sleep 1
> start_broker 5674
> {noformat}
> 2) Create ring queues and send there messages (it is enough to have 1 queue,
> having more should show the leak faster):
> {noformat}
> for i in $(seq 0 9); do
> qpid-config add queue FromKeyServer_$i --max-queue-size=1
> --max-queue-count=10 --limit-policy=ring --argument=x-qpid-priorities=10
> done
> while true; do
> for j in $(seq 1 10); do
> for i in $(seq 1 10); do
> for k in $(seq 0 9); do
> qpid-send -a FromKeyServer_$k -m 100
> --send-rate=50 -- priority=$(($((RANDOM))%10)) &
> done
> done
> wait
> while [ $(qpid-stat -q | grep broker-replicator | sed "s/Y//g"
> | awk '{ print $2 }' | sort -n | tail -n1) != "0" ]; do
> sleep 1
> done
> done
> date
> ps aux | grep qpidd | grep "port=5672" | awk -F "--store-dir" '{ print
> $1 }'
> done
> {noformat}
> (the "while [ $(qpid-stat -q | .." cycle is there just to slow down the
> message enqueues to ensure replication federation queues dont have big
> backlog - that would interfere with memory consumpiton observation)
> 3) Run those scripts and monitor memory consumption.
> - without using priority queues and sending messages without priorities, leak
> is evident as well - sometimes smaller, sometimes the same
> - valgrind (on some older versions I tested before more thoroughly) detects
> nothing (neither leaked memory or reachable at shutdown)
> - same leak is evident even with --ha-replicate=none
> - number of backup brokers does not affect the memory leak
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
