RE: FW: problems starting the version 18 broker and its Crash
You are welcome.
Nitin
-Original Message-
From: Alan Conway [mailto:acon...@redhat.com]
Sent: Monday, September 10, 2012 10:38 AM
To: dev@qpid.apache.org
Cc: Gordon Sim; Nitin Shah
Subject: RE: FW: problems starting the version 18 broker and its Crash
Thanks a lot for tracking this down. For some reason it happens on some
machines but not others. I'll put up a JIRA and a patch and fix this on trunk.
On Fri, 2012-09-07 at 09:43 -0400, Nitin Shah wrote:
> Hi,
>
> Thanks for the information. I started doing some investigation on the new
> release mainly because I could not see what we were ( if possible) doing
> wrong with the release. The broker would start executing and immediately one
> was getting an assert as shown below in the output I generated with running
> it under GDB. It asserts because it fails the test in file types.cpp in
> qpid/ha line 38 ( assert(value < count). I noticed that this is happening as
> a result of the call from the HaBroker::initialize() function line 90 in the
> HaBroker.cpp file where a QPID_LOG is being invoked.
>
> I believe the root of the problem is the BrokerInfo class constructor
> is not initializing the private class data called "BrokerStatus
> status" which is defined in file BrokerInfo.h . BrokerStatus is
> defined in types.h as an enum as follows
>
> enum BrokerStatus {
> JOINING,///< New broker, looking for primary
> CATCHUP,///< Backup: Connected to primary, catching
> up on state.
> READY, ///< Backup: Caught up, ready to take over.
> RECOVERING, ///< Primary: waiting for backups to connect
> & sync
> ACTIVE, ///< Primary: actively serving clients.
> STANDALONE ///< Not part of a cluster.
> };
>
> It seems like the assert happens on the second call to EnumBase::str() in
> types.cpp. The count was 6 and the value was some large uninitialized value.
>
> I initialized the status variable in the constructor to STANDALONE and the
> broker came up and worked fine.
>
> *I assume we are going to need a patch or an update for the broker to be
> used.
>
> NOTE:: If I start the broker with the "--log-enable warning" or for that
> matter others like notice, the broker comes up and works fine. But, if the
> broker is started with no log-enable parameters, IT CRASHES. This seems
> pretty strange as I thought you guys would have seen this.
>
> I am running the broker on a CentOs 6.2 system. We have been using version 16
> to date. The images are created a new and loaded on a VM, so the VM has no
> remnants of the version 16, so it is clean version 18 load and use.
>
> If I wish to continue using this release, can someone tell me what the
> correct process for getting a patch is? Alternatively, what is the patch I
> should apply, meaning can I add a initialize of the status parameter and what
> is the CORRECT enum value to set it to?
>
> Nitin
>
> Ps let me know if there any other information you need
>
> Thanks
>
>
>
>
>
> (gdb) where
> #0 0x758b58a5 in raise () from /lib64/libc.so.6
> #1 0x758b7085 in abort () from /lib64/libc.so.6
> #2 0x758aea1e in __assert_fail_base () from /lib64/libc.so.6
> #3 0x758aeae0 in __assert_fail () from /lib64/libc.so.6
> #4 0x750464fb in qpid::ha::EnumBase::str (this= optimized out>) at qpid/ha/types.cpp:38
> #5 0x75046533 in qpid::ha::operator<< (o=..., e=...) at
> qpid/ha/types.cpp:64
> #6 0x75013a5a in qpid::ha::operator<< (o=,
> b=)
> at qpid/ha/BrokerInfo.cpp:99
> #7 0x75029fb4 in operator<<
> (this=0x6507b0) at ../include/qpid/Msg.h:63
> #8 qpid::ha::HaBroker::initialize (this=0x6507b0) at
> qpid/ha/HaBroker.cpp:90
> #9 0x775d4861 in operator() (f=)
> at /usr/include/boost/bind/mem_fn_template.hpp:162
> #10 operator() qpid::Plugin::Target&>, boost::_bi::list1 >
> (f=) at /usr/include/boost/bind/bind.hpp:306
> #11 operator() (f=) at
> /usr/include/boost/bind/bind_template.hpp:47
> #12 for_each<__gnu_cxx::__normal_iterator std::vector > >,
> boost::_bi::bind_t qpid::Plugin::Target&>, boost::_bi::list2,
> boost::reference_wrapper > > > (f=)
> at
> /usr/lib/gcc/x86_64-redhat-linux/4.4.6/../../../../include/c++/4.4.6/b
> its/stl_algo.h:4200
> #13 qpid::(anonymous namespace)::each_plugin boost::_mfi::mf1,
> boost::_bi::list2,
> boost::reference_wrapper > > > (
> f=) at qpid/Plugin.cpp:73
> #14 0x775d48a2 in qpid::Plugin::initializeAll (t= optimi
RE: FW: problems starting the version 18 broker and its Crash
Thanks a lot for tracking this down. For some reason it happens on some
machines but not others. I'll put up a JIRA and a patch and fix this on
trunk.
On Fri, 2012-09-07 at 09:43 -0400, Nitin Shah wrote:
> Hi,
>
> Thanks for the information. I started doing some investigation on the new
> release mainly because I could not see what we were ( if possible) doing
> wrong with the release. The broker would start executing and immediately one
> was getting an assert as shown below in the output I generated with running
> it under GDB. It asserts because it fails the test in file types.cpp in
> qpid/ha line 38 ( assert(value < count). I noticed that this is happening as
> a result of the call from the HaBroker::initialize() function line 90 in the
> HaBroker.cpp file where a QPID_LOG is being invoked.
>
> I believe the root of the problem is the BrokerInfo class constructor is not
> initializing the private class data called "BrokerStatus status" which is
> defined in file BrokerInfo.h . BrokerStatus is defined in types.h as an enum
> as follows
>
> enum BrokerStatus {
> JOINING,///< New broker, looking for primary
> CATCHUP,///< Backup: Connected to primary, catching
> up on state.
> READY, ///< Backup: Caught up, ready to take over.
> RECOVERING, ///< Primary: waiting for backups to connect
> & sync
> ACTIVE, ///< Primary: actively serving clients.
> STANDALONE ///< Not part of a cluster.
> };
>
> It seems like the assert happens on the second call to EnumBase::str() in
> types.cpp. The count was 6 and the value was some large uninitialized value.
>
> I initialized the status variable in the constructor to STANDALONE and the
> broker came up and worked fine.
>
> *I assume we are going to need a patch or an update for the broker to be
> used.
>
> NOTE:: If I start the broker with the "--log-enable warning" or for that
> matter others like notice, the broker comes up and works fine. But, if the
> broker is started with no log-enable parameters, IT CRASHES. This seems
> pretty strange as I thought you guys would have seen this.
>
> I am running the broker on a CentOs 6.2 system. We have been using version 16
> to date. The images are created a new and loaded on a VM, so the VM has no
> remnants of the version 16, so it is clean version 18 load and use.
>
> If I wish to continue using this release, can someone tell me what the
> correct process for getting a patch is? Alternatively, what is the patch I
> should apply, meaning can I add a initialize of the status parameter and what
> is the CORRECT enum value to set it to?
>
> Nitin
>
> Ps let me know if there any other information you need
>
> Thanks
>
>
>
>
>
> (gdb) where
> #0 0x758b58a5 in raise () from /lib64/libc.so.6
> #1 0x758b7085 in abort () from /lib64/libc.so.6
> #2 0x758aea1e in __assert_fail_base () from /lib64/libc.so.6
> #3 0x758aeae0 in __assert_fail () from /lib64/libc.so.6
> #4 0x750464fb in qpid::ha::EnumBase::str (this= out>) at qpid/ha/types.cpp:38
> #5 0x75046533 in qpid::ha::operator<< (o=..., e=...) at
> qpid/ha/types.cpp:64
> #6 0x75013a5a in qpid::ha::operator<< (o=,
> b=)
> at qpid/ha/BrokerInfo.cpp:99
> #7 0x75029fb4 in operator<< (this=0x6507b0)
> at ../include/qpid/Msg.h:63
> #8 qpid::ha::HaBroker::initialize (this=0x6507b0) at qpid/ha/HaBroker.cpp:90
> #9 0x775d4861 in operator() (f=)
> at /usr/include/boost/bind/mem_fn_template.hpp:162
> #10 operator(),
> boost::_bi::list1 > (f=) at
> /usr/include/boost/bind/bind.hpp:306
> #11 operator() (f=) at
> /usr/include/boost/bind/bind_template.hpp:47
> #12 for_each<__gnu_cxx::__normal_iterator std::vector > >,
> boost::_bi::bind_t qpid::Plugin::Target&>, boost::_bi::list2,
> boost::reference_wrapper > > > (f=)
> at
> /usr/lib/gcc/x86_64-redhat-linux/4.4.6/../../../../include/c++/4.4.6/bits/stl_algo.h:4200
> #13 qpid::(anonymous namespace)::each_plugin boost::_mfi::mf1,
> boost::_bi::list2,
> boost::reference_wrapper > > > (
> f=) at qpid/Plugin.cpp:73
> #14 0x775d48a2 in qpid::Plugin::initializeAll (t= out>) at qpid/Plugin.cpp:91
>
>
>
>
>
> -Original Message-
> From: Gordon Sim [mailto:g...@redhat.com]
> Sent: Wednesday, September 05, 2012 12:29 PM
> To: dev@qpid.apache.org
> Subject: Re: FW: problems starting the version 18 broker
>
> Can you run it with full tracing on? i.e. --log-enable trace+
>
> What platform are you running on? Did make check pass when building? Did you
> build and install from source on that machine; did you uninstall the
> 0.16 release first or are they both installed in different locations?
>
> The second error appears to be coming from the HA module, possibly related to
> the options. Could you be picking up the 0.16 modules when starting the 0.18
>
RE: FW: problems starting the version 18 broker and its Crash
Gordon, Thanks for a quick turnaround of the patch. We will try it and see. Thanks again Nitin -Original Message- From: Gordon Sim [mailto:g...@redhat.com] Sent: Friday, September 07, 2012 10:21 AM To: dev@qpid.apache.org Subject: Re: FW: problems starting the version 18 broker and its Crash On 09/07/2012 02:43 PM, Nitin Shah wrote: > Hi, > > Thanks for the information. I started doing some investigation on the > new release mainly because I could not see what we were ( if > possible) doing wrong with the release. The broker would start > executing and immediately one was getting an assert as shown below in > the output I generated with running it under GDB. It asserts because > it fails the test in file types.cpp in qpid/ha line 38 ( assert(value > < count). I noticed that this is happening as a result of the call > from the HaBroker::initialize() function line 90 in the HaBroker.cpp > file where a QPID_LOG is being invoked. > > I believe the root of the problem is the BrokerInfo class constructor > is not initializing the private class data called "BrokerStatus > status" which is defined in file BrokerInfo.h . Good analysis; thanks! > *I assume we are going to need a patch or an update for the broker > to be used. > > NOTE:: If I start the broker with the "--log-enable warning" or for > that matter others like notice, the broker comes up and works fine. > But, if the broker is started with no log-enable parameters, IT > CRASHES. This seems pretty strange as I thought you guys would have > seen this. It actually works fine for me for repeated restarts, though if I run under valgrind the uninitialised value is reported and of course by inspection of the code you have correctly identified a problem. I guess we have just been unlucky in not hitting the problem... > I am running the broker on a CentOs 6.2 system. We have been using > version 16 to date. The images are created a new and loaded on a VM, > so the VM has no remnants of the version 16, so it is clean version > 18 load and use. > > If I wish to continue using this release, can someone tell me what the > correct process for getting a patch is? Alternatively, what is the > patch I should apply, meaning can I add a initialize of the status > parameter and what is the CORRECT enum value to set it to? The attached patch should take care of the crash. I believe the status is correctly set later on, just not in time for the logging statement. - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
Re: FW: problems starting the version 18 broker and its Crash
On 09/07/2012 02:43 PM, Nitin Shah wrote:
Hi,
Thanks for the information. I started doing some investigation on the
new release mainly because I could not see what we were ( if
possible) doing wrong with the release. The broker would start
executing and immediately one was getting an assert as shown below in
the output I generated with running it under GDB. It asserts because
it fails the test in file types.cpp in qpid/ha line 38 ( assert(value
< count). I noticed that this is happening as a result of the call
from the HaBroker::initialize() function line 90 in the HaBroker.cpp
file where a QPID_LOG is being invoked.
I believe the root of the problem is the BrokerInfo class constructor
is not initializing the private class data called "BrokerStatus
status" which is defined in file BrokerInfo.h .
Good analysis; thanks!
*I assume we are going to need a patch or an update for the
broker to be used.
NOTE:: If I start the broker with the "--log-enable warning" or for
that matter others like notice, the broker comes up and works fine.
But, if the broker is started with no log-enable parameters, IT
CRASHES. This seems pretty strange as I thought you guys would have
seen this.
It actually works fine for me for repeated restarts, though if I run
under valgrind the uninitialised value is reported and of course by
inspection of the code you have correctly identified a problem.
I guess we have just been unlucky in not hitting the problem...
I am running the broker on a CentOs 6.2 system. We have been using
version 16 to date. The images are created a new and loaded on a VM,
so the VM has no remnants of the version 16, so it is clean version
18 load and use.
If I wish to continue using this release, can someone tell me what
the correct process for getting a patch is? Alternatively, what is
the patch I should apply, meaning can I add a initialize of the
status parameter and what is the CORRECT enum value to set it to?
The attached patch should take care of the crash. I believe the status
is correctly set later on, just not in time for the logging statement.
Index: cpp/src/qpid/ha/BrokerInfo.cpp
===
--- cpp/src/qpid/ha/BrokerInfo.cpp (revision 1382034)
+++ cpp/src/qpid/ha/BrokerInfo.cpp (working copy)
@@ -44,7 +44,7 @@
using framing::FieldTable;
BrokerInfo::BrokerInfo(const std::string& host, uint16_t port_, const types::Uuid& id) :
-hostName(host), port(port_), systemId(id)
+hostName(host), port(port_), systemId(id), status(JOINING)
{
updateLogId();
}
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
RE: FW: problems starting the version 18 broker and its Crash
Hi,
Thanks for the information. I started doing some investigation on the new
release mainly because I could not see what we were ( if possible) doing wrong
with the release. The broker would start executing and immediately one was
getting an assert as shown below in the output I generated with running it
under GDB. It asserts because it fails the test in file types.cpp in qpid/ha
line 38 ( assert(value < count). I noticed that this is happening as a result
of the call from the HaBroker::initialize() function line 90 in the
HaBroker.cpp file where a QPID_LOG is being invoked.
I believe the root of the problem is the BrokerInfo class constructor is not
initializing the private class data called "BrokerStatus status" which is
defined in file BrokerInfo.h . BrokerStatus is defined in types.h as an enum as
follows
enum BrokerStatus {
JOINING,///< New broker, looking for primary
CATCHUP,///< Backup: Connected to primary, catching up
on state.
READY, ///< Backup: Caught up, ready to take over.
RECOVERING, ///< Primary: waiting for backups to connect &
sync
ACTIVE, ///< Primary: actively serving clients.
STANDALONE ///< Not part of a cluster.
};
It seems like the assert happens on the second call to EnumBase::str() in
types.cpp. The count was 6 and the value was some large uninitialized value.
I initialized the status variable in the constructor to STANDALONE and the
broker came up and worked fine.
*I assume we are going to need a patch or an update for the broker to be
used.
NOTE:: If I start the broker with the "--log-enable warning" or for that matter
others like notice, the broker comes up and works fine. But, if the broker is
started with no log-enable parameters, IT CRASHES. This seems pretty strange as
I thought you guys would have seen this.
I am running the broker on a CentOs 6.2 system. We have been using version 16
to date. The images are created a new and loaded on a VM, so the VM has no
remnants of the version 16, so it is clean version 18 load and use.
If I wish to continue using this release, can someone tell me what the correct
process for getting a patch is? Alternatively, what is the patch I should
apply, meaning can I add a initialize of the status parameter and what is the
CORRECT enum value to set it to?
Nitin
Ps let me know if there any other information you need
Thanks
(gdb) where
#0 0x758b58a5 in raise () from /lib64/libc.so.6
#1 0x758b7085 in abort () from /lib64/libc.so.6
#2 0x758aea1e in __assert_fail_base () from /lib64/libc.so.6
#3 0x758aeae0 in __assert_fail () from /lib64/libc.so.6
#4 0x750464fb in qpid::ha::EnumBase::str (this=)
at qpid/ha/types.cpp:38
#5 0x75046533 in qpid::ha::operator<< (o=..., e=...) at
qpid/ha/types.cpp:64
#6 0x75013a5a in qpid::ha::operator<< (o=,
b=)
at qpid/ha/BrokerInfo.cpp:99
#7 0x75029fb4 in operator<< (this=0x6507b0) at
../include/qpid/Msg.h:63
#8 qpid::ha::HaBroker::initialize (this=0x6507b0) at qpid/ha/HaBroker.cpp:90
#9 0x775d4861 in operator() (f=)
at /usr/include/boost/bind/mem_fn_template.hpp:162
#10 operator(),
boost::_bi::list1 > (f=) at
/usr/include/boost/bind/bind.hpp:306
#11 operator() (f=) at
/usr/include/boost/bind/bind_template.hpp:47
#12 for_each<__gnu_cxx::__normal_iterator > >,
boost::_bi::bind_t, boost::_bi::list2,
boost::reference_wrapper > > > (f=)
at
/usr/lib/gcc/x86_64-redhat-linux/4.4.6/../../../../include/c++/4.4.6/bits/stl_algo.h:4200
#13 qpid::(anonymous namespace)::each_plugin,
boost::_bi::list2, boost::reference_wrapper
> > > (
f=) at qpid/Plugin.cpp:73
#14 0x775d48a2 in qpid::Plugin::initializeAll (t=)
at qpid/Plugin.cpp:91
-Original Message-
From: Gordon Sim [mailto:g...@redhat.com]
Sent: Wednesday, September 05, 2012 12:29 PM
To: dev@qpid.apache.org
Subject: Re: FW: problems starting the version 18 broker
Can you run it with full tracing on? i.e. --log-enable trace+
What platform are you running on? Did make check pass when building? Did you
build and install from source on that machine; did you uninstall the
0.16 release first or are they both installed in different locations?
The second error appears to be coming from the HA module, possibly related to
the options. Could you be picking up the 0.16 modules when starting the 0.18
broker? (Trying with --module-dir pointing at a 0.18 modules only directory
would verify that).
On 09/05/2012 04:48 PM, Nitin Shah wrote:
> Ps if I run it in foreground then I get the following message
>
> 2012-09-05 11:39:45 [Broker] notice SASL disabled: No Authentication
> Performed
>
> 2012-09-05 11:39:45 [Network] notice Listening on TCP/TCP6 port 5672
>
> qpidd: qpid/ha/types.cpp:38: std::string qpid::ha::EnumBase::str()
> const: Assertion `value < count' failed.
>
> Aborte
