[
https://issues.apache.org/jira/browse/DISPATCH-895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16281302#comment-16281302
]
Saurabh Badhwar commented on DISPATCH-895:
------------------------------------------
On running qpid-dispatch under gdb with the coredump attached for analysis, it
seems like the crash happens in pni_add_tpwork when the following instrcution:
pn_connection_t *connection = delivery->link->session->connection;
is executed.
This probably seems like, either the link has been freed and is then being
referred to.
gdb -c /root/bz1519140-qdrouterd-segfault-coredump/coredump $(which qdrouterd)
..
Program terminated with signal 11, Segmentation fault.
#0 pni_add_tpwork (delivery=0x7f0d6c80ef00) at
/usr/src/debug/qpid-proton-0.16.0/proton-c/src/core/engine.c:727
727 pn_connection_t *connection = delivery->link->session->connection;
(gdb) bt
#0 pni_add_tpwork (delivery=0x7f0d6c80ef00) at
/usr/src/debug/qpid-proton-0.16.0/proton-c/src/core/engine.c:727
#1 0x00007f0d8bde1cae in pn_delivery_update (delivery=<optimized out>,
state=<optimized out>) at
/usr/src/debug/qpid-proton-0.16.0/proton-c/src/core/engine.c:2026
#2 0x00007f0d8c051cc8 in CORE_delivery_update (context=0x234e970,
dlv=0x7f0d7005c1d0, disp=39, settled=<optimized out>) at
/usr/src/debug/qpid-dispatch-0.8.0/src/router_node.c:1068
#3 0x00007f0d8c046c5d in qdr_connection_process (conn=0x7f0d6880f210) at
/usr/src/debug/qpid-dispatch-0.8.0/src/router_core/connections.c:306
#4 0x00007f0d8c0321d8 in writable_handler (container=0x2087740,
container=0x2087740, conn=<optimized out>, qd_conn=0x269c290) at
/usr/src/debug/qpid-dispatch-0.8.0/src/container.c:307
#5 handler (handler_context=0x2087740, conn_context=0x2197f40,
event=<optimized out>, qd_conn=0x269c290) at
/usr/src/debug/qpid-dispatch-0.8.0/src/container.c:614
#6 0x00007f0d8c056ccb in process_connector (cxtr=<optimized out>,
qd_server=0x230e4f0) at /usr/src/debug/qpid-dispatch-0.8.0/src/server.c:792
#7 thread_run (arg=<optimized out>) at
/usr/src/debug/qpid-dispatch-0.8.0/src/server.c:1012
#8 0x00007f0d8c0574e6 in qd_server_run (qd=0x1d28030) at
/usr/src/debug/qpid-dispatch-0.8.0/src/server.c:1421
#9 0x0000000000401bd7 in main_process
(config_path=config_path@entry=0x7ffed8973f2d
"/etc/qpid-dispatch/qdrouterd.conf",
python_pkgdir=python_pkgdir@entry=0x402321 "/usr/lib/qpid-dispatch/python",
fd=fd@entry=2) at /usr/src/debug/qpid-dispatch-0.8.0/router/src/main.c:147
#10 0x0000000000401844 in main (argc=3, argv=0x7ffed8972c68) at
/usr/src/debug/qpid-dispatch-0.8.0/router/src/main.c:353
> qpid-dispatch crashes with a SEGFAULT in libqpid-proton
> -------------------------------------------------------
>
> Key: DISPATCH-895
> URL: https://issues.apache.org/jira/browse/DISPATCH-895
> Project: Qpid Dispatch
> Issue Type: Bug
> Components: Routing Engine
> Affects Versions: 0.8.0
> Environment: OS: Red Hat Enterprise Linux 7.4
> Arch: x86_64
> Installed Packages
> Name : qpid-cpp-client
> Arch : x86_64
> Version : 1.36.0
> Release : 9.el7
> Size : 4.4 M
> Repo : installed
> From repo : rhel-server-7-satellite-6-beta-rpms
> Summary : Libraries for Qpid C++ client applications
> URL : http://qpid.apache.org
> License : ASL 2.0
> Description : Run-time libraries for AMQP client applications developed using
> Qpid
> : C++. Clients exchange messages with an AMQP message broker using
> : the AMQP protocol.
> Name : qpid-cpp-server
> Arch : x86_64
> Version : 1.36.0
> Release : 9.el7
> Size : 5.2 M
> Repo : installed
> From repo : rhel-server-7-satellite-6-beta-rpms
> Summary : An AMQP message broker daemon
> URL : http://qpid.apache.org
> License : ASL 2.0
> Description : A message broker daemon that receives stores and routes
> messages using
> : the open AMQP messaging protocol.
> Name : qpid-cpp-server-linearstore
> Arch : x86_64
> Version : 1.36.0
> Release : 9.el7
> Size : 797 k
> Repo : installed
> From repo : rhel-server-7-satellite-6-beta-rpms
> Summary : Red Hat persistence extension to the Qpid messaging sytem
> URL : http://qpid.apache.org
> License : ASL 2.0
> Description : Red Hat persistence extension to the Qpid AMQP broker:
> persistent message
> : storage using a libaio-based asynchronous journal.
> Name : qpid-dispatch-debuginfo
> Arch : x86_64
> Version : 0.8.0
> Release : 10.el7
> Size : 2.1 M
> Repo : installed
> Summary : Debug information for package qpid-dispatch
> URL : http://qpid.apache.org/
> License : ASL 2.0
> Description : This package provides debug information for package
> qpid-dispatch.
> : Debug information is useful when developing applications that
> use this
> : package or when debugging this package.
> Name : qpid-dispatch-router
> Arch : x86_64
> Version : 0.8.0
> Release : 10.el7
> Size : 1.0 M
> Repo : installed
> From repo : rhel-server-7-satellite-6-beta-rpms
> Summary : The Qpid Dispatch Router executable
> URL : http://qpid.apache.org/
> License : ASL 2.0
> Description : The Qpid Dispatch Router executable.
> Name : qpid-proton-c
> Arch : x86_64
> Version : 0.16.0
> Release : 7.el7
> Size : 334 k
> Repo : installed
> From repo : rhel-server-7-satellite-6-beta-rpms
> Summary : C libraries for Qpid Proton
> URL : http://qpid.apache.org/proton/
> License : ASL 2.0
> Description : C libraries for Qpid Proton.
> Name : qpid-proton-debuginfo
> Arch : x86_64
> Version : 0.16.0
> Release : 7.el7
> Size : 6.8 M
> Repo : installed
> Summary : Debug information for package qpid-proton
> URL : http://qpid.apache.org/proton/
> License : ASL 2.0
> Description : This package provides debug information for package qpid-proton.
> : Debug information is useful when developing applications that
> use this
> : package or when debugging this package.
> Name : qpid-qmf
> Arch : x86_64
> Version : 1.36.0
> Release : 9.el7
> Size : 476 k
> Repo : installed
> From repo : rhel-server-7-satellite-6-beta-rpms
> Summary : The QPID Management Framework
> URL : http://qpid.apache.org
> License : ASL 2.0
> Description : The Qpid Management Framework is a general-purpose management
> bus built on Qpid
> : messaging. It takes advantage of the scalability, security, and
> rich
> : capabilities of Qpid to provide flexible and easy-to-use
> manageability to a
> : large set of applications.
> Name : qpid-tools
> Arch : noarch
> Version : 1.36.0
> Release : 9.el7
> Size : 384 k
> Repo : installed
> From repo : rhel-server-7-satellite-6-beta-rpms
> Summary : Management and diagostic tools for Apache Qpid
> URL : http://qpid.apache.org
> License : ASL 2.0
> Description : Management and diagnostic tools for Apache Qpid brokers and
> clients.
> Reporter: Saurabh Badhwar
>
> During the work process associated with Red Hat Satellite, we found that
> Qpid-dispatch router started to crash abruptly without any changes being made
> to it. On checking the logs, it seems like qpid-dispatch-router experienced a
> SEGFAULT and crashed.
> Once this crash starts to happen there is no way we are able to restart
> qpid-dispatch since it continues to fail.
> A trace from the /var/log/messages
> [root@c10-h19-r730xd-vm1 qdrouterd]# cat /var/log/messages | grep qdrouterd
> Nov 28 00:59:50 c10-h19-r730xd-vm1 kernel: qdrouterd[28352]: segfault at 180
> ip 00007fdd86d37291 sp 00007fdd7951c2b8 error 4 in
> libqpid-proton.so.10.0.0[7fdd86d19000+4b000]
> Nov 28 00:59:50 c10-h19-r730xd-vm1 systemd: qdrouterd.service: main process
> exited, code=killed, status=11/SEGV
> Nov 28 00:59:50 c10-h19-r730xd-vm1 systemd: Unit qdrouterd.service entered
> failed state.
> Nov 28 00:59:50 c10-h19-r730xd-vm1 systemd: qdrouterd.service failed.
> Nov 30 03:44:04 c10-h19-r730xd-vm1 /usr/sbin/katello-service: *** status
> failed: qdrouterd ***
> Nov 30 03:45:58 c10-h19-r730xd-vm1 kernel: qdrouterd[26607]: segfault at 180
> ip 00007f9f4d230291 sp 00007f9f3fa152b8 error 4 in
> libqpid-proton.so.10.0.0[7f9f4d212000+4b000]
> Nov 30 03:45:58 c10-h19-r730xd-vm1 systemd: qdrouterd.service: main process
> exited, code=killed, status=11/SEGV
> Nov 30 03:45:58 c10-h19-r730xd-vm1 systemd: Unit qdrouterd.service entered
> failed state.
> Nov 30 03:45:58 c10-h19-r730xd-vm1 systemd: qdrouterd.service failed.
> Nov 30 03:50:12 c10-h19-r730xd-vm1 /usr/sbin/katello-service: *** status
> failed: qdrouterd ***
> Nov 30 04:06:07 c10-h19-r730xd-vm1 ansible-file: Invoked with
> directory_mode=None force=False remote_src=None
> path=/etc/systemd/system/qdrouterd.service.d owner=root follow=False
> group=root unsafe_writes=None state=directory content=NOT_LOGGING_PARAMETER
> serole=None diff_peek=None setype=None selevel=None original_basename=None
> regexp=None validate=None src=None seuser=None recurse=False delimiter=None
> mode=0755 attributes=None backup=None
> Nov 30 04:06:10 c10-h19-r730xd-vm1 ansible-stat: Invoked with
> checksum_algorithm=sha1 get_checksum=True
> path=/etc/systemd/system/qdrouterd.service.d/limits.conf checksum_algo=sha1
> follow=False get_md5=False get_mime=True get_attributes=True
> Nov 30 04:06:13 c10-h19-r730xd-vm1 ansible-copy: Invoked with
> directory_mode=None force=True remote_src=None owner=root follow=False
> local_follow=None group=root unsafe_writes=None setype=None
> content=NOT_LOGGING_PARAMETER serole=None
> dest=/etc/systemd/system/qdrouterd.service.d/limits.conf selevel=None
> original_basename=limits-qdrouterd.conf regexp=None validate=None
> src=/root/.ansible/tmp/ansible-tmp-1512032767.73-18163995376563/source
> seuser=None delimiter=None mode=0644 attributes=None backup=False
> Nov 30 04:06:15 c10-h19-r730xd-vm1 ansible-command: Invoked with warn=True
> executable=None _uses_shell=False _raw_params=restorecon -vR
> /etc/systemd/system/qdrouterd.service.d removes=None creates=None chdir=None
> stdin=None
> Nov 30 04:06:42 c10-h19-r730xd-vm1 kernel: qdrouterd[29818]: segfault at 180
> ip 00007f0863215291 sp 00007ffd3666b678 error 4 in
> libqpid-proton.so.10.0.0[7f08631f7000+4b000]
> Nov 30 04:06:42 c10-h19-r730xd-vm1 systemd: qdrouterd.service: main process
> exited, code=killed, status=11/SEGV
> Nov 30 04:06:42 c10-h19-r730xd-vm1 systemd: Unit qdrouterd.service entered
> failed state.
> Nov 30 04:06:42 c10-h19-r730xd-vm1 systemd: qdrouterd.service failed.
> Nov 30 04:06:58 c10-h19-r730xd-vm1 /usr/sbin/katello-service: *** status
> failed: qdrouterd ***
> Nov 30 04:08:02 c10-h19-r730xd-vm1 ansible-file: Invoked with
> directory_mode=None force=False remote_src=None
> path=/etc/systemd/system/qdrouterd.service.d owner=root follow=False
> group=root unsafe_writes=None state=directory content=NOT_LOGGING_PARAMETER
> serole=None diff_peek=None setype=None selevel=None original_basename=None
> regexp=None validate=None src=None seuser=None recurse=False delimiter=None
> mode=0755 attributes=None backup=None
> Nov 30 04:08:05 c10-h19-r730xd-vm1 ansible-stat: Invoked with
> checksum_algorithm=sha1 get_checksum=True
> path=/etc/systemd/system/qdrouterd.service.d/limits.conf checksum_algo=sha1
> follow=False get_md5=False get_mime=True get_attributes=True
> Nov 30 04:08:07 c10-h19-r730xd-vm1 ansible-file: Invoked with
> directory_mode=None force=False remote_src=None
> path=/etc/systemd/system/qdrouterd.service.d/limits.conf owner=root
> follow=False group=root unsafe_writes=None state=file
> content=NOT_LOGGING_PARAMETER serole=None diff_peek=None setype=None
> dest=/etc/systemd/system/qdrouterd.service.d/limits.conf selevel=None
> original_basename=limits-qdrouterd.conf regexp=None validate=None
> src=limits-qdrouterd.conf seuser=None recurse=False delimiter=None mode=0644
> attributes=None backup=None
> Nov 30 04:08:09 c10-h19-r730xd-vm1 ansible-command: Invoked with warn=True
> executable=None _uses_shell=False _raw_params=restorecon -vR
> /etc/systemd/system/qdrouterd.service.d removes=None creates=None chdir=None
> stdin=None
> Nov 30 04:08:20 c10-h19-r730xd-vm1 /usr/sbin/katello-service: *** status
> failed: qdrouterd ***
> Nov 30 04:08:47 c10-h19-r730xd-vm1 kernel: qdrouterd[31775]: segfault at 180
> ip 00007f56880e4291 sp 00007f56799852b8 error 4 in
> libqpid-proton.so.10.0.0[7f56880c6000+4b000]
> Nov 30 04:08:47 c10-h19-r730xd-vm1 systemd: qdrouterd.service: main process
> exited, code=killed, status=11/SEGV
> Nov 30 04:08:47 c10-h19-r730xd-vm1 systemd: Unit qdrouterd.service entered
> failed state.
> Nov 30 04:08:47 c10-h19-r730xd-vm1 systemd: qdrouterd.service failed.
> Nov 30 04:09:06 c10-h19-r730xd-vm1 kernel: qdrouterd[32150]: segfault at 180
> ip 00007fb455eea291 sp 00007fb446f8a2b8 error 4 in
> libqpid-proton.so.10.0.0[7fb455ecc000+4b000]
> Nov 30 04:09:06 c10-h19-r730xd-vm1 systemd: qdrouterd.service: main process
> exited, code=killed, status=11/SEGV
> Nov 30 04:09:06 c10-h19-r730xd-vm1 systemd: Unit qdrouterd.service entered
> failed state.
> Nov 30 04:09:06 c10-h19-r730xd-vm1 systemd: qdrouterd.service failed.
> Nov 30 04:09:21 c10-h19-r730xd-vm1 /usr/sbin/katello-service: *** status
> failed: qdrouterd ***
> Nov 30 04:11:20 c10-h19-r730xd-vm1 /usr/sbin/katello-service: *** status
> failed: qdrouterd ***
> From the messages, it seems like, the SEGFAULT happen in libqpid-proton.so
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
