[jira] [Commented] (DISPATCH-963) Router crash during shutdown in system_tests_distribution

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DISPATCH-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493936#comment-16493936
 ] 

ASF GitHub Bot commented on DISPATCH-963:
-

Github user ganeshmurthy closed the pull request at:

https://github.com/apache/qpid-dispatch/pull/292


> Router crash during shutdown in system_tests_distribution
> -
>
> Key: DISPATCH-963
> URL: https://issues.apache.org/jira/browse/DISPATCH-963
> Project: Qpid Dispatch
>  Issue Type: Improvement
>  Components: Tests
>Affects Versions: 1.0.1
>Reporter: Ganesh Murthy
>Assignee: Ganesh Murthy
>Priority: Major
> Fix For: 1.1.0
>
>
> The router crashes during shutdown in system_tests_distribution.py
> Here is the backtrace
>  
> {noformat}
> (gdb) bt
> #0  0x7f361ca5ae40 in pn_ep_decref (endpoint=0x7f35f01c2dd0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:447
> #1  0x7f361ca5b58b in pn_ep_decref (endpoint=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:445
> #2  0x7f361ca5f588 in pni_transport_unbind_handles 
> (handles=0x7f35f00764a0, reset_state=reset_state@entry=true) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:748
> #3  0x7f361ca5f666 in pni_transport_unbind_channels (channels=0x9d1ce0) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:761
> #4  0x7f361ca5f777 in pn_transport_unbind (transport=0xa863d0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:795
> #5  0x7f361ca5a63e in pn_connection_driver_release_connection 
> (d=d@entry=0xa86248) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:81
> #6  0x7f361ca5a679 in pn_connection_driver_destroy (d=d@entry=0xa86248) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:92
> #7  0x7f361c83a69c in pconnection_final_free (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:827
> #8  0x7f361c83b3ac in pconnection_cleanup (pc=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:843
> #9  0x7f361c83db37 in pconnection_forced_shutdown (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:878
> #10 pn_proactor_free (p=0x916fd0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:1815
> #11 0x7f361cce7bb5 in qd_server_free (qd_server=0x919190) at 
> /home/gmurthy/opensource/qpid-dispatch/src/server.c:1176
> #12 0x7f361cca878e in qd_dispatch_free (qd=0x6164b0) at 
> /home/gmurthy/opensource/qpid-dispatch/src/dispatch.c:318
> #13 0x00401864 in main_process (config_path=0x7ffcb36d88e2 "B.conf", 
> python_pkgdir=0x7ffcb36d88ec "/home/gmurthy/opensource/qpid-dispatch/python", 
> fd=2) at /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:116
> #14 0x004022b0 in main (argc=5, argv=0x7ffcb36d8158) at 
> /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:360
> (gdb){noformat}
>  
> Running the test under valgrind, it seems that the pn_proactor_free is trying 
> to free already freed link endpoint. Here are two outputs from valgrind
>  
> {noformat}
> Process 3493 error: exit code 42, expected 0
> qdrouterd -c B.conf -I /home/gmurthy/opensource/qpid-dispatch/python
> /home/gmurthy/opensource/qpid-dispatch/build/system_test.dir/system_tests_distribution/DistributionTests/setUpClass/B-2.cmd
> 
> ==3493== Invalid write of size 8
> ==3493==    at 0x50E82B0: pn_link_unbound (engine.c:1202)
> ==3493==    by 0x50EB5D0: pni_transport_unbind_handles (transport.c:746)
> ==3493==    by 0x50EB665: pni_transport_unbind_channels (transport.c:761)
> ==3493==    by 0x50EB776: pn_transport_unbind (transport.c:795)
> ==3493==    by 0x50E663D: pn_connection_driver_release_connection 
> (connection_driver.c:81)
> ==3493==    by 0x50E6678: pn_connection_driver_destroy 
> (connection_driver.c:92)
> ==3493==    by 0x530A69B: pconnection_final_free (epoll.c:827)
> ==3493==    by 0x530DB36: pconnection_forced_shutdown (epoll.c:878)
> ==3493==    by 0x530DB36: pn_proactor_free (epoll.c:1815)
> ==3493==    by 0x4EA5BB4: qd_server_free (server.c:1176)
> ==3493==    by 0x4E6678D: qd_dispatch_free (dispatch.c:318)
> ==3493==    by 0x401863: main_process (main.c:116)
> ==3493==    by 0x4022AF: main (main.c:360)
> ==3493==  Address 0x15c06188 is 376 bytes inside a block of size 488 free'd
> ==3493==    at 0x4C2DD18: free (vg_replace_malloc.c:530)
> ==3493==    by 0x50DD938: pn_class_decref (object.c:101)
> ==3493==    by 0x50EA03F: pn_event_finalize (event.c:226)
> ==3493==    by 0x50EA03F: pn_event_finalize_cast (event.c:271)
> ==3493==    by 

[jira] [Commented] (DISPATCH-963) Router crash during shutdown in system_tests_distribution

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/DISPATCH-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463264#comment-16463264
 ] 

ASF subversion and git services commented on DISPATCH-963:
--

Commit 21b7e664f085ed56ec423020b8352f500d8ede9e in qpid-dispatch's branch 
refs/heads/master from [~fgiorget]
[ https://git-wip-us.apache.org/repos/asf?p=qpid-dispatch.git;h=21b7e66 ]

DISPATCH-963 - Added decorator class that can be used to skip test methods, as 
unittest2 does not support skipping feature on python 2.6.


> Router crash during shutdown in system_tests_distribution
> -
>
> Key: DISPATCH-963
> URL: https://issues.apache.org/jira/browse/DISPATCH-963
> Project: Qpid Dispatch
>  Issue Type: Improvement
>  Components: Tests
>Affects Versions: 1.0.1
>Reporter: Ganesh Murthy
>Assignee: Ganesh Murthy
>Priority: Major
> Fix For: 1.1.0
>
>
> The router crashes during shutdown in system_tests_distribution.py
> Here is the backtrace
>  
> {noformat}
> (gdb) bt
> #0  0x7f361ca5ae40 in pn_ep_decref (endpoint=0x7f35f01c2dd0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:447
> #1  0x7f361ca5b58b in pn_ep_decref (endpoint=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:445
> #2  0x7f361ca5f588 in pni_transport_unbind_handles 
> (handles=0x7f35f00764a0, reset_state=reset_state@entry=true) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:748
> #3  0x7f361ca5f666 in pni_transport_unbind_channels (channels=0x9d1ce0) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:761
> #4  0x7f361ca5f777 in pn_transport_unbind (transport=0xa863d0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:795
> #5  0x7f361ca5a63e in pn_connection_driver_release_connection 
> (d=d@entry=0xa86248) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:81
> #6  0x7f361ca5a679 in pn_connection_driver_destroy (d=d@entry=0xa86248) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:92
> #7  0x7f361c83a69c in pconnection_final_free (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:827
> #8  0x7f361c83b3ac in pconnection_cleanup (pc=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:843
> #9  0x7f361c83db37 in pconnection_forced_shutdown (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:878
> #10 pn_proactor_free (p=0x916fd0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:1815
> #11 0x7f361cce7bb5 in qd_server_free (qd_server=0x919190) at 
> /home/gmurthy/opensource/qpid-dispatch/src/server.c:1176
> #12 0x7f361cca878e in qd_dispatch_free (qd=0x6164b0) at 
> /home/gmurthy/opensource/qpid-dispatch/src/dispatch.c:318
> #13 0x00401864 in main_process (config_path=0x7ffcb36d88e2 "B.conf", 
> python_pkgdir=0x7ffcb36d88ec "/home/gmurthy/opensource/qpid-dispatch/python", 
> fd=2) at /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:116
> #14 0x004022b0 in main (argc=5, argv=0x7ffcb36d8158) at 
> /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:360
> (gdb){noformat}
>  
> Running the test under valgrind, it seems that the pn_proactor_free is trying 
> to free already freed link endpoint. Here are two outputs from valgrind
>  
> {noformat}
> Process 3493 error: exit code 42, expected 0
> qdrouterd -c B.conf -I /home/gmurthy/opensource/qpid-dispatch/python
> /home/gmurthy/opensource/qpid-dispatch/build/system_test.dir/system_tests_distribution/DistributionTests/setUpClass/B-2.cmd
> 
> ==3493== Invalid write of size 8
> ==3493==    at 0x50E82B0: pn_link_unbound (engine.c:1202)
> ==3493==    by 0x50EB5D0: pni_transport_unbind_handles (transport.c:746)
> ==3493==    by 0x50EB665: pni_transport_unbind_channels (transport.c:761)
> ==3493==    by 0x50EB776: pn_transport_unbind (transport.c:795)
> ==3493==    by 0x50E663D: pn_connection_driver_release_connection 
> (connection_driver.c:81)
> ==3493==    by 0x50E6678: pn_connection_driver_destroy 
> (connection_driver.c:92)
> ==3493==    by 0x530A69B: pconnection_final_free (epoll.c:827)
> ==3493==    by 0x530DB36: pconnection_forced_shutdown (epoll.c:878)
> ==3493==    by 0x530DB36: pn_proactor_free (epoll.c:1815)
> ==3493==    by 0x4EA5BB4: qd_server_free (server.c:1176)
> ==3493==    by 0x4E6678D: qd_dispatch_free (dispatch.c:318)
> ==3493==    by 0x401863: main_process (main.c:116)
> ==3493==    by 0x4022AF: main (main.c:360)
> ==3493==  Address 0x15c06188 is 376 bytes inside a block of size 488 free'd
> 

[jira] [Commented] (DISPATCH-963) Router crash during shutdown in system_tests_distribution

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DISPATCH-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463265#comment-16463265
 ] 

ASF GitHub Bot commented on DISPATCH-963:
-

Github user asfgit closed the pull request at:

https://github.com/apache/qpid-dispatch/pull/298


> Router crash during shutdown in system_tests_distribution
> -
>
> Key: DISPATCH-963
> URL: https://issues.apache.org/jira/browse/DISPATCH-963
> Project: Qpid Dispatch
>  Issue Type: Improvement
>  Components: Tests
>Affects Versions: 1.0.1
>Reporter: Ganesh Murthy
>Assignee: Ganesh Murthy
>Priority: Major
> Fix For: 1.1.0
>
>
> The router crashes during shutdown in system_tests_distribution.py
> Here is the backtrace
>  
> {noformat}
> (gdb) bt
> #0  0x7f361ca5ae40 in pn_ep_decref (endpoint=0x7f35f01c2dd0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:447
> #1  0x7f361ca5b58b in pn_ep_decref (endpoint=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:445
> #2  0x7f361ca5f588 in pni_transport_unbind_handles 
> (handles=0x7f35f00764a0, reset_state=reset_state@entry=true) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:748
> #3  0x7f361ca5f666 in pni_transport_unbind_channels (channels=0x9d1ce0) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:761
> #4  0x7f361ca5f777 in pn_transport_unbind (transport=0xa863d0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:795
> #5  0x7f361ca5a63e in pn_connection_driver_release_connection 
> (d=d@entry=0xa86248) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:81
> #6  0x7f361ca5a679 in pn_connection_driver_destroy (d=d@entry=0xa86248) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:92
> #7  0x7f361c83a69c in pconnection_final_free (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:827
> #8  0x7f361c83b3ac in pconnection_cleanup (pc=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:843
> #9  0x7f361c83db37 in pconnection_forced_shutdown (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:878
> #10 pn_proactor_free (p=0x916fd0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:1815
> #11 0x7f361cce7bb5 in qd_server_free (qd_server=0x919190) at 
> /home/gmurthy/opensource/qpid-dispatch/src/server.c:1176
> #12 0x7f361cca878e in qd_dispatch_free (qd=0x6164b0) at 
> /home/gmurthy/opensource/qpid-dispatch/src/dispatch.c:318
> #13 0x00401864 in main_process (config_path=0x7ffcb36d88e2 "B.conf", 
> python_pkgdir=0x7ffcb36d88ec "/home/gmurthy/opensource/qpid-dispatch/python", 
> fd=2) at /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:116
> #14 0x004022b0 in main (argc=5, argv=0x7ffcb36d8158) at 
> /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:360
> (gdb){noformat}
>  
> Running the test under valgrind, it seems that the pn_proactor_free is trying 
> to free already freed link endpoint. Here are two outputs from valgrind
>  
> {noformat}
> Process 3493 error: exit code 42, expected 0
> qdrouterd -c B.conf -I /home/gmurthy/opensource/qpid-dispatch/python
> /home/gmurthy/opensource/qpid-dispatch/build/system_test.dir/system_tests_distribution/DistributionTests/setUpClass/B-2.cmd
> 
> ==3493== Invalid write of size 8
> ==3493==    at 0x50E82B0: pn_link_unbound (engine.c:1202)
> ==3493==    by 0x50EB5D0: pni_transport_unbind_handles (transport.c:746)
> ==3493==    by 0x50EB665: pni_transport_unbind_channels (transport.c:761)
> ==3493==    by 0x50EB776: pn_transport_unbind (transport.c:795)
> ==3493==    by 0x50E663D: pn_connection_driver_release_connection 
> (connection_driver.c:81)
> ==3493==    by 0x50E6678: pn_connection_driver_destroy 
> (connection_driver.c:92)
> ==3493==    by 0x530A69B: pconnection_final_free (epoll.c:827)
> ==3493==    by 0x530DB36: pconnection_forced_shutdown (epoll.c:878)
> ==3493==    by 0x530DB36: pn_proactor_free (epoll.c:1815)
> ==3493==    by 0x4EA5BB4: qd_server_free (server.c:1176)
> ==3493==    by 0x4E6678D: qd_dispatch_free (dispatch.c:318)
> ==3493==    by 0x401863: main_process (main.c:116)
> ==3493==    by 0x4022AF: main (main.c:360)
> ==3493==  Address 0x15c06188 is 376 bytes inside a block of size 488 free'd
> ==3493==    at 0x4C2DD18: free (vg_replace_malloc.c:530)
> ==3493==    by 0x50DD938: pn_class_decref (object.c:101)
> ==3493==    by 0x50EA03F: pn_event_finalize (event.c:226)
> ==3493==    by 0x50EA03F: pn_event_finalize_cast (event.c:271)
> ==3493==    by 

[jira] [Commented] (DISPATCH-963) Router crash during shutdown in system_tests_distribution

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DISPATCH-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463126#comment-16463126
 ] 

ASF GitHub Bot commented on DISPATCH-963:
-

GitHub user fgiorgetti opened a pull request:

https://github.com/apache/qpid-dispatch/pull/298

DISPATCH-963 - Added decorator class that can be used to skip test methods, 
as unittest2 does not support skipping feature on python 2.6.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fgiorgetti/qpid-dispatch 
fgiorgetti-DISPATCH-963

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/qpid-dispatch/pull/298.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #298


commit 21b7e664f085ed56ec423020b8352f500d8ede9e
Author: Fernando Giorgetti 
Date:   2018-05-03T21:54:29Z

DISPATCH-963 - Added decorator class that can be used to skip test methods, 
as unittest2 does not support skipping feature on python 2.6.




> Router crash during shutdown in system_tests_distribution
> -
>
> Key: DISPATCH-963
> URL: https://issues.apache.org/jira/browse/DISPATCH-963
> Project: Qpid Dispatch
>  Issue Type: Improvement
>  Components: Tests
>Affects Versions: 1.0.1
>Reporter: Ganesh Murthy
>Assignee: Ganesh Murthy
>Priority: Major
> Fix For: 1.1.0
>
>
> The router crashes during shutdown in system_tests_distribution.py
> Here is the backtrace
>  
> {noformat}
> (gdb) bt
> #0  0x7f361ca5ae40 in pn_ep_decref (endpoint=0x7f35f01c2dd0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:447
> #1  0x7f361ca5b58b in pn_ep_decref (endpoint=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:445
> #2  0x7f361ca5f588 in pni_transport_unbind_handles 
> (handles=0x7f35f00764a0, reset_state=reset_state@entry=true) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:748
> #3  0x7f361ca5f666 in pni_transport_unbind_channels (channels=0x9d1ce0) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:761
> #4  0x7f361ca5f777 in pn_transport_unbind (transport=0xa863d0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:795
> #5  0x7f361ca5a63e in pn_connection_driver_release_connection 
> (d=d@entry=0xa86248) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:81
> #6  0x7f361ca5a679 in pn_connection_driver_destroy (d=d@entry=0xa86248) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:92
> #7  0x7f361c83a69c in pconnection_final_free (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:827
> #8  0x7f361c83b3ac in pconnection_cleanup (pc=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:843
> #9  0x7f361c83db37 in pconnection_forced_shutdown (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:878
> #10 pn_proactor_free (p=0x916fd0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:1815
> #11 0x7f361cce7bb5 in qd_server_free (qd_server=0x919190) at 
> /home/gmurthy/opensource/qpid-dispatch/src/server.c:1176
> #12 0x7f361cca878e in qd_dispatch_free (qd=0x6164b0) at 
> /home/gmurthy/opensource/qpid-dispatch/src/dispatch.c:318
> #13 0x00401864 in main_process (config_path=0x7ffcb36d88e2 "B.conf", 
> python_pkgdir=0x7ffcb36d88ec "/home/gmurthy/opensource/qpid-dispatch/python", 
> fd=2) at /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:116
> #14 0x004022b0 in main (argc=5, argv=0x7ffcb36d8158) at 
> /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:360
> (gdb){noformat}
>  
> Running the test under valgrind, it seems that the pn_proactor_free is trying 
> to free already freed link endpoint. Here are two outputs from valgrind
>  
> {noformat}
> Process 3493 error: exit code 42, expected 0
> qdrouterd -c B.conf -I /home/gmurthy/opensource/qpid-dispatch/python
> /home/gmurthy/opensource/qpid-dispatch/build/system_test.dir/system_tests_distribution/DistributionTests/setUpClass/B-2.cmd
> 
> ==3493== Invalid write of size 8
> ==3493==    at 0x50E82B0: pn_link_unbound (engine.c:1202)
> ==3493==    by 0x50EB5D0: pni_transport_unbind_handles (transport.c:746)
> ==3493==    by 0x50EB665: pni_transport_unbind_channels (transport.c:761)
> ==3493==    by 0x50EB776: pn_transport_unbind (transport.c:795)
> ==3493==    by 0x50E663D: pn_connection_driver_release_connection 
> 

[jira] [Commented] (DISPATCH-963) Router crash during shutdown in system_tests_distribution

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/DISPATCH-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462984#comment-16462984
 ] 

ASF subversion and git services commented on DISPATCH-963:
--

Commit 4f5cc528736acf84bd81cd84ef6aea339fe966d2 in qpid-dispatch's branch 
refs/heads/master from [~ganeshmurthy]
[ https://git-wip-us.apache.org/repos/asf?p=qpid-dispatch.git;h=4f5cc52 ]

DISPATCH-963 - Check the version of Python and return without executing the 
test if version is 2.6. unittest2 does not support skipping tests


> Router crash during shutdown in system_tests_distribution
> -
>
> Key: DISPATCH-963
> URL: https://issues.apache.org/jira/browse/DISPATCH-963
> Project: Qpid Dispatch
>  Issue Type: Improvement
>  Components: Tests
>Affects Versions: 1.0.1
>Reporter: Ganesh Murthy
>Assignee: Ganesh Murthy
>Priority: Major
> Fix For: 1.1.0
>
>
> The router crashes during shutdown in system_tests_distribution.py
> Here is the backtrace
>  
> {noformat}
> (gdb) bt
> #0  0x7f361ca5ae40 in pn_ep_decref (endpoint=0x7f35f01c2dd0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:447
> #1  0x7f361ca5b58b in pn_ep_decref (endpoint=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:445
> #2  0x7f361ca5f588 in pni_transport_unbind_handles 
> (handles=0x7f35f00764a0, reset_state=reset_state@entry=true) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:748
> #3  0x7f361ca5f666 in pni_transport_unbind_channels (channels=0x9d1ce0) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:761
> #4  0x7f361ca5f777 in pn_transport_unbind (transport=0xa863d0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:795
> #5  0x7f361ca5a63e in pn_connection_driver_release_connection 
> (d=d@entry=0xa86248) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:81
> #6  0x7f361ca5a679 in pn_connection_driver_destroy (d=d@entry=0xa86248) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:92
> #7  0x7f361c83a69c in pconnection_final_free (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:827
> #8  0x7f361c83b3ac in pconnection_cleanup (pc=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:843
> #9  0x7f361c83db37 in pconnection_forced_shutdown (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:878
> #10 pn_proactor_free (p=0x916fd0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:1815
> #11 0x7f361cce7bb5 in qd_server_free (qd_server=0x919190) at 
> /home/gmurthy/opensource/qpid-dispatch/src/server.c:1176
> #12 0x7f361cca878e in qd_dispatch_free (qd=0x6164b0) at 
> /home/gmurthy/opensource/qpid-dispatch/src/dispatch.c:318
> #13 0x00401864 in main_process (config_path=0x7ffcb36d88e2 "B.conf", 
> python_pkgdir=0x7ffcb36d88ec "/home/gmurthy/opensource/qpid-dispatch/python", 
> fd=2) at /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:116
> #14 0x004022b0 in main (argc=5, argv=0x7ffcb36d8158) at 
> /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:360
> (gdb){noformat}
>  
> Running the test under valgrind, it seems that the pn_proactor_free is trying 
> to free already freed link endpoint. Here are two outputs from valgrind
>  
> {noformat}
> Process 3493 error: exit code 42, expected 0
> qdrouterd -c B.conf -I /home/gmurthy/opensource/qpid-dispatch/python
> /home/gmurthy/opensource/qpid-dispatch/build/system_test.dir/system_tests_distribution/DistributionTests/setUpClass/B-2.cmd
> 
> ==3493== Invalid write of size 8
> ==3493==    at 0x50E82B0: pn_link_unbound (engine.c:1202)
> ==3493==    by 0x50EB5D0: pni_transport_unbind_handles (transport.c:746)
> ==3493==    by 0x50EB665: pni_transport_unbind_channels (transport.c:761)
> ==3493==    by 0x50EB776: pn_transport_unbind (transport.c:795)
> ==3493==    by 0x50E663D: pn_connection_driver_release_connection 
> (connection_driver.c:81)
> ==3493==    by 0x50E6678: pn_connection_driver_destroy 
> (connection_driver.c:92)
> ==3493==    by 0x530A69B: pconnection_final_free (epoll.c:827)
> ==3493==    by 0x530DB36: pconnection_forced_shutdown (epoll.c:878)
> ==3493==    by 0x530DB36: pn_proactor_free (epoll.c:1815)
> ==3493==    by 0x4EA5BB4: qd_server_free (server.c:1176)
> ==3493==    by 0x4E6678D: qd_dispatch_free (dispatch.c:318)
> ==3493==    by 0x401863: main_process (main.c:116)
> ==3493==    by 0x4022AF: main (main.c:360)
> ==3493==  Address 0x15c06188 is 376 bytes inside a block of size 488 free'd
> 

[jira] [Commented] (DISPATCH-963) Router crash during shutdown in system_tests_distribution

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/DISPATCH-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457059#comment-16457059
 ] 

ASF subversion and git services commented on DISPATCH-963:
--

Commit 1a2bde109bdbf027e1645c68d4543852781e3a42 in qpid-dispatch's branch 
refs/heads/1.1.x from [~ganeshmurthy]
[ https://git-wip-us.apache.org/repos/asf?p=qpid-dispatch.git;h=1a2bde1 ]

DISPATCH-963 - Comment out tests that cause circular link routing issue. This 
issue will be addressed in an upcoming release


> Router crash during shutdown in system_tests_distribution
> -
>
> Key: DISPATCH-963
> URL: https://issues.apache.org/jira/browse/DISPATCH-963
> Project: Qpid Dispatch
>  Issue Type: Improvement
>  Components: Tests
>Affects Versions: 1.0.1
>Reporter: Ganesh Murthy
>Assignee: Ganesh Murthy
>Priority: Major
> Fix For: 1.1.0
>
>
> The router crashes during shutdown in system_tests_distribution.py
> Here is the backtrace
>  
> {noformat}
> (gdb) bt
> #0  0x7f361ca5ae40 in pn_ep_decref (endpoint=0x7f35f01c2dd0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:447
> #1  0x7f361ca5b58b in pn_ep_decref (endpoint=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:445
> #2  0x7f361ca5f588 in pni_transport_unbind_handles 
> (handles=0x7f35f00764a0, reset_state=reset_state@entry=true) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:748
> #3  0x7f361ca5f666 in pni_transport_unbind_channels (channels=0x9d1ce0) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:761
> #4  0x7f361ca5f777 in pn_transport_unbind (transport=0xa863d0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:795
> #5  0x7f361ca5a63e in pn_connection_driver_release_connection 
> (d=d@entry=0xa86248) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:81
> #6  0x7f361ca5a679 in pn_connection_driver_destroy (d=d@entry=0xa86248) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:92
> #7  0x7f361c83a69c in pconnection_final_free (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:827
> #8  0x7f361c83b3ac in pconnection_cleanup (pc=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:843
> #9  0x7f361c83db37 in pconnection_forced_shutdown (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:878
> #10 pn_proactor_free (p=0x916fd0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:1815
> #11 0x7f361cce7bb5 in qd_server_free (qd_server=0x919190) at 
> /home/gmurthy/opensource/qpid-dispatch/src/server.c:1176
> #12 0x7f361cca878e in qd_dispatch_free (qd=0x6164b0) at 
> /home/gmurthy/opensource/qpid-dispatch/src/dispatch.c:318
> #13 0x00401864 in main_process (config_path=0x7ffcb36d88e2 "B.conf", 
> python_pkgdir=0x7ffcb36d88ec "/home/gmurthy/opensource/qpid-dispatch/python", 
> fd=2) at /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:116
> #14 0x004022b0 in main (argc=5, argv=0x7ffcb36d8158) at 
> /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:360
> (gdb){noformat}
>  
> Running the test under valgrind, it seems that the pn_proactor_free is trying 
> to free already freed link endpoint. Here are two outputs from valgrind
>  
> {noformat}
> Process 3493 error: exit code 42, expected 0
> qdrouterd -c B.conf -I /home/gmurthy/opensource/qpid-dispatch/python
> /home/gmurthy/opensource/qpid-dispatch/build/system_test.dir/system_tests_distribution/DistributionTests/setUpClass/B-2.cmd
> 
> ==3493== Invalid write of size 8
> ==3493==    at 0x50E82B0: pn_link_unbound (engine.c:1202)
> ==3493==    by 0x50EB5D0: pni_transport_unbind_handles (transport.c:746)
> ==3493==    by 0x50EB665: pni_transport_unbind_channels (transport.c:761)
> ==3493==    by 0x50EB776: pn_transport_unbind (transport.c:795)
> ==3493==    by 0x50E663D: pn_connection_driver_release_connection 
> (connection_driver.c:81)
> ==3493==    by 0x50E6678: pn_connection_driver_destroy 
> (connection_driver.c:92)
> ==3493==    by 0x530A69B: pconnection_final_free (epoll.c:827)
> ==3493==    by 0x530DB36: pconnection_forced_shutdown (epoll.c:878)
> ==3493==    by 0x530DB36: pn_proactor_free (epoll.c:1815)
> ==3493==    by 0x4EA5BB4: qd_server_free (server.c:1176)
> ==3493==    by 0x4E6678D: qd_dispatch_free (dispatch.c:318)
> ==3493==    by 0x401863: main_process (main.c:116)
> ==3493==    by 0x4022AF: main (main.c:360)
> ==3493==  Address 0x15c06188 is 376 bytes inside a block of size 488 free'd
> ==3493==    at 

[jira] [Commented] (DISPATCH-963) Router crash during shutdown in system_tests_distribution

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/DISPATCH-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457028#comment-16457028
 ] 

ASF subversion and git services commented on DISPATCH-963:
--

Commit 788a2b74840c35efe357db6b65facd1728c5d6ac in qpid-dispatch's branch 
refs/heads/master from [~ganeshmurthy]
[ https://git-wip-us.apache.org/repos/asf?p=qpid-dispatch.git;h=788a2b7 ]

DISPATCH-963 - Comment out tests that cause circular link routing issue. This 
issue will be addressed in an upcoming release


> Router crash during shutdown in system_tests_distribution
> -
>
> Key: DISPATCH-963
> URL: https://issues.apache.org/jira/browse/DISPATCH-963
> Project: Qpid Dispatch
>  Issue Type: Improvement
>  Components: Tests
>Affects Versions: 1.0.1
>Reporter: Ganesh Murthy
>Assignee: Ganesh Murthy
>Priority: Major
> Fix For: 1.1.0
>
>
> The router crashes during shutdown in system_tests_distribution.py
> Here is the backtrace
>  
> {noformat}
> (gdb) bt
> #0  0x7f361ca5ae40 in pn_ep_decref (endpoint=0x7f35f01c2dd0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:447
> #1  0x7f361ca5b58b in pn_ep_decref (endpoint=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:445
> #2  0x7f361ca5f588 in pni_transport_unbind_handles 
> (handles=0x7f35f00764a0, reset_state=reset_state@entry=true) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:748
> #3  0x7f361ca5f666 in pni_transport_unbind_channels (channels=0x9d1ce0) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:761
> #4  0x7f361ca5f777 in pn_transport_unbind (transport=0xa863d0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:795
> #5  0x7f361ca5a63e in pn_connection_driver_release_connection 
> (d=d@entry=0xa86248) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:81
> #6  0x7f361ca5a679 in pn_connection_driver_destroy (d=d@entry=0xa86248) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:92
> #7  0x7f361c83a69c in pconnection_final_free (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:827
> #8  0x7f361c83b3ac in pconnection_cleanup (pc=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:843
> #9  0x7f361c83db37 in pconnection_forced_shutdown (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:878
> #10 pn_proactor_free (p=0x916fd0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:1815
> #11 0x7f361cce7bb5 in qd_server_free (qd_server=0x919190) at 
> /home/gmurthy/opensource/qpid-dispatch/src/server.c:1176
> #12 0x7f361cca878e in qd_dispatch_free (qd=0x6164b0) at 
> /home/gmurthy/opensource/qpid-dispatch/src/dispatch.c:318
> #13 0x00401864 in main_process (config_path=0x7ffcb36d88e2 "B.conf", 
> python_pkgdir=0x7ffcb36d88ec "/home/gmurthy/opensource/qpid-dispatch/python", 
> fd=2) at /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:116
> #14 0x004022b0 in main (argc=5, argv=0x7ffcb36d8158) at 
> /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:360
> (gdb){noformat}
>  
> Running the test under valgrind, it seems that the pn_proactor_free is trying 
> to free already freed link endpoint. Here are two outputs from valgrind
>  
> {noformat}
> Process 3493 error: exit code 42, expected 0
> qdrouterd -c B.conf -I /home/gmurthy/opensource/qpid-dispatch/python
> /home/gmurthy/opensource/qpid-dispatch/build/system_test.dir/system_tests_distribution/DistributionTests/setUpClass/B-2.cmd
> 
> ==3493== Invalid write of size 8
> ==3493==    at 0x50E82B0: pn_link_unbound (engine.c:1202)
> ==3493==    by 0x50EB5D0: pni_transport_unbind_handles (transport.c:746)
> ==3493==    by 0x50EB665: pni_transport_unbind_channels (transport.c:761)
> ==3493==    by 0x50EB776: pn_transport_unbind (transport.c:795)
> ==3493==    by 0x50E663D: pn_connection_driver_release_connection 
> (connection_driver.c:81)
> ==3493==    by 0x50E6678: pn_connection_driver_destroy 
> (connection_driver.c:92)
> ==3493==    by 0x530A69B: pconnection_final_free (epoll.c:827)
> ==3493==    by 0x530DB36: pconnection_forced_shutdown (epoll.c:878)
> ==3493==    by 0x530DB36: pn_proactor_free (epoll.c:1815)
> ==3493==    by 0x4EA5BB4: qd_server_free (server.c:1176)
> ==3493==    by 0x4E6678D: qd_dispatch_free (dispatch.c:318)
> ==3493==    by 0x401863: main_process (main.c:116)
> ==3493==    by 0x4022AF: main (main.c:360)
> ==3493==  Address 0x15c06188 is 376 bytes inside a block of size 488 free'd
> ==3493==    at 

[jira] [Commented] (DISPATCH-963) Router crash during shutdown in system_tests_distribution

[Alan Conway (JIRA)]

[ 
https://issues.apache.org/jira/browse/DISPATCH-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454325#comment-16454325
 ] 

Alan Conway commented on DISPATCH-963:
--

The proton bug has been fixed on master for proton-0.23 
https://issues.apache.org/jira/browse/PROTON-1832?focusedCommentId=16454298=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16454298

 

The affected dispatch tests now hang but leave diagnostics in the router log 
files which should help diagnose the remaining dispatch test or router issues, 
for example:

 
{code:java}
$ grep already 
/home/aconway/dispatch/bld/tests/system_test.dir/system_tests_distribution/DistributionTests/setUpClass/*
/home/aconway/dispatch/bld/tests/system_test.dir/system_tests_distribution/DistributionTests/setUpClass/A.log:2018-04-26
 09:53:11.592154 -0400 SERVER (trace) [5]:0 <- @close(24) [error=@error(29) 
[condition=:"amqp:invalid-field", description="link name already attached: 
sender_A_8"]] (/home/aconway/dispatch/src/server.c:103)
/home/aconway/dispatch/bld/tests/system_test.dir/system_tests_distribution/DistributionTests/setUpClass/B.log:2018-04-26
 09:53:11.590638 -0400 SERVER (trace) [1]:0 -> @close(24) [error=@error(29) 
[condition=:"amqp:invalid-field", description="link name already attached: 
sender_A_8"]] (/home/aconway/dispatch/src/server.c:103)
/home/aconway/dispatch/bld/tests/system_test.dir/system_tests_distribution/DistributionTests/setUpClass/B.log:2018-04-26
 09:53:11.591459 -0400 SERVER (info) Connection to 127.0.0.1:20900 failed: 
amqp:invalid-field link name already attached: sender_A_8 
(/home/aconway/dispatch/src/server.c:921){code}
 

> Router crash during shutdown in system_tests_distribution
> -
>
> Key: DISPATCH-963
> URL: https://issues.apache.org/jira/browse/DISPATCH-963
> Project: Qpid Dispatch
>  Issue Type: Improvement
>  Components: Tests
>Affects Versions: 1.0.1
>Reporter: Ganesh Murthy
>Assignee: Ganesh Murthy
>Priority: Major
>
> The router crashes during shutdown in system_tests_distribution.py
> Here is the backtrace
>  
> {noformat}
> (gdb) bt
> #0  0x7f361ca5ae40 in pn_ep_decref (endpoint=0x7f35f01c2dd0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:447
> #1  0x7f361ca5b58b in pn_ep_decref (endpoint=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:445
> #2  0x7f361ca5f588 in pni_transport_unbind_handles 
> (handles=0x7f35f00764a0, reset_state=reset_state@entry=true) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:748
> #3  0x7f361ca5f666 in pni_transport_unbind_channels (channels=0x9d1ce0) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:761
> #4  0x7f361ca5f777 in pn_transport_unbind (transport=0xa863d0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:795
> #5  0x7f361ca5a63e in pn_connection_driver_release_connection 
> (d=d@entry=0xa86248) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:81
> #6  0x7f361ca5a679 in pn_connection_driver_destroy (d=d@entry=0xa86248) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:92
> #7  0x7f361c83a69c in pconnection_final_free (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:827
> #8  0x7f361c83b3ac in pconnection_cleanup (pc=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:843
> #9  0x7f361c83db37 in pconnection_forced_shutdown (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:878
> #10 pn_proactor_free (p=0x916fd0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:1815
> #11 0x7f361cce7bb5 in qd_server_free (qd_server=0x919190) at 
> /home/gmurthy/opensource/qpid-dispatch/src/server.c:1176
> #12 0x7f361cca878e in qd_dispatch_free (qd=0x6164b0) at 
> /home/gmurthy/opensource/qpid-dispatch/src/dispatch.c:318
> #13 0x00401864 in main_process (config_path=0x7ffcb36d88e2 "B.conf", 
> python_pkgdir=0x7ffcb36d88ec "/home/gmurthy/opensource/qpid-dispatch/python", 
> fd=2) at /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:116
> #14 0x004022b0 in main (argc=5, argv=0x7ffcb36d8158) at 
> /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:360
> (gdb){noformat}
>  
> Running the test under valgrind, it seems that the pn_proactor_free is trying 
> to free already freed link endpoint. Here are two outputs from valgrind
>  
> {noformat}
> Process 3493 error: exit code 42, expected 0
> qdrouterd -c B.conf -I /home/gmurthy/opensource/qpid-dispatch/python
> 

[jira] [Commented] (DISPATCH-963) Router crash during shutdown in system_tests_distribution

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DISPATCH-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450310#comment-16450310
 ] 

ASF GitHub Bot commented on DISPATCH-963:
-

GitHub user ganeshmurthy opened a pull request:

https://github.com/apache/qpid-dispatch/pull/292

DISPATCH-963 - Added test name to creation of senders and receivers t…

…o make the names unique

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ganeshmurthy/qpid-dispatch ADDRESS-CONFIG-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/qpid-dispatch/pull/292.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #292


commit ca84e0dab3381a7f36f38c5c4a51b2e03525c174
Author: Ganesh Murthy 
Date:   2018-04-24T18:02:31Z

DISPATCH-963 - Added test name to creation of senders and receivers to make 
the names unique




> Router crash during shutdown in system_tests_distribution
> -
>
> Key: DISPATCH-963
> URL: https://issues.apache.org/jira/browse/DISPATCH-963
> Project: Qpid Dispatch
>  Issue Type: Improvement
>  Components: Tests
>Affects Versions: 1.0.1
>Reporter: Ganesh Murthy
>Assignee: Ganesh Murthy
>Priority: Major
>
> The router crashes during shutdown in system_tests_distribution.py
> Here is the backtrace
>  
> {noformat}
> (gdb) bt
> #0  0x7f361ca5ae40 in pn_ep_decref (endpoint=0x7f35f01c2dd0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:447
> #1  0x7f361ca5b58b in pn_ep_decref (endpoint=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:445
> #2  0x7f361ca5f588 in pni_transport_unbind_handles 
> (handles=0x7f35f00764a0, reset_state=reset_state@entry=true) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:748
> #3  0x7f361ca5f666 in pni_transport_unbind_channels (channels=0x9d1ce0) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:761
> #4  0x7f361ca5f777 in pn_transport_unbind (transport=0xa863d0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:795
> #5  0x7f361ca5a63e in pn_connection_driver_release_connection 
> (d=d@entry=0xa86248) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:81
> #6  0x7f361ca5a679 in pn_connection_driver_destroy (d=d@entry=0xa86248) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:92
> #7  0x7f361c83a69c in pconnection_final_free (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:827
> #8  0x7f361c83b3ac in pconnection_cleanup (pc=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:843
> #9  0x7f361c83db37 in pconnection_forced_shutdown (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:878
> #10 pn_proactor_free (p=0x916fd0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:1815
> #11 0x7f361cce7bb5 in qd_server_free (qd_server=0x919190) at 
> /home/gmurthy/opensource/qpid-dispatch/src/server.c:1176
> #12 0x7f361cca878e in qd_dispatch_free (qd=0x6164b0) at 
> /home/gmurthy/opensource/qpid-dispatch/src/dispatch.c:318
> #13 0x00401864 in main_process (config_path=0x7ffcb36d88e2 "B.conf", 
> python_pkgdir=0x7ffcb36d88ec "/home/gmurthy/opensource/qpid-dispatch/python", 
> fd=2) at /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:116
> #14 0x004022b0 in main (argc=5, argv=0x7ffcb36d8158) at 
> /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:360
> (gdb){noformat}
>  
> Running the test under valgrind, it seems that the pn_proactor_free is trying 
> to free already freed link endpoint. Here are two outputs from valgrind
>  
> {noformat}
> Process 3493 error: exit code 42, expected 0
> qdrouterd -c B.conf -I /home/gmurthy/opensource/qpid-dispatch/python
> /home/gmurthy/opensource/qpid-dispatch/build/system_test.dir/system_tests_distribution/DistributionTests/setUpClass/B-2.cmd
> 
> ==3493== Invalid write of size 8
> ==3493==    at 0x50E82B0: pn_link_unbound (engine.c:1202)
> ==3493==    by 0x50EB5D0: pni_transport_unbind_handles (transport.c:746)
> ==3493==    by 0x50EB665: pni_transport_unbind_channels (transport.c:761)
> ==3493==    by 0x50EB776: pn_transport_unbind (transport.c:795)
> ==3493==    by 0x50E663D: pn_connection_driver_release_connection 
> (connection_driver.c:81)
> ==3493==    by 0x50E6678: pn_connection_driver_destroy 
> (connection_driver.c:92)
> ==3493==    by 0x530A69B: 

[jira] [Commented] (DISPATCH-963) Router crash during shutdown in system_tests_distribution

[Alan Conway (JIRA)]

[ 
https://issues.apache.org/jira/browse/DISPATCH-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446290#comment-16446290
 ] 

Alan Conway commented on DISPATCH-963:
--

This may be entirely a proton bug, not a dispatch issues. The reproducer at 
PROTON-1832 shows that it is possible to crash proton by closing and 
immediately re-opening a link using the same name with a particular batching of 
events. This is definitely not illegal, it looks like proton is mixing up the 
order of the open and close.

> Router crash during shutdown in system_tests_distribution
> -
>
> Key: DISPATCH-963
> URL: https://issues.apache.org/jira/browse/DISPATCH-963
> Project: Qpid Dispatch
>  Issue Type: Improvement
>  Components: Tests
>Affects Versions: 1.0.1
>Reporter: Ganesh Murthy
>Assignee: Ganesh Murthy
>Priority: Major
>
> The router crashes during shutdown in system_tests_distribution.py
> Here is the backtrace
>  
> {noformat}
> (gdb) bt
> #0  0x7f361ca5ae40 in pn_ep_decref (endpoint=0x7f35f01c2dd0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:447
> #1  0x7f361ca5b58b in pn_ep_decref (endpoint=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:445
> #2  0x7f361ca5f588 in pni_transport_unbind_handles 
> (handles=0x7f35f00764a0, reset_state=reset_state@entry=true) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:748
> #3  0x7f361ca5f666 in pni_transport_unbind_channels (channels=0x9d1ce0) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:761
> #4  0x7f361ca5f777 in pn_transport_unbind (transport=0xa863d0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:795
> #5  0x7f361ca5a63e in pn_connection_driver_release_connection 
> (d=d@entry=0xa86248) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:81
> #6  0x7f361ca5a679 in pn_connection_driver_destroy (d=d@entry=0xa86248) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:92
> #7  0x7f361c83a69c in pconnection_final_free (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:827
> #8  0x7f361c83b3ac in pconnection_cleanup (pc=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:843
> #9  0x7f361c83db37 in pconnection_forced_shutdown (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:878
> #10 pn_proactor_free (p=0x916fd0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:1815
> #11 0x7f361cce7bb5 in qd_server_free (qd_server=0x919190) at 
> /home/gmurthy/opensource/qpid-dispatch/src/server.c:1176
> #12 0x7f361cca878e in qd_dispatch_free (qd=0x6164b0) at 
> /home/gmurthy/opensource/qpid-dispatch/src/dispatch.c:318
> #13 0x00401864 in main_process (config_path=0x7ffcb36d88e2 "B.conf", 
> python_pkgdir=0x7ffcb36d88ec "/home/gmurthy/opensource/qpid-dispatch/python", 
> fd=2) at /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:116
> #14 0x004022b0 in main (argc=5, argv=0x7ffcb36d8158) at 
> /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:360
> (gdb){noformat}
>  
> Running the test under valgrind, it seems that the pn_proactor_free is trying 
> to free already freed link endpoint. Here are two outputs from valgrind
>  
> {noformat}
> Process 3493 error: exit code 42, expected 0
> qdrouterd -c B.conf -I /home/gmurthy/opensource/qpid-dispatch/python
> /home/gmurthy/opensource/qpid-dispatch/build/system_test.dir/system_tests_distribution/DistributionTests/setUpClass/B-2.cmd
> 
> ==3493== Invalid write of size 8
> ==3493==    at 0x50E82B0: pn_link_unbound (engine.c:1202)
> ==3493==    by 0x50EB5D0: pni_transport_unbind_handles (transport.c:746)
> ==3493==    by 0x50EB665: pni_transport_unbind_channels (transport.c:761)
> ==3493==    by 0x50EB776: pn_transport_unbind (transport.c:795)
> ==3493==    by 0x50E663D: pn_connection_driver_release_connection 
> (connection_driver.c:81)
> ==3493==    by 0x50E6678: pn_connection_driver_destroy 
> (connection_driver.c:92)
> ==3493==    by 0x530A69B: pconnection_final_free (epoll.c:827)
> ==3493==    by 0x530DB36: pconnection_forced_shutdown (epoll.c:878)
> ==3493==    by 0x530DB36: pn_proactor_free (epoll.c:1815)
> ==3493==    by 0x4EA5BB4: qd_server_free (server.c:1176)
> ==3493==    by 0x4E6678D: qd_dispatch_free (dispatch.c:318)
> ==3493==    by 0x401863: main_process (main.c:116)
> ==3493==    by 0x4022AF: main (main.c:360)
> ==3493==  Address 0x15c06188 is 376 bytes inside a block of size 488 free'd
> ==3493==    at 0x4C2DD18: free (vg_replace_malloc.c:530)
> ==3493== 

[jira] [Commented] (DISPATCH-963) Router crash during shutdown in system_tests_distribution

[Alan Conway (JIRA)]

[ 
https://issues.apache.org/jira/browse/DISPATCH-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446278#comment-16446278
 ] 

Alan Conway commented on DISPATCH-963:
--

PROTON-1832 has a proton-only reproducer of the crash by creating 2 links with 
the same name.

> Router crash during shutdown in system_tests_distribution
> -
>
> Key: DISPATCH-963
> URL: https://issues.apache.org/jira/browse/DISPATCH-963
> Project: Qpid Dispatch
>  Issue Type: Improvement
>  Components: Tests
>Affects Versions: 1.0.1
>Reporter: Ganesh Murthy
>Assignee: Alan Conway
>Priority: Major
>
> The router crashes during shutdown in system_tests_distribution.py
> Here is the backtrace
>  
> {noformat}
> (gdb) bt
> #0  0x7f361ca5ae40 in pn_ep_decref (endpoint=0x7f35f01c2dd0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:447
> #1  0x7f361ca5b58b in pn_ep_decref (endpoint=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:445
> #2  0x7f361ca5f588 in pni_transport_unbind_handles 
> (handles=0x7f35f00764a0, reset_state=reset_state@entry=true) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:748
> #3  0x7f361ca5f666 in pni_transport_unbind_channels (channels=0x9d1ce0) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:761
> #4  0x7f361ca5f777 in pn_transport_unbind (transport=0xa863d0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:795
> #5  0x7f361ca5a63e in pn_connection_driver_release_connection 
> (d=d@entry=0xa86248) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:81
> #6  0x7f361ca5a679 in pn_connection_driver_destroy (d=d@entry=0xa86248) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:92
> #7  0x7f361c83a69c in pconnection_final_free (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:827
> #8  0x7f361c83b3ac in pconnection_cleanup (pc=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:843
> #9  0x7f361c83db37 in pconnection_forced_shutdown (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:878
> #10 pn_proactor_free (p=0x916fd0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:1815
> #11 0x7f361cce7bb5 in qd_server_free (qd_server=0x919190) at 
> /home/gmurthy/opensource/qpid-dispatch/src/server.c:1176
> #12 0x7f361cca878e in qd_dispatch_free (qd=0x6164b0) at 
> /home/gmurthy/opensource/qpid-dispatch/src/dispatch.c:318
> #13 0x00401864 in main_process (config_path=0x7ffcb36d88e2 "B.conf", 
> python_pkgdir=0x7ffcb36d88ec "/home/gmurthy/opensource/qpid-dispatch/python", 
> fd=2) at /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:116
> #14 0x004022b0 in main (argc=5, argv=0x7ffcb36d8158) at 
> /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:360
> (gdb){noformat}
>  
> Running the test under valgrind, it seems that the pn_proactor_free is trying 
> to free already freed link endpoint. Here are two outputs from valgrind
>  
> {noformat}
> Process 3493 error: exit code 42, expected 0
> qdrouterd -c B.conf -I /home/gmurthy/opensource/qpid-dispatch/python
> /home/gmurthy/opensource/qpid-dispatch/build/system_test.dir/system_tests_distribution/DistributionTests/setUpClass/B-2.cmd
> 
> ==3493== Invalid write of size 8
> ==3493==    at 0x50E82B0: pn_link_unbound (engine.c:1202)
> ==3493==    by 0x50EB5D0: pni_transport_unbind_handles (transport.c:746)
> ==3493==    by 0x50EB665: pni_transport_unbind_channels (transport.c:761)
> ==3493==    by 0x50EB776: pn_transport_unbind (transport.c:795)
> ==3493==    by 0x50E663D: pn_connection_driver_release_connection 
> (connection_driver.c:81)
> ==3493==    by 0x50E6678: pn_connection_driver_destroy 
> (connection_driver.c:92)
> ==3493==    by 0x530A69B: pconnection_final_free (epoll.c:827)
> ==3493==    by 0x530DB36: pconnection_forced_shutdown (epoll.c:878)
> ==3493==    by 0x530DB36: pn_proactor_free (epoll.c:1815)
> ==3493==    by 0x4EA5BB4: qd_server_free (server.c:1176)
> ==3493==    by 0x4E6678D: qd_dispatch_free (dispatch.c:318)
> ==3493==    by 0x401863: main_process (main.c:116)
> ==3493==    by 0x4022AF: main (main.c:360)
> ==3493==  Address 0x15c06188 is 376 bytes inside a block of size 488 free'd
> ==3493==    at 0x4C2DD18: free (vg_replace_malloc.c:530)
> ==3493==    by 0x50DD938: pn_class_decref (object.c:101)
> ==3493==    by 0x50EA03F: pn_event_finalize (event.c:226)
> ==3493==    by 0x50EA03F: pn_event_finalize_cast (event.c:271)
> ==3493==    by 0x50DD928: pn_class_decref (object.c:95)
> ==3493==  

[jira] [Commented] (DISPATCH-963) Router crash during shutdown in system_tests_distribution

[Chuck Rolke (JIRA)]

[ 
https://issues.apache.org/jira/browse/DISPATCH-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439506#comment-16439506
 ] 

Chuck Rolke commented on DISPATCH-963:
--

Using today's proton master (commit a80d54e6) the same tests crashes in other 
ways:
{quote}{{Core was generated by `qdrouterd -c B.conf -I 
/home/chug/git/qpid-dispatch/python'.}}
{{Program terminated with signal SIGSEGV, Segmentation fault.}}
{{[Current thread is 1 (Thread 0x7f0f57d11700 (LWP 20186))]}}
{{(gdb) p buf}}
{{$1 = (pn_buffer_t *) 0x7f0f0003}}
{{(gdb) p *buf}}
{{Cannot access memory at address 0x7f0f0003}}
{{(gdb) bt}}
{{#0  0x7f0f661d069e in pn_buffer_clear (buf=0x7f0f0003) at 
/home/chug/git/qpid-proton/c/src/core/buffer.c:257}}
{{#1  0x7f0f661d1e35 in pn_data_clear (data=0x7f0f38123ef0) at 
/home/chug/git/qpid-proton/c/src/core/codec.c:414}}
{{#2  0x7f0f661d659e in pn_data_copy (data=0x7f0f38123ef0, 
src=0x7f0f40172860) at /home/chug/git/qpid-proton/c/src/core/codec.c:1987}}
{{#3  0x7f0f664671f4 in qdr_terminus_copy (from=0x7f0f40093ca0, 
to=0x7f0f38117fd8) at 
/home/chug/git/qpid-dispatch/src/router_core/terminus.c:118}}
{{#4  0x7f0f6646e17c in CORE_link_second_attach (context=0x26089d0, 
link=0x7f0f380f0a20, source=0x7f0f40093ca0, target=0x7f0f40093ba0) at 
/home/chug/git/qpid-dispatch/src/router_node.c:1177}}
{{#5  0x7f0f66452cb3 in qdr_connection_process (conn=0x2789a60) at 
/home/chug/git/qpid-dispatch/src/router_core/connections.c:231}}
{{#6  0x7f0f6646b781 in AMQP_writable_conn_handler (type_context=0x26089d0, 
conn=0x276da60, context=0x0) at 
/home/chug/git/qpid-dispatch/src/router_node.c:167}}
{{#7  0x7f0f66431b6b in writable_handler (container=0x24810e0, 
conn=0x2763040, qd_conn=0x276da60) at 
/home/chug/git/qpid-dispatch/src/container.c:326}}
{{#8  0x7f0f66432577 in qd_container_handle_event (container=0x24810e0, 
event=0x7f0f38145710) at /home/chug/git/qpid-dispatch/src/container.c:548}}
{{#9  0x7f0f664730a2 in handle (qd_server=0x25eb820, e=0x7f0f38145710) at 
/home/chug/git/qpid-dispatch/src/server.c:940}}
{{#10 0x7f0f66473125 in thread_run (arg=0x25eb820) at 
/home/chug/git/qpid-dispatch/src/server.c:958}}
{{#11 0x7f0f65d9950b in start_thread (arg=0x7f0f57d11700) at 
pthread_create.c:465}}
{{#12 0x7f0f6505d16f in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:95}}{quote}
This seems to be very repeatable. It fails between tests 21 and 22:
{quote}{{30: test_21_linkroute_mesh_nonlocal 
(system_tests_distribution.DistributionTests) ... ok}}
{{30/46 Test #30: system_tests_distribution .***Timeout 
600.12 sec}}{quote}
 

 

 

> Router crash during shutdown in system_tests_distribution
> -
>
> Key: DISPATCH-963
> URL: https://issues.apache.org/jira/browse/DISPATCH-963
> Project: Qpid Dispatch
>  Issue Type: Improvement
>  Components: Tests
>Affects Versions: 1.0.1
>Reporter: Ganesh Murthy
>Priority: Major
>
> The router crashes during shutdown in system_tests_distribution.py
> Here is the backtrace
>  
> {noformat}
> (gdb) bt
> #0  0x7f361ca5ae40 in pn_ep_decref (endpoint=0x7f35f01c2dd0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:447
> #1  0x7f361ca5b58b in pn_ep_decref (endpoint=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:445
> #2  0x7f361ca5f588 in pni_transport_unbind_handles 
> (handles=0x7f35f00764a0, reset_state=reset_state@entry=true) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:748
> #3  0x7f361ca5f666 in pni_transport_unbind_channels (channels=0x9d1ce0) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:761
> #4  0x7f361ca5f777 in pn_transport_unbind (transport=0xa863d0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:795
> #5  0x7f361ca5a63e in pn_connection_driver_release_connection 
> (d=d@entry=0xa86248) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:81
> #6  0x7f361ca5a679 in pn_connection_driver_destroy (d=d@entry=0xa86248) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:92
> #7  0x7f361c83a69c in pconnection_final_free (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:827
> #8  0x7f361c83b3ac in pconnection_cleanup (pc=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:843
> #9  0x7f361c83db37 in pconnection_forced_shutdown (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:878
> #10 pn_proactor_free (p=0x916fd0) at 
> 

[jira] [Commented] (DISPATCH-963) Router crash during shutdown in system_tests_distribution

[Ganesh Murthy (JIRA)]

[ 
https://issues.apache.org/jira/browse/DISPATCH-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434376#comment-16434376
 ] 

Ganesh Murthy commented on DISPATCH-963:


The failing test output can be seen here - 
http://rhm-x3550-02.rhm.lab.eng.bos.redhat.com:8080/job/dispatch-master/OS=rhel7/323/testReport/projectroot/tests/system_tests_distribution/

> Router crash during shutdown in system_tests_distribution
> -
>
> Key: DISPATCH-963
> URL: https://issues.apache.org/jira/browse/DISPATCH-963
> Project: Qpid Dispatch
>  Issue Type: Improvement
>  Components: Tests
>Affects Versions: 1.0.1
>Reporter: Ganesh Murthy
>Priority: Major
>
> The router crashes during shutdown in system_tests_distribution.py
> Here is the backtrace
>  
> {noformat}
> (gdb) bt
> #0  0x7f361ca5ae40 in pn_ep_decref (endpoint=0x7f35f01c2dd0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:447
> #1  0x7f361ca5b58b in pn_ep_decref (endpoint=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:445
> #2  0x7f361ca5f588 in pni_transport_unbind_handles 
> (handles=0x7f35f00764a0, reset_state=reset_state@entry=true) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:748
> #3  0x7f361ca5f666 in pni_transport_unbind_channels (channels=0x9d1ce0) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:761
> #4  0x7f361ca5f777 in pn_transport_unbind (transport=0xa863d0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:795
> #5  0x7f361ca5a63e in pn_connection_driver_release_connection 
> (d=d@entry=0xa86248) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:81
> #6  0x7f361ca5a679 in pn_connection_driver_destroy (d=d@entry=0xa86248) 
> at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:92
> #7  0x7f361c83a69c in pconnection_final_free (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:827
> #8  0x7f361c83b3ac in pconnection_cleanup (pc=) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:843
> #9  0x7f361c83db37 in pconnection_forced_shutdown (pc=0xa85ca0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:878
> #10 pn_proactor_free (p=0x916fd0) at 
> /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:1815
> #11 0x7f361cce7bb5 in qd_server_free (qd_server=0x919190) at 
> /home/gmurthy/opensource/qpid-dispatch/src/server.c:1176
> #12 0x7f361cca878e in qd_dispatch_free (qd=0x6164b0) at 
> /home/gmurthy/opensource/qpid-dispatch/src/dispatch.c:318
> #13 0x00401864 in main_process (config_path=0x7ffcb36d88e2 "B.conf", 
> python_pkgdir=0x7ffcb36d88ec "/home/gmurthy/opensource/qpid-dispatch/python", 
> fd=2) at /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:116
> #14 0x004022b0 in main (argc=5, argv=0x7ffcb36d8158) at 
> /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:360
> (gdb){noformat}
>  
> Running the test under valgrind, it seems that the pn_proactor_free is trying 
> to free already freed link endpoint. Here are two outputs from valgrind
>  
> {noformat}
> Process 3493 error: exit code 42, expected 0
> qdrouterd -c B.conf -I /home/gmurthy/opensource/qpid-dispatch/python
> /home/gmurthy/opensource/qpid-dispatch/build/system_test.dir/system_tests_distribution/DistributionTests/setUpClass/B-2.cmd
> 
> ==3493== Invalid write of size 8
> ==3493==    at 0x50E82B0: pn_link_unbound (engine.c:1202)
> ==3493==    by 0x50EB5D0: pni_transport_unbind_handles (transport.c:746)
> ==3493==    by 0x50EB665: pni_transport_unbind_channels (transport.c:761)
> ==3493==    by 0x50EB776: pn_transport_unbind (transport.c:795)
> ==3493==    by 0x50E663D: pn_connection_driver_release_connection 
> (connection_driver.c:81)
> ==3493==    by 0x50E6678: pn_connection_driver_destroy 
> (connection_driver.c:92)
> ==3493==    by 0x530A69B: pconnection_final_free (epoll.c:827)
> ==3493==    by 0x530DB36: pconnection_forced_shutdown (epoll.c:878)
> ==3493==    by 0x530DB36: pn_proactor_free (epoll.c:1815)
> ==3493==    by 0x4EA5BB4: qd_server_free (server.c:1176)
> ==3493==    by 0x4E6678D: qd_dispatch_free (dispatch.c:318)
> ==3493==    by 0x401863: main_process (main.c:116)
> ==3493==    by 0x4022AF: main (main.c:360)
> ==3493==  Address 0x15c06188 is 376 bytes inside a block of size 488 free'd
> ==3493==    at 0x4C2DD18: free (vg_replace_malloc.c:530)
> ==3493==    by 0x50DD938: pn_class_decref (object.c:101)
> ==3493==    by 0x50EA03F: pn_event_finalize (event.c:226)
> ==3493==    by 0x50EA03F: pn_event_finalize_cast (event.c:271)
> ==3493==