[
https://issues.apache.org/jira/browse/DISPATCH-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17369701#comment-17369701
]
Ken Giusti commented on DISPATCH-2185:
--------------------------------------
More data - Threadsanitizer appears to confirm the state reference to a message:
{{69: WARNING: ThreadSanitizer: data race (pid=3406) }}
{{69: Read of size 8 at 0x7b4800022620 by thread T3 (mutexes: write M1507): }}
{{69: #0 qd_message_free
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/message.c:1037
(libqpid-dispatch.so+0x871b0) }}
{{69: #1 qdr_do_message_to_addr_free
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/router_core/router_core_thread.c:102
(libqpid-dispatch.so+0xcdff6) }}
{{69: #2 qdr_general_handler
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/router_core/router_core.c:942
(libqpid-dispatch.so+0xc78cb) }}
{{69: #3 qd_timer_visit
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/timer.c:317
(libqpid-dispatch.so+0xf878d) }}
{{69: #4 handle
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/server.c:1006
(libqpid-dispatch.so+0xf279e) }}
{{69: #5 thread_run
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/server.c:1121
(libqpid-dispatch.so+0xf6cb5) }}
{{69: #6 _thread_init
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/posix/threading.c:172
(libqpid-dispatch.so+0x97562) }}
{{69: }}
{{69: Previous write of size 8 at 0x7b4800022620 by thread T2: }}
{{69: #0 qd_message_stream_data_release
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/message.c:2680
(libqpid-dispatch.so+0x8876f) }}
{{69: #1 qdr_http1_out_data_fifo_cleanup
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/adaptors/http1/http1_adaptor.c:152
(libqpid-dispatch.so+0x43b51) }}
{{69: #2 _server_request_free
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/adaptors/http1/http1_server.c:1647
(libqpid-dispatch.so+0x4f1fd) }}
{{69: #3 _server_request_free
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/adaptors/http1/http1_server.c:1645
(libqpid-dispatch.so+0x4f853) }}
{{69: #4 _process_request
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/adaptors/http1/http1_server.c:734
(libqpid-dispatch.so+0x4f853) }}
{{69: #5 _handle_connection_events
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/adaptors/http1/http1_server.c:659
(libqpid-dispatch.so+0x4ff65) }}
{{69: #6 handle_event_with_context
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/server.c:802
(libqpid-dispatch.so+0xf284d) }}
{{69: #7 do_handle_raw_connection_event
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/server.c:808
(libqpid-dispatch.so+0xf284d) }}
{{69: #8 handle
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/server.c:1089
(libqpid-dispatch.so+0xf284d) }}
{{69: #9 thread_run
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/server.c:1121
(libqpid-dispatch.so+0xf6cb5) }}
{{69: #10 _thread_init
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/posix/threading.c:172
(libqpid-dispatch.so+0x97562) }}
{{69: }}
{{69: Location is heap block of size 384 at 0x7b4800022500 allocated by thread
T1: }}
{{69: #0 posix_memalign <null> (libtsan.so.0+0x32a23) }}
{{69: #1 qd_alloc
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/alloc_pool.c:396
(libqpid-dispatch.so+0x5d203) }}
{{69: #2 new_qd_message_t
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/message.c:91
(libqpid-dispatch.so+0x822c1) }}
{{69: #3 qd_message_copy
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/message.c:1099
(libqpid-dispatch.so+0x847e8) }}
{{69: #4 qdr_forward_new_delivery_CT
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/router_core/forwarder.c:153
(libqpid-dispatch.so+0xbe81b) }}
{{69: #5 qdr_forward_balanced_CT
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/router_core/forwarder.c:905
(libqpid-dispatch.so+0xbfead) }}
{{69: #6 qdr_forward_message_CT
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/router_core/forwarder.c:1120
(libqpid-dispatch.so+0xc32a0) }}
{{69: #7 qdr_link_forward_CT
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/router_core/transfer.c:572
(libqpid-dispatch.so+0xd6cd3) }}
{{69: #8 qdr_link_deliver_CT
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/router_core/transfer.c:862
(libqpid-dispatch.so+0xd8222) }}
{{69: #9 router_core_thread
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/router_core/router_core_thread.c:239
(libqpid-dispatch.so+0xced7b) }}
{{69: #10 _thread_init
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/posix/threading.c:172
(libqpid-dispatch.so+0x97562) }}
{{69: }}
{{69: Mutex M1507 (0x7b1000006280) created at: }}
{{69: #0 pthread_mutex_init <null> (libtsan.so.0+0x49603) }}
{{69: #1 sys_mutex
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/posix/threading.c:43
(libqpid-dispatch.so+0x975bc) }}
{{69: #2 qd_message
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/message.c:1005
(libqpid-dispatch.so+0x82867) }}
{{69: #3 _client_rx_headers_done_cb
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/adaptors/http1/http1_client.c:849
(libqpid-dispatch.so+0x454e8) }}
{{69: #4 process_headers_done
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/adaptors/http1/http1_codec.c:837
(libqpid-dispatch.so+0x411d8) }}
{{69: #5 parse_header
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/adaptors/http1/http1_codec.c:917
(libqpid-dispatch.so+0x411d8) }}
{{69: #6 decode_incoming
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/adaptors/http1/http1_codec.c:1294
(libqpid-dispatch.so+0x411d8) }}
{{69: #7 h1_codec_connection_rx_data
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/adaptors/http1/http1_codec.c:1339
(libqpid-dispatch.so+0x411d8) }}
{{69: #8 _handle_conn_read_event
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/adaptors/http1/http1_client.c:404
(libqpid-dispatch.so+0x45c89) }}
{{69: #9 _handle_connection_events
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/adaptors/http1/http1_client.c:511
(libqpid-dispatch.so+0x4827c) }}
{{69: #10 handle_event_with_context
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/server.c:802
(libqpid-dispatch.so+0xf284d) }}
{{69: #11 do_handle_raw_connection_event
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/server.c:808
(libqpid-dispatch.so+0xf284d) }}
{{69: #12 handle
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/server.c:1089
(libqpid-dispatch.so+0xf284d) }}
{{69: #13 thread_run
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/server.c:1121
(libqpid-dispatch.so+0xf6cb5) }}
{{69: #14 qd_server_run
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/src/server.c:1501
(libqpid-dispatch.so+0xf6f18) }}
{{69: #15 main_process
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/router/src/main.c:115
(qdrouterd+0x40287c) }}
{{69: #16 main
/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/router/src/main.c:369
(qdrouterd+0x4024fc) }}
> HTTP/1.x crash on message content sys_mutex_lock fail
> -----------------------------------------------------
>
> Key: DISPATCH-2185
> URL: https://issues.apache.org/jira/browse/DISPATCH-2185
> Project: Qpid Dispatch
> Issue Type: Bug
> Components: Protocol Adaptors
> Affects Versions: 1.16.0
> Reporter: Ken Giusti
> Assignee: Ken Giusti
> Priority: Critical
> Fix For: 1.17.0
>
>
> Hit during router shutdown after Http1AdaptorBadEndpointsTest:
> {{72: ======================================================================
> }}
> {{72: ERROR: tearDownClass
> (system_tests_http1_adaptor.Http1AdaptorBadEndpointsTest) }}
> {{72: ----------------------------------------------------------------------
> }}
> {{72: Traceback (most recent call last): }}
> {{72: File "/home/kgiusti/work/dispatch/qpid-dispatch/tests/system_test.py",
> line 865, in tearDownClass }}
> {{72: cls.tester.teardown() }}
> {{72: File "/home/kgiusti/work/dispatch/qpid-dispatch/tests/system_test.py",
> line 808, in teardown }}
> {{72: raise RuntimeError("Errors during teardown: \n\n%s" %
> "\n\n".join([str(e) for e in errors])) }}
> {{72: RuntimeError: Errors during teardown: }}
> {{72: }}
> {{72: Process 1257318 error: exit code -6, expected -1 }}
> {{72: qdrouterd -c TestBadEndpoints.conf -I
> /home/kgiusti/work/dispatch/qpid-dispatch/python }}
> {{72:
> /home/kgiusti/work/dispatch/qpid-dispatch/BUILD/tests/system_test.dir/system_tests_http1_adaptor/Http1AdaptorBadEndpointsTest/setUpClass/TestBadEndpoints-1.cmd
> }}
> {{72: >>>> }}
> {{72: qdrouterd:
> /home/kgiusti/work/dispatch/qpid-dispatch/src/posix/threading.c:58:
> sys_mutex_lock: Assertion `result == 0' failed. }}
> {{72: <<<< }}
>
> GDB thread backtrace:
> {{Thread 5 (Thread 0x7f7f9615f700 (LWP 1257324)): }}
> {{#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x9c09ac) at
> ../sysdeps/nptl/futex-internal.h:183 }}
> {{#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x9c0a40,
> cond=0x9c0980) at pthread_cond_wait.c:508 }}
> {{#2 __pthread_cond_wait (cond=0x9c0980, mutex=0x9c0a40) at
> pthread_cond_wait.c:638 }}
> {{#3 0x00007f7fa47a7a89 in sys_cond_wait (cond=<optimized out>,
> held_mutex=<optimized out>) at
> /home/kgiusti/work/dispatch/qpid-dispatch/src/posix/threading.c:92 }}
> {{#4 0x00007f7fa47c4dd7 in router_core_thread (arg=0x9c04c0) at
> /home/kgiusti/work/dispatch/qpid-dispatch/src/router_core/router_core_thread.c:221
> }}
> {{#5 0x00007f7fa46b5432 in start_thread (arg=<optimized out>) at
> pthread_create.c:477 }}
> {{#6 0x00007f7fa41f96d3 in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 }}
> {{Thread 4 (Thread 0x7f7f9578e700 (LWP 1257331)): }}
> {{#0 0x00007f7fa41f9a1e in epoll_wait (epfd=4, events=0x89c9f0, maxevents=16,
> timeout=timeout@entry=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 }}
> {{#1 0x00007f7fa4735e37 in poller_do_epoll (can_block=true, ts=0x8eb2a0,
> p=0x95e4a0) at
> /home/kgiusti/work/proton/qpid-proton/c/src/proactor/epoll.c:2496 }}
> {{#2 next_event_batch (p=0x95e4a0, can_block=true) at
> /home/kgiusti/work/proton/qpid-proton/c/src/proactor/epoll.c:2442 }}
> {{#3 0x00007f7fa47db54f in thread_run (arg=0x8d89f0) at
> /home/kgiusti/work/dispatch/qpid-dispatch/src/server.c:1106 }}
> {{#4 0x00007f7fa46b5432 in start_thread (arg=<optimized out>) at
> pthread_create.c:477 }}
> {{#5 0x00007f7fa41f96d3 in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 }}
> {{Thread 3 (Thread 0x7f7f94dfd700 (LWP 1257332)): }}
> {{#0 0x00007f7fa470cb97 in pn_transport_pending (transport=<optimized out>)
> at /home/kgiusti/work/proton/qpid-proton/c/src/core/transport.c:2838 }}
> {{#1 0x00007f7fa4707692 in pn_connection_driver_write_buffer
> (d=d@entry=0xeddc10) at
> /home/kgiusti/work/proton/qpid-proton/c/src/core/connection_driver.c:112 }}
> {{#2 0x00007f7fa4731030 in ensure_wbuf (pc=0xeddad0) at
> /home/kgiusti/work/proton/qpid-proton/c/src/proactor/epoll.c:1076 }}
> {{#3 write_flush (pc=pc@entry=0xeddad0) at
> /home/kgiusti/work/proton/qpid-proton/c/src/proactor/epoll.c:1076 }}
> {{#4 0x00007f7fa47325be in pconnection_batch_next (batch=0xeddc08) at
> /home/kgiusti/work/proton/qpid-proton/c/src/proactor/epoll.c:892 }}
> {{#5 0x00007f7fa47db55a in thread_run (arg=0x8d89f0) at
> /home/kgiusti/work/dispatch/qpid-dispatch/src/server.c:1111 }}
> {{#6 0x00007f7fa46b5432 in start_thread (arg=<optimized out>) at
> pthread_create.c:477 }}
> {{#7 0x00007f7fa41f96d3 in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 }}
> {{Thread 2 (Thread 0x7f7f9446c700 (LWP 1257333)): }}
> {{#0 futex_wait_cancelable (private=0, expected=0, futex_word=0xa3f734) at
> ../sysdeps/nptl/futex-internal.h:183 }}
> {{#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xa3f6e0,
> cond=0xa3f708) at pthread_cond_wait.c:508 }}
> {{#2 __pthread_cond_wait (cond=cond@entry=0xa3f708,
> mutex=mutex@entry=0xa3f6e0) at pthread_cond_wait.c:638 }}
> {{#3 0x00007f7fa4735ccb in suspend (ts=0xa3f6e0, p=0x95e4a0) at
> /home/kgiusti/work/proton/qpid-proton/c/src/proactor/epoll.c:393 }}
> {{#4 next_event_batch (p=0x95e4a0, can_block=true) at
> /home/kgiusti/work/proton/qpid-proton/c/src/proactor/epoll.c:2459 }}
> {{#5 0x00007f7fa47db54f in thread_run (arg=0x8d89f0) at
> /home/kgiusti/work/dispatch/qpid-dispatch/src/server.c:1106 }}
> {{#6 0x00007f7fa46b5432 in start_thread (arg=<optimized out>) at
> pthread_create.c:477 }}
> {{#7 0x00007f7fa41f96d3 in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 }}
> {{Thread 1 (Thread 0x7f7fa3902300 (LWP 1257318)): }}
> {{#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 }}
> {{#1 0x00007f7fa411d895 in __GI_abort () at abort.c:79 }}
> {{#2 0x00007f7fa411d769 in __assert_fail_base (fmt=0x7f7fa428b0d8 "%s%s%s:%u:
> %s%sAssertion `%s' failed.\n%n", assertion=0x7f7fa47ee88b "result == 0",
> file=0x7f7fa47ee840 "/home/kgiusti/work/dispatch/qpid-dispat\}}
> {{ch/src/posix/threading.c", line=58, function=<optimized out>) at
> assert.c:92 }}
> {{#3 0x00007f7fa412ce86 in __GI___assert_fail
> (assertion=assertion@entry=0x7f7fa47ee88b "result == 0",
> file=file@entry=0x7f7fa47ee840
> "/home/kgiusti/work/dispatch/qpid-dispatch/src/posix/threading.c",
> line=line@\}}
> {{entry=58, function=function@entry=0x7f7fa47ee998 <__PRETTY_FUNCTION__.10>
> "sys_mutex_lock") at assert.c:101 }}
> {{#4 0x00007f7fa47a79d1 in sys_mutex_lock (mutex=<optimized out>) at
> /home/kgiusti/work/dispatch/qpid-dispatch/src/posix/threading.c:58 }}
> {{#5 0x00007f7fa479eb1c in qd_message_stream_data_release
> (stream_data=0xcf0448) at
> /home/kgiusti/work/dispatch/qpid-dispatch/src/message.c:2640 }}
> {{#6 0x00007f7fa477c4a5 in qdr_http1_out_data_fifo_cleanup
> (out_data=out_data@entry=0xd96bb0) at
> /home/kgiusti/work/dispatch/qpid-dispatch/src/adaptors/http1/http1_adaptor.c:152
> }}
> {{#7 0x00007f7fa4782802 in _server_request_free (hreq=hreq@entry=0xd96b48) at
> /home/kgiusti/work/dispatch/qpid-dispatch/src/adaptors/http1/http1_server.c:1647
> }}
> {{#8 0x00007f7fa47829b7 in _server_request_free (hreq=0xd96b48) at
> /home/kgiusti/work/dispatch/qpid-dispatch/src/adaptors/http1/http1_server.c:1645
> }}
> {{#9 _process_request (hreq=0xd96b48) at
> /home/kgiusti/work/dispatch/qpid-dispatch/src/adaptors/http1/http1_server.c:734
> }}
> {{#10 0x00007f7fa4782e43 in _handle_connection_events (e=<optimized out>,
> qd_server=<optimized out>, context=0xa4ba88) at
> /home/kgiusti/work/dispatch/qpid-dispatch/src/adaptors/http1/http1_server.c:659
> }}
> {{#11 0x00007f7fa47d9311 in handle_event_with_context (context=<optimized
> out>, qd_server=<optimized out>, e=<optimized out>) at
> /home/kgiusti/work/dispatch/qpid-dispatch/src/server.c:802 }}
> {{#12 do_handle_raw_connection_event (qd_server=<optimized out>, e=<optimized
> out>) at /home/kgiusti/work/dispatch/qpid-dispatch/src/server.c:808 }}
> #13 handle (qd_server=qd_server@entry=0x8d89f0, e=e@entry=0xda4580,
> pn_conn=pn_conn@entry=0x0, ctx=ctx@entry=0x0) at
> /home/kgiusti/work/dispatch/qpid-dispatch/src/server.c:1089
> #14 0x00007f7fa47db668 in thread_run (arg=arg@entry=0x8d89f0) at
> /home/kgiusti/work/dispatch/qpid-dispatch/src/server.c:1121
> #15 0x00007f7fa47db830 in qd_server_run (qd=<optimized out>) at
> /home/kgiusti/work/dispatch/qpid-dispatch/src/server.c:1501
> #16 0x00000000004026a8 in main_process (config_path=0x7ffd8647ba86
> "TestBadEndpoints.conf", python_pkgdir=<optimized out>, test_hooks=<optimized
> out>, fd=2) at /home/kgiusti/work/dispatch/qpid-dispatch/router/s\
> rc/main.c:115
> #17 0x000000000040240b in main (argc=5, argv=0x7ffd8647a268) at
> /home/kgiusti/work/dispatch/qpid-dispatch/router/src/main.c:369
>
> {{Message content has apparently been freed:}}
>
> (gdb) p *stream_data->owning_message->content
> $9 = {
> lock = 0xef62c0,
> ref_count = 0,<<<<<<----------------------
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
