[
https://issues.apache.org/jira/browse/DISPATCH-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ken Giusti updated DISPATCH-1989:
---------------------------------
Fix Version/s: (was: 1.17.0)
> [libuv] ASAN use after free in connection close
> -----------------------------------------------
>
> Key: DISPATCH-1989
> URL: https://issues.apache.org/jira/browse/DISPATCH-1989
> Project: Qpid Dispatch
> Issue Type: Bug
> Components: Router Node, Routing Engine
> Affects Versions: 1.13.0, 1.14.0
> Reporter: Alex Ward
> Assignee: Clifford Jansen
> Priority: Minor
> Labels: libuv
> Attachments: mtrace.3.9.txt, mtrace.txt, qdrouterd.txt
>
>
> While we dig into the issue below I thought it worth reaching out to see if
> this is a familiar issue or in code that has changed much since 1.13 so may
> have likely already been fixed.
> We have a freebsd product that is using qdrouterd. We have had some crashes
> during closing of connections to a broker so built a qdrouterd with asan on.
> This is hitting the use-after-free issue in uv__run_closing_handles below
> fairly reliably.
> It appears from first inspection that uv__finish_close is trying to remove a
> handle from the handle_queue at line 300 but the handle_queue has reference
> into a pconnection_t that has already been freed.
> Is this an area of code that has had issues in the past? Is this likely to
> have been fixed in the latest release? Any suggestions on where to add
> printfs etc to identify the culprit?
>
>
> {{243static void uv__finish_close(uv_handle_t* handle) {}}
> {{244 uv_signal_t* sh;}}
> {{245}}
> {{246 /* Note: while the handle is in the UV_HANDLE_CLOSING state now, it's
> still}}
> {{247 * possible for it to be active in the sense that uv__is_active()
> returns}}
> {{248 * true.}}
> {{249 *}}
> {{250 * A good example is when the user calls uv_shutdown(), immediately
> followed}}
> {{251 * by uv_close(). The handle is considered active at this point because
> the}}
> {{252 * completion of the shutdown req is still pending.}}
> {{253 */}}
> {{254 assert(handle->flags & UV_HANDLE_CLOSING);}}
> {{255 assert(!(handle->flags & UV_HANDLE_CLOSED));}}
> {{256 handle->flags |= UV_HANDLE_CLOSED;}}
> {{257}}
> {{258 switch (handle->type) {}}
> {{259 case UV_PREPARE:}}
> {{260 case UV_CHECK:}}
> {{261 case UV_IDLE:}}
> {{262 case UV_ASYNC:}}
> {{263 case UV_TIMER:}}
> {{264 case UV_PROCESS:}}
> {{265 case UV_FS_EVENT:}}
> {{266 case UV_FS_POLL:}}
> {{267 case UV_POLL:}}
> {{268 break;}}
> {{269}}
> {{270 case UV_SIGNAL:}}
> {{271 /* If there are any caught signals "trapped" in the signal pipe,}}
> {{272 * we can't call the close callback yet. Reinserting the handle}}
> {{273 * into the closing queue makes the event loop spin but that's}}
> {{274 * okay because we only need to deliver the pending events.}}
> {{275 */}}
> {{276 sh = (uv_signal_t*) handle;}}
> {{277 if (sh->caught_signals > sh->dispatched_signals) {}}
> {{278 handle->flags ^= UV_HANDLE_CLOSED;}}
> {{279 uv__make_close_pending(handle); /* Back into the queue. */}}
> {{280 return;}}
> {{281 }}}
> {{282 break;}}
> {{283}}
> {{284 case UV_NAMED_PIPE:}}
> {{285 case UV_TCP:}}
> {{286 case UV_TTY:}}
> {{287 uv__stream_destroy((uv_stream_t*)handle);}}
> {{288 break;}}
> {{289}}
> {{290 case UV_UDP:}}
> {{291 uv__udp_finish_close((uv_udp_t*)handle);}}
> {{292 break;}}
> {{293}}
> {{294 default:}}
> {{295 assert(0);}}
> {{296 break;}}
> {{297 }}}
> {{298}}
> {{299 uv__handle_unref(handle);}}
> {{300 QUEUE_REMOVE(&handle->handle_queue);}}
> {{301}}
> {{302 if (handle->close_cb) {}}
> {{303 handle->close_cb(handle);}}
> {{304 }}}
> {{305}}}
>
> Here's the asan report
> {{10 ==13358==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x61d0002ec048 at pc 0x0008006a7f84 bp 0x7fffffffe8c0 sp 0x7fffffffe8b8}}
> {{11 WRITE of size 8 at 0x61d0002ec048 thread T0}}
> {{12 #0 0x8006a7f83 in uv__finish_close
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/libuv/bedrock/internal/x86_64.asan.sim/build_configure/../../../../distro/src/unix/core.c:300:3}}
> {{13 #1 0x8006a38db in uv__run_closing_handles
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/libuv/bedrock/internal/x86_64.asan.sim/build_configure/../../../../distro/src/unix/core.c:317:5}}
> {{14 #2 0x8006a3463 in uv_run
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/libuv/bedrock/internal/x86_64.asan.sim/build_configure/../../../../distro/src/unix/core.c:387:5}}
> {{15 #3 0x800b378b5 in leader_lead_lh
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-proton/distro/c/src/proactor/libuv.c:1039:5}}
> {{16 #4 0x800b37a5c in pn_proactor_wait
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-proton/distro/c/src/proactor/libuv.c:1072:15}}
> {{17 #5 0x8005b55c0 in thread_run
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-dispatch/distro/src/server.c:1051:36}}
> {{18 #6 0x8005b524b in qd_server_run
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-dispatch/distro/src/server.c:1453:5}}
> {{19 #7 0x2b8e20 in main_process
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-dispatch/distro/router/src/main.c:130:5}}
> {{20 #8 0x2b7ef2 in main
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-dispatch/distro/router/src/main.c:384:9}}
> {{21 #9 0x2b780a in _start
> /x/eng/bbrtp2/daemon/DOT/dev/daemon_base/bsd99/bedrock/private/x86_64/bsd_libcrt_files/lib/csu/amd64/../../../../../../../src/lib/csu/amd64/crt1.c:74:7}}
> {{22 #10 0x8002e4fff (<unknown module>)}}
> {{23 }}
> {{24 0x61d0002ec048 is located 1992 bytes inside of 2408-byte region
> [0x61d0002eb880,0x61d0002ec1e8)}}
> {{25 freed by thread T5 here:}}
> {{26 #0 0x28ca2d in free
> /x/eng/bbrtp2/daemon/DOT/dev/daemon_base/third_party/open_source/llvm/distro/compiler-rt/lib/asan/asan_malloc_linux.cpp:123:3}}
> {{27 #1 0x800b3aec7 in pconnection_free
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-proton/distro/c/src/proactor/libuv.c:363:3}}
> {{28 #2 0x800b3ae05 in on_close_pconnection_final
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-proton/distro/c/src/proactor/libuv.c:421:3}}
> {{29 #3 0x8006a7e28 in uv__finish_close
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/libuv/bedrock/internal/x86_64.asan.sim/build_configure/../../../../distro/src/unix/core.c:303:5}}
> {{30 #4 0x8006a38db in uv__run_closing_handles
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/libuv/bedrock/internal/x86_64.asan.sim/build_configure/../../../../distro/src/unix/core.c:317:5}}
> {{31 #5 0x8006a3463 in uv_run
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/libuv/bedrock/internal/x86_64.asan.sim/build_configure/../../../../distro/src/unix/core.c:387:5}}
> {{32 #6 0x800b378b5 in leader_lead_lh
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-proton/distro/c/src/proactor/libuv.c:1039:5}}
> {{33 #7 0x800b37a5c in pn_proactor_wait
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-proton/distro/c/src/proactor/libuv.c:1072:15}}
> {{34 #8 0x8005b55c0 in thread_run
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-dispatch/distro/src/server.c:1051:36}}
> {{35 #9 0x8005be392 in _thread_init
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-dispatch/distro/src/posix/threading.c:172:12}}
> {{36 #10 0x800eb20f4 in thread_start
> /x/eng/bbrtp2/daemon/DOT/dev/daemon_base/bsd99/bedrock/private/x86_64/bsd_libthr/lib/libthr/../../../../../../src/lib/libthr/thread/thr_create.c:300:16}}
> {{37 }}
> {{38 previously allocated by thread T5 here:}}
> {{39 #0 0x28cd12 in calloc
> /x/eng/bbrtp2/daemon/DOT/dev/daemon_base/third_party/open_source/llvm/distro/compiler-rt/lib/asan/asan_malloc_linux.cpp:154:3}}
> {{40 #1 0x800b38288 in pconnection
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-proton/distro/c/src/proactor/libuv.c:339:39}}
> {{41 #2 0x800b381e0 in pn_proactor_connect2
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-proton/distro/c/src/proactor/libuv.c:1168:23}}
> {{42 #3 0x800b4658d in pn_proactor_connect
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-proton/distro/c/src/proactor/proactor-internal.c:97:3}}
> {{43 #4 0x8005b8156 in try_open_lh
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-dispatch/distro/src/server.c:1169:5}}
> {{44 #5 0x8005b641e in try_open_cb
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-dispatch/distro/src/server.c:1278:14}}
> {{45 #6 0x8005bf6ad in qd_timer_visit
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-dispatch/distro/src/timer.c:201:9}}
> {{46 #7 0x8005b6ef8 (/usr/lib/libqpid-dispatch.so
> (distro/router/src/main.c:43)}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
