[jira] [Assigned] (DISPATCH-1989) [libuv] ASAN use after free in connection close
[
https://issues.apache.org/jira/browse/DISPATCH-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ken Giusti reassigned DISPATCH-1989:
Assignee: Clifford Jansen (was: Ken Giusti)
> [libuv] ASAN use after free in connection close
> ---
>
> Key: DISPATCH-1989
> URL: https://issues.apache.org/jira/browse/DISPATCH-1989
> Project: Qpid Dispatch
> Issue Type: Bug
> Components: Router Node, Routing Engine
>Affects Versions: 1.13.0, 1.14.0
>Reporter: Alex Ward
>Assignee: Clifford Jansen
>Priority: Minor
> Labels: libuv
> Fix For: 1.17.0
>
> Attachments: mtrace.3.9.txt, mtrace.txt, qdrouterd.txt
>
>
> While we dig into the issue below I thought it worth reaching out to see if
> this is a familiar issue or in code that has changed much since 1.13 so may
> have likely already been fixed.
> We have a freebsd product that is using qdrouterd. We have had some crashes
> during closing of connections to a broker so built a qdrouterd with asan on.
> This is hitting the use-after-free issue in uv__run_closing_handles below
> fairly reliably.
> It appears from first inspection that uv__finish_close is trying to remove a
> handle from the handle_queue at line 300 but the handle_queue has reference
> into a pconnection_t that has already been freed.
> Is this an area of code that has had issues in the past? Is this likely to
> have been fixed in the latest release? Any suggestions on where to add
> printfs etc to identify the culprit?
>
>
> {{243static void uv__finish_close(uv_handle_t* handle) {}}
> {{244 uv_signal_t* sh;}}
> {{245}}
> {{246 /* Note: while the handle is in the UV_HANDLE_CLOSING state now, it's
> still}}
> {{247 * possible for it to be active in the sense that uv__is_active()
> returns}}
> {{248 * true.}}
> {{249 *}}
> {{250 * A good example is when the user calls uv_shutdown(), immediately
> followed}}
> {{251 * by uv_close(). The handle is considered active at this point because
> the}}
> {{252 * completion of the shutdown req is still pending.}}
> {{253 */}}
> {{254 assert(handle->flags & UV_HANDLE_CLOSING);}}
> {{255 assert(!(handle->flags & UV_HANDLE_CLOSED));}}
> {{256 handle->flags |= UV_HANDLE_CLOSED;}}
> {{257}}
> {{258 switch (handle->type) {}}
> {{259 case UV_PREPARE:}}
> {{260 case UV_CHECK:}}
> {{261 case UV_IDLE:}}
> {{262 case UV_ASYNC:}}
> {{263 case UV_TIMER:}}
> {{264 case UV_PROCESS:}}
> {{265 case UV_FS_EVENT:}}
> {{266 case UV_FS_POLL:}}
> {{267 case UV_POLL:}}
> {{268 break;}}
> {{269}}
> {{270 case UV_SIGNAL:}}
> {{271 /* If there are any caught signals "trapped" in the signal pipe,}}
> {{272 * we can't call the close callback yet. Reinserting the handle}}
> {{273 * into the closing queue makes the event loop spin but that's}}
> {{274 * okay because we only need to deliver the pending events.}}
> {{275 */}}
> {{276 sh = (uv_signal_t*) handle;}}
> {{277 if (sh->caught_signals > sh->dispatched_signals) {}}
> {{278 handle->flags ^= UV_HANDLE_CLOSED;}}
> {{279 uv__make_close_pending(handle); /* Back into the queue. */}}
> {{280 return;}}
> {{281 }}}
> {{282 break;}}
> {{283}}
> {{284 case UV_NAMED_PIPE:}}
> {{285 case UV_TCP:}}
> {{286 case UV_TTY:}}
> {{287 uv__stream_destroy((uv_stream_t*)handle);}}
> {{288 break;}}
> {{289}}
> {{290 case UV_UDP:}}
> {{291 uv__udp_finish_close((uv_udp_t*)handle);}}
> {{292 break;}}
> {{293}}
> {{294 default:}}
> {{295 assert(0);}}
> {{296 break;}}
> {{297 }}}
> {{298}}
> {{299 uv__handle_unref(handle);}}
> {{300 QUEUE_REMOVE(&handle->handle_queue);}}
> {{301}}
> {{302 if (handle->close_cb) {}}
> {{303 handle->close_cb(handle);}}
> {{304 }}}
> {{305}}}
>
> Here's the asan report
> {{10 ==13358==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x61d0002ec048 at pc 0x0008006a7f84 bp 0x7fffe8c0 sp 0x7fffe8b8}}
> {{11 WRITE of size 8 at 0x61d0002ec048 thread T0}}
> {{12 #0 0x8006a7f83 in uv__finish_close
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/libuv/bedrock/internal/x86_64.asan.sim/build_configure/../../../../distro/src/unix/core.c:300:3}}
> {{13 #1 0x8006a38db in uv__run_closing_handles
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/libuv/bedrock/internal/x86_64.asan.sim/build_configure/../../../../distro/src/unix/core.c:317:5}}
> {{14 #2 0x8006a3463 in uv_run
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/libuv/bedrock/internal/x86_64.asan.sim/build_configure/../../../../distro/src/unix/core.c:387:5}}
> {{15 #3 0x800b378b5 in leader_lead_lh
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-proton/distro/c/src/proactor/libuv.c:1039:5}}
> {{16 #4 0x800b37a5c in pn_proactor_wait
> /x/eng/bbrtp3/users/alward/resilasan
[jira] [Assigned] (DISPATCH-1989) [libuv] ASAN use after free in connection close
[
https://issues.apache.org/jira/browse/DISPATCH-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ken Giusti reassigned DISPATCH-1989:
Assignee: Ken Giusti
> [libuv] ASAN use after free in connection close
> ---
>
> Key: DISPATCH-1989
> URL: https://issues.apache.org/jira/browse/DISPATCH-1989
> Project: Qpid Dispatch
> Issue Type: Bug
> Components: Router Node, Routing Engine
>Affects Versions: 1.13.0, 1.14.0
>Reporter: Alex Ward
>Assignee: Ken Giusti
>Priority: Minor
> Labels: libuv
> Attachments: mtrace.3.9.txt, mtrace.txt, qdrouterd.txt
>
>
> While we dig into the issue below I thought it worth reaching out to see if
> this is a familiar issue or in code that has changed much since 1.13 so may
> have likely already been fixed.
> We have a freebsd product that is using qdrouterd. We have had some crashes
> during closing of connections to a broker so built a qdrouterd with asan on.
> This is hitting the use-after-free issue in uv__run_closing_handles below
> fairly reliably.
> It appears from first inspection that uv__finish_close is trying to remove a
> handle from the handle_queue at line 300 but the handle_queue has reference
> into a pconnection_t that has already been freed.
> Is this an area of code that has had issues in the past? Is this likely to
> have been fixed in the latest release? Any suggestions on where to add
> printfs etc to identify the culprit?
>
>
> {{243static void uv__finish_close(uv_handle_t* handle) {}}
> {{244 uv_signal_t* sh;}}
> {{245}}
> {{246 /* Note: while the handle is in the UV_HANDLE_CLOSING state now, it's
> still}}
> {{247 * possible for it to be active in the sense that uv__is_active()
> returns}}
> {{248 * true.}}
> {{249 *}}
> {{250 * A good example is when the user calls uv_shutdown(), immediately
> followed}}
> {{251 * by uv_close(). The handle is considered active at this point because
> the}}
> {{252 * completion of the shutdown req is still pending.}}
> {{253 */}}
> {{254 assert(handle->flags & UV_HANDLE_CLOSING);}}
> {{255 assert(!(handle->flags & UV_HANDLE_CLOSED));}}
> {{256 handle->flags |= UV_HANDLE_CLOSED;}}
> {{257}}
> {{258 switch (handle->type) {}}
> {{259 case UV_PREPARE:}}
> {{260 case UV_CHECK:}}
> {{261 case UV_IDLE:}}
> {{262 case UV_ASYNC:}}
> {{263 case UV_TIMER:}}
> {{264 case UV_PROCESS:}}
> {{265 case UV_FS_EVENT:}}
> {{266 case UV_FS_POLL:}}
> {{267 case UV_POLL:}}
> {{268 break;}}
> {{269}}
> {{270 case UV_SIGNAL:}}
> {{271 /* If there are any caught signals "trapped" in the signal pipe,}}
> {{272 * we can't call the close callback yet. Reinserting the handle}}
> {{273 * into the closing queue makes the event loop spin but that's}}
> {{274 * okay because we only need to deliver the pending events.}}
> {{275 */}}
> {{276 sh = (uv_signal_t*) handle;}}
> {{277 if (sh->caught_signals > sh->dispatched_signals) {}}
> {{278 handle->flags ^= UV_HANDLE_CLOSED;}}
> {{279 uv__make_close_pending(handle); /* Back into the queue. */}}
> {{280 return;}}
> {{281 }}}
> {{282 break;}}
> {{283}}
> {{284 case UV_NAMED_PIPE:}}
> {{285 case UV_TCP:}}
> {{286 case UV_TTY:}}
> {{287 uv__stream_destroy((uv_stream_t*)handle);}}
> {{288 break;}}
> {{289}}
> {{290 case UV_UDP:}}
> {{291 uv__udp_finish_close((uv_udp_t*)handle);}}
> {{292 break;}}
> {{293}}
> {{294 default:}}
> {{295 assert(0);}}
> {{296 break;}}
> {{297 }}}
> {{298}}
> {{299 uv__handle_unref(handle);}}
> {{300 QUEUE_REMOVE(&handle->handle_queue);}}
> {{301}}
> {{302 if (handle->close_cb) {}}
> {{303 handle->close_cb(handle);}}
> {{304 }}}
> {{305}}}
>
> Here's the asan report
> {{10 ==13358==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x61d0002ec048 at pc 0x0008006a7f84 bp 0x7fffe8c0 sp 0x7fffe8b8}}
> {{11 WRITE of size 8 at 0x61d0002ec048 thread T0}}
> {{12 #0 0x8006a7f83 in uv__finish_close
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/libuv/bedrock/internal/x86_64.asan.sim/build_configure/../../../../distro/src/unix/core.c:300:3}}
> {{13 #1 0x8006a38db in uv__run_closing_handles
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/libuv/bedrock/internal/x86_64.asan.sim/build_configure/../../../../distro/src/unix/core.c:317:5}}
> {{14 #2 0x8006a3463 in uv_run
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/libuv/bedrock/internal/x86_64.asan.sim/build_configure/../../../../distro/src/unix/core.c:387:5}}
> {{15 #3 0x800b378b5 in leader_lead_lh
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-proton/distro/c/src/proactor/libuv.c:1039:5}}
> {{16 #4 0x800b37a5c in pn_proactor_wait
> /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-proton/distr
