[ https://issues.apache.org/jira/browse/PROTON-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459471#comment-17459471 ]
Ken Giusti commented on PROTON-2477: ------------------------------------ Hit something very similar, but with raw connections. Proton main branch + Dispatch main branch: [https://github.com/kgiusti/dispatch/runs/4458562073?check_suite_focus=true#step:9:6680] 2021-12-08T14:52:05.3292433Z 63: ::MulticastLinearTest::test_02_presettled_large_msg_rx_close FAILED 2021-12-08T14:53:55.8740316Z 73: 2021-12-08T14:53:55.8741374Z 73: Router EA1 output file: 2021-12-08T14:53:55.8742043Z 73: >>>> 2021-12-08T14:53:55.8742638Z 73: ================================================================= 2021-12-08T14:53:55.8744704Z 73: ==5523==ERROR: AddressSanitizer: heap-use-after-free on address 0x61b000031812 at pc 0x7f376866ff6d bp 0x7f375e8b91c0 sp 0x7f375e8b91b0 2021-12-08T14:53:55.8746349Z 73: READ of size 1 at 0x61b000031812 thread T3 2021-12-08T14:53:55.8747250Z 73: #0 0x7f376866ff6c in next_runnable ../c/src/proactor/epoll.c:2403 2021-12-08T14:53:55.8751431Z 73: #1 0x7f3768670e53 in next_event_batch ../c/src/proactor/epoll.c:2456 2021-12-08T14:53:55.8752274Z 73: #2 0x7f376867611a in pn_proactor_wait ../c/src/proactor/epoll.c:2715 2021-12-08T14:53:55.8752985Z 73: #3 0x556e145e860e in thread_run ../src/server.c:1118 2021-12-08T14:53:55.8753665Z 73: #4 0x556e14461fbb in _thread_init ../src/posix/threading.c:172 2021-12-08T14:53:55.8754859Z 73: #5 0x7f37681cc608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608) 2021-12-08T14:53:55.8755978Z 73: #6 0x7f37673c2292 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x122292) 2021-12-08T14:53:55.8756537Z 73: 2021-12-08T14:53:55.8757385Z 73: 0x61b000031812 is located 146 bytes inside of 1536-byte region [0x61b000031780,0x61b000031d80) 2021-12-08T14:53:55.8758072Z 73: freed by thread T3 here: 2021-12-08T14:53:55.8759053Z 73: #0 0x7f37688397cf in __interceptor_free (/lib/x86_64-linux-gnu/libasan.so.5+0x10d7cf) 2021-12-08T14:53:55.8760086Z 73: #1 0x7f376867aad8 in praw_connection_cleanup ../c/src/proactor/epoll_raw_connection.c:162 2021-12-08T14:53:55.8761078Z 73: #2 0x7f376867d10b in pni_raw_connection_done ../c/src/proactor/epoll_raw_connection.c:419 2021-12-08T14:53:55.8761955Z 73: #3 0x7f3768676276 in pn_proactor_done ../c/src/proactor/epoll.c:2737 2021-12-08T14:53:55.8762659Z 73: #4 0x556e145e88b5 in thread_run ../src/server.c:1151 2021-12-08T14:53:55.8763356Z 73: #5 0x556e14461fbb in _thread_init ../src/posix/threading.c:172 2021-12-08T14:53:55.8764451Z 73: #6 0x7f37681cc608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608) 2021-12-08T14:53:55.8765115Z 73: 2021-12-08T14:53:55.8765619Z 73: previously allocated by thread T3 here: 2021-12-08T14:53:55.8766779Z 73: #0 0x7f3768839dc6 in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10ddc6) 2021-12-08T14:53:55.8767729Z 73: #1 0x7f376867aafa in pn_raw_connection ../c/src/proactor/epoll_raw_connection.c:168 2021-12-08T14:53:55.8768694Z 73: #2 0x556e1433c278 in _create_client_connection ../src/adaptors/http1/http1_client.c:175 2021-12-08T14:53:55.8769637Z 73: #3 0x556e1433ce86 in _handle_listener_events ../src/adaptors/http1/http1_client.c:206 2021-12-08T14:53:55.8770500Z 73: #4 0x556e145e3d83 in handle_event_with_context ../src/server.c:814 2021-12-08T14:53:55.8771246Z 73: #5 0x556e145e3e0a in do_handle_listener ../src/server.c:825 2021-12-08T14:53:55.8771932Z 73: #6 0x556e145e6a2f in handle ../src/server.c:1024 2021-12-08T14:53:55.8772579Z 73: #7 0x556e145e86b1 in thread_run ../src/server.c:1133 2021-12-08T14:53:55.8773265Z 73: #8 0x556e14461fbb in _thread_init ../src/posix/threading.c:172 2021-12-08T14:53:55.8774378Z 73: #9 0x7f37681cc608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608) 2021-12-08T14:53:55.8775042Z 73: 2021-12-08T14:53:55.8775479Z 73: Thread T3 created by T0 here: 2021-12-08T14:53:55.8776427Z 73: #0 0x7f3768766805 in pthread_create (/lib/x86_64-linux-gnu/libasan.so.5+0x3a805) 2021-12-08T14:53:55.8777279Z 73: #1 0x556e1446212a in sys_thread ../src/posix/threading.c:181 2021-12-08T14:53:55.8777972Z 73: #2 0x556e145f0137 in qd_server_run ../src/server.c:1525 2021-12-08T14:53:55.8778627Z 73: #3 0x556e1464b7da in main_process ../router/src/main.c:115 2021-12-08T14:53:55.8786102Z 73: #4 0x556e1464d7de in main ../router/src/main.c:369 2021-12-08T14:53:55.8787133Z 73: #5 0x7f37672c70b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2) 2021-12-08T14:53:55.8787735Z 73: 2021-12-08T14:53:55.8788930Z 73: SUMMARY: AddressSanitizer: heap-use-after-free ../c/src/proactor/epoll.c:2403 in next_runnable 2021-12-08T14:53:55.8789884Z 73: Shadow bytes around the buggy address: 2021-12-08T14:53:55.8790711Z 73: 0x0c367fffe2b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 2021-12-08T14:53:55.8791373Z 73: 0x0c367fffe2c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 2021-12-08T14:53:55.8792044Z 73: 0x0c367fffe2d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 2021-12-08T14:53:55.8793395Z 73: 0x0c367fffe2e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 2021-12-08T14:53:55.8801941Z 73: 0x0c367fffe2f0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 2021-12-08T14:53:55.8802636Z 73: =>0x0c367fffe300: fd fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd 2021-12-08T14:53:55.8803292Z 73: 0x0c367fffe310: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 2021-12-08T14:53:55.8804426Z 73: 0x0c367fffe320: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 2021-12-08T14:53:55.8805101Z 73: 0x0c367fffe330: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 2021-12-08T14:53:55.8812882Z 73: 0x0c367fffe340: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 2021-12-08T14:53:55.8813589Z 73: 0x0c367fffe350: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 2021-12-08T14:53:55.8814387Z 73: Shadow byte legend (one shadow byte represents 8 application bytes): 2021-12-08T14:53:55.8815090Z 73: Addressable: 00 2021-12-08T14:53:55.8828899Z 73: Partially addressable: 01 02 03 04 05 06 07 2021-12-08T14:53:55.8829418Z 73: Heap left redzone: fa 2021-12-08T14:53:55.8829871Z 73: Freed heap region: fd 2021-12-08T14:53:55.8830304Z 73: Stack left redzone: f1 2021-12-08T14:53:55.8830741Z 73: Stack mid redzone: f2 2021-12-08T14:53:55.8831173Z 73: Stack right redzone: f3 2021-12-08T14:53:55.8831617Z 73: Stack after return: f5 2021-12-08T14:53:55.8832057Z 73: Stack use after scope: f8 2021-12-08T14:53:55.8832481Z 73: Global redzone: f9 2021-12-08T14:53:55.8832903Z 73: Global init order: f6 2021-12-08T14:53:55.8833320Z 73: Poisoned by user: f7 2021-12-08T14:53:55.8833773Z 73: Container overflow: fc 2021-12-08T14:53:55.8834211Z 73: Array cookie: ac 2021-12-08T14:53:55.8834825Z 73: Intra object redzone: bb 2021-12-08T14:53:55.8835272Z 73: ASan internal: fe 2021-12-08T14:53:55.8835693Z 73: Left alloca redzone: ca 2021-12-08T14:53:55.8836147Z 73: Right alloca redzone: cb 2021-12-08T14:53:55.8836557Z 73: Shadow gap: cc 2021-12-08T14:53:55.8836934Z 73: ==5523==ABORTING 2021-12-08T14:53:55.8837260Z 73: 2021-12-08T14:53:55.8837535Z 73: <<<< 2021-12-08T14:53:55.8837925Z 73: 2021-12-08T14:53:55.8838293Z 73: Router EA1 output file: 2021-12-08T14:53:55.8838649Z 73: >>>> 2021-12-08T14:53:55.8838990Z 73: ================================================================= 2021-12-08T14:53:55.8840262Z 73: ==5523==ERROR: AddressSanitizer: heap-use-after-free on address 0x61b000031812 at pc 0x7f376866ff6d bp 0x7f375e8b91c0 sp 0x7f375e8b91b0 2021-12-08T14:53:55.8841138Z 73: READ of size 1 at 0x61b000031812 thread T3 2021-12-08T14:53:55.8841732Z 73: #0 0x7f376866ff6c in next_runnable ../c/src/proactor/epoll.c:2403 2021-12-08T14:53:55.8842410Z 73: #1 0x7f3768670e53 in next_event_batch ../c/src/proactor/epoll.c:2456 2021-12-08T14:53:55.8843083Z 73: #2 0x7f376867611a in pn_proactor_wait ../c/src/proactor/epoll.c:2715 2021-12-08T14:53:55.8843702Z 73: #3 0x556e145e860e in thread_run ../src/server.c:1118 2021-12-08T14:53:55.8844298Z 73: #4 0x556e14461fbb in _thread_init ../src/posix/threading.c:172 2021-12-08T14:53:55.8845281Z 73: #5 0x7f37681cc608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608) 2021-12-08T14:53:55.8846239Z 73: #6 0x7f37673c2292 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x122292) 2021-12-08T14:53:55.8846733Z 73: 2021-12-08T14:53:55.8847449Z 73: 0x61b000031812 is located 146 bytes inside of 1536-byte region [0x61b000031780,0x61b000031d80) 2021-12-08T14:53:55.8848061Z 73: freed by thread T3 here: 2021-12-08T14:53:55.8848919Z 73: #0 0x7f37688397cf in __interceptor_free (/lib/x86_64-linux-gnu/libasan.so.5+0x10d7cf) 2021-12-08T14:53:55.8849809Z 73: #1 0x7f376867aad8 in praw_connection_cleanup ../c/src/proactor/epoll_raw_connection.c:162 2021-12-08T14:53:55.8850794Z 73: #2 0x7f376867d10b in pni_raw_connection_done ../c/src/proactor/epoll_raw_connection.c:419 2021-12-08T14:53:55.8851556Z 73: #3 0x7f3768676276 in pn_proactor_done ../c/src/proactor/epoll.c:2737 2021-12-08T14:53:55.8852164Z 73: #4 0x556e145e88b5 in thread_run ../src/server.c:1151 2021-12-08T14:53:55.8852776Z 73: #5 0x556e14461fbb in _thread_init ../src/posix/threading.c:172 2021-12-08T14:53:55.8853727Z 73: #6 0x7f37681cc608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608) 2021-12-08T14:53:55.8854311Z 73: 2021-12-08T14:53:55.8854738Z 73: previously allocated by thread T3 here: 2021-12-08T14:53:55.8855605Z 73: #0 0x7f3768839dc6 in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10ddc6) 2021-12-08T14:53:55.8856420Z 73: #1 0x7f376867aafa in pn_raw_connection ../c/src/proactor/epoll_raw_connection.c:168 2021-12-08T14:53:55.8857297Z 73: #2 0x556e1433c278 in _create_client_connection ../src/adaptors/http1/http1_client.c:175 2021-12-08T14:53:55.8858127Z 73: #3 0x556e1433ce86 in _handle_listener_events ../src/adaptors/http1/http1_client.c:206 2021-12-08T14:53:55.8858880Z 73: #4 0x556e145e3d83 in handle_event_with_context ../src/server.c:814 2021-12-08T14:53:55.8859532Z 73: #5 0x556e145e3e0a in do_handle_listener ../src/server.c:825 2021-12-08T14:53:55.8860125Z 73: #6 0x556e145e6a2f in handle ../src/server.c:1024 2021-12-08T14:53:55.8860670Z 73: #7 0x556e145e86b1 in thread_run ../src/server.c:1133 2021-12-08T14:53:55.8861273Z 73: #8 0x556e14461fbb in _thread_init ../src/posix/threading.c:172 2021-12-08T14:53:55.8862231Z 73: #9 0x7f37681cc608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608) 2021-12-08T14:53:55.8862808Z 73: 2021-12-08T14:53:55.8863188Z 73: Thread T3 created by T0 here: 2021-12-08T14:53:55.8864012Z 73: #0 0x7f3768766805 in pthread_create (/lib/x86_64-linux-gnu/libasan.so.5+0x3a805) 2021-12-08T14:53:55.8864833Z 73: #1 0x556e1446212a in sys_thread ../src/posix/threading.c:181 2021-12-08T14:53:55.8865434Z 73: #2 0x556e145f0137 in qd_server_run ../src/server.c:1525 2021-12-08T14:53:55.8866020Z 73: #3 0x556e1464b7da in main_process ../router/src/main.c:115 2021-12-08T14:53:55.8866595Z 73: #4 0x556e1464d7de in main ../router/src/main.c:369 2021-12-08T14:53:55.8867419Z 73: #5 0x7f37672c70b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2) 2021-12-08T14:53:55.8868353Z 73: 2021-12-08T14:53:55.8869258Z 73: SUMMARY: AddressSanitizer: heap-use-after-free ../c/src/proactor/epoll.c:2403 in next_runnable 2021-12-08T14:53:55.8870097Z 73: Shadow bytes around the buggy address: 2021-12-08T14:53:55.8870662Z 73: 0x0c367fffe2b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 2021-12-08T14:53:55.8871241Z 73: 0x0c367fffe2c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 2021-12-08T14:53:55.8871831Z 73: 0x0c367fffe2d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 2021-12-08T14:53:55.8872410Z 73: 0x0c367fffe2e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 2021-12-08T14:53:55.8872998Z 73: 0x0c367fffe2f0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 2021-12-08T14:53:55.8873581Z 73: =>0x0c367fffe300: fd fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd 2021-12-08T14:53:55.8874144Z 73: 0x0c367fffe310: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 2021-12-08T14:53:55.8874726Z 73: 0x0c367fffe320: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 2021-12-08T14:53:55.8875286Z 73: 0x0c367fffe330: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 2021-12-08T14:53:55.8875862Z 73: 0x0c367fffe340: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 2021-12-08T14:53:55.8876434Z 73: 0x0c367fffe350: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 2021-12-08T14:53:55.8877111Z 73: Shadow byte legend (one shadow byte represents 8 application bytes): 2021-12-08T14:53:55.8877719Z 73: Addressable: 00 2021-12-08T14:53:55.8878192Z 73: Partially addressable: 01 02 03 04 05 06 07 2021-12-08T14:53:55.8878792Z 73: Heap left redzone: fa 2021-12-08T14:53:55.8879245Z 73: Freed heap region: fd 2021-12-08T14:53:55.8879673Z 73: Stack left redzone: f1 2021-12-08T14:53:55.8880109Z 73: Stack mid redzone: f2 2021-12-08T14:53:55.8880537Z 73: Stack right redzone: f3 2021-12-08T14:53:55.8880977Z 73: Stack after return: f5 2021-12-08T14:53:55.8881403Z 73: Stack use after scope: f8 2021-12-08T14:53:55.8881834Z 73: Global redzone: f9 2021-12-08T14:53:55.8882260Z 73: Global init order: f6 2021-12-08T14:53:55.8882683Z 73: Poisoned by user: f7 2021-12-08T14:53:55.8883124Z 73: Container overflow: fc 2021-12-08T14:53:55.8883563Z 73: Array cookie: ac 2021-12-08T14:53:55.8883990Z 73: Intra object redzone: bb 2021-12-08T14:53:55.8884414Z 73: ASan internal: fe 2021-12-08T14:53:55.8884846Z 73: Left alloca redzone: ca 2021-12-08T14:53:55.8885282Z 73: Right alloca redzone: cb 2021-12-08T14:53:55.8885703Z 73: Shadow gap: cc 2021-12-08T14:53:55.8886074Z 73: ==5523==ABORTING > ASAN use-after-free of proactor pconnection > ------------------------------------------- > > Key: PROTON-2477 > URL: https://issues.apache.org/jira/browse/PROTON-2477 > Project: Qpid Proton > Issue Type: Bug > Components: proton-c > Reporter: Ken Giusti > Assignee: Clifford Jansen > Priority: Major > Attachments: ASAN.txt > > > qpid-dispatch github actions CI has hit this ASAN issue a couple of times > since enabling use of latest proton-c/main in our CI tests (see attached). > Appears to show a pconnection being freed at the end of batch processing, > then accessing that freed pconnection while waiting for the next event. > > [https://github.com/apache/qpid-dispatch/runs/4513058827?check_suite_focus=true#step:9:7347] > > {noformat} > ==4956==ERROR: AddressSanitizer: heap-use-after-free on address > 0x616000171412 at pc 0x7f7144626f6d bp 0x7ffe23b9a600 sp 0x7ffe23b9a5f0 > 63: E READ of size 1 at 0x616000171412 thread T0 > 63: E #0 0x7f7144626f6c in next_runnable > ../c/src/proactor/epoll.c:2403 > 63: E #1 0x7f7144627e53 in next_event_batch > ../c/src/proactor/epoll.c:2456 > 63: E #2 0x7f714462d11a in pn_proactor_wait > ../c/src/proactor/epoll.c:2715 > 63: E #3 0x556f559f860e in thread_run ../src/server.c:1118 > 63: E #4 0x556f55a001cf in qd_server_run ../src/server.c:1527 > 63: E #5 0x556f55a5b7ea in main_process ../router/src/main.c:115 > 63: E #6 0x556f55a5d7ee in main ../router/src/main.c:369 > 63: E #7 0x7f714327e0b2 in __libc_start_main > (/lib/x86_64-linux-gnu/libc.so.6+0x270b2) > 63: E #8 0x556f5571574d in _start > (/home/runner/work/qpid-dispatch/qpid-dispatch/qpid-dispatch/build/router/qdrouterd+0x56874d) > 63: E > 63: E 0x616000171412 is located 146 bytes inside of 576-byte region > [0x616000171380,0x6160001715c0) > 63: E freed by thread T0 here: > 63: E #0 0x7f71447f07cf in __interceptor_free > (/lib/x86_64-linux-gnu/libasan.so.5+0x10d7cf) > 63: E #1 0x7f714460d565 in pconnection_final_free > ../c/src/proactor/epoll.c:832 > 63: E #2 0x7f714460d8bc in pconnection_cleanup > ../c/src/proactor/epoll.c:848 > 63: E #3 0x7f71446104ab in pconnection_done > ../c/src/proactor/epoll.c:1048 > 63: E #4 0x7f714462d20e in pn_proactor_done > ../c/src/proactor/epoll.c:2725 > 63: E #5 0x556f559f88b5 in thread_run ../src/server.c:1151 > 63: E #6 0x556f55a001cf in qd_server_run ../src/server.c:1527 > 63: E #7 0x556f55a5b7ea in main_process ../router/src/main.c:115 > 63: E #8 0x556f55a5d7ee in main ../router/src/main.c:369 > 63: E #9 0x7f714327e0b2 in __libc_start_main > (/lib/x86_64-linux-gnu/libc.so.6+0x270b2) > 63: E > 63: E previously allocated by thread T2 here: > 63: E #0 0x7f71447f0bc8 in malloc > (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8) > 63: E #1 0x7f714461dbda in pn_listener_accept2 > ../c/src/proactor/epoll.c:1883 > 63: E #2 0x7f7144638bd3 in pn_listener_accept > ../c/src/proactor/proactor-internal.c:94 > 63: E #3 0x556f559efbe1 in on_accept ../src/server.c:622 > 63: E #4 0x556f559f44fc in handle_listener ../src/server.c:865 > 63: E #5 0x556f559f3d83 in handle_event_with_context > ../src/server.c:814 > 63: E #6 0x556f559f3e0a in do_handle_listener > ../src/server.c:825 > 63: E #7 0x556f559f6a2f in handle ../src/server.c:1024 > 63: E #8 0x556f559f86b1 in thread_run ../src/server.c:1133 > 63: E #9 0x556f55871fbb in _thread_init > ../src/posix/threading.c:172 > 63: E #10 0x7f7144183608 in start_thread > (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608) > 63: E > 63: E Thread T2 created by T0 here: > 63: E #0 0x7f714471d805 in pthread_create > (/lib/x86_64-linux-gnu/libasan.so.5+0x3a805) > 63: E #1 0x556f5587212a in sys_thread > ../src/posix/threading.c:181 > 63: E #2 0x556f55a00137 in qd_server_run ../src/server.c:1525 > 63: E #3 0x556f55a5b7ea in main_process ../router/src/main.c:115 > 63: E #4 0x556f55a5d7ee in main ../router/src/main.c:369 > 63: E #5 0x7f714327e0b2 in __libc_start_main > (/lib/x86_64-linux-gnu/libc.so.6+0x270b2) > 63: E > 63: E SUMMARY: AddressSanitizer: heap-use-after-free > ../c/src/proactor/epoll.c:2403 in next_runnable > 63: E Shadow bytes around the buggy address: > 63: E 0x0c2c80026230: fd fd fd fd fd fd fd fd fd fd fd fd fd fd > fd fd > 63: E 0x0c2c80026240: fd fd fd fd fd fd fd fd fd fd fd fd fd fd > fd fd > 63: E 0x0c2c80026250: fd fd fd fd fd fd fd fd fa fa fa fa fa fa > fa fa > 63: E 0x0c2c80026260: fa fa fa fa fa fa fa fa fa fa fa fa fa fa > fa fa > 63: E 0x0c2c80026270: fd fd fd fd fd fd fd fd fd fd fd fd fd fd > fd fd > 63: E =>0x0c2c80026280: fd fd[fd]fd fd fd fd fd fd fd fd fd fd fd > fd fd > 63: E 0x0c2c80026290: fd fd fd fd fd fd fd fd fd fd fd fd fd fd > fd fd > 63: E 0x0c2c800262a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd > fd fd > 63: E 0x0c2c800262b0: fd fd fd fd fd fd fd fd fa fa fa fa fa fa > fa fa > 63: E 0x0c2c800262c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa > fa fa > 63: E 0x0c2c800262d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > 00 00 > 63: E Shadow byte legend (one shadow byte represents 8 application > bytes): {noformat} > -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org