[ https://issues.apache.org/jira/browse/TS-3078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Leif Hedstrom updated TS-3078: ------------------------------ Fix Version/s: 5.2.0 > MIMEHdrImpl::unmarshal crash caused by cache corruption > ------------------------------------------------------- > > Key: TS-3078 > URL: https://issues.apache.org/jira/browse/TS-3078 > Project: Traffic Server > Issue Type: Bug > Components: Cache, MIME > Affects Versions: 4.0.2 > Reporter: kang li > Labels: crash > Fix For: 5.2.0 > > > {code} > (gdb) bt > #0 0x00000000005c22c6 in unmarshal (this=0x2aaed05f61f8, > offset=46930308587840) at MIME.cc:3534 > #1 MIMEHdrImpl::unmarshal (this=0x2aaed05f61f8, offset=46930308587840) at > MIME.cc:3590 > #2 0x00000000005b4bcb in HdrHeap::unmarshal (this=0x2aaed05f6140, > buf_length=<value optimized out>, obj_type=<value optimized out>, > found_obj=<value optimized out>, > block_ref=<value optimized out>) at HdrHeap.cc:926 > #3 0x00000000005b8be1 in HTTPInfo::unmarshal (buf=0x2aaed05f6048 > "\355\336\315\253", len=3280, block_ref=0x2aae080feec0) at HTTP.cc:1948 > #4 0x0000000000635e43 in unmarshal_helper (this=0x2aae10081b90, event=<value > optimized out>, e=<value optimized out>) at Cache.cc:2066 > #5 CacheVC::handleReadDone (this=0x2aae10081b90, event=<value optimized > out>, e=<value optimized out>) at Cache.cc:2195 > #6 0x00000000005f1845 in handleEvent (this=<value optimized out>, > event=<value optimized out>, data=<value optimized out>) > at ../../iocore/eventsystem/I_Continuation.h:146 > #7 AIOCallbackInternal::io_complete (this=<value optimized out>, > event=<value optimized out>, data=<value optimized out>) at > ../../iocore/aio/P_AIO.h:123 > #8 0x00000000006a990f in handleEvent (this=0x2aadf0809010, e=0x2aae3812b710, > calling_code=1) at I_Continuation.h:146 > #9 EThread::process_event (this=0x2aadf0809010, e=0x2aae3812b710, > calling_code=1) at UnixEThread.cc:141 > #10 0x00000000006aa48b in EThread::execute (this=0x2aadf0809010) at > UnixEThread.cc:192 > #11 0x00000000006a87aa in spawn_thread_internal (a=0x13e0a80) at Thread.cc:88 > #12 0x00002aadeaf3c9d1 in start_thread () from /lib64/libpthread.so.0 > #13 0x00000034cfce8b6d in clone () from /lib64/libc.so.6 > (gdb) p m_freetop > $1 = 892220471 > (gdb) p m_field_slots > $2 = {{m_ptr_name = 0x33203a2274616c22 <Address 0x33203a2274616c22 out of > bounds>, m_ptr_value = 0x3939333338312e33 <Address 0x3939333338312e33 out of > bounds>, > m_next_dup = 0x3a226e6f6c22202c, m_wks_idx = 11552, m_len_name = 14137, > m_len_value = 3289390, m_n_v_raw_printable = 0 '\000', > m_n_v_raw_printable_pad = 3 '\003', > m_readiness = 3 '\003', m_flags = 0 '\000'}, {m_ptr_name = > 0x766c4ccefc939174 <Address 0x766c4ccefc939174 out of bounds>, > m_ptr_value = 0x222056def09983ac <Address 0x222056def09983ac out of > bounds>, m_next_dup = 0x203a4d1444c0d5b3, m_wks_idx = 21538, m_len_name = > 9048, > m_len_value = 7890260, m_n_v_raw_printable = 1 '\001', > m_n_v_raw_printable_pad = 0 '\000', m_readiness = 2 '\002', m_flags = 1 > '\001'}, { > m_ptr_name = 0x656f7722202c2273 <Address 0x656f7722202c2273 out of > bounds>, m_ptr_value = 0x3039373231203a22 <Address 0x3039373231203a22 out of > bounds>, > m_next_dup = 0x697a22202c393739, m_wks_idx = 8816, m_len_name = 8250, > m_len_value = 3553058, m_n_v_raw_printable = 0 '\000', > m_n_v_raw_printable_pad = 1 '\001', > m_readiness = 3 '\003', m_flags = 0 '\000'}, {m_ptr_name = > 0x7d5d4b2bf0819670 <Address 0x7d5d4b2bf0819670 out of bounds>, > m_ptr_value = 0x796b8e1844d2836c <Address 0x796b8e1844d2836c out of > bounds>, m_next_dup = 0x756c8c24f2da9b62, m_wks_idx = 8805, m_len_name = > 31546, > m_len_value = 7152160, m_n_v_raw_printable = 1 '\001', > m_n_v_raw_printable_pad = 0 '\000', m_readiness = 2 '\002', m_flags = 1 > '\001'}, { > m_ptr_name = 0x22207b203a227370 <Address 0x22207b203a227370 out of > bounds>, m_ptr_value = 0x22203a2272646461 <Address 0x22203a2272646461 out of > bounds>, > m_next_dup = 0x626d452034323332, m_wks_idx = 29285, m_len_name = 22304, > m_len_value = 6582127, m_n_v_raw_printable = 1 '\001', > m_n_v_raw_printable_pad = 1 '\001', > m_readiness = 3 '\003', m_flags = 1 '\001'}, {m_ptr_name = > 0x6c654b202c724420 <Address 0x6c654b202c724420 out of bounds>, > {code} > This only happened in one machine in four nodes. The frequency is about > several days one time. After disable plugins and restart ATS the problem > still exist. And there are also some warning in traffic.out. One interesting > thing is that the warning all happened "after 776 bytes". > {code} > WARNING: Unmarshal failed due to unknow obj type 173 after 776 bytes---- > Dumping header heap @ 0x2b37b251e140 - len 2250 ------ > 0x2b37b251e140: 0xdcbafeed 0x0 0xb251e4b8 0x2b37 > 0x2b37b251e150: 0xb251e1c8 0x2b37 0x378 0x0 > 0x2b37b251e160: 0x0 0x0 0x0 0x0 > 0x2b37b251e170: 0x0 0x0 0x956b2af0 0x2b39 > 0x2b37b251e180: 0xb251e4b8 0x2b37 0x552 0x312e312f > 0x2b37b251e190: 0x31323420 0x63655220 0x0 0x0 > 0x2b37b251e1a0: 0xd646e75 0x6361430a 0x432d6568 0x72746e6f > 0x2b37b251e1b0: 0x0 0x0 0xd657461 0x6e6f430a > 0x2b37b251e1c0: 0x89 0x7079542d 0x3003 0x1 > 0x2b37b251e1d0: 0x10001 0x480004 0xb251e448 0x2b37 > 0x2b37b251e1e0: 0xb251e4b8 0x2b37 0x6e0003 0x0 > 0x2b37b251e1f0: 0xb251e1f8 0x2b37 0x25004 0x2b35 > 0x2b37b251e200: 0x1000401 0x0 0x6ffffff1 0xfffffffe > 0x2b37b251e210: 0xffffffff 0xfeffffff 0x0 0x0 > 0x2b37b251e220: 0x0 0x0 0x0 0x2b00 > 0x2b37b251e230: 0xb251e238 0x2b37 0x21005 0xb > 0x2b37b251e240: 0x0 0x0 0xb251e4bb 0x2b37 > 0x2b37b251e250: 0xb251e4bf 0x2b37 0x0 0x0 > 0x2b37b251e260: 0x4001e 0x68000024 0xb251e4e3 0x2b37 > 0x2b37b251e270: 0xb251e4e9 0x2b37 0x0 0x0 > 0x2b37b251e280: 0x60004 0x68000003 0xb251e4ec 0x2b37 > 0x2b37b251e290: 0xb251e4fa 0x2b37 0x0 0x0 > 0x2b37b251e2a0: 0xeffff 0x6800012e 0xb251e628 0x2b37 > 0x2b37b251e2b0: 0xb251e63c 0x2b37 0xb251e388 0x2b37 > 0x2b37b251e2c0: 0x14ffff 0x68000018 0xb251e654 0x2b37 > 0x2b37b251e2d0: 0xb251e663 0x2b37 0x0 0x0 > 0x2b37b251e2e0: 0xfffff 0x68000001 0xb251e664 0x2b37 > 0x2b37b251e2f0: 0xb251e674 0x2b37 0x0 0x0 > 0x2b37b251e300: 0x10ffff 0x68000105 0xb251e779 0x2b37 > 0x2b37b251e310: 0xb251e782 0x2b37 0x0 0x0 > 0x2b37b251e320: 0x9000b 0x6800000c 0xb251e78e 0x2b37 > 0x2b37b251e330: 0xb251e79d 0x2b37 0x0 0x0 > 0x2b37b251e340: 0xf0078 0x6800001c 0x4dd2e5e1 0x2b35 > 0x2b37b251e350: 0x4dd2e5e4 0x2b35 0x0 0x0 > 0x2b37b251e360: 0x30042 0x78000086 0x6b5e2416 0x2b3a > 0x2b37b251e370: 0x6b5e2422 0x2b3a 0x0 0x0 > 0x2b37b251e380: 0xa000c 0x79000005 0xb251e842 0x2b37 > 0x2b37b251e390: 0xb251e856 0x2b37 0x0 0x0 > 0x2b37b251e3a0: 0x14ffff 0x20000018 0xec9d3913 0x2b37 > 0x2b37b251e3b0: 0xec9d391a 0x2b37 0x0 0x0 > 0x2b37b251e3c0: 0x70043 0x60000015 0x0 0x0 > 0x2b37b251e3d0: 0x0 0x0 0x0 0x0 > 0x2b37b251e3e0: 0x0 0x0 0x0 0x0 > 0x2b37b251e3f0: 0x0 0x0 0x0 0x0 > 0x2b37b251e400: 0x42a53c96 0x4a4246a8 0x515e4ead 0x5aea5690 > 0x2b37b251e410: 0x605d5c77 0x6a726583 0x70606ccd 0x794174c1 > 0x2b37b251e420: 0x737f01 0x9510525 0x10c70ff4 0x1a6e174f > 0x2b37b251e430: 0x23821c2f 0x28fd244b 0x30412ca1 0x3be137e2 > 0x2b37b251e440: 0x42a53c96 0x4a4246a8 0x515e4ead 0x5aea5690 > 0x2b37b251e450: 0x605d5c77 0x6a726583 0x70606ccd 0x794174c1 > 0x2b37b251e460: 0x737f01 0x9510525 0x1c2f1a6e 0x244b2382 > 0x2b37b251e470: 0x2ca12aa3 0x37e23041 0x3c963be1 0x46a842a5 > 0x2b37b251e480: 0x4ead4a42 0x5690515e 0x5c775aea 0x6583605d > 0x2b37b251e490: 0x6ccd6a72 0x74c17060 0x7f017941 0x5250044 > 0x2b37b251e4a0: 0xff40951 0x244b10c7 0x2ca12870 0x37e23135 > 0x2b37b251e4b0: 0x3fbc3b89 0x45424236 0x4ead48a0 0x5690535e > 0x2b37b251e4c0: 0x5c775aea 0x6583624d 0x6ccd6a7b 0x74c17060 > 0x2b37b251e4d0: 0x7ef97941 0x5250045 0xebe0951 0x164a12ea > 0x2b37b251e4e0: 0x1c2f1a6e 0x244b2240 0x2ca12870 0x37e23135 > 0x2b37b251e4f0: 0x3fb83b89 0x45424331 0x4ead48a0 0x5690535e > 0x2b37b251e500: 0x5c775b70 0x65f4624d 0x6ccd6a7b 0x74c17060 > 0x2b37b251e510: 0x7ef97941 0x5250045 0xebe0951 0x164a12ea > 0x2b37b251e520: 0x1c2f1a6e 0x244b2240 0x2ca12870 0x36ab3135 > 0x2b37b251e530: 0x3fb83b89 0x45424331 0x4ead48a0 0x5690535e > 0x2b37b251e540: 0x5c775b70 0x65f4624d 0x6ccd6a7b 0x755d7060 > 0x2b37b251e550: 0x7ef97941 0x5250045 0xebe0951 0x164a12ea > 0x2b37b251e560: 0x1c2f1a6e 0x244b2240 0x2ca12870 0x36ab3135 > 0x2b37b251e570: 0x3fb83b89 0x45424331 0x4ead48a0 0x5690535e > 0x2b37b251e580: 0x5c775829 0x65f4624d 0x6ccd6a7b 0x755d7060 > 0x2b37b251e590: 0x7ef97941 0x5250045 0xebe095e 0x164a12ea > 0x2b37b251e5a0: 0x1c2f1a6e 0xe402240 0x45914591 0x460d4591 > 0x2b37b251e5b0: 0x5a7b0cd3 0x460d460d 0x5455460d 0x460d0cd3 > 0x2b37b251e5c0: 0xcd35455 0x5455460d 0x54550cd3 0x543e0cd3 > 0x2b37b251e5d0: 0x543e0e7e 0x543e460d 0x543e0c4d 0x543c0c4d > 0x2b37b251e5e0: 0x543c0c4d 0x460d0fc4 0xfc433b0 0xfc45685 > 0x2b37b251e5f0: 0x460d5685 0x5a7b5685 0x61ae5e94 0x4004434f > 0x2b37b251e600: 0x56854bde 0x56850fc4 0x5e945a7b 0x68d72780 > 0x2b37b251e610: 0x33b06f7c 0x5a7b5685 0x5e942780 0x278061ae > 0x2b37b251e620: 0x40044004 0x56850c67 0x40041b4d 0x5a7b5685 > 0x2b37b251e630: 0x22885e94 0x5a7b5685 0x4bde1ea0 0x56850c50 > 0x2b37b251e640: 0x5e945a7b 0x56852288 0x1ea05a7b 0xcbd4bde > 0x2b37b251e650: 0x5a7b5685 0x22885e94 0x5a7b5685 0x61ae5fd1 > 0x2b37b251e660: 0x134f4ebf 0x5a7b57ec 0x5fd1329b 0x1b4d1782 > ... > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)