[
https://issues.apache.org/jira/browse/TS-3078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kang li updated TS-3078:
------------------------
Affects Version/s: 4.0.2
> MIMEHdrImpl::unmarshal crash caused by cache corruption
> -------------------------------------------------------
>
> Key: TS-3078
> URL: https://issues.apache.org/jira/browse/TS-3078
> Project: Traffic Server
> Issue Type: Bug
> Components: Cache, MIME
> Affects Versions: 4.0.2
> Reporter: kang li
>
> {code}
> (gdb) bt
> #0 0x00000000005c22c6 in unmarshal (this=0x2aaed05f61f8,
> offset=46930308587840) at MIME.cc:3534
> #1 MIMEHdrImpl::unmarshal (this=0x2aaed05f61f8, offset=46930308587840) at
> MIME.cc:3590
> #2 0x00000000005b4bcb in HdrHeap::unmarshal (this=0x2aaed05f6140,
> buf_length=<value optimized out>, obj_type=<value optimized out>,
> found_obj=<value optimized out>,
> block_ref=<value optimized out>) at HdrHeap.cc:926
> #3 0x00000000005b8be1 in HTTPInfo::unmarshal (buf=0x2aaed05f6048
> "\355\336\315\253", len=3280, block_ref=0x2aae080feec0) at HTTP.cc:1948
> #4 0x0000000000635e43 in unmarshal_helper (this=0x2aae10081b90, event=<value
> optimized out>, e=<value optimized out>) at Cache.cc:2066
> #5 CacheVC::handleReadDone (this=0x2aae10081b90, event=<value optimized
> out>, e=<value optimized out>) at Cache.cc:2195
> #6 0x00000000005f1845 in handleEvent (this=<value optimized out>,
> event=<value optimized out>, data=<value optimized out>)
> at ../../iocore/eventsystem/I_Continuation.h:146
> #7 AIOCallbackInternal::io_complete (this=<value optimized out>,
> event=<value optimized out>, data=<value optimized out>) at
> ../../iocore/aio/P_AIO.h:123
> #8 0x00000000006a990f in handleEvent (this=0x2aadf0809010, e=0x2aae3812b710,
> calling_code=1) at I_Continuation.h:146
> #9 EThread::process_event (this=0x2aadf0809010, e=0x2aae3812b710,
> calling_code=1) at UnixEThread.cc:141
> #10 0x00000000006aa48b in EThread::execute (this=0x2aadf0809010) at
> UnixEThread.cc:192
> #11 0x00000000006a87aa in spawn_thread_internal (a=0x13e0a80) at Thread.cc:88
> #12 0x00002aadeaf3c9d1 in start_thread () from /lib64/libpthread.so.0
> #13 0x00000034cfce8b6d in clone () from /lib64/libc.so.6
> (gdb) p m_freetop
> $1 = 892220471
> (gdb) p m_field_slots
> $2 = {{m_ptr_name = 0x33203a2274616c22 <Address 0x33203a2274616c22 out of
> bounds>, m_ptr_value = 0x3939333338312e33 <Address 0x3939333338312e33 out of
> bounds>,
> m_next_dup = 0x3a226e6f6c22202c, m_wks_idx = 11552, m_len_name = 14137,
> m_len_value = 3289390, m_n_v_raw_printable = 0 '\000',
> m_n_v_raw_printable_pad = 3 '\003',
> m_readiness = 3 '\003', m_flags = 0 '\000'}, {m_ptr_name =
> 0x766c4ccefc939174 <Address 0x766c4ccefc939174 out of bounds>,
> m_ptr_value = 0x222056def09983ac <Address 0x222056def09983ac out of
> bounds>, m_next_dup = 0x203a4d1444c0d5b3, m_wks_idx = 21538, m_len_name =
> 9048,
> m_len_value = 7890260, m_n_v_raw_printable = 1 '\001',
> m_n_v_raw_printable_pad = 0 '\000', m_readiness = 2 '\002', m_flags = 1
> '\001'}, {
> m_ptr_name = 0x656f7722202c2273 <Address 0x656f7722202c2273 out of
> bounds>, m_ptr_value = 0x3039373231203a22 <Address 0x3039373231203a22 out of
> bounds>,
> m_next_dup = 0x697a22202c393739, m_wks_idx = 8816, m_len_name = 8250,
> m_len_value = 3553058, m_n_v_raw_printable = 0 '\000',
> m_n_v_raw_printable_pad = 1 '\001',
> m_readiness = 3 '\003', m_flags = 0 '\000'}, {m_ptr_name =
> 0x7d5d4b2bf0819670 <Address 0x7d5d4b2bf0819670 out of bounds>,
> m_ptr_value = 0x796b8e1844d2836c <Address 0x796b8e1844d2836c out of
> bounds>, m_next_dup = 0x756c8c24f2da9b62, m_wks_idx = 8805, m_len_name =
> 31546,
> m_len_value = 7152160, m_n_v_raw_printable = 1 '\001',
> m_n_v_raw_printable_pad = 0 '\000', m_readiness = 2 '\002', m_flags = 1
> '\001'}, {
> m_ptr_name = 0x22207b203a227370 <Address 0x22207b203a227370 out of
> bounds>, m_ptr_value = 0x22203a2272646461 <Address 0x22203a2272646461 out of
> bounds>,
> m_next_dup = 0x626d452034323332, m_wks_idx = 29285, m_len_name = 22304,
> m_len_value = 6582127, m_n_v_raw_printable = 1 '\001',
> m_n_v_raw_printable_pad = 1 '\001',
> m_readiness = 3 '\003', m_flags = 1 '\001'}, {m_ptr_name =
> 0x6c654b202c724420 <Address 0x6c654b202c724420 out of bounds>,
> {code}
> This only happened in one machine in four nodes. The frequency is about
> several days one time. After disable plugins and restart ATS the problem
> still exist. And there are also some warning in traffic.out. One interesting
> thing is that the warning all happened "after 776 bytes".
> {code}
> WARNING: Unmarshal failed due to unknow obj type 173 after 776 bytes----
> Dumping header heap @ 0x2b37b251e140 - len 2250 ------
> 0x2b37b251e140: 0xdcbafeed 0x0 0xb251e4b8 0x2b37
> 0x2b37b251e150: 0xb251e1c8 0x2b37 0x378 0x0
> 0x2b37b251e160: 0x0 0x0 0x0 0x0
> 0x2b37b251e170: 0x0 0x0 0x956b2af0 0x2b39
> 0x2b37b251e180: 0xb251e4b8 0x2b37 0x552 0x312e312f
> 0x2b37b251e190: 0x31323420 0x63655220 0x0 0x0
> 0x2b37b251e1a0: 0xd646e75 0x6361430a 0x432d6568 0x72746e6f
> 0x2b37b251e1b0: 0x0 0x0 0xd657461 0x6e6f430a
> 0x2b37b251e1c0: 0x89 0x7079542d 0x3003 0x1
> 0x2b37b251e1d0: 0x10001 0x480004 0xb251e448 0x2b37
> 0x2b37b251e1e0: 0xb251e4b8 0x2b37 0x6e0003 0x0
> 0x2b37b251e1f0: 0xb251e1f8 0x2b37 0x25004 0x2b35
> 0x2b37b251e200: 0x1000401 0x0 0x6ffffff1 0xfffffffe
> 0x2b37b251e210: 0xffffffff 0xfeffffff 0x0 0x0
> 0x2b37b251e220: 0x0 0x0 0x0 0x2b00
> 0x2b37b251e230: 0xb251e238 0x2b37 0x21005 0xb
> 0x2b37b251e240: 0x0 0x0 0xb251e4bb 0x2b37
> 0x2b37b251e250: 0xb251e4bf 0x2b37 0x0 0x0
> 0x2b37b251e260: 0x4001e 0x68000024 0xb251e4e3 0x2b37
> 0x2b37b251e270: 0xb251e4e9 0x2b37 0x0 0x0
> 0x2b37b251e280: 0x60004 0x68000003 0xb251e4ec 0x2b37
> 0x2b37b251e290: 0xb251e4fa 0x2b37 0x0 0x0
> 0x2b37b251e2a0: 0xeffff 0x6800012e 0xb251e628 0x2b37
> 0x2b37b251e2b0: 0xb251e63c 0x2b37 0xb251e388 0x2b37
> 0x2b37b251e2c0: 0x14ffff 0x68000018 0xb251e654 0x2b37
> 0x2b37b251e2d0: 0xb251e663 0x2b37 0x0 0x0
> 0x2b37b251e2e0: 0xfffff 0x68000001 0xb251e664 0x2b37
> 0x2b37b251e2f0: 0xb251e674 0x2b37 0x0 0x0
> 0x2b37b251e300: 0x10ffff 0x68000105 0xb251e779 0x2b37
> 0x2b37b251e310: 0xb251e782 0x2b37 0x0 0x0
> 0x2b37b251e320: 0x9000b 0x6800000c 0xb251e78e 0x2b37
> 0x2b37b251e330: 0xb251e79d 0x2b37 0x0 0x0
> 0x2b37b251e340: 0xf0078 0x6800001c 0x4dd2e5e1 0x2b35
> 0x2b37b251e350: 0x4dd2e5e4 0x2b35 0x0 0x0
> 0x2b37b251e360: 0x30042 0x78000086 0x6b5e2416 0x2b3a
> 0x2b37b251e370: 0x6b5e2422 0x2b3a 0x0 0x0
> 0x2b37b251e380: 0xa000c 0x79000005 0xb251e842 0x2b37
> 0x2b37b251e390: 0xb251e856 0x2b37 0x0 0x0
> 0x2b37b251e3a0: 0x14ffff 0x20000018 0xec9d3913 0x2b37
> 0x2b37b251e3b0: 0xec9d391a 0x2b37 0x0 0x0
> 0x2b37b251e3c0: 0x70043 0x60000015 0x0 0x0
> 0x2b37b251e3d0: 0x0 0x0 0x0 0x0
> 0x2b37b251e3e0: 0x0 0x0 0x0 0x0
> 0x2b37b251e3f0: 0x0 0x0 0x0 0x0
> 0x2b37b251e400: 0x42a53c96 0x4a4246a8 0x515e4ead 0x5aea5690
> 0x2b37b251e410: 0x605d5c77 0x6a726583 0x70606ccd 0x794174c1
> 0x2b37b251e420: 0x737f01 0x9510525 0x10c70ff4 0x1a6e174f
> 0x2b37b251e430: 0x23821c2f 0x28fd244b 0x30412ca1 0x3be137e2
> 0x2b37b251e440: 0x42a53c96 0x4a4246a8 0x515e4ead 0x5aea5690
> 0x2b37b251e450: 0x605d5c77 0x6a726583 0x70606ccd 0x794174c1
> 0x2b37b251e460: 0x737f01 0x9510525 0x1c2f1a6e 0x244b2382
> 0x2b37b251e470: 0x2ca12aa3 0x37e23041 0x3c963be1 0x46a842a5
> 0x2b37b251e480: 0x4ead4a42 0x5690515e 0x5c775aea 0x6583605d
> 0x2b37b251e490: 0x6ccd6a72 0x74c17060 0x7f017941 0x5250044
> 0x2b37b251e4a0: 0xff40951 0x244b10c7 0x2ca12870 0x37e23135
> 0x2b37b251e4b0: 0x3fbc3b89 0x45424236 0x4ead48a0 0x5690535e
> 0x2b37b251e4c0: 0x5c775aea 0x6583624d 0x6ccd6a7b 0x74c17060
> 0x2b37b251e4d0: 0x7ef97941 0x5250045 0xebe0951 0x164a12ea
> 0x2b37b251e4e0: 0x1c2f1a6e 0x244b2240 0x2ca12870 0x37e23135
> 0x2b37b251e4f0: 0x3fb83b89 0x45424331 0x4ead48a0 0x5690535e
> 0x2b37b251e500: 0x5c775b70 0x65f4624d 0x6ccd6a7b 0x74c17060
> 0x2b37b251e510: 0x7ef97941 0x5250045 0xebe0951 0x164a12ea
> 0x2b37b251e520: 0x1c2f1a6e 0x244b2240 0x2ca12870 0x36ab3135
> 0x2b37b251e530: 0x3fb83b89 0x45424331 0x4ead48a0 0x5690535e
> 0x2b37b251e540: 0x5c775b70 0x65f4624d 0x6ccd6a7b 0x755d7060
> 0x2b37b251e550: 0x7ef97941 0x5250045 0xebe0951 0x164a12ea
> 0x2b37b251e560: 0x1c2f1a6e 0x244b2240 0x2ca12870 0x36ab3135
> 0x2b37b251e570: 0x3fb83b89 0x45424331 0x4ead48a0 0x5690535e
> 0x2b37b251e580: 0x5c775829 0x65f4624d 0x6ccd6a7b 0x755d7060
> 0x2b37b251e590: 0x7ef97941 0x5250045 0xebe095e 0x164a12ea
> 0x2b37b251e5a0: 0x1c2f1a6e 0xe402240 0x45914591 0x460d4591
> 0x2b37b251e5b0: 0x5a7b0cd3 0x460d460d 0x5455460d 0x460d0cd3
> 0x2b37b251e5c0: 0xcd35455 0x5455460d 0x54550cd3 0x543e0cd3
> 0x2b37b251e5d0: 0x543e0e7e 0x543e460d 0x543e0c4d 0x543c0c4d
> 0x2b37b251e5e0: 0x543c0c4d 0x460d0fc4 0xfc433b0 0xfc45685
> 0x2b37b251e5f0: 0x460d5685 0x5a7b5685 0x61ae5e94 0x4004434f
> 0x2b37b251e600: 0x56854bde 0x56850fc4 0x5e945a7b 0x68d72780
> 0x2b37b251e610: 0x33b06f7c 0x5a7b5685 0x5e942780 0x278061ae
> 0x2b37b251e620: 0x40044004 0x56850c67 0x40041b4d 0x5a7b5685
> 0x2b37b251e630: 0x22885e94 0x5a7b5685 0x4bde1ea0 0x56850c50
> 0x2b37b251e640: 0x5e945a7b 0x56852288 0x1ea05a7b 0xcbd4bde
> 0x2b37b251e650: 0x5a7b5685 0x22885e94 0x5a7b5685 0x61ae5fd1
> 0x2b37b251e660: 0x134f4ebf 0x5a7b57ec 0x5fd1329b 0x1b4d1782
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
