Hello,
I have a tap interface connected to a noisy LAN and I found that a certain type
of IGMP packet will sometimes cause a crash (backtrace at the end) in
ip4_fib_mtrie_lookup_step_one(). More specifically its an IGMP packet with the
router alert IP option. Here's a packet trace:
00:02:41:522429: virtio-input
virtio: hw_if_index 6 next-index 4 vring 0 len 54
hdr: flags 0x00 gso_type 0x00 hdr_len 0 gso_size 0 csum_start 0 csum_offset
0 num_buffers 1
00:02:41:522430: ethernet-input
IP4: 00:0c:29:1f:43:a4 -> 01:00:5e:00:00:16
00:02:41:522430: ip4-input
IGMP: 172.20.2.194 -> 224.0.0.22
version 4, header length 24
tos 0xc0, ttl 1, length 40, checksum 0x5523
fragment id 0x0000, flags DONT_FRAGMENT
00:02:41:522431: ip4-options
option:[0x94,0x4,0x0,0x0]
00:02:41:522431: ip4-local
IGMP: 172.20.2.194 -> 224.0.0.22
version 4, header length 24
tos 0xc0, ttl 1, length 40, checksum 0x5523
fragment id 0x0000, flags DONT_FRAGMENT
00:02:41:522434: igmp-input
sw_if_index 6 next-index 0
membership_report_v3: code 0, checksum 0xfbf4
00:02:41:522435: error-drop
igmp-input: IGMP not enabled on this interface
I found that when the crash occurs vnet_buffer(b)->ip.fib_index is ~0 in
ip4_local_check_src(). Here's an example debug print just added just after "if
(PREDICT_FALSE (last_check->src.as_u32 != ip0->src_address.as_u32))" in
ip4_local_check_src()
Usual case:
ip4_local_check_src: (00000000 != 0101A8C0), buf 0x7f6b6301b900, vlib_tx
4294967295 fib index 0
When crash happens:
ip4_local_check_src: (00000000 != 0100A8C0), buf 0x7f6b63a00000, vlib_tx
4294967295 fib index 4294967295
I think the problem is that vnet_buffer(b)->ip.fib_index isn't set anywhere in
this processing chain (ip4-input -> ip4-options -> ip4-local). This can cause
an invalid pointer to be used when looking up the mtrie in
ip4_local_check_src(). Normally the fib_index metadata is assigned by
ip4-lookup via ip_lookup_set_buffer_fib_index(). But since the packet doesn't
traverse that node the metadata is unset. I'm guessing that due to luck and/or
initialization the fib_index metadata is usually zero, so the crash won't
happen until the metadata is modified elsewhere and then the buffer is reused
for this IGMP packet with router alert. I hope this is what's happening and
it's not something more nefarious like memory corruption.
I made the following change at the top of ip4_local_check_src (taken from
ip_lookup_set_buffer_fib_index())
const dpo_id_t *dpo0;
load_balance_t *lb0;
u32 lbi0;
+ ip4_main_t *im = &ip4_main;
vnet_buffer (b)->ip.fib_index =
+ vec_elt (im->fib_index_by_sw_if_index, vnet_buffer
(b)->sw_if_index[VLIB_RX]);
+ vnet_buffer (b)->ip.fib_index =
vnet_buffer (b)->sw_if_index[VLIB_TX] != ~0 ?
vnet_buffer (b)->sw_if_index[VLIB_TX] : vnet_buffer (b)->ip.fib_index;
With this change I was unable to trigger the crash. Don't know if this is a
proper fix though.
Here's the backtrace (some of the line numbers might be offset due to my
debugging):
Thread 1 "vpp_main" received signal SIGSEGV, Segmentation fault.
0x00007f73861c2748 in ip4_fib_mtrie_lookup_step_one
(dst_address=0x7f717de38e1a, m=<optimized out>) at
/home/jeff/vpp/src/vnet/ip/ip4_mtrie.h:229
229 /home/jeff/vpp/src/vnet/ip/ip4_mtrie.h: No such file or directory.
(gdb) bt
#0 0x00007f73861c2748 in ip4_fib_mtrie_lookup_step_one
(dst_address=0x7f717de38e1a, m=<optimized out>) at
/home/jeff/vpp/src/vnet/ip/ip4_mtrie.h:229
#1 ip4_local_check_src (error0=<synthetic pointer>, last_check=<synthetic
pointer>, ip0=0x7f717de38e0e, b=<optimized out>)
at /home/jeff/vpp/src/vnet/ip/ip4_forward.c:1352
#2 ip4_local_inline (vm=<optimized out>, node=<optimized out>,
frame=<optimized out>, head_of_feature_arc=<optimized out>)
at /home/jeff/vpp/src/vnet/ip/ip4_forward.c:1586
#3 0x00007f7385c70014 in dispatch_node (last_time_stamp=17304359695215669,
frame=0x7f718dcaf300, dispatch_state=VLIB_NODE_STATE_POLLING,
type=VLIB_NODE_TYPE_INTERNAL, node=0x7f7184ed2ec0, vm=0x7f7385ec9980
<vlib_global_main>) at /home/jeff/vpp/src/vlib/main.c:989
#4 dispatch_pending_node (vm=vm@entry=0x7f7385ec9980 <vlib_global_main>,
pending_frame_index=pending_frame_index@entry=3,
last_time_stamp=last_time_stamp@entry=17304359695215669) at
/home/jeff/vpp/src/vlib/main.c:1139
#5 0x00007f7385c719fd in vlib_main_or_worker_loop (is_main=1,
vm=0x7f7385ec9980 <vlib_global_main>) at /home/jeff/vpp/src/vlib/main.c:1555
#6 vlib_main_loop (vm=0x7f7385ec9980 <vlib_global_main>) at
/home/jeff/vpp/src/vlib/main.c:1629
#7 vlib_main (vm=vm@entry=0x7f7385ec9980 <vlib_global_main>,
input=input@entry=0x7f7184dfffa0) at /home/jeff/vpp/src/vlib/main.c:1820
#8 0x00007f7385cab453 in thread0 (arg=140134144842112) at
/home/jeff/vpp/src/vlib/unix/main.c:607
#9 0x00007f73855917cc in clib_calljmp () from
/usr/lib/x86_64-linux-gnu/libvppinfra.so.18.10
#10 0x00007ffd505eee30 in ?? ()
#11 0x00007f7385cac4e7 in vlib_unix_main (argc=<optimized out>, argv=<optimized
out>) at /home/jeff/vpp/src/vlib/unix/main.c:676
#12 0x0000000000000000 in ?? ()
(gdb) up
#1 ip4_local_check_src (error0=<synthetic pointer>, last_check=<synthetic
pointer>, ip0=0x7f717de38e0e, b=<optimized out>)
at /home/jeff/vpp/src/vnet/ip/ip4_forward.c:1352
1352 /home/jeff/vpp/src/vnet/ip/ip4_forward.c: No such file or directory.
(gdb) p/x *ip0
$15 = {{ip_version_and_header_length = 0x46, tos = 0xc0, length = 0x2800,
fragment_id = 0x0, flags_and_fragment_offset = 0x40, ttl = 0x1, protocol = 0x2,
checksum = 0x8183, {{src_address = {data = {0xac, 0x14, 0xd4, 0x63},
data_u32 = 0x63d414ac, as_u8 = {0xac, 0x14, 0xd4, 0x63}, as_u16 = {0x14ac,
0x63d4}, as_u32 = 0x63d414ac}, dst_address = {data = {0xe0, 0x0,
0x0, 0x16}, data_u32 = 0x160000e0, as_u8 = {0xe0, 0x0, 0x0, 0x16}, as_u16 = {
0xe0, 0x1600}, as_u32 = 0x160000e0}}, address_pair = {src = {data =
{0xac, 0x14, 0xd4, 0x63}, data_u32 = 0x63d414ac, as_u8 = {0xac, 0x14,
0xd4, 0x63}, as_u16 = {0x14ac, 0x63d4}, as_u32 = 0x63d414ac}, dst =
{data = {0xe0, 0x0, 0x0, 0x16}, data_u32 = 0x160000e0, as_u8 = {0xe0,
0x0, 0x0, 0x16}, as_u16 = {0xe0, 0x1600}, as_u32 = 0x160000e0}}}},
{checksum_data_64 = {0x4000002800c046, 0x63d414ac81830201},
checksum_data_64_32 = {0x160000e0}}, {checksum_data_32 = {0x2800c046,
0x400000, 0x81830201, 0x63d414ac, 0x160000e0}}}
Let me know if additional information is needed or if there are any other
questions.
Thanks,
Jeff
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#10843): https://lists.fd.io/g/vpp-dev/message/10843
Mute This Topic: https://lists.fd.io/mt/27373338/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
