Hoi,
Following reproduces the drop.c:77 assertion:
create loopback interface instance 0
set interface ip address loop0 10.0.0.1/32
set interface state GigabitEthernet3/0/1 up
set interface state loop0 up
set interface state loop0 down
set interface ip address del loop0 10.0.0.1/32
delete loopback interface intfc loop0
set interface state GigabitEthernet3/0/1 down
set interface state GigabitEthernet3/0/1 up
comment { the following crashes VPP }
set interface state GigabitEthernet3/0/1 down
I found that adding IPv6 addresses does not provoke the crash, while adding
IPv4 addresses to loop0 does provoke it.
groet,
Pim
On Wed, Apr 6, 2022 at 3:56 PM Pim van Pelt via lists.fd.io <pim=
ipng...@lists.fd.io> wrote:
> Hoi,
>
> The crash I observed is now gone, thanks!
>
> VPP occasionally hits an ASSERT related to error counters at drop.c:77 --
> I'll try to see if I can get a reproduction, but it may take a while, and
> it may be transient.
>
>
> *11: /home/pim/src/vpp/src/vlib/drop.c:77 (counter_index) assertion `ci <
> n->n_errors' fails*
>
> Thread 14 "vpp_wk_11" received signal SIGABRT, Aborted.
> [Switching to Thread 0x7fff4bbfd700 (LWP 182685)]
> __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
> 50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
> (gdb) bt
> #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
> #1 0x00007ffff6a5f859 in __GI_abort () at abort.c:79
> #2 0x00000000004072e3 in os_panic () at
> /home/pim/src/vpp/src/vpp/vnet/main.c:413
> #3 0x00007ffff6daea29 in debugger () at
> /home/pim/src/vpp/src/vppinfra/error.c:84
> #4 0x00007ffff6dae7fa in _clib_error (how_to_die=2, function_name=0x0,
> line_number=0, fmt=0x7ffff6f9d19c "%s:%d (%s) assertion `%s' fails")
> at /home/pim/src/vpp/src/vppinfra/error.c:143
> #5 0x00007ffff6f782d9 in counter_index (vm=0x7fffa09fb2c0, e=3416) at
> /home/pim/src/vpp/src/vlib/drop.c:77
> #6 0x00007ffff6f77c57 in process_drop_punt (vm=0x7fffa09fb2c0,
> node=0x7fffa0c79b00, frame=0x7fff97168140,
> disposition=ERROR_DISPOSITION_DROP)
> at /home/pim/src/vpp/src/vlib/drop.c:224
> #7 0x00007ffff6f77957 in error_drop_node_fn_hsw (vm=0x7fffa09fb2c0,
> node=0x7fffa0c79b00, frame=0x7fff97168140)
> at /home/pim/src/vpp/src/vlib/drop.c:248
> #8 0x00007ffff6f0b10d in dispatch_node (vm=0x7fffa09fb2c0,
> node=0x7fffa0c79b00, type=VLIB_NODE_TYPE_INTERNAL,
> dispatch_state=VLIB_NODE_STATE_POLLING, frame=0x7fff97168140,
> last_time_stamp=5318787653101516) at /home/pim/src/vpp/src/vlib/main.c:961
> #9 0x00007ffff6f0bb60 in dispatch_pending_node (vm=0x7fffa09fb2c0,
> pending_frame_index=5, last_time_stamp=5318787653101516)
> at /home/pim/src/vpp/src/vlib/main.c:1120
> #10 0x00007ffff6f06e0f in vlib_main_or_worker_loop (vm=0x7fffa09fb2c0,
> is_main=0) at /home/pim/src/vpp/src/vlib/main.c:1587
> #11 0x00007ffff6f06537 in vlib_worker_loop (vm=0x7fffa09fb2c0) at
> /home/pim/src/vpp/src/vlib/main.c:1721
> #12 0x00007ffff6f44ef4 in vlib_worker_thread_fn (arg=0x7fff98eabec0) at
> /home/pim/src/vpp/src/vlib/threads.c:1587
> #13 0x00007ffff6f3ffe5 in vlib_worker_thread_bootstrap_fn
> (arg=0x7fff98eabec0) at /home/pim/src/vpp/src/vlib/threads.c:426
> #14 0x00007ffff6e61609 in start_thread (arg=<optimized out>) at
> pthread_create.c:477
> #15 0x00007ffff6b5c163 in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
> (gdb) up 4
> #4 0x00007ffff6dae7fa in _clib_error (how_to_die=2, function_name=0x0,
> line_number=0, fmt=0x7ffff6f9d19c "%s:%d (%s) assertion `%s' fails")
> at /home/pim/src/vpp/src/vppinfra/error.c:143
> 143 debugger ();
> (gdb) up
> #5 0x00007ffff6f782d9 in counter_index (vm=0x7fffa09fb2c0, e=3416) at
> /home/pim/src/vpp/src/vlib/drop.c:77
> 77 ASSERT (ci < n->n_errors);
> (gdb) list
> 72
> 73 ni = vlib_error_get_node (&vm->node_main, e);
> 74 n = vlib_get_node (vm, ni);
> 75
> 76 ci = vlib_error_get_code (&vm->node_main, e);
> 77 ASSERT (ci < n->n_errors);
> 78
> 79 ci += n->error_heap_index;
> 80
> 81 return ci;
>
> On Wed, Apr 6, 2022 at 1:53 PM Damjan Marion (damarion) <
> damar...@cisco.com> wrote:
>
>>
>> This seems to be day one issue, and my patch just exposed it.
>> Current interface deletion code is not removing node stats entries.
>>
>> So if you delete interface and then create one with the same name,
>> stats entry is already there, and creation of new entry fails.
>>
>> Hope this helps:
>>
>> https://gerrit.fd.io/r/c/vpp/+/35900
>>
>> —
>> Damjan
>>
>>
>>
>> > On 05.04.2022., at 22:13, Pim van Pelt <p...@ipng.nl> wrote:
>> >
>> > Hoi,
>> >
>> > Here's a minimal repro that reliably crashes VPP at head for me, does
>> not crash before gerrit 35640:
>> >
>> > create loopback interface instance 0
>> > create bond id 0 mode lacp load-balance l34
>> > create bond id 1 mode lacp load-balance l34
>> > delete loopback interface intfc loop0
>> > delete bond BondEthernet0
>> > delete bond BondEthernet1
>> > create bond id 0 mode lacp load-balance l34
>> > delete bond BondEthernet0
>> > comment { the next command crashes VPP }
>> > create loopback interface instance 0
>> >
>> >
>> >
>> > On Tue, Apr 5, 2022 at 9:48 PM Pim van Pelt <p...@ipng.nl> wrote:
>> > Hoi,
>> >
>> > There is a crashing regression in VPP after
>> https://gerrit.fd.io/r/c/vpp/+/35640
>> >
>> > With that change merged, VPP crashes upon creation and deletion of
>> interfaces. Winding back the repo until before 35640 does not crash. The
>> crash happens in
>> > 0: /home/pim/src/vpp/src/vlib/stats/stats.h:115 (vlib_stats_get_entry)
>> assertion `entry_index < vec_len (sm->directory_vector)' fails
>> >
>> > (gdb) bt
>> > #0 __GI_raise (sig=sig@entry=6) at
>> ../sysdeps/unix/sysv/linux/raise.c:50
>> > #1 0x00007ffff6a5e859 in __GI_abort () at abort.c:79
>> > #2 0x00000000004072e3 in os_panic () at
>> /home/pim/src/vpp/src/vpp/vnet/main.c:413
>> > #3 0x00007ffff6dada29 in debugger () at
>> /home/pim/src/vpp/src/vppinfra/error.c:84
>> > #4 0x00007ffff6dad7fa in _clib_error (how_to_die=2, function_name=0x0,
>> line_number=0, fmt=0x7ffff6f9c19c "%s:%d (%s) assertion `%s' fails")
>> > at /home/pim/src/vpp/src/vppinfra/error.c:143
>> > #5 0x00007ffff6f39605 in vlib_stats_get_entry (sm=0x7ffff6fce5e8
>> <vlib_stats_main>, entry_index=4294967295)
>> > at /home/pim/src/vpp/src/vlib/stats/stats.h:115
>> > #6 0x00007ffff6f39273 in vlib_stats_remove_entry
>> (entry_index=4294967295) at /home/pim/src/vpp/src/vlib/stats/stats.c:135
>> > #7 0x00007ffff6ee36d9 in vlib_register_errors (vm=0x7fff96800740,
>> node_index=718, n_errors=0, error_strings=0x0, counters=0x0)
>> > at /home/pim/src/vpp/src/vlib/error.c:149
>> > #8 0x00007ffff70b8e0c in setup_tx_node (vm=0x7fff96800740,
>> node_index=718, dev_class=0x7fff973f9fb0) at
>> /home/pim/src/vpp/src/vnet/interface.c:816
>> > #9 0x00007ffff70b7f26 in vnet_register_interface (vnm=0x7ffff7f579a0
>> <vnet_main>, dev_class_index=31, dev_instance=0, hw_class_index=29,
>> > hw_instance=7) at /home/pim/src/vpp/src/vnet/interface.c:1085
>> > #10 0x00007ffff7129efd in vnet_eth_register_interface
>> (vnm=0x7ffff7f579a0 <vnet_main>, r=0x7fff4b288f18)
>> > at /home/pim/src/vpp/src/vnet/ethernet/interface.c:376
>> > #11 0x00007ffff712bd05 in vnet_create_loopback_interface
>> (sw_if_indexp=0x7fff4b288fb8, mac_address=0x7fff4b288fb2 "", is_specified=1
>> '\001',
>> > user_instance=0) at
>> /home/pim/src/vpp/src/vnet/ethernet/interface.c:883
>> > #12 0x00007ffff712fecf in create_simulated_ethernet_interfaces
>> (vm=0x7fff96800740, input=0x7fff4b2899d0, cmd=0x7fff973c7e38)
>> > at /home/pim/src/vpp/src/vnet/ethernet/interface.c:930
>> > #13 0x00007ffff6ed65e8 in vlib_cli_dispatch_sub_commands
>> (vm=0x7fff96800740, cm=0x42c2f0 <vlib_global_main+48>,
>> input=0x7fff4b2899d0,
>> > parent_command_index=1161) at /home/pim/src/vpp/src/vlib/cli.c:592
>> > #14 0x00007ffff6ed6358 in vlib_cli_dispatch_sub_commands
>> (vm=0x7fff96800740, cm=0x42c2f0 <vlib_global_main+48>,
>> input=0x7fff4b2899d0,
>> > parent_command_index=33) at /home/pim/src/vpp/src/vlib/cli.c:549
>> > #15 0x00007ffff6ed6358 in vlib_cli_dispatch_sub_commands
>> (vm=0x7fff96800740, cm=0x42c2f0 <vlib_global_main+48>,
>> input=0x7fff4b2899d0,
>> > parent_command_index=0) at /home/pim/src/vpp/src/vlib/cli.c:549
>> > #16 0x00007ffff6ed5528 in vlib_cli_input (vm=0x7fff96800740,
>> input=0x7fff4b2899d0, function=0x0, function_arg=0)
>> > at /home/pim/src/vpp/src/vlib/cli.c:695
>> > #17 0x00007ffff6f61f21 in unix_cli_exec (vm=0x7fff96800740,
>> input=0x7fff4b289e78, cmd=0x7fff973c99d8) at
>> /home/pim/src/vpp/src/vlib/unix/cli.c:3454
>> > #18 0x00007ffff6ed65e8 in vlib_cli_dispatch_sub_commands
>> (vm=0x7fff96800740, cm=0x42c2f0 <vlib_global_main+48>,
>> input=0x7fff4b289e78,
>> > parent_command_index=0) at /home/pim/src/vpp/src/vlib/cli.c:592
>> > #19 0x00007ffff6ed5528 in vlib_cli_input (vm=0x7fff96800740,
>> input=0x7fff4b289e78, function=0x7ffff6f55960 <unix_vlib_cli_output>,
>> function_arg=1)
>> > at /home/pim/src/vpp/src/vlib/cli.c:695
>> >
>> > This is caught by a local regression test (
>> https://github.com/pimvanpelt/vppcfg/tree/main/intest) that executes a
>> bunch of CLI statements, and I have a set of transitions there which I can
>> probably narrow down to an exact repro case.
>> >
>> > On Fri, Apr 1, 2022 at 3:08 PM Pim van Pelt via lists.fd.io <pim=
>> ipng...@lists.fd.io> wrote:
>> > Hoi,
>> >
>> > As a followup - I tried to remember why I copied class VPPStats() and
>> friends into my own repository, but that may be because it's not exported
>> in __init__.py. Should it be? I pulled in the latest changed Damjan made to
>> vpp_stats.py into my own repo, and my app runs again. Is it possibly worth
>> our while to add the VPPStats() class to the exported classes in vpp_papi ?
>> >
>> > groet,
>> > Pim
>> >
>> > On Fri, Apr 1, 2022 at 2:50 PM Pim van Pelt via lists.fd.io <pim=
>> ipng...@lists.fd.io> wrote:
>> > Hoi,
>> >
>> > I noticed that my VPP SNMP Agent no longer works with the python API at
>> HEAD, and my attention was drawn to this change:
>> > https://gerrit.fd.io/r/c/vpp/+/35640
>> > stats: convert error counters to normal counters
>> >
>> >
>> > At HEAD, src/vpp-api/python/vpp_papi/vpp_stats.py now fails 4 out of 6
>> tests with the same error as my application:
>> > struct.error: offset -140393469444104 out of range for 1073741824-byte
>> buffer
>> > ..
>> > Ran 6 tests in 0.612s
>> > FAILED (errors=4)
>> >
>> > Damjan, Ole, any clues?
>> >
>> > groet,
>> > Pim
>> > --
>> > Pim van Pelt <p...@ipng.nl>
>> > PBVP1-RIPE - http://www.ipng.nl/
>> >
>> >
>> >
>> >
>> >
>> > --
>> > Pim van Pelt <p...@ipng.nl>
>> > PBVP1-RIPE - http://www.ipng.nl/
>> >
>> >
>> >
>> >
>> >
>> > --
>> > Pim van Pelt <p...@ipng.nl>
>> > PBVP1-RIPE - http://www.ipng.nl/
>> >
>> >
>> > --
>> > Pim van Pelt <p...@ipng.nl>
>> > PBVP1-RIPE - http://www.ipng.nl/
>> >
>> >
>> >
>>
>>
>
> --
> Pim van Pelt <p...@ipng.nl>
> PBVP1-RIPE - http://www.ipng.nl/
>
>
>
>
--
Pim van Pelt <p...@ipng.nl>
PBVP1-RIPE - http://www.ipng.nl/
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21211): https://lists.fd.io/g/vpp-dev/message/21211
Mute This Topic: https://lists.fd.io/mt/90274515/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
