Adrián Moreno via discuss <[email protected]> writes:
> On Tue, Apr 29, 2025 at 01:48:42PM +0800, chenyongchang--- via discuss wrote:
>>
>> Hello，
>> In a high-traffic scenario, when modifying the bond-rebalance-interval
>> configuration for an OVS-DPDK bond interface,
>> we observed that OVS-DPDK generated USERSPACE_INVALID_PORT_DROP errors.
>>
>> After analysis, executing the command ovs-vsctl set port dpdk_tun_port
>> other_config:bond-rebalance-interval=1000
>> triggered the following process, ultimately leading to the
>> USERSPACE_INVALID_PORT_DROP errors:
>>
>> 1. Execution of memset(bond->hash, 0, hash_len);
>> Call stack:
>> #0 bond_entry_reset (bond=0x4c64bc0) at ofproto/bond.c:1852
>> #1 0x0000000001a2a238 in bond_reconfigure (bond=0x4c64bc0, s=0x7fff6d1dec10)
>> at ofproto/bond.c:514
>> #2 0x0000000001a4e253 in bundle_set (ofproto_=0x4c21110,
>> aux=0x4c39d90, s=0x7fff6d1deb90) at ofproto/ofproto-dpif.c:3484
>> #3 0x0000000001a31b27 in ofproto_bundle_register (ofproto=0x4c21110,
>> aux=0x4c39d90, s=0x7fff6d1deb90) at ofproto/ofproto.c:1430
>> #4 0x0000000001a1c80e in port_configure (port=0x4c39d90) at
>> vswitchd/bridge.c:1384
>> #5 0x0000000001a1b7b3 in bridge_reconfigure (ovs_cfg=0x4bb37c0) at
>> vswitchd/bridge.c:1005
>> #6 0x0000000001a223e7 in bridge_run () at vswitchd/bridge.c:3423
>> #7 0x0000000001a27b9e in main (argc=11, argv=0x7fff6d1def38) at
>> vswitchd/ovs-vswitchd.c:129
>>
>> 2. Execution of member_map[i] = OFPP_NONE
>> Call stack:
>> #0 bond_add_lb_output_buckets (bond=0x37220f0) at ofproto/bond.c:2135
>> #1 0x0000000001a29b4f in update_recirc_rules__ (bond=0x37220f0) at
>> ofproto/bond.c:356
>> #2 0x0000000001a29ebe in update_recirc_rules (bond=0x37220f0) at
>> ofproto/bond.c:426
>> #3 0x0000000001a2a262 in bond_reconfigure (bond=0x37220f0, s=0x7fffffffe230)
>> at ofproto/bond.c:520
>> #4 0x0000000001a4e292 in bundle_set (ofproto_=0x366afa0,
>> aux=0x3713290, s=0x7fffffffe1b0) at ofproto/ofproto-dpif.c:3484
>> #5 0x0000000001a31b66 in ofproto_bundle_register (ofproto=0x366afa0,
>> aux=0x3713290, s=0x7fffffffe1b0) at ofproto/ofproto.c:1430
>> #6 0x0000000001a1c80e in port_configure (port=0x3713290) at
>> vswitchd/bridge.c:1384
>> #7 0x0000000001a1b7b3 in bridge_reconfigure (ovs_cfg=0x3660180) at
>> vswitchd/bridge.c:1005
>> #8 0x0000000001a223b7 in bridge_run () at vswitchd/bridge.c:3422
>> #9 0x0000000001a27b92 in main (argc=1, argv=0x7fffffffe558)
>>
>> 3.PMD thread sending packets found port_no=0xffffffff
>> Call stack:
>> #0 dp_execute_output_action (pmd=0x7fff68731010, packets_=0x7fff53ff8f50,
>> should_steal=true, port_no=4294967295)
>> at lib/dpif-netdev.c:9273
>> #1 0x0000000001acaf6d in dp_execute_lb_output_action
>> (pmd=0x7fff68731010, packets_=0x7fff53ff9ca0, should_steal=true,
>> bond=1) at lib/dpif-netdev.c:9350
>> #2 0x0000000001acb0b6 in dp_execute_cb (aux_=0x7fff53ff9b30,
>> packets_=0x7fff53ff9ca0, a=0x7fff4800f074, should_steal=true)
>> at lib/dpif-netdev.c:9379
>> #3 0x0000000001b526b5 in odp_execute_actions (dp=0x7fff53ff9b30,
>> batch=0x7fff53ff9ca0, steal=true,
>> actions=0x7fff4800f074, actions_len=8, dp_execute_action=0x1acafc0
>> <dp_execute_cb>) at lib/odp-execute.c:1016
>> #4 0x0000000001acbc8e in dp_netdev_execute_actions (pmd=0x7fff68731010,
>> packets=0x7fff53ff9ca0, should_steal=true,
>> flow=0x7fff4800ea70, actions=0x7fff4800f074, actions_len=8) at
>> lib/dpif-netdev.c:9698
>> #5 0x0000000001ac8133 in packet_batch_per_flow_execute
>> (batch=0x7fff53ff9c90, pmd=0x7fff68731010)
>> at lib/dpif-netdev.c:8338
>> #6 0x0000000001aca3ad in dp_netdev_input__ (pmd=0x7fff68731010,
>> packets=0x7fff53ffbdf0, md_is_valid=false, port_no=4)
>> at lib/dpif-netdev.c:9055
>> #7 0x0000000001aca3ff in dp_netdev_input (pmd=0x7fff68731010,
>> packets=0x7fff53ffbdf0, port_no=4) at lib/dpif-netdev.c:9064
>> #8 0x0000000001ac0da2 in dp_netdev_process_rxq_port (pmd=0x7fff68731010,
>> rxq=0x3720220, port_no=4)
>> at lib/dpif-netdev.c:5690
>> #9 0x0000000001ac566a in pmd_thread_main (f_=0x7fff68731010) at
>> lib/dpif-netdev.c:7334
>> #10 0x0000000001bc4b1b in ovsthread_wrapper (aux_=0x3711920) at
>> lib/ovs-thread.c:422
>> #11 0x00007ffff76f4802 in start_thread () from /lib64/libc.so.6
>> --Type <RET> for more, q to quit, c to continue without paging--
>> #12 0x00007ffff7694314 in clone () from /lib64/libc.so.6
>>
>> The main issue arises from a timing discrepancy between the main
>> thread and the PMD thread when operating on pmd->tx_bonds,
>> which causes the PMD to temporarily resolve the egress interface to
>> 0xffffffff (an invalid value).
>> What solutions does the community propose to address this problem?
>
> Reconfiguring the bonding flows for a simple change in the
> rebalance_interval seems an overkill. It was added so that users could
> disable rebalancing but just increasing or decreasing the interval
> (without initial or final values being zero) should not trigger a bond
> reset.
You mean by doing something like:
diff --git a/ofproto/bond.c b/ofproto/bond.c
index 3859ddca08..86e21607e5 100644
--- a/ofproto/bond.c
+++ b/ofproto/bond.c
@@ -459,8 +459,14 @@ bond_reconfigure(struct bond *bond, const struct
bond_settings *s)
}
if (bond->rebalance_interval != s->rebalance_interval) {
+ /* Recompute the next rebalance interval by moving the next_rebalance
+ * to be offset by the new interval. Then let the rebalance code
+ * trigger a rebalance based on the new details. In this case, if
+ * all that was updated is the rebalance interval, we can skip
+ * triggering the rest of the port reconfigure mechanism. */
+ int old_start_time = bond->next_rebalance - bond->rebalance_interval;
bond->rebalance_interval = s->rebalance_interval;
- revalidate = true;
+ bond->next_rebalance = old_start_time + bond->rebalance_interval;
}
if (bond->balance != s->balance) {
> Also, if bond_reconfigure resets the bond hashes, we should probaly not
> wait until bond_run() calls bond_update_post_recirc_rules__() to initialize
> them. Even for recirc-driven bonds, this makes an initial update of
> post-recirc rules with all hashes being zero.
This might require a bit deeper look, and I consider it a bit different
anyway. The above patch should at least allow for updating the
rebalance interval, but it keeps the idea of deferring the work until
the actual bond run has been invoked. NOTE: I didn't test the above
change in any way. I need to recheck if the reconfigure code is run
inline with the rest of the bond code for the dpif - if not the
bond->next_rebalance is unsafe. Consider this just illustration.
> I'll take a deeper look probably next week.
>
>>
>> our ovs version 2.17.5 lts.
>
> Note 2.17.5 is not EOL.
2.17.11 is the most recent release for 2.17, but our current LTS is 3.3,
and the most recent release is 3.3.5 - so it would be a good idea to
move forward.
> Thanks.
> Adrián
>
> _______________________________________________
> discuss mailing list
> [email protected]
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
_______________________________________________
discuss mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
