On Wed, May 8, 2024 at 6:01 PM Gavin McKee via discuss <ovs-discuss@openvswitch.org> wrote: > > @Numan Siddique > > If we enable conditional monitoring here , will this help ? > How does transport zones help with something like this ? Do they > limit the amount of processing . We only have a single VM on this > node , so single LSP , Logical Switch etc that is actually needed or > used. > Would v24.03.0 > https://github.com/ovn-org/ovn/commit/1622526ff2102525e1bbf2ca262842c71d6b9b33 > help here ? >
I think conditional monitoring should work as ovn-controllers will not get updates for changes it's not interested. You can easily test this out. I'm not sure about the transport zones. Even if you use transport zones, conditional monitoring should be enabled, otherwise ovn-controller will get updates for all SB DB changes. I don't think the 24.03 commit you linked to would help in your case as the issue is with ovn-controller main thread blocking the pinctrl thread from processing the DNS packet. Numan > Gav > > On Wed, 8 May 2024 at 14:43, Gavin McKee <gavmcke...@googlemail.com> wrote: > > > > Ok so > > > > 1. Customers depend on the internal DNS reccords, so this is needed > > for production operations > > 2. I can take a look at the updates - would using conditional > > monitoring work here? We have ovn-monitor-all=true , would this help > > at all ? > > 3 & 4 . Is that something the community can help with? Is that a > > viable long term fix we could maybe get a patch for ? > > > > Gav > > > > On Wed, 8 May 2024 at 14:30, Numan Siddique <num...@ovn.org> wrote: > > > > > > On Wed, May 8, 2024 at 3:20 PM Gavin McKee via discuss > > > <ovs-discuss@openvswitch.org> wrote: > > > > > > > > Hi, > > > > > > > > Can someone help me understand why this issue occurs > > > > > > > > > > > > ovn-controller 23.09.1 > > > > Open vSwitch Library 3.2.2 > > > > > > > > We have an issue with some machines intermittently unable to resolve > > > > DNS for external domains (example dig +noall +answer > > > > harmonic-openai-canada.openai.azure.com) > > > > > > > > In the OVN controller log I see the following > > > > > > > > 2024-05-08T12:12:35.596Z|30138|inc_proc_eng|INFO|node: > > > > logical_flow_output, recompute (missing handler for input SB_dns) took > > > > 7950ms > > > > 2024-05-08T14:50:29.747Z|30312|inc_proc_eng|INFO|node: > > > > logical_flow_output, recompute (missing handler for input SB_dns) took > > > > 8634ms > > > > 2024-05-08T14:50:46.673Z|30329|inc_proc_eng|INFO|node: > > > > logical_flow_output, recompute (missing handler for input SB_dns) took > > > > 8774ms > > > > 2024-05-08T14:54:40.781Z|30353|inc_proc_eng|INFO|node: > > > > logical_flow_output, recompute (missing handler for input SB_dns) took > > > > 8535ms > > > > 2024-05-08T14:58:43.381Z|30433|inc_proc_eng|INFO|node: > > > > logical_flow_output, recompute (missing handler for input SB_dns) took > > > > 8541ms > > > > 2024-05-08T14:58:56.802Z|30488|inc_proc_eng|INFO|node: > > > > logical_flow_output, recompute (missing handler for input SB_dns) took > > > > 8820ms > > > > 2024-05-08T15:02:50.704Z|30512|inc_proc_eng|INFO|node: > > > > logical_flow_output, recompute (missing handler for input SB_dns) took > > > > 8739ms > > > > 2024-05-08T15:03:05.206Z|30529|inc_proc_eng|INFO|node: > > > > logical_flow_output, recompute (missing handler for input SB_dns) took > > > > 8686ms > > > > 2024-05-08T15:08:39.441Z|30569|inc_proc_eng|INFO|node: > > > > logical_flow_output, recompute (missing handler for input SB_dns) took > > > > 9167ms > > > > 2024-05-08T15:09:09.152Z|30603|inc_proc_eng|INFO|node: > > > > logical_flow_output, recompute (missing handler for input SB_dns) took > > > > 8985ms > > > > 2024-05-08T15:12:14.361Z|30632|inc_proc_eng|INFO|node: > > > > logical_flow_output, recompute (missing handler for input SB_dns) took > > > > 8569ms > > > > 2024-05-08T15:13:52.535Z|30705|inc_proc_eng|INFO|node: > > > > logical_flow_output, recompute (missing handler for input SB_dns) took > > > > 8764ms > > > > 2024-05-08T15:14:53.989Z|30732|inc_proc_eng|INFO|node: > > > > logical_flow_output, recompute (missing handler for input SB_dns) took > > > > 8802ms > > > > 2024-05-08T15:16:30.911Z|30757|inc_proc_eng|INFO|node: > > > > logical_flow_output, recompute (missing handler for input SB_dns) took > > > > 8776ms > > > > 2024-05-08T15:17:09.371Z|30784|inc_proc_eng|INFO|node: > > > > logical_flow_output, recompute (missing handler for input SB_dns) took > > > > 9062ms > > > > > > > > Why would this happen and is there something I can do about it ? Are > > > > there more logs needed ? > > > > > > This indicates that your deployment is creating, updating or deleting a > > > DNS row > > > in the Northbound database and in turn ovn-northd is updating the SB DNS > > > rows. > > > When ovn-controller receives the Southbound DNS updates, it falls back > > > to a full recompute > > > because we are not handling these changes incrementally. Since OVN > > > native DNS is configured > > > in your deployment, each DNS packet is sent to ovn-controller for lookup. > > > Even though a separate pinctrl thread handles packet-ins, dns > > > handling is blocked until > > > the main ovn-controller thread releases a mutex [1]. > > > > > > There are a few ways to resolve this > > > > > > 1. Disable native OVN DNS if you're not using this feature. To > > > disable, don't create any DNS records in the OVN Northbound db. > > > 2. Investigate why your deployment is updating the NB DBS table and > > > avoid it if its not required. > > > 3. Implement a handler for SB DNS so that ovn-controller do not fall > > > back to a full recompute > > > 4. Avoid locking on the mutex for DNS handling in pinctrl thread [1]. > > > > > > (3) or (4) requires code changes. > > > > > > Thanks > > > Numan > > > > > > [1] - https://github.com/ovn-org/ovn/blob/main/controller/pinctrl.c#L3807 > > > > > > > > > > > > > > Gav > > > > _______________________________________________ > > > > discuss mailing list > > > > disc...@openvswitch.org > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > > > > > _______________________________________________ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
