> On 5 Jul 2023, at 20:07, Vladislav Odintsov <odiv...@gmail.com> wrote: > > Hi Dumitru, > > thanks for the quick response! > >> On 5 Jul 2023, at 19:53, Dumitru Ceara <dce...@redhat.com> wrote: >> >> On 7/5/23 17:14, Vladislav Odintsov wrote: >>> Hi, >>> >> >> Hi Vladislav, >> >>> we’ve noticed there is a huge ovn-controller memory consumption introduced >>> with [0] comparing to version without its changes in ovn-controller.c part >>> (just OVS submodule bump without ovn-controller changes doesn’t trigger >>> such behaviour). >>> >> >> Thanks for reporting this! >> >>> On an empty host connected to working cluster ovn-controller normally >>> consumes ~175 MiB RSS, and the same host updated to a version with commit >>> [0] consumes 3.3 GiB RSS. The start time and CPU consumption during process >>> start of an affected version is higher that for the normal version. >>> >>> Before upgrade (normal functioning): >>> >>> #ovn-appctl -t ovn-controller memory/show ;ps-mem -p $(pidof >>> ovn-controller)| grep ovn >>> idl-cells-OVN_Southbound:343855 idl-cells-Open_vSwitch:14131 >>> ofctrl_desired_flow_usage-KB:76 ofctrl_installed_flow_usage-KB:60 >>> ofctrl_sb_flow_ref_usage-KB:18 >>> 174.2 MiB + 327.5 KiB = 174.5 MiB ovn-controller >>> >>> After upgrade to an affected version I’ve checked memory report while >>> ovn-controller was starting and at some time there was a bigger amount of >>> OVN_Southbound cells comparing to "after start" time: >>> >>> during start: >>> >>> # ovn-appctl -t ovn-controller memory/show ;ps-mem -p $(pidof >>> ovn-controller)| grep ovn >>> idl-cells-OVN_Southbound:11388742 idl-cells-Open_vSwitch:14131 >>> idl-outstanding-txns-Open_vSwitch:1 >>> 3.3 GiB + 327.0 KiB = 3.3 GiB ovn-controller >>> >>> after start: >>> >>> # ovn-appctl -t ovn-controller memory/show ;ps-mem -p $(pidof >>> ovn-controller)| grep ovn >>> idl-cells-OVN_Southbound:343896 idl-cells-Open_vSwitch:14131 >>> idl-outstanding-txns-Open_vSwitch:1 >>> 3.3 GiB + 327.0 KiB = 3.3 GiB ovn-controller >>> >>> >>> cells during start: >>> 11388742 >>> >>> cells after start: >>> 343896 >>> >> >> Are you running with ovn-monitor-all=true on this host? > > No, it has default false. > >> >> I guess it's unlikely but I'll try just in case: would it be possible to >> share the SB database? > > Unfortunately, no. But I can say it’s about 450 M in size after compaction. > And there are about 1M mac_bindings if it’s important :).
But if you are interested in any specific information, let me know it, I can check. > >> >>> I guess it could be connected with this problem. Can anyone look at this >>> and comment please? >>> >> >> Does the memory usage persist after SB is upgraded too? I see the >> number of SB idl-cells goes down eventually which means that eventually >> the periodic malloc_trim() call would free up memory. We trim on idle >> since >> https://github.com/ovn-org/ovn/commit/b4c593d23bd959e98fcc9ada4a973ac933579ded >> > > In this upgrade DB schemas not upgraded, so they’re up to date. > >> Are you using a different allocator? E.g., jemalloc. > > No, this issue reproduces with gcc. > >> >>> >>> 0: >>> https://github.com/ovn-org/ovn/commit/1b0dbde940706e5de6e60221be78a278361fa76d >>> >>> >>> >>> Regards, >>> Vladislav Odintsov >>> >> >> Regards, >> Dumitru >> >> _______________________________________________ >> dev mailing list >> d...@openvswitch.org <mailto:d...@openvswitch.org> >> <mailto:d...@openvswitch.org> >> https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > Regards, > Vladislav Odintsov > > _______________________________________________ > dev mailing list > d...@openvswitch.org <mailto:d...@openvswitch.org> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev Regards, Vladislav Odintsov _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
