Hi Dumitru, thanks for the quick response!
> On 5 Jul 2023, at 19:53, Dumitru Ceara <dce...@redhat.com> wrote: > > On 7/5/23 17:14, Vladislav Odintsov wrote: >> Hi, >> > > Hi Vladislav, > >> we’ve noticed there is a huge ovn-controller memory consumption introduced >> with [0] comparing to version without its changes in ovn-controller.c part >> (just OVS submodule bump without ovn-controller changes doesn’t trigger such >> behaviour). >> > > Thanks for reporting this! > >> On an empty host connected to working cluster ovn-controller normally >> consumes ~175 MiB RSS, and the same host updated to a version with commit >> [0] consumes 3.3 GiB RSS. The start time and CPU consumption during process >> start of an affected version is higher that for the normal version. >> >> Before upgrade (normal functioning): >> >> #ovn-appctl -t ovn-controller memory/show ;ps-mem -p $(pidof >> ovn-controller)| grep ovn >> idl-cells-OVN_Southbound:343855 idl-cells-Open_vSwitch:14131 >> ofctrl_desired_flow_usage-KB:76 ofctrl_installed_flow_usage-KB:60 >> ofctrl_sb_flow_ref_usage-KB:18 >> 174.2 MiB + 327.5 KiB = 174.5 MiB ovn-controller >> >> After upgrade to an affected version I’ve checked memory report while >> ovn-controller was starting and at some time there was a bigger amount of >> OVN_Southbound cells comparing to "after start" time: >> >> during start: >> >> # ovn-appctl -t ovn-controller memory/show ;ps-mem -p $(pidof >> ovn-controller)| grep ovn >> idl-cells-OVN_Southbound:11388742 idl-cells-Open_vSwitch:14131 >> idl-outstanding-txns-Open_vSwitch:1 >> 3.3 GiB + 327.0 KiB = 3.3 GiB ovn-controller >> >> after start: >> >> # ovn-appctl -t ovn-controller memory/show ;ps-mem -p $(pidof >> ovn-controller)| grep ovn >> idl-cells-OVN_Southbound:343896 idl-cells-Open_vSwitch:14131 >> idl-outstanding-txns-Open_vSwitch:1 >> 3.3 GiB + 327.0 KiB = 3.3 GiB ovn-controller >> >> >> cells during start: >> 11388742 >> >> cells after start: >> 343896 >> > > Are you running with ovn-monitor-all=true on this host? No, it has default false. > > I guess it's unlikely but I'll try just in case: would it be possible to > share the SB database? Unfortunately, no. But I can say it’s about 450 M in size after compaction. And there are about 1M mac_bindings if it’s important :). > >> I guess it could be connected with this problem. Can anyone look at this and >> comment please? >> > > Does the memory usage persist after SB is upgraded too? I see the > number of SB idl-cells goes down eventually which means that eventually > the periodic malloc_trim() call would free up memory. We trim on idle > since > https://github.com/ovn-org/ovn/commit/b4c593d23bd959e98fcc9ada4a973ac933579ded > In this upgrade DB schemas not upgraded, so they’re up to date. > Are you using a different allocator? E.g., jemalloc. No, this issue reproduces with gcc. > >> >> 0: >> https://github.com/ovn-org/ovn/commit/1b0dbde940706e5de6e60221be78a278361fa76d >> >> >> >> Regards, >> Vladislav Odintsov >> > > Regards, > Dumitru > > _______________________________________________ > dev mailing list > d...@openvswitch.org <mailto:d...@openvswitch.org> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev Regards, Vladislav Odintsov _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
