Hi, thanks for the scripts, so here's the output for a 24G core dump: https://pastebin.com/hWa3R9Fx there's 271 entries of 4MB - does it seem something we should take a closer look at?
Thanks, Oleg On Tue, Feb 26, 2019 at 3:26 AM Ben Pfaff <b...@ovn.org> wrote: > Some combinations of kernel bonding with Open vSwitch don't necessarily > work that well. I have forgotten which ones are problematic or why. > However, the problems are functional ones (the bonds don't work well), > not operational ones like memory leaks. A memory leak would be a bug > whether or not the configuration used kernel bonds in a way that we > discourage. > > With small OpenFlow flow tables (maybe a few thousand flows) and a small > number of ports (maybe up to 100 or so), using the Linux kernel module > (not DPDK), I'd be surprised to see OVS use more than 100 MB. I'd also > be surprised to see OVS memory usage grow past, say, the first day or > usage. > > With large numbers of OpenFlow flows (e.g. hundreds of thousands), my > rule of thumb is that OVS tends to use about 1 to 2 kB RAM per flow. > > A dump of a 500 MB process could still be enlightening. Usually, when > there is a memory leak, one sees an extraordinary number of unfreed > memory blocks of a single size, and by looking at their size and their > contents one can often figure out what allocated them, and that usually > leads one to figure out why they did not get freed. > > On Mon, Feb 25, 2019 at 09:46:28PM +0000, Fernando Casas Schössow wrote: > > > > I read in a few places that mixing OS networking features (like bonding) > and OVS is not a good idea and that the recommendation is to do everything > at OVS level. That's why I assumed the configuration was not ok (even when > it worked correctly for around two years albeit the high memory usage I > detected). > > > > How many MB of RAM would you consider normal in a small setup like this > one? Just to make myself an idea. > > > > I just finished a maintenance window on this server that required a > reboot. > > Right after reboot ovs-vswitchd is using 14MB of RAM. > > I will keep monitoring the process memory usage usage and report back > after two weeks or so. > > > > Would it make sense to get a process dump for analysis even if memory > usage is not going as high (several GBs) as before the config change? In > other words, if I find that the process memory usage grows up to around > 500MB but then becomes steady and is not growing anymore would it make > sense to collect a dump for analysis? > > > > On lun, feb 25, 2019 at 5:48 PM, Ben Pfaff <b...@ovn.org> wrote: > > Both configurations should work, so probably you did find a bug causing > a memory leak in the former configuration. 464 MB actually sounds like a > lot also. On Sun, Feb 24, 2019 at 02:58:02PM +0000, Fernando Casas Schössow > wrote: > > Hi Ben, In my case I think I found the cause of the issue, and it was > indeed a misconfiguration on my side. Yet I'm not really sure why the > misconfiguration was causing the high memory usage on OVS. The server has 4 > NICs. Bonded in two bonds of two. The problem I think it was that the > bonding was done at OS level (Linux kernel bonding) instead of at OVS > level. So there were two interfaces at OS level (bond0 and bond1) with > bond0 added to OVS as an uplink port. I changed that configuration, removed > all the bonding at OS level and instead created the bonds at OVS level. > Then I restarted the service so I can monitor memory usage. After this > change, memory usage growth from 10MB (at service start) to 464MB after a > few hours and then stayed at that level until today (a week later). I'm > still monitoring the process memory usage but as I said is steady for > almost a week so I will keep monitoring it for a couple more weeks just in > case and report back. Thanks. Kind regards, Fernando On sáb, feb 23, 2019 > at 12:23 AM, Ben Pfaff <b...@ovn.org<mailto:b...@ovn.org>> wrote: It's odd > that two people would notice the same problem at the same time on old > branches. Anyway, I'm attaching the scripts I have. They are rough. The > second one invokes the first one as a subprocess; it is probably the one > you should use. I might have to walk you through how to use it, or write > better documentation myself. Anyway, it should be a start. On Wed, Feb 20, > 2019 at 07:15:26PM +0400, Oleg Bondarev wrote: Ah, sorry, I missed > "ovs-vswitchd memory consumption behavior" thread. So I guess I'm also > interested in the scripts for analyzing the heap in a core dump :) Thanks, > Oleg On Wed, Feb 20, 2019 at 7:00 PM Oleg Bondarev <obonda...@mirantis.com > <mailto:obonda...@mirantis.com><mailto:obonda...@mirantis.com>> wrote: > > Hi, > > OVS 2.8.0, uptime 197 days, 44G RAM. > ovs-appctl memory/show > reports: > "handlers:35 ofconns:4 ports:73 revalidators:13 rules:1099 udpif > > keys:686" > > Similar data on other nodes of the OpenStack cluster. > > Seems usage grows gradually over time. > Are there any known issues, like > > http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-14970? > Please > advise on the best way to debug. > > Thanks, > Oleg > > > _______________________________________________ discuss mailing list > disc...@openvswitch.org<mailto:disc...@openvswitch.org><mailto: > disc...@openvswitch.org> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > > > > >
_______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
