Re: [ovs-discuss] ovs-vswitchd process huge memory consumption

[Oleg Bondarev]

Hi Ben,

So I decided to try find the leak with help of Address Sanitizer as you
recommended.
Would the following approach make sense?:

 - build OVS from sources with flags CFLAGS="-g -O2 -fsanitize=leak
-fno-omit-frame-pointer -fno-common"
 - service openvswitch-switch stop
 - replace binary */usr/lib/openvswitch-switch/ovs-vswitchd*
 - service openvswitch-switch start
 - load the cluster to cause possible mem leaks
 - look for address sanitizer logs in syslog

Thanks,
Oleg

On Thu, Mar 7, 2019 at 1:33 PM Oleg Bondarev  wrote:

> Hi Ben,
>
> attaching the full dump-heap script output for a 11G core dump, probably
> it can bring some more clarity.
>
> Thanks,
> Oleg
>
> On Thu, Mar 7, 2019 at 11:54 AM Oleg Bondarev 
> wrote:
>
>>
>>
>> On Wed, Mar 6, 2019 at 7:01 PM Oleg Bondarev 
>> wrote:
>>
>>>
>>> I'm thinking if this can be malloc() not returning memory to the system
>>> after peak loads:
>>> *"Occasionally, free can actually return memory to the operating system
>>> and make the process smaller. Usually, all it can do is allow a later call
>>> to malloc to reuse the space. In the meantime, the space remains in your
>>> program as part of a free-list used internally by malloc." [1]*
>>>
>>> Does it sound sane? If yes, what would be a best way to check that?
>>>
>>
>> Seems that's not the case. On one of the nodes memory usage by
>> ovs-vswitchd grew from 84G to 87G for the past week, and on other nodes
>> grows gradually as well.
>>
>>
>>>
>>> [1] http://www.gnu.org/software/libc/manual/pdf/libc.pdf
>>>
>>> Thanks,
>>> Oleg
>>>
>>> On Wed, Mar 6, 2019 at 12:34 PM Oleg Bondarev 
>>> wrote:
>>>
 Hi,

 On Wed, Mar 6, 2019 at 1:08 AM Ben Pfaff  wrote:

> Starting from 0x30, this looks like a "minimatch" data structure, which
> is a kind of compressed bitwise match against a flow.
>
> 0030:    4014    
> 0040:     fa16 3e2b c5d5   0022  
>
> 0058:    4014    
> 0068:          0fff  
>
> I think this corresponds to a flow of this form:
>
>
> pkt_mark=0xc5d5/0x,skb_priority=0x3e2bfa16,reg13=0,mpls_label=2,mpls_tc=1,mpls_ttl=0,mpls_bos=0
>
> Is that at all meaningful?  Does it match anything that appears in the
> OpenFlow flow table?
>

 Not sure, actually fa:16:3e:2b:c5:d5 is a mac address of a neutron port
 (this is an OpenStack cluster) - the port is a VM port.
 fa:16:3e/fa:16:3f - are standard neutron mac prefixes. That makes me
 think those might be some actual eth packets (broadcasts?) that somehow
 stuck in memory..
 So I didn't find anything similar in the flow tables. I'm attaching
 flows of all 5 OVS bridges on the node.


>
> Are you using the kernel or DPDK datapath?
>

 It's kernel datapath, no DPDK. Ubuntu with 4.13.0-45  kernel.


>
> On Tue, Mar 05, 2019 at 08:42:14PM +0400, Oleg Bondarev wrote:
> > Hi,
> >
> > thanks for your help!
> >
> > On Tue, Mar 5, 2019 at 7:26 PM Ben Pfaff  wrote:
> >
> > > You're talking about the email where you dumped out a repeating
> sequence
> > > from some blocks?  That might be the root of the problem, if you
> can
> > > provide some more context.  I didn't see from the message where you
> > > found the sequence (was it just at the beginning of each of the 4
> MB
> > > blocks you reported separately, or somewhere else), how many
> copies of
> > > it, or if you were able to figure out how long each of the blocks
> was.
> > > If you can provide that information I might be able to learn some
> > > things.
> > >
> >
> > Yes, those were beginnings of 0x400 size blocks reported by the
> script.
> > I also checked 0x800 blocks reported and the content is the same.
> > Examples of how those blocks end:
> >  - https://pastebin.com/D9M6T2BA
> >  - https://pastebin.com/gNT7XEGn
> >  - https://pastebin.com/fqy4XDbN
> >
> > So basically contents of the blocks are sequences of:
> >
> > *0020:     6500     e...*
> > *0030:    4014      ..@.*
> > *0040:     fa16 3e2b c5d5   ..>+*
> > *0050:  0022      4014  ..."..@.*
> > *0060:          *
> > *0070:      0fff    *
> >
> > following each other and sometimes separated by sequences like this:
> >
> > *1040: 6861 6e64 6c65 7232 3537     handler257..*
> >
> > I ran the scripts against several core dumps of several compute
> nodes with
> > the issue and
> > the 

Re: [ovs-discuss] ovs-vswitchd process huge memory consumption

[Oleg Bondarev]

On Wed, Mar 6, 2019 at 7:01 PM Oleg Bondarev  wrote:

>
> I'm thinking if this can be malloc() not returning memory to the system
> after peak loads:
> *"Occasionally, free can actually return memory to the operating system
> and make the process smaller. Usually, all it can do is allow a later call
> to malloc to reuse the space. In the meantime, the space remains in your
> program as part of a free-list used internally by malloc." [1]*
>
> Does it sound sane? If yes, what would be a best way to check that?
>

Seems that's not the case. On one of the nodes memory usage by ovs-vswitchd
grew from 84G to 87G for the past week, and on other nodes grows gradually
as well.


>
> [1] http://www.gnu.org/software/libc/manual/pdf/libc.pdf
>
> Thanks,
> Oleg
>
> On Wed, Mar 6, 2019 at 12:34 PM Oleg Bondarev 
> wrote:
>
>> Hi,
>>
>> On Wed, Mar 6, 2019 at 1:08 AM Ben Pfaff  wrote:
>>
>>> Starting from 0x30, this looks like a "minimatch" data structure, which
>>> is a kind of compressed bitwise match against a flow.
>>>
>>> 0030:    4014    
>>> 0040:     fa16 3e2b c5d5   0022  
>>>
>>> 0058:    4014    
>>> 0068:          0fff  
>>>
>>> I think this corresponds to a flow of this form:
>>>
>>>
>>> pkt_mark=0xc5d5/0x,skb_priority=0x3e2bfa16,reg13=0,mpls_label=2,mpls_tc=1,mpls_ttl=0,mpls_bos=0
>>>
>>> Is that at all meaningful?  Does it match anything that appears in the
>>> OpenFlow flow table?
>>>
>>
>> Not sure, actually fa:16:3e:2b:c5:d5 is a mac address of a neutron port
>> (this is an OpenStack cluster) - the port is a VM port.
>> fa:16:3e/fa:16:3f - are standard neutron mac prefixes. That makes me
>> think those might be some actual eth packets (broadcasts?) that somehow
>> stuck in memory..
>> So I didn't find anything similar in the flow tables. I'm attaching flows
>> of all 5 OVS bridges on the node.
>>
>>
>>>
>>> Are you using the kernel or DPDK datapath?
>>>
>>
>> It's kernel datapath, no DPDK. Ubuntu with 4.13.0-45  kernel.
>>
>>
>>>
>>> On Tue, Mar 05, 2019 at 08:42:14PM +0400, Oleg Bondarev wrote:
>>> > Hi,
>>> >
>>> > thanks for your help!
>>> >
>>> > On Tue, Mar 5, 2019 at 7:26 PM Ben Pfaff  wrote:
>>> >
>>> > > You're talking about the email where you dumped out a repeating
>>> sequence
>>> > > from some blocks?  That might be the root of the problem, if you can
>>> > > provide some more context.  I didn't see from the message where you
>>> > > found the sequence (was it just at the beginning of each of the 4 MB
>>> > > blocks you reported separately, or somewhere else), how many copies
>>> of
>>> > > it, or if you were able to figure out how long each of the blocks
>>> was.
>>> > > If you can provide that information I might be able to learn some
>>> > > things.
>>> > >
>>> >
>>> > Yes, those were beginnings of 0x400 size blocks reported by the
>>> script.
>>> > I also checked 0x800 blocks reported and the content is the same.
>>> > Examples of how those blocks end:
>>> >  - https://pastebin.com/D9M6T2BA
>>> >  - https://pastebin.com/gNT7XEGn
>>> >  - https://pastebin.com/fqy4XDbN
>>> >
>>> > So basically contents of the blocks are sequences of:
>>> >
>>> > *0020:     6500     e...*
>>> > *0030:    4014      ..@.*
>>> > *0040:     fa16 3e2b c5d5   ..>+*
>>> > *0050:  0022      4014  ..."..@.*
>>> > *0060:          *
>>> > *0070:      0fff    *
>>> >
>>> > following each other and sometimes separated by sequences like this:
>>> >
>>> > *1040: 6861 6e64 6c65 7232 3537     handler257..*
>>> >
>>> > I ran the scripts against several core dumps of several compute nodes
>>> with
>>> > the issue and
>>> > the picture is pretty much the same: 0x400 blocks and less
>>> 0x800
>>> > blocks.
>>> > I checked the core dump from a compute node where OVS memory
>>> consumption
>>> > was ok:
>>> > no such block sizes reported.
>>> >
>>> >
>>> > >
>>> > > On Tue, Mar 05, 2019 at 09:07:55AM +0400, Oleg Bondarev wrote:
>>> > > > Hi Ben,
>>> > > >
>>> > > > I didn't have a chance to debug the scripts yet, but just in case
>>> you
>>> > > > missed my last email with examples of repeatable blocks
>>> > > > and sequences - do you think we still need to analyze further,
>>> will the
>>> > > > scripts tell more about the heap?
>>> > > >
>>> > > > Thanks,
>>> > > > Oleg
>>> > > >
>>> > > > On Thu, Feb 28, 2019 at 10:14 PM Ben Pfaff  wrote:
>>> > > >
>>> > > > > On Tue, Feb 26, 2019 at 01:41:45PM +0400, Oleg Bondarev wrote:
>>> > > > > > Hi,
>>> > > > > >
>>> > > > > > thanks for the scripts, so here's the output for a 24G core
>>> dump:
>>> > > > > > https://pastebin.com/hWa3R9Fx
>>> > > 

Re: [ovs-discuss] ovs-vswitchd process huge memory consumption

[Oleg Bondarev]

I'm thinking if this can be malloc() not returning memory to the system
after peak loads:
*"Occasionally, free can actually return memory to the operating system and
make the process smaller. Usually, all it can do is allow a later call to
malloc to reuse the space. In the meantime, the space remains in your
program as part of a free-list used internally by malloc." [1]*

Does it sound sane? If yes, what would be a best way to check that?

[1] http://www.gnu.org/software/libc/manual/pdf/libc.pdf

Thanks,
Oleg

On Wed, Mar 6, 2019 at 12:34 PM Oleg Bondarev 
wrote:

> Hi,
>
> On Wed, Mar 6, 2019 at 1:08 AM Ben Pfaff  wrote:
>
>> Starting from 0x30, this looks like a "minimatch" data structure, which
>> is a kind of compressed bitwise match against a flow.
>>
>> 0030:    4014    
>> 0040:     fa16 3e2b c5d5   0022  
>>
>> 0058:    4014    
>> 0068:          0fff  
>>
>> I think this corresponds to a flow of this form:
>>
>>
>> pkt_mark=0xc5d5/0x,skb_priority=0x3e2bfa16,reg13=0,mpls_label=2,mpls_tc=1,mpls_ttl=0,mpls_bos=0
>>
>> Is that at all meaningful?  Does it match anything that appears in the
>> OpenFlow flow table?
>>
>
> Not sure, actually fa:16:3e:2b:c5:d5 is a mac address of a neutron port
> (this is an OpenStack cluster) - the port is a VM port.
> fa:16:3e/fa:16:3f - are standard neutron mac prefixes. That makes me think
> those might be some actual eth packets (broadcasts?) that somehow
> stuck in memory..
> So I didn't find anything similar in the flow tables. I'm attaching flows
> of all 5 OVS bridges on the node.
>
>
>>
>> Are you using the kernel or DPDK datapath?
>>
>
> It's kernel datapath, no DPDK. Ubuntu with 4.13.0-45  kernel.
>
>
>>
>> On Tue, Mar 05, 2019 at 08:42:14PM +0400, Oleg Bondarev wrote:
>> > Hi,
>> >
>> > thanks for your help!
>> >
>> > On Tue, Mar 5, 2019 at 7:26 PM Ben Pfaff  wrote:
>> >
>> > > You're talking about the email where you dumped out a repeating
>> sequence
>> > > from some blocks?  That might be the root of the problem, if you can
>> > > provide some more context.  I didn't see from the message where you
>> > > found the sequence (was it just at the beginning of each of the 4 MB
>> > > blocks you reported separately, or somewhere else), how many copies of
>> > > it, or if you were able to figure out how long each of the blocks was.
>> > > If you can provide that information I might be able to learn some
>> > > things.
>> > >
>> >
>> > Yes, those were beginnings of 0x400 size blocks reported by the
>> script.
>> > I also checked 0x800 blocks reported and the content is the same.
>> > Examples of how those blocks end:
>> >  - https://pastebin.com/D9M6T2BA
>> >  - https://pastebin.com/gNT7XEGn
>> >  - https://pastebin.com/fqy4XDbN
>> >
>> > So basically contents of the blocks are sequences of:
>> >
>> > *0020:     6500     e...*
>> > *0030:    4014      ..@.*
>> > *0040:     fa16 3e2b c5d5   ..>+*
>> > *0050:  0022      4014  ..."..@.*
>> > *0060:          *
>> > *0070:      0fff    *
>> >
>> > following each other and sometimes separated by sequences like this:
>> >
>> > *1040: 6861 6e64 6c65 7232 3537     handler257..*
>> >
>> > I ran the scripts against several core dumps of several compute nodes
>> with
>> > the issue and
>> > the picture is pretty much the same: 0x400 blocks and less 0x800
>> > blocks.
>> > I checked the core dump from a compute node where OVS memory consumption
>> > was ok:
>> > no such block sizes reported.
>> >
>> >
>> > >
>> > > On Tue, Mar 05, 2019 at 09:07:55AM +0400, Oleg Bondarev wrote:
>> > > > Hi Ben,
>> > > >
>> > > > I didn't have a chance to debug the scripts yet, but just in case
>> you
>> > > > missed my last email with examples of repeatable blocks
>> > > > and sequences - do you think we still need to analyze further, will
>> the
>> > > > scripts tell more about the heap?
>> > > >
>> > > > Thanks,
>> > > > Oleg
>> > > >
>> > > > On Thu, Feb 28, 2019 at 10:14 PM Ben Pfaff  wrote:
>> > > >
>> > > > > On Tue, Feb 26, 2019 at 01:41:45PM +0400, Oleg Bondarev wrote:
>> > > > > > Hi,
>> > > > > >
>> > > > > > thanks for the scripts, so here's the output for a 24G core
>> dump:
>> > > > > > https://pastebin.com/hWa3R9Fx
>> > > > > > there's 271 entries of 4MB - does it seem something we should
>> take a
>> > > > > closer
>> > > > > > look at?
>> > > > >
>> > > > > I think that this output really just indicates that the script
>> failed.
>> > > > > It analyzed a lot of regions but didn't output anything useful.
>> If it
>> > > > > had worked properly, it would have told us a lot about data
>> 

Re: [ovs-discuss] ovs-vswitchd process huge memory consumption

[Ben Pfaff]

Starting from 0x30, this looks like a "minimatch" data structure, which
is a kind of compressed bitwise match against a flow.

0030:    4014    
0040:     fa16 3e2b c5d5   0022   
   
0058:    4014    
0068:          0fff  

I think this corresponds to a flow of this form:

pkt_mark=0xc5d5/0x,skb_priority=0x3e2bfa16,reg13=0,mpls_label=2,mpls_tc=1,mpls_ttl=0,mpls_bos=0

Is that at all meaningful?  Does it match anything that appears in the
OpenFlow flow table?

Are you using the kernel or DPDK datapath?

On Tue, Mar 05, 2019 at 08:42:14PM +0400, Oleg Bondarev wrote:
> Hi,
> 
> thanks for your help!
> 
> On Tue, Mar 5, 2019 at 7:26 PM Ben Pfaff  wrote:
> 
> > You're talking about the email where you dumped out a repeating sequence
> > from some blocks?  That might be the root of the problem, if you can
> > provide some more context.  I didn't see from the message where you
> > found the sequence (was it just at the beginning of each of the 4 MB
> > blocks you reported separately, or somewhere else), how many copies of
> > it, or if you were able to figure out how long each of the blocks was.
> > If you can provide that information I might be able to learn some
> > things.
> >
> 
> Yes, those were beginnings of 0x400 size blocks reported by the script.
> I also checked 0x800 blocks reported and the content is the same.
> Examples of how those blocks end:
>  - https://pastebin.com/D9M6T2BA
>  - https://pastebin.com/gNT7XEGn
>  - https://pastebin.com/fqy4XDbN
> 
> So basically contents of the blocks are sequences of:
> 
> *0020:     6500     e...*
> *0030:    4014      ..@.*
> *0040:     fa16 3e2b c5d5   ..>+*
> *0050:  0022      4014  ..."..@.*
> *0060:          *
> *0070:      0fff    *
> 
> following each other and sometimes separated by sequences like this:
> 
> *1040: 6861 6e64 6c65 7232 3537     handler257..*
> 
> I ran the scripts against several core dumps of several compute nodes with
> the issue and
> the picture is pretty much the same: 0x400 blocks and less 0x800
> blocks.
> I checked the core dump from a compute node where OVS memory consumption
> was ok:
> no such block sizes reported.
> 
> 
> >
> > On Tue, Mar 05, 2019 at 09:07:55AM +0400, Oleg Bondarev wrote:
> > > Hi Ben,
> > >
> > > I didn't have a chance to debug the scripts yet, but just in case you
> > > missed my last email with examples of repeatable blocks
> > > and sequences - do you think we still need to analyze further, will the
> > > scripts tell more about the heap?
> > >
> > > Thanks,
> > > Oleg
> > >
> > > On Thu, Feb 28, 2019 at 10:14 PM Ben Pfaff  wrote:
> > >
> > > > On Tue, Feb 26, 2019 at 01:41:45PM +0400, Oleg Bondarev wrote:
> > > > > Hi,
> > > > >
> > > > > thanks for the scripts, so here's the output for a 24G core dump:
> > > > > https://pastebin.com/hWa3R9Fx
> > > > > there's 271 entries of 4MB - does it seem something we should take a
> > > > closer
> > > > > look at?
> > > >
> > > > I think that this output really just indicates that the script failed.
> > > > It analyzed a lot of regions but didn't output anything useful.  If it
> > > > had worked properly, it would have told us a lot about data blocks that
> > > > had been allocated and freed.
> > > >
> > > > The next step would have to be to debug the script.  It definitely
> > > > worked for me before, because I have fixed at least 3 or 4 bugs based
> > on
> > > > it, but it also definitely is a quick hack and not something that I can
> > > > stand behind.  I'm not sure how to debug it at a distance.  It has a
> > > > large comment that describes what it's trying to do.  Maybe that would
> > > > help you, if you want to try to debug it yourself.  I guess it's also
> > > > possible that glibc has changed its malloc implementation; if so, then
> > > > it would probably be necessary to start over and build a new script.
> > > >
> >
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] ovs-vswitchd process huge memory consumption

[Oleg Bondarev]

On Tue, Mar 5, 2019 at 9:00 PM Ben Pfaff  wrote:

> OK.  That gives me something to investigate.  I'll see what I can find
> out.
>
> You're running 64-bit kernel and userspace, x86-64?
>

Yes.


>
> On Tue, Mar 05, 2019 at 08:42:14PM +0400, Oleg Bondarev wrote:
> > On Tue, Mar 5, 2019 at 7:26 PM Ben Pfaff  wrote:
> >
> > > You're talking about the email where you dumped out a repeating
> sequence
> > > from some blocks?  That might be the root of the problem, if you can
> > > provide some more context.  I didn't see from the message where you
> > > found the sequence (was it just at the beginning of each of the 4 MB
> > > blocks you reported separately, or somewhere else), how many copies of
> > > it, or if you were able to figure out how long each of the blocks was.
> > > If you can provide that information I might be able to learn some
> > > things.
> > >
> >
> > Yes, those were beginnings of 0x400 size blocks reported by the
> script.
> > I also checked 0x800 blocks reported and the content is the same.
> > Examples of how those blocks end:
> >  - https://pastebin.com/D9M6T2BA
> >  - https://pastebin.com/gNT7XEGn
> >  - https://pastebin.com/fqy4XDbN
> >
> > So basically contents of the blocks are sequences of:
> >
> > *0020:     6500     e...*
> > *0030:    4014      ..@.*
> > *0040:     fa16 3e2b c5d5   ..>+*
> > *0050:  0022      4014  ..."..@.*
> > *0060:          *
> > *0070:      0fff    *
> >
> > following each other and sometimes separated by sequences like this:
> >
> > *1040: 6861 6e64 6c65 7232 3537     handler257..*
> >
> > I ran the scripts against several core dumps of several compute nodes
> with
> > the issue and
> > the picture is pretty much the same: 0x400 blocks and less 0x800
> > blocks.
> > I checked the core dump from a compute node where OVS memory consumption
> > was ok:
> > no such block sizes reported.
> >
> >
> > >
> > > On Tue, Mar 05, 2019 at 09:07:55AM +0400, Oleg Bondarev wrote:
> > > > Hi Ben,
> > > >
> > > > I didn't have a chance to debug the scripts yet, but just in case you
> > > > missed my last email with examples of repeatable blocks
> > > > and sequences - do you think we still need to analyze further, will
> the
> > > > scripts tell more about the heap?
> > > >
> > > > Thanks,
> > > > Oleg
> > > >
> > > > On Thu, Feb 28, 2019 at 10:14 PM Ben Pfaff  wrote:
> > > >
> > > > > On Tue, Feb 26, 2019 at 01:41:45PM +0400, Oleg Bondarev wrote:
> > > > > > Hi,
> > > > > >
> > > > > > thanks for the scripts, so here's the output for a 24G core dump:
> > > > > > https://pastebin.com/hWa3R9Fx
> > > > > > there's 271 entries of 4MB - does it seem something we should
> take a
> > > > > closer
> > > > > > look at?
> > > > >
> > > > > I think that this output really just indicates that the script
> failed.
> > > > > It analyzed a lot of regions but didn't output anything useful.
> If it
> > > > > had worked properly, it would have told us a lot about data blocks
> that
> > > > > had been allocated and freed.
> > > > >
> > > > > The next step would have to be to debug the script.  It definitely
> > > > > worked for me before, because I have fixed at least 3 or 4 bugs
> based
> > > on
> > > > > it, but it also definitely is a quick hack and not something that
> I can
> > > > > stand behind.  I'm not sure how to debug it at a distance.  It has
> a
> > > > > large comment that describes what it's trying to do.  Maybe that
> would
> > > > > help you, if you want to try to debug it yourself.  I guess it's
> also
> > > > > possible that glibc has changed its malloc implementation; if so,
> then
> > > > > it would probably be necessary to start over and build a new
> script.
> > > > >
> > >
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] ovs-vswitchd process huge memory consumption

[Ben Pfaff]

OK.  That gives me something to investigate.  I'll see what I can find
out.

You're running 64-bit kernel and userspace, x86-64?

On Tue, Mar 05, 2019 at 08:42:14PM +0400, Oleg Bondarev wrote:
> On Tue, Mar 5, 2019 at 7:26 PM Ben Pfaff  wrote:
> 
> > You're talking about the email where you dumped out a repeating sequence
> > from some blocks?  That might be the root of the problem, if you can
> > provide some more context.  I didn't see from the message where you
> > found the sequence (was it just at the beginning of each of the 4 MB
> > blocks you reported separately, or somewhere else), how many copies of
> > it, or if you were able to figure out how long each of the blocks was.
> > If you can provide that information I might be able to learn some
> > things.
> >
> 
> Yes, those were beginnings of 0x400 size blocks reported by the script.
> I also checked 0x800 blocks reported and the content is the same.
> Examples of how those blocks end:
>  - https://pastebin.com/D9M6T2BA
>  - https://pastebin.com/gNT7XEGn
>  - https://pastebin.com/fqy4XDbN
> 
> So basically contents of the blocks are sequences of:
> 
> *0020:     6500     e...*
> *0030:    4014      ..@.*
> *0040:     fa16 3e2b c5d5   ..>+*
> *0050:  0022      4014  ..."..@.*
> *0060:          *
> *0070:      0fff    *
> 
> following each other and sometimes separated by sequences like this:
> 
> *1040: 6861 6e64 6c65 7232 3537     handler257..*
> 
> I ran the scripts against several core dumps of several compute nodes with
> the issue and
> the picture is pretty much the same: 0x400 blocks and less 0x800
> blocks.
> I checked the core dump from a compute node where OVS memory consumption
> was ok:
> no such block sizes reported.
> 
> 
> >
> > On Tue, Mar 05, 2019 at 09:07:55AM +0400, Oleg Bondarev wrote:
> > > Hi Ben,
> > >
> > > I didn't have a chance to debug the scripts yet, but just in case you
> > > missed my last email with examples of repeatable blocks
> > > and sequences - do you think we still need to analyze further, will the
> > > scripts tell more about the heap?
> > >
> > > Thanks,
> > > Oleg
> > >
> > > On Thu, Feb 28, 2019 at 10:14 PM Ben Pfaff  wrote:
> > >
> > > > On Tue, Feb 26, 2019 at 01:41:45PM +0400, Oleg Bondarev wrote:
> > > > > Hi,
> > > > >
> > > > > thanks for the scripts, so here's the output for a 24G core dump:
> > > > > https://pastebin.com/hWa3R9Fx
> > > > > there's 271 entries of 4MB - does it seem something we should take a
> > > > closer
> > > > > look at?
> > > >
> > > > I think that this output really just indicates that the script failed.
> > > > It analyzed a lot of regions but didn't output anything useful.  If it
> > > > had worked properly, it would have told us a lot about data blocks that
> > > > had been allocated and freed.
> > > >
> > > > The next step would have to be to debug the script.  It definitely
> > > > worked for me before, because I have fixed at least 3 or 4 bugs based
> > on
> > > > it, but it also definitely is a quick hack and not something that I can
> > > > stand behind.  I'm not sure how to debug it at a distance.  It has a
> > > > large comment that describes what it's trying to do.  Maybe that would
> > > > help you, if you want to try to debug it yourself.  I guess it's also
> > > > possible that glibc has changed its malloc implementation; if so, then
> > > > it would probably be necessary to start over and build a new script.
> > > >
> >
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] ovs-vswitchd process huge memory consumption

[Oleg Bondarev]

Hi,

thanks for your help!

On Tue, Mar 5, 2019 at 7:26 PM Ben Pfaff  wrote:

> You're talking about the email where you dumped out a repeating sequence
> from some blocks?  That might be the root of the problem, if you can
> provide some more context.  I didn't see from the message where you
> found the sequence (was it just at the beginning of each of the 4 MB
> blocks you reported separately, or somewhere else), how many copies of
> it, or if you were able to figure out how long each of the blocks was.
> If you can provide that information I might be able to learn some
> things.
>

Yes, those were beginnings of 0x400 size blocks reported by the script.
I also checked 0x800 blocks reported and the content is the same.
Examples of how those blocks end:
 - https://pastebin.com/D9M6T2BA
 - https://pastebin.com/gNT7XEGn
 - https://pastebin.com/fqy4XDbN

So basically contents of the blocks are sequences of:

*0020:     6500     e...*
*0030:    4014      ..@.*
*0040:     fa16 3e2b c5d5   ..>+*
*0050:  0022      4014  ..."..@.*
*0060:          *
*0070:      0fff    *

following each other and sometimes separated by sequences like this:

*1040: 6861 6e64 6c65 7232 3537     handler257..*

I ran the scripts against several core dumps of several compute nodes with
the issue and
the picture is pretty much the same: 0x400 blocks and less 0x800
blocks.
I checked the core dump from a compute node where OVS memory consumption
was ok:
no such block sizes reported.


>
> On Tue, Mar 05, 2019 at 09:07:55AM +0400, Oleg Bondarev wrote:
> > Hi Ben,
> >
> > I didn't have a chance to debug the scripts yet, but just in case you
> > missed my last email with examples of repeatable blocks
> > and sequences - do you think we still need to analyze further, will the
> > scripts tell more about the heap?
> >
> > Thanks,
> > Oleg
> >
> > On Thu, Feb 28, 2019 at 10:14 PM Ben Pfaff  wrote:
> >
> > > On Tue, Feb 26, 2019 at 01:41:45PM +0400, Oleg Bondarev wrote:
> > > > Hi,
> > > >
> > > > thanks for the scripts, so here's the output for a 24G core dump:
> > > > https://pastebin.com/hWa3R9Fx
> > > > there's 271 entries of 4MB - does it seem something we should take a
> > > closer
> > > > look at?
> > >
> > > I think that this output really just indicates that the script failed.
> > > It analyzed a lot of regions but didn't output anything useful.  If it
> > > had worked properly, it would have told us a lot about data blocks that
> > > had been allocated and freed.
> > >
> > > The next step would have to be to debug the script.  It definitely
> > > worked for me before, because I have fixed at least 3 or 4 bugs based
> on
> > > it, but it also definitely is a quick hack and not something that I can
> > > stand behind.  I'm not sure how to debug it at a distance.  It has a
> > > large comment that describes what it's trying to do.  Maybe that would
> > > help you, if you want to try to debug it yourself.  I guess it's also
> > > possible that glibc has changed its malloc implementation; if so, then
> > > it would probably be necessary to start over and build a new script.
> > >
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] ovs-vswitchd process huge memory consumption

[Ben Pfaff]

You're talking about the email where you dumped out a repeating sequence
from some blocks?  That might be the root of the problem, if you can
provide some more context.  I didn't see from the message where you
found the sequence (was it just at the beginning of each of the 4 MB
blocks you reported separately, or somewhere else), how many copies of
it, or if you were able to figure out how long each of the blocks was.
If you can provide that information I might be able to learn some
things.

On Tue, Mar 05, 2019 at 09:07:55AM +0400, Oleg Bondarev wrote:
> Hi Ben,
> 
> I didn't have a chance to debug the scripts yet, but just in case you
> missed my last email with examples of repeatable blocks
> and sequences - do you think we still need to analyze further, will the
> scripts tell more about the heap?
> 
> Thanks,
> Oleg
> 
> On Thu, Feb 28, 2019 at 10:14 PM Ben Pfaff  wrote:
> 
> > On Tue, Feb 26, 2019 at 01:41:45PM +0400, Oleg Bondarev wrote:
> > > Hi,
> > >
> > > thanks for the scripts, so here's the output for a 24G core dump:
> > > https://pastebin.com/hWa3R9Fx
> > > there's 271 entries of 4MB - does it seem something we should take a
> > closer
> > > look at?
> >
> > I think that this output really just indicates that the script failed.
> > It analyzed a lot of regions but didn't output anything useful.  If it
> > had worked properly, it would have told us a lot about data blocks that
> > had been allocated and freed.
> >
> > The next step would have to be to debug the script.  It definitely
> > worked for me before, because I have fixed at least 3 or 4 bugs based on
> > it, but it also definitely is a quick hack and not something that I can
> > stand behind.  I'm not sure how to debug it at a distance.  It has a
> > large comment that describes what it's trying to do.  Maybe that would
> > help you, if you want to try to debug it yourself.  I guess it's also
> > possible that glibc has changed its malloc implementation; if so, then
> > it would probably be necessary to start over and build a new script.
> >
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] ovs-vswitchd process huge memory consumption

[Oleg Bondarev]

Hi Ben,

I didn't have a chance to debug the scripts yet, but just in case you
missed my last email with examples of repeatable blocks
and sequences - do you think we still need to analyze further, will the
scripts tell more about the heap?

Thanks,
Oleg

On Thu, Feb 28, 2019 at 10:14 PM Ben Pfaff  wrote:

> On Tue, Feb 26, 2019 at 01:41:45PM +0400, Oleg Bondarev wrote:
> > Hi,
> >
> > thanks for the scripts, so here's the output for a 24G core dump:
> > https://pastebin.com/hWa3R9Fx
> > there's 271 entries of 4MB - does it seem something we should take a
> closer
> > look at?
>
> I think that this output really just indicates that the script failed.
> It analyzed a lot of regions but didn't output anything useful.  If it
> had worked properly, it would have told us a lot about data blocks that
> had been allocated and freed.
>
> The next step would have to be to debug the script.  It definitely
> worked for me before, because I have fixed at least 3 or 4 bugs based on
> it, but it also definitely is a quick hack and not something that I can
> stand behind.  I'm not sure how to debug it at a distance.  It has a
> large comment that describes what it's trying to do.  Maybe that would
> help you, if you want to try to debug it yourself.  I guess it's also
> possible that glibc has changed its malloc implementation; if so, then
> it would probably be necessary to start over and build a new script.
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] ovs-vswitchd process huge memory consumption

[Fernando Casas Schössow]

I just replied to the other thread (since I'm running a different version of 
OVS -2.10.1-) and added ovs-...@openvswitch.org mailing list as well. Oleg, 
maybe you also want to add the dev list in case they can help on this?

Basically I still observe the continuous memory usage increase by ovs-vswitchd. 
The growth is almost linear since the process starts.

I will try to run the scripts against the dump I collected as well but I don't 
really have debugging skills to analyze a dump so it may be of little help even 
if the scripts generate a good output.

On jue, feb 28, 2019 at 7:14 PM, Ben Pfaff  wrote:
On Tue, Feb 26, 2019 at 01:41:45PM +0400, Oleg Bondarev wrote:
Hi, thanks for the scripts, so here's the output for a 24G core dump: 
https://pastebin.com/hWa3R9Fx there's 271 entries of 4MB - does it seem 
something we should take a closer look at?
I think that this output really just indicates that the script failed. It 
analyzed a lot of regions but didn't output anything useful. If it had worked 
properly, it would have told us a lot about data blocks that had been allocated 
and freed. The next step would have to be to debug the script. It definitely 
worked for me before, because I have fixed at least 3 or 4 bugs based on it, 
but it also definitely is a quick hack and not something that I can stand 
behind. I'm not sure how to debug it at a distance. It has a large comment that 
describes what it's trying to do. Maybe that would help you, if you want to try 
to debug it yourself. I guess it's also possible that glibc has changed its 
malloc implementation; if so, then it would probably be necessary to start over 
and build a new script.


___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] ovs-vswitchd process huge memory consumption

[Ben Pfaff]

On Tue, Feb 26, 2019 at 01:41:45PM +0400, Oleg Bondarev wrote:
> Hi,
> 
> thanks for the scripts, so here's the output for a 24G core dump:
> https://pastebin.com/hWa3R9Fx
> there's 271 entries of 4MB - does it seem something we should take a closer
> look at?

I think that this output really just indicates that the script failed.
It analyzed a lot of regions but didn't output anything useful.  If it
had worked properly, it would have told us a lot about data blocks that
had been allocated and freed.

The next step would have to be to debug the script.  It definitely
worked for me before, because I have fixed at least 3 or 4 bugs based on
it, but it also definitely is a quick hack and not something that I can
stand behind.  I'm not sure how to debug it at a distance.  It has a
large comment that describes what it's trying to do.  Maybe that would
help you, if you want to try to debug it yourself.  I guess it's also
possible that glibc has changed its malloc implementation; if so, then
it would probably be necessary to start over and build a new script.
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] ovs-vswitchd process huge memory consumption

[Oleg Bondarev]

Hi Ben,

so here're examples of how those repeatable blocks look like:
https://pastebin.com/JBUaeX44
https://pastebin.com/wKreHDJf
https://pastebin.com/f41knqgn

All those blocks are mostly filled with such sequence:
"    fa16 3e39 83c4   ..>9
  0030      4014  ...0..@.
          
      0fff    
     6500     e...
    4014      ..@."

where "fa 16 3e xx xx xx" are openstack neutron ports mac addresses
(actually VMs MACs as confirmed on the env with issue).
Can we figure something out of this?

Thanks,
Oleg

On Wed, Feb 27, 2019 at 8:41 AM Oleg Bondarev 
wrote:

>
>
> On Tue, Feb 26, 2019 at 1:41 PM Oleg Bondarev 
> wrote:
>
>> Hi,
>>
>> thanks for the scripts, so here's the output for a 24G core dump:
>> https://pastebin.com/hWa3R9Fx
>> there's 271 entries of 4MB - does it seem something we should take a
>> closer look at?
>>
>
> not 4 but ~67Mb of course.
>
>
>>
>> Thanks,
>> Oleg
>>
>> On Tue, Feb 26, 2019 at 3:26 AM Ben Pfaff  wrote:
>>
>>> Some combinations of kernel bonding with Open vSwitch don't necessarily
>>> work that well.  I have forgotten which ones are problematic or why.
>>> However, the problems are functional ones (the bonds don't work well),
>>> not operational ones like memory leaks.  A memory leak would be a bug
>>> whether or not the configuration used kernel bonds in a way that we
>>> discourage.
>>>
>>> With small OpenFlow flow tables (maybe a few thousand flows) and a small
>>> number of ports (maybe up to 100 or so), using the Linux kernel module
>>> (not DPDK), I'd be surprised to see OVS use more than 100 MB.  I'd also
>>> be surprised to see OVS memory usage grow past, say, the first day or
>>> usage.
>>>
>>> With large numbers of OpenFlow flows (e.g. hundreds of thousands), my
>>> rule of thumb is that OVS tends to use about 1 to 2 kB RAM per flow.
>>>
>>> A dump of a 500 MB process could still be enlightening.  Usually, when
>>> there is a memory leak, one sees an extraordinary number of unfreed
>>> memory blocks of a single size, and by looking at their size and their
>>> contents one can often figure out what allocated them, and that usually
>>> leads one to figure out why they did not get freed.
>>>
>>> On Mon, Feb 25, 2019 at 09:46:28PM +, Fernando Casas Schössow wrote:
>>> >
>>> > I read in a few places that mixing OS networking features (like
>>> bonding) and OVS is not a good idea and that the recommendation is to do
>>> everything at OVS level. That's why I assumed the configuration was not ok
>>> (even when it worked correctly for around two years albeit the high memory
>>> usage I detected).
>>> >
>>> > How many MB of RAM would you consider normal in a small setup like
>>> this one? Just to make myself an idea.
>>> >
>>> > I just finished a maintenance window on this server that required a
>>> reboot.
>>> > Right after reboot ovs-vswitchd is using 14MB of RAM.
>>> > I will keep monitoring the process memory usage usage and report back
>>> after two weeks or so.
>>> >
>>> > Would it make sense to get a process dump for analysis even if memory
>>> usage is not going as high (several GBs) as before the config change? In
>>> other words, if I find that the process memory usage grows up to around
>>> 500MB but then becomes steady and is not growing anymore would it make
>>> sense to collect a dump for analysis?
>>> >
>>> > On lun, feb 25, 2019 at 5:48 PM, Ben Pfaff  wrote:
>>> > Both configurations should work, so probably you did find a bug
>>> causing a memory leak in the former configuration. 464 MB actually sounds
>>> like a lot also. On Sun, Feb 24, 2019 at 02:58:02PM +, Fernando Casas
>>> Schössow wrote:
>>> > Hi Ben, In my case I think I found the cause of the issue, and it was
>>> indeed a misconfiguration on my side. Yet I'm not really sure why the
>>> misconfiguration was causing the high memory usage on OVS. The server has 4
>>> NICs. Bonded in two bonds of two. The problem I think it was that the
>>> bonding was done at OS level (Linux kernel bonding) instead of at OVS
>>> level. So there were two interfaces at OS level (bond0 and bond1) with
>>> bond0 added to OVS as an uplink port. I changed that configuration, removed
>>> all the bonding at OS level and instead created the bonds at OVS level.
>>> Then I restarted the service so I can monitor memory usage. After this
>>> change, memory usage growth from 10MB (at service start) to 464MB after a
>>> few hours and then stayed at that level until today (a week later). I'm
>>> still monitoring the process memory usage but as I said is steady for
>>> almost a week so I will keep monitoring it for a couple more weeks just in
>>> case and report back. Thanks. Kind regards, Fernando On sáb, feb 23, 2019
>>> at 12:23 AM, Ben Pfaff mailto:b...@ovn.org>> wrote: It's
>>> odd that 

Re: [ovs-discuss] ovs-vswitchd process huge memory consumption

[Oleg Bondarev]

On Tue, Feb 26, 2019 at 1:41 PM Oleg Bondarev 
wrote:

> Hi,
>
> thanks for the scripts, so here's the output for a 24G core dump:
> https://pastebin.com/hWa3R9Fx
> there's 271 entries of 4MB - does it seem something we should take a
> closer look at?
>

not 4 but ~67Mb of course.


>
> Thanks,
> Oleg
>
> On Tue, Feb 26, 2019 at 3:26 AM Ben Pfaff  wrote:
>
>> Some combinations of kernel bonding with Open vSwitch don't necessarily
>> work that well.  I have forgotten which ones are problematic or why.
>> However, the problems are functional ones (the bonds don't work well),
>> not operational ones like memory leaks.  A memory leak would be a bug
>> whether or not the configuration used kernel bonds in a way that we
>> discourage.
>>
>> With small OpenFlow flow tables (maybe a few thousand flows) and a small
>> number of ports (maybe up to 100 or so), using the Linux kernel module
>> (not DPDK), I'd be surprised to see OVS use more than 100 MB.  I'd also
>> be surprised to see OVS memory usage grow past, say, the first day or
>> usage.
>>
>> With large numbers of OpenFlow flows (e.g. hundreds of thousands), my
>> rule of thumb is that OVS tends to use about 1 to 2 kB RAM per flow.
>>
>> A dump of a 500 MB process could still be enlightening.  Usually, when
>> there is a memory leak, one sees an extraordinary number of unfreed
>> memory blocks of a single size, and by looking at their size and their
>> contents one can often figure out what allocated them, and that usually
>> leads one to figure out why they did not get freed.
>>
>> On Mon, Feb 25, 2019 at 09:46:28PM +, Fernando Casas Schössow wrote:
>> >
>> > I read in a few places that mixing OS networking features (like
>> bonding) and OVS is not a good idea and that the recommendation is to do
>> everything at OVS level. That's why I assumed the configuration was not ok
>> (even when it worked correctly for around two years albeit the high memory
>> usage I detected).
>> >
>> > How many MB of RAM would you consider normal in a small setup like this
>> one? Just to make myself an idea.
>> >
>> > I just finished a maintenance window on this server that required a
>> reboot.
>> > Right after reboot ovs-vswitchd is using 14MB of RAM.
>> > I will keep monitoring the process memory usage usage and report back
>> after two weeks or so.
>> >
>> > Would it make sense to get a process dump for analysis even if memory
>> usage is not going as high (several GBs) as before the config change? In
>> other words, if I find that the process memory usage grows up to around
>> 500MB but then becomes steady and is not growing anymore would it make
>> sense to collect a dump for analysis?
>> >
>> > On lun, feb 25, 2019 at 5:48 PM, Ben Pfaff  wrote:
>> > Both configurations should work, so probably you did find a bug causing
>> a memory leak in the former configuration. 464 MB actually sounds like a
>> lot also. On Sun, Feb 24, 2019 at 02:58:02PM +, Fernando Casas Schössow
>> wrote:
>> > Hi Ben, In my case I think I found the cause of the issue, and it was
>> indeed a misconfiguration on my side. Yet I'm not really sure why the
>> misconfiguration was causing the high memory usage on OVS. The server has 4
>> NICs. Bonded in two bonds of two. The problem I think it was that the
>> bonding was done at OS level (Linux kernel bonding) instead of at OVS
>> level. So there were two interfaces at OS level (bond0 and bond1) with
>> bond0 added to OVS as an uplink port. I changed that configuration, removed
>> all the bonding at OS level and instead created the bonds at OVS level.
>> Then I restarted the service so I can monitor memory usage. After this
>> change, memory usage growth from 10MB (at service start) to 464MB after a
>> few hours and then stayed at that level until today (a week later). I'm
>> still monitoring the process memory usage but as I said is steady for
>> almost a week so I will keep monitoring it for a couple more weeks just in
>> case and report back. Thanks. Kind regards, Fernando On sáb, feb 23, 2019
>> at 12:23 AM, Ben Pfaff mailto:b...@ovn.org>> wrote: It's odd
>> that two people would notice the same problem at the same time on old
>> branches. Anyway, I'm attaching the scripts I have. They are rough. The
>> second one invokes the first one as a subprocess; it is probably the one
>> you should use. I might have to walk you through how to use it, or write
>> better documentation myself. Anyway, it should be a start. On Wed, Feb 20,
>> 2019 at 07:15:26PM +0400, Oleg Bondarev wrote: Ah, sorry, I missed
>> "ovs-vswitchd memory consumption behavior" thread. So I guess I'm also
>> interested in the scripts for analyzing the heap in a core dump :) Thanks,
>> Oleg On Wed, Feb 20, 2019 at 7:00 PM Oleg Bondarev <
>> obonda...@mirantis.com> obonda...@mirantis.com>> wrote: > Hi, > > OVS 2.8.0, uptime 197 days,
>> 44G RAM. > ovs-appctl memory/show reports: > "handlers:35 ofconns:4
>> ports:73 revalidators:13 rules:1099 udpif > 

Re: [ovs-discuss] ovs-vswitchd process huge memory consumption

[Oleg Bondarev]

Hi,

thanks for the scripts, so here's the output for a 24G core dump:
https://pastebin.com/hWa3R9Fx
there's 271 entries of 4MB - does it seem something we should take a closer
look at?

Thanks,
Oleg

On Tue, Feb 26, 2019 at 3:26 AM Ben Pfaff  wrote:

> Some combinations of kernel bonding with Open vSwitch don't necessarily
> work that well.  I have forgotten which ones are problematic or why.
> However, the problems are functional ones (the bonds don't work well),
> not operational ones like memory leaks.  A memory leak would be a bug
> whether or not the configuration used kernel bonds in a way that we
> discourage.
>
> With small OpenFlow flow tables (maybe a few thousand flows) and a small
> number of ports (maybe up to 100 or so), using the Linux kernel module
> (not DPDK), I'd be surprised to see OVS use more than 100 MB.  I'd also
> be surprised to see OVS memory usage grow past, say, the first day or
> usage.
>
> With large numbers of OpenFlow flows (e.g. hundreds of thousands), my
> rule of thumb is that OVS tends to use about 1 to 2 kB RAM per flow.
>
> A dump of a 500 MB process could still be enlightening.  Usually, when
> there is a memory leak, one sees an extraordinary number of unfreed
> memory blocks of a single size, and by looking at their size and their
> contents one can often figure out what allocated them, and that usually
> leads one to figure out why they did not get freed.
>
> On Mon, Feb 25, 2019 at 09:46:28PM +, Fernando Casas Schössow wrote:
> >
> > I read in a few places that mixing OS networking features (like bonding)
> and OVS is not a good idea and that the recommendation is to do everything
> at OVS level. That's why I assumed the configuration was not ok (even when
> it worked correctly for around two years albeit the high memory usage I
> detected).
> >
> > How many MB of RAM would you consider normal in a small setup like this
> one? Just to make myself an idea.
> >
> > I just finished a maintenance window on this server that required a
> reboot.
> > Right after reboot ovs-vswitchd is using 14MB of RAM.
> > I will keep monitoring the process memory usage usage and report back
> after two weeks or so.
> >
> > Would it make sense to get a process dump for analysis even if memory
> usage is not going as high (several GBs) as before the config change? In
> other words, if I find that the process memory usage grows up to around
> 500MB but then becomes steady and is not growing anymore would it make
> sense to collect a dump for analysis?
> >
> > On lun, feb 25, 2019 at 5:48 PM, Ben Pfaff  wrote:
> > Both configurations should work, so probably you did find a bug causing
> a memory leak in the former configuration. 464 MB actually sounds like a
> lot also. On Sun, Feb 24, 2019 at 02:58:02PM +, Fernando Casas Schössow
> wrote:
> > Hi Ben, In my case I think I found the cause of the issue, and it was
> indeed a misconfiguration on my side. Yet I'm not really sure why the
> misconfiguration was causing the high memory usage on OVS. The server has 4
> NICs. Bonded in two bonds of two. The problem I think it was that the
> bonding was done at OS level (Linux kernel bonding) instead of at OVS
> level. So there were two interfaces at OS level (bond0 and bond1) with
> bond0 added to OVS as an uplink port. I changed that configuration, removed
> all the bonding at OS level and instead created the bonds at OVS level.
> Then I restarted the service so I can monitor memory usage. After this
> change, memory usage growth from 10MB (at service start) to 464MB after a
> few hours and then stayed at that level until today (a week later). I'm
> still monitoring the process memory usage but as I said is steady for
> almost a week so I will keep monitoring it for a couple more weeks just in
> case and report back. Thanks. Kind regards, Fernando On sáb, feb 23, 2019
> at 12:23 AM, Ben Pfaff mailto:b...@ovn.org>> wrote: It's odd
> that two people would notice the same problem at the same time on old
> branches. Anyway, I'm attaching the scripts I have. They are rough. The
> second one invokes the first one as a subprocess; it is probably the one
> you should use. I might have to walk you through how to use it, or write
> better documentation myself. Anyway, it should be a start. On Wed, Feb 20,
> 2019 at 07:15:26PM +0400, Oleg Bondarev wrote: Ah, sorry, I missed
> "ovs-vswitchd memory consumption behavior" thread. So I guess I'm also
> interested in the scripts for analyzing the heap in a core dump :) Thanks,
> Oleg On Wed, Feb 20, 2019 at 7:00 PM Oleg Bondarev  > wrote: >
> Hi, > > OVS 2.8.0, uptime 197 days, 44G RAM. > ovs-appctl memory/show
> reports: > "handlers:35 ofconns:4 ports:73 revalidators:13 rules:1099 udpif
> > keys:686" > > Similar data on other nodes of the OpenStack cluster. >
> Seems usage grows gradually over time. > Are there any known issues, like >
> 

Re: [ovs-discuss] ovs-vswitchd process huge memory consumption

[Ben Pfaff]

Some combinations of kernel bonding with Open vSwitch don't necessarily
work that well.  I have forgotten which ones are problematic or why.
However, the problems are functional ones (the bonds don't work well),
not operational ones like memory leaks.  A memory leak would be a bug
whether or not the configuration used kernel bonds in a way that we
discourage.

With small OpenFlow flow tables (maybe a few thousand flows) and a small
number of ports (maybe up to 100 or so), using the Linux kernel module
(not DPDK), I'd be surprised to see OVS use more than 100 MB.  I'd also
be surprised to see OVS memory usage grow past, say, the first day or
usage.

With large numbers of OpenFlow flows (e.g. hundreds of thousands), my
rule of thumb is that OVS tends to use about 1 to 2 kB RAM per flow.

A dump of a 500 MB process could still be enlightening.  Usually, when
there is a memory leak, one sees an extraordinary number of unfreed
memory blocks of a single size, and by looking at their size and their
contents one can often figure out what allocated them, and that usually
leads one to figure out why they did not get freed.

On Mon, Feb 25, 2019 at 09:46:28PM +, Fernando Casas Schössow wrote:
> 
> I read in a few places that mixing OS networking features (like bonding) and 
> OVS is not a good idea and that the recommendation is to do everything at OVS 
> level. That's why I assumed the configuration was not ok (even when it worked 
> correctly for around two years albeit the high memory usage I detected).
> 
> How many MB of RAM would you consider normal in a small setup like this one? 
> Just to make myself an idea.
> 
> I just finished a maintenance window on this server that required a reboot.
> Right after reboot ovs-vswitchd is using 14MB of RAM.
> I will keep monitoring the process memory usage usage and report back after 
> two weeks or so.
> 
> Would it make sense to get a process dump for analysis even if memory usage 
> is not going as high (several GBs) as before the config change? In other 
> words, if I find that the process memory usage grows up to around 500MB but 
> then becomes steady and is not growing anymore would it make sense to collect 
> a dump for analysis?
> 
> On lun, feb 25, 2019 at 5:48 PM, Ben Pfaff  wrote:
> Both configurations should work, so probably you did find a bug causing a 
> memory leak in the former configuration. 464 MB actually sounds like a lot 
> also. On Sun, Feb 24, 2019 at 02:58:02PM +, Fernando Casas Schössow wrote:
> Hi Ben, In my case I think I found the cause of the issue, and it was indeed 
> a misconfiguration on my side. Yet I'm not really sure why the 
> misconfiguration was causing the high memory usage on OVS. The server has 4 
> NICs. Bonded in two bonds of two. The problem I think it was that the bonding 
> was done at OS level (Linux kernel bonding) instead of at OVS level. So there 
> were two interfaces at OS level (bond0 and bond1) with bond0 added to OVS as 
> an uplink port. I changed that configuration, removed all the bonding at OS 
> level and instead created the bonds at OVS level. Then I restarted the 
> service so I can monitor memory usage. After this change, memory usage growth 
> from 10MB (at service start) to 464MB after a few hours and then stayed at 
> that level until today (a week later). I'm still monitoring the process 
> memory usage but as I said is steady for almost a week so I will keep 
> monitoring it for a couple more weeks just in case and report back. Thanks. 
> Kind regards, Fernando On sáb, feb 23, 2019 at 12:23 AM, Ben Pfaff 
> mailto:b...@ovn.org>> wrote: It's odd that two people would 
> notice the same problem at the same time on old branches. Anyway, I'm 
> attaching the scripts I have. They are rough. The second one invokes the 
> first one as a subprocess; it is probably the one you should use. I might 
> have to walk you through how to use it, or write better documentation myself. 
> Anyway, it should be a start. On Wed, Feb 20, 2019 at 07:15:26PM +0400, Oleg 
> Bondarev wrote: Ah, sorry, I missed "ovs-vswitchd memory consumption 
> behavior" thread. So I guess I'm also interested in the scripts for analyzing 
> the heap in a core dump :) Thanks, Oleg On Wed, Feb 20, 2019 at 7:00 PM Oleg 
> Bondarev 
> mailto:obonda...@mirantis.com>>
>  wrote: > Hi, > > OVS 2.8.0, uptime 197 days, 44G RAM. > ovs-appctl 
> memory/show reports: > "handlers:35 ofconns:4 ports:73 revalidators:13 
> rules:1099 udpif > keys:686" > > Similar data on other nodes of the OpenStack 
> cluster. > Seems usage grows gradually over time. > Are there any known 
> issues, like > http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-14970? 
> > Please advise on the best way to debug. > > Thanks, > Oleg > > 
> ___ discuss mailing list 
> disc...@openvswitch.org
>  

Re: [ovs-discuss] ovs-vswitchd process huge memory consumption

[Fernando Casas Schössow]

I read in a few places that mixing OS networking features (like bonding) and 
OVS is not a good idea and that the recommendation is to do everything at OVS 
level. That's why I assumed the configuration was not ok (even when it worked 
correctly for around two years albeit the high memory usage I detected).

How many MB of RAM would you consider normal in a small setup like this one? 
Just to make myself an idea.

I just finished a maintenance window on this server that required a reboot.
Right after reboot ovs-vswitchd is using 14MB of RAM.
I will keep monitoring the process memory usage usage and report back after two 
weeks or so.

Would it make sense to get a process dump for analysis even if memory usage is 
not going as high (several GBs) as before the config change? In other words, if 
I find that the process memory usage grows up to around 500MB but then becomes 
steady and is not growing anymore would it make sense to collect a dump for 
analysis?

On lun, feb 25, 2019 at 5:48 PM, Ben Pfaff  wrote:
Both configurations should work, so probably you did find a bug causing a 
memory leak in the former configuration. 464 MB actually sounds like a lot 
also. On Sun, Feb 24, 2019 at 02:58:02PM +, Fernando Casas Schössow wrote:
Hi Ben, In my case I think I found the cause of the issue, and it was indeed a 
misconfiguration on my side. Yet I'm not really sure why the misconfiguration 
was causing the high memory usage on OVS. The server has 4 NICs. Bonded in two 
bonds of two. The problem I think it was that the bonding was done at OS level 
(Linux kernel bonding) instead of at OVS level. So there were two interfaces at 
OS level (bond0 and bond1) with bond0 added to OVS as an uplink port. I changed 
that configuration, removed all the bonding at OS level and instead created the 
bonds at OVS level. Then I restarted the service so I can monitor memory usage. 
After this change, memory usage growth from 10MB (at service start) to 464MB 
after a few hours and then stayed at that level until today (a week later). I'm 
still monitoring the process memory usage but as I said is steady for almost a 
week so I will keep monitoring it for a couple more weeks just in case and 
report back. Thanks. Kind regards, Fernando On sáb, feb 23, 2019 at 12:23 AM, 
Ben Pfaff mailto:b...@ovn.org>> wrote: It's odd that two people 
would notice the same problem at the same time on old branches. Anyway, I'm 
attaching the scripts I have. They are rough. The second one invokes the first 
one as a subprocess; it is probably the one you should use. I might have to 
walk you through how to use it, or write better documentation myself. Anyway, 
it should be a start. On Wed, Feb 20, 2019 at 07:15:26PM +0400, Oleg Bondarev 
wrote: Ah, sorry, I missed "ovs-vswitchd memory consumption behavior" thread. 
So I guess I'm also interested in the scripts for analyzing the heap in a core 
dump :) Thanks, Oleg On Wed, Feb 20, 2019 at 7:00 PM Oleg Bondarev 
mailto:obonda...@mirantis.com>>
 wrote: > Hi, > > OVS 2.8.0, uptime 197 days, 44G RAM. > ovs-appctl memory/show 
reports: > "handlers:35 ofconns:4 ports:73 revalidators:13 rules:1099 udpif > 
keys:686" > > Similar data on other nodes of the OpenStack cluster. > Seems 
usage grows gradually over time. > Are there any known issues, like > 
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-14970? > Please advise 
on the best way to debug. > > Thanks, > Oleg > > 
___ discuss mailing list 
disc...@openvswitch.org
 https://mail.openvswitch.org/mailman/listinfo/ovs-discuss


___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] ovs-vswitchd process huge memory consumption

[Ben Pfaff]

Both configurations should work, so probably you did find a bug causing
a memory leak in the former configuration.

464 MB actually sounds like a lot also.

On Sun, Feb 24, 2019 at 02:58:02PM +, Fernando Casas Schössow wrote:
> Hi Ben,
> 
> In my case I think I found the cause of the issue, and it was indeed a 
> misconfiguration on my side.
> Yet I'm not really sure why the misconfiguration was causing the high memory 
> usage on OVS.
> 
> The server has 4 NICs. Bonded in two bonds of two.
> The problem I think it was that the bonding was done at OS level (Linux 
> kernel bonding) instead of at OVS level.
> So there were two interfaces at OS level (bond0 and bond1) with bond0 added 
> to OVS as an uplink port.
> 
> I changed that configuration, removed all the bonding at OS level and instead 
> created the bonds at OVS level. Then I restarted the service so I can monitor 
> memory usage.
> After this change, memory usage growth from 10MB (at service start) to 464MB 
> after a few hours and then stayed at that level until today (a week later).
> 
> I'm still monitoring the process memory usage but as I said is steady for 
> almost a week so I will keep monitoring it for a couple more weeks just in 
> case and report back.
> 
> Thanks.
> Kind regards,
> 
> Fernando
> 
> On sáb, feb 23, 2019 at 12:23 AM, Ben Pfaff  wrote:
> It's odd that two people would notice the same problem at the same time on 
> old branches. Anyway, I'm attaching the scripts I have. They are rough. The 
> second one invokes the first one as a subprocess; it is probably the one you 
> should use. I might have to walk you through how to use it, or write better 
> documentation myself. Anyway, it should be a start. On Wed, Feb 20, 2019 at 
> 07:15:26PM +0400, Oleg Bondarev wrote:
> Ah, sorry, I missed "ovs-vswitchd memory consumption behavior" thread. So I 
> guess I'm also interested in the scripts for analyzing the heap in a core 
> dump :) Thanks, Oleg On Wed, Feb 20, 2019 at 7:00 PM Oleg Bondarev 
> mailto:obonda...@mirantis.com>> wrote: > Hi, > > OVS 
> 2.8.0, uptime 197 days, 44G RAM. > ovs-appctl memory/show reports: > 
> "handlers:35 ofconns:4 ports:73 revalidators:13 rules:1099 udpif > keys:686" 
> > > Similar data on other nodes of the OpenStack cluster. > Seems usage grows 
> gradually over time. > Are there any known issues, like > 
> http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-14970? > Please advise 
> on the best way to debug. > > Thanks, > Oleg > >
> ___ discuss mailing list 
> disc...@openvswitch.org 
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> 
> 
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] ovs-vswitchd process huge memory consumption

[Fernando Casas Schössow]

Hi Ben,

In my case I think I found the cause of the issue, and it was indeed a 
misconfiguration on my side.
Yet I'm not really sure why the misconfiguration was causing the high memory 
usage on OVS.

The server has 4 NICs. Bonded in two bonds of two.
The problem I think it was that the bonding was done at OS level (Linux kernel 
bonding) instead of at OVS level.
So there were two interfaces at OS level (bond0 and bond1) with bond0 added to 
OVS as an uplink port.

I changed that configuration, removed all the bonding at OS level and instead 
created the bonds at OVS level. Then I restarted the service so I can monitor 
memory usage.
After this change, memory usage growth from 10MB (at service start) to 464MB 
after a few hours and then stayed at that level until today (a week later).

I'm still monitoring the process memory usage but as I said is steady for 
almost a week so I will keep monitoring it for a couple more weeks just in case 
and report back.

Thanks.
Kind regards,

Fernando

On sáb, feb 23, 2019 at 12:23 AM, Ben Pfaff  wrote:
It's odd that two people would notice the same problem at the same time on old 
branches. Anyway, I'm attaching the scripts I have. They are rough. The second 
one invokes the first one as a subprocess; it is probably the one you should 
use. I might have to walk you through how to use it, or write better 
documentation myself. Anyway, it should be a start. On Wed, Feb 20, 2019 at 
07:15:26PM +0400, Oleg Bondarev wrote:
Ah, sorry, I missed "ovs-vswitchd memory consumption behavior" thread. So I 
guess I'm also interested in the scripts for analyzing the heap in a core dump 
:) Thanks, Oleg On Wed, Feb 20, 2019 at 7:00 PM Oleg Bondarev 
mailto:obonda...@mirantis.com>> wrote: > Hi, > > OVS 
2.8.0, uptime 197 days, 44G RAM. > ovs-appctl memory/show reports: > 
"handlers:35 ofconns:4 ports:73 revalidators:13 rules:1099 udpif > keys:686" > 
> Similar data on other nodes of the OpenStack cluster. > Seems usage grows 
gradually over time. > Are there any known issues, like > 
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-14970? > Please advise 
on the best way to debug. > > Thanks, > Oleg > >
___ discuss mailing list 
disc...@openvswitch.org 
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss


___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] ovs-vswitchd process huge memory consumption

[Ben Pfaff]

It's odd that two people would notice the same problem at the same time
on old branches.

Anyway, I'm attaching the scripts I have.  They are rough.  The second
one invokes the first one as a subprocess; it is probably the one you
should use.  I might have to walk you through how to use it, or write
better documentation myself.  Anyway, it should be a start.

On Wed, Feb 20, 2019 at 07:15:26PM +0400, Oleg Bondarev wrote:
> Ah, sorry, I missed "ovs-vswitchd memory consumption behavior" thread.
> So I guess I'm also interested in the scripts for analyzing the heap in a
> core dump :)
> 
> Thanks,
> Oleg
> 
> On Wed, Feb 20, 2019 at 7:00 PM Oleg Bondarev 
> wrote:
> 
> > Hi,
> >
> > OVS 2.8.0, uptime 197 days, 44G RAM.
> > ovs-appctl memory/show reports:
> > "handlers:35 ofconns:4 ports:73 revalidators:13 rules:1099 udpif
> > keys:686"
> >
> > Similar data on other nodes of the OpenStack cluster.
> > Seems usage grows gradually over time.
> > Are there any known issues, like
> > http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-14970?
> > Please advise on the best way to debug.
> >
> > Thanks,
> > Oleg
> >
> >

> ___
> discuss mailing list
> disc...@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

#! /usr/bin/perl

use strict;
#use warnings;
use Fcntl;

my ($core, $start, $len, $vma) = @ARGV;
$start = oct($start) if $start =~ /^0/;
$len = oct($len) if $len =~ /^0/;
$vma = oct($vma) if defined($vma) && ($vma =~ /^0/);
open (CORE, '<', $core) or die "$core: open: $!\n";

my $bits = 64;
my $bytes = $bits / 8;

my $template;
if ($bits == 64) {
$template = 'QQ';
} elsif ($bits == 32) {
$template = 'LL';
} else {
die;
}

seek (CORE, $start, Fcntl::SEEK_SET) or die "$core: seek: $!\n";
my $ofs = $start;
my $n = 0;
my $n_inuse = 0;
my $last_size = 0;
my $last_ofs = 0;
my $bytes_inuse = 0;
my %hist;
my %loc;
while ($len > $bytes * 2) {
my $header;
read (CORE, $header, $bytes * 2) or die "$core: read: $!\n";
$ofs += $bytes * 2;


my ($prev_size, $size) = unpack($template, $header);
#print "$prev_size, $size\n";
die if $size < $bytes * 2;
die "$size byte block at offset $ofs" if $size > 10485760;
$n++;
if ($size & 1) {
#printf "0x%x, %d used\n", ($last_ofs - $start) + $vma, $last_size - 
$bytes * 2;
$n_inuse++;
my $sample = !$hist{$last_size}++ || int(rand($hist{$last_size})) < 1;
$loc{$last_size} = $last_ofs if $sample;
$bytes_inuse += $last_size;
} else {
#printf "0x%x, %u free\n", ($last_ofs - $start) + $vma, $last_size - 
$bytes * 2;
}
$last_size = $size & ~7;
$last_ofs = $ofs;

$len -= $size & ~7;

my $content_len = ($size & ~7) - $bytes * 2;
my $content;
read (CORE, $content, $content_len);
$ofs += $content_len;
}
print "$n blocks, $n_inuse blocks in use, $bytes_inuse bytes in use\n";

print "In use:\n";
foreach my $size (sort { $a <=> $b }(keys(%hist))) {
my $realsize = $size - 2 * $bytes;
printf "%6d bytes x %6d = %10d (0x%x)\n",
  $realsize, $hist{$size}, $realsize * $hist{$size}, $loc{$size};

if ($realsize == 96) {
printf "starting at vaddr 0x%x:\n", $loc{$size} - $start + $vma;
system("hd -n $realsize -s $loc{$size} $core");
}
}
#! /usr/bin/perl

use strict;
#use warnings;
use Fcntl;

my ($core) = @ARGV;

# Notes:
#
# - Each heap begins with a 'heap_info' structure, which is 32 bytes long
#   on 64-bit systems:
#
#   typedef struct _heap_info
#   {
# mstate ar_ptr; /* Arena for this heap. */
# struct _heap_info *prev; /* Previous heap. */
# size_t size;   /* Current size in bytes. */
# size_t mprotect_size; /* Size in bytes that has been mprotected
#  PROT_READ|PROT_WRITE.  */
# /* Make sure the following data is properly aligned, particularly
#that sizeof (heap_info) + 2 * SIZE_SZ is a multiple of
#MALLOC_ALIGNMENT. */
# char pad[-6 * SIZE_SZ & MALLOC_ALIGN_MASK];
#   } heap_info;
#
# - heap_info.ar_ptr points to a 'struct malloc_state' (typedefed to mstate),
#   which is a 2192-byte structure (on 64-bit systems):
#
#   struct malloc_state
#   {
# /* Serialize access.  */
# __libc_lock_define (, mutex);
#
# /* Flags (formerly in max_fast).  */
# int flags;
#
# /* Set if the fastbin chunks contain recently inserted free blocks.  
*/
# /* Note this is a bool but not all targets support atomics on 
booleans.  */
# int have_fastchunks;
#
# /* Fastbins */
# mfastbinptr fastbinsY[NFASTBINS];
#
# /* Base of the topmost chunk -- not otherwise kept in a bin */
# mchunkptr top;
#
# /* The remainder from the most recent split of a small request */
# mchunkptr last_remainder;
#
# /* Normal bins packed as described above */
# mchunkptr 

Re: [ovs-discuss] ovs-vswitchd process huge memory consumption

[Oleg Bondarev]

Ah, sorry, I missed "ovs-vswitchd memory consumption behavior" thread.
So I guess I'm also interested in the scripts for analyzing the heap in a
core dump :)

Thanks,
Oleg

On Wed, Feb 20, 2019 at 7:00 PM Oleg Bondarev 
wrote:

> Hi,
>
> OVS 2.8.0, uptime 197 days, 44G RAM.
> ovs-appctl memory/show reports:
> "handlers:35 ofconns:4 ports:73 revalidators:13 rules:1099 udpif
> keys:686"
>
> Similar data on other nodes of the OpenStack cluster.
> Seems usage grows gradually over time.
> Are there any known issues, like
> http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-14970?
> Please advise on the best way to debug.
>
> Thanks,
> Oleg
>
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

[ovs-discuss] ovs-vswitchd process huge memory consumption

[Oleg Bondarev]

Hi,

OVS 2.8.0, uptime 197 days, 44G RAM.
ovs-appctl memory/show reports:
"handlers:35 ofconns:4 ports:73 revalidators:13 rules:1099 udpif
keys:686"

Similar data on other nodes of the OpenStack cluster.
Seems usage grows gradually over time.
Are there any known issues, like
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-14970?
Please advise on the best way to debug.

Thanks,
Oleg
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss