Re: [j-nsp] [c-nsp] LACP between router VMs (James Bensley)

[adamv0025]

I can't afford wasting time figuring out whether the feature is broken per se 
or due to use in logical-system and there's the problem of missing features as 
you mentioned, so all in all, can't really use logical-systems in virtual or 
physical lab. 
I find the only limiting factor in large builds to be the system memory. CPU 
wise the VMs are still rock solid even if "load average" is  ~4-5.  
And for simulation purposes you don't need to be concerned with speed. -hence 
no need to worry about the VT support, these router VMs will happily run in 
pure QEMU. 

adam
> -Original Message-
> From: Chris Burton [mailto:chris.bur...@speakeasy.net]
> Sent: Sunday, December 03, 2017 6:37 AM
> To: adamv0...@netconsultings.com; juniper-nsp@puck.nether.net
> Subject: Re: [j-nsp] [c-nsp] LACP between router VMs (James Bensley)
> 
> For better or worse I do remember those days, though I was referring to
> recent hardware switches/bridges (should have clarified that).  To my
> knowledge that only applies to things like the STP protocols, but I could be
> wrong, again would need to read through the specifications again to be sure
> as that is not a use case I have needed thus far.
> 
> Scaling the vMX for testing/lab/PoC deployments can be challenging, but I
> have been able to get large topologies off of a single older model dual Xeon
> E5-2670 server using logical systems, total of seven vMX instances and 84
> routers using the aforementioned logical systems, at that point I run into
> thermal limits because of the VFP CPU usage (even in lite-mode, which only
> appears to be related to the number of interfaces rather than the packet
> processing, which seems to be the same in lite and performance modes
> hence the CPU usage).  Some months back I tried to see how large of a
> topology I could build with five servers I had access to, I was able to get to
> seven vMX instances per server with
> 12 logical systems per vMX instance which gave me a 420 router topology
> using the trial license, so you can scale a lab/PoC setup quite nicely. Only
> downside to using logical systems is they do not support everything that a
> non-LS would support, the biggest missing feature in my case being EVPN.
> 
> Another item I have been testing is the vQFX which has much less CPU
> demand since the interfaces are bound to the VCP instead of the VFP, but I
> have run into many other issues with that and have not tested it as
> thoroughly, it is also just a alpha/beta release from Juniper at the moment.
> 
> -C
> 
> 
> On 12/02/2017 03:40 AM, adamv0...@netconsultings.com wrote:
> > Hey,
> >
> >> local link and not forwarded by the soft bridge by default (I do not know
> of
> >> any hardware bridges that allow you to disabled this restriction, if you
> know
> >> of any I would be interested.
> >>
> > My understanding is that Carrier-Ethernet grade switches/routers should
> allow you to peer/drop/tunnel/forward L2 protocols.
> > If you're in be business long enough you may remember migrations from
> leased-lines to frame-relay and then from FR to MPLS and then from L3VPNs
> to L2VPNs to complete the circle.
> > These L2 services especially the point-to-point ones, that's where
> customers pretty much expect the same properties as they used to have in
> leased-lines or FR services, basically just a pipe where MTU is not an issue
> and can transport anything from L2 up so they can run their own MPLS/DC
> networks over these pipes.
> >
> >> Out of curiosity what is your use case that you need to use LACP to
> >> communicate with VMs?
> >>
> > Large scale ISP network simulations (for proof of concept testing of various
> designs/migrations/etc).
> > This allows me to verify my designs, how the technology works on selected
> code versions -if there are any bugs, interworking between vendors.
> > And there are the provisioning and network monitoring systems, new SDN
> approaches that can be tested in this virtual environment, you name it.
> > Since it's all virtual one can simulate complete networks rather than scaled
> down slices used in physical labs, so I can see the effects of topology-based
> route-reflection in terms of routes distribution, the effects of node or link
> failures on traffic-engineering and possible congestion as a result across the
> whole backbone all in 1:1 scale, but the important point is to make the
> simulated control-plane as close to reality as possible hence the need for
> LACP.
> >
> > Speaking of scale, the fact that the VFP is always at 100% CPU is not 
> > helping
> -reminds me of the good old Dynamips -but at least there you could fix it
> with an idle value.
> > Having hundreds of these VNFs running is not very green.
> >
> >
> > adam
> >
> >
> >


___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] [c-nsp] LACP between router VMs (James Bensley)

[James Bensley]

Amazon recently announced bare metal instances which provide access to
the CPU virtual instruction set such as Intel VT-x, so although I
haven't had a chance to look into it much just yet, I'm hoping one can
spin up vMX/ASR9Kv etc. using these instances in Amazon and just
offload the scaling issue to them:

https://aws.amazon.com/about-aws/whats-new/2017/11/announcing-amazon-ec2-bare-metal-instances-preview/


Cheers,
James.
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] [c-nsp] LACP between router VMs (James Bensley)

[Chris Burton]

For better or worse I do remember those days, though I was referring to 
recent hardware switches/bridges (should have clarified that).  To my 
knowledge that only applies to things like the STP protocols, but I 
could be wrong, again would need to read through the specifications 
again to be sure as that is not a use case I have needed thus far.


Scaling the vMX for testing/lab/PoC deployments can be challenging, but 
I have been able to get large topologies off of a single older model 
dual Xeon E5-2670 server using logical systems, total of seven vMX 
instances and 84 routers using the aforementioned logical systems, at 
that point I run into thermal limits because of the VFP CPU usage (even 
in lite-mode, which only appears to be related to the number of 
interfaces rather than the packet processing, which seems to be the same 
in lite and performance modes hence the CPU usage).  Some months back I 
tried to see how large of a topology I could build with five servers I 
had access to, I was able to get to seven vMX instances per server with 
12 logical systems per vMX instance which gave me a 420 router topology 
using the trial license, so you can scale a lab/PoC setup quite nicely.  
Only downside to using logical systems is they do not support everything 
that a non-LS would support, the biggest missing feature in my case 
being EVPN.


Another item I have been testing is the vQFX which has much less CPU 
demand since the interfaces are bound to the VCP instead of the VFP, but 
I have run into many other issues with that and have not tested it as 
thoroughly, it is also just a alpha/beta release from Juniper at the moment.


-C


On 12/02/2017 03:40 AM, adamv0...@netconsultings.com wrote:

Hey,


local link and not forwarded by the soft bridge by default (I do not know of
any hardware bridges that allow you to disabled this restriction, if you know
of any I would be interested.


My understanding is that Carrier-Ethernet grade switches/routers should allow 
you to peer/drop/tunnel/forward L2 protocols.
If you're in be business long enough you may remember migrations from 
leased-lines to frame-relay and then from FR to MPLS and then from L3VPNs to 
L2VPNs to complete the circle.
These L2 services especially the point-to-point ones, that's where customers 
pretty much expect the same properties as they used to have in leased-lines or 
FR services, basically just a pipe where MTU is not an issue and can transport 
anything from L2 up so they can run their own MPLS/DC networks over these pipes.
 

Out of curiosity what is your use case that you need to use LACP to
communicate with VMs?


Large scale ISP network simulations (for proof of concept testing of various 
designs/migrations/etc).
This allows me to verify my designs, how the technology works on selected code 
versions -if there are any bugs, interworking between vendors.
And there are the provisioning and network monitoring systems, new SDN 
approaches that can be tested in this virtual environment, you name it.
Since it's all virtual one can simulate complete networks rather than scaled 
down slices used in physical labs, so I can see the effects of topology-based 
route-reflection in terms of routes distribution, the effects of node or link 
failures on traffic-engineering and possible congestion as a result across the 
whole backbone all in 1:1 scale, but the important point is to make the 
simulated control-plane as close to reality as possible hence the need for LACP.
  
Speaking of scale, the fact that the VFP is always at 100% CPU is not helping -reminds me of the good old Dynamips -but at least there you could fix it with an idle value.

Having hundreds of these VNFs running is not very green.
  


adam





___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] [c-nsp] LACP between router VMs (James Bensley)

[adamv0025]

Hey,

> local link and not forwarded by the soft bridge by default (I do not know of
> any hardware bridges that allow you to disabled this restriction, if you know
> of any I would be interested.  
>
My understanding is that Carrier-Ethernet grade switches/routers should allow 
you to peer/drop/tunnel/forward L2 protocols. 
If you're in be business long enough you may remember migrations from 
leased-lines to frame-relay and then from FR to MPLS and then from L3VPNs to 
L2VPNs to complete the circle. 
These L2 services especially the point-to-point ones, that's where customers 
pretty much expect the same properties as they used to have in leased-lines or 
FR services, basically just a pipe where MTU is not an issue and can transport 
anything from L2 up so they can run their own MPLS/DC networks over these 
pipes. 

> 
> Out of curiosity what is your use case that you need to use LACP to
> communicate with VMs?
> 
Large scale ISP network simulations (for proof of concept testing of various 
designs/migrations/etc).
This allows me to verify my designs, how the technology works on selected code 
versions -if there are any bugs, interworking between vendors. 
And there are the provisioning and network monitoring systems, new SDN 
approaches that can be tested in this virtual environment, you name it.   
Since it's all virtual one can simulate complete networks rather than scaled 
down slices used in physical labs, so I can see the effects of topology-based 
route-reflection in terms of routes distribution, the effects of node or link 
failures on traffic-engineering and possible congestion as a result across the 
whole backbone all in 1:1 scale, but the important point is to make the 
simulated control-plane as close to reality as possible hence the need for LACP.
 
Speaking of scale, the fact that the VFP is always at 100% CPU is not helping 
-reminds me of the good old Dynamips -but at least there you could fix it with 
an idle value. 
Having hundreds of these VNFs running is not very green. 
 

adam

___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] [c-nsp] LACP between router VMs (James Bensley)

[Vincent Bernat]

 ❦  1 décembre 2017 21:43 -0800, Chris Burton  :

> LACP and LLDP (and the like) are link local only protocols (would need
> to dig through the specifications to find the exact rules), they
> should not be forwarded by the bridge

802.1Q-2005, 8.6.3 and 8.13.4. However, some bridges are allowed to
forward those frames: two port mac relays can forward frames when
destination is 01-80-C2-00-00-03 (802.1Q-2011, dunno which part since
it's not freely available) and QinQ bridges can forward frames when the
destination is 01-80-C2-00-00-00 (LLDP frames can use those MAC
addresses since 802.1AB-2009, dunno if there is something similar for
LACP).
-- 
Use the fundamental control flow constructs.
- The Elements of Programming Style (Kernighan & Plauger)
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] [c-nsp] LACP between router VMs (James Bensley)

[Chris Burton]

Glad the conversation helped.  The note about the scripts was a general 
FYI in case there are others looking to use the scripts as there have 
been many conversations about the use of LLDP and LACP with the vMX in 
the past, both on this thread as well as other chats, forums, etc... 
just included for completeness.


LACP and LLDP (and the like) are link local only protocols (would need 
to dig through the specifications to find the exact rules), they should 
not be forwarded by the bridge, otherwise you could break the protocols 
themselves, in the case of LLDP it may be a simple as creating 
troubleshooting nightmare in terms of bad information, but in the case 
of LACP it could cause real harm, that is why they are only transmitted 
on the local link and not forwarded by the soft bridge by default (I do 
not know of any hardware bridges that allow you to disabled this 
restriction, if you know of any I would be interested.  Disabling this 
behavior on soft switch breaks that rule, that being said in the 
virtualized network world there are many things that have changed, so my 
statement was more of a general warning to not implement unless you 
understand and are willing to accept the possible consequences.


I have been running both OVS and Linux Bridge (with modified Kernel) in 
various labs, both large and small for quite some time without any 
issues, so my guess is it would be fine with the above caveats and ample 
precautions, though in production I use physical NICs and SR-IOV based 
NICs so I have not attempted to implement this bypass in production and 
do not know all of the consequences for production deployment.


Out of curiosity what is your use case that you need to use LACP to 
communicate with VMs?


Cheers,

-C


On 11/12/2017 05:36 AM, adamv0...@netconsultings.com wrote:

Chris Burton
Sent: Saturday, November 11, 2017 11:50 PM

This has been discussed a few times on this list and other forums. That being
said, if you are looking to do this with Linux bridge you will need to modify
the net/br_private.h header library to remove the mask restriction placed on
using group_fwd_mask setting in /sys/class/net and recompile the kernel to
allow it forward LACP BPDU's.


My question regarding the Linux bridge was out of curiosity and to my surprise 
spawned a very fruitful discussion which I learned a lot from.
So thank you everyone.
I'm actually using OVS and the "sudo ovs-vsctl set bridge br-1 
other-config:forward-bpdu=true" did the trick.


If you are looking to use Openvswitch you can do as Sergio mentioned and
set the OVS bridge to allow forwarding of LACP BPDU's after you manually
add each interface, but if you want to use the default vmx.sh script to bind
the interfaces with OVS you will need to modify the config/vmx-
junosdev.conf file and add in "use_ovs_transport: True" as a key, and
depending on what version of OS you are running (I presume Ubuntu 16.04)
you may also need to modify the sub-script that vmx.sh --bind-dev calls to
prevent it from checking for the "openvswitch-datapath-source" package. I
have only tested OVS 2.6 (Ubuntu cloud repo) with version 16.1 of VMX, but
unless the sub-scripts are different for earlier or later versions it should 
work
(but it is unlikely to be supported since I do not see mention of using OVS in
any of the official Juniper documents, and they likely will not support this
even if OVS is supported as it breaks standards, same goes for LB mod's).


I'm not using any juniper scripts at all, I need complete control about all 
aspects of the setup and there are VMs of different vendors so I'm using custom 
scripts and procedures to create XMLs, define and start VMs and to interconnect 
them in OVS.



It should be understood though that you are breaking standards
compatibility and could create massive problems on the network, so running
this type of configuration outside of a lab environment should fall under the
do not do it category. If you are going to do this production you should use
physical NICs and SR-IOV with VF’s which eliminates the issue as was
mentioned by James.


As I mentioned earlier, I don't need to talk LACP to PF just LACP between VFs.
Can you elaborate on the "breaking standards compatibility" please?
In my setup I just need hundreds of simple p2p links (basically emulating 
fibres) between VMs so I need the OVS (or other bridge) not to intervene.


adam






___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] [c-nsp] LACP between router VMs (James Bensley)

[adamv0025]

> Chris Burton
> Sent: Saturday, November 11, 2017 11:50 PM
> 
> This has been discussed a few times on this list and other forums. That being
> said, if you are looking to do this with Linux bridge you will need to modify
> the net/br_private.h header library to remove the mask restriction placed on
> using group_fwd_mask setting in /sys/class/net and recompile the kernel to
> allow it forward LACP BPDU's.
> 
My question regarding the Linux bridge was out of curiosity and to my surprise 
spawned a very fruitful discussion which I learned a lot from. 
So thank you everyone. 
I'm actually using OVS and the "sudo ovs-vsctl set bridge br-1 
other-config:forward-bpdu=true" did the trick. 

> If you are looking to use Openvswitch you can do as Sergio mentioned and
> set the OVS bridge to allow forwarding of LACP BPDU's after you manually
> add each interface, but if you want to use the default vmx.sh script to bind
> the interfaces with OVS you will need to modify the config/vmx-
> junosdev.conf file and add in "use_ovs_transport: True" as a key, and
> depending on what version of OS you are running (I presume Ubuntu 16.04)
> you may also need to modify the sub-script that vmx.sh --bind-dev calls to
> prevent it from checking for the "openvswitch-datapath-source" package. I
> have only tested OVS 2.6 (Ubuntu cloud repo) with version 16.1 of VMX, but
> unless the sub-scripts are different for earlier or later versions it should 
> work
> (but it is unlikely to be supported since I do not see mention of using OVS in
> any of the official Juniper documents, and they likely will not support this
> even if OVS is supported as it breaks standards, same goes for LB mod's).
> 
I'm not using any juniper scripts at all, I need complete control about all 
aspects of the setup and there are VMs of different vendors so I'm using custom 
scripts and procedures to create XMLs, define and start VMs and to interconnect 
them in OVS. 


> It should be understood though that you are breaking standards
> compatibility and could create massive problems on the network, so running
> this type of configuration outside of a lab environment should fall under the
> do not do it category. If you are going to do this production you should use
> physical NICs and SR-IOV with VF’s which eliminates the issue as was
> mentioned by James.
> 
As I mentioned earlier, I don't need to talk LACP to PF just LACP between VFs. 
Can you elaborate on the "breaking standards compatibility" please? 
In my setup I just need hundreds of simple p2p links (basically emulating 
fibres) between VMs so I need the OVS (or other bridge) not to intervene.   


adam


___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] [c-nsp] LACP between router VMs (James Bensley)

[Chris Burton]

This has been discussed a few times on this list and other forums. That 
being said, if you are looking to do this with Linux bridge you will 
need to modify the net/br_private.h header library to remove the mask 
restriction placed on using group_fwd_mask setting in /sys/class/net and 
recompile the kernel to allow it forward LACP BPDU's.


If you are looking to use Openvswitch you can do as Sergio mentioned and 
set the OVS bridge to allow forwarding of LACP BPDU's after you manually 
add each interface, but if you want to use the default vmx.sh script to 
bind the interfaces with OVS you will need to modify the 
config/vmx-junosdev.conf file and add in "use_ovs_transport: True" as a 
key, and depending on what version of OS you are running (I presume 
Ubuntu 16.04) you may also need to modify the sub-script that vmx.sh 
--bind-dev calls to prevent it from checking for the 
"openvswitch-datapath-source" package. I have only tested OVS 2.6 
(Ubuntu cloud repo) with version 16.1 of VMX, but unless the sub-scripts 
are different for earlier or later versions it should work (but it is 
unlikely to be supported since I do not see mention of using OVS in any 
of the official Juniper documents, and they likely will not support this 
even if OVS is supported as it breaks standards, same goes for LB mod's).


It should be understood though that you are breaking standards 
compatibility and could create massive problems on the network, so 
running this type of configuration outside of a lab environment should 
fall under the do not do it category. If you are going to do this 
production you should use physical NICs and SR-IOV with VF’s which 
eliminates the issue as was mentioned by James.


-C


On 11/09/2017 07:30 PM, Sergio D. wrote:

You can use the following on ovs:

ovs-vsctl set bridge  other-config:forward-bpdu=true

according to the ovs-vsctl docs[0] this includes 01:80:c2:00:00:0x which is
LACP.

[0] http://openvswitch.org/support/dist-docs/ovs-vswitchd.conf.db.5.html



1. Re: [c-nsp] LACP between router VMs (James Bensley)


--

Message: 1
Date: Wed, 8 Nov 2017 18:21:07 +
From: James Bensley 
To: adamv0...@netconsultings.com
Cc: "cisco-...@puck.nether.net" ,
 juniper-nsp 
Subject: Re: [j-nsp] [c-nsp] LACP between router VMs
Message-ID:
 

Re: [j-nsp] [c-nsp] LACP between router VMs (James Bensley)

[Sergio D.]

You can use the following on ovs:

ovs-vsctl set bridge  other-config:forward-bpdu=true

according to the ovs-vsctl docs[0] this includes 01:80:c2:00:00:0x which is
LACP.

[0] http://openvswitch.org/support/dist-docs/ovs-vswitchd.conf.db.5.html


>
>1. Re: [c-nsp] LACP between router VMs (James Bensley)
>
>
> --
>
> Message: 1
> Date: Wed, 8 Nov 2017 18:21:07 +
> From: James Bensley 
> To: adamv0...@netconsultings.com
> Cc: "cisco-...@puck.nether.net" ,
> juniper-nsp 
> Subject: Re: [j-nsp] [c-nsp] LACP between router VMs
> Message-ID:
>