* Andrey Kostin <ank...@podolsk.ru> [2019-03-15 20:50]: > I'm interested to hear about experience of running EVPN/VXLAN, particularly > with QFX10k as L3 gateway and QFX5k as spine/leaves. As per docs, it should > be immune to any single switch downtime, so might be a candidate to really > redundant design.
All right here it goes: I can't speak for QFX10k as spine but we have QFX5100 Leaf/Spine setups with EVPN/VXLAN running right now. Switch downtime is no problem at all, we unplugged a running switch, shut down ports, unplugged cables between leaf & spine or leaf & client all while there was storage traffic (NFS) active in the setup. Worst thing that happend was that IOPS went down from 400k/s to 100k/s for 1-3 seconds. What did bother us was that you are limited (at least on QFX5100) in the amount of "VLANs" (VNIs). We were testing with 30 client full-trunk ports per leaf and with that amount you can only provision around 500 VLANs before you get errors and basically it seems you run out of memory for bridge domains on the switch. This seems to be a limitation by the chips used in the QFX5100, at least that's what I got when I asked about it. You can check if you know where: root@SW-A:RE:0% ifsmon -Id | grep IFBD IFBD : 12884 0 root@SW-A:RE:0% ifsmon -Id | grep Bridge Bridge Domain : 3502 0 These numbers combined need to be <= 16382. And if you get over the limit these nice errors occur: dcf_ng_get_vxlan_ifbd_hw_token: Max vxlan ifbd hw token reached 16382 ifbd_create_node: VXLAN IFBD hw token couldn't be allocated for <xe-...> Workaround is to decrease VLANs or trunk config. Also you absolutely NEED LACP from servers to the fabric. 17.4 has enhancements which will put the client ports in LACP standby when the leaf gets separated from all spines. > As a downside I see the more complex configuration at least. Adding > vlan means adding routing instance etc. There are also other > questions, about convergence, scalability, how stable it is and code > maturity. We have it automated with Ansible. Management access happens over OOB (Mgmt) ports and everything is pushed by Ansible playbooks. Ansible generates configuration from templates and pushes it to the switches via netconf. I never would want to do this by hand. This demands a certain level of structuring by every team (network, people doing the cabling, server team) but it works out well for structured setups. Our switch config looks like this: -------------------------------------------------------------------------- user@sw1-spine-pod1> show configuration ## Last commit: 2019-03-11 03:13:49 CET by user ## Image name: jinstall-host-qfx-5-flex-17.4R2-S2.3-signed.tgz version 17.4R1-S3.3; groups { /* Created by Ansible */ evpn-defaults { /* OMITTED */ }; /* Created by Ansible */ evpn-spine-defaults { /* OMITTED */ }; /* Created by Ansible */ evpn-spine-1 { /* OMITTED */ }; /* Created by Ansible - Empty group for maintenance operations */ client-interfaces; } apply-groups [ evpn-defaults evpn-spine-defaults evpn-spine-1 ]; -------------------------------------------------------------------------- So everything Ansible does is contained in apply-groups and is hidden. You can immediately spot if something is configured by hand. For code we're currently running on the 17.4 train which works mostly fine, we had a few problems with third party 40G optics but these should be fixed in the newest 17.4 service release. Also we had a problem where new Spine/Leaf links did not come up but these vanished after rebooting/upgrading the spines. In daily operations it proves to be quite stable. Best Regards Sebastian -- GPG Key: 0x58A2D94A93A0B9CE (F4F6 B1A3 866B 26E9 450A 9D82 58A2 D94A 93A0 B9CE) 'Are you Death?' ... IT'S THE SCYTHE, ISN'T IT? PEOPLE ALWAYS NOTICE THE SCYTHE. -- Terry Pratchett, The Fifth Elephant _______________________________________________ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp