As I said, I'm not excluding any protocols from MACsec. With that configuration, LLDP apparently doesn't work "outside the tunnel"--I never see any directly attached neighbors. LLDP does work between the MACsec endpoints--they show as if they are directly connected neighbors. I'm fine with that result in my case.
The solution to eliminate the spurious framing errors appears to be: Ask your carrier to shut off LLDP/CDP/any other L2 protocols running on their interfaces directly attached to your MACsec endpoint devices. On Tue, Apr 21, 2020 at 02:59:53PM +0000, Richard McGovern wrote: > Based upon Chuck’s reply: > > Well, that was an easy fix on my MX480s: > > set protocols lldp interface xe-0/0/1 disable > > Now I'm not seeing CRC errors incrementing 2-3 times per minute on the > EX3400s connected directly to the MX480s. > > I'm not excluding any protocols from MACsec--LLDP runs end-to-end between > the EX3400s just fine. > > Don’t exclude LLDP from MACSEC and either stop or block LLDP from the > Carrier/ISP. In Chuck’s case the Carrier (to the EX3400s) was his MX. Then > EX3400s should see each other via LLDP, but not see the carrier. I am not > sure if today, you have an LLDP neighbor with your Carrier/ISP or not? > > This is the way I now read his response. > > Yes? > > > Richard McGovern > Sr Sales Engineer, Juniper Networks > 978-618-3342 > > I’d rather be lucky than good, as I know I am not good > I don’t make the news, I just report it > > [signature_1140633420] > > From: james list <jameslis...@gmail.com> > Date: Tuesday, April 21, 2020 at 10:53 AM > To: Richard McGovern <rmcgov...@juniper.net> > Cc: Chuck Anderson <c...@wpi.edu>, Juniper List <juniper-nsp@puck.nether.net> > Subject: Re: [j-nsp] [EXT] EX4300: Framing error with macsec enabled > > [External Email. Be cautious of content] > > Hi Richard > lldp and lacp are excluded: > > > > @EX4300-A> show configuration security macsec | display set > > > set security macsec connectivity-association MAC security-mode > static-cak > > > set security macsec connectivity-association MAC pre-shared-key ckn > xxxx > > > set security macsec connectivity-association MAC pre-shared-key cak > > > "tttttvvvv" > > > set security macsec connectivity-association MAC exclude-protocol lldp > > > set security macsec connectivity-association MAC exclude-protocol lacp > > > set security macsec interfaces ge-0/0/0 connectivity-association MAC > > I did not catch the connection with Framing errors counter... > > Please detail if you can. > > Cheers > > > Il giorno mar 21 apr 2020 alle ore 15:27 Richard McGovern > <rmcgov...@juniper.net<mailto:rmcgov...@juniper.net>> ha scritto: > Chuck, I thought you were running both LLDP and LACP outside the MACSEC > tunnel, no? > > (Optional) Exclude a protocol from MACsec: > [edit security macsec connectivity-association connectivity-association-name] > user@switch# set exclude-protocol protocol-name > For instance, if you did not want Link Level Discovery Protocol (LLDP) to be > secured using MACsec: > > [edit security macsec connectivity-association ca-dynamic1] > user@switch# set exclude-protocol lldp > When this option is enabled, MACsec is disabled for all packets of the > specified protocol—in this case, LLDP—that are sent or received on the link. > > BEST PRACTICEWe recommend that any protocol other than MACsec being used on > the MACsec connection, such as LLDP, LACP, STP, or layer 3 routing protocols, > should be excluded and moved outside of the MACsec tunnel. > > Is this not working properly for LLDP? > > Rich > > Richard McGovern > Sr Sales Engineer, Juniper Networks > 978-618-3342 > > I’d rather be lucky than good, as I know I am not good > I don’t make the news, I just report it > > > On 4/19/20, 7:31 PM, "Chuck Anderson" <c...@wpi.edu<mailto:c...@wpi.edu>> > wrote: > > Well, that was an easy fix on my MX480s: > > set protocols lldp interface xe-0/0/1 disable > > Now I'm not seeing CRC errors incrementing 2-3 times per minute on the > EX3400s connected directly to the MX480s. > > I'm not excluding any protocols from MACsec--LLDP runs end-to-end between > the EX3400s just fine. > > Check if the carrier is running LLDP or CDP or similar. > > > On Sun, Apr 19, 2020 at 07:16:46PM -0400, Chuck Anderson wrote: > > Yes, I see CRC errors on EX3400s with MACsec termination, but only on > one side. > > > > Here is my topology: > > > > From A to B: > > > > [EX3400-A]-->--[push-vlan-tag-on-MX480]-->-L2 > vlan-->-[Carrier-ASR9k-pop-vlan-tag]-->--[EX3400-B] > > MACsec L2 connection L2 > xconnect MACsec > > > > From B to A: > > > > [EX3400-A]--<--[pop-vlan-tag-on-MX480]--<-L2 > vlan--<-[Carrier-ASR9k-push-vlan-tag]--<--[EX3400-B] > > MACsec L2 connection L2 > xconnect MACsec > > > > I also have a redundant path with EX3400-C (different local switch) and > EX3400-B (same remote switch). > > > > I see the CRC errors increasing at a rate of about 2-3 per minute, but > only on EX3400-A and EXX3400-C. > > > > All EX3400s were initially running 15.1X53-D57. Now A and C are > running 18.2R3-S2 and B is running 15.1X53-D592. But the problem has been > consistent throughout all releases, no improvement with upgrades. > > > > I wonder if something the carrier's ASR9k is sending down the VLAN > towards EX3400-A and -C is causing this? If not, maybe it is the MX480s > sending something locally to EX3400-A and -C? > > > > The following PRs don't seem relevant--I'm not doing anywhere close to > 60% utilization: > > > > > https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR1261567 > > > > And I'm not seeing "runts": > > > > > https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR1469663 > > > > I'm only seeing Framing errors (CRC/Align errors): > > > > admin@ex3400-a> show interfaces extensive xe-0/2/0 |match 22791 > > Errors: 227911, Drops: 0, Framing errors: 227911, Runts: 0, Policed > discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: > 0, FIFO errors: 0, Resource errors: 0 > > CRC/Align errors 227911 0 > > > > A few seconds later, it increased to 227913: > > > > MAC statistics: Receive Transmit > > Total octets 17953647117156 3221741316352 > > Total packets 13200126465 7010832956 > > Unicast packets 13194022205 7004785539 > > Broadcast packets 5272 0 > > Multicast packets 6098988 6047417 > > CRC/Align errors 227913 0 > > FIFO errors 0 0 > > MAC control frames 0 0 > > MAC pause frames 0 0 > > Oversized frames 0 > > Jabber frames 0 > > Fragment frames 0 > > VLAN tagged frames 13196813130 > > Code violations 0 > > > > Rate is only 24 Mbps, 2200 pps: > > > > admin@ex3400-a> show interfaces extensive xe-0/2/0 |match "bps|pps" > > Link-level type: Ethernet, MTU: 9192, LAN-PHY mode, Speed: 10Gbps, > BPDU Error: None, Loop Detect PDU Error: None, Ethernet-Switching Error: > None, MAC-REWRITE Error: None, > > Input bytes : 17954037433802 24242136 bps > > Output bytes : 3221749625049 498504 bps > > Input packets: 13200416854 2190 pps > > Output packets: 7010941872 830 pps > > > > > > > > On Sun, Apr 19, 2020 at 09:37:23AM +0200, james list wrote: > > > Dear experts, > > > I've an EX4300 (Junos 17.3R3-S3.3) which have a constant Framing error > > > counter increase also if the traffic is very low. > > > Interface is connected to a WAN link from a carrier and bw is 1 Gbs > but > > > traffic max is actually 100 Mbs and on average 10 Mbs. > > > On this interface I've enabled macsec, if I disable macsec the issue > is not > > > in place but unfortunately macsec is mandatory to be kept enabled. > > > > > > I cannot sniff since the packet is encrypted but to me it seems that > > > traffic is not lost, if I have 100 Mbs inside from WAN I see 100 Mbs > > > outside to DataCenter. > > > > > > Due to the fact that monitoring system contantly raise an alert, I'd > like > > > to know how to fix it or at least let the EX4300 do not raise the > counter > > > increase. > > > > > > I've opened a JTAC case but they found a PR which is currently > related to a > > > Broadcom chipset raising framing errors during spikes (ie 70% of the > > > interface bandwidth). > > > > > > > https://kb.juniper.net/InfoCenter/index?page=content&id=KB32264&actp=METADATA > > > > > > Also enabling flow-control as described in the KB do not change the > > > behaviour. > > > > > > I'm wondering if there could the option we're receiving some sort on > > > "unknown protocol" from the carrier (I remeber Cisco has something > like > > > that) or could be an harware issue.. > > > > > > On the other side of the link, the other EX4300 (side B) do not > experience > > > the same issue but the traffic is mostly from side B to side A. > > > > > > Here an example of the output, statistics cleared and after 1 minute > I get > > > 12 framing errors with 2 Mbs running on the WAN link: > > > > > > @EX4300-A> show interfaces ae0 extensive > > > Physical interface: ae0, Enabled, Physical link is Up > > > Interface index: 220, SNMP ifIndex: 549, Generation: 131 > > > Description: xxx > > > Link-level type: Ethernet, MTU: 9192, Speed: 1Gbps, BPDU Error: > None, > > > Ethernet-Switching Error: None, MAC-REWRITE Error: None, > > > Loopback: Disabled, Source filtering: Disabled, Flow control: > Enabled, > > > Minimum links needed: 1, Minimum bandwidth needed: 1bps > > > Device flags : Present Running > > > Interface flags: SNMP-Traps Internal: 0x0 > > > Current address: cc:e5:94:11:43:23, Hardware address: > cc:e5:94:11:43:23 > > > Last flapped : 2020-04-19 02:05:05 CEST (06:50:45 ago) > > > Statistics last cleared: 2020-04-19 09:11:22 CEST (00:01:00 ago) > > > Traffic statistics: > > > Input bytes : 10014863 2205456 bps > > > Output bytes : 4095720 582456 bps > > > Input packets: 33292 624 pps > > > Output packets: 33023 568 pps > > > IPv6 transit statistics: > > > Input bytes : 0 > > > Output bytes : 0 > > > Input packets: 0 > > > Output packets: 0 > > > Input errors: > > > Errors: 12, Drops: 0, Framing errors: 12, Runts: 0, Giants: 0, > Policed > > > discards: 0, Resource errors: 0 > > > Output errors: > > > Carrier transitions: 0, Errors: 0, Drops: 0, MTU errors: 0, > Resource > > > errors: 0 > > > Egress queues: 12 supported, 11 in use > > > > > > > > > @EX4300-A> show interfaces ge-0/0/0 extensive > > > Physical interface: ge-0/0/0, Enabled, Physical link is Up > > > Interface index: 649, SNMP ifIndex: 509, Generation: 140 > > > Description: WAN link > > > Link-level type: Ethernet, MTU: 9192, LAN-PHY mode, Link-mode: > > > Full-duplex, Speed: 1000mbps, BPDU Error: None, Loop Detect PDU > Error: None, > > > Ethernet-Switching Error: None, Source filtering: Disabled > > > Ethernet-Switching Error: None, MAC-REWRITE Error: None, Loopback: > > > Disabled, Flow control: Enabled, Auto-negotiation: Enabled, > > > Remote fault: Online, Media type: Copper, IEEE 802.3az Energy > Efficient > > > Ethernet: Disabled, Auto-MDIX: Enabled > > > Device flags : Present Running > > > Interface flags: SNMP-Traps Internal: 0x0 > > > Link flags : None > > > CoS queues : 12 supported, 12 maximum usable queues > > > Hold-times : Up 0 ms, Down 0 ms > > > Current address: cc:e5:94:11:43:23, Hardware address: > cc:e5:94:11:43:23 > > > Last flapped : 2020-03-28 18:43:04 CET (3w0d 13:30 ago) > > > Statistics last cleared: 2020-04-19 09:11:18 CEST (00:02:18 ago) > > > Traffic statistics: > > > Input bytes : 21782579 932296 bps > > > Output bytes : 17898068 498704 bps > > > Input packets: 76844 569 pps > > > Output packets: 82594 590 pps > > > IPv6 transit statistics: > > > Input bytes : 0 > > > Output bytes : 0 > > > Input packets: 0 > > > Output packets: 0 > > > Input errors: > > > Errors: 28, Drops: 0, Framing errors: 28, Runts: 0, Policed > discards: > > > 0, L3 incompletes: 0, L2 channel errors: 0, > > > L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0 > > > > > > > > > Here part of the config: > > > > > > @EX4300-A> show configuration interfaces ge-0/0/0 | display set > > > set interfaces ge-0/0/0 ether-options auto-negotiation > > > set interfaces ge-0/0/0 ether-options flow-control > > > set interfaces ge-0/0/0 ether-options 802.3ad ae0 > > > > > > > > > @EX4300-A> show configuration interfaces ae0 | display set > > > set interfaces ae0 mtu 9192 > > > set interfaces ae0 aggregated-ether-options flow-control > > > set interfaces ae0 aggregated-ether-options lacp active > > > set interfaces ae0 aggregated-ether-options lacp periodic fast > > > set interfaces ae0 unit 0 family ethernet-switching interface-mode > trunk > > > set interfaces ae0 unit 0 family ethernet-switching vlan members 2228 > > > set interfaces ae0 unit 0 family ethernet-switching vlan members > 2552-2553 > > > set interfaces ae0 unit 0 family ethernet-switching filter input QOS > > > > > > > > > @EX4300-A> show configuration security macsec | display set > > > set security macsec connectivity-association MAC security-mode > static-cak > > > set security macsec connectivity-association MAC pre-shared-key ckn > xxxx > > > set security macsec connectivity-association MAC pre-shared-key cak > > > "tttttvvvv" > > > set security macsec connectivity-association MAC exclude-protocol lldp > > > set security macsec connectivity-association MAC exclude-protocol lacp > > > set security macsec interfaces ge-0/0/0 connectivity-association MAC > > > > > > > > > Dear all, an help is appreciated and welcomme, please let me thank in > > > advance anyone will give an hint. > > > > > > Cheers > > > James _______________________________________________ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp