On Thu, Sep 24, 2009 at 12:13:41PM +0300, Pekka Savola wrote: > Have you tested MX also during RIB or FIB changes? I'd like to use > this as a soapbox. On Juniper platforms, when a next-hop changes, > this results in first DELETE and then (maybe much later) ADD > operations in FIB. We have noticed that when BGP next-hop changes for > 300K prefixes, this has led to up to 7 minutes of packet loss due to a > lack of FIB entry for a destination. This was seen on T-series, but MX > has the same architectural limitations. This slowness was introduced > sometime in JunOS 8.x. And this is "working as designed" and "no way > to speed it up". > > Hopefully at some point someone will do performance testing in these > kind of scenarios as well; I don't recall seeing such a test myself.
Well, we saw something similar (that I've described on this list several times) but not exactly what you described, starting somewhere in 7.x and continuing until 8.5 where it was fixed on most platforms (including MX). Basically what happens is during a path change when it adds the new path and removes the old, something will cause the update process to stall, resulting in many minutes taken to install the new path into the hardware. If you did a show route on it while it is trying to make the update, you'll see a - on the old path and a + on the new path as it tries to update the FIB. It seemed like "something" would cause one particular update to stall, which would block that and all further updates behind that from taking effect for some number of minutes. Back on the M160 this could be 30 minutes to in some cases hours, but when we saw it on MX 7 minutes was a good average value. Eventually whatever route update was stalling things would go through, and then all the other routes that were stuck behind it would come flooding through. If you did a "show bgp summary" during this event, the "stuck" routes would show up in the "Pending" state counter up top. This normally didn't cause any packet loss for us, as it would keep forwarding via the old path until the new one became active. If the next-hop became invalid it would pull the routes immediately (or at least, we never got a complaint or saw an example where it didn't, in the many hundreds of times we observed this behavior over the years). The only time we would see packet loss was when the old path would drop BGP and not be able to continue to forward traffic, but not drop link. We complained about this for years and years, it went from "we don't see this" to "we can't reproduce this" to "we have no idea whats causing this". Eventually at 8.5 it become "we changed a whole bunch of things to try and fix this", but they never did tell us exactly what. After 8.5 we have not seen this problem since, at least on the MX platform (and we tossed all our M160s into the nearest landfill). Ironically enough the problem still exists on J-series running JUNOS (real JUNOS, we stopped testing after it was unfixed in the last JUNOS version to be released before the switch to -ES), despite there not actually being a hardware fib that needs updating. This pretty much put a kibosh on plans to use J-series as a BGP route-reflector, since BGP will not propagate the route change to other neighbors until it successfully installs the new path in its own FIB. If anyone from Juniper wants to step forward and describe what the problem or fix was I'd be interested in hearing it, our original issue was spread out over too many years and too many PRs to ever get anyone to even understand the question let alone give us a straight answer. -- Richard A Steenbergen <r...@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC) _______________________________________________ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp