Re: Whats so difficult about ISSU
http://www.juniper.net/techpubs/en_US/junos/topics/concept/issu-oveview.html The Juniper ISSU guide. You need two things: 1. Separation of the control plane and forwarding plane 2. 2 routing engines in the same chassis -- the non active RE upgrades first, then when its up and running the active one goes into upgrade mode and control fails over to the secondary RE which is running the upgraded version of the software. I assume it works on any vendor that has 2 REs in the same chassis and the fwd and control planes are separated, and there is a redundancy protocol running between the two REs(like Graceful Switchover on Juniper gear). On 11/09/2012 01:42 AM, Kenneth McRae wrote: Juniper also offers it on the EX virtual switching platform. Works if you have the correct version of JunOS. On Thu, Nov 8, 2012 at 3:38 PM, Zaid Ali z...@zaidali.com wrote: Cisco Nexus platform does it pretty well so they have achieved it. Zaid On Nov 8, 2012, at 3:22 PM, Kasper Adel wrote: Hello, We've been hearing about ISSU for so many years and i didnt hear that any vendor was able to achieve it yet. What is the technical reason behind that? If i understand correctly, the way it will be done would be simply to have extra ASICs/HW to be able to build dual circuits accessing the same memory, and gracefully switch from one to another. Is that right? Thanks, Kim
Re: Whats so difficult about ISSU
I would argue no. The Class 5 softswitches that are around now are off-the-shelf cPCI or ACTA hardware running Linux or some other *nix. The TDM - IP cards are the only sticky point there to be upgraded, but since everything is a mid-plane, you can do rolling N:1 upgrades across the cards with minimal (sub 400msec) impact. There's not a ton special secret sauce there.. To the other point, they probably process way more than 2mbps/s of control traffic during busy hour, especially in geo-redundant configurations as lots of things have to be synchronized. I think you're talking more on the order of 50-120mbps.. Yet all of this works pretty damn well. -- Tim On Mon, Nov 12, 2012 at 12:21 AM, Kasper Adel karim.a...@gmail.com wrote: Hi Frank, Is it because C5 softswitches have expensive hardware, advanced software and dual asics? I would have never imagined that any vendor is capable of upgrading fpd's/ASICs ucode without a hit unless there are multiple chips continuously syncing with each other. Regards, Kim On Monday, November 12, 2012, Frank Bulk wrote: We do it on our Class 5 softswitch ... and it works consistently. There may be a few seconds, once, where a new call can't be made, but most people will re-dial. It just works. It can be done, but the product has to be built with that in mind. Frank -Original Message- From: Kasper Adel [mailto:karim.a...@gmail.com javascript:;] Sent: Thursday, November 08, 2012 5:23 PM To: NANOG list Subject: Whats so difficult about ISSU Hello, We've been hearing about ISSU for so many years and i didnt hear that any vendor was able to achieve it yet. What is the technical reason behind that? If i understand correctly, the way it will be done would be simply to have extra ASICs/HW to be able to build dual circuits accessing the same memory, and gracefully switch from one to another. Is that right? Thanks, Kim
RE: Whats so difficult about ISSU
Compared to our CMTS, our class 5 softswitch cost us less money. Yet our CMTS vendor stopped talking about hitless software upgrades 2 years ago because the upgrade path (from, to, and which software releases you can use) is so limited it's hardly practical. Frank From: Kasper Adel [mailto:karim.a...@gmail.com] Sent: Monday, November 12, 2012 12:22 AM To: Frank Bulk Cc: NANOG list Subject: Re: Whats so difficult about ISSU Hi Frank, Is it because C5 softswitches have expensive hardware, advanced software and dual asics? I would have never imagined that any vendor is capable of upgrading fpd's/ASICs ucode without a hit unless there are multiple chips continuously syncing with each other. Regards, Kim On Monday, November 12, 2012, Frank Bulk wrote: We do it on our Class 5 softswitch ... and it works consistently. There may be a few seconds, once, where a new call can't be made, but most people will re-dial. It just works. It can be done, but the product has to be built with that in mind. Frank -Original Message- From: Kasper Adel [mailto:karim.a...@gmail.com javascript:; ] Sent: Thursday, November 08, 2012 5:23 PM To: NANOG list Subject: Whats so difficult about ISSU Hello, We've been hearing about ISSU for so many years and i didnt hear that any vendor was able to achieve it yet. What is the technical reason behind that? If i understand correctly, the way it will be done would be simply to have extra ASICs/HW to be able to build dual circuits accessing the same memory, and gracefully switch from one to another. Is that right? Thanks, Kim
RE: Whats so difficult about ISSU
Our softswitch vendor talks about control plane bandwidths for geo-redundant configurations on the low end of your numbers. I'd have to drag out the slide deck to see exactly what they recommended. My point is that carrier-class products have demonstrated it's possible. Frank From: Tim Jackson [mailto:jackson@gmail.com] Sent: Monday, November 12, 2012 9:36 AM To: Kasper Adel Cc: Frank Bulk; NANOG list Subject: Re: Whats so difficult about ISSU I would argue no. The Class 5 softswitches that are around now are off-the-shelf cPCI or ACTA hardware running Linux or some other *nix. The TDM - IP cards are the only sticky point there to be upgraded, but since everything is a mid-plane, you can do rolling N:1 upgrades across the cards with minimal (sub 400msec) impact. There's not a ton special secret sauce there.. To the other point, they probably process way more than 2mbps/s of control traffic during busy hour, especially in geo-redundant configurations as lots of things have to be synchronized. I think you're talking more on the order of 50-120mbps.. Yet all of this works pretty damn well. -- Tim On Mon, Nov 12, 2012 at 12:21 AM, Kasper Adel karim.a...@gmail.com wrote: Hi Frank, Is it because C5 softswitches have expensive hardware, advanced software and dual asics? I would have never imagined that any vendor is capable of upgrading fpd's/ASICs ucode without a hit unless there are multiple chips continuously syncing with each other. Regards, Kim On Monday, November 12, 2012, Frank Bulk wrote: We do it on our Class 5 softswitch ... and it works consistently. There may be a few seconds, once, where a new call can't be made, but most people will re-dial. It just works. It can be done, but the product has to be built with that in mind. Frank -Original Message- From: Kasper Adel [mailto:karim.a...@gmail.com javascript:;] Sent: Thursday, November 08, 2012 5:23 PM To: NANOG list Subject: Whats so difficult about ISSU Hello, We've been hearing about ISSU for so many years and i didnt hear that any vendor was able to achieve it yet. What is the technical reason behind that? If i understand correctly, the way it will be done would be simply to have extra ASICs/HW to be able to build dual circuits accessing the same memory, and gracefully switch from one to another. Is that right? Thanks, Kim
Re: Whats so difficult about ISSU
On (2012-11-11 08:50 +0900), Randy Bush wrote: linux has become a fad in the vendor community. it seems to lend legitimacy to their products in some way, witness this discussion. but linux has the gpl poison. so, any code that they wish to keep proprietary is in userland. I've sometimes wondered why Linux is so common, and not FreeBSD. Is it easier to hire people if you use Linux? Or is GPL not really problematic issue, as you can hide your intellectual property in binary kernel modules? -- ++ytti
Re: Whats so difficult about ISSU
I've sometimes wondered why Linux is so common, and not FreeBSD. juniper is currently freebsd Is it easier to hire people if you use Linux? i think it's just perceived as having more customer acceptance. Or is GPL not really problematic issue my lawyer tells me it is very problematic randy
Re: Whats so difficult about ISSU
Saku Ytti (saku) writes: I've sometimes wondered why Linux is so common, and not FreeBSD. Historical reasons and good timing. Is it easier to hire people if you use Linux? As opposed to... ? Or is GPL not really problematic issue, as you can hide your intellectual property in binary kernel modules? You can't. The GPL has provisions for that. Common mistake, several lawsuits have shown. As Randy pointed out, Juniper is FreeBSD inside, and NetApp uses it as well (+ number of other vendors that don't advertise it because they don't have to). Phil
Re: Whats so difficult about ISSU
In article xs4all.m2lie9nhko.wl%ra...@psg.com you write: linux has become a fad in the vendor community. it seems to lend legitimacy to their products in some way, witness this discussion. but linux has the gpl poison. so, any code that they wish to keep proprietary is in userland. Which isn't really a problem, none of the control plane stuff needs to run in the kernel. The only thing that needs to run in the kernel is the device driver(s) to talk to the forwarding plane hardware, but if you use ethernet or infiniband for that communication you don't need any proprietary drivers. Mike.
Re: Whats so difficult about ISSU
On (2012-11-11 08:50 +0900), Randy Bush wrote: linux has become a fad in the vendor community. it seems to lend legitimacy to their products in some way, witness this discussion. but linux has the gpl poison. so, any code that they wish to keep proprietary is in userland. I've sometimes wondered why Linux is so common, and not FreeBSD. Is it easier to hire people if you use Linux? Or is GPL not really problematic issue, as you can hide your intellectual property in binary kernel modules? Years ago, Linux was relatively immature and FreeBSD wasn't. Vendors like Juniper, NetApp, Apple, etc., took whatever suited them from FreeBSD, often ran it on x86, and went on their way. The relatively low legal bar presented little problem, as was intended. Over the years, Linux was ported to more platforms, and has matured a good bit, so now the obvious choice when someone was looking for a cheap platform to build a residential CPE or NAT gateway or whatever became Linux. At the same time, large companies have poured lots of dollars and man-hours into Linux, and this has improved code quality and maturity. However, there are still legal issues relating to the GPL. If you're on supported CPU's, the BSD's are likely to be a better choice if you want to avoid legal entanglements. Otherwise, if you don't mind code disclosure, Linux supports more platforms. Both are relatively mature, feature-full operating systems when used for embedded applications. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Re: Whats so difficult about ISSU
On Sun, Nov 11, 2012 at 1:45 AM, Saku Ytti s...@ytti.fi wrote: ... Or is GPL not really problematic issue, as you can hide your intellectual property in binary kernel modules? GPLv2, which governs the Linux Kernel, does tolorate use of binary kernel modules under some conditions (the classic example is the nVidia driver blob which uses a GPL shim). Regardless, most lawyers would advise a company to avoid being a test case for some of the poorly defined terms used in the license, including derivative work. A recent paper discussing the issue can be found at: LOADED QUESTION: EXAMINING LOADABLE KERNEL MODULES UNDER THE GENERAL PUBLIC LICENSE V2 http://digital.law.washington.edu/dspace-law/bitstream/handle/1773.1/1115/7WJLTA265.pdf?sequence=8 Gary
Re: Whats so difficult about ISSU
On Sun, Nov 11, 2012 at 12:41 PM, Joe Greco jgr...@ns.sol.net wrote: If you're on supported CPU's, the BSD's are likely to be a better choice if you want to avoid legal entanglements. Otherwise, if you don't mind code disclosure, Linux supports more platforms. Both are relatively mature, feature-full operating systems when used for embedded applications. If your silicon vendor supports BSD's, of course. From my (little) experience most vendors SDK will be available to Linux and vxWorks but not BSD. This limits companies that are building equipments based on third parties ASIC to use anything but Linux. My 0.02... Felipe
Re: Whats so difficult about ISSU
On 11/11/12, Miquel van Smoorenburg mik...@xs4all.net wrote: Which isn't really a problem, none of the control plane stuff needs to run in the kernel. The only thing that needs to run in the kernel is the device driver(s) to talk to the forwarding plane Yes. But avoiding kernel mode is a consideration, even before GPL. Perhaps GPL is just another force to discourage developers from doing what they shouldn't be doing anyways -- which is to insert complicated code in the kernel itself to do application-specific things, instead of providing hardware interfaces for applications. You introduce risks if you run control plane things in kernel mode ring0 and not separate control plane functions into user processes. Risks that buggy code will be executed with privilege and corrupt critical data. Risks that a buffer overflow in the SNMP code will crash the kernel and cause the entire control unit to reboot. If instead, each control function is a separate user process, running without privilege in protected mode, then you have a larger amount of fault isolation provided by the hardware -- restart the SNMP process automatically, but leave ISISd/Bgpd alone, and no kernel panic... hardware, but if you use ethernet or infiniband for that communication you don't need any proprietary drivers. Mike. -- -JH
Re: Whats so difficult about ISSU
On Sun, Nov 11, 2012 at 7:31 AM, Felipe Zanchet Grazziotin fel...@starbyte.net wrote: ... If your silicon vendor supports BSD's, of course. From my (little) experience most vendors SDK will be available to Linux and vxWorks but not BSD. This limits companies that are building equipments based on third parties ASIC to use anything but Linux. You are right, of course, since the silicon vendors customers decide what they want the device to support, and that is (currently) Linux and VxWorks. Some BSD folk are trying to change that, by investing their time in the patches/ports needed to support additional embedded processor types/derivatives and make it a viable platform. There is even a Raspberry Pi port now available for FreeBSD as I recall. Ideally those efforts will produce a viable ecosystem for BSD in this space. Gary
Re: Whats so difficult about ISSU
On 11/8/12, Mikael Abrahamsson swm...@swm.pp.se wrote: On Thu, 8 Nov 2012, Phil wrote: NSR isn't ISSU. The equipment vendors call upgrades with NSR failover, ISSU; if their marketing people feel that a 0.5 or 6 second hit is good enough.. If you care about the 0.5 seconds, it's important you speak their language, and require that vague expressions such as In-Service Software Update be clarified. Personally, I don't trust any of it; routers should have regular maintenance windows, period, with a minimum duration of 30 minutes. And software updates to fix known bugs should be done regularly, and during those windows. NSR for ISSU, or ISSU with a small hit called ISSU, is likely inexpensive for the network equipment vendors, because they already invested hundreds of thousands of developer hours in implementing and validating NSR functionality to provide redundancy against device failure. The process of replacing code on a hot device, and restructuring any stored data to match expectations of the new code, without suspending or delaying execution of any code during that process, is possible, but a non-trivial problem: whose solutions add complexity (and therefore a higher risk of bugs and unexpected results) to the upgrade process. You might reduce the hit from 0.5 seconds to 0.01 seconds by implementing true in-place upgrade 90% of the time; but 10% of the time, the online upgrade either fails, because of an issue with the online patch, or unexpected interactions between partially patched functional units, result in a period of incorrect device operation --- until the patching finishes, and continued use of bad data even after patching finished. ISSU contains the wording in service. 6 seconds of outage isn't in service. 0.5 seconds of outage isn't in service. I could accept a few microseconds of outage as being ISSU, but tenths of seconds isn't in service. What is the maximum percentage more would your organization be able to justify paying the network equipment vendor for routers/switches, to reduce the ISSU hit from 0.5 seconds to a few microseconds? :) The main remaining hurdle is updating microcode on linecards, they still need to be rebooted after an upgrade. -- Mikael Abrahamssonemail: swm...@swm.pp.se -- -JH
RE: Whats so difficult about ISSU
We do it on our Class 5 softswitch ... and it works consistently. There may be a few seconds, once, where a new call can't be made, but most people will re-dial. It just works. It can be done, but the product has to be built with that in mind. Frank -Original Message- From: Kasper Adel [mailto:karim.a...@gmail.com] Sent: Thursday, November 08, 2012 5:23 PM To: NANOG list Subject: Whats so difficult about ISSU Hello, We've been hearing about ISSU for so many years and i didnt hear that any vendor was able to achieve it yet. What is the technical reason behind that? If i understand correctly, the way it will be done would be simply to have extra ASICs/HW to be able to build dual circuits accessing the same memory, and gracefully switch from one to another. Is that right? Thanks, Kim
Re: Whats so difficult about ISSU
Hi Frank, Is it because C5 softswitches have expensive hardware, advanced software and dual asics? I would have never imagined that any vendor is capable of upgrading fpd's/ASICs ucode without a hit unless there are multiple chips continuously syncing with each other. Regards, Kim On Monday, November 12, 2012, Frank Bulk wrote: We do it on our Class 5 softswitch ... and it works consistently. There may be a few seconds, once, where a new call can't be made, but most people will re-dial. It just works. It can be done, but the product has to be built with that in mind. Frank -Original Message- From: Kasper Adel [mailto:karim.a...@gmail.com javascript:;] Sent: Thursday, November 08, 2012 5:23 PM To: NANOG list Subject: Whats so difficult about ISSU Hello, We've been hearing about ISSU for so many years and i didnt hear that any vendor was able to achieve it yet. What is the technical reason behind that? If i understand correctly, the way it will be done would be simply to have extra ASICs/HW to be able to build dual circuits accessing the same memory, and gracefully switch from one to another. Is that right? Thanks, Kim
Re: Whats so difficult about ISSU
On 11/12/12 1:21 AM, Kasper Adel wrote: Is it because C5 softswitches have expensive hardware, advanced software and dual asics? I would have never imagined that any vendor is capable of upgrading fpd's/ASICs ucode without a hit unless there are multiple chips continuously syncing with each other. And they only have to process maybe 2mbit/s of control traffic during busy hour. The rest is handled by dedicated hardware/ASIC's. Each one has a fully redundant hardware circuit pack and a bunch of monitoring to switch over in case one fails. Even then, it's fun when some one screws up and reboots it :) -- Bryan Fields 727-409-1194 - Voice 727-214-2508 - Fax http://bryanfields.net
Re: Whats so difficult about ISSU
On Mon, 12 Nov 2012, Bryan Fields wrote: And they only have to process maybe 2mbit/s of control traffic during busy hour. The rest is handled by dedicated hardware/ASIC's. Each one has a fully redundant hardware circuit pack and a bunch of monitoring to switch over in case one fails. I'd imagine it's also because some are written in a language especially designed for the task. http://en.wikipedia.org/wiki/Erlang_(programming_language) ... It supports hot swapping, so that code can be changed without stopping a system.[2] I've been told some people are doing routing control plane implementations in erlang just because of these features, but I'd imagine there is a hurdle getting enough programmers who are experienced in the language. -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: Whats so difficult about ISSU
On (2012-11-09 20:24 -0500), Pete Lumbis wrote: So each IOSd process 'show proc cpu' are separate threads to linux? Yep. The show platform software... commands are used to look at things in To be honest I'm very sceptical about this. I fully accept that IOSd is multithreaded. But I'm having difficulties accepting that each IOSd process would be own thread scheduled by Linux and that native/IOS run-to-completion scheduler isn't used at all. But we're out of scope for ISSU thread in nanog-ml, I think. I do appreciate always when vendors pitch in in public forums, so thank you for that Pete. -- ++ytti
Re: Whats so difficult about ISSU
On (2012-11-10 10:43 +0200), Saku Ytti wrote: So each IOSd process 'show proc cpu' are separate threads to linux? Yep. The show platform software... commands are used to look at things in To be honest I'm very sceptical about this. I fully accept that IOSd is multithreaded. But I'm having difficulties accepting that each IOSd process would be own thread scheduled by Linux and that native/IOS run-to-completion scheduler isn't used at all. Someone who has ASR1004 just ran 'ps auxH' and it's running 3 threads. So IOS XE control-plane is for most parts run-to-completion and relies on classic IOS scheduler and memory-management. I'm not saying this is inherently bad, it is least overhead way to do it, but it also sets requirement for code quality unrealistically high. I'm pretty sure it would be easier to start from scratch and loan IOS BGP, ISIS, OSPF etc code to new project than to try to make IOS be pre-emptive and SMP safe. I'm also not saying IOS XE is bad, JunOS is same design and many consider JunOS good design (I'm not one of them). I think XR and NX-OS are better, more modern examples how to do things now, when we have some extra CPU time in control-plane. It would be interesting ti know what are the roles of these 3 threads. If I'd have to guess, one is IOS control-plane, one is emulation of IOS interrupts and one is abstraction for hardware forwarding. But this is likely far off. -- ++ytti
Re: Whats so difficult about ISSU
as to whether ios/xe is rtc, you may want to see my preso at the last nanog. randy
Re: Whats so difficult about ISSU
On (2012-11-11 00:14 +0900), Randy Bush wrote: as to whether ios/xe is rtc, you may want to see my preso at the last nanog. NANOG56? I only found RPKI Propagation by you. Direct URL would be appreciated. But I really have 0 doubt that IOSd is run-to-completion, exactly like RPD is. But IOSd indeed seems to have 3 OS threads, while RPD is single threaded. As I understand RPD does have green threads though, but that is not helping at all making things simple for JNPR, more if anything, it's making things more complex. If native process separation and even native thread separation cannot be made scale (I'm highly suspicious why it couldn't). Then I wonder why vendors don't use some existing VM, instead of inventing their own. Many existing are free and support green threads and native thread and many-to-many mapping between, so you could get benefit of minimal overhead of green thread and you get benefit of OS level threading (SMP, scheduling) to compartmentalize processes. -- ++ytti
Re: Whats so difficult about ISSU
On 10/11/2012 16:48, Saku Ytti wrote: If native process separation and even native thread separation cannot be made scale (I'm highly suspicious why it couldn't). As the old joke goes, once you introduce threading to fix a problem, you end up with evmeonr eprboelm.s Then I wonder why vendors don't use some existing VM, instead of inventing their own. because of legacy issues. It's easier to rewrite from scratch than to adapt existing code. Nick
Re: Whats so difficult about ISSU
as to whether ios/xe is rtc, you may want to see my preso at the last nanog. NANOG56? I only found RPKI Propagation by you. Direct URL would be appreciated. Look towards the end of the presentation and you'll find run to completion... Steinar Haug, Nethelp consulting, sth...@nethelp.no
Re: Whats so difficult about ISSU
NANOG56? I only found RPKI Propagation by you. Direct URL would be appreciated. apologies. i slid a second preso into my time allotment, and i thought the archive maintainer was going to catenate them. here is the second preso, some measurements of the tcp behavior of bgp. http://archive.psg.com/121022.nanog-bgp-tcp.pdf But I really have 0 doubt that IOSd is run-to-completion nor i. but, as i said but did not write in my preso, consider the following: linux has become a fad in the vendor community. it seems to lend legitimacy to their products in some way, witness this discussion. but linux has the gpl poison. so, any code that they wish to keep proprietary is in userland. randy
Re: Whats so difficult about ISSU
I can't speak for JunOS, but none of the new IOS operating systems are run to completion. This includes IOS-XE, XR and NX-OS. On Fri, Nov 9, 2012 at 2:36 AM, Saku Ytti s...@ytti.fi wrote: When we start on that premise, we can do silly things like write run-to-completion operating systems like IOS and JunOS (rpd). Which means single guy making one bad judgement call, and whole OS is bad. Of course run-to-completion is most optimum way to execute code, if your code is flawless, but that ship has sailed. Possibly when IOS started CPU time was premium and it was cheaper to through code review money at the problem. But today it clearly is cheaper to add power to control plane and have levels of abstraction in control-plane which saves the system from bad code, i.e. design your control-plane assuming code you deliver isn't good.
Re: Whats so difficult about ISSU
On (2012-11-09 08:02 -0500), Pete Lumbis wrote: I can't speak for JunOS, but none of the new IOS operating systems are run to completion. This includes IOS-XE, XR and NX-OS. Really? I thought IOS XE is Linux control-plane on top of where you have monolithic IOSd process? I had chat with Michael Beesley when ASR1k was coming up, and he said Cisco has plans to remove processes from IOS and directly on top of Linux in XE, starting with BGP. But I don't think that has materialized? To me JunOS and IOS XE look very much same, NIX control-plane and magic process with has its own memory management and cooperative multitasking/scheduling? -- ++ytti
Re: Whats so difficult about ISSU
I apologize, I realized I forgot a critical word in my reply. The new Cisco OSes are /NOT/ run to completion. For IOS-XE we have Linux in charge of the scheduler with a multi-threaded IOSd process responsible for the control plane. I'm not familiar with movements to put processes directly on top of the kernel, but this would be a lot more like the NX-OS model where a process like BGP can crash without taking down the system (or the critical IOSd process for example). The down side of this model is that control plane scaling, due to message passing, starts to have a lot of overhead. You can see this in the fact that the NX-OS routing scale is not where IOS-XE is. -Pete On Fri, Nov 9, 2012 at 8:27 AM, Saku Ytti s...@ytti.fi wrote: On (2012-11-09 08:02 -0500), Pete Lumbis wrote: I can't speak for JunOS, but none of the new IOS operating systems are run to completion. This includes IOS-XE, XR and NX-OS. Really? I thought IOS XE is Linux control-plane on top of where you have monolithic IOSd process? I had chat with Michael Beesley when ASR1k was coming up, and he said Cisco has plans to remove processes from IOS and directly on top of Linux in XE, starting with BGP. But I don't think that has materialized? To me JunOS and IOS XE look very much same, NIX control-plane and magic process with has its own memory management and cooperative multitasking/scheduling? -- ++ytti
Re: Whats so difficult about ISSU
On (2012-11-09 13:33 -0500), Pete Lumbis wrote: I apologize, I realized I forgot a critical word in my reply. The new Cisco OSes are /NOT/ run to completion. I did not notice that :). I assumed not was there, and was arguing that I thought IOS XE still is. I know XR and NX-OS aren't. For IOS-XE we have Linux in charge of the scheduler with a multi-threaded IOSd process responsible for the control plane. I'm I'm sceptical if this means there isn't normal IOS run-to-completion scheduler, certainly not all ios processes are separate threads to linux kernel? But I guess this is moving target. Would be interesting to hear how many threads, what are threads relative priorities, what runs in each thread etc. But anyhow just to hear it is threaded, is good news. Does this mean, IOSd can capitalize on multiple cores? (Something JunOS cannot do today) critical IOSd process for example). The down side of this model is that control plane scaling, due to message passing, starts to have a lot of overhead. You can see this in the fact that the NX-OS routing scale is not where IOS-XE is. Yup, luckily you guys stopped freescale pq3 and switch to xeon in ng nexus sup (unfortunately you also killed CMP, which I think every vendor should have). I think the overhead is worth it, built correctly you can scale horizontally and just keep throwing faster RP CPU at it. -- ++ytti
Re: Whats so difficult about ISSU
On Fri, Nov 9, 2012 at 4:00 PM, Saku Ytti s...@ytti.fi wrote: For IOS-XE we have Linux in charge of the scheduler with a multi-threaded IOSd process responsible for the control plane. I'm I'm sceptical if this means there isn't normal IOS run-to-completion scheduler, certainly not all ios processes are separate threads to linux kernel? But I guess this is moving target. Would be interesting to hear how many threads, what are threads relative priorities, what runs in each thread etc. But anyhow just to hear it is threaded, is good news. Does this mean, IOSd can capitalize on multiple cores? (Something JunOS cannot do today) I do not believe that the linux scheduler is run to completion, but to be honest I'm not 100% certain. I know a big reason for IOS-XE was to be able to operate in multicore environments. From a high level you have IOSd as a process with each traditional process (BGP, OSPF, IP Input) as a thread within IOSd. Overall IOS-XE is Linux managing a few processes: IOSd, FMan-RP, CMan-RP (and a few others) FMan deals with adjacencies and CMan deals with modules/cards and IOSd all the interesting stuff. Since Linux is the piece actually running the show IOS-XE gets all the memory management and scheduling benefits that linux has.
Re: Whats so difficult about ISSU
On (2012-11-09 16:58 -0500), Pete Lumbis wrote: I do not believe that the linux scheduler is run to completion, but to be honest I'm not 100% certain. I know a big reason for IOS-XE was to It certainly is not, I'm not proposing it is. I'm saying it is bit of a stretch to believe that IOSd does not have own legacy scheduler and memory management as pulling that switch would have been quite major rework. be able to operate in multicore environments. From a high level you have IOSd as a process with each traditional process (BGP, OSPF, IP Input) as a thread within IOSd. Overall IOS-XE is Linux managing a few processes: IOSd, FMan-RP, CMan-RP (and a few others) FMan deals with adjacencies and CMan deals with modules/cards and IOSd all the interesting stuff. Since Linux is the piece actually running the show IOS-XE gets all the memory management and scheduling benefits that linux has. So each IOSd process 'show proc cpu' are separate threads to linux? -- ++ytti
Re: Whats so difficult about ISSU
So each IOSd process 'show proc cpu' are separate threads to linux? Yep. The show platform software... commands are used to look at things in software, outside of IOSd. Don't get me started on how absurd the command lengths are. === ASR#show proc cpu CPU utilization for five seconds: 0%/0%; one minute: 0%; five minutes: 0% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 1 2 16125 0.00% 0.00% 0.00% 0 Chunk Manager 2 12446 402522 30 0.00% 0.00% 0.00% 0 Load Meter 32927 97933 29 0.00% 0.00% 0.00% 0 CDP Protocol ASR#show platform software process list rp active Name PidPPid Group Id StatusPriority Size -- init 1 0 1 S 20 2207744 kthreadd 2 0 0 S 15 0 ksoftirqd/03 2 0 S 15 0 watchdog/0 4 2 0 S 4294967196 0 events/0 5 2 0 S 15 0 khelper6 2 0 S 15 0 netns 9 2 0 S 15 0 linux_iosd-imag26181 24860 26181 R 20 2010271744 . ASR#show plat soft proc slot rp active monitor cycles 5 top - 17:14:45 up 23 days, 7:15, 0 users, load average: 0.10, 0.11, 0.09 Tasks: 126 total, 2 running, 124 sleeping, 0 stopped, 0 zombie Cpu(s): 1.8%us, 3.0%sy, 0.0%ni, 95.1%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 3874968k total, 1760588k used, 2114380k free, 129520k buffers Swap:0k total,0k used,0k free, 1122644k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 25395 root 20 0 26784 15m 13m S 9.8 0.4 288:48.06 imand 23350 root 20 0 28004 11m 8628 S 5.9 0.3 219:03.66 cmand 13631 root 20 0 2648 1148 884 R 2.0 0.0 0:00.01 top 26181 root 20 0 1917m 406m 143m R 2.0 10.7 365:07.12 linux_iosd-imag 28395 root 20 0 101m 90m 6328 S 2.0 2.4 13:08.97 smand 1 root 20 0 2156 640 552 S 0.0 0.0 0:04.66 init ===
Whats so difficult about ISSU
Hello, We've been hearing about ISSU for so many years and i didnt hear that any vendor was able to achieve it yet. What is the technical reason behind that? If i understand correctly, the way it will be done would be simply to have extra ASICs/HW to be able to build dual circuits accessing the same memory, and gracefully switch from one to another. Is that right? Thanks, Kim
Re: Whats so difficult about ISSU
Cisco Nexus platform does it pretty well so they have achieved it. Zaid On Nov 8, 2012, at 3:22 PM, Kasper Adel wrote: Hello, We've been hearing about ISSU for so many years and i didnt hear that any vendor was able to achieve it yet. What is the technical reason behind that? If i understand correctly, the way it will be done would be simply to have extra ASICs/HW to be able to build dual circuits accessing the same memory, and gracefully switch from one to another. Is that right? Thanks, Kim
Re: Whats so difficult about ISSU
Juniper also offers it on the EX virtual switching platform. Works if you have the correct version of JunOS. On Thu, Nov 8, 2012 at 3:38 PM, Zaid Ali z...@zaidali.com wrote: Cisco Nexus platform does it pretty well so they have achieved it. Zaid On Nov 8, 2012, at 3:22 PM, Kasper Adel wrote: Hello, We've been hearing about ISSU for so many years and i didnt hear that any vendor was able to achieve it yet. What is the technical reason behind that? If i understand correctly, the way it will be done would be simply to have extra ASICs/HW to be able to build dual circuits accessing the same memory, and gracefully switch from one to another. Is that right? Thanks, Kim
Re: Whats so difficult about ISSU
The major vendors have figured it out for the most part by moving to stateful synchronization between control plane modules and implementing non-stop routing. ALU has supported ISSU on minor releases for many years and just added support for major releases. The Cisco Nexus ISSU works well, I've done an upgrade on a 5K switch and it was completely hitless. Juniper and Cisco with the 9K have gone through some hurdles but ISSU is actually usable now if the software versions support it. The main remaining hurdle is updating microcode on linecards, they still need to be rebooted after an upgrade. Phil On Nov 8, 2012, at 6:22 PM, Kasper Adel karim.a...@gmail.com wrote: Hello, We've been hearing about ISSU for so many years and i didnt hear that any vendor was able to achieve it yet. What is the technical reason behind that? If i understand correctly, the way it will be done would be simply to have extra ASICs/HW to be able to build dual circuits accessing the same memory, and gracefully switch from one to another. Is that right? Thanks, Kim
Re: Whats so difficult about ISSU
What i was asking is full ISSU, even with micro code. I assume between Major release there will be microcode upgrade most of the time. On Fri, Nov 9, 2012 at 2:48 AM, Phil bedard.p...@gmail.com wrote: The major vendors have figured it out for the most part by moving to stateful synchronization between control plane modules and implementing non-stop routing. ALU has supported ISSU on minor releases for many years and just added support for major releases. The Cisco Nexus ISSU works well, I've done an upgrade on a 5K switch and it was completely hitless. Juniper and Cisco with the 9K have gone through some hurdles but ISSU is actually usable now if the software versions support it. The main remaining hurdle is updating microcode on linecards, they still need to be rebooted after an upgrade. Phil On Nov 8, 2012, at 6:22 PM, Kasper Adel karim.a...@gmail.com wrote: Hello, We've been hearing about ISSU for so many years and i didnt hear that any vendor was able to achieve it yet. What is the technical reason behind that? If i understand correctly, the way it will be done would be simply to have extra ASICs/HW to be able to build dual circuits accessing the same memory, and gracefully switch from one to another. Is that right? Thanks, Kim
Re: Whats so difficult about ISSU
I have executed successfully on the MX960 with no issues.. EX on the other hand, really depends on your version of JunOS. On Thu, Nov 8, 2012 at 4:19 PM, Alex dreamwave...@yahoo.com wrote: http://www.juniper.net/**techpubs/en_US/junos/topics/** concept/issu-oveview.htmlhttp://www.juniper.net/techpubs/en_US/junos/topics/concept/issu-oveview.html The Juniper ISSU guide. You need two things: 1. Separation of the control plane and forwarding plane 2. 2 routing engines in the same chassis -- the non active RE upgrades first, then when its up and running the active one goes into upgrade mode and control fails over to the secondary RE which is running the upgraded version of the software. I assume it works on any vendor that has 2 REs in the same chassis and the fwd and control planes are separated, and there is a redundancy protocol running between the two REs(like Graceful Switchover on Juniper gear). On 11/09/2012 01:42 AM, Kenneth McRae wrote: Juniper also offers it on the EX virtual switching platform. Works if you have the correct version of JunOS. On Thu, Nov 8, 2012 at 3:38 PM, Zaid Ali z...@zaidali.com wrote: Cisco Nexus platform does it pretty well so they have achieved it. Zaid On Nov 8, 2012, at 3:22 PM, Kasper Adel wrote: Hello, We've been hearing about ISSU for so many years and i didnt hear that any vendor was able to achieve it yet. What is the technical reason behind that? If i understand correctly, the way it will be done would be simply to have extra ASICs/HW to be able to build dual circuits accessing the same memory, and gracefully switch from one to another. Is that right? Thanks, Kim
Re: Whats so difficult about ISSU
I have performed micro code upgrades using ISSU on the Juniper platform. On Thu, Nov 8, 2012 at 4:52 PM, Kasper Adel karim.a...@gmail.com wrote: What i was asking is full ISSU, even with micro code. I assume between Major release there will be microcode upgrade most of the time. On Fri, Nov 9, 2012 at 2:48 AM, Phil bedard.p...@gmail.com wrote: The major vendors have figured it out for the most part by moving to stateful synchronization between control plane modules and implementing non-stop routing. ALU has supported ISSU on minor releases for many years and just added support for major releases. The Cisco Nexus ISSU works well, I've done an upgrade on a 5K switch and it was completely hitless. Juniper and Cisco with the 9K have gone through some hurdles but ISSU is actually usable now if the software versions support it. The main remaining hurdle is updating microcode on linecards, they still need to be rebooted after an upgrade. Phil On Nov 8, 2012, at 6:22 PM, Kasper Adel karim.a...@gmail.com wrote: Hello, We've been hearing about ISSU for so many years and i didnt hear that any vendor was able to achieve it yet. What is the technical reason behind that? If i understand correctly, the way it will be done would be simply to have extra ASICs/HW to be able to build dual circuits accessing the same memory, and gracefully switch from one to another. Is that right? Thanks, Kim
Re: Whats so difficult about ISSU
Does that mean they are the only vendor capable of doing this today?
I am interested in the technology behind this if this is something public,
any ideas?
Thx
On Friday, November 9, 2012, Kenneth McRae wrote:
I have performed micro code upgrades using ISSU on the Juniper platform.
On Thu, Nov 8, 2012 at 4:52 PM, Kasper Adel
karim.a...@gmail.comjavascript:_e({}, 'cvml', 'karim.a...@gmail.com');
wrote:
What i was asking is full ISSU, even with micro code. I assume between
Major release there will be microcode upgrade most of the time.
On Fri, Nov 9, 2012 at 2:48 AM, Phil
bedard.p...@gmail.comjavascript:_e({}, 'cvml', 'bedard.p...@gmail.com');
wrote:
The major vendors have figured it out for the most part by moving to
stateful synchronization between control plane modules and implementing
non-stop routing.
ALU has supported ISSU on minor releases for many years and just added
support for major releases.
The Cisco Nexus ISSU works well, I've done an upgrade on a 5K switch and
it was completely hitless.
Juniper and Cisco with the 9K have gone through some hurdles but ISSU is
actually usable now if the software versions support it.
The main remaining hurdle is updating microcode on linecards, they still
need to be rebooted after an upgrade.
Phil
On Nov 8, 2012, at 6:22 PM, Kasper Adel
karim.a...@gmail.comjavascript:_e({}, 'cvml', 'karim.a...@gmail.com');
wrote:
Hello,
We've been hearing about ISSU for so many years and i didnt hear that
any
vendor was able to achieve it yet.
What is the technical reason behind that?
If i understand correctly, the way it will be done would be simply to
have
extra ASICs/HW to be able to build dual circuits accessing the same
memory,
and gracefully switch from one to another. Is that right?
Thanks,
Kim
Re: Whats so difficult about ISSU
I know some people here have mentioned good experiences with ISSU on
Nexus. I don't doubt that it usually works right, but in my latest
experience with upgrading NX-OS on dual-SUP'ed 7k's, it was hitless
if, by hitless, you mean ~20% packet loss while troubleshooting with
TAC before we found that we had to remove and re-apply QoS policies
from every interface.
Also, depending on the update, linecards might have to be reset.
Oliver
-
Oliver Garraux
Check out my blog: www.GetSimpliciti.com/blog
Follow me on Twitter: twitter.com/olivergarraux
On Thu, Nov 8, 2012 at 8:00 PM, Kasper Adel karim.a...@gmail.com wrote:
Does that mean they are the only vendor capable of doing this today?
I am interested in the technology behind this if this is something public,
any ideas?
Thx
On Friday, November 9, 2012, Kenneth McRae wrote:
I have performed micro code upgrades using ISSU on the Juniper platform.
On Thu, Nov 8, 2012 at 4:52 PM, Kasper Adel
karim.a...@gmail.comjavascript:_e({}, 'cvml', 'karim.a...@gmail.com');
wrote:
What i was asking is full ISSU, even with micro code. I assume between
Major release there will be microcode upgrade most of the time.
On Fri, Nov 9, 2012 at 2:48 AM, Phil
bedard.p...@gmail.comjavascript:_e({}, 'cvml', 'bedard.p...@gmail.com');
wrote:
The major vendors have figured it out for the most part by moving to
stateful synchronization between control plane modules and implementing
non-stop routing.
ALU has supported ISSU on minor releases for many years and just added
support for major releases.
The Cisco Nexus ISSU works well, I've done an upgrade on a 5K switch and
it was completely hitless.
Juniper and Cisco with the 9K have gone through some hurdles but ISSU is
actually usable now if the software versions support it.
The main remaining hurdle is updating microcode on linecards, they still
need to be rebooted after an upgrade.
Phil
On Nov 8, 2012, at 6:22 PM, Kasper Adel
karim.a...@gmail.comjavascript:_e({}, 'cvml', 'karim.a...@gmail.com');
wrote:
Hello,
We've been hearing about ISSU for so many years and i didnt hear that
any
vendor was able to achieve it yet.
What is the technical reason behind that?
If i understand correctly, the way it will be done would be simply to
have
extra ASICs/HW to be able to build dual circuits accessing the same
memory,
and gracefully switch from one to another. Is that right?
Thanks,
Kim
Re: Whats so difficult about ISSU
Heh you will find vendors avoid using the term hitless. I can't think of any
router which supports ISSU that is truly hitless. The ASR9K ISSU states it
will sustain less than 6 seconds of loss...
ISSU is still rife with caveats and incompatibilities as well if you are doing
more advanced things.
Phil
On Nov 8, 2012, at 8:22 PM, Oliver Garraux oli...@g.garraux.net wrote:
I know some people here have mentioned good experiences with ISSU on
Nexus. I don't doubt that it usually works right, but in my latest
experience with upgrading NX-OS on dual-SUP'ed 7k's, it was hitless
if, by hitless, you mean ~20% packet loss while troubleshooting with
TAC before we found that we had to remove and re-apply QoS policies
from every interface.
Also, depending on the update, linecards might have to be reset.
Oliver
-
Oliver Garraux
Check out my blog: www.GetSimpliciti.com/blog
Follow me on Twitter: twitter.com/olivergarraux
On Thu, Nov 8, 2012 at 8:00 PM, Kasper Adel karim.a...@gmail.com wrote:
Does that mean they are the only vendor capable of doing this today?
I am interested in the technology behind this if this is something public,
any ideas?
Thx
On Friday, November 9, 2012, Kenneth McRae wrote:
I have performed micro code upgrades using ISSU on the Juniper platform.
On Thu, Nov 8, 2012 at 4:52 PM, Kasper Adel
karim.a...@gmail.comjavascript:_e({}, 'cvml', 'karim.a...@gmail.com');
wrote:
What i was asking is full ISSU, even with micro code. I assume between
Major release there will be microcode upgrade most of the time.
On Fri, Nov 9, 2012 at 2:48 AM, Phil
bedard.p...@gmail.comjavascript:_e({}, 'cvml',
'bedard.p...@gmail.com');
wrote:
The major vendors have figured it out for the most part by moving to
stateful synchronization between control plane modules and implementing
non-stop routing.
ALU has supported ISSU on minor releases for many years and just added
support for major releases.
The Cisco Nexus ISSU works well, I've done an upgrade on a 5K switch and
it was completely hitless.
Juniper and Cisco with the 9K have gone through some hurdles but ISSU is
actually usable now if the software versions support it.
The main remaining hurdle is updating microcode on linecards, they still
need to be rebooted after an upgrade.
Phil
On Nov 8, 2012, at 6:22 PM, Kasper Adel
karim.a...@gmail.comjavascript:_e({}, 'cvml', 'karim.a...@gmail.com');
wrote:
Hello,
We've been hearing about ISSU for so many years and i didnt hear that
any
vendor was able to achieve it yet.
What is the technical reason behind that?
If i understand correctly, the way it will be done would be simply to
have
extra ASICs/HW to be able to build dual circuits accessing the same
memory,
and gracefully switch from one to another. Is that right?
Thanks,
Kim
Re: Whats so difficult about ISSU
On Thu, 8 Nov 2012, Phil wrote: The major vendors have figured it out for the most part by moving to stateful synchronization between control plane modules and implementing non-stop routing. NSR isn't ISSU. ISSU contains the wording in service. 6 seconds of outage isn't in service. 0.5 seconds of outage isn't in service. I could accept a few microseconds of outage as being ISSU, but tenths of seconds isn't in service. The main remaining hurdle is updating microcode on linecards, they still need to be rebooted after an upgrade. ... and as long as this is the case, there is no ISSU. There is only shorter outages during upgrade compared to a complete reboot. -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: Whats so difficult about ISSU
On Thu, Nov 8, 2012 at 8:13 PM, Mikael Abrahamsson swm...@swm.pp.se wrote: On Thu, 8 Nov 2012, Phil wrote: The major vendors have figured it out for the most part by moving to stateful synchronization between control plane modules and implementing non-stop routing. NSR isn't ISSU. ISSU contains the wording in service. 6 seconds of outage isn't in service. 0.5 seconds of outage isn't in service. I could accept a few microseconds of outage as being ISSU, but tenths of seconds isn't in service. The main remaining hurdle is updating microcode on linecards, they still need to be rebooted after an upgrade. ... and as long as this is the case, there is no ISSU. There is only shorter outages during upgrade compared to a complete reboot. This. There are some wonderfully reconfigurable router hardwares out in the world, and platforms that can dynamically program their forwarding hardware make this seem possible. It's possible to build things such that portions of a single box can be upgraded at a time. With multiple links, or forwarding-paths out to a remote destination, it seems to me that if the upgrade process could just coordinate things and update each piece of forwarding hardware while letting traffic cut over and waiting for it to come back before moving on. I could envision a Juniper M/TX box, where MPLS FRR or an ae interface across FPCs could take backup traffic while a PFE is upgraded. Of course, every possible path would need to be able to survive an FPC being down, and the process would have to have hooks into protocols to know when everything is switched back.
Re: Whats so difficult about ISSU
In vendor-speak ISSU usually refers to 'minimal traffic impact' upgrade. Definition of minimal varies from vendor to vendor and from upgrade to upgrade, depending of which parts of the code need to be upgraded. In general, traffic loss during ISSU is an order of magnitude less than by reloading the whole box or line card as with conventional upgrade. On high level, the ISSU can be divided to two areas: * Control plane / controller card software upgrade * Forwarding plane / line card software upgrade Control card software upgrade is the easy part. In 1+1 controller design, the standby controller card is upgraded first. Next, control card switchover is performed. And last, the remaining controller card is upgraded. Line card upgrade is the more tricky part. On high level, the line card can be divided into forwarding plane and control plane (yes - there is CPU complex on line cards as well). The control plane part of the line card can be upgraded separately and then restarted. If line-card CPU is responsible for generating OSPF hellos, the OSPF session might time out during the restart. However, for most protocols, graceful restart extensions help over any such issues. While the control plane is rebooting, the forwarding bits on the line card continue packet forwarding. The forwarding plane upgrade of the line card is the tricky part. This is the part that will cause the 'short outage' during ISSU. If the code upgrade needs to touch microcode or FPGA code, you will be seeing some traffic loss. It is just the way these chips are built - you cannot reprogram FPGA without taking the FPGA out of service first. The same applies to network processors as well. In theory you could duplicate these forwarding plane chips on line cards and implement simple switch before the PHY. However, I doubt if any vendor has gone this way as it would push line card prices much higher. If your SLAs are built so that no packet loss is acceptable, you need to work around the ISSU limitations: * Use line-level protection on adjacent line cards (LAG, APS1+1, MSP1+1) - when primary card goes down, the backup card will carry the traffic * When upgrading a transit router, route traffic via redundant path before starting transit router upgrade BR, Juuso is such that no traffic loss whatsoever is acceptable, be sure to On Thu, Nov 8, 2012 at 3:22 PM, Kasper Adel karim.a...@gmail.com wrote: Hello, We've been hearing about ISSU for so many years and i didnt hear that any vendor was able to achieve it yet. What is the technical reason behind that? If i understand correctly, the way it will be done would be simply to have extra ASICs/HW to be able to build dual circuits accessing the same memory, and gracefully switch from one to another. Is that right? Thanks, Kim
Re: Whats so difficult about ISSU
On (2012-11-09 01:22 +0200), Kasper Adel wrote: We've been hearing about ISSU for so many years and i didnt hear that any vendor was able to achieve it yet. What is the technical reason behind that? I'd say generally code quality in routers is really really bad, I'm not sure why this is. I think one problem is, that we start on premise that code will be written correctly. When we start on that premise, we can do silly things like write run-to-completion operating systems like IOS and JunOS (rpd). Which means single guy making one bad judgement call, and whole OS is bad. Of course run-to-completion is most optimum way to execute code, if your code is flawless, but that ship has sailed. Possibly when IOS started CPU time was premium and it was cheaper to through code review money at the problem. But today it clearly is cheaper to add power to control plane and have levels of abstraction in control-plane which saves the system from bad code, i.e. design your control-plane assuming code you deliver isn't good. Take a page from erlang team on design principles. I think Arista is walking the right path. They have (hopefully) stable and simplistic state-storage process, from which separate processes can download their states when they crash, which can make crashing virtually transparent to operator. However I think Arista is still running single BGPd etc, I think you should at least rung iBGP and eBGP or maybe even peer gruops in different daemons, so when you get bad UPDATE, it'll crash your eBGPs or one peer-group, instead of all neighbours. Or of course if you keep TCP state and various bgp RIBs in separate location, you won't need to tear down the TCP just because you crash. Someone might argue the overhead is too large, but is it though? MX routers ship with 4 cores RP, out of which you're using 1 core. The overhead isn't that high. Some people write positive things about ISSU in reply, only box where I've seen it work reliably is CAT4500 switches. I've not seen it working in routers. On MX960 my personal hit miss ratio is like 4/5 ISSU work, 1/5 have failed catastrophically, like suddenly PFE is dropping packets as if FW filter was applied, while none is. So we've stopped using ISSU. Point of ISSU is, you're not doing change management notices to your customers, so then it positively has to work, or you're in breach of contract. -- ++ytti
