Re: [NANOG] IOS rootkits
On Sun, May 25, 2008 at 4:26 PM, Christian [EMAIL PROTECTED] wrote: any news of the presentation surfacing anywhere? interested to details of what was discussed yeah. where's the beef?* *not that I don't think said beef exists.
Re: [NANOG] IOS rootkits
On Mon, 19 May 2008, Deepak Jain wrote: Wouldn't this level of verification/authentication of running code be a pretty trivial function via RANCID or similar tool? Absolutely, and it actually makes sense. The problem though is that it is one again an escalation war and counter-inventions keep happening. RANCID will connect remotely and use the local tools to get results, these local tools or their esults can be altered. I understand *why* we are worried about rootkits on individual servers. On essentially closed platforms this isn't going to be rocket science. It may seem odd by today's BCPs, but booting up from golden images via write-protected hardware or TFTP or similar is pretty straightforward -- especially for those of us who run large server farms. That is a neat idea, you mean something like a magic card? Well, the rootkit could still hide in memory, or heck, on the video card if it likes. While XR is not implemented your best bet is reflashing with an updated version, screws up the memory allocations which is apparently a difficult problem to overcome. A POP or node could certainly keep a few servers around that are a permanent repository of these items for all the devices that get images. If you can't trust the boot rom, well, that's an entirely separate matter. I think the issue with rootkits whether server or embedded device is more about infection vector than the maliciousness that could be caused AFTER a compromise has occurred. Here is very much disagree with you. Imagine what you can do with a Trojan horse on a computer, say a server. You could, in effective terms, use it as your own. You'd own it. The same is true for a router. You could sniff the network, steal traffic, use it as a bridge to connect to potnetially any part of your network, hide traffic, etc. The potential for attackrs is almosy cool. Gadi. Deepak Jain Dragos Ruiu wrote: The question this presentation begs for me... is how many of the folks on this list do integrity checking on their routers? You can no longer say this isn't necessary :-). I know FX and a few others are working on toolsets for this... I'll probably have other comments after I see the presentation. This development has all sort of implications for binary signing requirements, etc... cheers, --dr -- World Security Pros. Cutting Edge Training, Tools, and Techniques London, U.K. May 21/22 - 2008http://cansecwest.com pgpkey http://dragos.com/ kyxpgp ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
On Sat, May 17, 2008 at 5:45 PM, [EMAIL PROTECTED] wrote: It's the people who pop up and smear Gadi that I really wonder about. There seems to be no good reason for this, unless possibly they are blackhats of some sort. I remember a few years ago when William Leibzon posted about his work which eventually became completewhois.com and several blackhats popped up and tried to smear him. So when people attack Gadi or anyone else with no substantive facts to justify those attacks, I always assume that they are part of the criminal gangs who drive network abuse in the 21st century. Of course they may just be harmless fools who think that they will become better network operators if they can become part of the in group. Who knows... Actually, Michael, folks who have problems with Gadi, William, and certain other offenders are mainly annoyed with the quantity (high) and quality (low) of their posts. That you seem to have a blind spot in the direction of this particular explanation is dismaying but not surprising. Paul ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
Wouldn't this level of verification/authentication of running code be a pretty trivial function via RANCID or similar tool? I understand *why* we are worried about rootkits on individual servers. On essentially closed platforms this isn't going to be rocket science. It may seem odd by today's BCPs, but booting up from golden images via write-protected hardware or TFTP or similar is pretty straightforward -- especially for those of us who run large server farms. A POP or node could certainly keep a few servers around that are a permanent repository of these items for all the devices that get images. If you can't trust the boot rom, well, that's an entirely separate matter. I think the issue with rootkits whether server or embedded device is more about infection vector than the maliciousness that could be caused AFTER a compromise has occurred. Deepak Jain Dragos Ruiu wrote: The question this presentation begs for me... is how many of the folks on this list do integrity checking on their routers? You can no longer say this isn't necessary :-). I know FX and a few others are working on toolsets for this... I'll probably have other comments after I see the presentation. This development has all sort of implications for binary signing requirements, etc... cheers, --dr -- World Security Pros. Cutting Edge Training, Tools, and Techniques London, U.K. May 21/22 - 2008http://cansecwest.com pgpkey http://dragos.com/ kyxpgp ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
I understand *why* we are worried about rootkits on individual servers. On essentially closed platforms this isn't going to be rocket science. It may seem odd by today's BCPs, but booting up from golden images via write-protected hardware or TFTP or similar is pretty straightforward Since todays bootstrap codes are in EEPROM (or equivalent), if you get root once, you can have root forever. Faking file system content (and real time replacing of code) is the core of any current (good) Linux/Mac/Windows rootkit. Cisco/Juniper/Force10/whatever is just another platform to do the same if you can replace the bootstrap. Modular IOS might even make it easier to do dynamic code insertion. There are platforms (Xbox?, Tivo?, etc.) that try to do cryptographic validation of the code they are loading. Network devices are not yet doing a true cryptograhic validation as far as I know, although one could imagine that that might be a next step to protect against that specific threat (although I seem to recall that bypassing the Xbox validations only took a few months, so it is harder than it first appears to get right). Gary ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
Buhrmaster, Gary wrote: I understand *why* we are worried about rootkits on individual servers. On essentially closed platforms this isn't going to be rocket science. It may seem odd by today's BCPs, but booting up from golden images via write-protected hardware or TFTP or similar is pretty straightforward Since todays bootstrap codes are in EEPROM (or equivalent), if you get root once, you can have root forever. Faking file system content (and real time replacing of code) is the core of any current (good) Linux/Mac/Windows rootkit. Cisco/Juniper/Force10/whatever is just another platform to do the same if you can replace the bootstrap. Modular IOS might even make it easier to do dynamic code insertion. There are platforms (Xbox?, Tivo?, etc.) that try to do cryptographic validation of the code they are loading. Network devices are not yet doing a true cryptograhic validation as far as I know, although one could imagine that that might be a next step to protect against that specific threat (although I seem to recall that bypassing the Xbox validations only took a few months, so it is harder than it first appears to get right). I think that is exactly the point. Once a box has been thoroughly compromised, its almost impossible to bring it back to a known, good state without a complete (reformat). In the case of embedded HW, that may include wiping/rewriting the EEPROMs to a known good state. I don't think this is going to be outside of the purview of Network Operators for very long, no matter what the case. Anti-virii and such are somewhat interesting in the end-system model, but when downtimes need to be scheduled significantly in advance for network operations you either a) prevent infection by much tighter controls at the get-go or b) provide a high-trust way to keep the systems in a known good-state. This, of course, assumes true bugs are kept to a minimum. It does raise significant security concerns for those networks that have employees/contractors/etc with turn-over that could leave a parting gift in their respective networks. Changing passwords isn't really sufficient anymore. DJ ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
Mark Smith wrote: On Sat, 17 May 2008 09:34:19 -0500 [EMAIL PROTECTED] wrote: On Sat, May 17, 2008 at 04:47:02PM +0930, Matthew Moyle-Croft wrote: I'm sure it'll be good for a number of security providers to hawk their wares. If the way of running this isn't out in the wild and it's actually dangerous then a pox on anyone who releases it, especially to gain publicity at the expensive of network operators sleep and well being. May you never find a reliable route ever again. I personally like Gadi's work, but not as much as I like getting my packets to their destination. I personally don't quite understand why netops keep buying proprietary, closed technology for routers, but I'm not and have never been a netop so I'm sure there's good reasons. To me it seems that if you need reliable router hardware, you can buy that from a vendor, but in theory I don't see why the software for routers couldn't be much more open. When I can, I reflash my WAPs with DD-WRT, because at least then I understand the system (and you can't secure what you don't understand), but I am not saying that's much of a comparison. Have you read and security validated every line of open code you're running? Even if you've only read and security validated 99% of it, you're still trusting that the other 1% doesn't have any vulnerabilities in it. There are people who routinely deal in absolutes. we generally call them mathematicians... The rest of us have to operate on a certain amount of uncertainty. Ken's goal I think in 1985 was to open people's eyes to an area of uncertainty which was then relatively poorly understood. It was infeasible in 1985 and certainly remains so outside the confines of some really narrowly focused areas to audit a significant percentage of the code you run. Then again, even if you have audited every line of code, and it is 100% secure, who's to say the compiler used to compile it is ... so you'll have to audit that too. ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
On 17-May-08, at 3:12 AM, Suresh Ramasubramanian wrote: On Sat, May 17, 2008 at 12:47 PM, Matthew Moyle-Croft [EMAIL PROTECTED] wrote: If the way of running this isn't out in the wild and it's actually dangerous then a pox on anyone who releases it, especially to gain publicity at the expensive of network operators sleep and well being. May you never find a reliable route ever again. This needs fixing. It doesnt need publicity at security conferences till after cisco gets presented this stuff first and asked to release an emergency patch. Bullshit. There is nothing to patch. It needs to be presented at conferences, exactly because people will play ostrich and stick their heads in the sand and pretend it can't happen to them, and do nothing about it until someone shows them, yes it can happen and here is how Which is exactly why we've accepted this talk. We've all known this is a possibility for years, but I haven't seen significant motion forward on this until we announced this talk. So in a fashion, this has already helped make people more realistic about their infrastructure devices. And the discussions, and idea interchange that will happen between the smart folks at the conference will undoubtedly usher forth other related issues and creative solutions. Problems don't get fixed until you talk about them. cheers, --dr -- World Security Pros. Cutting Edge Training, Tools, and Techniques London, U.K. May 21/22 - 2008http://cansecwest.com pgpkey http://dragos.com/ kyxpgp ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
Let's put it this way. 1. Yes there's nothing to patch, as such 2. It can be prevented by what's widely regarded as BCP on router security, and has been covered at *nog, in cisco training material, etc etc for quite some time now. I am much less concerned about security conferences discussing this than about the (highly uninformed) publicity that accompanies these conferences. Yes, this sounds a lot more like the bugtraq v/s full disclosure discussion than I'm comfortable with, but I still think this could have been handled a lot better. --srs On Sun, May 18, 2008 at 7:27 PM, Dragos Ruiu [EMAIL PROTECTED] wrote: Bullshit. There is nothing to patch. It needs to be presented at conferences, exactly because people will play ostrich and stick their heads in the sand and pretend it can't happen to them, and do nothing about it until someone shows them, yes it can happen and here is how Which is exactly why we've accepted this talk. We've all known this is a possibility for years, but I haven't seen significant motion forward on this until we announced this talk. So in a fashion, this has already helped make people more realistic about their infrastructure devices. And the discussions, and idea interchange that will happen between the smart folks at the conference will undoubtedly usher forth other related issues and creative solutions. Problems don't get fixed until you talk about them. cheers, --dr ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
On Sun, 18 May 2008, Dragos Ruiu wrote: On 17-May-08, at 3:12 AM, Suresh Ramasubramanian wrote: On Sat, May 17, 2008 at 12:47 PM, Matthew Moyle-Croft [EMAIL PROTECTED] wrote: If the way of running this isn't out in the wild and it's actually dangerous then a pox on anyone who releases it, especially to gain publicity at the expensive of network operators sleep and well being. May you never find a reliable route ever again. This needs fixing. It doesnt need publicity at security conferences till after cisco gets presented this stuff first and asked to release an emergency patch. Bullshit. There is nothing to patch. It needs to be presented at conferences, exactly because people will play ostrich and stick their heads in the sand and pretend it can't happen to them, and do nothing about it until someone shows them, yes it can happen and here is how Which is exactly why we've accepted this talk. We've all known this is a possibility for years, but I haven't seen significant motion forward on this until we announced this talk. So in a fashion, this has already helped make people more realistic about their infrastructure devices. And the discussions, and idea interchange that will happen between the smart folks at the conference will undoubtedly usher forth other related issues and creative solutions. Problems don't get fixed until you talk about them. Dragus, while I hold full disclosure very close and it is dear to my heart, I admit the fact that it can be harmful. Let me link that to network operations. People forget history. A few years back I had a chat with Aleph1 on the first days of bugtraq. He reminded me how things are not always black and white. Full disclosure, while preferable in my ideology, is not the best solution for all. One of the reasons bugtraq was created is because vendors did not care about security, not to mention have a capability to handle security issues, or avoid them to begin with. Full disclosure made a lot of progress for us, and while still a useful tool, with some vendors it has become far more useful to report to them and let them provide with a solution first. In the case of routers which are used for infrastructure as well as critical infrastructure, it is my strong belief that full disclosure is, at least at face value, a bad idea. I'd like to think Cisco, which has shown capability in the past, is as responsible as it should be on these issues. Experience tells me they have a ways to go yet even if they do have good processes in place with good people to employ them. I'd also like to think tier-1 and tier-2 providers get patches first before such releases. This used to somewhat be the case, last I checked it no longer is -- for legitimate concerns by Cisco. has this changed? So, if we don't patch the infrastructure up first, and clients don't know of problems until they are public for their own security (an argument that holds water only so much) perhaps it is the time for full disclosure to be considered a viable alternative. All that aside, this is a rootkit, not a vulnerability. There is no inherent vulnerability to patch (unless it is very local). There is the vulnerability of operators who don't so far even consider trojan horses as a threat, and the fact tools don't exist for them to do something once they do. Gadi. cheers, --dr -- World Security Pros. Cutting Edge Training, Tools, and Techniques London, U.K. May 21/22 - 2008http://cansecwest.com pgpkey http://dragos.com/ kyxpgp ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
On Sun, 18 May 2008, Suresh Ramasubramanian wrote: Let's put it this way. 1. Yes there's nothing to patch, as such 2. It can be prevented by what's widely regarded as BCP on router security, and has been covered at *nog, in cisco training material, etc etc for quite some time now. I am much less concerned about security conferences discussing this than about the (highly uninformed) publicity that accompanies these conferences. Yes, this sounds a lot more like the bugtraq v/s full disclosure discussion than I'm comfortable with, but I still think this could have been handled a lot better. It's easy to blame researchers for doing their studies, but the fact is, if one whitehat researcher has done work on it, it is already exploited in the wild. Gadi. --srs On Sun, May 18, 2008 at 7:27 PM, Dragos Ruiu [EMAIL PROTECTED] wrote: Bullshit. There is nothing to patch. It needs to be presented at conferences, exactly because people will play ostrich and stick their heads in the sand and pretend it can't happen to them, and do nothing about it until someone shows them, yes it can happen and here is how Which is exactly why we've accepted this talk. We've all known this is a possibility for years, but I haven't seen significant motion forward on this until we announced this talk. So in a fashion, this has already helped make people more realistic about their infrastructure devices. And the discussions, and idea interchange that will happen between the smart folks at the conference will undoubtedly usher forth other related issues and creative solutions. Problems don't get fixed until you talk about them. cheers, --dr ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
On 18-May-08, at 7:11 AM, Suresh Ramasubramanian wrote: 2. It can be prevented by what's widely regarded as BCP on router security, and has been covered at *nog, in cisco training material, etc etc for quite some time now. I am much less concerned about security conferences discussing this than about the (highly uninformed) publicity that accompanies these conferences. I'm not going to touch the disclosure or not debate... it's been done. But I will agree to disagree with you about the above two points. First of all about prevention, I'm not at all sure about this being covered by existing router security planning / BCP. I don't believe most operators reflash their routers periodically, nor check existing images (particularly because the tools for this integrity verification don't even exist). If I'm wrong about this I would love to be corrected with pointers to the tools. Regarding the second point, I also lament the often liberal doses of alarmism/FUD that get plastered over the popular media whenever complicated technical issues are discussed - but unless we have some have the discussions, and information dispersal, then the misconceptions have no chance of being dispelled. The threat of misinformed press does not seem to be sufficient to justify censuring open discussion of the issues imho. One of the thing I truly enjoy about the conferences we organize, is seeing the synergism that occurs when multiple minds focus on these security issues at the conferences. When the analysis is parallelized over multiple brains, inevitably the creative solutions that occur from the congregation of different viewpoints and ideas is pleasantly surprising, and powerful. I've seen numerous examples of this: even just last April I had a chance to be a fly on the wall at a discussion between Jacob Appelbaum and Theo DeRaadt talking about the cold memory attacks research Jacob started - the result of which was that during the discussion it was realized that with the addition of about 30 lines of code in the power fail interrupt handler a large segment of those attacks could be nullified, as they are now on OpenBSD. If the discussion hadn't happened, the creative solution to it would have never arisen. These kinds of out of the box solutions frequently arise out of multi-person debate and free association that follows discussions of serious issues - no-one has the whole picture and adding other's viewpoints often brings superior solutions to problems up. So in my opinion the benefits of discussing serious issues at conferences far outweigh the potential drawbacks of misguided media coverage of them. What I infer from your post is that you are of the opinion that issues such as this rootkit prototype should be reported to CSIRT and then shuffled under a carpet. To which I respond that that kind of attitude has led to what I currently consider to be an inappropriate level of concern and awareness amongst service providers of the seriousness of this threat. Cisco has some great guys, but surely discussion of this threat amongst the wider security community will lead to more and better solutions than Cisco operating in a vacuum. And more importantly this issue is not a Cisco issue - the basic threat vector should be a concern to other infrastructure equipment manufacturers too. Until we talk about it, we cannot find the right responses to the problem, and experts talking about it usually leads to better and more comprehensive solutions than single persons or smaller groups working in isolation. cheers, --dr -- World Security Pros. Cutting Edge Training, Tools, and Techniques London, U.K. May 21/22 - 2008http://eusecwest.com pgpkey http://dragos.com/ kyxpgp ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
Dragos Ruiu wrote: First of all about prevention, I'm not at all sure about this being covered by existing router security planning / BCP. I don't believe most operators reflash their routers periodically, nor check existing images (particularly because the tools for this integrity verification don't even exist). If I'm wrong about this I would love to be corrected with pointers to the tools. I have 6 years worth of rancid logs for every time the reported number of blocks in use on my flash changes, I imagine others do as well. That's hardly the silver bullet however. We as I imagine others do expended a fair amount of cycles monitoring who it is that our routers are talking to and protecting the integrity of the communications channels that they use (bgp, ospf, ssh, tftp etc), If a router has a tcp connection to someplace it shouldn't we'll probably know about it. If it's announcing a prefix it shouldn't be, we'll probably know about it, those are the easy ones though. There are some things one might consider adding in terms of auditing, comparing the running image more closely to the one in flash for example, peroidic checksum of the on onflash image, after downloading to another host would be another. I'm not sure that I'd trust the later given the rooted box can I suppose hand you an unmodified version of the subverted image. In the end if you subvert a router, presumably you're doing it for a purpose and given what the device does, that purpose is probably detectable in a well instrumented network. It is desirable I expect to insure that any locally stored security credentials that might be subverted not be usable when connecting to another router, that applies in a absence of root kits however. Regarding the second point, I also lament the often liberal doses of alarmism/FUD that get plastered over the popular media whenever complicated technical issues are discussed - but unless we have some have the discussions, and information dispersal, then the misconceptions have no chance of being dispelled. The threat of misinformed press does not seem to be sufficient to justify censuring open discussion of the issues imho. ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
On Sun, 18 May 2008, Joel Jaeggli wrote: Dragos Ruiu wrote: First of all about prevention, I'm not at all sure about this being covered by existing router security planning / BCP. I don't believe most operators reflash their routers periodically, nor check existing images (particularly because the tools for this integrity verification don't even exist). If I'm wrong about this I would love to be corrected with pointers to the tools. I have 6 years worth of rancid logs for every time the reported number of blocks in use on my flash changes, I imagine others do as well. That's hardly the silver bullet however. We as I imagine others do expended a fair amount of cycles monitoring who it is that our routers are talking to and protecting the integrity of the communications channels that they use (bgp, ospf, ssh, tftp etc), If a router has a tcp connection to someplace it shouldn't we'll probably know about it. If it's announcing a prefix it shouldn't be, we'll probably know about it, those are the easy ones though. I am very happy to hear you do these... very useful and will catch quite a bit. There are some things one might consider adding in terms of auditing, comparing the running image more closely to the one in flash for example, peroidic checksum of the on onflash image, after downloading to another host would be another. I'm not sure that I'd trust the later given the rooted box can I suppose hand you an unmodified version of the subverted image. The result from your check can easily be modified, first thing I would have changed is the checker. Say you did this from a usb stick--I'd just hide the rootkit in memory. In the end if you subvert a router, presumably you're doing it for a purpose and given what the device does, that purpose is probably detectable in a well instrumented network. Subversion may not be the goal. A router is perfect for faking outgoing traffic. This traffic can contain stolen sniffed or relayed data. It is desirable I expect to insure that any locally stored security credentials that might be subverted not be usable when connecting to another router, that applies in a absence of root kits however. ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
Gadi Evron wrote: On Sun, 18 May 2008, Joel Jaeggli wrote: Dragos Ruiu wrote: First of all about prevention, I'm not at all sure about this being covered by existing router security planning / BCP. I don't believe most operators reflash their routers periodically, nor check existing images (particularly because the tools for this integrity verification don't even exist). If I'm wrong about this I would love to be corrected with pointers to the tools. I have 6 years worth of rancid logs for every time the reported number of blocks in use on my flash changes, I imagine others do as well. That's hardly the silver bullet however. We as I imagine others do expended a fair amount of cycles monitoring who it is that our routers are talking to and protecting the integrity of the communications channels that they use (bgp, ospf, ssh, tftp etc), If a router has a tcp connection to someplace it shouldn't we'll probably know about it. If it's announcing a prefix it shouldn't be, we'll probably know about it, those are the easy ones though. I am very happy to hear you do these... very useful and will catch quite a bit. There are some things one might consider adding in terms of auditing, comparing the running image more closely to the one in flash for example, peroidic checksum of the on onflash image, after downloading to another host would be another. I'm not sure that I'd trust the later given the rooted box can I suppose hand you an unmodified version of the subverted image. The result from your check can easily be modified, first thing I would have changed is the checker. That is a normal thing to do with rootkits (return bogus results). Which is part of the reason I suggested that method I did. Short of pulling the flash you're not going to get a fully unbiased view of what's it on it thusly the audit process has some limitations. A TCPA style boot process would be a better approach. It's certainly not a quick fix since it in general can't be retrofited to existing products. Say you did this from a usb stick--I'd just hide the rootkit in memory. In the end if you subvert a router, presumably you're doing it for a purpose and given what the device does, that purpose is probably detectable in a well instrumented network. Subversion may not be the goal. A router is perfect for faking outgoing traffic. This traffic can contain stolen sniffed or relayed data. If my device is now taking marching orders from a third party then by definition it is subverted, regardless of agency or activity. sub verte - turn from under ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
On Sun, 18 May 2008, Joel Jaeggli wrote: The result from your check can easily be modified, first thing I would have changed is the checker. That is a normal thing to do with rootkits (return bogus results). Which is part of the reason I suggested that method I did. Short of pulling the flash you're not going to get a fully unbiased view of what's it on it thusly the audit process has some limitations. A TCPA style boot process would be a better approach. It's certainly not a quick fix since it in general can't be retrofited to existing products. EuSecWest released this interview about the rootkit with its creator, Sebastian Muniz of Core Security, it also mentions a third party product to detect some of these issues. Thank whatever diety we like for FX's work, as obviously Cisco isn't there yet. http://eusecwest.com/sebastian-muniz-da-ios-rootkit.html Say you did this from a usb stick--I'd just hide the rootkit in memory. In the end if you subvert a router, presumably you're doing it for a purpose and given what the device does, that purpose is probably detectable in a well instrumented network. Subversion may not be the goal. A router is perfect for faking outgoing traffic. This traffic can contain stolen sniffed or relayed data. If my device is now taking marching orders from a third party then by definition it is subverted, regardless of agency or activity. sub verte - turn from under ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
http://eusecwest.com/sebastian-muniz-da-ios-rootkit.html its worth a digg... http://digg.com/security/Da_IOS_Rootkit regards -- Use your imagination not to scare yourself to death but to inspire yourself to life. Les enfants teribbles - research and deployment Marc Manthey - head of research and innovation Hildeboldplatz 1a D - 50672 Köln - Germany Tel.:0049-221-3558032 Mobil:0049-1577-3329231 jabber :[EMAIL PROTECTED] blog : http://www.let.de ipv6 http://www.ipsix.org xing : https://www.xing.com/profile/Marc_Manthey ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
On Mon, May 19, 2008 at 2:03 AM, Dragos Ruiu [EMAIL PROTECTED] wrote: So in my opinion the benefits of discussing serious issues at conferences far outweigh the potential drawbacks of misguided media coverage of them. What I infer from your post is that you are of the opinion that issues such Well, there are any number of closed, no media, relevant people only conferences, or communities like nsp-sec, that come in useful Report to CSIRT by all means but that doesnt imply brush it under the carpet. Getting releases out and fixes (if only router management bcp like in Joel Jaeggli's post) without various people spreading FUD about it should certainly be an achievable goal? srs ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
What if some good comes from this root kit? For instance, what if it lets us fix things like DOM on non-Cisco XENPAKs and SFPs? Or lets us un-cripple our 6500 chassis to run the code we want? Of course, given the messenger, I'm sure it's just hype to help bolster Gadi's security practice, and will prove to be no big deal. Paul ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
Simon Lockhart wrote: How long before we need to install Anti-virus / Anti-root-kit software on our routers? Nah - we'll just replace them all with Macs. They don't need anti-virus ... :-) MMC Simon ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Suresh Ramasubramanian wrote: On Sat, May 17, 2008 at 12:47 PM, Matthew Moyle-Croft [EMAIL PROTECTED] wrote: If the way of running this isn't out in the wild and it's actually dangerous then a pox on anyone who releases it, especially to gain publicity at the expensive of network operators sleep and well being. May you never find a reliable route ever again. This needs fixing. It doesnt need publicity at security conferences till after cisco gets presented this stuff first and asked to release an emergency patch. --srs According to Cisco, there is nothing to patch: http://www.cisco.com/warp/public/707/cisco-sr-20080516-rootkits.shtml Jon Kibler - -- Jon R. Kibler Chief Technical Officer Advanced Systems Engineering Technology, Inc. Charleston, SC USA o: 843-849-8214 c: 843-224-2494 s: 843-564-4224 My PGP Fingerprint is: BAA2 1F2C 5543 5D25 4636 A392 515C 5045 CF39 4253 -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkgusjEACgkQUVxQRc85QlO5kACfaZtij86HqIH540xeH+Uh/NyI ccQAnjiRCMFnLxk/Ew9EuUKDzdLN6HQZ =BCdw -END PGP SIGNATURE- == Filtered by: TRUSTEM.COM's Email Filtering Service http://www.trustem.com/ No Spam. No Viruses. Just Good Clean Email. ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
On Sat, 17 May 2008, Suresh Ramasubramanian wrote: On Sat, May 17, 2008 at 12:47 PM, Matthew Moyle-Croft [EMAIL PROTECTED] wrote: If the way of running this isn't out in the wild and it's actually dangerous then a pox on anyone who releases it, especially to gain publicity at the expensive of network operators sleep and well being. May you never find a reliable route ever again. This needs fixing. It doesnt need publicity at security conferences till after cisco gets presented this stuff first and asked to release an emergency patch. I'd like to discuss: 1. What is it we are talking about. 2. Why it is serious. 3. What we can do to defend ourselves. I'll be brief as this is not a briefing. You are absolutely right on the sentiment, but miss the point on this particular issue. I agree with you that in most cases, software vulnerability issues should be resolved with the vendor first, especially where critical infrastructure is involved. This is not only about exploiting a vulnerability. In this case it the the very realization that these issues exist (namely being able to run Trojan horses on IOS systems AND/or hiding their presense) is what we are discussing. Router security as far as most operators are concerned includes the following issues: software version (now update), configuration, ACL and authentication (password) security. I include subjects such as BGP MD5 in configuration. These issues are indeed important and very neglected, after all, how many 0wned routers can be found that respond to cisco/cisco? The main difference here is that we are now at a cross-roads where the face of router security changes, It is that the realization that: 1. A router is not an hardware device, it is an embedded device with a software operating system. As such it is as vulnerable to malware (wide-spreading--worm, or targeted--Trojan horse) as a Windows machine is.) 2. There are no real tools today for us to be able to detect such malicious activity on a router, listing processes doesn't cut it. 3. What tools exist, which I hope to secure permission to discuss later on, are only from third parties. This is not about fear mongering, it's about facing reality how about how Cisco handles security threats to their customer base before such an issue becomes a public concern--namely, ignoring its very existence, at least as far as the public can see. The point is, I don't want to rely on third parties for my router's security, even if I trust the said third party. Gadi. ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
On Sat, 17 May 2008, Simon Lockhart wrote: On Sat May 17, 2008 at 04:47:02PM +0930, Matthew Moyle-Croft wrote: Paul Wall wrote: What if some good comes from this root kit? I'm sure it'll be good for a number of security providers to hawk their wares. How long before we need to install Anti-virus / Anti-root-kit software on our routers? Very astute. Sadly, this is already being done by a few people I know. No AV vendor has such a tool to offer you, so don't bother asking them. The question is, can you afford not to? The answer may be yes, you can afford for your router to be a spying machine for the enemy/competitor, and you can afford for it to be a bot participating in DDoS (as currently, for example, many *nix routers are known to be). The question is who can't afford for these things to happen... Gadi. Simon -- Simon Lockhart | * Sun Server Colocation * ADSL * Domain Registration * Director|* Domain Web Hosting * Internet Consultancy * Bogons Ltd | * http://www.bogons.net/ * Email: [EMAIL PROTECTED] * ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
The question is who can't afford for these things to happen... Gadi. I can't help but feel you're pushing fear to further some other interest here Gadi. Do you actually have live examples of this or able to demonstrate it or are you just theorising about it all? MMC ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
On Sat, 17 May 2008, Matthew Moyle-Croft wrote: The question is who can't afford for these things to happen... Gadi. I can't help but feel you're pushing fear to further some other interest here Gadi. It is alright to have feelings. Gadi. ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
On Sat, 17 May 2008 07:03:58 -0500 (CDT) Gadi Evron [EMAIL PROTECTED] wrote: On Sat, 17 May 2008, Matthew Moyle-Croft wrote: The question is who can't afford for these things to happen... Gadi. I can't help but feel you're pushing fear to further some other interest here Gadi. It is alright to have feelings. The rational thing to do is to move beyond fear. -- Sheep are slow and tasty, and therefore must remain constantly alert. - Bruce Schneier, Beyond Fear ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
It is alright to have feelings. Gadi. So I ask again, expecting nothing but another flippant answer: Do you actually have live examples of this or able to demonstrate it or are you just theorising about it all? MMC ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
I'd love to know what magical mystical protection your routers have that will enable them to avoid the same fate as every other device and operating system has. There's only one thing up there that doesn't have known rootkits in the wild. Yet. The question isn't IF routers have security vunerabilities, but whether Gadi has an example he can demonstrate now of installing a root kit on an IOS router NOW or not. MMC ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
On Sat, 17 May 2008, Matthew Moyle-Croft wrote: It is alright to have feelings. Gadi. So I ask again, expecting nothing but another flippant answer: I will honour you flame-bait, but only once. Do you actually have live examples of this or able to demonstrate it or are you just theorising about it all? Your question is irrelevant to our discussion, as I obviously base myself on the first email in this thread discussing the poc (?) about to be released, and my own statements from that first email in which I mention I will not discuss my own experience on the subject of rootkit risks and solutions until said poc (?) is released due to matters of honour. MMC ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
On Sat, 17 May 2008, Matthew Moyle-Croft wrote: I'd love to know what magical mystical protection your routers have that will enable them to avoid the same fate as every other device and operating system has. There's only one thing up there that doesn't have known rootkits in the wild. Yet. The question isn't IF routers have security vunerabilities Nope, the question is not about if routers have security vulnerabilities. The question is how operators and organizations can defend their routers against rootkits, and cisco's practices. MMC ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
On Sat, May 17, 2008 at 04:47:02PM +0930, Matthew Moyle-Croft wrote: I'm sure it'll be good for a number of security providers to hawk their wares. If the way of running this isn't out in the wild and it's actually dangerous then a pox on anyone who releases it, especially to gain publicity at the expensive of network operators sleep and well being. May you never find a reliable route ever again. I personally like Gadi's work, but not as much as I like getting my packets to their destination. I personally don't quite understand why netops keep buying proprietary, closed technology for routers, but I'm not and have never been a netop so I'm sure there's good reasons. To me it seems that if you need reliable router hardware, you can buy that from a vendor, but in theory I don't see why the software for routers couldn't be much more open. When I can, I reflash my WAPs with DD-WRT, because at least then I understand the system (and you can't secure what you don't understand), but I am not saying that's much of a comparison. So, speaking of hawking wares... ;-) Since I see some disclosure discussions brewing here, so I thought I'd mention that I have a free online book on security, and I'm trying to capture all the arguments about disclosure policies so that they don't ever have to be rehashed. Instead, we can just point someone to it, and move on. Here's the section on disclosure: http://www.subspacefield.org/security/security_concepts.html#tth_sEc25.1 I'm numbering them for your convenience, so that if for some reason you want to state a particular argument, you can compress the conversation by simply giving its index. ;-) HHOS, Travis -- Crypto ergo sum. https://www.subspacefield.org/~travis/ If you are a spammer, please email [EMAIL PROTECTED] to get blacklisted. ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
Gadi Evron wrote: The question isn't IF routers have security vunerabilities Nope, the question is not about if routers have security vulnerabilities. The question is how operators and organizations can defend their routers against rootkits, and cisco's practices. The existence proof of a root kit does little if anything to change how one protects and secures the control plane. ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
* Joel Jaeggli: The existence proof of a root kit does little if anything to change how one protects and secures the control plane. | Network administrators are not able to observe Lawful Intercept is | enabled. No Lawful Intercept program messages or error messages are ever | displayed on the console. http://www.cisco.com/en/US/docs/ios/12_0s/feature/guide/lawf_int.html This is a Sony-style rootkit, but it certainly demonstrate that the concept is feasible (surprise). ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
The question isn't IF routers have security vunerabilities, but whether Gadi has an example he can demonstrate now of installing a root kit on an IOS router NOW or not. That's not really the question. In fact, there are two questions. First, are routers really embedded devices running a software operating system? Secondly who can you trust in regards to security of your routers. On the first question, I don't think anyone will argue that routers are not capable of being compromised by software. Some may argue that compromising the software from the public Internet is virtually impossible and statistically unlikely, but most organizations now realize that hard shell security is a fantasy. The real danger is an insider who has enable on the router and who takes money to install a trojan, or the reseller who sells you a router with trojans already installed. Let's face it, if the NSA now believes there is a serious risk of counterfeit hardware that has been modified to contain hardware trojans, then the much easier to achieve software trojans should be a greater risk, and therefore worthy of attention. But the second question is the more interesting one in the context of NANOG. Can we trust Gadi? Can we trust the people who pop up and try to smear Gadi in some way? I haven't a clear answer here except to say that Gadi is a well-known person whose biases and possible motives (consultancy work) are well known. Same thing could be said about Cisco or Microsoft and this may make Gadi (or Cisco) more trustable about some things and less trustable about others. But everybody on this list deals with certainties like this every day. It's the people who pop up and smear Gadi that I really wonder about. There seems to be no good reason for this, unless possibly they are blackhats of some sort. I remember a few years ago when William Leibzon posted about his work which eventually became completewhois.com and several blackhats popped up and tried to smear him. So when people attack Gadi or anyone else with no substantive facts to justify those attacks, I always assume that they are part of the criminal gangs who drive network abuse in the 21st century. Of course they may just be harmless fools who think that they will become better network operators if they can become part of the in group. Who knows... Personally, I am not particularly disturbed that security vulnerabilities are announced with few substantive details. That's just the way things are normally done in the real world. --Michael Dillon ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
On Sat, 17 May 2008, Felix 'FX' Lindner wrote: But I don't see a reason for panic and Cisco is at least partially right with their response ( http://www.cisco.com/en/US/products/products_security_response09186a0080997783.html ) to the whole issue: someone still needs a privilege level 15 VTY on your router and no matter what press is currently making of the rootkit, this prerequisite step is non-trivial (or should be, depending on your configuration). On this rootkit and IOS security and how it works FX's word is of the most qualified. cheers FX -- Recurity Labs GmbH | Felix 'FX' Lindner http://www.recurity-labs.com | [EMAIL PROTECTED] Wrangelstrasse 4 | Fon: +49 30 69539993-0 10997 Berlin | PGP: A740 DE51 9891 19DF 0D05 Germany | 13B3 1759 C388 C92D 6BBB HRB 105213 B, Amtsgericht Charlottenburg, GF Felix Lindner ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
On Sun, 18 May 2008, Mark Smith wrote: Reflections on Trusting Trust http://cm.bell-labs.com/who/ken/trust.html That is the #1 paper on security anyone can read, and reading your email I was about to ask if you ever read it. It certainly is my fav. Thanks for reminding us all of the url. Gadi. ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
On Sun, 18 May 2008 09:29:47 +0930 Mark Smith [EMAIL PROTECTED] wrote: On Sat, 17 May 2008 09:34:19 -0500 [EMAIL PROTECTED] wrote: On Sat, May 17, 2008 at 04:47:02PM +0930, Matthew Moyle-Croft wrote: I'm sure it'll be good for a number of security providers to hawk their wares. If the way of running this isn't out in the wild and it's actually dangerous then a pox on anyone who releases it, especially to gain publicity at the expensive of network operators sleep and well being. May you never find a reliable route ever again. I personally like Gadi's work, but not as much as I like getting my packets to their destination. I personally don't quite understand why netops keep buying proprietary, closed technology for routers, but I'm not and have never been a netop so I'm sure there's good reasons. To me it seems that if you need reliable router hardware, you can buy that from a vendor, but in theory I don't see why the software for routers couldn't be much more open. When I can, I reflash my WAPs with DD-WRT, because at least then I understand the system (and you can't secure what you don't understand), but I am not saying that's much of a comparison. snip As the cliche goes, If you want something done properly, you have to do it yourself. If you can't do it (all) yourself, because you don't have the time and the expertise, then inherently you have to place a should have been have the time and/or the expertise level of trust in other people. Regards, Mark. -- Sheep are slow and tasty, and therefore must remain constantly alert. - Bruce Schneier, Beyond Fear ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog -- Sheep are slow and tasty, and therefore must remain constantly alert. - Bruce Schneier, Beyond Fear ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
Gadi, Please try to keep the self-promotion to a minimum, and come back when you have meaningful data to share with operators. Examples would include a list of affected platforms and code revisions, as well as preventative measures. Thank you, Paul On Fri, May 16, 2008 at 9:06 PM, Gadi Evron [EMAIL PROTECTED] wrote: At the upcoming EusecWest Sebastian Muniz will apparently unveil an IOS rootkit. skip below for the news item itself. We've had discussions on this before, here and elsewhere. I've been heavily attacked on the subject of considering router security as an issue when compared to routing security. I have a lot to say about this, looking into this threat for a few years now and having engaged different organizations within Cisco on the subject in the past. Due to what I refer to as an NDA of honour I will just relay the following until it is officially public, then consider what should be made public, including: 1. Current defense startegies possible with Cisco gear 2. Third party defense strategies (yes, they now exist) 2. Cisco response (no names or exact quotes will likely be given) 3. A bet on when such a rootkit would be public, and who won it (participants are.. relevant people). From: http://www.networkworld.com/news/2008/051408-hacker-writes-rootkit-for-ciscos.html A security researcher has developed malicious rootkit software for Cisco's routers, a development that has placed increasing scrutiny on the routers that carry the majority of the Internet's traffic. Sebastian Muniz, a researcher with Core Security Technologies, developed the software, which he will unveil on May 22 at the EuSecWest conference in London. Gadi Evron. ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] IOS rootkits
On Fri, 16 May 2008, Paul Wall wrote: Gadi, Please try to keep the self-promotion to a minimum, and come back when you have meaningful data to share with operators. Examples would include a list of affected platforms and code revisions, as well as preventative measures. Name on the door, money to be sent via paypal. I will sign my playgirl cover for 5 USD each. This is operational, and it is about me saying na na na na na, na na na na na na to a discussion from two years ago. I have every intention to gloat, but I will keep it to a minimum. Yes? Gadi. On Fri, May 16, 2008 at 9:06 PM, Gadi Evron [EMAIL PROTECTED] wrote: At the upcoming EusecWest Sebastian Muniz will apparently unveil an IOS rootkit. skip below for the news item itself. We've had discussions on this before, here and elsewhere. I've been heavily attacked on the subject of considering router security as an issue when compared to routing security. I have a lot to say about this, looking into this threat for a few years now and having engaged different organizations within Cisco on the subject in the past. Due to what I refer to as an NDA of honour I will just relay the following until it is officially public, then consider what should be made public, including: 1. Current defense startegies possible with Cisco gear 2. Third party defense strategies (yes, they now exist) 2. Cisco response (no names or exact quotes will likely be given) 3. A bet on when such a rootkit would be public, and who won it (participants are.. relevant people). From: http://www.networkworld.com/news/2008/051408-hacker-writes-rootkit-for-ciscos.html A security researcher has developed malicious rootkit software for Cisco's routers, a development that has placed increasing scrutiny on the routers that carry the majority of the Internet's traffic. Sebastian Muniz, a researcher with Core Security Technologies, developed the software, which he will unveil on May 22 at the EuSecWest conference in London. Gadi Evron. ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
