Re: YUM security issues...
On 21 July 2008, Josh Bressers wrote: On 19 July 2008, Justin Cappos wrote: By the way, did you remove the ability for mirror admins to select a subnet where they'll serve all of the traffic? We're particularly concerned about this issue in the short term. We took our mirror down (mirror1.lockdownhosting.com) quite a while ago so we can't check for ourselves. I don't know the answer to this, so I'm adding the Fedora Infrastructure list to the CC. For you Fedora Infrastructure guys, this is regarding this paper: http://www.cs.arizona.edu/people/justin/packagemanagersecurity/ Thanks. Sadly I'm resending this. The Fedora Infrastructure group doesn't have their act together, so my original message is stuck in a moderator queue nobody has the password for. I've subscribed to the list for the purpose of sending this mail. Can someone respond to the above question from Justin. Thanks. -- JB ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
WP MU Help
Hi all, I've come here before talking about getting a news.fp.o site set up and running, and we've had a test instance up in the past with Lyceum but we decided to move in the direction of MU. Bret McMillan has been working very hard on this over the past several months and has now got a test instance up on publictest13. He has set-up a public git repo so that others can get the code he's been working on at: http://fedorapeople.org/~bretm/wordpress-mu.git And says to focus on the fedora branch as bretm-security is effectively dead now that those changes are upstream. There is a short to do list of tasks he still needs to accomplish, and we're looking for anyone who's interested in lending a hand to push this along. The todo list is (directly from Bret): 1) update wp-config.php generation to account for FORCE_SSL_LOGIN FORCE_SSL_ADMIN directives 2) figure out why blog/ url-path is missing for the primary blog when installed w/ VHOST == no, causes: https://bugzilla.redhat.com/show_bug.cgi?id=455801 3) do the formal RPM submission admin stuff 4) integrating w/ fedora account system; I have written a plugin internally for delegating authentication to mod_auth_*... once it clears the RH process for open-sourcing, can just use that 5) munging the puppet configs for publictest13; I'm slowly learning puppet, and have no idea what the norms are for Fedora Infrastructure in this regard. Guidance/help greatly appreciated. Any help would be hugely appreciated, along with any guidance on how to proceed from where we are now to going live with this :) Best, Jon ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: YUM security issues...
On Fri, 25 Jul 2008, Mike McGrath wrote: On Fri, 25 Jul 2008, Josh Bressers wrote: On 21 July 2008, Josh Bressers wrote: On 19 July 2008, Justin Cappos wrote: By the way, did you remove the ability for mirror admins to select a subnet where they'll serve all of the traffic? We're particularly concerned about this issue in the short term. We took our mirror down (mirror1.lockdownhosting.com) quite a while ago so we can't check for ourselves. AFAIK, this service is still in place and working fine. Though I am a little confused about the question. It sounds like you'd like to direct all subnet traffic to a specific mirror. But you're also saying you took your mirror down. Are you worried people in your subnet are being directed to a down mirror? -Mike ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: YUM security issues...
On Fri, 2008-07-25 at 10:37 -0500, Mike McGrath wrote: AFAIK, this service is still in place and working fine. Though I am a little confused about the question. It sounds like you'd like to direct all subnet traffic to a specific mirror. But you're also saying you took your mirror down. Are you worried people in your subnet are being directed to a down mirror? More like taking over a subnet and directing all clients at a rouge mirror. -- Jesse Keating Fedora -- Freedom² is a feature! signature.asc Description: This is a digitally signed message part ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: YUM security issues...
On 25 July 2008, Mike McGrath wrote: On Fri, 25 Jul 2008, Mike McGrath wrote: On Fri, 25 Jul 2008, Josh Bressers wrote: On 21 July 2008, Josh Bressers wrote: On 19 July 2008, Justin Cappos wrote: By the way, did you remove the ability for mirror admins to select a subnet where they'll serve all of the traffic? We're particularly concerned about this issue in the short term. We took our mirror down (mirror1.lockdownhosting.com) quite a while ago so we can't check for ourselves. AFAIK, this service is still in place and working fine. Though I am a little confused about the question. It sounds like you'd like to direct all subnet traffic to a specific mirror. But you're also saying you took your mirror down. Are you worried people in your subnet are being directed to a down mirror? No, the problem is what happens when a malicious mirror claims a subnet? This is currently being viewed as a security issue due to this research: http://www.cs.arizona.edu/people/justin/packagemanagersecurity/attacks-on-package-managers.html -- JB ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: YUM security issues...
On Fri, Jul 25, 2008 at 10:43:59AM -0500, Mike McGrath wrote: On Fri, 25 Jul 2008, Jesse Keating wrote: On Fri, 2008-07-25 at 10:37 -0500, Mike McGrath wrote: AFAIK, this service is still in place and working fine. Though I am a little confused about the question. It sounds like you'd like to direct all subnet traffic to a specific mirror. But you're also saying you took your mirror down. Are you worried people in your subnet are being directed to a down mirror? More like taking over a subnet and directing all clients at a rouge mirror. nod that makes more sense. Domsch? Yes, this is a known challenge with subnet delegation in MirrorManager. We're trusting package signing (and soon, repodata signing) to prevent rogue mirrors from issuing unsigned data. In addition, I'm working on adding in a way to prevent stale mirrors (with signed content) from being used. -- Matt Domsch Linux Technology Strategist, Dell Office of the CTO linux.dell.com www.dell.com/linux ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: YUM security issues...
On Fri, 25 Jul 2008, Matt Domsch wrote: On Fri, Jul 25, 2008 at 10:43:59AM -0500, Mike McGrath wrote: On Fri, 25 Jul 2008, Jesse Keating wrote: On Fri, 2008-07-25 at 10:37 -0500, Mike McGrath wrote: AFAIK, this service is still in place and working fine. Though I am a little confused about the question. It sounds like you'd like to direct all subnet traffic to a specific mirror. But you're also saying you took your mirror down. Are you worried people in your subnet are being directed to a down mirror? More like taking over a subnet and directing all clients at a rouge mirror. nod that makes more sense. Domsch? Yes, this is a known challenge with subnet delegation in MirrorManager. We're trusting package signing (and soon, repodata signing) to prevent rogue mirrors from issuing unsigned data. In addition, I'm working on adding in a way to prevent stale mirrors (with signed content) from being used. Perhaps it might also be a good idea to add a comment to the default yum.conf for gpgcheck explaining what a bad idea it is to set to 0. I could imagine people setting it to 0 not understanding what they're doing. Especially if they're familiar with gpg's encryption bits, but not its signing functionality. -Mike ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: YUM security issues...
On Fri, Jul 25, 2008 at 12:46:15PM -0400, Josh Bressers wrote: On 25 July 2008, Matt Domsch wrote: Yes, this is a known challenge with subnet delegation in MirrorManager. We're trusting package signing (and soon, repodata signing) to prevent rogue mirrors from issuing unsigned data. In addition, I'm working on adding in a way to prevent stale mirrors (with signed content) from being used. How does one get this subnet delegation though? Can I request any subnet I want, or do we do some sort of verification? At present there is no verification (I'm not at all sure how one _could_ verify except by ARIN co delegation). However there are limits as to how large a block can be requested. Nothing larger than a IPv4 /16 can be automatically requested. Fedora Infrastructure admins can add larger blocks, and request ARIN co data when doing so. What happens if the client decided its mirror is bad, I presume it will go off and find a better one, even with delegation? Yes, the mirrorlist returned includes quite a few mirrors, in priority order. -- Matt Domsch Linux Technology Strategist, Dell Office of the CTO linux.dell.com www.dell.com/linux ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: YUM security issues...
On 25 July 2008, Matt Domsch wrote: On Fri, Jul 25, 2008 at 12:46:15PM -0400, Josh Bressers wrote: On 25 July 2008, Matt Domsch wrote: Yes, this is a known challenge with subnet delegation in MirrorManager. We're trusting package signing (and soon, repodata signing) to prevent rogue mirrors from issuing unsigned data. In addition, I'm working on adding in a way to prevent stale mirrors (with signed content) from being used. How does one get this subnet delegation though? Can I request any subnet I want, or do we do some sort of verification? At present there is no verification (I'm not at all sure how one _could_ verify except by ARIN co delegation). However there are limits as to how large a block can be requested. Nothing larger than a IPv4 /16 can be automatically requested. Fedora Infrastructure admins can add larger blocks, and request ARIN co data when doing so. That's a lot of IPs though. Can I request multiple /16s, or only one? How many mirrors are doing this? Does the mirror have to be part of the /16 to request it? Thanks for the patience here. I'm trying to understand the risk we're dealing with. Thanks. -- JB ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Public demo of amber and eventual production instance
Hi, So sometime next week I'd like to link to the publictest10.fedoraproject.org/amber site and ask for feedback. In the meantime I'm going to install the latest changes on it, and load it up with all the data from F9. Just a heads-up, and a humble request not to break things (like the FAS instance I'm using) too much next week. If anyone has specific plans in that area, ping me and we can work things out. Second, Fedora Applications/Amber is eventually going to need an actual production server ready around the Fedora 10 release date. I'd like to get to the point where I can make a formal request for aforementioned resources. What do I need to do to get there? Thanks, -RN -- Robin Norwood Red Hat, Inc. The Sage does nothing, yet nothing remains undone. -Lao Tzu, Te Tao Ching ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: YUM security issues...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Matt Domsch wrote: On Fri, Jul 25, 2008 at 12:46:15PM -0400, Josh Bressers wrote: On 25 July 2008, Matt Domsch wrote: Yes, this is a known challenge with subnet delegation in MirrorManager. We're trusting package signing (and soon, repodata signing) to prevent rogue mirrors from issuing unsigned data. In addition, I'm working on adding in a way to prevent stale mirrors (with signed content) from being used. How does one get this subnet delegation though? Can I request any subnet I want, or do we do some sort of verification? At present there is no verification (I'm not at all sure how one _could_ verify except by ARIN co delegation). However there are limits as to how large a block can be requested. Nothing larger than a IPv4 /16 can be automatically requested. Fedora Infrastructure admins can add larger blocks, and request ARIN co data when doing so. What happens if the client decided its mirror is bad, I presume it will go off and find a better one, even with delegation? Yes, the mirrorlist returned includes quite a few mirrors, in priority order. Our testing showed that when our client was in a MirrorManager-defined CIDR block for a mirror, the returned mirrorlist included only the single mirror. -- It's dangerous either way, of course, but I'm just wondering if our testing was faulty, if this has changed since we tested, or if it might be behaving differently than you expect. Possibly you tested with a block that was already defined by other mirrors and so multiple entries were returned in the mirrorist? That's just a guess, we didn't test with a block that was defined by more than one mirror (as far as we knew, at least). - -- Justin Samuel https://www.cs.arizona.edu/~jsamuel/ gpg: 0xDDF1F3EE [66EF 84E2 F184 B140 712B 55A7 2B96 AB8F DDF1 F3EE] -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIih0cK5arj93x8+4RAklWAKC+Lewfd+pixUvL2MvbdCYxnjHBpQCdHtNd x5BQsM6GqW5zKpJt+RH8Vco= =w9yV -END PGP SIGNATURE- ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
RE: YUM security issues...
Fedora 7 definitely behaves differently than Fedora 8 and 9. The behavior I describe began with F8. For F7 and earlier, the yum policy would chose any random mirror from the returned list, so having many mirrors on the list, some of which are unreachable from inside an organization, would be bad. The yum default policy was changed to treat the mirrorlist as a priority list, so MM returns a longer list. -- Matt Domsch Linux Technology Strategist, Dell Office of the CTO linux.dell.com www.dell.com/linux -Original Message- From: Justin Samuel [mailto:[EMAIL PROTECTED] On Behalf Of Justin Samuel Sent: Friday, July 25, 2008 1:36 PM To: Domsch, Matt Cc: Josh Bressers; Mike McGrath; fedora-infrastructure-list@redhat.com; Justin Cappos; [EMAIL PROTECTED] Subject: Re: YUM security issues... -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Matt Domsch wrote: On Fri, Jul 25, 2008 at 12:46:15PM -0400, Josh Bressers wrote: On 25 July 2008, Matt Domsch wrote: Yes, this is a known challenge with subnet delegation in MirrorManager. We're trusting package signing (and soon, repodata signing) to prevent rogue mirrors from issuing unsigned data. In addition, I'm working on adding in a way to prevent stale mirrors (with signed content) from being used. How does one get this subnet delegation though? Can I request any subnet I want, or do we do some sort of verification? At present there is no verification (I'm not at all sure how one _could_ verify except by ARIN co delegation). However there are limits as to how large a block can be requested. Nothing larger than a IPv4 /16 can be automatically requested. Fedora Infrastructure admins can add larger blocks, and request ARIN co data when doing so. What happens if the client decided its mirror is bad, I presume it will go off and find a better one, even with delegation? Yes, the mirrorlist returned includes quite a few mirrors, in priority order. Our testing showed that when our client was in a MirrorManager-defined CIDR block for a mirror, the returned mirrorlist included only the single mirror. -- It's dangerous either way, of course, but I'm just wondering if our testing was faulty, if this has changed since we tested, or if it might be behaving differently than you expect. Possibly you tested with a block that was already defined by other mirrors and so multiple entries were returned in the mirrorist? That's just a guess, we didn't test with a block that was defined by more than one mirror (as far as we knew, at least). - -- Justin Samuel https://www.cs.arizona.edu/~jsamuel/ gpg: 0xDDF1F3EE [66EF 84E2 F184 B140 712B 55A7 2B96 AB8F DDF1 F3EE] -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIih0cK5arj93x8+4RAklWAKC+Lewfd+pixUvL2MvbdCYxnjHBpQCdHtNd x5BQsM6GqW5zKpJt+RH8Vco= =w9yV -END PGP SIGNATURE- ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: YUM security issues...
On 25 July 2008, Matt Domsch wrote: On Fri, Jul 25, 2008 at 01:52:26PM -0400, Josh Bressers wrote: That's a lot of IPs though. Can I request multiple /16s, or only one? As many as you like. And recall, such changes are made using your FAS credentials. Are these ever checked? Does say a mail get generated every time someone adds one of these? My fear would be that someone could blanket quite a large IP space without anyone noticing. Granted that would no doubt generate a huge volume of traffic, but if they're serving up a frozen repo, they probably won't be pushing all that much data. How many mirrors are doing this? 374 total Hosts 185 have at least 1 netblock entry 94 of these are private - don't serve the public wow, that's quite a few. I wasn't expecting numbers this high honestly. Does the mirror have to be part of the /16 to request it? no. Take for example Dell's mirrors. Netblock 143.166/16 is Dell US, but the mirror IPs are located inside the 10/8 private space. OK, so here is the problem the way I see it, signing the repository won't fix it. I'll try to explain this clearly, Justin can yell at me if I've gotten any of this wrong. So let's say Mallory (the bad guy) decides that he wants to host a malicious mirror and wait for a nasty security flaw. He sets up his mirror and even claims some IP subnets to serve. Bob and Alice are happily installing valid updates from him for some period of time. Since Mallory has claimed to serve a specific subnet, he has a rather impressive view of what Bob and Alice have installed. Now let's say there is a horrible security bug found in a mail server. Mallory knows for a fact that Bob and Alice both have it installed as he's been their mirror for a while. Mallory stops updating his mirror, so none of the users being served will get the mail server updates. Mallory also knows the IP address of the vulnerable clients and can easily break into their systems. So from what I understand MirrorManager will check on the mirrors to ensure they're not out of date. Mallory knows this and makes sure that when MirrorManager connects to his mirror, it lies and serves up current metadata. So here is the problem. The repodata was valid. The packages are signed. Even if we sign the repodata, this attack works. Being able to acquire an IP block simply makes this attack easier to do. It's still very possible that a bad mirror will wait for users to connect, serve up old content then use this knowledge to break into their system. What this problem boils down to, is we need a way for clients to ask MirrorManager what the current valid repo data is. Ideally we want the results to be signed in some manner so it can't be spoofed. Some thoughts I've had are: 1) Have MirrorManager use https and return some repo verification data. 2) Sign the repo data, and if it's older than X, don't use it (I don't like this solution, but it's probably the easiest, just push out a new signed repo file once a day, even if nothing changes.) 3) Always get repo data from fedoraproject.org (probably not practical due to resource issues) 4) use DNS, have the client query repodata sha1sum.repo.fedoraproject.org if the lookup fails, the repo is invalid. (this is really cheap from a resource standpoint, but hard to do technically) 5) ??? Thanks. -- JB ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: YUM security issues...
Josh Bressers wrote: On 25 July 2008, Matt Domsch wrote: On Fri, Jul 25, 2008 at 01:52:26PM -0400, Josh Bressers wrote: That's a lot of IPs though. Can I request multiple /16s, or only one? As many as you like. And recall, such changes are made using your FAS credentials. Are these ever checked? Does say a mail get generated every time someone adds one of these? My fear would be that someone could blanket quite a large IP space without anyone noticing. Granted that would no doubt generate a huge volume of traffic, but if they're serving up a frozen repo, they probably won't be pushing all that much data. How many mirrors are doing this? 374 total Hosts 185 have at least 1 netblock entry 94 of these are private - don't serve the public wow, that's quite a few. I wasn't expecting numbers this high honestly. Does the mirror have to be part of the /16 to request it? no. Take for example Dell's mirrors. Netblock 143.166/16 is Dell US, but the mirror IPs are located inside the 10/8 private space. OK, so here is the problem the way I see it, signing the repository won't fix it. I'll try to explain this clearly, Justin can yell at me if I've gotten any of this wrong. So let's say Mallory (the bad guy) decides that he wants to host a malicious mirror and wait for a nasty security flaw. He sets up his mirror and even claims some IP subnets to serve. Bob and Alice are happily installing valid updates from him for some period of time. Since Mallory has claimed to serve a specific subnet, he has a rather impressive view of what Bob and Alice have installed. Now let's say there is a horrible security bug found in a mail server. Mallory knows for a fact that Bob and Alice both have it installed as he's been their mirror for a while. Mallory stops updating his mirror, so none of the users being served will get the mail server updates. Mallory also knows the IP address of the vulnerable clients and can easily break into their systems. So from what I understand MirrorManager will check on the mirrors to ensure they're not out of date. Mallory knows this and makes sure that when MirrorManager connects to his mirror, it lies and serves up current metadata. So here is the problem. The repodata was valid. The packages are signed. Even if we sign the repodata, this attack works. Being able to acquire an IP block simply makes this attack easier to do. It's still very possible that a bad mirror will wait for users to connect, serve up old content then use this knowledge to break into their system. What this problem boils down to, is we need a way for clients to ask MirrorManager what the current valid repo data is. Ideally we want the results to be signed in some manner so it can't be spoofed. Some thoughts I've had are: 1) Have MirrorManager use https and return some repo verification data. We'd need to push repo verification data for both the latest and previous repo information. MirrorManager would have to be updated with new data as part of the updates/rawhide push so that it's always up to date with the mirror. We'll have to revise mirrormanager's caching model so that it always has access to the latest repository verification information. 2) Sign the repo data, and if it's older than X, don't use it (I don't like this solution, but it's probably the easiest, just push out a new signed repo file once a day, even if nothing changes.) This is going to have to have some policy attached to it. Do we continue to sign the repo data for EOL releases because people use them even though we tell people they're EOL? 3) Always get repo data from fedoraproject.org (probably not practical due to resource issues) This is the easiest to implement. It means the small repomd.xml file always comes from our server. But the rest of the metadata can come from the individual mirrors. However, it does mean that *very* large swaths of the mirrors will be unable to serve packages to users at any time because their metadata won't match with ours for some period after we have an update pushed. Maybe we could do this with versioned repodata and the client can decide how far back in time or how many past revisions it is willing to accept. 4) use DNS, have the client query repodata sha1sum.repo.fedoraproject.org if the lookup fails, the repo is invalid. (this is really cheap from a resource standpoint, but hard to do technically) I don't know enough about implementing this one to comment. Also, all of these solutions need client-side support. That being the case, it should be generic enough that other repomd consuming clients and distributions will be willing to use it. Otherwise we'll be securing our updates and mirror infrastructure with the default package manager but doing nothing for the ecosystem as a whole. -Toshio signature.asc Description: OpenPGP digital signature ___
Re: YUM security issues...
On Fri, 2008-07-25 at 18:41 -0700, Toshio Kuratomi wrote: 3) Always get repo data from fedoraproject.org (probably not practical due to resource issues) This is the easiest to implement. It means the small repomd.xml file always comes from our server. But the rest of the metadata can come from the individual mirrors. However, it does mean that *very* large swaths of the mirrors will be unable to serve packages to users at any time because their metadata won't match with ours for some period after we have an update pushed. Maybe we could do this with versioned repodata and the client can decide how far back in time or how many past revisions it is willing to accept. We don't need to version the metadata, we have timestamps in them already. We can easily do a comparison from this timestamp to now. And we can set this number to be different from the base repo as to the updates repo. But as you've already mentioned we're stuck with the question of EOL'd releases and how to deal with things deeply out of date. I can make yum throw out warnings and alerts but at what point does it actually STOP doing anything and does that not open us up to a different kind of DoS? I don't know enough about implementing this one to comment. Also, all of these solutions need client-side support. That being the case, it should be generic enough that other repomd consuming clients and distributions will be willing to use it. Otherwise we'll be securing our updates and mirror infrastructure with the default package manager but doing nothing for the ecosystem as a whole. We need to make sure that whatever we implement is trivially done so by people running a local downstream branch of fedora or centos or what-have-you. Or, as you say, we've saved ourselves and screwed everyone else. -sv ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
