Re: Possibly moving Debian services to a CDN
Hi James, On 08/02/14 at 13:48 +0800, James Bromberger wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 8/02/2014 11:46 AM, Luca Filipozzi wrote: Have you been trying to reach out to other CDN providers about supporting Debian? I know of discussions with Amazon CloudFront, but I remember some technical blockers? Could the DPL be of some help to you in that process? I am in active discussion with another CDN provider and I should restart the CloudFront conversation. There are technical considerations with Fastly, also, that Tollef will work through. We've always been of the opinion that we need two CDN providers. We're just as concerned about vendor lock-in as anyone. Hello Luca, all, http://cloudfront.debian.net/ is continuing to do some traffic. It is also offering CDN acceleration for debian-cd and cdimage. We set up a second CDN distribution, http://cloudfront-security.debian.net/; however at this point in time CloudFront does not support IPv6. CloudFront now has 51 edge locations worldwide (having added Rio de Janerio, Taipei, Manila, Marseille and Warsaw recently) and supports custom SSL certificates if (Debian) want to do this. Wasn't there an issue (that might not be CloudFront-specific) about the refresh times of files in dists/? I remember reading about something like that. As to lock in - CloudFront is an http cache network - stop handing out the cloudfront.debian.net URL and traffic naturally drops off; nothing else do to. Sure, the hypothetical scenario which I'm worried about is: - there's only one CDN provider willing to support Debian, and able to meet our technical requirements - we switch everything to that CDN, and shut down our mirrors network - the CDN provider decides that, after all, it doesn't want to support Debian anymore Clearly, that's a long way from the current status, but that's something that we should keep in mind, and I was just slightly worried that it was not mentioned in Tollef's status update. Lucas -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20140208084147.gc30...@xanadu.blop.info
Re: Possibly moving Debian services to a CDN
On Sat, Feb 08, 2014 at 07:53:56PM +0800, James Bromberger wrote: On 8/02/2014 4:41 PM, Lucas Nussbaum wrote: [..] Sure, the hypothetical scenario which I'm worried about is: - there's only one CDN provider willing to support Debian, and able to meet our technical requirements - we switch everything to that CDN, and shut down our mirrors network - the CDN provider decides that, after all, it doesn't want to support Debian anymore I don't think Debian should shut down the mirror network; at least on a national level. For example, right now I am configuring Debian AMIs within China, and the only mirror I can access from there is ftp.cn.debian.org. I don't want the current mirror network be dropped in favor of a CDN, for the same good reason of being independent of a too little group of CDN providers willing/able to carry Debian. Clearly, that's a long way from the current status, but that's something that we should keep in mind, and I was just slightly worried that it was not mentioned in Tollef's status update. I think maintaining mirrors *and* balancing across mirrors and CDNs is a possible way forward with something like http.debian.net. I'm more than happy to help scale-out http.debian.net to multiple PoPs if we want to address that as well... That would be great, thanks James for your help, we can discuss this privately on mirr...@debian.org. -- Simon Paillard -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20140208123115.gc...@mraw.org
Re: Possibly moving Debian services to a CDN
On Sat, 08 Feb 2014, Simon Paillard wrote: I don't think Debian should shut down the mirror network; at least on a national level. For example, right now I am configuring Debian AMIs within China, and the only mirror I can access from there is ftp.cn.debian.org. I don't want the current mirror network be dropped in favor of a CDN, for the same good reason of being independent of a too little group of CDN providers willing/able to carry Debian. The goal should be that we provide users with the best means to get packages quickly: low latency for requests, high bandwidth for transfers, and soon after a dinstall run. Users shouldn't have to pick their mirror manually, they shouldn't have to update their configuration if anything breaks - nothing they would pick should ever visibly break for end-users. A user should be able to tell their system give me debian [, I don't care where it comes from]. In the end it matters little how we achieve these goals, but we should work towards them. -- | .''`. ** Debian ** Peter Palfrader | : :' : The universal http://www.palfrader.org/ | `. `' Operating System | `-http://www.debian.org/ -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20140208131607.gb25...@anguilla.noreply.org
Re: Possibly moving Debian services to a CDN
On Sat, Feb 08, 2014 at 02:16:07PM +0100, Peter Palfrader wrote: On Sat, 08 Feb 2014, Simon Paillard wrote: I don't think Debian should shut down the mirror network; at least on a national level. For example, right now I am configuring Debian AMIs within China, and the only mirror I can access from there is ftp.cn.debian.org. I don't want the current mirror network be dropped in favor of a CDN, for the same good reason of being independent of a too little group of CDN providers willing/able to carry Debian. [..] A user should be able to tell their system give me debian [, I don't care where it comes from]. We agree. The mirror network would still exist, as the basis for http.d.n (or an alternate solution if any). If having very low latency is too expensive according to DSA (too many PoPs), then in my opinion we can sacrifice a bit the latency to keep independence. In the end it matters little how we achieve these goals, but we should work towards them. We disagree on this, but in my opinion, we already achieve this with http.d.n (except it's not DSA-sponsored and as consequence not official). -- Simon Paillard -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20140208140103.gd...@mraw.org
Re: Possibly moving Debian services to a CDN
On Sat, 08 Feb 2014, Simon Paillard wrote: In the end it matters little how we achieve these goals, but we should work towards them. We disagree on this, but in my opinion, we already achieve this with http.d.n (except it's not DSA-sponsored and as consequence not official). http.d.n is a nice idea, but I think the redirects are expensive latency wise. Even worse, it fails the 'must not visibly break' requirement. We regularly see broken apt-get runs with http.d.n in our sources.lists. -- | .''`. ** Debian ** Peter Palfrader | : :' : The universal http://www.palfrader.org/ | `. `' Operating System | `-http://www.debian.org/ -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20140208143335.gc25...@anguilla.noreply.org
Re: Possibly moving Debian services to a CDN
]] Lucas Nussbaum Thanks a lot for this status update. I'm very much in favor of exploring ways to make the Debian infrastructure easier to manage, and using a CDN sounds like a great way to do so. It's great that things worked out with Fastly (any plans for a more public announcement?). I'd like to move some other bits over and get some of the technical hurdles worked out first. Amongst those are the problems of ensuring that downstream caches see a consistent view, whether we're talking about sites such as planet (where the plan is to pull in images to be served from the planet.d.o domain rather than from the blog posters domain) or backports. The considerations that apply to backports would also apply if/when we want to push the main archive through. In short, the problem is as follows: We have a static-master with the master copy of the web site. This tree gets synced to a bunch of mirrors (currently three). In some cases, a mirror might be unreachable, be down or otherwise not be updateable. If it's not updated, we want to ensure that mirror is not used until it's up-to-date. How we solve this problem is going to differ by CDN, but common to all of them is we need tight control over timing to ensure users never encounter out-of-sync mirrors. I have some ideas how to do this for Varnish-based CDNs, but I'm not sure how to solve it for some of the other CDNs, so we'll need to talk to them. However, in [1], I raised one main non-technical concern that is not mentioned in your mail: I fear that, by moving to CDNs without ensuring that there are a sufficient number of CDN providers willing and able to support Debian, we could end up in a lock-in situation with a specific CDN provider (after all, there are not so many of them, and even a smaller number could be able to deal with our technical requirements). [1] https://lists.debian.org/debian-project/2013/10/msg00074.html You're just mirroring what we talked about in https://lists.debian.org/debian-project/2013/10/msg00029.html there. We're so far just reducing latency for users, those sites were served purely by the static mirror hosts up until recently and we had no load problems there, so if we want to, we could easily pull back to using our own infrastructure again. Of course, as long as we have the infrastructure to go back to the old way of doing things, it is not a big problem. So I'm not worried at the moment. But one of the end goals of using CDN is to reduce the number of Debian PoP (have Debian machines in a fewer number of datacenters, to make them easier to manage). Once we do that, it will be very hard to go back. We're not going to reduce the number of POPs significantly by using a CDN, and it's not the goal either. https://lists.debian.org/debian-project/2013/06/msg00164.html talks more about the initial motivations for using a CDN. Have you been trying to reach out to other CDN providers about supporting Debian? I know of discussions with Amazon CloudFront, but I remember some technical blockers? I'd like to get the services working well with one CDN before I start engaging others. Could the DPL be of some help to you in that process? I talked with James Bromberger at LCA, so contact is already established there. We're also talking with at least one other CDN, so I don't think we need any help in that area right now. We'll let you know. -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/m2fvnt2go1@rahvafeir.err.no
Re: Possibly moving Debian services to a CDN
On 30/01/14 at 13:53 +0100, Tollef Fog Heen wrote: ]] Tollef Fog Heen Hi all, - the various bits and bobs that are currently hosted on static.debian.org I thought it's time for a small update about this. As of about an hour ago, planet and metadata.ftp-master are now served from the Fastly CDN, and it all seems to be working quite smoothly. We've uncovered some bits we want to make work better, such as adding and removing backend servers automatically when they become unavailable or are added to the static DNS RR, purging content from the caches when it's updated and possibly some other minor bits. This does sadly mean we don't currently have IPv6 for those two services, something that's being worked on by Fastly. As for the privacy concerns raised in the thread, I've had quite a lot of discussions with Fastly about how they operate wrt privacy. They don't store request-related logs (only billing information), so there are no URLs, cookie, client IPs or similar being stored. Varnish has an ephemeral log which they go through a couple of times a minute where some of that information is present, but it never leaves the host (unless we enable logging to an endpoint we control). I'm quite content with how they're handling the privacy concerns. In the interest of full disclosure I should also mention that I'm starting to work for Fastly in a few days time. I don't believe that has influenced my views or judgements here. Hi Tollef, Thanks a lot for this status update. I'm very much in favor of exploring ways to make the Debian infrastructure easier to manage, and using a CDN sounds like a great way to do so. It's great that things worked out with Fastly (any plans for a more public announcement?). However, in [1], I raised one main non-technical concern that is not mentioned in your mail: I fear that, by moving to CDNs without ensuring that there are a sufficient number of CDN providers willing and able to support Debian, we could end up in a lock-in situation with a specific CDN provider (after all, there are not so many of them, and even a smaller number could be able to deal with our technical requirements). [1] https://lists.debian.org/debian-project/2013/10/msg00074.html Of course, as long as we have the infrastructure to go back to the old way of doing things, it is not a big problem. So I'm not worried at the moment. But one of the end goals of using CDN is to reduce the number of Debian PoP (have Debian machines in a fewer number of datacenters, to make them easier to manage). Once we do that, it will be very hard to go back. Have you been trying to reach out to other CDN providers about supporting Debian? I know of discussions with Amazon CloudFront, but I remember some technical blockers? Could the DPL be of some help to you in that process? Cheers. Lucas signature.asc Description: Digital signature
Re: Possibly moving Debian services to a CDN
On Fri, Feb 07, 2014 at 02:08:26PM +0100, Lucas Nussbaum wrote: On 30/01/14 at 13:53 +0100, Tollef Fog Heen wrote: ]] Tollef Fog Heen Hi all, - the various bits and bobs that are currently hosted on static.debian.org I thought it's time for a small update about this. As of about an hour ago, planet and metadata.ftp-master are now served from the Fastly CDN, and it all seems to be working quite smoothly. We've uncovered some bits we want to make work better, such as adding and removing backend servers automatically when they become unavailable or are added to the static DNS RR, purging content from the caches when it's updated and possibly some other minor bits. This does sadly mean we don't currently have IPv6 for those two services, something that's being worked on by Fastly. As for the privacy concerns raised in the thread, I've had quite a lot of discussions with Fastly about how they operate wrt privacy. They don't store request-related logs (only billing information), so there are no URLs, cookie, client IPs or similar being stored. Varnish has an ephemeral log which they go through a couple of times a minute where some of that information is present, but it never leaves the host (unless we enable logging to an endpoint we control). I'm quite content with how they're handling the privacy concerns. In the interest of full disclosure I should also mention that I'm starting to work for Fastly in a few days time. I don't believe that has influenced my views or judgements here. Hi Tollef, Thanks a lot for this status update. I'm very much in favor of exploring ways to make the Debian infrastructure easier to manage, and using a CDN sounds like a great way to do so. It's great that things worked out with Fastly (any plans for a more public announcement?). However, in [1], I raised one main non-technical concern that is not mentioned in your mail: I fear that, by moving to CDNs without ensuring that there are a sufficient number of CDN providers willing and able to support Debian, we could end up in a lock-in situation with a specific CDN provider (after all, there are not so many of them, and even a smaller number could be able to deal with our technical requirements). [1] https://lists.debian.org/debian-project/2013/10/msg00074.html Of course, as long as we have the infrastructure to go back to the old way of doing things, it is not a big problem. So I'm not worried at the moment. But one of the end goals of using CDN is to reduce the number of Debian PoP (have Debian machines in a fewer number of datacenters, to make them easier to manage). Once we do that, it will be very hard to go back. Have you been trying to reach out to other CDN providers about supporting Debian? I know of discussions with Amazon CloudFront, but I remember some technical blockers? Could the DPL be of some help to you in that process? I am in active discussion with another CDN provider and I should restart the CloudFront conversation. There are technical considerations with Fastly, also, that Tollef will work through. We've always been of the opinion that we need two CDN providers. We're just as concerned about vendor lock-in as anyone. Thank you for the offer of DPL help. I'll loop you in. Luca -- Luca Filipozzi http://www.crowdrise.com/SupportDebian signature.asc Description: Digital signature
Re: Possibly moving Debian services to a CDN
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 8/02/2014 11:46 AM, Luca Filipozzi wrote: Have you been trying to reach out to other CDN providers about supporting Debian? I know of discussions with Amazon CloudFront, but I remember some technical blockers? Could the DPL be of some help to you in that process? I am in active discussion with another CDN provider and I should restart the CloudFront conversation. There are technical considerations with Fastly, also, that Tollef will work through. We've always been of the opinion that we need two CDN providers. We're just as concerned about vendor lock-in as anyone. Hello Luca, all, http://cloudfront.debian.net/ is continuing to do some traffic. It is also offering CDN acceleration for debian-cd and cdimage. We set up a second CDN distribution, http://cloudfront-security.debian.net/; however at this point in time CloudFront does not support IPv6. CloudFront now has 51 edge locations worldwide (having added Rio de Janerio, Taipei, Manila, Marseille and Warsaw recently) and supports custom SSL certificates if (Debian) want to do this. Using the Debian http redirection service at http://http.debian.net/ is perhaps a preferred approach as it allows real time redirection to the desired CDN service. I'm happy to give any DD access to the AWS account that is running this; please mail me (gpg signed) off list. As to lock in - CloudFront is an http cache network - stop handing out the cloudfront.debian.net URL and traffic naturally drops off; nothing else do to. James - -- /Mobile:/ +61 422 166 708, /Email:/ james_AT_rcpt.to -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (MingW32) iQIcBAEBAgAGBQJS9cVEAAoJEK7IKHSdhcU8G7IQAI32rECjmo8/lNeLP5Z4jn+T KqqmEqoyKAX1cDFP0PFHZZPocWeTZM9p5B5u+hnGcf4gj8umjmFM7XzP3EwlSYM1 QOiOlRfGuNsp871Rnhwdw3pslEiOX2vdg7V4Mjcbo+NX0MNY+Vu2XqDoie5SSgE0 eSR/0quV9o9a22A6CkO5lmmtgOnWSSYsFUFOutdPseQmTKYn5J9Qc4wLOIO4Z0CI X+ANOlK6RMeg4CwiBNezxa4xkTQcg6TvqOt9r0/rxWNCYsWPnCyFel4Iykctqt99 yjrkMga8dDWgKqegB7awINu701z9SVbVqpWkiM+E3+5ZpDDshbwf78uR6BiliXP6 ZY860SSXhX12EWrlEIcamZXZMbh0xZrWITuFksoXMnQZK45S6TU5JYPhVPnWoLou kM99eBcJhNyiqZawH1nTulsP4DM9vYRDt+kb9g3L8HSBhkYmCN0Gz5iPzEpXZ/Ef b4VhKOegPfiT1cjHLqltNxZvJ8gN1L4X4LJ2bVu56qDCgqj83r5vtRVfI5RI53TB U8itPGsZHgxXtOG8Hqyji8YjUT7i60ZAok9kMKgZ7lBdNt1Cv0SYN8yRl2pf1Ru5 pqyIjxUsBSNFFPkSAh01cXrHvhK+EonbxGFcrW2xSdXXLr5KgkM9kN/vjtzgLPfS t1zIIRKge1YN+5z4n35T =2A80 -END PGP SIGNATURE-
Re: Possibly moving Debian services to a CDN
]] Florian Weimer * Tollef Fog Heen: There's not really anything to be fixed, since you shouldn't be using HTTPS for that host yet. Can't they serve it off an IP address that doesn't answer on 443/TCP, to avoid confusion? It would be technically possible to do so, but the network's not set up that way and I see little point in spending effort on changing that. (If we do change it, we should change it to be served using HTTPS and with a local copy of resources, else you'll just end up with warnings about mixed-content pages.) -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/878utrulwg@xoog.err.no
Re: Possibly moving Debian services to a CDN
* Tollef Fog Heen: There's not really anything to be fixed, since you shouldn't be using HTTPS for that host yet. Can't they serve it off an IP address that doesn't answer on 443/TCP, to avoid confusion? -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/8738jzuchj@mid.deneb.enyo.de
Re: Possibly moving Debian services to a CDN
Craig Small schrieb: Interesting way they've stapled all the names together on the certificate too. I didn't know you could do that, See http://en.wikipedia.org/wiki/SubjectAltName. Regards, -thh -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/ldp.1402011352@landroval.ancalagon.de
Re: Possibly moving Debian services to a CDN
]] Tollef Fog Heen Hi all, - the various bits and bobs that are currently hosted on static.debian.org I thought it's time for a small update about this. As of about an hour ago, planet and metadata.ftp-master are now served from the Fastly CDN, and it all seems to be working quite smoothly. We've uncovered some bits we want to make work better, such as adding and removing backend servers automatically when they become unavailable or are added to the static DNS RR, purging content from the caches when it's updated and possibly some other minor bits. This does sadly mean we don't currently have IPv6 for those two services, something that's being worked on by Fastly. As for the privacy concerns raised in the thread, I've had quite a lot of discussions with Fastly about how they operate wrt privacy. They don't store request-related logs (only billing information), so there are no URLs, cookie, client IPs or similar being stored. Varnish has an ephemeral log which they go through a couple of times a minute where some of that information is present, but it never leaves the host (unless we enable logging to an endpoint we control). I'm quite content with how they're handling the privacy concerns. In the interest of full disclosure I should also mention that I'm starting to work for Fastly in a few days time. I don't believe that has influenced my views or judgements here. Cheers, -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/877g9h1vh8@qurzaw.varnish-software.com
Re: Possibly moving Debian services to a CDN
Tollef Fog Heen writes (Re: Possibly moving Debian services to a CDN): As for the privacy concerns raised in the thread, I've had quite a lot of discussions with Fastly about how they operate wrt privacy. They don't store request-related logs (only billing information), so there are no URLs, cookie, client IPs or similar being stored. Varnish has an ephemeral log which they go through a couple of times a minute where some of that information is present, but it never leaves the host (unless we enable logging to an endpoint we control). I'm quite content with how they're handling the privacy concerns. Thanks for looking into that. I know that not everyone shares these worries but I do and I really appreciate you taking them serious. Your comments are reassuring and, speaking personally, I'm satisfied. In the interest of full disclosure I should also mention that I'm starting to work for Fastly in a few days time. I don't believe that has influenced my views or judgements here. You are of course right to mention this. Personally I find it increases rather than reduces my confidence in the assurances earlier in your message, and in the situation in general. Thank you. Regards, Ian. -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/21226.28765.549741.548...@chiark.greenend.org.uk
Re: Possibly moving Debian services to a CDN
]] Craig Small On Thu, Jan 30, 2014 at 01:53:55PM +0100, Tollef Fog Heen wrote: I thought it's time for a small update about this. As of about an hour ago, planet and metadata.ftp-master are now served from the Fastly CDN, and it all seems to be working quite smoothly. https://planet.debian.org/ gives a big scary certificate warning. Well, yes. We didn't have HTTPS before and we don't have it now either. One reason to not have it is that you'd have a ton of «insecure third party resource» warnings. Ganneff said he'd take a look at having Planet download those resources locally when he has some free time. Until then, planet is HTTP-only as before. (We want to do this anyway to avoid leaking information about people who read planet to those running the hosting for various blogs.) Interesting way they've stapled all the names together on the certificate too. I didn't know you could do that, but, you might like to tell them to fix that certificate. There's not really anything to be fixed, since you shouldn't be using HTTPS for that host yet. -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/878utw8zks@xoog.err.no
Re: Possibly moving Debian services to a CDN
]] anarcat I would also point out that none of the CDNs *publish* the software they develop as free software. They may *use* free software, but they built their business on proprietary software on top of our hard work. Maybe not all of their software, but a blanket statement like that is simply wrong, take a look at https://github.com/fastly for some of what fastly's done at least. I'm not sure if Amazon has done something similar, but it wouldn't surprise me if they're pushing fixes for components upstream. Why wouldn't they? -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87zjoxbg0p@xoog.err.no
Re: Possibly moving Debian services to a CDN
On 2013-11-20 16:43:35, Tollef Fog Heen wrote: ]] anarcat On 2013-11-14 10:37:21, Tollef Fog Heen wrote: Yes. If you're just anycasting an IP, you'll get pretty poor performance. Can you expand on that? BGP anycast will just get you the closest one in term of metrics. This is probably the fewest number of cheapest hops. There's no guarantee those hops are going to be the shortest or fastest. It doesn't mean you will necessarily get poor performance, you may, but then it's part of the idea of running BGP: you choose the paths.. But I see your point: it's not simply plugging in an AS number and announcing the netblock if you're not already running that AS. :) I guess what I am saying is that doing incremental improvements over the mirror infrastructure should be considered. I am worried that migrating to a commercial CDN will be detrimental to the current infrastructure, which are based on a spirit of free access and open knowledge, something commercial CDNs seem to be alien to... We've waited for somebody to step up and actually do that. Nobody is doing it. Lots of proof-of-concept services out there, but nothing that's solid, tested and production ready. How long would you like us to keep the current state before we actually do something about it? Right, I hear you. I guess I am worried about forcing CDNs down the users' throats. I'd like us to keep a decentralised infrastructure that doesn't depend on those CDNs and allow people to run their own mirrors. Hopefully that will still be possible. A. -- Ils versent un pauvre miel sur leurs mots pourris et te parlent de pénurie Et sur ta faim, sur tes amis, ils aiguisent leur appétit - Richard Desjardins, La maison est ouverte pgpFSf4CzCO7v.pgp Description: PGP signature
Re: Possibly moving Debian services to a CDN
]] anarcat On 2013-11-20 16:43:35, Tollef Fog Heen wrote: ]] anarcat On 2013-11-14 10:37:21, Tollef Fog Heen wrote: Yes. If you're just anycasting an IP, you'll get pretty poor performance. Can you expand on that? BGP anycast will just get you the closest one in term of metrics. This is probably the fewest number of cheapest hops. There's no guarantee those hops are going to be the shortest or fastest. It doesn't mean you will necessarily get poor performance, you may, but then it's part of the idea of running BGP: you choose the paths.. Not really. You choose some. There are other players that will manipulate paths to save themselves money or bandwidth too, so it's all a bit like a real-time strategy game. I guess I am worried about forcing CDNs down the users' throats. I'd like us to keep a decentralised infrastructure that doesn't depend on those CDNs and allow people to run their own mirrors. Hopefully that will still be possible. Nobody has talked about taking away rsync which you can use to run your own mirror. debmirror and others also support mirroring over HTTP so you'll always be able to able to mirror if you want to. Whether that's possible has little or no bearing on if ftp.$cc.d.o is the default apt download location or if it's some other CDN-ed name. (Which isn't what we're talking about initially, but a point we might eventually get to, some years down the line.) -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87y54hbg0l@xoog.err.no
Re: Possibly moving Debian services to a CDN
On 2013-11-20 14:49:34, Thijs Kinkhorst wrote: On Wed, November 20, 2013 19:03, anarcat wrote I see your point, although I think there is a few orders of magnitudes between adding improvements to the current mirror infrastructures versus soldering PCBs... Debian's core activity is building the best OS in the world. Distributing the actual result of that is, at least in my opinion, a thing very peripheral to that activity. The fact that Debian has always shied away from making CD's and distrbuting those to users but rather left that to commercial and non-commercial parties, indicates that this is not a novel development. We allow (and promoted) commercial CD services. I see delivery over the network not as a fundamentally different thing than delivery on CD. It's not the thing Debian is necessarily good at. Let us concentrate on the OS building. Hosting mirrors wasn't our core activity either 20 years ago, yet we had to distribute this thing somehow. I would argue that distributing Debian, regardless of the mechanism (CDNs, mirror infrastructure, CDs) is as much part of making the software real as pushing packages to ftp-master... which are based on a spirit of free access and open knowledge, something commercial CDNs seem to be alien to... I have not seen any indication that commercial CDN's would necessarily be alien to free access and open knowledge, in fact, it would seem rather contrary to their nature to hinder access to content. To take an example of one of the proposed CDNs, Amazon has been actively destroying data on people's e-readers, 1984 of all books. This would qualify as simply hostile. I would also point out that none of the CDNs *publish* the software they develop as free software. They may *use* free software, but they built their business on proprietary software on top of our hard work. That is what I meant. A. -- Software gets slower faster than hardware gets faster. - Wirth's law pgpoLr0jVm6u1.pgp Description: PGP signature
Re: Possibly moving Debian services to a CDN
On 2013-11-14 10:37:21, Tollef Fog Heen wrote: ]] anarcat On 2013-11-14 05:20:12, Tollef Fog Heen wrote: ]] anarcat I'm not aware of any open source 40GE PHYs for instance? Most of what I've seen done is around SDN, which is all nice and good, but doesn't actually make the PHY and MAC firmware free. I haven't been trawling around, so maybe it's further ahead than I thought. Even if you restrict yourself to the main UEFI implementation, can you even get anything free for a modern server there? With warranties? Having to void your warranty to run a free UEFI implementation is not something we'd like to do. Well, those are pretty specific examples, which apply to all hardware we are currently using. With that argument, we might as well say we are not running free software at all. :) There's some inevitable proprietary hardware out there, sure, but it shouldn't be an argument to letting more proprietary software into our infrastructure. This is not a two-tone discussion and trying to make it one will not lead to a useful outcome. And saying that because there's proprietary firmware in your BIOS it's okay to offload all of Debian's infrastructure to a non-free CDN is okay seems to me to be a slippery slope. Nobody has talked about moving all of Debian's infrastructure. Okay, some. The argument remains: it's still a slippery slope argument. :) The concern for me is moving a critical part of the infrastructure there. If we are just talking about adding resources, I think that's an interesting idea, but the title of this thread and the original post were suggested to move some of that critical infrastructure over proprietary and heavily surveilled commercial providers. I am uncomfortable with that idea. It's not a vote, and it's easy for the people who do not have to do anything but send mails to a mailing list to say «we should spend more effort». Well, if it's not a vote, and if my opinion doesn't actually matter, why are we discussing this on -project in the first place? We were hoping to get some useful feedback. I hope you did! It certainly encouraged me to look further into hosting our own Debian mirror now. :) And the improvements necessary to get this to a commercial-grade CDN doesn't seem to me like much more: some IP alias on the mirror machine, and BGP announcements. Am I missing something? Yes. If you're just anycasting an IP, you'll get pretty poor performance. Can you expand on that? You need monitoring to make sure the mirror is up to date and something that automatically updates DNS when it isn't, and puts it back in when it is. That is a problem we're having already, and that we'll probably have with commercial CDNs, or at least that we'll have to resolve so get a consistent state across the mirrors. You need to herd the mirror operators, keep them happy, make sure they're using the right tools so you don't get transient breakages in the middle of a mirror run. Right. We could start with a smaller subset... If you're going to do anycast, you'll need to have BGP announcements sent from a diverse set of places. This seems like something we have, with the variety of mirrors out there. :) You need to monitor your BGP stuff, trace down why you get suboptimal routing for a given user and get that fixed. I am not sure how or why you could get suboptimal routing through BGP links, but I am fairly new with this technology (a few years) so maybe I am missing something here again. You'll need to run GeoDNS and correlate that with routing so hopefully, the user hits the best server. I was thinking that using anycast would alleviate the need for geoDNS hacks... And that's just a few items off the top of my head. I responded to those, I hope you don't mind having that discussion. :) If you are tired of it or were just giving examples, feel free to disregard. Running a CDN is not hard. Running a CDN well, over time, is hard. It's something that DSA would not add value by doing ourselves, just like we would not add value by creating our own OOB solutions or soldering together our own UPSes. It's not that we can't, it's that it's cheaper and more reliable to buy a ready-made solution and most important of all: it's not part of our core mission. It's a means to an end, not an end goal in itself. I see your point, although I think there is a few orders of magnitudes between adding improvements to the current mirror infrastructures versus soldering PCBs... It just seems to me that providing the basic tools for sponsors to join the CDN doesn't seem that complicated. After all, DSA doesn't manage all mirrors right now. I guess what I am saying is that doing incremental improvements over the mirror infrastructure should be considered. I am worried that migrating to a commercial CDN will be detrimental to the current infrastructure, which are based on a spirit of free access and open knowledge, something commercial CDNs
Re: Possibly moving Debian services to a CDN
On Wed, November 20, 2013 19:03, anarcat wrote And saying that because there's proprietary firmware in your BIOS it's okay to offload all of Debian's infrastructure to a non-free CDN is okay seems to me to be a slippery slope. Nobody has talked about moving all of Debian's infrastructure. Okay, some. The argument remains: it's still a slippery slope argument. :) The slippery slope argument is often considered a logical fallacy and not without reason. The argument implies that by making choice A we would automatically not be stopping to make the 'worse' choice B, while Debian at any and all moments keeps the freedom to do A and not B. I see your point, although I think there is a few orders of magnitudes between adding improvements to the current mirror infrastructures versus soldering PCBs... Debian's core activity is building the best OS in the world. Distributing the actual result of that is, at least in my opinion, a thing very peripheral to that activity. The fact that Debian has always shied away from making CD's and distrbuting those to users but rather left that to commercial and non-commercial parties, indicates that this is not a novel development. We allow (and promoted) commercial CD services. I see delivery over the network not as a fundamentally different thing than delivery on CD. It's not the thing Debian is necessarily good at. Let us concentrate on the OS building. which are based on a spirit of free access and open knowledge, something commercial CDNs seem to be alien to... I have not seen any indication that commercial CDN's would necessarily be alien to free access and open knowledge, in fact, it would seem rather contrary to their nature to hinder access to content. Thijs -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/741f124a517fd73222781224a56f4fcb.squir...@aphrodite.kinkhorst.nl
Re: Possibly moving Debian services to a CDN
]] anarcat On 2013-11-14 10:37:21, Tollef Fog Heen wrote: Yes. If you're just anycasting an IP, you'll get pretty poor performance. Can you expand on that? BGP anycast will just get you the closest one in term of metrics. This is probably the fewest number of cheapest hops. There's no guarantee those hops are going to be the shortest or fastest. You need monitoring to make sure the mirror is up to date and something that automatically updates DNS when it isn't, and puts it back in when it is. That is a problem we're having already, and that we'll probably have with commercial CDNs, or at least that we'll have to resolve so get a consistent state across the mirrors. Yes, and we're doing a terrible job at it. It's a manual job right now, and there's essentially a single person doing that job. I'm not trying to pick on him, since he's doing as well as he can, but it's the wrong way to go about solving the problem. CDNs have infrastructure for distributing purges, so we'd just go «nuke all your Release and Packages files» and assuming the CDN isn't too bad, they'd be gone a few hundred milliseconds later. If you're going to do anycast, you'll need to have BGP announcements sent from a diverse set of places. This seems like something we have, with the variety of mirrors out there. :) We don't. Debian doesn't run its own AS, we don't have peering agreements and we don't announce anything anywhere. We don't have any interest in doing so either. It's not what we think is fun, nor what we're good at. I guess what I am saying is that doing incremental improvements over the mirror infrastructure should be considered. I am worried that migrating to a commercial CDN will be detrimental to the current infrastructure, which are based on a spirit of free access and open knowledge, something commercial CDNs seem to be alien to... We've waited for somebody to step up and actually do that. Nobody is doing it. Lots of proof-of-concept services out there, but nothing that's solid, tested and production ready. How long would you like us to keep the current state before we actually do something about it? -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87mwkyda0o@xoog.err.no
Re: Possibly moving Debian services to a CDN
]] anarcat All the tools currently running the Debian mirror architecture. Some mirrors may run an FTP mirror on a non-free software, but they don't *have* to, and we unfortunately can't control that. No, they can't. You can't route a packet through the public internet without it hitting non-free software. Heck, you can't get a normal server that will run without non-free software. (If nothing else, firmware for the ethernet controller or the firmware for the EC or disks.) This is not a two-tone discussion and trying to make it one will not lead to a useful outcome. Yes, it's extra work, but it's feasible. The question is: do we want to keep on running our own CDN, or do we want to give up? I say we should keep doing it. Autonomy is important for our community. And a commercial CDN will come with strings attached - Gimp just moved off Sourceforge for that reason... It's not a vote, and it's easy for the people who do not have to do anything but send mails to a mailing list to say «we should spend more effort». -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87fvqz8eub@qurzaw.varnish-software.com
Re: Possibly moving Debian services to a CDN
On 2013-11-14 05:20:12, Tollef Fog Heen wrote: ]] anarcat All the tools currently running the Debian mirror architecture. Some mirrors may run an FTP mirror on a non-free software, but they don't *have* to, and we unfortunately can't control that. No, they can't. You can't route a packet through the public internet without it hitting non-free software. You can't *right now*. I assume you are talking in part of those cisco switches that litter the network, and I can tell you a lot of people are tired of those - for example Facebook (of all places...) is building an open alternative for those. As much as I can, we try to build infrastructure that is free. At every step of the way people tell me don't do that, use cisco switches dumbass. But somewhat, I still manage to use as free hardware and free software as I can. Heck, you can't get a normal server that will run without non-free software. (If nothing else, firmware for the ethernet controller or the firmware for the EC or disks.) Sure, but we at least try. A lot of great people are working on coreboot and such initiatives that go a long way, probably farther that we've ever been, in making sure that the stack is as open as we can. This is not a two-tone discussion and trying to make it one will not lead to a useful outcome. And saying that because there's proprietary firmware in your BIOS it's okay to offload all of Debian's infrastructure to a non-free CDN is okay seems to me to be a slippery slope. Sure, there are shades of gray here, but i'd try to avoid going further down the non-free path. Yes, it's extra work, but it's feasible. The question is: do we want to keep on running our own CDN, or do we want to give up? I say we should keep doing it. Autonomy is important for our community. And a commercial CDN will come with strings attached - Gimp just moved off Sourceforge for that reason... It's not a vote, and it's easy for the people who do not have to do anything but send mails to a mailing list to say «we should spend more effort». Well, if it's not a vote, and if my opinion doesn't actually matter, why are we discussing this on -project in the first place? We have been considering making our own mirror for Debian here. What kept us is of course labour, but also bandwidth costs. And the problem is mostly that those are not clearly established variables: if we knew how much a mirror would consume in terms of bandwidth, we would be in a better position to make such a decision. Right now, the documentation seems a little outdated, saying that we need a T1 or better... From what I understand, the labour involved in running a mirror right now is simply running the ftpsync cronjob and maintaining a few GB of disk space (1TB for the full archive) on a good connexion (which still needs some clarification, FWICT). And the improvements necessary to get this to a commercial-grade CDN doesn't seem to me like much more: some IP alias on the mirror machine, and BGP announcements. Am I missing something? A. -- Voter, c'est abdiquer - Élisée Reclus pgpRwhaSylRk8.pgp Description: PGP signature
Re: Possibly moving Debian services to a CDN
]] anarcat On 2013-11-14 05:20:12, Tollef Fog Heen wrote: ]] anarcat All the tools currently running the Debian mirror architecture. Some mirrors may run an FTP mirror on a non-free software, but they don't *have* to, and we unfortunately can't control that. No, they can't. You can't route a packet through the public internet without it hitting non-free software. You can't *right now*. I assume you are talking in part of those cisco switches that litter the network, and I can tell you a lot of people are tired of those - for example Facebook (of all places...) is building an open alternative for those. I'm not aware of any open source 40GE PHYs for instance? Most of what I've seen done is around SDN, which is all nice and good, but doesn't actually make the PHY and MAC firmware free. I haven't been trawling around, so maybe it's further ahead than I thought. Heck, you can't get a normal server that will run without non-free software. (If nothing else, firmware for the ethernet controller or the firmware for the EC or disks.) Sure, but we at least try. A lot of great people are working on coreboot and such initiatives that go a long way, probably farther that we've ever been, in making sure that the stack is as open as we can. Even if you restrict yourself to the main UEFI implementation, can you even get anything free for a modern server there? With warranties? Having to void your warranty to run a free UEFI implementation is not something we'd like to do. This is not a two-tone discussion and trying to make it one will not lead to a useful outcome. And saying that because there's proprietary firmware in your BIOS it's okay to offload all of Debian's infrastructure to a non-free CDN is okay seems to me to be a slippery slope. Nobody has talked about moving all of Debian's infrastructure. It's not a vote, and it's easy for the people who do not have to do anything but send mails to a mailing list to say «we should spend more effort». Well, if it's not a vote, and if my opinion doesn't actually matter, why are we discussing this on -project in the first place? We were hoping to get some useful feedback. [...] And the improvements necessary to get this to a commercial-grade CDN doesn't seem to me like much more: some IP alias on the mirror machine, and BGP announcements. Am I missing something? Yes. If you're just anycasting an IP, you'll get pretty poor performance. You need monitoring to make sure the mirror is up to date and something that automatically updates DNS when it isn't, and puts it back in when it is. You need to herd the mirror operators, keep them happy, make sure they're using the right tools so you don't get transient breakages in the middle of a mirror run. If you're going to do anycast, you'll need to have BGP announcements sent from a diverse set of places. You need to monitor your BGP stuff, trace down why you get suboptimal routing for a given user and get that fixed. You'll need to run GeoDNS and correlate that with routing so hopefully, the user hits the best server. And that's just a few items off the top of my head. Running a CDN is not hard. Running a CDN well, over time, is hard. It's something that DSA would not add value by doing ourselves, just like we would not add value by creating our own OOB solutions or soldering together our own UPSes. It's not that we can't, it's that it's cheaper and more reliable to buy a ready-made solution and most important of all: it's not part of our core mission. It's a means to an end, not an end goal in itself. -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/m2iovv6lla@rahvafeir.err.no
Re: Possibly moving Debian services to a CDN
On 2013-10-29 06:19:14, Tollef Fog Heen wrote: ]] Stefano Zacchiroli For the specific case of CDN offerings to the Debian Project, the point---well, my point, I respect the fact that others disagree it's a problem---is whether we're going to force our user to receive the Free Software we're distributing via infrastructures built using non-free software. That problem would exist even if the companies behind those services were big Free Software advocates, which just happen to have a single service (the CDN) built using non-free software. You seem to be under the impression that CDN implies non-free software. Fastly uses Varnish (which is free software). Cloudfront uses Apache (which is free software). I'm sure there are CDNs using non-free software too, but that doesn't seem particularly relevant. One CDN may use one piece of free software, but do they really embrace the free software spirit the way the Debian project does? Do they publish all the tools that make their mirrors run? Amazon, for example, clearly do not free all the software that run their mirror network. All the tools currently running the Debian mirror architecture. Some mirrors may run an FTP mirror on a non-free software, but they don't *have* to, and we unfortunately can't control that. If we go with Cloudfront or Fastly, we *know* that they *will* use non-free software to manage their architecture. Furthermore, to take the example of Fastly, they have only 11 locations - 6 in the US, 3 in Europe and 2 in New Zealand. Notice: none in china, for example. This is certainly less than the current mirror infrastructure. Amazon doesn't have mirrors in China either (although they have one in Hong Kong). Same with Internap. So just saying CDN doesn't necessarily solve our problems, even assuming they give us the service for free. Finally, as an organisation that recently started working with BGP announcements, AS numbers and IP allocation, I have to say that it is rather easy to configure. Making this an option for mirrors could be an attractive option that could significantly enhance the current set of mirrors. Debian would simply need to get an IP allocation (or lease/borrow one, I am sure one HP wouldn't mind giving away a /24 out of it's /8 :P) and run BGP on mirrors. Yes, it's extra work, but it's feasible. The question is: do we want to keep on running our own CDN, or do we want to give up? I say we should keep doing it. Autonomy is important for our community. And a commercial CDN will come with strings attached - Gimp just moved off Sourceforge for that reason... A. -- Si les triangles avaient un Dieu, ils lui donneraient trois côtés. - Montesquieu, Lettres persanes pgp2zIcMXTGpU.pgp Description: PGP signature
Re: Possibly moving Debian services to a CDN
On 29/10/13 at 10:54 +0100, Stefano Zacchiroli wrote: Just a minor nitpick on the point below; for the more general discussion I stand behind the opinion I've previously posted in this thread. On Sun, Oct 20, 2013 at 04:00:42PM +0200, Lucas Nussbaum wrote: After all, if we could use and point to 3-4 CDNs that are advocating Free Software, isn't it better to show that such core Internet services can be run using Free Software? I don't think whether CDN offerings (and/or the companies behind them) are *advocating* Free Software or not is relevant. IMO it's neither relevant for this discussion, nor it's relevant in general to decide the kind of relationships Debian should have with companies. Basically all medium-to-big sized companies are split on how much they advocate Free Software. You invariably find company parts that are hard-core Free Software advocates, and other parts that don't care and turn a living by developing non-free software and services that deprive users of the software freedoms they are entitled to. For the specific case of CDN offerings to the Debian Project, the point---well, my point, I respect the fact that others disagree it's a problem---is whether we're going to force our user to receive the Free Software we're distributing via infrastructures built using non-free software. That problem would exist even if the companies behind those services were big Free Software advocates, which just happen to have a single service (the CDN) built using non-free software. Right; my point is about CDN providers that are public about their use of Free Software (and possibly about their contributions to Free Software) for their CDN infrastructure. Not about companies advocating Free Software in general -- I agree that this would not be relevant. Lucas -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131031143858.ga27...@xanadu.blop.info
Re: Possibly moving Debian services to a CDN
Just a minor nitpick on the point below; for the more general discussion I stand behind the opinion I've previously posted in this thread. On Sun, Oct 20, 2013 at 04:00:42PM +0200, Lucas Nussbaum wrote: After all, if we could use and point to 3-4 CDNs that are advocating Free Software, isn't it better to show that such core Internet services can be run using Free Software? I don't think whether CDN offerings (and/or the companies behind them) are *advocating* Free Software or not is relevant. IMO it's neither relevant for this discussion, nor it's relevant in general to decide the kind of relationships Debian should have with companies. Basically all medium-to-big sized companies are split on how much they advocate Free Software. You invariably find company parts that are hard-core Free Software advocates, and other parts that don't care and turn a living by developing non-free software and services that deprive users of the software freedoms they are entitled to. For the specific case of CDN offerings to the Debian Project, the point---well, my point, I respect the fact that others disagree it's a problem---is whether we're going to force our user to receive the Free Software we're distributing via infrastructures built using non-free software. That problem would exist even if the companies behind those services were big Free Software advocates, which just happen to have a single service (the CDN) built using non-free software. Cheers. -- Stefano Zacchiroli . . . . . . . z...@upsilon.cc . . . . o . . . o . o Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o Former Debian Project Leader . . @zack on identi.ca . . o o o . . . o . « the first rule of tautology club is the first rule of tautology club » signature.asc Description: Digital signature
Re: Possibly moving Debian services to a CDN
]] Stefano Zacchiroli For the specific case of CDN offerings to the Debian Project, the point---well, my point, I respect the fact that others disagree it's a problem---is whether we're going to force our user to receive the Free Software we're distributing via infrastructures built using non-free software. That problem would exist even if the companies behind those services were big Free Software advocates, which just happen to have a single service (the CDN) built using non-free software. You seem to be under the impression that CDN implies non-free software. Fastly uses Varnish (which is free software). Cloudfront uses Apache (which is free software). I'm sure there are CDNs using non-free software too, but that doesn't seem particularly relevant. -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87zjpsv0pp@qurzaw.varnish-software.com
Re: Possibly moving Debian services to a CDN
On Tue, Oct 29, 2013 at 11:19:14AM +0100, Tollef Fog Heen wrote: You seem to be under the impression that CDN implies non-free software. Oh, no, not at all. I'm just saying that we should judge CDN offerings on the basis of the kind of software they're build upon, and not on the basis of how much the companies offering them advocate Free Software. -- Stefano Zacchiroli . . . . . . . z...@upsilon.cc . . . . o . . . o . o Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o Former Debian Project Leader . . @zack on identi.ca . . o o o . . . o . « the first rule of tautology club is the first rule of tautology club » signature.asc Description: Digital signature
Re: Possibly moving Debian services to a CDN
]] Stefano Zacchiroli On Tue, Oct 29, 2013 at 11:19:14AM +0100, Tollef Fog Heen wrote: You seem to be under the impression that CDN implies non-free software. Oh, no, not at all. I'm just saying that we should judge CDN offerings on the basis of the kind of software they're build upon, and not on the basis of how much the companies offering them advocate Free Software. I don't believe we ask mirror operators what OS their mirror runs on or whether it's free software today. While I'd like both them and a CDN to use free software, this doesn't look like it'll change anything from current practice. -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/871u34us83@qurzaw.varnish-software.com
Re: Possibly moving Debian services to a CDN
On Tue, Oct 29, 2013 at 02:22:36PM +0100, Tollef Fog Heen wrote: I don't believe we ask mirror operators what OS their mirror runs on or whether it's free software today. While I'd like both them and a CDN to use free software, this doesn't look like it'll change anything from current practice. We're running in circles :-) See: https://lists.debian.org/debian-project/2013/10/msg00058.html -- Stefano Zacchiroli . . . . . . . z...@upsilon.cc . . . . o . . . o . o Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o Former Debian Project Leader . . @zack on identi.ca . . o o o . . . o . « the first rule of tautology club is the first rule of tautology club » signature.asc Description: Digital signature
Re: Possibly moving Debian services to a CDN
This one time, at band camp, Simon Paillard said: Hi, On Sat, Oct 19, 2013 at 11:07:05PM +0200, Martin Zobel-Helas wrote: On Wed Oct 16, 2013 at 21:01:08 +0200, Simon Paillard wrote: On Wed, Oct 16, 2013 at 07:54:04PM +0100, Stephen Gran wrote: This one time, at band camp, Simon Paillard said: [..] * Our (mirrors@d.o) is to have http.d.n made official so that we can have an instance like on each continent, then use GeoDNS, like security.d.o mirrors, to achieve 2 goals: avoid SPOF, be closer to users. in long term, i would like to get rid of our GeoDNS setup. I guess you have a rationale for this ? Which issues do you have with GeoDNS ? The major design problem is that it points you to a mirror that is close to your DNS server, rather than a mirror that is close to you. Also, it doesn't actually point you to the closest mirror, just a mirror on the same continent as you (unless you're in Africa or Asia, in which case oh well). This is sometimes good enough (as is the case for Europe, and mostly the case for America), but probably not for South America, where peering arrangements between countries can actually be worse than links to somewhere further away like the US. Anyway, I've gotten the sense that this conversation has now completed the entire Debian pattern of: I have an idea that might make things better No!!! You are worse than Hitler I give up So I'm probably going to stop posting about it now. Cheers, -- - | ,''`.Stephen Gran | | : :' :sg...@debian.org | | `. `'Debian user, admin, and developer | |`- http://www.debian.org | - signature.asc Description: Digital signature
Re: Possibly moving Debian services to a CDN
On Sun, Oct 20, 2013 at 09:53:35AM +0100, Stephen Gran wrote: This one time, at band camp, Simon Paillard said: On Sat, Oct 19, 2013 at 11:07:05PM +0200, Martin Zobel-Helas wrote: On Wed Oct 16, 2013 at 21:01:08 +0200, Simon Paillard wrote: On Wed, Oct 16, 2013 at 07:54:04PM +0100, Stephen Gran wrote: This one time, at band camp, Simon Paillard said: [..] * Our (mirrors@d.o) is to have http.d.n made official so that we can have an instance like on each continent, then use GeoDNS, like security.d.o mirrors, to achieve 2 goals: avoid SPOF, be closer to users. in long term, i would like to get rid of our GeoDNS setup. I guess you have a rationale for this ? Which issues do you have with GeoDNS ? The major design problem is that it points you to a mirror that is close to your DNS server, rather than a mirror that is close to you. In our idea, it would just points to a close http.d.n redirector, not to a close mirror. The point you raises applies to a solution like cdn.debian.net. -- Simon Paillard -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131020090257.gl26...@mraw.org
Re: Possibly moving Debian services to a CDN
* Simon Paillard: * My own experience is different, http.d.n redirects to ftp2.fr, which i got 10,2Mo/s, while cloudfront.d.n (Amazon) gives 5Mo/s. This matches my experience. One of the CDNs we use at work does not seem to pre-replicate content world-wide (which is not too surprising, I suspect this would be a poor trade-off for them), so when I download an obscure package that none of our customers in close network proximity to me has downloaded yet from the same CDN node, I get relatively poor download rates. Subsequent downloads of the same package are faster, but performance is still not quite comparable to local community-provided package mirrors. -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87txgc8ecc@mid.deneb.enyo.de
Re: Possibly moving Debian services to a CDN
Stephen Gran wrote: This one time, at band camp, Raphael Geissert said: They do require a bit more work as they can not be added to the mirrors master list - they are dumb caching proxies that can not guarantee the consistency of the view of the archive they provide. Here's where I disagree with Russ as a piece of software is needed to address that, it's not all about the network. That's mostly because we're not actually 'using' them now - we're just allowing them to cache. Most CDNs have a decache mechanism of some sort or other that we could use on mirror pulses, or we could tune the cache headers to actually make it possible for CDNs to do the right thing, etc. I'm aware of those methods to expire objects and I can tell you that they are already in use for cloudfront.d.n. Even with them I still need to re- enable cloudfront.d.n from http.debian.net as from time to time it fails a consistency check and gets banned. Looking at report.txt right now it seems that some got banned again. Cheers, -- Raphael Geissert - Debian Developer www.debian.org - get.debian.net -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/l409ae$l24$1...@ger.gmane.org
Re: Possibly moving Debian services to a CDN
On 13/10/13 at 08:44 +0200, Tollef Fog Heen wrote:
We appreciate feedback while we continue our investigation of CDNs.
Hi,
I'm trying to summarize the discussion so far and add my own
understanding/thoughts, in a set of Q A.
Q: What problem are we trying to solve? What's the current status?
==
The Debian project needs to distribute content over HTTP; mainly
packages (on ftp.d.o and security.d.o) and websites (such as www.d.o).
For that, a set of machines running Debian and managed by DSA and
located in various datacenters all over the world are used.
Additionally, for ftp.d.o, Debian relies on mirrors provided by third
parties. Those mirrors are not managed by DSA, and might be running
non-free software (that could include primary mirrors, ie
ftp.*.debian.org).
Mirroring the Debian archive is tricky, as files need to be copied in
the correct order (to avoid having files in dists/ point to files
not yet copied in pool/). A script (ftpsync[1]) is provided by the
mirrors team, but is not used on every mirror AFAIK.
[1] http://www.debian.org/mirror/ftpmirror
The performance of our {packages,website} delivery network is an
interesting question. Like many things on the Internet, it's related to
a mix of bandwidth, latency, and application behaviour (e.g. use of HTTP
keep-alive). More and more, the dominating factor in network
performance is latency (as others are easier to optimize), and the
only way to reduce it is to have servers close (geographically or
network-wise) to end users. Benchmarking mirrors by measuring
bandwidth is generally not very relevant.
This raises several challenges:
- DSA needs to interact with many datacenters, often for only one
machine. This is very time-consuming.
- The mirrors team needs to constantly monitor mirrors and notify mirror
operators in case of problems. Notifications are automated, but DNS
updates to *.debian.org when a mirror fails are not.
- There are parts of the world that are not so well covered. For
example, http://deb.li/y8GA is the current map of security.d.o
mirrors (which are all managed by DSA), we don't have any point of
presence in Asia, which causes poor performance. There are discussions
in progress to buy a server and host it somewhere in Asia, and the cost
for Debian would be between $1500 and $2500 depending on the server's
specs.
One solution that has been developed is http.d.n. It's a redirector
service that redirects to the closest working mirror (the mirror
checking is automated). However, the http.d.n machine is still
centralized: round-trip time to it is still a problem, so, if the
service would become official, several geographically-distributed
instances of the service would have to be set up. Also, as each request
goes through a http.d.n redirect, there's a lot of additional latency.
If we want those http.d.n redirector machines to be managed by DSA
(which is probably something we want), it doesn't really improve the
situation in terms of machines DSA has to managed.
Q: What are CDNs? How do they compare to our mirrors network?
=
Content Delivery Networks (Akamai, Fastly, Amazon Cloudfront, etc. [1])
can be seen as giant location-aware caching networks. They provide
local points of presence and manage global caching of external data
inside the CDN network.
[1] http://www.cdnplanet.com/cdns/
As a solution based on caching, they work and perform quite differently
from our mirrors (where the Debian archive is fully replicated). It's
not easy to compare their performance, especially if you want to
consider access patterns on the mirrors (file sizes, long tail
distribution, etc.)
Q: Do CDNs raise more security/privacy concerns than our mirrors?
=
Not easy to answer. I'm inclined to say that they both raise about
the same amount of concerns. There's more discussion about those points
in the subthread starting at
http://lists.debian.org/2a773832-09f2-4adb-9b10-2a554b6dd...@2013.bluespice.org
Q: How does that meet with Debian's Social Contract and Free Software in
general?
Some CDNs use Free Software. As data points, Fastly[1,2] uses and
contributes to Varnish, and the frontend servers of Amazon Cloudfront
are running Apache.
[1] http://www.fastly.com/about
[2] http://www.fastly.com/about/open-source
Building a CDN is mostly an infrastructure problem: bring PoP in many
parts of the world, manage those servers, etc. It would be about Free
Infrastructure more than Free Software.
How much do we (Debian) care about Free Infrastructure?
The Social Contract says:
1. Debian will remain 100% free
[..] We promise that the Debian system and all its components will be
free according to these guidelines. [..] We will never make the system
require
Re: Possibly moving Debian services to a CDN
On 20/10/2013 5:55 PM, Raphael Geissert wrote: Stephen Gran wrote: That's mostly because we're not actually 'using' them now - we're just allowing them to cache. Most CDNs have a decache mechanism of some sort or other that we could use on mirror pulses, or we could tune the cache headers to actually make it possible for CDNs to do the right thing, etc. I'm aware of those methods to expire objects and I can tell you that they are already in use for cloudfront.d.n. Even with them I still need to re- enable cloudfront.d.n from http.debian.net as from time to time it fails a consistency check and gets banned. Looking at report.txt right now it seems that some got banned again. It's been some time since i have delved into these deeply again, but I'd love to continue to tune down the cache expiry headers on cloudfront.debian.net as needed; for those not aware I covered this in detail in my presentation at Debconf. Here's the summary of the 'introduced' Cache-Control Max-Age headers that are currently in place: Default: 24 hours (Cloudfront Default on objects that have no headers) /debian/dists/*: 15 minutes (default for all files, overridden by subsequent rules) /debian/dists/(unstable|sid)/.*: 5 minutes /debian/dists/.*\.diff/[\d-]+\.gz: 2 hours - these are datestamped filenames and don't change once uploaded /debian/dists/.*\.diff/(Index)?: 10 seconds /debian/dists/.*/(Contents-.*\.(bz|gz)|(In)?Release(\.gpg)?)?: 20 seconds /debian/dists/(unstable|sid)/(Contents-.*\.(bz|gz)|(In)?Release(\.gpg)?)?: 10 seconds /debian/dists/.*/i18n/(Index|Translation-.*)?: 10 seconds /debian/dists/.*/(binary-.*|source)/(Packages(\..*)?|Sources(\..*)?|Release)?: 10 seconds /debian/project/.*: 10 seconds These rules are all running on a tiny little Apache instance, becuase upstream mirrors do not have any Cache-Control headers. I am hoping that if we are happy refining these cache times, we can migrate these rules to an upstream HTTP server (this is currently using ftp.debian.org) and do away with an 'interstitial' server that is squirting these headers into the response. 0 seconds would work but probably overload things - that's no caching at all for every edge hit. And 1 second means that for objects in heavy use, we'll have 43 hits/sec, so something a little longer than that... hence the 10/20 seconds lines above. CloudFront is currently 43 locations worldwide, and continuing to expand. That's much less than the 400 mirrors in the Debian list. I would not recommend abandoning mirrors. James (Its just mod proxy and mod headers in Apache) (PS: I am travelling this week and possibly slower at answering email - ping jameseb_AT_amazon.com if you need me urgently) (PPS: Happy to give any DD/DM or read access into the AWS Account - just send me a signed email - and read/write (ie, for DSA) if you want). -- /Mobile:/ +61 422 166 708, /Email:/ james_AT_rcpt.to
Re: Possibly moving Debian services to a CDN
This one time, at band camp, Ian Jackson said: I don't see a clear explanation of what the motivation is to switch to a commercial CDN. Can you clarify ? That will help us understand what we would be giving up if we decline to make this change. There are probably several having to do with effort and effectiveness, but I'll just share one story that's been ongoing. At debconf in Spain, I spoke with several people about the idea of putting in a security mirror in the vicinity of China. We are still talking about this, and users in China have made do with .. I don't know what in the meantime. Getting things closer to our users makes things nicer for them - time to first byte can be appalling when viewing our web page from some parts of the world, especially when on a mobile device. Time to first byte matters less when downloading large packages, but again, if we can get caching with large bandwidth, it's a better experience for the end users. I count at least a dozen d.o machines that are nothing but mirrors of one thing or another, and we still don't have very good coverage. There are lots and lots of subdomains (lintian, popcon, people, etc) that are in basically one place. That's a single point of failure, and also, half the time, on the far side of the world from our users. Sure, we could add more mirror machines, and we could make more and more mirror infrastructure and build our own CDN. At some point, though, it makes sense to me to say, other people already do this really well. We can carry on the way we are, but I think we'll lose out on improving throughput for our end users, getting close to users in parts of the world that have been difficult for us to find hardware or hosting, and reduce the workload on DSA while making more services more available. I understand your concerns about privacy, although I think they are perhaps unrealistic. You're asking, in a world where security services tap uplink cables and passively record metadata about all traffic, for me to get upset about whether someone logs an IP or not. Once the transit is proven to be unsafe, the endpoints no longer matter, in my mind. That being said, I don't think that at least the first iteration of this should force anyone off of their preferred mirror. I don't think we want to rush into this, and I don't think we want to throw out all the work we've done already to make our home-brewed CDN. I think we can let them coexist for a while, and see how we do. We have had one report that someone gets better throughput off of their existing mirror than they do from one of the CDNs offering their support. If we get better coverage for a small number of users and worse coverage for most of our existing users, that's clearly not a tradeoff worth making. I guess what I'm saying is, I can see lots of ways that this can make things better. Right now, we don't have any metrics about the tradeoffs, all we have are emotional responses. I'd like to start trying, so we can collect those metrics and make an informed decision. We can always say, thanks but no thanks later, right? Cheers, -- - | ,''`.Stephen Gran | | : :' :sg...@debian.org | | `. `'Debian user, admin, and developer | |`- http://www.debian.org | - signature.asc Description: Digital signature
Re: Possibly moving Debian services to a CDN
Stephen Gran wrote: We have had one report that someone gets better throughput off of their existing mirror than they do from one of the CDNs offering their support. If we get better coverage for a small number of users and worse coverage for most of our existing users, that's clearly not a tradeoff worth making. One advantage of us being the ones controlling the service is that we can easily incorporate the contributions from those commercial CDNs into the existing mirrors network. As mentioned by James Bromberger during his DC13 presentation, cloudfront is already in use by http.debian.net for users who would actually benefit from it. I personally welcome those contributions from commercial CDNs to improve the final service to our users. If any company other than AWS would like to offer its service to Debian I'm happy to work on adding them to http.d.n. They do require a bit more work as they can not be added to the mirrors master list - they are dumb caching proxies that can not guarantee the consistency of the view of the archive they provide. Here's where I disagree with Russ as a piece of software is needed to address that, it's not all about the network. Cheers, -- Raphael Geissert - Debian Developer www.debian.org - get.debian.net -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/l3tobl$mt8$1...@ger.gmane.org
Re: Possibly moving Debian services to a CDN
This one time, at band camp, Raphael Geissert said: They do require a bit more work as they can not be added to the mirrors master list - they are dumb caching proxies that can not guarantee the consistency of the view of the archive they provide. Here's where I disagree with Russ as a piece of software is needed to address that, it's not all about the network. That's mostly because we're not actually 'using' them now - we're just allowing them to cache. Most CDNs have a decache mechanism of some sort or other that we could use on mirror pulses, or we could tune the cache headers to actually make it possible for CDNs to do the right thing, etc. They're currently more work than machines that actually have the files on disk because we treat them the same way, not because they are actually worse, if you see what I mean. Cheers, -- - | ,''`.Stephen Gran | | : :' :sg...@debian.org | | `. `'Debian user, admin, and developer | |`- http://www.debian.org | - signature.asc Description: Digital signature
Re: Possibly moving Debian services to a CDN
Hi, On Wed Oct 16, 2013 at 21:01:08 +0200, Simon Paillard wrote: Hi, On Wed, Oct 16, 2013 at 07:54:04PM +0100, Stephen Gran wrote: This one time, at band camp, Simon Paillard said: We already have a network of almost 400 packages mirrors around the world. Using http.d.n, provides the CDN layer (not as much as optimal as anycast), so we don't need to sort ourselves peering issues etc. The mirrors do a very good job of being near to our users in most cases (we have had some difficulty with things like security mirror coverage, but that's not your issue). I know many people are happy with http.d.n, but you have to understand that it's not the same thing at all as a CDN - it's a single host, and it's in a single location. Time to first byte will still suffer if you're coming from Australia. Obviously, it's due to current unsponsored deployment. * For a package of several kB, the first bytes (actually ~800B, measured) are not meaningful * Our (mirrors@d.o) is to have http.d.n made official so that we can have an instance like on each continent, then use GeoDNS, like security.d.o mirrors, to achieve 2 goals: avoid SPOF, be closer to users. in long term, i would like to get rid of our GeoDNS setup. Martin -- Martin Zobel-Helas zo...@debian.orgDebian System Administrator Debian GNU/Linux Developer Debian Listmaster http://about.me/zobel Debian Webmaster GPG Fingerprint: 6B18 5642 8E41 EC89 3D5D BDBB 53B1 AC6D B11B 627B -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131019210705.gi3...@ftbfs.de
Re: Possibly moving Debian services to a CDN
Hi, On Sat, Oct 19, 2013 at 11:07:05PM +0200, Martin Zobel-Helas wrote: On Wed Oct 16, 2013 at 21:01:08 +0200, Simon Paillard wrote: On Wed, Oct 16, 2013 at 07:54:04PM +0100, Stephen Gran wrote: This one time, at band camp, Simon Paillard said: [..] * Our (mirrors@d.o) is to have http.d.n made official so that we can have an instance like on each continent, then use GeoDNS, like security.d.o mirrors, to achieve 2 goals: avoid SPOF, be closer to users. in long term, i would like to get rid of our GeoDNS setup. I guess you have a rationale for this ? Which issues do you have with GeoDNS ? -- Simon Paillard -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131019221953.gj26...@mraw.org
Re: Possibly moving Debian services to a CDN
Tollef Fog Heen writes (Re: Possibly moving Debian services to a CDN): I'm fundamentally of the opinion that if the NSA or a similar organisation wants to track you and is willing to expend that effort on tracking you in particular, there is just about nothing you can do about it. This is true, but largely irrelevant. That the NSA can get something if they really want to is true - but the question is how much of a price they want to pay. By making things more difficult for them, we reduce the effectiveness of their surveillance capability. They then have to be more selective in their targeting, or divert resources from other projects, etc. As you note, we can't actually control it, just like we can't do it today, so the difference becomes «lots of mirrors, vulnerable to smaller attackers, but hard to coordinate MITM-ing» vs «fewer mirrors/CDN nodes, requires more effort from attackers, easier to MITM». I don't think it makes that much of a difference in terms of cost if the attacker has that many resources and is willing to expend the effort. It seems you disagree, and I don't really see us agreeing here, as it's a question of tradeoffs and you weigh your tradeoffs differently than I do. In my view the important question is not whether an attacker like the NSA has the capability to get what it wants when it really cares. The important question for Debian in this context is how much attacks would cost (not just in money but also in risk, effort, political clout, etc.). It seems to me that obtaining blanket logs about Debian users from a commercial CDN (or small set of CDNs) would be easy and cheap for the NSA and give significant and valuable information (what packages are installed and what security updates are done) about the vast majority of Debian users. Indeed I would be amazed if the NSA don't already routinely collect or scan all traffic to the big CDNs. In contrast, some parts of our current mirror networks are weak against monitoring but it is very easy for a user to (for example) select a mirror they think will be more trustworthy, and attacking our current mirror network in that way would involve strong-arming, subverting or hacking a much greater set of organisations and systems. I imagine the NSA would want to confine such compromises to those mirrors where they think they're not likely to get caught. I share Ingo's privacy concerns. I don't see a clear explanation of what the motivation is to switch to a commercial CDN. Can you clarify ? That will help us understand what we would be giving up if we decline to make this change. Ian. -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/21089.17961.750016.418...@chiark.greenend.org.uk
Re: Possibly moving Debian services to a CDN
Am 15.10.2013 um 15:51 schrieb Tollef Fog Heen tfh...@err.no: I'm fundamentally of the opinion that if the NSA or a similar organisation wants to track you and is willing to expend that effort on tracking you in particular, there is just about nothing you can do about it. As you note, we can't actually control it, just like we can't do it today, so the difference becomes «lots of mirrors, vulnerable to smaller attackers, but hard to coordinate MITM-ing» vs «fewer mirrors/CDN nodes, requires more effort from attackers, easier to MITM». I don't think it makes that much of a difference in terms of cost if the attacker has that many resources and is willing to expend the effort. It seems you disagree, and I don't really see us agreeing here, as it's a question of tradeoffs and you weigh your tradeoffs differently than I do. Yes. As a German and activist I value privacy as a fundamental right very high. I know that other people prefer ease of use and don't care whether their government can spy on them. As you said, it's (way too often) a trade-off each person has to choose. 2That's a valid point of you, thanks! The use of HTTPS should be encouraged, of course. How would HTTPS with a CDN work? I would believe that the CDN provider will use some kind of SSL proxy or SSL interception techniques. Otherwise you would have the same problems with managing HTTPS with the current mirror network. There are probably these possible ways: a) CDN provides an HTTPS entry point, but connects to the underlying mirror by plain HTTP. b) CDN uses DPI and SSL interception to break end-to-end encryption You upload your cert and key to the CDN, which then does HTTPS to the client. Whether they do HTTPS to the backend or not depends on the CDN. I know at least some do. Hmmm, uploading a SSL key to a CDN seems not the right thing to me. Anyway, I think the discussion about using a CDN is not about technical aspects, but it's a political debate that needs to b held and finally a political decision have to made whether Debian as a Free/Libre Software project/distribution wants to use a CDN and accept the risks that come with that or not. Right, there are technical hurdles we need to overcome. If we can't overcome those in a reasonable fashion, the whole exercise becomes pointless. Not quite. Even if some technical issues get solved, some general privacy issue will still stand. And in general it comes down to the question: what is freedom and how much freedom do we want? Debian as a free software project stands for high level of freedom, like the legendary discussions about non-free and sometimes even contrib showed. This is just a different (high) level of freedom: whether we want centralized services that has the power to control us (I'm exaggerating a little bit here ;)) or want we de-centralized services, even if they have some minor drawbacks in ease of use? It's basically the same question whether you want to use Windows/OSX or Linux. You can get the black boxes of Windows and OSX and have the ease of use, or you can use Linux because you value the freedom and security that comes with it. It's a basic question of freedom, not only a technical one. Personally I believe, that using a CDN would make live of DSA more easier (you wrote something in a different mail today that current CDN breaks on a weekly basis. Can you elaborate this, maybe on wiki.d.o?) and it might be easier for users. The breakage I'm seeing is from apt-get update failing on various hosts around the world. It's usually fine if it's retried 5s later. And yes, the goal here is to free up volunteer time as well as get a better experience for the end user. Yes, I understand your motivation and partially agree with it. :-) I would really appreciate a pro/con page about this topic in the wiki. I think a wiki page is better to follow the arguments than you can do on a mailing list. OTOH I have great privacy concerns of using a CDN. And when the current mirror network will still be maintained, where's the benefit for DSA and the users then at all? Having freedom of choice is always good, so I'd be fine with keeping current mirror network, but having a cdn.debian.org in parallel. When doing fresh installations people should be made aware of privacy concerns when using the CDN (like: Using a CDN might be easier and faster for you, but Debian doesn't control the CDN and cannot guarantee privacy and data protection). That implies we can guarantee privacy and data protection for other mirrors, which we can't. Yes, partially. Contrary to using a CDN where it's privacy and data protection is at large (either it is or it is not granted), a misbehaving single mirror would only violate privacy of those using that particular mirror. -- Ciao...// Fon: 0381-2744150 Ingo \X/ http://blog.windfluechter.net gpg pubkey:
Re: Possibly moving Debian services to a CDN
Am 18.10.2013 um 16:31 schrieb Ian Jackson ijack...@chiark.greenend.org.uk: Tollef Fog Heen writes (Re: Possibly moving Debian services to a CDN): I'm fundamentally of the opinion that if the NSA or a similar organisation wants to track you and is willing to expend that effort on tracking you in particular, there is just about nothing you can do about it. This is true, but largely irrelevant. That the NSA can get something if they really want to is true - but the question is how much of a price they want to pay. By making things more difficult for them, we reduce the effectiveness of their surveillance capability. They then have to be more selective in their targeting, or divert resources from other projects, etc. Exactly. It seems to me that obtaining blanket logs about Debian users from a commercial CDN (or small set of CDNs) would be easy and cheap for the NSA and give significant and valuable information (what packages are installed and what security updates are done) about the vast majority of Debian users. Indeed I would be amazed if the NSA don't already routinely collect or scan all traffic to the big CDNs. If not scanned by the NSA, it's like that your traffic will be monitored by GCHQ or others. Using near-by mirrors will reduce that risk of being monitored by foreign services. Keep in mind that in nearly all countries it is forbidden by law to spy on own citizens. And I'm sure that all big CDNs are already cooperating with secret services around the world. Additionally, as I already wrote in another mail, I think that Debian as a free software project has some sort of social responsibility to ensure peoples freedom and civil rights. Driving them to use a CDN that can be suited to limit their civil rights and freedom contradict the freedom Debian wants to achieve with its social contract to some degree. -- Ciao...// Fon: 0381-2744150 Ingo \X/ http://blog.windfluechter.net gpg pubkey: http://www.juergensmann.de/ij_public_key.asc -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/72796a55-0316-4f6a-b6d4-aa20b4c22...@2013.bluespice.org
Re: Possibly moving Debian services to a CDN
On Sun, Oct 13, 2013 at 08:44:56AM +0200, Tollef Fog Heen wrote: We appreciate feedback while we continue our investigation of CDNs. Hi Tollef, thanks for bringing this discussion to -project. I'm myself against switching to a CDN, but it might be due to a lack of information on my part, so I'd love if DSA could fill in my gaps. Debian is a Free Software project. I think that is what should drive our choices and nothing else. As such I'd hate seeing Debian moving, by default, to a content delivery solution that is made of proprietary software parts. That would be very bad for the Debian Project, as it would send the message that while we do create an entirely Free OS for others, we are fine with using proprietary software to do so (Mako has expressed this concept much better than I could possibly do in his Free Software Needs Free Tools essay http://mako.cc/writing/hill-free_tools.html ). In my mind, using a CDN made of non-free parts would be exactly the same as using a proprietary BTS, or replacing the DBMS that backs dak with Oracle Database. I do realize that most of the value of a CDN is not in its software parts. But I'm under the impression there is still quite a bit of software behind commercial CDN offerings. So my question is: would the CDN providers we're going to choose be able to ensure that the software parts behind their offerings to us are 100% Free Software? I don't think we have enough leverage to impose that. But if it is the case my non 100% Free Software concern above would certainly disappear. Another way of making my concern moot would be to use the CDN only as a non-default option that users should explicitly choose, and label it as some non official service. That would reduce the impression that the Debian Project endorses infrastructures that rely on non-free software. For instance, we could repurpose the cdn.debian.net name (assuming the current maintainer is fine with the idea) and present it as an option to our users. A corollary of this is that it would be difficult to use the CDN for things like www.debian.org (probably making moot many of the advantages you're looking for). An unrelated concern is that of technical independence. I do see a lot of value in Debian social experiment (quoting here a very nice way of calling it that Lucas has come up to) of trying to do almost everything by ourselves, from packaging to legal, from marketing to sysadm'ing. Of course it comes with a lot of drawbacks, and I don't think it is something rooted in Debian principles. But it is a very nice characteristic of our Project, and I think we should be very careful before giving it up, even if in only in specific areas. In your mail you've addressed the concern of excessive dependence on a *single* CDN provider, mentioning that you're looking into how easily switch from one provider to another. Would it be equally easy to get rid of the CDN --- or switch to a more home brew (set of) CDN(s) --- if things go awry? (FWIW, I'm not myself worried about dependency on money / hw coming from companies, I'm very well aware that Debian needs quite a bit of resources from companies. But that concern is very much mitigated by the diversity in donors, or at least by the fact that we can have that diversity. Technical dependency is IMHO much worse, because to get diversity there you need to have in place, beforehand, the needed abstraction layers.) Cheers. -- Stefano Zacchiroli . . . . . . . z...@upsilon.cc . . . . o . . . o . o Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o Former Debian Project Leader . . @zack on identi.ca . . o o o . . . o . « the first rule of tautology club is the first rule of tautology club » signature.asc Description: Digital signature
Re: Possibly moving Debian services to a CDN
On Wed, October 16, 2013 15:01, Stefano Zacchiroli wrote: I do realize that most of the value of a CDN is not in its software parts. But I'm under the impression there is still quite a bit of software behind commercial CDN offerings. So my question is: would the CDN providers we're going to choose be able to ensure that the software parts behind their offerings to us are 100% Free Software? It is of course a good target to strive for, but putting such a demand on a service is probably not fair, because our current distribution system also does not run on 100% Free Software. We do not know what software our mirrors are running (we can make a guess that there will be Debian hosts in there, but just as well we can be quite sure there will be some Solaris or commercial Linux variants) so it's just as opaque as a commercial CDN offering would be. The principled point that everything Debian does should involve 100% Free Software is not the status quo nor is it attainable. We will deliver bits using equipment running Cisco IOS, and we will not have the source of the bank software that processes our donations. The question is therefore: do we consider such CDN services to be more like network- or financial services, which we source externally, or is it a core aspect of developing Debian? Cheers, Thijs -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/459f0e787bedca53b8a40256c4050c1f.squir...@aphrodite.kinkhorst.nl
Re: Possibly moving Debian services to a CDN
On Wed, Oct 16, 2013 at 03:40:48PM +0200, Thijs Kinkhorst wrote: It is of course a good target to strive for, but putting such a demand on a service is probably not fair, because our current distribution system also does not run on 100% Free Software. We do not know what software our mirrors are running (we can make a guess that there will be Debian hosts in there, but just as well we can be quite sure there will be some Solaris or commercial Linux variants) so it's just as opaque as a commercial CDN offering would be. I disagree with that, both on a principle basis and on a practical one. Right now, as a project, we do not take any stance on the free-ness of our mirror network. It's some sort of don't ask, don't tell regime. In practical terms I'm myself convinced that many mirrors are run entirely on Free Software (see about network equipment below), and I can guarantee that was the case for mirrors I've been running myself in the past. Switching *by default* to a CDN could make the situation much better (if the offer is 100% Free Software, which is unfortunately unlikely) or much worse (the most likely case). If the situation is going to get much worse, I frankly prefer the status quo. BTW, I forgot to add a very important note to the previous mail. I realize that switching to a CDN could make things much better for DSA, and that's why it pains me to object to the idea by being someone that is not contributing in anyway to the DSA team. I do that only because I think we're here on a topic where what I believe to be Debian principles are at stake. The principled point that everything Debian does should involve 100% Free Software is not the status quo nor is it attainable. We will deliver bits using equipment running Cisco IOS, and we will not have the source of the bank software that processes our donations. The question is therefore: do we consider such CDN services to be more like network- or financial services, which we source externally, or is it a core aspect of developing Debian? Right, that's indeed the question. And I realize that different people will place their bars at different levels. My bar is that content delivery is indeed a core aspect for a project producing a Free Software distribution. YMMV, and I've no desire of trying to convince otherwise people who place their bars differently. But this thread was about opinions on the idea; this is mine :-) -- Stefano Zacchiroli . . . . . . . z...@upsilon.cc . . . . o . . . o . o Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o Former Debian Project Leader . . @zack on identi.ca . . o o o . . . o . « the first rule of tautology club is the first rule of tautology club » signature.asc Description: Digital signature
Re: Possibly moving Debian services to a CDN
Hi, (cc debian-mirr...@lists.debian.org, the list where such discussions can happen) On Sun, Oct 13, 2013 at 09:00:04PM -0700, Russ Allbery wrote: Joey Hess jo...@debian.org writes: But apparently not one solved by free software included in Debian. Perhaps it's worth avoiding using it if that will help encourage the development of libre alternatives. CDNs aren't really software problems. They're infrastructure and network peering problems. I think all the software required is in Debian, but the data centers, peering arragements, route advertisements, and so forth are things that only make sense to do at a larger scale than a single project. We already have a network of almost 400 packages mirrors around the world. Using http.d.n, provides the CDN layer (not as much as optimal as anycast), so we don't need to sort ourselves peering issues etc. We can do a home-brewed CDN -- that, after all, is what the various services referenced in the original message are. But one of the commercial CDNs will have better performance and better load distribution than one can do with software-only solutions without the peering setup and data center distribution. * Commercial CDN have no knowledge of debian archive datamodel and constraints, which leads to inconsistency (and consequently hash sum dismatch). Or this would imply just disabling caching for problematic files, or change apt/dak so that the archive is less impacted by synchro atomicity issues. * My own experience is different, http.d.n redirects to ftp2.fr, which i got 10,2Mo/s, while cloudfront.d.n (Amazon) gives 5Mo/s. -- Simon Paillard mirr...@debian.org team member -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131016182336.gq5...@mraw.org
Re: Possibly moving Debian services to a CDN
This one time, at band camp, Simon Paillard said: Hi, We already have a network of almost 400 packages mirrors around the world. Using http.d.n, provides the CDN layer (not as much as optimal as anycast), so we don't need to sort ourselves peering issues etc. The mirrors do a very good job of being near to our users in most cases (we have had some difficulty with things like security mirror coverage, but that's not your issue). I know many people are happy with http.d.n, but you have to understand that it's not the same thing at all as a CDN - it's a single host, and it's in a single location. Time to first byte will still suffer if you're coming from Australia. We can do a home-brewed CDN -- that, after all, is what the various services referenced in the original message are. But one of the commercial CDNs will have better performance and better load distribution than one can do with software-only solutions without the peering setup and data center distribution. * Commercial CDN have no knowledge of debian archive datamodel and constraints, which leads to inconsistency (and consequently hash sum dismatch). Or this would imply just disabling caching for problematic files, or change apt/dak so that the archive is less impacted by synchro atomicity issues. Or do things like set long cache time on all files, and invalidate the cache on the metadata files once sync is complete. There are ways to solve those issues, just as there are ways to solve them when we're transferring them with rsync. * My own experience is different, http.d.n redirects to ftp2.fr, which i got 10,2Mo/s, while cloudfront.d.n (Amazon) gives 5Mo/s. This is an important data point. If cloudfront is outperformed by our current mirror infrastructure, it becomes less appealing. After all, we're trying to make things better, not worse. Cheers, -- - | ,''`.Stephen Gran | | : :' :sg...@debian.org | | `. `'Debian user, admin, and developer | |`- http://www.debian.org | - signature.asc Description: Digital signature
Re: Possibly moving Debian services to a CDN
Hi, On Wed, Oct 16, 2013 at 07:54:04PM +0100, Stephen Gran wrote: This one time, at band camp, Simon Paillard said: We already have a network of almost 400 packages mirrors around the world. Using http.d.n, provides the CDN layer (not as much as optimal as anycast), so we don't need to sort ourselves peering issues etc. The mirrors do a very good job of being near to our users in most cases (we have had some difficulty with things like security mirror coverage, but that's not your issue). I know many people are happy with http.d.n, but you have to understand that it's not the same thing at all as a CDN - it's a single host, and it's in a single location. Time to first byte will still suffer if you're coming from Australia. Obviously, it's due to current unsponsored deployment. * For a package of several kB, the first bytes (actually ~800B, measured) are not meaningful * Our (mirrors@d.o) is to have http.d.n made official so that we can have an instance like on each continent, then use GeoDNS, like security.d.o mirrors, to achieve 2 goals: avoid SPOF, be closer to users. -- Simon Paillard mirrors team member -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131016190108.gt5...@mraw.org
Re: Possibly moving Debian services to a CDN
]] Nikolaus Rath - You can use an IP anonymizing service such as Tor. Are you suggesting to download debian packages over tor? Last time I used it, I got about 25 kB/s of bandwidth. But even if that has changed, I'm pretty sure the tor network isn't intended for bulk transfer of the debian archive... «such as». They're not the only one, and I wasn't suggesting running a full mirror over it. Downloading your daily package updates over it is likely to be slower than your regular network connection, but when I just tested it was at the level of some megabits. If you're actually seriously concerned about privacy and worried about government-level organisations using package download data for tracking you, you should use something like tor, yes. I personally think that's overly paranoid, but I'm not going to tell people they're not allowed to turn their paranoia dial all the way up. (Using paranoia here for lack of a better word; no disrespect meant.) -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87wqle7ito@xoog.err.no
Re: Possibly moving Debian services to a CDN
]] Ingo Jürgensmann Am 14.10.2013 um 07:29 schrieb Tollef Fog Heen tfh...@err.no: - I would like us to have agreements with any donors that they're not allowed to use the information for anything but operational issues. We can't tell them not to log (because that's really hard on a technical level), but we can restrict what they can do with the logs. True. You can request agreements, but as the whole NSA affair is showing: it doesn't matter when it comes down to NSA Co. There are secret courts with secret decisions and National Security Letters for silencing the providers, although internal agreements like Safe Harbor do exist. So, whereas agreements can be made, there will be no way for Debian to control whether they are being held or not. I'm fundamentally of the opinion that if the NSA or a similar organisation wants to track you and is willing to expend that effort on tracking you in particular, there is just about nothing you can do about it. As you note, we can't actually control it, just like we can't do it today, so the difference becomes «lots of mirrors, vulnerable to smaller attackers, but hard to coordinate MITM-ing» vs «fewer mirrors/CDN nodes, requires more effort from attackers, easier to MITM». I don't think it makes that much of a difference in terms of cost if the attacker has that many resources and is willing to expend the effort. It seems you disagree, and I don't really see us agreeing here, as it's a question of tradeoffs and you weigh your tradeoffs differently than I do. 2) Integrity concerns: although Debian uses signed package lists and hashed packages, using a CDN would raise the chances that there might be attack vectors by manipulating the traffic. Maybe not be the will of the running company, but there are other groups that might have interest and the power to intercept traffic and manipulating it. This is, of course, also true to current mirror sites, but a centralized CDN will be more convenient to such kind of attackers. Given we don't use HTTPs and such today, you don't know if the traffic is actually going to the mirror you think it's going to, so this isn't really different from today. With a CDN we could actually push more of the traffic to HTTPS if we wanted. This isn't feasible with today's mirror network. That's a valid point of you, thanks! The use of HTTPS should be encouraged, of course. How would HTTPS with a CDN work? I would believe that the CDN provider will use some kind of SSL proxy or SSL interception techniques. Otherwise you would have the same problems with managing HTTPS with the current mirror network. There are probably these possible ways: a) CDN provides an HTTPS entry point, but connects to the underlying mirror by plain HTTP. b) CDN uses DPI and SSL interception to break end-to-end encryption You upload your cert and key to the CDN, which then does HTTPS to the client. Whether they do HTTPS to the backend or not depends on the CDN. I know at least some do. [...] Anyway, I think the discussion about using a CDN is not about technical aspects, but it's a political debate that needs to b held and finally a political decision have to made whether Debian as a Free/Libre Software project/distribution wants to use a CDN and accept the risks that come with that or not. Right, there are technical hurdles we need to overcome. If we can't overcome those in a reasonable fashion, the whole exercise becomes pointless. Personally I believe, that using a CDN would make live of DSA more easier (you wrote something in a different mail today that current CDN breaks on a weekly basis. Can you elaborate this, maybe on wiki.d.o?) and it might be easier for users. The breakage I'm seeing is from apt-get update failing on various hosts around the world. It's usually fine if it's retried 5s later. And yes, the goal here is to free up volunteer time as well as get a better experience for the end user. OTOH I have great privacy concerns of using a CDN. And when the current mirror network will still be maintained, where's the benefit for DSA and the users then at all? Having freedom of choice is always good, so I'd be fine with keeping current mirror network, but having a cdn.debian.org in parallel. When doing fresh installations people should be made aware of privacy concerns when using the CDN (like: Using a CDN might be easier and faster for you, but Debian doesn't control the CDN and cannot guarantee privacy and data protection). That implies we can guarantee privacy and data protection for other mirrors, which we can't. -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87siw27ibt@xoog.err.no
Re: Possibly moving Debian services to a CDN
This one time, at band camp, Joey Hess said: Ultimately, we are of the opinion that the content delivery problem is a solved one But apparently not one solved by free software included in Debian. Perhaps it's worth avoiding using it if that will help encourage the development of libre alternatives. As others have pointed out, this isn't the problem at all. However, if this is a topic that interests you, you could have a look at the software requirements for mirror operators that we currently have. Cheers, -- - | ,''`.Stephen Gran | | : :' :sg...@debian.org | | `. `'Debian user, admin, and developer | |`- http://www.debian.org | - signature.asc Description: Digital signature
Re: Possibly moving Debian services to a CDN
Am 14.10.2013 um 07:29 schrieb Tollef Fog Heen tfh...@err.no: 1) Privacy concerns: Debian would deliver much more data to business companies than necessary. Keep in mind that personalized data is one of the most valuable things to data miners. Currently I choose one mirror site to pull my packages from. I can freely choose that mirror on basis of location, bandwidth, personal likes or, let's say, privacy reasons because I know that this specific mirror doesn't log my IPs. When using a CDN, at least in that way I understood your proposal, I'm not free to choose anymore. The company running that CDN will obtain all of data like how many machines are behind a subnet or IP, what kind of machines (intel, sparc, powerpc, m68k, ...) and might know if I forget to update a machine (security). This is absolutely a valid concern. I have a few mitigation strategies and one observation: - You can still run your own mirror. We need that ourselves and like I wrote in the initial email, we need to find a way that keeps rsync working. Yeah, running my own mirror is an option for me. I did run a backports.org mirror in the past and was thinking of expanding it to a full-blown mirror. But that, of course, is not an option for Joe Average User. - You can use an IP anonymizing service such as Tor. We know that NSA and GCHQ are running Tor exit nodes. And yet they don't have the capacity to track all TOR traffic globally, but only, with great cost/effort, single users can be tracked. Apparently this might just be a matter of time, competing with counter measures from the TOR project. - You can use a local proxy that hides the details of how many nodes, etc. you have. There are ways to distinguish nodes/users behind a proxy by using fingerprinting, latency checks and other stuff. Yes, I know it will be rather unlikely that someone will do that for Debian updates, but until some weeks ago I couldn't think of secret services that will do a Full Take of intercontinental sea cables like the GCHQ is doing. The lesson from that is: if a secret service like NSA or GCHQ want to know something, no effort is too big. All the Debian project can do, is to drive the costs high for such kind of surveillance. Or to put it other way around: Debian should avoid it to make it more easy for them. - I would like us to have agreements with any donors that they're not allowed to use the information for anything but operational issues. We can't tell them not to log (because that's really hard on a technical level), but we can restrict what they can do with the logs. True. You can request agreements, but as the whole NSA affair is showing: it doesn't matter when it comes down to NSA Co. There are secret courts with secret decisions and National Security Letters for silencing the providers, although internal agreements like Safe Harbor do exist. So, whereas agreements can be made, there will be no way for Debian to control whether they are being held or not. The observation is that we currently don't have any such control over mirror operators. They are, AFAIK, free to use whatever information they collect for whatever purpose they would like. Granted. That's maybe something Debian can address as well in the future. But having many mirror operators result in: - higher costs for controlling them - each mirror operator only sees its own traffic - each mirror site will be subject to the specific law in that country (higher data protection level in Germany for example) Well, I think you got the point already... ;-) 2) Integrity concerns: although Debian uses signed package lists and hashed packages, using a CDN would raise the chances that there might be attack vectors by manipulating the traffic. Maybe not be the will of the running company, but there are other groups that might have interest and the power to intercept traffic and manipulating it. This is, of course, also true to current mirror sites, but a centralized CDN will be more convenient to such kind of attackers. Given we don't use HTTPs and such today, you don't know if the traffic is actually going to the mirror you think it's going to, so this isn't really different from today. With a CDN we could actually push more of the traffic to HTTPS if we wanted. This isn't feasible with today's mirror network. That's a valid point of you, thanks! The use of HTTPS should be encouraged, of course. How would HTTPS with a CDN work? I would believe that the CDN provider will use some kind of SSL proxy or SSL interception techniques. Otherwise you would have the same problems with managing HTTPS with the current mirror network. There are probably these possible ways: a) CDN provides an HTTPS entry point, but connects to the underlying mirror by plain HTTP. b) CDN uses DPI and SSL interception to break end-to-end encryption For example using Cisco WAAS is a nice and decent way to minimize TCP traffic on a low
Re: Possibly moving Debian services to a CDN
Tollef Fog Heen tfh...@err.no writes: ... Nobody has suggested removing the mirror network. What's being discussed is using a CDN for some .d.o services. That was certainly not clear from your original post. I certainly read you as suggesting that some services could be moved to third-party CDN(s), with an eye to moving ftp.debian.org there to, with the implication that the mirror network would then become mostly redundant. I would suggest that that's the scenario that is causing people to argue against you, so if that's not what you were suggesting, perhaps you should try to express your plans again to get the discussion closer to what you think you were suggesting. Cheers, Phil. -- |)| Philip Hands [+44 (0)20 8530 9560]http://www.hands.com/ |-| HANDS.COM Ltd.http://ftp.uk.debian.org/ |(| 10 Onslow Gardens, South Woodford, London E18 1NE ENGLAND pgpNV9vbU3taC.pgp Description: PGP signature
Re: Possibly moving Debian services to a CDN
]] Philip Hands Tollef Fog Heen tfh...@err.no writes: ... Nobody has suggested removing the mirror network. What's being discussed is using a CDN for some .d.o services. That was certainly not clear from your original post. I certainly read you as suggesting that some services could be moved to third-party CDN(s), with an eye to moving ftp.debian.org there to, with the implication that the mirror network would then become mostly redundant. «Become redundant» is not the same as being removed, though. It would initially be something we ran alongside the regular mirror network (anything else would be crazy for what I think are obvious reasons). If our experiences are then positive, we might want to stop relying on the mirror network in say, d-i, but there's not central planning committee shutting down any mirrors. Local mirrors choose whether they want to carry Debian or not, and I suspect many of them will want to use the resources for other things if the usage falls below a threshold. Whether that actually happens or not amounts to predicting the future, something I'm not going to try to do. Does that make it clearer, or is it still confusing? -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87txgk5839@qurzaw.varnish-software.com
Re: Possibly moving Debian services to a CDN
On Mon, 14 Oct 2013, Paul Wise wrote: On Mon, Oct 14, 2013 at 2:16 AM, Joey Hess wrote: But apparently not one solved by free software included in Debian. Perhaps it's worth avoiding using it if that will help encourage the development of libre alternatives. I guess the hardest part of the problem is logistics to get machines and disks into the network of every ISP in the world and put those on an anycast IP address. While building the anycast infrasctructure is quite easy as long as you have the ASN and IP blocks, getting it deployed is anything but trivial... and the problems are _not_ technical. It is not just logistics: getting the anycast blocks routed and announced properly (remember, most of those will be global nodes, not reduced-visibility nodes!) is a very large annoyance. Let's stick to something DNSSEC-based, please. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131014155546.gb29...@khazad-dum.debian.net
Re: Possibly moving Debian services to a CDN
Tollef Fog Heen tfh...@err.no writes: 1) Privacy concerns: Debian would deliver much more data to business companies than necessary. Keep in mind that personalized data is one of the most valuable things to data miners. Currently I choose one mirror site to pull my packages from. I can freely choose that mirror on basis of location, bandwidth, personal likes or, let's say, privacy reasons because I know that this specific mirror doesn't log my IPs. When using a CDN, at least in that way I understood your proposal, I'm not free to choose anymore. The company running that CDN will obtain all of data like how many machines are behind a subnet or IP, what kind of machines (intel, sparc, powerpc, m68k, ...) and might know if I forget to update a machine (security). This is absolutely a valid concern. I have a few mitigation strategies and one observation: - You can still run your own mirror. We need that ourselves and like I wrote in the initial email, we need to find a way that keeps rsync working. - You can use an IP anonymizing service such as Tor. Are you suggesting to download debian packages over tor? Last time I used it, I got about 25 kB/s of bandwidth. But even if that has changed, I'm pretty sure the tor network isn't intended for bulk transfer of the debian archive... Best, Nikolaus -- Encrypted emails preferred. PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C »Time flies like an arrow, fruit flies like a Banana.« -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87mwmbiqa4@rath.org
Re: Possibly moving Debian services to a CDN
On Mon, Oct 14, 2013 at 1:36 AM, Tollef Fog Heen tfh...@err.no wrote: ]] Paul Wise About the archive mirrors, some reworded thoughts from the DPL IRC channel when this came up a few days ago: pabs [...] I think the current state of affairs is fine; I don't believe you're one of the person who is doing the legwork in maintaining any of the CDNs we're currently running, so how would you know that? Because it's not visibly broken (most of the time)? I see it breaking on a weekly basis, if not more often. [...] Removing the mirror network won't be possible anyway, people are still going to create mirrors, especially ISPs will for their customers; due to quotas and distant mirrors being much slower. Nobody has suggested removing the mirror network. What's being discussed is using a CDN for some .d.o services. If somebody wants to continue running their mirror they will of course be free to do so. bguptaNot all CDNs support IPv6. We will want to use CDNs that do support IPv6. It's one of the technical bits that need to fall into place before we will want to switch. I would rather expand the mirror network. Does that mean you're volunteering for the task of doing this and maintaining the various existing CDNs? Yes, the hardest part of this is Debian itself would need to basically build its own distribution network. Whether this is something I would personal volunteer to help with, if we decide not to go the route of using CDNs, the answer is yes. However, I don't think in this case, whether or not I would volunteer for this should be a factor in the decision, and will share my more fleshed out point of view than the couple of lines I shared on IRC. In believe the considerations here fall into a number of areas: 1) Technical. e.g. - CNAMEs, IPv6 support, security concerns related to caching, and support for protocols other than HTTP under the same DNS name as the HTTP services (rsync, SMTP, etc.). I believe these are the biggest challenges that Debian would find moving to using CDNs, but trust that if the DSA is seriously considering this, they know what these issues are and would address them before making any changes. 2) DFSG concerns - These blackbox services may or may not be built using Free Software. We really have no way of knowing for sure, since we would be abdicating the actual responsibility of running software. That said, if our end users and Debian itself are not required to use proprietary protocols or tools, I think this issue isn't as major of a concern that it might seem, and believe that a CDN would be classified as a network service, as defined by the FSF. RMS has an interesting blog post on this topic [1], that I largely agree with. Although there are other issues raised, I don't believe they would impact our decision whether or not to use a CDN, but I encourage everyone to read this before making any decisions. My take on this, is that using a CDN does not violate the DFSG, but defer to more experienced hands on this particular issue. 3) Privacy - There have been issues raised about logging by these third party CDNs. My sense is that if the CDNs do not replace our mirror network, and people are free to continue to use existing mirrors, my take is this change may not introduce new privacy concerns. 4) Commercial nature of CDN services - As Tollef correctly points out, Debian does rely on monetary and in-kind donations from a number of for-profit enterprises. A CDN does not change this, so I think this particular point should not be a major factor in our decision. 5) Relationships - We do have relationships with the members of our mirror network. An open question remains, Would migrating to CDN services damage these relationships? I suspect that if we allow the mirror network to remain in place, and we communicate a solid case for this change to our mirror network partners, prior to making it, the majority would likely support the change. 6) Single point of failure - While technically any properly designed CDN should be a distributed system resilient to single failures, the fact remains, that a CDN as a whole, if it were to be compromised or have a systematic failure (technical or other), could be catastrophic for Debian, and our users. That said, I believe Toleff's proposal to work with multiple CDNs, is likely a good way to address this concern, which could further be mitigated by keeping our mirror network in place. One thing to bear in mind, is that if any single CDN makes accommodations/changes for our unique technical needs, this means that it is less likely that these CDNs would be easily interchangeable. One way to work around this, is to make sure from the start we are working with at least two networks. 7) User experience - Does the use of CDNs improve the end user experience? From my understanding and experience working with CDNs, the answer is almost certainly yes, but defer to the DSA to confirm this is in fact the case. Obviously, if the
Possibly moving Debian services to a CDN
Dear Project, The System Administration Team (DSA) are considering moving some of the static hosting that Debian currently provides from our infrastructure to one or more CDNs. We have received feedback indicating that a broader discussion is desired. Debian has, for over a decade, operated its own form of a content delivery network (multiple variants, actually) by leveraging our own infrastructure as master sources and community-provided infrastructure (primarily from universities and regional network providers) for local distribution. This «proto-CDN» has required users to select the mirror 'closest' to them. Recent efforts (http.debian.net, cdn.debian.net) have attempted to alleviate the configuration challenge of pre-selecting a mirror node. In the commercial world, this challenge has been addressed through a mix of anycast DNS redirection and geo/BGP-based DNS views to local distribution nodes hosted at ISPs. Akamai is the best known CDN but other significant players include Amazon and Fastly and a host of other more specialised CDNs for example for video. In the DSA team's view, CDNs have become sufficiently mature for Debian to consider leveraging external service providers for our CDN needs. We have approached several providers and they have agreed, in principle, to sponsor bandwidth and storage for Debian's CDN needs. This allows us to consider combining the efforts of http.debian.net, cdn.debian.net, static.debian.org and the mirror network under a single effort to provide our users with the most transparent access to Debian public resources as possible. We appreciate that there are some sensitivities regarding the use of commercial service providers and/or our reliance upon them. Our mitigation strategy is to utilize multiple CDN service providers so that we can survive the loss of any single one (with quick change-over via DNS record modification). The concern regarding commercial entities support Debian activity is somewhat misdirected given our reliance on sponsors (often commercial) to support Debian and DebConf. For many years, Debian survived on the good graces of HP, for example, who provided cash and in-kind donations. Ultimately, we are of the opinion that the content delivery problem is a solved one and it behooves us to investigate whether CDNs can benefit Debian. There are several technical challenges that we must overcome. In particular, CDN offerings are very focussed on HTTP/HTTPS while Debian has a strong reliance (and strong desire to continue to use) other protocols such as rsync. Also, since CDNs primarily utilize CNAME records, they are incompatible with email service for that particular domain name. The address t...@security.debian.org is a good example here. In addition, using a CNAME means all services are redirected to the CDN, not just HTTP, which is incompatible with rsync. We are working with CDN service providers to find a resolution to these technical challenges and we hope to be able to report successful resolution in the near future. The services that we consider would benefit from a move to a CDN are: - ftp.debian.org - www.debian.org - security.debian.org - the various bits and bobs that are currently hosted on static.debian.org There has been concerns that switching to a CDN would harm our existing relationships with mirror operators and make it impossible to go back if we later wanted to do so. The ftp mirror network is one of the most important mirror networks, so we wouldn't have to start with that. We could start with (for instance) www.debian.org and only later move the actual package mirrors over once we are confident CDNs are not a passing fad. It will also take time to coordinate switching to a CDN with all the country operators, we do not wish to undermine the existing relationships or upset anybody needlessly. Assuming that our experiences are positive, we don't believe it will be interesting to go back, and even if one CDN folds, they are fast becoming a commodity so we think switching to another will be fairly easy. We appreciate feedback while we continue our investigation of CDNs. Thanks for your interest, -- Tollef Fog Heen for the Debian System Administration team -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/m24n8lacuv@rahvafeir.err.no
Re: Possibly moving Debian services to a CDN
Tollef Fog Heen: The System Administration Team (DSA) are considering moving some of the static hosting that Debian currently provides from our infrastructure to one or more CDNs. We have received feedback indicating that a broader discussion is desired. Debian has, for over a decade, operated its own form of a content delivery network (multiple variants, actually) by leveraging our own infrastructure as master sources and community-provided infrastructure (primarily from universities and regional network providers) for local distribution. Obviously, if DSA are thinking of replacing this infrastructure, it means that it is problematic. Could we get a summary of the problems with our current infrastructure? The following is the only one I can identify in your email: This «proto-CDN» has required users to select the mirror 'closest' to them. Recent efforts (http.debian.net, cdn.debian.net) have attempted to alleviate the configuration challenge of pre-selecting a mirror node. Are these recent efforts not solving the problem of choosing the best mirror? If not, why? -- Lunar.''`. lu...@debian.org: :Ⓐ : # apt-get install anarchism `. `'` `- signature.asc Description: Digital signature
Re: Possibly moving Debian services to a CDN
On Sun, Oct 13, 2013 at 08:44:56AM +0200, Tollef Fog Heen wrote: Dear Project, The System Administration Team (DSA) are considering moving some of the static hosting that Debian currently provides from our infrastructure to one or more CDNs. We have received feedback indicating that a broader discussion is desired. Debian has, for over a decade, operated its own form of a content delivery network (multiple variants, actually) by leveraging our own infrastructure as master sources and community-provided infrastructure (primarily from universities and regional network providers) for local distribution. It works - tell anyone to use ftp.[country name].debian.org and it just works. If ftp.[mycountry].debian.org is down, use ftp.[neighbour country] .debian.org - and, crucially, it's all under one debian.org domain. If managers/software licensing mavens/project funding authorities etc. question where your software is actualy from - it's from Debian themselves. That's a really, really good thing. We appreciate feedback while we continue our investigation of CDNs. Thanks for your interest, -- Tollef Fog Heen for the Debian System Administration team This is (potentially) good news for laptop/desktop users: instant access from closest mirrors. This is also an increasing trend: Firefox, Raspbian, Archlinux (I think) are all CDN served. If you run behind restrictive firewall policies / in corporate land, it's not nearly so hot. Static long lasting mirrors are really useful here when you have to ask your network admin. to unblock firewalls for each IP address. If you want to configure 10 servers, say, or to build repeated groups of servers over a long time, it's good to have consistency and the existing network provides this. If the trend globally is to CDNs, however, it will be hard to buck the trend for the long term. Just my 0.02€ Andy signature.asc Description: Digital signature
Re: Possibly moving Debian services to a CDN
Interesting idea. If the Debian website is served to users directly from non-free software, and so is the archive, I have to wonder what would be the FSF's reaction to this. It seems to have some potential to burn bridges that I'd otherwise hope would get mended. Ultimately, we are of the opinion that the content delivery problem is a solved one But apparently not one solved by free software included in Debian. Perhaps it's worth avoiding using it if that will help encourage the development of libre alternatives. -- see shy jo signature.asc Description: Digital signature
Re: Possibly moving Debian services to a CDN
Am 13.10.2013 um 08:44 schrieb Tollef Fog Heen tfh...@err.no: The System Administration Team (DSA) are considering moving some of the static hosting that Debian currently provides from our infrastructure to one or more CDNs. We have received feedback indicating that a broader discussion is desired. [...] We appreciate feedback while we continue our investigation of CDNs. Although I understand that there will be some benefits of using a CDN, I see some issues as well: 1) Privacy concerns: Debian would deliver much more data to business companies than necessary. Keep in mind that personalized data is one of the most valuable things to data miners. Currently I choose one mirror site to pull my packages from. I can freely choose that mirror on basis of location, bandwidth, personal likes or, let's say, privacy reasons because I know that this specific mirror doesn't log my IPs. When using a CDN, at least in that way I understood your proposal, I'm not free to choose anymore. The company running that CDN will obtain all of data like how many machines are behind a subnet or IP, what kind of machines (intel, sparc, powerpc, m68k, ...) and might know if I forget to update a machine (security). 2) Integrity concerns: although Debian uses signed package lists and hashed packages, using a CDN would raise the chances that there might be attack vectors by manipulating the traffic. Maybe not be the will of the running company, but there are other groups that might have interest and the power to intercept traffic and manipulating it. This is, of course, also true to current mirror sites, but a centralized CDN will be more convenient to such kind of attackers. 3) Surveillance concerns: together with 1) and 2) goes this one... Using a CDN would make it easier to secret services to collect data, because they have a single point where they can get all wanted data from instead of monitoring several providers and connections. 4) Dependency concerns: as a project Debian should be as independent as possible. Using a CDN provider will create a big dependency to a specific company, although we might be able to shift companies from time to time. Using multiple CDN providers will mitigate that concern a little bit, but only to a certain degree. Having too many CDN providers will be as difficult to handle as now the many FTP mirror donators. So, there's some trade-off anyway. So, after all my strongest concerns are 1), 2) and 3), of course. I'm not a big fan of centralized solutions, but more a great friend of de-centralised ones. Having monocultures is always a bad thing and using large CDNs is driving that kind of monoculture. Diversity is enrichment and should be chosen whenever possible. -- Ciao...// Fon: 0381-2744150 Ingo \X/ http://blog.windfluechter.net gpg pubkey: http://www.juergensmann.de/ij_public_key.asc -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/2a773832-09f2-4adb-9b10-2a554b6dd...@2013.bluespice.org
Re: Possibly moving Debian services to a CDN
About the archive mirrors, some reworded thoughts from the DPL IRC channel when this came up a few days ago: pabs I would suggest forwarding/bouncing this mail to the debian-mirrors list. I think the current state of affairs is fine; some CDNs in use (cloudfront.d.n), some normal mirrors, mirror.d.o/http.d.n checks all of them for freshness. Removing the mirror network won't be possible anyway, people are still going to create mirrors, especially ISPs will for their customers; due to quotas and distant mirrors being much slower. bguptaNot all CDNs support IPv6. I would rather expand the mirror network. There is something appealing to me about the decentralized nature of a volunteer set of mirrors. If one provider was our CDN provider, and they didn't want to keep donating the service.. that could be problematic. -- bye, pabs http://wiki.debian.org/PaulWise -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/caktje6fi1fy8tq-19gezp71wq6wrefkgojtkpn-webtxzon...@mail.gmail.com
Re: Possibly moving Debian services to a CDN
On Mon, Oct 14, 2013 at 2:16 AM, Joey Hess wrote: But apparently not one solved by free software included in Debian. Perhaps it's worth avoiding using it if that will help encourage the development of libre alternatives. I guess the hardest part of the problem is logistics to get machines and disks into the network of every ISP in the world and put those on an anycast IP address. -- bye, pabs http://wiki.debian.org/PaulWise -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/caktje6fp5zrmjvg1r2qmzypq1vtz9o8ibjkhzrddszvshqw...@mail.gmail.com
Re: Possibly moving Debian services to a CDN
Joey Hess jo...@debian.org writes: Ultimately, we are of the opinion that the content delivery problem is a solved one But apparently not one solved by free software included in Debian. Perhaps it's worth avoiding using it if that will help encourage the development of libre alternatives. CDNs aren't really software problems. They're infrastructure and network peering problems. I think all the software required is in Debian, but the data centers, peering arragements, route advertisements, and so forth are things that only make sense to do at a larger scale than a single project. We can do a home-brewed CDN -- that, after all, is what the various services referenced in the original message are. But one of the commercial CDNs will have better performance and better load distribution than one can do with software-only solutions without the peering setup and data center distribution. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87zjqcmri3@windlord.stanford.edu
Re: Possibly moving Debian services to a CDN
]] Russ Allbery Joey Hess jo...@debian.org writes: Ultimately, we are of the opinion that the content delivery problem is a solved one But apparently not one solved by free software included in Debian. Perhaps it's worth avoiding using it if that will help encourage the development of libre alternatives. Quite a few of the CDNs use free software. Both Varnish and squid are used, for instance. CDNs aren't really software problems. They're infrastructure and network peering problems. I think all the software required is in Debian, but the data centers, peering arragements, route advertisements, and so forth are things that only make sense to do at a larger scale than a single project. This is the main reason. The commercial CDNs are in a position where they have more manpower and can scale better than we can because of volume. We can do a home-brewed CDN -- that, after all, is what the various services referenced in the original message are. But one of the commercial CDNs will have better performance and better load distribution than one can do with software-only solutions without the peering setup and data center distribution. We are already running CDNs, multiple of them: The mirror network, the security archive network, the web pages and a few more. What we don't have is the manpower and the infrastructure to run and maintain this as well as a CDN that does this as its primary business. -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/m2txgk8mrl@rahvafeir.err.no
Re: Possibly moving Debian services to a CDN
]] Ingo Jürgensmann 1) Privacy concerns: Debian would deliver much more data to business companies than necessary. Keep in mind that personalized data is one of the most valuable things to data miners. Currently I choose one mirror site to pull my packages from. I can freely choose that mirror on basis of location, bandwidth, personal likes or, let's say, privacy reasons because I know that this specific mirror doesn't log my IPs. When using a CDN, at least in that way I understood your proposal, I'm not free to choose anymore. The company running that CDN will obtain all of data like how many machines are behind a subnet or IP, what kind of machines (intel, sparc, powerpc, m68k, ...) and might know if I forget to update a machine (security). This is absolutely a valid concern. I have a few mitigation strategies and one observation: - You can still run your own mirror. We need that ourselves and like I wrote in the initial email, we need to find a way that keeps rsync working. - You can use an IP anonymizing service such as Tor. - You can use a local proxy that hides the details of how many nodes, etc. you have. - I would like us to have agreements with any donors that they're not allowed to use the information for anything but operational issues. We can't tell them not to log (because that's really hard on a technical level), but we can restrict what they can do with the logs. The observation is that we currently don't have any such control over mirror operators. They are, AFAIK, free to use whatever information they collect for whatever purpose they would like. 2) Integrity concerns: although Debian uses signed package lists and hashed packages, using a CDN would raise the chances that there might be attack vectors by manipulating the traffic. Maybe not be the will of the running company, but there are other groups that might have interest and the power to intercept traffic and manipulating it. This is, of course, also true to current mirror sites, but a centralized CDN will be more convenient to such kind of attackers. Given we don't use HTTPs and such today, you don't know if the traffic is actually going to the mirror you think it's going to, so this isn't really different from today. With a CDN we could actually push more of the traffic to HTTPS if we wanted. This isn't feasible with today's mirror network. 3) Surveillance concerns: together with 1) and 2) goes this one... Using a CDN would make it easier to secret services to collect data, because they have a single point where they can get all wanted data from instead of monitoring several providers and connections. CDNs generally don't have central logging at the request level. There's just no way for them to do that with the data rates you're looking at. Also, can be mitigated with chucking HTTPS at the problem. 4) Dependency concerns: as a project Debian should be as independent as possible. Using a CDN provider will create a big dependency to a specific company, although we might be able to shift companies from time to time. Using multiple CDN providers will mitigate that concern a little bit, but only to a certain degree. Having too many CDN providers will be as difficult to handle as now the many FTP mirror donators. So, there's some trade-off anyway. As I wrote in the initial email: CDNs are becoming a commodity. Switching from one provider to another isn't hard, and we already have offers from multiple CDNs, so I'm not particularly worried about this. Were it harder to switch, it would be different, but luckily, it's fairly easy. -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/m2ppr88lp0@rahvafeir.err.no
Re: Possibly moving Debian services to a CDN
]] Paul Wise About the archive mirrors, some reworded thoughts from the DPL IRC channel when this came up a few days ago: pabs [...] I think the current state of affairs is fine; I don't believe you're one of the person who is doing the legwork in maintaining any of the CDNs we're currently running, so how would you know that? Because it's not visibly broken (most of the time)? I see it breaking on a weekly basis, if not more often. [...] Removing the mirror network won't be possible anyway, people are still going to create mirrors, especially ISPs will for their customers; due to quotas and distant mirrors being much slower. Nobody has suggested removing the mirror network. What's being discussed is using a CDN for some .d.o services. If somebody wants to continue running their mirror they will of course be free to do so. bguptaNot all CDNs support IPv6. We will want to use CDNs that do support IPv6. It's one of the technical bits that need to fall into place before we will want to switch. I would rather expand the mirror network. Does that mean you're volunteering for the task of doing this and maintaining the various existing CDNs? -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/m2li1w8ldp@rahvafeir.err.no
Re: Possibly moving Debian services to a CDN
]] Andrew M.A. Cater On Sun, Oct 13, 2013 at 08:44:56AM +0200, Tollef Fog Heen wrote: Dear Project, The System Administration Team (DSA) are considering moving some of the static hosting that Debian currently provides from our infrastructure to one or more CDNs. We have received feedback indicating that a broader discussion is desired. Debian has, for over a decade, operated its own form of a content delivery network (multiple variants, actually) by leveraging our own infrastructure as master sources and community-provided infrastructure (primarily from universities and regional network providers) for local distribution. It works - tell anyone to use ftp.[country name].debian.org and it just works. If ftp.[mycountry].debian.org is down, use ftp.[neighbour country] .debian.org - and, crucially, it's all under one debian.org domain. Whether we use a CDN or not does not change that at all. (We might want to move everything towards using ftp.debian.org and just geolocate/anycast the CDN nodes and long-term deprecate the country based domains.) If managers/software licensing mavens/project funding authorities etc. question where your software is actualy from - it's from Debian themselves. To the same degree that it will be in the future. We don't run most of the mirrors ourselves, they're run by a bunch of third parties. Some of those are doing an excellent job, some are not. This is (potentially) good news for laptop/desktop users: instant access from closest mirrors. This is also an increasing trend: Firefox, Raspbian, Archlinux (I think) are all CDN served. One of the CDN offers we have are from the people that run the CDN for the pypi (Python packages) network. If you run behind restrictive firewall policies / in corporate land, it's not nearly so hot. Static long lasting mirrors are really useful here when you have to ask your network admin. to unblock firewalls for each IP address. That's not really that different from today. We move security mirrors every so often and they're geolocated so the ones you get in Europe and the US are not necessarily the same. If you want to configure 10 servers, say, or to build repeated groups of servers over a long time, it's good to have consistency and the existing network provides this. If the trend globally is to CDNs, however, it will be hard to buck the trend for the long term. If you want to, you can always run a local mirror. We are not trying to change that. There is nothing in the current setup that inherently give you any such guarantees. -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/m2k3hg8lbn@rahvafeir.err.no
