Re: Source packages appropriate by default?
On 24 July 2013 11:08, Scott Kitterman wrote: > On Wednesday, July 24, 2013 11:00:40 AM Daniel J Blueman wrote: >> Perhaps we have two issues here: > >> The 20% additional download due to sources [1] would help both issues, >> but perhaps of bigger impact, trusting the country-level mirror for >> the security updates? > ... > You aren't. Security updates are pushed first to security.ubuntu.com and then > copied to archive.ubuntu.com and mirrored from there. The security pocket > isn't mirrored so you always hit it directly and if a country mirror lags, you > get the package from security.ubuntu.com. Also, the signing key is the same > Ubuntu archive signing key whether you're getting a package form > archive.ubuntu.com or a country mirror, so you aren't trusting the country > mirror cryptographically either. What I meant, if the country-level archive is sync'd every 12-24 hours, would it be sufficient to download the security pocket from .archive.ubuntu.com? It is mirrored, so this would alleviate the second issue. Daniel -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Source packages appropriate by default?
On Wednesday, July 24, 2013 11:00:40 AM Daniel J Blueman wrote: > Perhaps we have two issues here: > The 20% additional download due to sources [1] would help both issues, > but perhaps of bigger impact, trusting the country-level mirror for > the security updates? ... You aren't. Security updates are pushed first to security.ubuntu.com and then copied to archive.ubuntu.com and mirrored from there. The security pocket isn't mirrored so you always hit it directly and if a country mirror lags, you get the package from security.ubuntu.com. Also, the signing key is the same Ubuntu archive signing key whether you're getting a package form archive.ubuntu.com or a country mirror, so you aren't trusting the country mirror cryptographically either. Scott K -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Source packages appropriate by default?
Perhaps we have two issues here: - the download during installs or first index update is 6-7MB extra, which makes a real difference when installing lots of computers - downloads from security.ubuntu.com being slow (eg 1-5KB/s) as it's >500ms away The 20% additional download due to sources [1] would help both issues, but perhaps of bigger impact, trusting the country-level mirror for the security updates? Daniel --- [1] Get:1 http://security.ubuntu.com precise-security Release.gpg [198 B] Get:2 http://security.ubuntu.com precise-security Release [49.6 kB] Get:3 http://security.ubuntu.com precise-security/main Sources [83.5 kB] Get:4 http://security.ubuntu.com precise-security/restricted Sources [2494 B] Get:5 http://security.ubuntu.com precise-security/universe Sources [27.1 kB] Get:6 http://security.ubuntu.com precise-security/multiverse Sources [1383 B] Get:7 http://security.ubuntu.com precise-security/main amd64 Packages [296 kB] Get:8 http://security.ubuntu.com precise-security/restricted amd64 Packages [4627 B] Get:9 http://security.ubuntu.com precise-security/universe amd64 Packages [77.7 kB] Get:10 http://security.ubuntu.com precise-security/multiverse amd64 Packages [2186 B] Get:11 http://security.ubuntu.com precise-security/main i386 Packages [311 kB] Get:12 http://security.ubuntu.com precise-security/restricted i386 Packages [4620 B] Get:13 http://security.ubuntu.com precise-security/universe i386 Packages [80.5 kB] Get:14 http://security.ubuntu.com precise-security/multiverse i386 Packages [2371 B] Get:15 http://security.ubuntu.com precise-security/main TranslationIndex [74 B] Get:16 http://security.ubuntu.com precise-security/multiverse TranslationIndex [71 B] Get:17 http://security.ubuntu.com precise-security/restricted TranslationIndex [72 B] Get:18 http://security.ubuntu.com precise-security/universe TranslationIndex [73 B] On 24 July 2013 10:46, Robie Basak wrote: > On Tue, Jul 23, 2013 at 09:31:15PM -0400, Scott Kitterman wrote: >> Before we run off and expend a lot more effort on this, I'd like to >> see something other than handwaving that this is really is a >> significant issue. > > [size comparisions snipped] > > My concern is latency, not size. How many round trips will we save this > way? For cloud images using Amazon S3 mirrors, for example, each request > is quite a bit slower AIUI, and apt-get doesn't currently support > concurrent requests to a single server. > > This is a pain for instances that start up with cloud-init and > immediately have to update sources and install things before they can > become functional. It'd be nice to see the delay from "juju deploy" to > having a live service running get shorter. Same for "juju add-unit". > Admittedly an alternative means to achieve this could be to have > cloud-init remove the deb-src lines first, but it seems a shame to leave > others behind if this really does improve things. > > I agree that I should come up with actual figures before pushing ahead > for this reason. > > -- > Ubuntu-devel-discuss mailing list > Ubuntu-devel-discuss@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss -- Daniel J Blueman -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Source packages appropriate by default?
On Tue, Jul 23, 2013 at 09:31:15PM -0400, Scott Kitterman wrote: > Before we run off and expend a lot more effort on this, I'd like to > see something other than handwaving that this is really is a > significant issue. [size comparisions snipped] My concern is latency, not size. How many round trips will we save this way? For cloud images using Amazon S3 mirrors, for example, each request is quite a bit slower AIUI, and apt-get doesn't currently support concurrent requests to a single server. This is a pain for instances that start up with cloud-init and immediately have to update sources and install things before they can become functional. It'd be nice to see the delay from "juju deploy" to having a live service running get shorter. Same for "juju add-unit". Admittedly an alternative means to achieve this could be to have cloud-init remove the deb-src lines first, but it seems a shame to leave others behind if this really does improve things. I agree that I should come up with actual figures before pushing ahead for this reason. -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Source packages appropriate by default?
Or 90/110K per day per computer for Precise. I guess what was getting me is the additional 6-7MB during install or first update: http://archive.ubuntu.com/ubuntu/dists/precise/universe/source/ 4.8M/5.9M http://archive.ubuntu.com/ubuntu/dists/precise/main/source/ 912K/1.1M On 24 July 2013 09:31, Scott Kitterman wrote: > On Tuesday, July 23, 2013 08:21:40 AM Jordon Bedwell wrote: >> On Tue, Jul 23, 2013 at 6:32 AM, Scott Kitterman > wrote: >> > Assuming add-apt-repository was installed by default, it's close. I think >> > something like this might be reasonable (imagine some policykit or >> > whatever it is called now magic here): >> > >> > $ sudo apt-get source hello >> > Reading package lists... Done >> > Building dependency tree >> > Reading state information... Done >> > E: You must put some 'source' URIs in your sources.list >> > Would you like 'source' URIs to be added? (y/N) >> > Y >> > deb-src lines have been added to your sources.list. >> > ... >> > Get:9 http://archive.ubuntu.com saucy/main Sources [1,001 kB] >> > Get:10 http://archive.ubuntu.com saucy/restricted Sources [6,578 B] >> > Get:11 http://archive.ubuntu.com saucy/universe Sources [6,071 kB] >> > >> > In other words, it's, I think, possible to make it roughly as easy as it >> > is >> > now to get source without having the sources.list "cluttered". For users >> > of our releases, I doubt it saves much, but that would be a way to do it >> > that both avoids whatever amount of bandwidth usage is involved until the >> > user opts in to it, but preserves ready access to the source that I think >> > is important. >> Depending on how clever and one-off you want to be you could also just >> give them the http url to the source as well. It shouldn't be that >> hard to guess since apt already has most of the information needed to >> just generate the URL from a chosen apt server in the normal deb. >> This would allow for one-off downloads (for example somebody needs to >> look at the way debian does some of it's compiles so they can >> replicate without a package so they grab the source for nginx -- >> that's a one-off IMO if they would never use any other source >> package.) >> >> Though I personally like a default command that would be something >> like add-apt-default-sources so you can also give them the ability to >> run that command and disable sources too (but you can already do that >> via the GUI and terminal by editing /etc/apt/sources.list and such.) > > Before we run off and expend a lot more effort on this, I'd like to see > something other than handwaving that this is really is a significant issue. > > /ubuntu/dists/raring-security/main/source > > [ ] Release 24-Jul-2013 01:16 106 > [ ] Sources.bz2 24-Jul-2013 01:16 32K > [ ] Sources.gz 24-Jul-2013 01:16 38K > > For end users, how much is really downloaded? > > /ubuntu/dists/raring-updates/main/source > > [ ] Release 24-Jul-2013 01:16 105 > [ ] Sources.bz2 24-Jul-2013 01:16 50K > [ ] Sources.gz 24-Jul-2013 01:16 62K > > /ubuntu/dists/raring-updates/universe/source > > [ ] Release 24-Jul-2013 01:16 109 > [ ] Sources.bz2 24-Jul-2013 01:16 64K > [ ] Sources.gz 24-Jul-2013 01:16 77K > > It doesn't seem like a lot. > > Scott K > > -- > ubuntu-devel mailing list > ubuntu-de...@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel -- Daniel J Blueman -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Source packages appropriate by default?
On Tuesday, July 23, 2013 08:21:40 AM Jordon Bedwell wrote: > On Tue, Jul 23, 2013 at 6:32 AM, Scott Kitterman wrote: > > Assuming add-apt-repository was installed by default, it's close. I think > > something like this might be reasonable (imagine some policykit or > > whatever it is called now magic here): > > > > $ sudo apt-get source hello > > Reading package lists... Done > > Building dependency tree > > Reading state information... Done > > E: You must put some 'source' URIs in your sources.list > > Would you like 'source' URIs to be added? (y/N) > > Y > > deb-src lines have been added to your sources.list. > > ... > > Get:9 http://archive.ubuntu.com saucy/main Sources [1,001 kB] > > Get:10 http://archive.ubuntu.com saucy/restricted Sources [6,578 B] > > Get:11 http://archive.ubuntu.com saucy/universe Sources [6,071 kB] > > > > In other words, it's, I think, possible to make it roughly as easy as it > > is > > now to get source without having the sources.list "cluttered". For users > > of our releases, I doubt it saves much, but that would be a way to do it > > that both avoids whatever amount of bandwidth usage is involved until the > > user opts in to it, but preserves ready access to the source that I think > > is important. > Depending on how clever and one-off you want to be you could also just > give them the http url to the source as well. It shouldn't be that > hard to guess since apt already has most of the information needed to > just generate the URL from a chosen apt server in the normal deb. > This would allow for one-off downloads (for example somebody needs to > look at the way debian does some of it's compiles so they can > replicate without a package so they grab the source for nginx -- > that's a one-off IMO if they would never use any other source > package.) > > Though I personally like a default command that would be something > like add-apt-default-sources so you can also give them the ability to > run that command and disable sources too (but you can already do that > via the GUI and terminal by editing /etc/apt/sources.list and such.) Before we run off and expend a lot more effort on this, I'd like to see something other than handwaving that this is really is a significant issue. /ubuntu/dists/raring-security/main/source [ ] Release 24-Jul-2013 01:16 106 [ ] Sources.bz2 24-Jul-2013 01:16 32K [ ] Sources.gz 24-Jul-2013 01:16 38K For end users, how much is really downloaded? /ubuntu/dists/raring-updates/main/source [ ] Release 24-Jul-2013 01:16 105 [ ] Sources.bz2 24-Jul-2013 01:16 50K [ ] Sources.gz 24-Jul-2013 01:16 62K /ubuntu/dists/raring-updates/universe/source [ ] Release 24-Jul-2013 01:16 109 [ ] Sources.bz2 24-Jul-2013 01:16 64K [ ] Sources.gz 24-Jul-2013 01:16 77K It doesn't seem like a lot. Scott K -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Source packages appropriate by default?
On Tue, Jul 23, 2013 at 6:32 AM, Scott Kitterman wrote: > Assuming add-apt-repository was installed by default, it's close. I think > something like this might be reasonable (imagine some policykit or whatever it > is called now magic here): > > $ sudo apt-get source hello > Reading package lists... Done > Building dependency tree > Reading state information... Done > E: You must put some 'source' URIs in your sources.list > Would you like 'source' URIs to be added? (y/N) > Y > deb-src lines have been added to your sources.list. > ... > Get:9 http://archive.ubuntu.com saucy/main Sources [1,001 kB] > Get:10 http://archive.ubuntu.com saucy/restricted Sources [6,578 B] > Get:11 http://archive.ubuntu.com saucy/universe Sources [6,071 kB] > > In other words, it's, I think, possible to make it roughly as easy as it is > now to get source without having the sources.list "cluttered". For users of > our releases, I doubt it saves much, but that would be a way to do it that > both avoids whatever amount of bandwidth usage is involved until the user opts > in to it, but preserves ready access to the source that I think is important. Depending on how clever and one-off you want to be you could also just give them the http url to the source as well. It shouldn't be that hard to guess since apt already has most of the information needed to just generate the URL from a chosen apt server in the normal deb. This would allow for one-off downloads (for example somebody needs to look at the way debian does some of it's compiles so they can replicate without a package so they grab the source for nginx -- that's a one-off IMO if they would never use any other source package.) Though I personally like a default command that would be something like add-apt-default-sources so you can also give them the ability to run that command and disable sources too (but you can already do that via the GUI and terminal by editing /etc/apt/sources.list and such.) -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Source packages appropriate by default?
On Tuesday, July 23, 2013 08:12:16 AM Robie Basak wrote: > On Tue, Jul 23, 2013 at 03:02:02AM -0400, Scott Kitterman wrote: > > So those are a couple of examples of what I think is definitely not what > > we > > want. I'm open to discussion about alternate ways to preserve easy access > > to the source. > > How about: > > $ sudo apt-get source hello > Reading package lists... Done > Building dependency tree > Reading state information... Done > E: You must put some 'source' URIs in your sources.list > E: Type "add-apt-repository sources" to do this automatically for you. > $ sudo add-apt-repository sources > deb-src lines have been added to your sources.list. > Now type "apt-get update", and then "apt-get source ..." will work. > $ sudo apt-get update > (...) > $ sudo apt-get source hello > (works) > > To do this, we'd need to patch apt to add the second error line, and > implement "sources" to add-apt-repository. Assuming add-apt-repository was installed by default, it's close. I think something like this might be reasonable (imagine some policykit or whatever it is called now magic here): $ sudo apt-get source hello Reading package lists... Done Building dependency tree Reading state information... Done E: You must put some 'source' URIs in your sources.list Would you like 'source' URIs to be added? (y/N) Y deb-src lines have been added to your sources.list. ... Get:9 http://archive.ubuntu.com saucy/main Sources [1,001 kB] Get:10 http://archive.ubuntu.com saucy/restricted Sources [6,578 B] Get:11 http://archive.ubuntu.com saucy/universe Sources [6,071 kB] ... apt-get source lightdm-kde Reading package lists... Done Building dependency tree Reading state information... Done NOTICE: 'lightdm-kde' packaging is maintained in the 'Git' version control system at: git://git.debian.org/pkg-kde/kde-extras/lightdm-kde.git Need to get 1,386 kB of source archives. Get:1 http://archive.ubuntu.com/ubuntu/ saucy/universe lightdm-kde 0.3.2.1-1ubuntu2 (dsc) [1,543 B] Get:2 http://archive.ubuntu.com/ubuntu/ saucy/universe lightdm-kde 0.3.2.1-1ubuntu2 (tar) [1,379 kB] Get:3 http://archive.ubuntu.com/ubuntu/ saucy/universe lightdm-kde 0.3.2.1-1ubuntu2 (diff) [5,088 B] Fetched 1,386 kB in 1s (807 kB/s) apt-get source lightdm-kde Reading package lists... Done Building dependency tree Reading state information... Done NOTICE: 'lightdm-kde' packaging is maintained in the 'Git' version control system at: git://git.debian.org/pkg-kde/kde-extras/lightdm-kde.git Need to get 1,386 kB of source archives. Get:1 http://archive.ubuntu.com/ubuntu/ saucy/universe lightdm-kde 0.3.2.1-1ubuntu2 (dsc) [1,543 B] Get:2 http://archive.ubuntu.com/ubuntu/ saucy/universe lightdm-kde 0.3.2.1-1ubuntu2 (tar) [1,379 kB] Get:3 http://archive.ubuntu.com/ubuntu/ saucy/universe lightdm-kde 0.3.2.1-1ubuntu2 (diff) [5,088 B] Fetched 1,386 kB in 1s (807 kB/s) (and so on) In other words, it's, I think, possible to make it roughly as easy as it is now to get source without having the sources.list "cluttered". For users of our releases, I doubt it saves much, but that would be a way to do it that both avoids whatever amount of bandwidth usage is involved until the user opts in to it, but preserves ready access to the source that I think is important. Scott K -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Source packages appropriate by default?
Hi Daniel (2013.07.23_08:13:47_+0200) > For the other 99% of users, where practicality is more important than > immediate access to source, we end up wasting ~10% of Canonical and > our mirror's bandwidth on the source updates. Can you back that up with evidence? As I (and a few other people) have repeatedly said in this thread: The release pocket lists aren't changed after release. Only -updates, -security, -backports and -proposed change, and they are all small because they are an overlay on the release pocket. SR -- Stefano Rivera http://tumbleweed.org.za/ H: +27 21 461 1230 C: +27 72 419 8559 -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Source packages appropriate by default?
On Tue, Jul 23, 2013 at 03:02:02AM -0400, Scott Kitterman wrote: > So those are a couple of examples of what I think is definitely not what we > want. I'm open to discussion about alternate ways to preserve easy access to > the source. How about: $ sudo apt-get source hello Reading package lists... Done Building dependency tree Reading state information... Done E: You must put some 'source' URIs in your sources.list E: Type "add-apt-repository sources" to do this automatically for you. $ sudo add-apt-repository sources deb-src lines have been added to your sources.list. Now type "apt-get update", and then "apt-get source ..." will work. $ sudo apt-get update (...) $ sudo apt-get source hello (works) To do this, we'd need to patch apt to add the second error line, and implement "sources" to add-apt-repository. Robie -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Source packages appropriate by default?
On Tuesday, July 23, 2013 06:59:43 AM Robie Basak wrote: > On Tue, Jul 23, 2013 at 01:51:46AM -0400, Scott Kitterman wrote: > > I think most developers would believe the current situation is > > appropriate. > > I disagree. > > > By default users have the same access to source and binary packages and > > for a free software distribution, that is the ethically correct approach. > Indeed, but you never replied to my original response to your concern. > By "same access", do you specifically require the mechanism to be to > keep users' local apt caches maintained with source entries? If so, why > is such a mechanism necessary to fit the spirit of Free Software? If the > user still has easy access to the source, why is this not sufficient? > > I'm happy to discuss what "easy access" might actually mean, but I see > no reason that it should require the waste of users' bandwidth and time. Sorry. I didn't mean to ignore you. What's easy? For example, I think "install more packages to get the tools to get the source" (use pull-lp-source in ubuntu-dev-tools) doesn't qualify. There are tons of documentation all over the web and other places as well that assume apt-get source works. I think access using installed tools that are normally used for the job (wget is installed (I think) by default, but I don't think having to go to a web page to find a URL and then wget'ing the components of the source package is easy either. So those are a couple of examples of what I think is definitely not what we want. I'm open to discussion about alternate ways to preserve easy access to the source. Scott K -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
