Re: About the recent DD retirements
On Sat, 24 Jan 2015, Anthony Towns wrote: > Archive, mirrors, lintian, piuparts, other qa stuff, yep. Right. > Promotion to testing [1], maybe? I think so; I know I personally would like to have a set of bits of R packages which didn't change on about the schedule of a stable release. > BTS, I somewhat don't think so? Probably just a single pseudo-package for coordinating groups of automatically packaged packages; if someone really wants to file more than one or two bugs, then it's probably a sign that the package should become a fully-supported package. [Or the person filing bugs should get more involved and help promote it to becoming a real package.] > NEW-esque license review seems impractical. Yeah; for debian-r, we assume that CRAN has the license listed correctly, and we just go from there. [Well, the awesome cran2deb stuff does it; I just modified it and built the results.] > Probably also wants an additional db tracking what upstream > commit/whatever was converted to which Debian-ised version. Right. > [1] Oh, dude! I finally thought of a possible rename for britney: > pro-test as in promotion to testing. Sounds good to me. -- Don Armstrong http://www.donarmstrong.com "You have many years to live--do things you will be proud to remember when you are old." -- Shinka proverb. (John Brunner _Stand On Zanzibar_ p413) -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20150124073130.gp21...@teltox.donarmstrong.com
Re: About the recent DD retirements
On Fri, Jan 23, 2015 at 10:03:08AM -0800, Don Armstrong wrote: > On Fri, 23 Jan 2015, Anthony Towns wrote: > > CRAN has about 6k (~250 in Debian), > Just to piggyback here, debian-r.debian.net has about 8.6k of these > packages (bioc, cran, and omegahat). Talk about giving 110%! [0] > > Yes, I really think Debian should have 300k+ packages, including > > everything in all the language archives > Handling this will probably require second-class packages, where we use > automated tools to package the package, and hope for the best. In some > ways, we're already starting to go that way,[1] but it would be nice to > use all of Debian's infrastructure even for those second-class packages, > too. Archive, mirrors, lintian, piuparts, other qa stuff, yep. Promotion to testing [1], maybe? BTS, I somewhat don't think so? NEW-esque license review seems impractical. Probably also wants an additional db tracking what upstream commit/whatever was converted to which Debian-ised version. Cheers, aj [0] Or more accurately 110% of 110% of 110% of 110% [1] Oh, dude! I finally thought of a possible rename for britney: pro-test as in promotion to testing. -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20150124071757.gc17...@master.debian.org
Re: About the recent DD retirements
On Fri, Jan 23, 2015 at 11:29:19AM -0600, Gunnar Wolf wrote: > Anthony Towns dijo [Fri, Jan 23, 2015 at 10:57:55AM +]: > > (Yes, I really think Debian should have 300k+ packages, including > > everything in all the language archives, no matter how special purposes > > (compare against the chiark* packages eg). > My answer to this is that... A distribution should mostly cater to > users. That means, we should target applications, not libraries. So I'm going to disagree with that in two ways. First, and more trivially, is that there are plenty of applications people want to run, that depend on unpackaged libraries that are a PITA to package. Etherpad is the example I already gave. Openrocket is another -- it's packaged as an installer that "downloads the pre-build OpenRocket .jar file from the upstream site and instals it"; because openrocket upstream likes using cool new java libraries for features and java libraries are a pain to package. ie, not having those libraries easily available in Debianised form /is/ one of the things that prevents people from running apps on Debian. Second, and more philosophically... Free software's about users not just being passive consumers, but treating them like co-developers. From fsf.org: "Free software developers guarantee everyone equal rights to their programs; any user can study the source code, modify it, and share the program." If we *really* believe that, then we shouldn't just be about getting users some nice eary to install prepackaged apps, we should be about getting them the tools they need to make those apps do whatever they want, and even make apps of their own. Once upon a time the C library, a compiler and a debugger were state of the art for doing that; nowdays, pulling together bits of code from the hundreds of thousands of pre-written open source modules is. Oh, and as a bonus: third -- how much easier would packaging applications be, if all the libraries they used (and all the libraries we currently maintain) we're already packaged, essentially for free? > By limiting our scope to what is actually wanted (i.e. by applications > that have been ITPed or RFPed, Per the wnpp pages, there are: * 868 packages being worked on * 3434 requested packages That's a total of 4302, compared to 21554 source packages, that's just under 20%, so about two years worth of backlog at 10% growth in source packages per year. ie, Debian's already not coming close to keeping up with the "actually wanted" stuff. (Also, note that quite a lot of those are modules from various language sites already) Cheers, aj -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20150124064646.gb17...@master.debian.org
Re: About language specific package management tools
On Fri, Jan 23, 2015 at 11:51:32AM +, Jonathan McDowell wrote: > On Fri, Jan 23, 2015 at 10:57:55AM +, Anthony Towns wrote: > > It takes a couple of minutes to download something using pip or > > npm; how long does it take to get a python or nodejs Debianized and > > installable? (eg: learning that npm2deb exists, how to use it, what else > > you have to do to have a package, building the package, and getting apt > > access to the package -- which in turn presumably includes setting up > > and distributing an archive key) > > > > In an ideal world, users would just be able to say "apt-get install > > lib-whatever-perl" and have it. At worst, they might have to modify > > their apt sources explicitly to say "yes, I know there's a lot of crap > > on CPAN that doesn't necessarily receive good security updates, I know > > what I'm doing". > > > > There's two ways that could be achieved: > > > > - having automated scripts pull everything from CPAN (et al), package > >it as debs, and publish it > > > > - having about 14,000 new DDs each individally maintaining 10-20 > >library packages > > > > But if the answer is "oh, you want to use some random nodejs package? just > > npm it into /opt. if you want there's some tools to help start you off > > in packaging it too" > > > > (Yes, I really think Debian should have 300k+ packages, including > > If this is being done in an automated fashion is there not a third > option? Teach apt and associated tools about the language specific > repositories. They'd do the download from CPAN or wherever, do the > conversion, and pass to dpkg. On the fly, no need to expand the archive > and no need to wait for the latest and greatest if you're that way > inclined. For extra bonus points teach cpan, gem etc to still work but > register the package + files with dpkg. Sure, that's an option. It seems more a gentoo-style approach than a Debian style approach to me. The main drawback to my mind is that I don't think it's as feasible: - if cpan/pypi/etc change their formats and the conversion scripts need updating, it'd have to be done on every system, rather than in one place - you'd still require a build environment just to use the modules on and end-user system - your end-user system relies on multiple mirror networks all working, rather than just the debian mirror network working - modifying apt is probably harder than just writing scripts that generate debs (On the other hand, just pointing users at urls and having them download stuff has fewer potential licensing issues) > I think there are some issues with automated packaging which would mean > that you'd still want hand crafted bits, You definitely want hand-crafted bits somewhere; ideally those would go in the upstream sources. Having them in a separate git repo could work too though; though that adds another dependency if you're expecting every user to have access to it. > and there's the question of how > you pin to a "stable" version (though I think often the reason > people are pulling in from external sources is because the version in > stable simply isn't recent enough, rather than unavailable) but it'd be Yep, that's certainly a question. You might be able to automate it via dependency analysis, britney-style, or use popcon-style statistics. I don't know how you'd choose versions either; would experimental/unstable/testing/stable be enough varieties? Would they even be useful? How would you deal with developing against one version of a module versus running an app using a different version? Coinstalled packages? Containers? > kinda cool to have: > > cpan http://cpan.etla.org/ > cran http://mirrors.ebi.ac.uk/CRAN/ > > etc in /etc/apt/sources.list and have it just work. I guess my theory was more along the lines of either: deb http://http.debian.net/debian/ testing main contrib extras (with them all combined into the "extras" component, and sharing the pool with Debian main), or deb http://http.debian.net/debian-extras/ testing cpan cran pypi deb http://http.debian.net/debian-extras/ unstable cpan npm (with a separate pool/ and separated by upstream archive) Same/similar result, different behind the scenes implementation. > You could probably > treat each different source as a different suite to aid with apt > pinning (and by default preferring the Debian version rather than the > external version). With the examples above it'd be: Pin: release c=extras or Pin: release o=Debian-extras or Pin: release o=Debian-extras, c=cpan, a=unstable Cheers, aj -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20150124051340.ga17...@master.debian.org
Re: About the recent DD retirements
On Fri, 23 Jan 2015, Anthony Towns wrote: > CRAN has about 6k (~250 in Debian), Just to piggyback here, debian-r.debian.net has about 8.6k of these packages (bioc, cran, and omegahat). > Yes, I really think Debian should have 300k+ packages, including > everything in all the language archives Handling this will probably require second-class packages, where we use automated tools to package the package, and hope for the best. In some ways, we're already starting to go that way,[1] but it would be nice to use all of Debian's infrastructure even for those second-class packages, too. 1: I mean, I've already done this myself for parts of CRAN. -- Don Armstrong http://www.donarmstrong.com But if, after all, we are on the wrong track, what then? Only disappointed human hopes, nothing more. And even if we perish, what will it matter in the endless cycles of eternity? -- Fridtjof Nansen _Farthest North_ p152 -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20150123180308.gj21...@teltox.donarmstrong.com
Re: About the recent DD retirements
Anthony Towns dijo [Fri, Jan 23, 2015 at 10:57:55AM +]: > On Thu, Jan 22, 2015 at 07:02:51PM +0800, Paul Wise wrote: > > On Thu, Jan 22, 2015 at 6:28 PM, Anthony Towns wrote: > > > - there are "archive networks" for most programming languages these days: > > >CPAN, CRAN, Hackage, PyPI, RubyGems, NPM, CCAN, etc. Installing > > >software from these sources is often necessary for Debian users, but > > >doesn't mesh well with packaged software (unlss you're a DD and can > > >package it yourself). Since it's all free software, I don't really > > >see why Debian doesn't have a set of automatic tools to repackage > > >all that software, so it's all just an "apt-get" away. > > We do: dh-make-perl, npm2deb, gem2deb, stdeb, cabal-debian etc which > > are intended to be wrapped by debdry to eliminate much of the initial > > packaging process. > > Sure, that works great if your model is "there are a few thousand pieces > of interesting software, and a few hundred packagers, each of whom can > maintain tens of them". But CPAN has 30k modules (~3k in Debian), CRAN > has about 6k (~250 in Debian), Hackage has 7k (~700 packaged), PyPI has > about 54k (2500 packaged), RubyGems has about 95k (~6000 packaged?), > npm has about 120k (266 packaged?). [0] > > There's obviously a seriously long tail of stuff that's not very > interesting to many people in those numbers, but Debian's still at least > an order of magnitude short of any of them. > (...) > In an ideal world, users would just be able to say "apt-get install > lib-whatever-perl" and have it. At worst, they might have to modify > their apt sources explicitly to say "yes, I know there's a lot of crap > on CPAN that doesn't necessarily receive good security updates, I know > what I'm doing". We have talked about this problem since long ago. I'm presently not involved in the (great!) pkg-perl group, but back in 2007 I wrote an article and presented this talk at the Vienna YAPC: http://gwolf.org/files/integrating_perl_in_distro.pdf http://gwolf.org/files/integrating_perl_in_distro_-_presentation.pdf Around that time there was talk in the pkg-perl group about packaging *all* of the CPAN. One of the factors that made us decide against it is that in Debian we care about quality, not just quantity. And not all of the available Perl modules have the same maintenance level (and Perl is quite an exemplary community WRT their quality levels). Having all modules packaged would mean we DDs would have to answer through the BTS for any shortcomings in the different Perl (or Ruby, or R, or TeX, or Hackage, or Python, or Node.js, or Drupal, or Whatnot) modules. Hardly feasible. > - having automated scripts pull everything from CPAN (et al), package >it as debs, and publish it > (...) > But if the answer is "oh, you want to use some random nodejs package? just > npm it into /opt. if you want there's some tools to help start you off > in packaging it too" > > (Yes, I really think Debian should have 300k+ packages, including > everything in all the language archives, no matter how special purposes > (compare against the chiark* packages eg). My answer to this is that... A distribution should mostly cater to users. That means, we should target applications, not libraries. Yes, most of us are programmers, and we are a special kind of users — But programmers often prefer anyway working with either a particular library version they are comfortable with, or with the bleeding edge, or whatnot. Programmers will often look outside of the distribution, because they will want specific bits at different points in time. I believe it is the programmers' products (the applications) are closer to what we should aim to package. If an application requires a given set of dependencies we don't have yet fulfilled, we should work on them. And yes, that might mean tweaking it so it works with the versions of the libraries we have on the distribution — As we need to provide an always-coherent, always-coinstallable set of packages. By limiting our scope to what is actually wanted (i.e. by applications that have been ITPed or RFPed, or for the *relatively few* specific librares deemed as worth having on their own because there's an obvious need for them, or whatnot), we can expect to keep excelling in overall quality. If we were to open the scope to just-about-everything, our distribution's quality would surely drop. > > > - perhaps it's all been fixed since I last looked, but "web apps" still > > >don't seem to be a "solved problem" to me. If you install, say, > > >libreoffice, you run apt-get (or whatever), then you run libreoffice, > > >and you're done. But if you want to install wordpress, you have a whole > > >bunch of additional steps to go through [1]. > > We have a web app policy but it is fairly abandoned. > > Isn't that statement alone a pretty clear indication that Debian's not > addressing the packaging problems of today? Yes. Web ap
Re: About language specific package management tools
On Fri, Jan 23, 2015 at 10:57:55AM +, Anthony Towns wrote: > It takes a couple of minutes to download something using pip or > npm; how long does it take to get a python or nodejs Debianized and > installable? (eg: learning that npm2deb exists, how to use it, what else > you have to do to have a package, building the package, and getting apt > access to the package -- which in turn presumably includes setting up > and distributing an archive key) > > In an ideal world, users would just be able to say "apt-get install > lib-whatever-perl" and have it. At worst, they might have to modify > their apt sources explicitly to say "yes, I know there's a lot of crap > on CPAN that doesn't necessarily receive good security updates, I know > what I'm doing". > > There's two ways that could be achieved: > > - having automated scripts pull everything from CPAN (et al), package >it as debs, and publish it > > - having about 14,000 new DDs each individally maintaining 10-20 >library packages > > But if the answer is "oh, you want to use some random nodejs package? just > npm it into /opt. if you want there's some tools to help start you off > in packaging it too" > > (Yes, I really think Debian should have 300k+ packages, including If this is being done in an automated fashion is there not a third option? Teach apt and associated tools about the language specific repositories. They'd do the download from CPAN or wherever, do the conversion, and pass to dpkg. On the fly, no need to expand the archive and no need to wait for the latest and greatest if you're that way inclined. For extra bonus points teach cpan, gem etc to still work but register the package + files with dpkg. I think there are some issues with automated packaging which would mean that you'd still want hand crafted bits, and there's the question of how you pin to a "stable" version (though I think often the reason people are pulling in from external sources is because the version in stable simply isn't recent enough, rather than unavailable) but it'd be kinda cool to have: cpan http://cpan.etla.org/ cran http://mirrors.ebi.ac.uk/CRAN/ etc in /etc/apt/sources.list and have it just work. You could probably treat each different source as a different suite to aid with apt pinning (and by default preferring the Debian version rather than the external version). J. -- I reckon that me and you should rule the world. signature.asc Description: Digital signature
Re: About the recent DD retirements
On Thu, Jan 22, 2015 at 07:02:51PM +0800, Paul Wise wrote: > On Thu, Jan 22, 2015 at 6:28 PM, Anthony Towns wrote: > > - there are "archive networks" for most programming languages these days: > >CPAN, CRAN, Hackage, PyPI, RubyGems, NPM, CCAN, etc. Installing > >software from these sources is often necessary for Debian users, but > >doesn't mesh well with packaged software (unlss you're a DD and can > >package it yourself). Since it's all free software, I don't really > >see why Debian doesn't have a set of automatic tools to repackage > >all that software, so it's all just an "apt-get" away. > We do: dh-make-perl, npm2deb, gem2deb, stdeb, cabal-debian etc which > are intended to be wrapped by debdry to eliminate much of the initial > packaging process. Sure, that works great if your model is "there are a few thousand pieces of interesting software, and a few hundred packagers, each of whom can maintain tens of them". But CPAN has 30k modules (~3k in Debian), CRAN has about 6k (~250 in Debian), Hackage has 7k (~700 packaged), PyPI has about 54k (2500 packaged), RubyGems has about 95k (~6000 packaged?), npm has about 120k (266 packaged?). [0] There's obviously a seriously long tail of stuff that's not very interesting to many people in those numbers, but Debian's still at least an order of magnitude short of any of them. So what's that mean if you're either developing or installing random software? 1) ignore anything not in Debian and be hamstrung 2) use the language native packaging stuff 3) invest a lot of time skilling up on the various Debian utilities and package it yourself (including repackaging every update) 4) invest even more time so that your packages can be included in Debian It takes a couple of minutes to download something using pip or npm; how long does it take to get a python or nodejs Debianized and installable? (eg: learning that npm2deb exists, how to use it, what else you have to do to have a package, building the package, and getting apt access to the package -- which in turn presumably includes setting up and distributing an archive key) In an ideal world, users would just be able to say "apt-get install lib-whatever-perl" and have it. At worst, they might have to modify their apt sources explicitly to say "yes, I know there's a lot of crap on CPAN that doesn't necessarily receive good security updates, I know what I'm doing". There's two ways that could be achieved: - having automated scripts pull everything from CPAN (et al), package it as debs, and publish it - having about 14,000 new DDs each individally maintaining 10-20 library packages But if the answer is "oh, you want to use some random nodejs package? just npm it into /opt. if you want there's some tools to help start you off in packaging it too" (Yes, I really think Debian should have 300k+ packages, including everything in all the language archives, no matter how special purposes (compare against the chiark* packages eg). By my count, jessie's at 20,700 source packages in main, which is about an 11% per annum growth rate since lenny, and indeed since sarge in 2005 [1]. If we'd stayed at about a 35% pa growth rate since woody then we'd be at around 200k source packages now, which I think would be the right order of magnitude for everything that's actually buildable in CPAN, and PyPI and so on being just an apt-get away.) > > - perhaps it's all been fixed since I last looked, but "web apps" still > >don't seem to be a "solved problem" to me. If you install, say, > >libreoffice, you run apt-get (or whatever), then you run libreoffice, > >and you're done. But if you want to install wordpress, you have a whole > >bunch of additional steps to go through [1]. > We have a web app policy but it is fairly abandoned. Isn't that statement alone a pretty clear indication that Debian's not addressing the packaging problems of today? > There are some external projects in this space, like sandstorm and juju. (And that one too... Also, unless I'm mistaken neither are packaged in Debian; apparently juju got dropped over a year and a half ago, bug#716754) > The wordpress wiki page seems to miss the wp-setup script that is in > the package now. The manpage says "For now, it's mainly for the package's internal usage though." > > - application isolation is a popular issue now, often solved by blatting > >whole OS images around (VMs, docker, ...). That brings up a whole host > >of packaging issues (how do you create the image, how do you keep it up > >to date, how do you distribute the image) > systemd has some great isolation features that use the same Linux > kernel APIs as docker/etc. > > The GNOME folks are working on application isolation for the desktop: > > http://blogs.gnome.org/mclasen/2015/01/21/sandboxed-applications-for-gnome/ Why is that something "the GNOME folks" are doing, but "the Debian folks" aren't? If Debian's just about packagin
