Re: RFC: Private CPAN In A Box
David Cantrell da...@cantrell.org.uk writes: On Wed, Jun 01, 2011 at 10:51:41AM +0200, Steffen Schwigon wrote: I used CPAN::Mini a lot but now start using *real* CPAN mirrors for the distroprefs mechanism that CPAN.pm provides (needed to work around any known problem on CPAN). You can, of course, use distroprefs with any CPAN-like mirror, including a mini-CPAN. Technically yes. But the existing distroprefs yaml files maintained by ANDK in CPAN (and reused by me) are not synced in CPAN::Mini. I once synced it manually with rsync on top of a CPAN::Mini mirror but I got tired by that eventually. Kind regards, Steffen -- Steffen Schwigon s...@renormalist.net Dresden Perl Mongers http://dresden-pm.org/
Re: RFC: Private CPAN In A Box
Steffen Schwigon s...@renormalist.net writes: David Cantrell da...@cantrell.org.uk writes: You can, of course, use distroprefs with any CPAN-like mirror, including a mini-CPAN. Technically yes. But the existing distroprefs yaml files maintained by ANDK in CPAN (and reused by me) are not synced in CPAN::Mini. Too early in the morning my brain leaks like a wiki… It's the patch collection I mean[1], that is referenced from some distroprefs and that is not contained in a CPAN::Mini mirror. The yaml files are part of the CPAN distro. These in turn I usually rsync manually from CPAN[2] or github directly to keep up to date with Andreas' work. Steffen Footnotes: [1] http://mirror.de.leaseweb.net/CPAN/authors/id/A/AN/ANDK/patches/ [2] http://cpansearch.perl.org/src/ANDK/CPAN-1.9600/distroprefs/ -- Steffen Schwigon s...@renormalist.net Dresden Perl Mongers http://dresden-pm.org/
Re: RFC: Private CPAN In A Box
David Cantrell da...@cantrell.org.uk writes: FWIW, my solution (which I haven't tried) would be: * make an up-to-date mirror of the most recent versions of everything on the CPAN, using CPAN::Mini; * use CPAN::Mini::Inject to add your own modules to the mirror. I used CPAN::Mini a lot but now start using *real* CPAN mirrors for the distroprefs mechanism that CPAN.pm provides (needed to work around any known problem on CPAN). I *think* (but haven't tried yet) that combining both might be good: - a local real CPAN mirror for the latest full CPAN - an additional CPAN::Mini (plus ::Inject) to add your own lcoal modules AFARI CPAN.pm looks into other mirrors once it doesn't find a module, which would happen for local modules only in CPAN::Mini. Kind regards, Steffen -- Steffen Schwigon s...@renormalist.net Dresden Perl Mongers http://dresden-pm.org/
Re: RFC: Private CPAN In A Box
Jeffrey Thalhammer j...@imaginative-software.com writes: Now this is where it gets interesting. Rather than install in site_perl, it installs into a directory right inside my project (let's call it dlib for dependent libraries). At the same time, it stashes the tar balls in a directory structure suitable for a CPAN mirror, which is also right inside my project directory (let's call it lpan, for local perl archive network). Finally, it generates the 02packages for the lpan/. Everything in lpan/ will be placed under revision control too. I like the idea. Of course the revision control should not be tied to subversion but kept to the developer who uses cvs/svn/git/hg/whatever. I realize this doesn't solve the big problem of automatically building a particular CPAN mirror with a magical mix modules. Which is ok imho, as that “big” problem usually was just an intermediate one, introduced to solve the actual distro bundle problem. It comes back when you want to restore a snapshot back in time from nothing, like deploy an app for Perl 5.6, but that's worth to skip for now. IMHO. Kind regards, Steffen -- Steffen Schwigon s...@renormalist.net Dresden Perl Mongers http://dresden-pm.org/
Re: RFC: Private CPAN In A Box
Jeffrey Thalhammer j...@imaginative-software.com writes: [build private CPANs] Just to add an experience for completeness of this discussion: I once tried to snapshot a complete MINICPAN with git every time I verified a CPAN::Mini sync with my application's “cpan upgrade ; make test”. But it did not work out well due to size and speed, even for the damn fast git. That *might* be relevant again also for the dlib/lpan idea, when the dependencies for your app are exploding. Kind regards, Steffen -- Steffen Schwigon s...@renormalist.net Dresden Perl Mongers http://dresden-pm.org/
Re: RFC: Private CPAN In A Box
Hi, On Wed, Jun 1, 2011 at 9:51 AM, Steffen Schwigon s...@renormalist.net wrote: I *think* (but haven't tried yet) that combining both might be good: - a local real CPAN mirror for the latest full CPAN - an additional CPAN::Mini (plus ::Inject) to add your own lcoal modules AFARI CPAN.pm looks into other mirrors once it doesn't find a module, which would happen for local modules only in CPAN::Mini. Never tried this either, but maybe start with an empty local CPAN and CPAN::Inject into that. Then use that one as the first mirror on your CPAN.pm configuration... Bye, -- Pedro Melo @pedromelo http://www.simplicidade.org/ http://about.me/melo xmpp:m...@simplicidade.org mailto:m...@simplicidade.org
Re: RFC: Private CPAN In A Box
On Wed, Jun 01, 2011 at 10:51:41AM +0200, Steffen Schwigon wrote: I used CPAN::Mini a lot but now start using *real* CPAN mirrors for the distroprefs mechanism that CPAN.pm provides (needed to work around any known problem on CPAN). You can, of course, use distroprefs with any CPAN-like mirror, including a mini-CPAN. Distroprefs are awesome, and not anything like well-known enough. I *think* (but haven't tried yet) that combining both might be good: - a local real CPAN mirror for the latest full CPAN - an additional CPAN::Mini (plus ::Inject) to add your own lcoal modules AFARI CPAN.pm looks into other mirrors once it doesn't find a module, which would happen for local modules only in CPAN::Mini. The urllist will indeed take multiple sources. This isn't something I've ever done though, so I don't know how well it works or whether it introduces any interesting gotchas. If you do do this, I suggest that one of the edge-cases you should test before relying on it is what happens if your local CPAN has a more up-to-date version of some module than what the real CPAN has. This could happen because you've got locally applied patches which the upstream author hasn't yet applied, or if you've not yet released the latest version of something to the CPAN, for example. -- David Cantrell | even more awesome than a panda-fur coat Seven o'clock in the morning is something that happens to those less fortunate than me
Re: RFC: Private CPAN In A Box
In article 02c001cc1a59$6865b810$39312830$@activestate.com, Jan Dubois j...@activestate.com wrote: AFAICT there is no good index for backpan though. There's a good index, but it lives on one of my private computers and is rather large. :)
Re: RFC: Private CPAN In A Box
In article BANLkTinzErQAnW4qthz=t1jdd8p3iou...@mail.gmail.com, David Golden xda...@gmail.com wrote: * You run the dependency analysis tool against your tarball, using the perl as a base and using the local+BackPAN to resolve dependencies. * Tool spits out the ordered list of tarballs (probably as URLs) needed to install the application I have this bit done but not public yet.
Re: RFC: Private CPAN In A Box
On May 25, 2011, at 4:16 AM, David Golden wrote: My though is to write a separate tool for this rather than try to make it work with either M::B or dzil. Agreed. A separate tool could be later integrated with either of those. Also, because you wouldn't want the tools themselves polluting the pure environment that you're trying to build in. Jeffrey Thalhammer Imaginative Software Systems vcard: http://www.imaginative-software.com/contact/jeff.vcf
Re: RFC: Private CPAN In A Box
On May 24, 2011, at 10:51 PM, Cosimo Streppone wrote: When you start a second project that uses a subset of project1 dependencies, what do you do? Do you duplicate the lpan/ and dlib/ folders? If you had to maintain patches, do you apply the patches to all your projects lpan/ and dlib/ folders? These dlib/ and lpan/ could be shared among projects, but they would lose, I imagine, part of the self-contained-ness of the original idea. I did think about this a bit. The dlib/ is always private to the project -- it is a generated directory that contains the fruit of installing the dependencies from the lpan. As you pointed out, most non-trivial applications consist of several distributions. In that case, the lpan could be moved outside the project directory, and you could use an svn:external to link it to each of the associated projects. So the project remains physically self-contained, but logically shares the same lpan with related projects. But I don't know how you would do this with git. Jeffrey Thalhammer Imaginative Software Systems vcard: http://www.imaginative-software.com/contact/jeff.vcf
Re: RFC: Private CPAN In A Box
Hi, On Wed, May 25, 2011 at 7:03 AM, Jeffrey Thalhammer j...@imaginative-software.com wrote: On May 24, 2011, at 10:51 PM, Cosimo Streppone wrote: When you start a second project that uses a subset of project1 dependencies, what do you do? Do you duplicate the lpan/ and dlib/ folders? If you had to maintain patches, do you apply the patches to all your projects lpan/ and dlib/ folders? These dlib/ and lpan/ could be shared among projects, but they would lose, I imagine, part of the self-contained-ness of the original idea. I did think about this a bit. The dlib/ is always private to the project -- it is a generated directory that contains the fruit of installing the dependencies from the lpan. As you pointed out, most non-trivial applications consist of several distributions. In that case, the lpan could be moved outside the project directory, and you could use an svn:external to link it to each of the associated projects. So the project remains physically self-contained, but logically shares the same lpan with related projects. But I don't know how you would do this with git. The same way. Create a separate repository with the shared dependencies, and include it on your projects as a submodule (git version of svn externals). Bye, -- Pedro Melo http://www.simplicidade.org/ xmpp:m...@simplicidade.org mailto:m...@simplicidade.org
Re: RFC: Private CPAN In A Box
On May 24, 2011, at 10:51 PM, Cosimo Streppone wrote: When you start a second project that uses a subset of project1 dependencies, what do you do? We should probably clarify what we mean by project and application. I think of an application as all the code that will run against a particular set of dependencies. A project is just a distribution, which may depend on other distributions (modules, really). An application will consist of one or more related projects. All the projects within an application will share a common lpan. But each application will have its own lpan. Jeffrey Thalhammer Imaginative Software Systems vcard: http://www.imaginative-software.com/contact/jeff.vcf
Re: RFC: Private CPAN In A Box
On May 24, 2011, at 10:51 PM, Cosimo Streppone wrote: If you had to maintain patches, do you apply the patches to all your projects lpan/ and dlib/ folders? [Sorry for the fragmented response. I'm a bit scatter-brained at this hour] Locally patched modules are an interesting case. You could stash your patched tarballs elsewhere in the repository, and then manually inject them (by copying) them into the lpan for each application that needs them. Or perhaps, you could establish a separate private CPAN just for patched distros and use CPAN::Site to produce an 02packages that falls back to the public CPAN for everything else. That would give you a mechanism for distributing patched distros (or any of your private distros) throughout the organization. Either way, each application ultimately gets a separate copy of the patched distro in its lpan. The only difference is how that distro get's delivered into the lpan. Off to bed now. Jeffrey Thalhammer Imaginative Software Systems vcard: http://www.imaginative-software.com/contact/jeff.vcf
Re: RFC: Private CPAN In A Box
On Wed, May 25, 2011 at 1:19 AM, Jeffrey Thalhammer j...@imaginative-software.com wrote: I realize this doesn't solve the big problem of automatically building a particular CPAN mirror with a magical mix modules. But based on my own development patterns, I think this model could provide a workable framework for organically evolving application dependencies, while providing a stable and reproducible dependency stack at the same time. I hadn't quite taken it that far in my thinking, but I'm glad to have a use case to consider as I work. My mental model to date has been along these lines: * You're working off a particular version of perl (/opt/perl/5.XX/) which has no site_lib modules installed * Modules you're adding as dependencies get installed into your local::lib (possibly per application, if you need that separation) * When you're done with your application, you tar it up and inject it into a repository that can be accessed via a URL * You add your tarball to a local+BackPAN index (local supersedes BackPAN) * You run the dependency analysis tool against your tarball, using the perl as a base and using the local+BackPAN to resolve dependencies. * Tool spits out the ordered list of tarballs (probably as URLs) needed to install the application If you need to patch a module from CPAN, you treat it just like an app, publish locally and add it to the local index. To deploy, on some other machine, you run a deployment tool against the matching perl on that machine, giving it the ordered list of URLs. For instances where you might have App A and App B, the tool should be able to factor out a common set of dependencies, which could be deployed separately. Then with the right ordering of library paths, you could have App A use version X of a module, while App B uses version Y of a module and otherwise, they use a common set of modules. I hadn't considered the team development case much. My snap reaction is that you would regularly deploy to a shared development library path rather than install modules to a personal local::lib during development. As long as the local+BackPAN index can reverse your dependency (module name and version) to the right tarball (including locally patched ones), it all should just work. My though is to write a separate tool for this rather than try to make it work with either M::B or dzil. -- David
Re: RFC: Private CPAN In A Box
The problem is that there is no way for a single application to use multiple versions of a module. So it's easy to imagine that very specific module prereqs could be in conflict and require manual override. That's not the right answer. What we need is application level dependencies that are specific and non conflicting and then to make it easy to install into app specific lib directories. That's what im On May 23, 2011 5:22 PM, David E. Wheeler da...@kineticode.com wrote: On May 20, 2011, at 11:04 AM, Jeffrey Thalhammer wrote: I'm not attempting to build an all-encompasing dependency management system. I've tried to avoid thinking about it in terms of CPAN as we know it, since in theory, that is an implementation detail. But in practice, I'm will stipulate that a dependency is just a Perl module in some CPAN repository. Yeah, that's what's confusing me. To me, a dependency is a Perl module in some CPAN repository that the module I'm trying to install right now depends on. That is, it's listed in in the prereqs section of a META.json file. The trouble with CPANs is that they are a moving target -- modules are constantly updated, added, and removed[1]. This creates problems for developers that want to use the CPAN tool chain, but need to have a stable set of dependencies. At the same time, they need flexibility to evolve those dependencies in a systematic way. Well, the new version spec in CPAN Meta Spec 2.0 helps with this. You can specify very precise requirements. Thanks for bringing PGXN to my attention. The architecture is very similar to what I was thinking. I will dig deeper to see if we can leverage PGXN. [1] I think a lot of this trouble would go away if the CPAN tool chain simply permitted authors to express precisely which $VERSION of something they require. So I have to wonder if that is the right place to focus, rather than building something on top of the tool chain. You can. http://search.cpan.org/~dagolden/CPAN-Meta-2.110930/lib/CPAN/Meta/Spec.pm#Version_Ranges Best, David
Re: RFC: Private CPAN In A Box
On Tue, 24 May 2011 07:22:02 +1000, David E. Wheeler
da...@kineticode.com wrote:
On May 20, 2011, at 11:04 AM, Jeffrey Thalhammer wrote:
[1] I think a lot of this trouble would go away if the CPAN tool chain
simply permitted authors to express precisely which $VERSION of
something they require.
Hi David,
You can.
http://search.cpan.org/~dagolden/CPAN-Meta-2.110930/lib/CPAN/Meta/Spec.pm#Version_Ranges
Whaaat!? :-)
Let's see if I understand this correctly.
If, as an example, in this file,
http://cpansearch.perl.org/src/COSIMO/Net-Prober-0.03/META.json
I say:
prereqs : {
...
runtime : {
requires : {
Net::Ping : = 2.35
}
},
...
},
Where current Net::Ping on CPAN is 2.36,
than this will install Net::Ping 2.35 as
prerequisite?
Or, if Net::Ping on the system is already 2.36
it will refuse to proceed?
I have tried fiddling with Makefile.PL/META.yml/META.json
without success.
--
Cosimo
Re: RFC: Private CPAN In A Box
On Monday, May 23, 2011, David Golden da...@autopragmatic.com wrote: The problem is that there is no way for a single application to use multiple versions of a module. So it's easy to imagine that very specific module prereqs could be in conflict and require manual override. That's not the right answer. What we need is application level dependencies that are specific and non conflicting and then to make it easy to install into app specific lib directories. That's what im The SEND button should be harder to press by accident... That's what I'm working on in June - trying to tie all the various lessons from other toolchain and meta tools into something that can be for Perl apps what cpan(p|m) is for Perl modules. I think people are trying to make cpan solve the application level problem and that's why custom CPAN mirror seems like the answer. David
Re: RFC: Private CPAN In A Box
On Mon, May 23, 2011 at 8:38 PM, Cosimo Streppone cos...@streppone.it wrote:
I say:
prereqs : {
...
runtime : {
requires : {
Net::Ping : = 2.35
}
},
...
},
Where current Net::Ping on CPAN is 2.36,
than this will install Net::Ping 2.35 as
prerequisite?
Or, if Net::Ping on the system is already 2.36
it will refuse to proceed?
I think the current CPAN clients will all refuse to proceed. This is
what I meant when I said that version ranges don't accomplish the goal
because the computer can't know what the right thing to do is.
-- David
Re: RFC: Private CPAN In A Box
On Mon, May 23, 2011 at 11:52:39AM -0700, Jeffrey Thalhammer wrote: After looking at David Wheeler's PGXN and David Cantrell's CPAN snapshots, I've come up with a story. Tell me if you think this makes sense (or not): Historically, CPAN mirrors have always been static files. In other words, the paths in the 02packages file always pointed directly to some physical file on disk. This made it possible to access mirrors the with the older ftp:// and file:// protocols supported by CPAN.pm. It also made synchronizing the mirrors easy with tools like rsync. This has worked well for a long time. But the need to support older protocols limits the flexibility of a CPAN mirror. If you drop support for ftp:// and file://, and just focus on http:// then things can become more interesting. This open the door for a CPAN server (like the PGXN::Manager) that can do all sorts of things. Your mirror only has to support file:// if you want it to live on a user's local disk, and only has to support ftp:// if you want to give FTP access. I realise that this is obvious, but please bear with me :-) You can support whatever craziness you want over HTTP. The only place in CPAN::Config which mentions the mirror's URL is this: urllist = [q[http://cp5.6.2an.barnyard.co.uk/]], Crucially, it derives URLs for both the index and the distributions from that. So, if you have a magic mirror and are willing to do the heavy lifting on the server (and please bear in mind it's *very* heavy lifting - my daily rebuilds of static 02packages files take about 7 hours, and to build a mirror like cp2011-05an takes about 12 hours) you could simply configure CPAN.pm to use: urllist = [q[http://example.com/perl/5.8.8/asat/2008-05-04Z11:43:22]], CPAN.pm would then get its index by requesting: http://example.com/perl/5.8.8/asat/2008-05-04Z11:43:22/modules/02packages.details.txt.gz and you can, of course, construct whatever the hell you want in response, provided it has the right format. That file would normally read something like ... Description: This is a whitespace-seperated file. Description: Each line is modulename moduleversion filename. Line-Count: 33800 Last-Updated: Tue, 24 May 2011 06:22:15 GMT ::Crypt::DH 0.04 B/BI/BINGOS/-Crypt-DH-0.04.tar.gz ABI 1.0 M/MA/MALAY/ABI-1.0.tar.gz ACH::Builder 0.03 T/TK/TKEEFER/ACH-Builder-0.03.tar.gz ACME::Error 0.03 C/CW/CWEST/ACME-Error-0.03.tar.gz but there's no reason why the meat of it couldn't be stuff like ... ::Crypt::DH distcontaining/::Crypt::DH which CPAN.pm would resolve to ... http://example.com/perl/5.8.8/asat/2008-05-04Z11:43:22/authors/id/distcontaining/::Crypt::DH which, apart from the annoying 'authors/id/' in the middle, which could be trivially dealt with using mod_rewrite, looks deliciously RESTish. -- David Cantrell | Reality Engineer, Ministry of Information Your call is important to me. To see if it's important to you I'm going to make you wait on hold for five minutes. All calls are recorded for blackmail and amusement purposes.
Re: RFC: Private CPAN In A Box
On May 24, 2011, at 3:04 AM, David Golden wrote: Or, if Net::Ping on the system is already 2.36 it will refuse to proceed? I think the current CPAN clients will all refuse to proceed. This is what I meant when I said that version ranges don't accomplish the goal because the computer can't know what the right thing to do is Ah, but sometime this summer the clients (and therefor the computer) will? Best, David
Re: RFC: Private CPAN In A Box
On Tue, May 24, 2011 at 12:14 PM, David E. Wheeler da...@kineticode.com wrote: On May 24, 2011, at 3:04 AM, David Golden wrote: Or, if Net::Ping on the system is already 2.36 it will refuse to proceed? I think the current CPAN clients will all refuse to proceed. This is what I meant when I said that version ranges don't accomplish the goal because the computer can't know what the right thing to do is Ah, but sometime this summer the clients (and therefor the computer) will? I doubt it. I have plans for an application installer that will pre-compute dependencies and reliably set up per-application lib directories. Different applications could have different versions of a module, but a single application will only have one version available to it. This is not something that the CPAN clients are really designed to do (though they can be made to do the right things with INSTALLBASE and so on). A better specification and design for application dependencies and installation will avoid the need to create N custom CPAN views as a hack to make CPAN clients try to be application deployment tools. -- David
Re: RFC: Private CPAN In A Box
On May 24, 2011, at 9:18 AM, David Golden wrote: I doubt it. I have plans for an application installer that will pre-compute dependencies and reliably set up per-application lib directories. Different applications could have different versions of a module, but a single application will only have one version available to it. This is not something that the CPAN clients are really designed to do (though they can be made to do the right things with INSTALLBASE and so on). Well, are you using the same version ranges spec for the META.json file? Seems to me that this might result in a stand-alone library that, given prereqs, could determine what should be installed. Then the existing clients could use it too, eh? A better specification and design for application dependencies and installation will avoid the need to create N custom CPAN views as a hack to make CPAN clients try to be application deployment tools. Yeah, that'll be nice. Best, David
Re: RFC: Private CPAN In A Box
On Tue, May 24, 2011 at 12:25 PM, David E. Wheeler da...@kineticode.com wrote: Well, are you using the same version ranges spec for the META.json file? Seems to me that this might result in a stand-alone library that, given prereqs, could determine what should be installed. Then the existing clients could use it too, eh? That's the idea, though I've not worked out the details. But for example, I'd like to use the configuration output data (MYMETA.json) to determine three sets of modules (tarballs, really): - latest versions on CPAN - currently installed versions on developers machine - minimum versions on CPAN that satisfy the prereq specs It's possible that version ranges could conflict, in which case some or all of those might not be achievable. Ideally, the dependencies would be determined against a specific version of perl without additional libraries installed. Then, given one of those three ordered list of tarballs that satisfy all prereqs, it should be possibly to repeatably deploy an application with a known set of module versions, even as the latest on CPAN evolves. Of the three sets, users would have to decide which suit their specific needs/purpose, or could substitute versions, etc. That's my vision in a nutshell. I have some people in NY that I'm going to do some brainstorming with and then I'll try to get a proof of concept code out next month. -- David
Re: RFC: Private CPAN In A Box
On Tue, May 24, 2011 at 10:17 PM, David Golden xda...@gmail.com wrote: Then, given one of those three ordered list of tarballs that satisfy all prereqs, it should be possibly to repeatably deploy an application with a known set of module versions, even as the latest on CPAN evolves. That could get a bit troublesome with the current best practice to delete older versions of module from CPAN. Leon
Re: RFC: Private CPAN In A Box
On Tue, May 24, 2011 at 4:32 PM, Leon Timmermans faw...@gmail.com wrote: On Tue, May 24, 2011 at 10:17 PM, David Golden xda...@gmail.com wrote: Then, given one of those three ordered list of tarballs that satisfy all prereqs, it should be possibly to repeatably deploy an application with a known set of module versions, even as the latest on CPAN evolves. That could get a bit troublesome with the current best practice to delete older versions of module from CPAN. http://backpan.perl.org/ I don't remember when they started keeping the history -- I've seen stuff as far back as 1997. -- Davie
RE: RFC: Private CPAN In A Box
On Tue, 24 May 2011, David Golden wrote: On Tue, May 24, 2011 at 4:32 PM, Leon Timmermans faw...@gmail.com wrote: On Tue, May 24, 2011 at 10:17 PM, David Golden xda...@gmail.com wrote: Then, given one of those three ordered list of tarballs that satisfy all prereqs, it should be possibly to repeatably deploy an application with a known set of module versions, even as the latest on CPAN evolves. That could get a bit troublesome with the current best practice to delete older versions of module from CPAN. And a somewhat unfortunate practice, as it makes it hard to create diffs between arbitrary module versions. And with the new fast-mirror mechanism, I'm not sure that the aggressive pruning of CPAN should still be considered a best practice, as it makes the web-based diff on search.cpan.org pretty useless. For example when trying to understand this Locale-Codes changelog entry: | 3.16 2011-03-01 sbeck | * NEW CODE(s) I would first try to look here: http://search.cpan.org/tools/Locale-Codes-3.16 But it is not working because even the previous release has already been deleted. http://backpan.perl.org/ I don't remember when they started keeping the history -- I've seen stuff as far back as 1997. Yes, backpan has the complete history with very few exceptions (I think some website scraping modules were deleted from backpan when the website owner threatened legal action). AFAICT there is no good index for backpan though. I have to audit diffs to older module versions quite often, so I have my own tools, and found that even the links are not comprehensive; I have to crawl both backpan/modules/by-module and backpan/modules/by-authors/id to make sure I find *all* the different versions. Cheers, -Jan
Re: RFC: Private CPAN In A Box
On Tue, May 24, 2011 at 5:27 PM, Jan Dubois j...@activestate.com wrote: AFAICT there is no good index for backpan though. I have to audit diffs to older module versions quite often, so I have my own tools, and found that even the links are not comprehensive; I have to crawl both backpan/modules/by-module and backpan/modules/by-authors/id to make sure I find *all* the different versions. I believe that brian foy created a pretty comprehensive one -- I think he may have actually investigated how module files changed from tarball to tarball. (E.g. Module $VERSION was the same in two tarballs, but the file MD5's are different). The hard problem is that PAUSE doesn't have any maintainer/co-maintainer history -- so given a historical DAGOLDEN/Foo-Bar-1.23.tar.gz and RJBS/Foo-Bar-1.23.tar.gz, there's no way to tell that one or the other was the official indexed version. You can look at who has maintainer rights *today* and assume from that who was authorized when the tarball was uploaded. I don't know (yet) how many collisions like that there are. I suspect it's a relatively small number that can be flagged and resolved with a bit of research and email. -- David
Re: RFC: Private CPAN In A Box
On May 24, 2011, at 1:17 PM, David Golden wrote: Then, given one of those three ordered list of tarballs that satisfy all prereqs, it should be possibly to repeatably deploy an application with a known set of module versions, even as the latest on CPAN evolves. . Wow! This is getting juicy. It's easy for me to get carried away with all the intricacies of dependency management. But for my task at hand, I think it might be the wrong problem. So for the sake of this discussion, let's put aside the issue of how a private CPAN (really, the 02packages) could be constructed to have a magical mixture of modules. Instead, I want to think about how a private CPAN mirror fits into the development cycle. This might be easiest by walking through a hypothetical scenario... Let's suppose I wan to write a new application in Perl. By application, I really just mean some set of modules and scripts. And let's assume that my development, testing, and deployment environments all have a perfectly clean version of perl -- no modules have been added or updated from the core. I want my app to leverage the CPAN tool chain, so I setup a CPAN-style distribution for my source code. It has lib/, bin/, and t/ directories, and all the usual trappings. I'm also going to use Module::Build. And of course, all this is in some source control system (assume svn for now). After a couple of Mountain-Dews, I have some code and tests and everything passes. Commit. Now I want to add some feature that requires a module from CPAN. So I go shopping on search.cpan.org and find one I like called Frobulator::Simple. At this point, I might go ask my sysadmin to install Frobulator::Simple (and all its dependencies) in /usr/lib/perl5 on the development machine. Or if I'm smart, I might use local::lib to install it somewhere in my home directory. But these both suck, because changing /usr/lib/perl5 will affect everyone else, and who knows what else I might have installed in my home directory. So instead, let's say I add Frobulator::Simple to the requires list (or build_requires, or whatever) in my Build.PL. Now I run the installdeps target, which fetches the latest Frobulator::Simple and all its dependencies from a public CPAN and installs them. Now this is where it gets interesting. Rather than install in site_perl, it installs into a directory right inside my project (let's call it dlib for dependent libraries). At the same time, it stashes the tar balls in a directory structure suitable for a CPAN mirror, which is also right inside my project directory (let's call it lpan, for local perl archive network). Finally, it generates the 02packages for the lpan/. Everything in lpan/ will be placed under revision control too. Now I have all my application code and my dependencies in one local, revision-controlled location. When I run the build target, all my modules are copied into the blib/ directory as usual. And when I run the test target, both blib/ and dlib/ are placed on @INC.If the tests pass, then commit. If not, then fix my code. Or I can svn revert my lpan/ directory, change the requires list in Build.PL to try a new module, and run the installdeps target again. Rinse and repeat as needed. This goes on for a while as my application evolves until I have several dependencies. Dependencies that I want to keep are in the growing dpan/ directory and committed to svn. Meanwhile, the dlib directory is clouding up with all the modules I've experimented with. Periodically, I want to test the whole stack. So now I run the realclean target, which wipes out dlib/. Next, I run an installdepslocal target, which basically does an installdeps, but this time it pulls everything from the lpan. This effectively flushes out any modules that fail since, over time, I may have upgraded some of their dependencies in my lpan/ and those upgrades may not be backward compatible. If some of the dependencies do fail, it is my job to figure out what to do. This might mean removing a newer distro from the lpan/ in favor of an older one. Or maybe I'll make a patched version of the distro and put it in the lpan/. Or perhaps I'll just switch to a different dependency altogether. How this all gets done is out of scope for this discussion, but I imagine that we could generate some kind of map that tracks all the dependencies for the app. This could help developers see the interdependencies and figure out how an upgrade or downgrade will impact the app. You could also have special build targets that assist with adding, removing, or upgrading specific sub-trees in the dependency graph. But for now, let's assume I've got a working stack.. When I run the test target again, I'm now testing my code against a pure set of dependencies. The traditional disttest target might also be enhanced to first run installdepslocal in the dist directory. Once my application code is golden,
Re: RFC: Private CPAN In A Box
On Wed, 25 May 2011 15:19:49 +1000, Jeffrey Thalhammer j...@imaginative-software.com wrote: On May 24, 2011, at 1:17 PM, David Golden wrote: Then, given one of those three ordered list of tarballs that satisfy all prereqs, it should be possibly to repeatably deploy an application with a known set of module versions, even as the latest on CPAN evolves. [...] Let's suppose I wan to write a new application in Perl. By application, [...] Rather than install in site_perl, it installs into a directory right inside my project (let's call it dlib for dependent libraries). [...] it stashes the tar balls in a directory structure [...] which is also inside my project directory (let's call it lpan ...) Everything in lpan/ will be placed under revision control too. Interesting! What do you think? When you start a second project that uses a subset of project1 dependencies, what do you do? Do you duplicate the lpan/ and dlib/ folders? If you had to maintain patches, do you apply the patches to all your projects lpan/ and dlib/ folders? These dlib/ and lpan/ could be shared among projects, but they would lose, I imagine, part of the self-contained-ness of the original idea. -- Cosimo
Re: RFC: Private CPAN In A Box
After looking at David Wheeler's PGXN and David Cantrell's CPAN snapshots, I've come up with a story. Tell me if you think this makes sense (or not): Historically, CPAN mirrors have always been static files. In other words, the paths in the 02packages file always pointed directly to some physical file on disk. This made it possible to access mirrors the with the older ftp:// and file:// protocols supported by CPAN.pm. It also made synchronizing the mirrors easy with tools like rsync. This has worked well for a long time. But the need to support older protocols limits the flexibility of a CPAN mirror. If you drop support for ftp:// and file://, and just focus on http:// then things can become more interesting. This open the door for a CPAN server (like the PGXN::Manager) that can do all sorts of things. For example, it could serve up distros that are spread across multiple remote machines. Or they could be stored in version control system or database. Or you could present a view of the mirror that meets certain criteria, such as by time, or cpantesters results, or by author, or by kwalitee scores, or by some other metadata (like http://cp2011-05an.barnyard.co.uk/). The concept of a CPAN server also provides a convenient hook for plugging in other services, like a private PAUSE or MetaCPAN. I'm not saying that *the* CPAN should work this way. I'm just suggesting that a *private* CPAN could work this way. Jeffrey Thalhammer Imaginative Software Systems vcard: http://www.imaginative-software.com/contact/jeff.vcf
Re: RFC: Private CPAN In A Box
On May 20, 2011, at 11:04 AM, Jeffrey Thalhammer wrote: I'm not attempting to build an all-encompasing dependency management system. I've tried to avoid thinking about it in terms of CPAN as we know it, since in theory, that is an implementation detail. But in practice, I'm will stipulate that a dependency is just a Perl module in some CPAN repository. Yeah, that's what's confusing me. To me, a dependency is a Perl module in some CPAN repository that the module I'm trying to install right now depends on. That is, it's listed in in the prereqs section of a META.json file. The trouble with CPANs is that they are a moving target -- modules are constantly updated, added, and removed[1]. This creates problems for developers that want to use the CPAN tool chain, but need to have a stable set of dependencies. At the same time, they need flexibility to evolve those dependencies in a systematic way. Well, the new version spec in CPAN Meta Spec 2.0 helps with this. You can specify very precise requirements. Thanks for bringing PGXN to my attention. The architecture is very similar to what I was thinking. I will dig deeper to see if we can leverage PGXN. [1] I think a lot of this trouble would go away if the CPAN tool chain simply permitted authors to express precisely which $VERSION of something they require. So I have to wonder if that is the right place to focus, rather than building something on top of the tool chain. You can. http://search.cpan.org/~dagolden/CPAN-Meta-2.110930/lib/CPAN/Meta/Spec.pm#Version_Ranges Best, David
Re: RFC: Private CPAN In A Box
On May 23, 2011, at 11:52 AM, Jeffrey Thalhammer wrote: Historically, CPAN mirrors have always been static files. In other words, the paths in the 02packages file always pointed directly to some physical file on disk. This made it possible to access mirrors the with the older ftp:// and file:// protocols supported by CPAN.pm. It also made synchronizing the mirrors easy with tools like rsync. Yep. PGXN is the same, although I've built an API server that has a dynamic interface for searching. This has worked well for a long time. But the need to support older protocols limits the flexibility of a CPAN mirror. If you drop support for ftp:// and file://, and just focus on http:// then things can become more interesting. This open the door for a CPAN server (like the PGXN::Manager) that can do all sorts of things. For example, it could serve up distros that are spread across multiple remote machines. Or they could be stored in version control system or database. Or you could present a view of the mirror that meets certain criteria, such as by time, or cpantesters results, or by author, or by kwalitee scores, or by some other metadata (like http://cp2011-05an.barnyard.co.uk/).The concept of a CPAN server also provides a convenient hook for plugging in other services, like a private PAUSE or MetaCPAN. I'm not saying that *the* CPAN should work this way. I'm just suggesting that a *private* CPAN could work this way. What I've tried to do with PGXN is to get the best of both worlds. At the lowest level, the mirror generated by PGXN::Manager should be very simple, with all static files. But many of those static files are JSON files, and so every mirror is a lightweight REST API server without any additional work required. Then anyone can build a service using the API. Or they can build new mirrors that do additional stuff with the data, which is what the API server does (adds additional info to some JSON files, generates HTML pages, manages full-text index, etc.). So take the static stuff, and build something dynamic on top of it, either using its API or by adding to it. Best, David
Re: RFC: Private CPAN In A Box
On May 20, 2011, at 3:51 AM, Jeffrey Thalhammer wrote: Once again, I'm faced with building another private CPAN. But this time, I have an opportunity to build something that could have broader appeal in the Perl community. In fact, the explicit goal is to produce an open source, turnkey framework for creating, deploying, and maintaining a private CPAN. FWIW, I've written one, called PGXN. The various parts are managed here: https://github.com/pgxn/ PGXN::Manager == PAUSE PGXN::API PGXN::Site == search.cpan.org pgxnclient = CPAN.pm The client was actually written by someone else, in Python, just using the API. https://github.com/dvarrazzo/pgxnclient/ He was able to do it pretty quickly because almost everything is represented by a static JSON file. The mirror and API APIs are documented here: https://github.com/pgxn/pgxn-api/wiki Might be of interest to you, FWIW. I wrote it for PostgreSQL Extensions, but tried to implement a generalized solution that could be used by any language or community -- or privately, if you like. So with that in mind, I'm looking for feedback on what you might want from such a framework. Here are some questions I've been asking myself -- hopefully they will help stir your mind: How would it provide an identifiable reproducible stack of dependencies that developers can use to write, test, and deploy their code against? Do you mean dependencies to build the framework itself, or how to identify dependencies in distributions released to the network? META.json works great for both of these, IMHO. How would it allow one to define and deliver a standardized set of dependencies to all your environments? Not sure I understand the question here… How would it enable developers to experiment with dependencies that are not part of the standard environment? What is the standard environment? How would it enable different teams to work against different versions of their dependencies? You mean other than just preserving earlier versions of distributions? How would it enable teams to independently upgrade, add, and remove their dependencies in a controlled, reproducible manner? I think your use of the term dependencies confuses me. What are dependencies? How would teams use it to distribute and share their own modules and applications within the organization? I don't understand. You upload your module or app to the upload server. Which parts of the larger CPAN ecosystem would be most valuable in a private CPAN (e.g. CPAN Testers, AnnoCPAN, search.cpan.org, CPAN Ratings)? That's really going to depend on the requirements of the organization, I would think. Some will think testing is important; others that ratings or wikis are important. How would one migrate their legacy code into CPAN-style distributions? Fortunately, all one needs to do is add a META.json file and you're read to go. Unless you want a single method of installing distributions, in which case you would either need to build an installation client that recognizes and executes many different approaches to building, or get your users to adopt one. How would one migrate from their existing dependency management infrastructure to a private CPAN? Again, just define dependencies in the META.json file. How might one want incorporate a private CPAN with their other development infrastructure, such as bug trackers continuous integration servers? Provide a robust API. :-) Thanks for sharing your thoughts! HTH, sorry I get a bit lost with some of your questions. Best, David
Re: RFC: Private CPAN In A Box
Hi Jeffrey, On 20 May 2011 08:51, Jeffrey Thalhammer j...@imaginative-software.com wrote: NOTE: This was also posted on perlmonks at http://perlmonks.org/?node_id=905878. I'm trying to reach a wide audience, so I'm posting it here too. Over the last few years, I've helped build private CPANs (DarkPANs or DPANs as brian d foy calls them) for 3 different organizations. Each time I cobbled together some combination of different CPAN::Site, CPAN::Mini::Inject and CPAN modules with various shell scripts, commit hooks, and cron jobs. Although they were generally effective, I feel they were clunky, highly specialized, and hard to maintain. You might want to check out: http://beta.metacpan.org/ the front and backend are here: https://github.com/cpan-api http://blogs.perl.org/mt/mt-search.fcgi?limit=20search=metacpan has some history and discuss with them on irc.perl.org #metacpan - I've pointed them to your perlmonk post https://github.com/szabgab/CPAN-Digger also exists - I think Gabour is discussing if there is any overlap with #metacpan which they're discussing. http://blogs.perl.org/users/michael_j/2011/04/deploying-perl-code-with-git-locallib-minicpan-and-cpanminus.html is actually the setup we use at work, but this only works because we only have one version of all code for all projects. Hope that helps. Leo
Re: RFC: Private CPAN In A Box
On May 20, 2011, at 8:07 AM, David E. Wheeler wrote: HTH, sorry I get a bit lost with some of your questions. No, this is very helpful. I think you've shown that I need to take a step back and look at the underlying problem. I'm not attempting to build an all-encompasing dependency management system. I've tried to avoid thinking about it in terms of CPAN as we know it, since in theory, that is an implementation detail. But in practice, I'm will stipulate that a dependency is just a Perl module in some CPAN repository. As I see it, there are two sides to the problem. The first is creating tools for creating, managing, and using a private CPAN repository. There is already a lot of prior art for this, but I feel that it is fragmented. So this might just be a matter of assembling the right combination of existing technologies. The second (and probably more difficult) side is establishing patterns for using these tools, which enable developers to write, test, and deploy their code against an identifiable and reproducible set of dependencies. I acknowledge that one size will not fit all, but I strongly believe there is (or should be) some common ground here. The trouble with CPANs is that they are a moving target -- modules are constantly updated, added, and removed[1]. This creates problems for developers that want to use the CPAN tool chain, but need to have a stable set of dependencies. At the same time, they need flexibility to evolve those dependencies in a systematic way. Thanks for bringing PGXN to my attention. The architecture is very similar to what I was thinking. I will dig deeper to see if we can leverage PGXN. [1] I think a lot of this trouble would go away if the CPAN tool chain simply permitted authors to express precisely which $VERSION of something they require. So I have to wonder if that is the right place to focus, rather than building something on top of the tool chain. Jeffrey Thalhammer Imaginative Software Systems vcard: http://www.imaginative-software.com/contact/jeff.vcf
Re: RFC: Private CPAN In A Box
On May 20, 2011, at 8:29 AM, Leo Lapworth wrote: http://beta.metacpan.org/ https://github.com/szabgab/CPAN-Digger Thanks! MetaCPAN and CPAN::Digger are definitely part of the picture. But I see these more as adjunct features of a private CPAN. I think the meat of the problem lies in how one actually utilizes a private CPAN in the development cycle. http://blogs.perl.org/users/michael_j/2011/04/deploying-perl-code-with-git-locallib-minicpan-and-cpanminus.html That's exactly what I'm talking about. I think this kind of pattern is popping up in a lot of organizations. Somehow, I want to try and package that idea in a way that might appeal to a wide audience. Jeffrey Thalhammer Imaginative Software Systems vcard: http://www.imaginative-software.com/contact/jeff.vcf
Re: RFC: Private CPAN In A Box
On Fri, May 20, 2011 at 11:04:24AM -0700, Jeffrey Thalhammer wrote: The trouble with CPANs is that they are a moving target -- modules are constantly updated, added, and removed[1]. http://cp2010-05an.barnyard.co.uk/ and all its date/time-stamped friends and relations aren't. A feature I keep meaning to add is some way for users to pick one of my mirrors and then further restrict it so that, for example, you might want the CPAN as at the beginning of 2011 but with Some::Module (and all its pre-requisites) stuck at some earlier point in time, or even a way of applying that to an up-to-date CPAN mirror (an up-to-date mirror is just a special case of my mirrors anyway, one with no filtering at all). I want to do this in a way that: * won't require manual intervention from me to set up a new restriction * will work with any CPAN client * is efficient with storage and CPU * provides free ponies and kittens -- David Cantrell | Minister for Arbitrary Justice
