Re: RFC: Private CPAN In A Box

[David Golden]

On Wed, May 25, 2011 at 1:19 AM, Jeffrey Thalhammer
 wrote:
> I realize this doesn't solve the "big" problem of automatically building a 
> particular CPAN mirror with a magical mix modules.  But based on my own 
> development patterns, I think this model could provide a workable framework 
> for organically evolving application dependencies, while providing a stable 
> and reproducible dependency stack at the same time.

I hadn't quite taken it that far in my thinking, but I'm glad to have
a use case to consider as I work.

My mental model to date has been along these lines:

* You're working off a particular version of perl (/opt/perl/5.XX/)
which has no site_lib modules installed

* Modules you're adding as dependencies get installed into your
local::lib  (possibly per application, if you need that separation)

* When you're done with your application, you tar it up and inject it
into a repository that can be accessed via a URL

* You add your tarball to a "local+BackPAN" index (local supersedes BackPAN)

* You run the dependency analysis tool against your tarball, using the
perl as a base and using the local+BackPAN to resolve dependencies.

* Tool spits out the ordered list of tarballs (probably as URLs)
needed to install the application

If you need to patch a module from CPAN, you treat it just like an
app, publish locally and add it to the local index.

To deploy, on some other machine, you run a deployment tool against
the matching perl on that machine, giving it the ordered list of URLs.

For instances where you might have App A and App B, the tool should be
able to factor out a common set of dependencies, which could be
deployed separately.  Then with the right ordering of library paths,
you could have App A use version X of a module, while App B uses
version Y of a module and otherwise, they use a common set of modules.

I hadn't considered the team development case much.  My snap reaction
is that you would regularly "deploy" to a shared development library
path rather than install modules to a personal local::lib during
development.  As long as the local+BackPAN index can reverse your
dependency (module name and version) to the right tarball (including
locally patched ones), it all should just work.

My though is to write a separate tool for this rather than try to make
it work with either M::B or dzil.

-- David

Re: RFC: Private CPAN In A Box

[David Cantrell]

On Tue, May 24, 2011 at 10:19:49PM -0700, Jeffrey Thalhammer wrote:

> It's easy for me to get carried away with all the intricacies of dependency 
> management.  But for my task at hand, I think it might be the wrong problem.  
> So for the sake of this discussion, let's put aside the issue of how a 
> private CPAN (really, the 02packages) could be constructed to have a magical 
> mixture of modules.  Instead, I want to think about how a private CPAN mirror 
> fits into the development cycle.  This might be easiest by walking through a 
> hypothetical scenario...
> 
> Let's suppose I wan to write a new application in Perl.  By "application", I 
> really just mean some set of modules and scripts.  And let's assume that my 
> development, testing, and deployment environments all have a perfectly clean 
> version of perl -- no modules have been added or updated from the core.
> 
> I want my app to leverage the CPAN tool chain, so I setup a CPAN-style 
> distribution for my source code.  It has lib/, bin/,  and t/ directories, and 
> all the usual trappings.  I'm also going to use Module::Build.  And of 
> course, all this is in some source control system (assume svn for now).
> 
> After a couple of Mountain-Dews, I have some code and tests and everything 
> passes.  Commit.  Now I want to add some feature that requires a module from 
> CPAN.  So I go shopping on search.cpan.org and find one I like called 
> Frobulator::Simple...
> 
> So instead, let's say I add Frobulator::Simple to the  "requires" list (or 
> "build_requires", or whatever) in my Build.PL.  Now I run the "installdeps" 
> target, which fetches the latest Frobulator::Simple and all its dependencies 
> from a public CPAN and installs them.
> 
> Now this is where it gets interesting.  Rather than install in site_perl, it 
> installs into a directory right inside my project (let's call it "dlib" for 
> "dependent libraries").  At the same time, it stashes the tar balls in a 
> directory structure suitable for a CPAN mirror, ...

FWIW, my solution (which I haven't tried) would be:

* make an up-to-date mirror of the most recent versions of everything on
  the CPAN, using CPAN::Mini;
* use CPAN::Mini::Inject to add your own modules to the mirror.

To keep your mirror up to date with new releases to the CPAN, use the
funky fast update mechanism to get change notifications, and, for each
new distribution:
  * download it;
  * inject it into a "staging" CPAN;
  * try to test your application, using a clean perl and the staging
CPAN to resolve dependencies;
  * if it passes, inject the new distribution into your "real" CPAN;
  * if it fails, remove it from the staging CPAN

Optionally, if adding a distribution to the CPAN makes your app fail its
tests, put it into a third CPAN mirror and notify your devs that it's
there and that they should fix the app and then re-inject the
distribution into your CPAN mirror.  However, I can see problems
here.  First, they won't do it because they have more important* things
to do such as adding features or fixing their own bugs; second, you
should probably address those failures in order, meaning that that work
stream will eventually get blocked entirely when something like Catalyst
has an incompatible change that completely fucks your application over;
third, what do you do about dists which fail their own tests but don't
make your app fail its tests?

* to management

-- 
David Cantrell | Hero of the Information Age

"IMO, the primary historical significance of Unix is that it marks the
time in computer history where CPUs became so cheap that it was possible
to build an operating system without adult supervision."
 -- Russ Holsclaw in a.f.c