Now that we have the efs-deploy-config repos setup, the next big issue
to tackle is how we specify and manage dependencies in this shared
data.
Problem: right now, the data in these repos contains BOTH generic and
site-specific information about dependencies. I think it's reasonable
to assert the project namespace, so the list of projects on which
something depends can be shared, and made generic. A lot of these
hooks only work if you follow the metaproj/project naming conventions
we use, for one thing. What is site-specific, and the real problem,
are the specific releasealiases that are specified in efsdeploy.conf.
Note that platforms are also problematic, but I'll address those
separately.
Here's how this is done right now....
In files like efsdeploy/globals.conf, I've generically specified the
dependcies via macros like this:
[depends]
c_runtime = $oss_zlib_release
Then, the specific releasealias is in efsdeploy.conf:
[macros]
oss_zlib_release = oss/zlib/1.2.3
In many cases, the hooks query these macros and use them in all sorts
of ways, so we need a means of refering to oss/zlib/1.2.3 in both the
config files, like globals.conf, as well as programmatically via the
EFS::Deploy::Config API in the hooks. We need a means of mapping a
project (eg. oss/zlib) to the MPR that we use to refer to it. The
simple macro convention doesn't scale well, when you consider that
metaproj and project names do not map cleanly to the macros, since we
allow a broader range of characters into these names than are allowed
in perl variable names (although we could certainly allow them in the
macro expansion mechanism since we own it).
I am considering extending the macro expansion logic to support this syntax:
[configure]
optional_arguments = --with-zlib=/efs/dist/$depends_mpr{ oss/zlib
}/.exec/$platform
In a hook, you would get these using:
my $oss_zlib_release = $build->depends_mpr( q{oss/zlib} ); # NOTE:
The variable name's ONLY used in the hook, could be $foo if you want
The bigger issue is how does the VALUE of $depends_mpr{ oss/zlib } get
determined? This is why I'm moving the "depends" step to be right
after download. This command will now perform auto-discovery of these
releasealias values, for each project that has been specified as a
dependency. If the [depends] section is written this way:
[depends]
c_runtime = gnu/libiconv oss/openssl
path_build = gnu/libiconv
Now the fun part -- what's the algorithm for expanding M/P into an
M/P/R? Let's back up and recall some implicit best practices for how
we manage dependencies....
[*] When rebuilding a release (-buildXXX+1), changing the dependencies
should NOT happen automatically
When I build gnu/foo/x.y.z-buildXXX, and get it working perfectly with
a long list of dependencies, I want to be able to create buildXXX+1,
and rebuild it with identical results. If the dependencies are
auto-discovered and changed automatically, then the build may work
just fine, but it wll be easy to miss the fact that you're building
with new dependencies. Fortunately, we can bootstrap the defaults by
querying the EFS dependency data for the releasealias (if we're
rebuilding then this already exists, and there will already be
registered dependencies for it).
If you're building something for the first time, then the
auto-discovery mechanism can do it's thing. If you RE-build, then
you'll get the same dependencies, unless you take action to change
them (I'll come back to the mechanism for doing this). Note that this
solved the problem of how to manage site-specific dependencies, too!!!
[*] When building a new version, the dependencies should be
bootstrapped from the old one
What I mean is, suppose you have foo/bar/1.0 installed, and you decide
to install foo/bar/1.1. Since 1.1 doesn't exist yet, dependency
auto-discovery may very well find radically different dependency
dependencies than those used by 1.0. I want the auto-discovery
mechanism to start with the 1.0 dependencies, but conditionally
UPGRADE them for me. Since this is a new release, this is where you
normally WANT to do these upgrades.
So, what's the upgrade logic, and how do we control it? Here's some
ideas for upgrade rules....
[*] Upgrades are monotonically increasing
OK, this one should be obvious -- never downgrade automatically. This
has obvious implications for how we do version sorting, but let's
assume we can work that out (and I think it's very solvable). If you
want to downgrade, then you'll have to do so by hand (which will just
mean specifying an override in efsdeploy.conf, most likely)
[*] If the current dependency is prod, do not upgrade to one that is dev
Suppose foo/bar/1.0 has a dependency on gnu/stuff/1.2, which your site
has setstage to prod. Suppose also that gnu/stuff/1.3 exists, but is
still dev. By default, auto-discovery would NOT upgrade to the dev
release. Going from dev to prod is not a problem of course.
More of these will surely fall out of the woodwork as we work with the
early implementations, of course.
Next issue, how do we manage this data? Right now, the [depends]
section is in globals.conf. I am undecided on moving this to a
dedicated file, but that can always be done later. The *data* will
still be in the globals->depends part of the config namespace. The
data *might* be platform/compiler specific, so the auto-generated file
will be created in
/efs/dev/m/p/r/build/$platform/$compiler/autodepends.conf. Note that
this will NOT be in the build trees in /var/tmp. This is because (a)
the build trees are volatile, and (b) when registering the
dependencies, we'll need to easily collect all of this data, so
caching it in the baseline build directory will make it easily
visible, and persistent.
This file will have a format something like this:
[depends]
M1/P1 = R1
M2/P2 = R2
...
When the depends command runs, we'll be able to display the results of
the dependency analysis in a table, and highlight the possible
upgrades, too. We can implement something like:
efsdeploy depends-upgrade
which will apply ALL of the dependency upgrades that were found. I
don't want to invest too much time in designing a bunch of management
tools up front, until we've worked with the underlying mechanisms a
bit more in the real world, but you can get an idea for where this is
going from this rant, I hope.
I should have some prototype code working in a few days, as I work
through the problem...
_______________________________________________
EFS-dev mailing list
[email protected]
http://mailman.openefs.org/mailman/listinfo/efs-dev
