On Fri, Dec 30, 2011 at 12:09 PM, Phillip Moore
<[email protected]> wrote:
> More thoughts, and some significant progress in this area....
>
> I spent most of yesterday collecting the efsedploy rules for
> EVERYTHING I've built into /efs/dist over the last few months (it's a
> lot), by copying the src directory to:
>
> ~/dev/efs/deploy-config/$metaproj/$project
OK, so once everything in that directory has been sanitized of ALL
site-specific information, then we have to figure out how to manage
it. Here's what I'm currently thinking, although this is going to
evolve, of course.
First of all, note that efsdeploy is going to start whining at you to
switch from efs/deploy-config to efs/deploy-site, because I want to
use the name deploy-config for all of this data. Deal with it....
It's *trivial* to switch, and takes about 5-10 minutes, if you type
slow.
I want to create 3 types of git repo to manage this data:
deploy-config-$metaproj-$project.git
deploy-config-$metaproj.git
deploy-config.git
For things like gnu/gcc, we'll obviously create a project-specific git
repo, and for large metaprojs where we expect a lot of similarity
among the projects, we can create metaproj-specific ones. The
default, global git repo would contain all the small, simple stuff,
like oss/zlib. For starters, I expect to create these:
deploy-config-gnu-gcc.git (which will be used for rhel/gcc as well)
deploy-config-gnu-gcclib.git (also for rhel/gcclib)
deploy-config-gnu.git
deploy-config-perl5-core.git
deploy-config-perl5.git
deploy-config-apache.git (might get it's own system, too -- we'll see...)
And of course the generic one. What I like about this is we always
migrate things from one to the other pretty easily. if we find that,
say oss/openssl has grown complex enough, we can yank it out of
deploy-config, and create deploy-config-oss-openssl.
So how do we deploy this data? Having it well managed is git is
great, but how to we access it when building things with efsdeploy,
and where does it get copied/cached?
Let's start with the generic repo first. Just as we use
efs/deploy-site/current to abstract the site-specific config
information, I think we can do the following:
deploy-config.git => /efs/dist/efs/deploy-config/current
The metaproj- and project-specific ones would then map to:
deploy-config-$metaproj.git =>
/efs/dist/$metaproj/deploy-config-$metaproj/current
deploy-config-$metaproj-project.git =>
/efs/dist/$metaproj/deploy-config-$metaproj-$project/current
This would allow us to publish, probably date-based, any of these
repositories with the "latest" set of efsdeploy build rules.
Note that the default rules go into the efs metaproj, obviously, but
we can still have a "deploy-config-efs.git" repo if we want, with no
conflict.
It is very straight forward to code a solution that allows us to
automate keeping the local copies of these rules uptodate as they
change. I will almost certainly have a first pass at this within the
next month. However, what is NOT clear is just how to use this
information in efsdeploy when building release.
Reproducibility concerns me. The rules are going to evolve, and when
we make gnu/gcc rule changes to build, say 4.7.0, we don't want to
break builds of 4.4.6, and yet *testing* that is extremely expensive.
For that reason, I think the contents of the efsdeploy directory
should be CACHED in the release, rather than read from these projects
during the build. Just as we are going to provide generic dependency
specs (see email from 30 minutes ago), and expanding those into
specific releasealiases to be used for the duration of the build, I
think we should do the same for the project-specific build rules, or
at least make it optional.
In theory, if we just have efsdeploy search for these rules the same
way it searches for system-specific (i,e, gnu, perl5, etc) rules, and
then site-specific rules, then I could actually build EVERYTHING I
have in /efs/dist with EMPTY source directories!! If a project is
supported by one of these repos, then you can build a new release with
nothing more than:
efs create project ...
efs create release ...
cd ..../src
efsdeploy down:up
The contents of the src directory would contain NOTHING but the
changes you had to make (hooks, configs, whatever) to get the release
to build. Those changes should then be re-integrated with the git
repo in a controlled fashion, so that the next person building that
MPR has no pain. The specific workflow for how a new change gets
rolled into the published git repos will need to be worked out, but I
think that will be straight forward.
Now, obviously, in order to *develop* changes to the rules, we'll need
a simply means of overriding the path to these published rules.
Maybe you want to install the latest set of gnu/gcc rules, but not
make them current until you've actually done a test-build of the
releases you care about. Maybe something in efsdeploy.conf (which
will now be a site/release-specific file, by definition) like this.
Say we wanted to test out some local changes right from the source
tree (I've been doing this with symlinks for now):
[rules]
$metaproj/$project = /home/efsops/dev/efs/deploy-config-gnu-gcc
or, perhaps, if we use date-based releases, you could install the
latest update into /efs/dist, and test it out this way:
[rules]
$metaproj/$project = /efs/dist/gnu/deploy-config-gnu-gcc/20111230
Alternately, you could just rsync the efsdeploy directory right into a
release, and work with a copy.
OK, that's enough of Phil's rantings for one day. Not that anyone's
paying attention, but you will see commits that implement many of
these features over the next few weeks.
_______________________________________________
EFS-dev mailing list
[email protected]
http://mailman.openefs.org/mailman/listinfo/efs-dev
