Yeah, but not anonymously. The remote spec won't work with a git: or http: URL
pmoore@Renegade$ git archive --remote=git://git.openefs.org/efs-deploy-config.git --format tar --prefix=test | tar vxf - fatal: read error: Connection reset by peer Right now, I think a commit hook that intelligently publishes a date-based tarball, using something like this: git archive --format=tar --prefix=. | gzip -c > ~/data/packages/releases/$repo-YYYYMMDD-HHMMSS.tar.gz I'd add a $repo-latest.tar.gz symlink, and keep 2 or 3 old versions. Then, we simply update a trivial little file ~/data/packages/releases/index.txt which lists all the file names in the ~/data/packages/releases directory. This will allow us to get the list without parsing any damn HTML (yuck). Just wget http://ftp.openefs.org/packages/releases/index.txt and now you have the complete list of what's been published. The mechanism to populate the /efs/dev/efs/deploy-config autoreleases will now download these tarballs into /efs/dev/efs/deploy-config/next/src. When determining whether or not there's anything to update, we're doing a very simple comparison of the filenames in the index.txt file, and the contents of that src directory. if anything's changed, then we go ahead and create a new autorelease, download the changes, and install. On Wed, Jan 4, 2012 at 10:09 AM, DENNIS SHIM, BLOOMBERG/ 731 LEXIN <[email protected]> wrote: > I am in the process of getting familiar with git myself, but I think 'git > archive' will get you the repo without the .git metadata. > > > ----- Original Message ----- > From: [email protected] > To: [email protected] > At: 1/04 9:55:59 > > This is a great idea except that I have no clue how git works, obviously.... > > I had confused "git checkout" with "svn export", and now that I look, > I can't find a way to accomlish this after all. What I wanted might > not be possible with git -- namely a way to download the repo, and > just get a working tree with no repo metadata. > > What I want is the equivalant result of "svn export", which gives you > HEAD of your SVN repo, without all the .svn dirs. > > Now, obviously, you can do this: > > git clone $url . > rm -r .git > > But that will NEVER scale, as the size of the git history grows. > > Maybe the better mechanism is to have a commit hook which does this, > and publishes a tarball on ftp.openefs.org with a "latest" symlink. > Then the code can use wget and tar to achieve this goal, rather than > using git directly. > > If one of you knows of a means to do this using git, directly, please > let me know. I will continue researching this... > > On Wed, Jan 4, 2012 at 8:28 AM, Phillip Moore <[email protected]> > wrote: >> I came up with an alternate way to manage deploying these >> deploy-config projects, that will make it trivial to keep them >> uptodate, AND deal with the fact that we're managing them in multiple >> repos. >> >> First of all, for flexibility, I'm still going to implement the search >> mechanism for the efsdeploy directory as I described before. However, >> based on the way I've structured the git repos, you can actually do a >> "git checkout" and drop them all into the same root directory? >> >> I'm going to try this today, since it so damn simple. >> efsdeploy_config_update will be the script that does the following: >> >> efs create autorelease efs deploy-config >> cd /efs/dev/efs/deploy-config/next/install/common >> git checkout http://git.openefs.org/deploy-config >> git checkout http://git.openefs.org/deploy-config-aix >> git checkout http://git.openefs.org/deploy-config-gnu >> .... >> efs dist autorelease efs deploy-config >> >> Now, you have /efs/dist/efs/deploy-config/current/common with ALL of >> the published git configs. >> >> Note that because ALL of these repos are structures with a >> metaproj/project structure, they can ALL co-exist in the same >> directory tree (if you use checkout, I think -- I haven't tried this >> yet, but since you don't get the .git directory, I don't see why this >> won't work -- I'll figure out how to make it work :-P) >> >> Even better, we can drop a simple file into the root of each repo, >> giving the name of the "child" repos in the obvious hierarchy here. >> For example, in the root of deploy-config, the contents of >> subrepos.txt might be: >> >> deploy-config-aix >> deploy-config-gnu >> deploy-config-rhel >> deploy-config-sunos >> >> The subrepos.txt file in deploy-config-gnu will have to live in the >> gnu subdir, to avoid clashes, but then, since the top tells us to >> checkout deploy-config-gnu, we then know to look for the next >> subrepos.txt file in ./gnu. This will then contain: >> >> deploy-config-gnu-gcc >> deploy-config-gnu-gcclib >> >> This will give us the full flexibility of an easy to use, well managed >> default (you only get the published, commited master branch), with the >> ability to create and manage your own local repos as well. For >> example, there will never be an "fsf" metaproj in the OpenEFS >> namespace, and in practice, you've going to be migrating stuff to gnu, >> I assume, but if you wanted to maintain your own deploy-config-fsf git >> repo, that works fine. You would simply manage it in: >> >> /efs/dist/fsf/deploy-config-fsf >> >> I can even support publishing this using efsdeploy_config_update via >> CLI args, if you wanted to use the same, simple mechanism. >> >> This is starting to come together very nicely, and now all we really >> need are.... >> >> Users :-( >> >> On Fri, Dec 30, 2011 at 12:57 PM, Phillip Moore >> <[email protected]> wrote: >>> On Fri, Dec 30, 2011 at 12:09 PM, Phillip Moore >>> <[email protected]> wrote: >>>> More thoughts, and some significant progress in this area.... >>>> >>>> I spent most of yesterday collecting the efsedploy rules for >>>> EVERYTHING I've built into /efs/dist over the last few months (it's a >>>> lot), by copying the src directory to: >>>> >>>> ~/dev/efs/deploy-config/$metaproj/$project >>> >>> OK, so once everything in that directory has been sanitized of ALL >>> site-specific information, then we have to figure out how to manage >>> it. Here's what I'm currently thinking, although this is going to >>> evolve, of course. >>> >>> First of all, note that efsdeploy is going to start whining at you to >>> switch from efs/deploy-config to efs/deploy-site, because I want to >>> use the name deploy-config for all of this data. Deal with it.... >>> It's *trivial* to switch, and takes about 5-10 minutes, if you type >>> slow. >>> >>> I want to create 3 types of git repo to manage this data: >>> >>> deploy-config-$metaproj-$project.git >>> deploy-config-$metaproj.git >>> deploy-config.git >>> >>> For things like gnu/gcc, we'll obviously create a project-specific git >>> repo, and for large metaprojs where we expect a lot of similarity >>> among the projects, we can create metaproj-specific ones. The >>> default, global git repo would contain all the small, simple stuff, >>> like oss/zlib. For starters, I expect to create these: >>> >>> deploy-config-gnu-gcc.git (which will be used for rhel/gcc as well) >>> deploy-config-gnu-gcclib.git (also for rhel/gcclib) >>> deploy-config-gnu.git >>> deploy-config-perl5-core.git >>> deploy-config-perl5.git >>> deploy-config-apache.git (might get it's own system, too -- we'll see...) >>> >>> And of course the generic one. What I like about this is we always >>> migrate things from one to the other pretty easily. if we find that, >>> say oss/openssl has grown complex enough, we can yank it out of >>> deploy-config, and create deploy-config-oss-openssl. >>> >>> So how do we deploy this data? Having it well managed is git is >>> great, but how to we access it when building things with efsdeploy, >>> and where does it get copied/cached? >>> >>> Let's start with the generic repo first. Just as we use >>> efs/deploy-site/current to abstract the site-specific config >>> information, I think we can do the following: >>> >>> deploy-config.git => /efs/dist/efs/deploy-config/current >>> >>> The metaproj- and project-specific ones would then map to: >>> >>> deploy-config-$metaproj.git => >>> /efs/dist/$metaproj/deploy-config-$metaproj/current >>> deploy-config-$metaproj-project.git => >>> /efs/dist/$metaproj/deploy-config-$metaproj-$project/current >>> >>> This would allow us to publish, probably date-based, any of these >>> repositories with the "latest" set of efsdeploy build rules. >>> Note that the default rules go into the efs metaproj, obviously, but >>> we can still have a "deploy-config-efs.git" repo if we want, with no >>> conflict. >>> >>> It is very straight forward to code a solution that allows us to >>> automate keeping the local copies of these rules uptodate as they >>> change. I will almost certainly have a first pass at this within the >>> next month. However, what is NOT clear is just how to use this >>> information in efsdeploy when building release. >>> >>> Reproducibility concerns me. The rules are going to evolve, and when >>> we make gnu/gcc rule changes to build, say 4.7.0, we don't want to >>> break builds of 4.4.6, and yet *testing* that is extremely expensive. >>> For that reason, I think the contents of the efsdeploy directory >>> should be CACHED in the release, rather than read from these projects >>> during the build. Just as we are going to provide generic dependency >>> specs (see email from 30 minutes ago), and expanding those into >>> specific releasealiases to be used for the duration of the build, I >>> think we should do the same for the project-specific build rules, or >>> at least make it optional. >>> >>> In theory, if we just have efsdeploy search for these rules the same >>> way it searches for system-specific (i,e, gnu, perl5, etc) rules, and >>> then site-specific rules, then I could actually build EVERYTHING I >>> have in /efs/dist with EMPTY source directories!! If a project is >>> supported by one of these repos, then you can build a new release with >>> nothing more than: >>> >>> efs create project ... >>> efs create release ... >>> cd ..../src >>> efsdeploy down:up >>> >>> The contents of the src directory would contain NOTHING but the >>> changes you had to make (hooks, configs, whatever) to get the release >>> to build. Those changes should then be re-integrated with the git >>> repo in a controlled fashion, so that the next person building that >>> MPR has no pain. The specific workflow for how a new change gets >>> rolled into the published git repos will need to be worked out, but I >>> think that will be straight forward. >>> >>> Now, obviously, in order to *develop* changes to the rules, we'll need >>> a simply means of overriding the path to these published rules. >>> Maybe you want to install the latest set of gnu/gcc rules, but not >>> make them current until you've actually done a test-build of the >>> releases you care about. Maybe something in efsdeploy.conf (which >>> will now be a site/release-specific file, by definition) like this. >>> Say we wanted to test out some local changes right from the source >>> tree (I've been doing this with symlinks for now): >>> >>> [rules] >>> $metaproj/$project = /home/efsops/dev/efs/deploy-config-gnu-gcc >>> >>> or, perhaps, if we use date-based releases, you could install the >>> latest update into /efs/dist, and test it out this way: >>> >>> [rules] >>> $metaproj/$project = /efs/dist/gnu/deploy-config-gnu-gcc/20111230 >>> >>> Alternately, you could just rsync the efsdeploy directory right into a >>> release, and work with a copy. >>> >>> OK, that's enough of Phil's rantings for one day. Not that anyone's >>> paying attention, but you will see commits that implement many of >>> these features over the next few weeks. > _______________________________________________ > EFS-dev mailing list > [email protected] > http://mailman.openefs.org/mailman/listinfo/efs-dev > _______________________________________________ > EFS-dev mailing list > [email protected] > http://mailman.openefs.org/mailman/listinfo/efs-dev _______________________________________________ EFS-dev mailing list [email protected] http://mailman.openefs.org/mailman/listinfo/efs-dev
