Man I haven't played with in YEARS... I don't know if the infrastructure's available, but I'll look into it.
Thanks for the pointer!! On Thu, Jan 5, 2012 at 12:06 PM, Kevin Green <[email protected]> wrote: > Do you have something like runsocks(1) available? That will socksify > the outbound trafic to the git:// protocol port so it gets through the > socks gateway. Try: > > $ runsocks git archive ... > > > e.g., have the same problem here: > > $ git archive --remote=git://git.openefs.org/efs-deploy-config.git --format > tar --prefix=test HEAD > /tmp/tarball.tar.gz > fatal: unable to connect to git.openefs.org: > git.openefs.org[0: 140.211.167.249]: errno=No route to host > > > $ runsocks git archive --remote=git://git.openefs.org/efs-deploy-config.git > --format tar --prefix=test HEAD > /tmp/tarball.tar.gz > $ ls -l /tmp/tarball.tar.gz > -rw-rw-r-- 1 kgreen cc7673 522240 Jan 5 12:06 /tmp/tarball.tar.gz > $ > > hth > > --Kevin > > On 01/05/12 10:43:14, Phillip Moore wrote: >> How this for timing... >> >> As soon as I sent this email, I started testing >> efsdeploy_config_update, only to discover that while http: URLs are >> proxied just fine of course, git: URLs are not. Since git archive >> only works with git: URLs, I am screwed. Getting this setup with our >> proxy infrastructure at work will take months, and more meetings than >> I want to be involved in, only to have it not work reliably... >> >> OK, enough cynicism. This means my hand is forced, and I am going to >> have to implement the tarball approach. >> >> This is one of those times when having an inactive development >> community is a win. I'm going to spend some time deploying the >> efs-site changes I made yesterday, which means I'm probably going to >> be screwing up the website, and the git repos (not in a destructive >> way, I hope :-) during the day. >> >> >> On Thu, Jan 5, 2012 at 10:34 AM, Phillip Moore >> <[email protected]> wrote: >> > Thanks for that information -- I didn't know about clone --depth. >> > >> > Well, in this particular case, once the git daemon was configured >> > correctly, I was able to make git archive work just fine, so that >> > scalability issue is resolved. git archive is very fast, since >> > there's no history involved. >> > >> > However, I do agree that git as the primary software distribution >> > mechanism is not a very good design. Among my many (too many) side >> > projects, I'm working on some post-receive hooks to automatically >> > publish the deploy-config data as timestamped tarballs (these git >> > repos are a little special). I have reworked the code that manages >> > the git repos, and once I get that deployed, I'll be able to introduce >> > some more hooks to do creative things. >> > >> > The efsdeploy_config_update script, which right now just supports >> > downloading via git, will be extended to support downloading the >> > published releases as well. If I may fantasize about actually having >> > some users some day (pause to stare the window and dream....) I would >> > expect sites that are actively involved in EFS development to use the >> > git based method, but sites that just want to use the published, >> > published data would use the tarballs. >> > >> > In any event, the mechanism's evolving very rapidly, and I'm still in >> > the experimental proof of concept phase with respect to managing the >> > efsdeploy config rules. I think this model is going to work, but >> > time will tell... >> > >> > On Wed, Jan 4, 2012 at 11:11 AM, Kevin Green >> > <[email protected]> wrote: >> >> Hi, >> >> >> >> Well, you could potentially work around the scale issue by using a >> >> shallow clone (--depth 1). I'll use git.git as an example since you >> >> have a small history repo with examples below: >> >> >> >> $ time git clone --depth 1 git://github.com/git/git.git >> >> Cloning into 'git'... >> >> remote: Counting objects: 26189, done. >> >> remote: Compressing objects: 100% (13591/13591), done. >> >> remote: Total 26189 (delta 21763), reused 15849 (delta 12193) >> >> Receiving objects: 100% (26189/26189), 9.05 MiB | 1.13 MiB/s, done. >> >> Resolving deltas: 100% (21763/21763), done. >> >> >> >> real 0m16.753s >> >> user 0m3.931s >> >> sys 0m0.609s >> >> $ cd git >> >> [master]$ git log >> >> commit 4570aeb0d85f3b5ff274b6d5a651c2ee06d25d76 >> >> Merge: 228c341 28755db >> >> Author: Junio C Hamano <[email protected]> >> >> Date: Tue Jan 3 14:09:28 2012 -0800 >> >> >> >> Merge branch 'pw/p4-docs-and-tests' >> >> >> >> * pw/p4-docs-and-tests: >> >> git-p4: document and test submit options >> >> git-p4: test and document --use-client-spec >> >> git-p4: test --keep-path >> >> git-p4: test --max-changes >> >> git-p4: document and test --import-local >> >> git-p4: honor --changesfile option and test >> >> git-p4: document and test clone --branch >> >> git-p4: test cloning with two dirs, clarify doc >> >> git-p4: clone does not use --git-dir >> >> git-p4: introduce asciidoc documentation >> >> rename git-p4 tests >> >> >> >> commit 228c3418356d06d0596408bee1c863e53ca27d58 >> >> Author: Junio C Hamano <[email protected]> >> >> Date: Tue Jan 3 13:48:00 2012 -0800 >> >> >> >> Merge branch 'maint' >> >> >> >> * maint: >> >> docs: describe behavior of relative submodule URLs >> >> fix hang in git fetch if pointed at a 0 length bundle >> >> Documentation: read-tree --prefix works with existing subtrees >> >> Add MYMETA.json to perl/.gitignore >> >> >> >> commit 28755dbaa5213032b2da202652c214a9f94ff853 >> >> Author: Pete Wyckoff <[email protected]> >> >> Date: Sat Dec 24 21:07:40 2011 -0500 >> >> >> >> git-p4: document and test submit options >> >> >> >> Clarify there is a -M option, but no -C. These are both >> >> configurable through variables. >> >> >> >> Explain that the allowSubmit variable takes a comma-separated >> >> list of branch names. >> >> >> >> Catch earlier an invalid branch name given as an argument to >> >> "git p4 clone". >> >> >> >> Test option --origin, variable allowSubmit, and explicit master >> >> branch name. >> >> >> >> Signed-off-by: Pete Wyckoff <[email protected]> >> >> Signed-off-by: Junio C Hamano <[email protected]> >> >> >> >> >> >> >> >> And then compare that with the time to check out the full repo: >> >> >> >> [master]$ cd .. >> >> $ rm -rf git >> >> $ time git clone git://github.com/git/git.git >> >> Cloning into 'git'... >> >> remote: Counting objects: 127389, done. >> >> remote: Compressing objects: 100% (41918/41918), done. >> >> remote: Total 127389 (delta 92731), reused 117665 (delta 83665) >> >> Receiving objects: 100% (127389/127389), 27.95 MiB | 1.35 MiB/s, done. >> >> Resolving deltas: 100% (92731/92731), done. >> >> >> >> real 0m46.661s >> >> user 0m14.107s >> >> sys 0m1.865s >> >> >> >> >> >> Since you don't care about the history in your use case, you can use a >> >> shallow >> >> clone to pull down the least amount of data necessary... >> >> >> >> >> >> >> >> I think the idea of providing a tarball on the server side is the way to >> >> go >> >> though... git really is a distributed code management tool meant for >> >> keeping >> >> track of change. It's not ideally suited for pure distribution. Use the >> >> simple >> >> git-archive (which also will do the gzip compression for you) on the >> >> backend, >> >> auto-generated by a git hook whenever code is updated there and just pull >> >> that >> >> down to the client. >> >> >> >> --Kevin >> >> >> >> On 01/04/12 10:23:42, Phillip Moore wrote: >> >>> Well, "git archive" comes very close to what we want, but it only >> >>> works against remote repositories using ssh, so that's not going to >> >>> work for any of the real world sites that are using (or hopefully will >> >>> soon be using) EFS. >> >>> >> >>> This really seems like a short coming in git, really. if you can >> >>> anonymously clone an entire repo, it should be easy to get just a >> >>> working directory for the HEAD of master anonymously, too. >> >>> >> >>> I think we need to come up with a mechanism for auto-generating a >> >>> "latest" tarball for each of these via a commit hook, so I'll go take >> >>> a look at the code Jerry wrote to implement the hooks we have today, >> >>> and see how we extend that to add a new one. The creation of the >> >>> tarball will end up being a VERY short script, since a one-liner with >> >>> git/gzip can create it. >> >>> >> >>> On Wed, Jan 4, 2012 at 9:55 AM, Phillip Moore >> >>> <[email protected]> wrote: >> >>> > This is a great idea except that I have no clue how git works, >> >>> > obviously.... >> >>> > >> >>> > I had confused "git checkout" with "svn export", and now that I look, >> >>> > I can't find a way to accomlish this after all. What I wanted might >> >>> > not be possible with git -- namely a way to download the repo, and >> >>> > just get a working tree with no repo metadata. >> >>> > >> >>> > What I want is the equivalant result of "svn export", which gives you >> >>> > HEAD of your SVN repo, without all the .svn dirs. >> >>> > >> >>> > Now, obviously, you can do this: >> >>> > >> >>> > git clone $url . >> >>> > rm -r .git >> >>> > >> >>> > But that will NEVER scale, as the size of the git history grows. >> >>> > >> >>> > Maybe the better mechanism is to have a commit hook which does this, >> >>> > and publishes a tarball on ftp.openefs.org with a "latest" symlink. >> >>> > Then the code can use wget and tar to achieve this goal, rather than >> >>> > using git directly. >> >>> > >> >>> > If one of you knows of a means to do this using git, directly, please >> >>> > let me know. I will continue researching this... >> >>> > >> >>> > On Wed, Jan 4, 2012 at 8:28 AM, Phillip Moore >> >>> > <[email protected]> wrote: >> >>> >> I came up with an alternate way to manage deploying these >> >>> >> deploy-config projects, that will make it trivial to keep them >> >>> >> uptodate, AND deal with the fact that we're managing them in multiple >> >>> >> repos. >> >>> >> >> >>> >> First of all, for flexibility, I'm still going to implement the search >> >>> >> mechanism for the efsdeploy directory as I described before. However, >> >>> >> based on the way I've structured the git repos, you can actually do a >> >>> >> "git checkout" and drop them all into the same root directory? >> >>> >> >> >>> >> I'm going to try this today, since it so damn simple. >> >>> >> efsdeploy_config_update will be the script that does the following: >> >>> >> >> >>> >> efs create autorelease efs deploy-config >> >>> >> cd /efs/dev/efs/deploy-config/next/install/common >> >>> >> git checkout http://git.openefs.org/deploy-config >> >>> >> git checkout http://git.openefs.org/deploy-config-aix >> >>> >> git checkout http://git.openefs.org/deploy-config-gnu >> >>> >> .... >> >>> >> efs dist autorelease efs deploy-config >> >>> >> >> >>> >> Now, you have /efs/dist/efs/deploy-config/current/common with ALL of >> >>> >> the published git configs. >> >>> >> >> >>> >> Note that because ALL of these repos are structures with a >> >>> >> metaproj/project structure, they can ALL co-exist in the same >> >>> >> directory tree (if you use checkout, I think -- I haven't tried this >> >>> >> yet, but since you don't get the .git directory, I don't see why this >> >>> >> won't work -- I'll figure out how to make it work :-P) >> >>> >> >> >>> >> Even better, we can drop a simple file into the root of each repo, >> >>> >> giving the name of the "child" repos in the obvious hierarchy here. >> >>> >> For example, in the root of deploy-config, the contents of >> >>> >> subrepos.txt might be: >> >>> >> >> >>> >> deploy-config-aix >> >>> >> deploy-config-gnu >> >>> >> deploy-config-rhel >> >>> >> deploy-config-sunos >> >>> >> >> >>> >> The subrepos.txt file in deploy-config-gnu will have to live in the >> >>> >> gnu subdir, to avoid clashes, but then, since the top tells us to >> >>> >> checkout deploy-config-gnu, we then know to look for the next >> >>> >> subrepos.txt file in ./gnu. This will then contain: >> >>> >> >> >>> >> deploy-config-gnu-gcc >> >>> >> deploy-config-gnu-gcclib >> >>> >> >> >>> >> This will give us the full flexibility of an easy to use, well managed >> >>> >> default (you only get the published, commited master branch), with the >> >>> >> ability to create and manage your own local repos as well. For >> >>> >> example, there will never be an "fsf" metaproj in the OpenEFS >> >>> >> namespace, and in practice, you've going to be migrating stuff to gnu, >> >>> >> I assume, but if you wanted to maintain your own deploy-config-fsf git >> >>> >> repo, that works fine. You would simply manage it in: >> >>> >> >> >>> >> /efs/dist/fsf/deploy-config-fsf >> >>> >> >> >>> >> I can even support publishing this using efsdeploy_config_update via >> >>> >> CLI args, if you wanted to use the same, simple mechanism. >> >>> >> >> >>> >> This is starting to come together very nicely, and now all we really >> >>> >> need are.... >> >>> >> >> >>> >> Users :-( >> >>> >> >> >>> >> On Fri, Dec 30, 2011 at 12:57 PM, Phillip Moore >> >>> >> <[email protected]> wrote: >> >>> >>> On Fri, Dec 30, 2011 at 12:09 PM, Phillip Moore >> >>> >>> <[email protected]> wrote: >> >>> >>>> More thoughts, and some significant progress in this area.... >> >>> >>>> >> >>> >>>> I spent most of yesterday collecting the efsedploy rules for >> >>> >>>> EVERYTHING I've built into /efs/dist over the last few months (it's >> >>> >>>> a >> >>> >>>> lot), by copying the src directory to: >> >>> >>>> >> >>> >>>> ~/dev/efs/deploy-config/$metaproj/$project >> >>> >>> >> >>> >>> OK, so once everything in that directory has been sanitized of ALL >> >>> >>> site-specific information, then we have to figure out how to manage >> >>> >>> it. Here's what I'm currently thinking, although this is going to >> >>> >>> evolve, of course. >> >>> >>> >> >>> >>> First of all, note that efsdeploy is going to start whining at you to >> >>> >>> switch from efs/deploy-config to efs/deploy-site, because I want to >> >>> >>> use the name deploy-config for all of this data. Deal with it.... >> >>> >>> It's *trivial* to switch, and takes about 5-10 minutes, if you type >> >>> >>> slow. >> >>> >>> >> >>> >>> I want to create 3 types of git repo to manage this data: >> >>> >>> >> >>> >>> deploy-config-$metaproj-$project.git >> >>> >>> deploy-config-$metaproj.git >> >>> >>> deploy-config.git >> >>> >>> >> >>> >>> For things like gnu/gcc, we'll obviously create a project-specific >> >>> >>> git >> >>> >>> repo, and for large metaprojs where we expect a lot of similarity >> >>> >>> among the projects, we can create metaproj-specific ones. The >> >>> >>> default, global git repo would contain all the small, simple stuff, >> >>> >>> like oss/zlib. For starters, I expect to create these: >> >>> >>> >> >>> >>> deploy-config-gnu-gcc.git (which will be used for rhel/gcc as >> >>> >>> well) >> >>> >>> deploy-config-gnu-gcclib.git (also for rhel/gcclib) >> >>> >>> deploy-config-gnu.git >> >>> >>> deploy-config-perl5-core.git >> >>> >>> deploy-config-perl5.git >> >>> >>> deploy-config-apache.git (might get it's own system, too -- we'll >> >>> >>> see...) >> >>> >>> >> >>> >>> And of course the generic one. What I like about this is we always >> >>> >>> migrate things from one to the other pretty easily. if we find that, >> >>> >>> say oss/openssl has grown complex enough, we can yank it out of >> >>> >>> deploy-config, and create deploy-config-oss-openssl. >> >>> >>> >> >>> >>> So how do we deploy this data? Having it well managed is git is >> >>> >>> great, but how to we access it when building things with efsdeploy, >> >>> >>> and where does it get copied/cached? >> >>> >>> >> >>> >>> Let's start with the generic repo first. Just as we use >> >>> >>> efs/deploy-site/current to abstract the site-specific config >> >>> >>> information, I think we can do the following: >> >>> >>> >> >>> >>> deploy-config.git => /efs/dist/efs/deploy-config/current >> >>> >>> >> >>> >>> The metaproj- and project-specific ones would then map to: >> >>> >>> >> >>> >>> deploy-config-$metaproj.git => >> >>> >>> /efs/dist/$metaproj/deploy-config-$metaproj/current >> >>> >>> deploy-config-$metaproj-project.git => >> >>> >>> /efs/dist/$metaproj/deploy-config-$metaproj-$project/current >> >>> >>> >> >>> >>> This would allow us to publish, probably date-based, any of these >> >>> >>> repositories with the "latest" set of efsdeploy build rules. >> >>> >>> Note that the default rules go into the efs metaproj, obviously, but >> >>> >>> we can still have a "deploy-config-efs.git" repo if we want, with no >> >>> >>> conflict. >> >>> >>> >> >>> >>> It is very straight forward to code a solution that allows us to >> >>> >>> automate keeping the local copies of these rules uptodate as they >> >>> >>> change. I will almost certainly have a first pass at this within >> >>> >>> the >> >>> >>> next month. However, what is NOT clear is just how to use this >> >>> >>> information in efsdeploy when building release. >> >>> >>> >> >>> >>> Reproducibility concerns me. The rules are going to evolve, and when >> >>> >>> we make gnu/gcc rule changes to build, say 4.7.0, we don't want to >> >>> >>> break builds of 4.4.6, and yet *testing* that is extremely expensive. >> >>> >>> For that reason, I think the contents of the efsdeploy directory >> >>> >>> should be CACHED in the release, rather than read from these projects >> >>> >>> during the build. Just as we are going to provide generic >> >>> >>> dependency >> >>> >>> specs (see email from 30 minutes ago), and expanding those into >> >>> >>> specific releasealiases to be used for the duration of the build, I >> >>> >>> think we should do the same for the project-specific build rules, or >> >>> >>> at least make it optional. >> >>> >>> >> >>> >>> In theory, if we just have efsdeploy search for these rules the same >> >>> >>> way it searches for system-specific (i,e, gnu, perl5, etc) rules, and >> >>> >>> then site-specific rules, then I could actually build EVERYTHING I >> >>> >>> have in /efs/dist with EMPTY source directories!! If a project is >> >>> >>> supported by one of these repos, then you can build a new release >> >>> >>> with >> >>> >>> nothing more than: >> >>> >>> >> >>> >>> efs create project ... >> >>> >>> efs create release ... >> >>> >>> cd ..../src >> >>> >>> efsdeploy down:up >> >>> >>> >> >>> >>> The contents of the src directory would contain NOTHING but the >> >>> >>> changes you had to make (hooks, configs, whatever) to get the release >> >>> >>> to build. Those changes should then be re-integrated with the git >> >>> >>> repo in a controlled fashion, so that the next person building that >> >>> >>> MPR has no pain. The specific workflow for how a new change gets >> >>> >>> rolled into the published git repos will need to be worked out, but I >> >>> >>> think that will be straight forward. >> >>> >>> >> >>> >>> Now, obviously, in order to *develop* changes to the rules, we'll >> >>> >>> need >> >>> >>> a simply means of overriding the path to these published rules. >> >>> >>> Maybe you want to install the latest set of gnu/gcc rules, but not >> >>> >>> make them current until you've actually done a test-build of the >> >>> >>> releases you care about. Maybe something in efsdeploy.conf (which >> >>> >>> will now be a site/release-specific file, by definition) like this. >> >>> >>> Say we wanted to test out some local changes right from the source >> >>> >>> tree (I've been doing this with symlinks for now): >> >>> >>> >> >>> >>> [rules] >> >>> >>> $metaproj/$project = /home/efsops/dev/efs/deploy-config-gnu-gcc >> >>> >>> >> >>> >>> or, perhaps, if we use date-based releases, you could install the >> >>> >>> latest update into /efs/dist, and test it out this way: >> >>> >>> >> >>> >>> [rules] >> >>> >>> $metaproj/$project = /efs/dist/gnu/deploy-config-gnu-gcc/20111230 >> >>> >>> >> >>> >>> Alternately, you could just rsync the efsdeploy directory right into >> >>> >>> a >> >>> >>> release, and work with a copy. >> >>> >>> >> >>> >>> OK, that's enough of Phil's rantings for one day. Not that anyone's >> >>> >>> paying attention, but you will see commits that implement many of >> >>> >>> these features over the next few weeks. >> >>> _______________________________________________ >> >>> EFS-dev mailing list >> >>> [email protected] >> >>> http://mailman.openefs.org/mailman/listinfo/efs-dev >> >> _______________________________________________ >> >> EFS-dev mailing list >> >> [email protected] >> >> http://mailman.openefs.org/mailman/listinfo/efs-dev >> _______________________________________________ >> EFS-dev mailing list >> [email protected] >> http://mailman.openefs.org/mailman/listinfo/efs-dev > _______________________________________________ > EFS-dev mailing list > [email protected] > http://mailman.openefs.org/mailman/listinfo/efs-dev _______________________________________________ EFS-dev mailing list [email protected] http://mailman.openefs.org/mailman/listinfo/efs-dev
