Do you have something like runsocks(1) available? That will socksify the outbound trafic to the git:// protocol port so it gets through the socks gateway. Try:
$ runsocks git archive ... e.g., have the same problem here: $ git archive --remote=git://git.openefs.org/efs-deploy-config.git --format tar --prefix=test HEAD > /tmp/tarball.tar.gz fatal: unable to connect to git.openefs.org: git.openefs.org[0: 140.211.167.249]: errno=No route to host $ runsocks git archive --remote=git://git.openefs.org/efs-deploy-config.git --format tar --prefix=test HEAD > /tmp/tarball.tar.gz $ ls -l /tmp/tarball.tar.gz -rw-rw-r-- 1 kgreen cc7673 522240 Jan 5 12:06 /tmp/tarball.tar.gz $ hth --Kevin On 01/05/12 10:43:14, Phillip Moore wrote: > How this for timing... > > As soon as I sent this email, I started testing > efsdeploy_config_update, only to discover that while http: URLs are > proxied just fine of course, git: URLs are not. Since git archive > only works with git: URLs, I am screwed. Getting this setup with our > proxy infrastructure at work will take months, and more meetings than > I want to be involved in, only to have it not work reliably... > > OK, enough cynicism. This means my hand is forced, and I am going to > have to implement the tarball approach. > > This is one of those times when having an inactive development > community is a win. I'm going to spend some time deploying the > efs-site changes I made yesterday, which means I'm probably going to > be screwing up the website, and the git repos (not in a destructive > way, I hope :-) during the day. > > > On Thu, Jan 5, 2012 at 10:34 AM, Phillip Moore > <[email protected]> wrote: > > Thanks for that information -- I didn't know about clone --depth. > > > > Well, in this particular case, once the git daemon was configured > > correctly, I was able to make git archive work just fine, so that > > scalability issue is resolved. git archive is very fast, since > > there's no history involved. > > > > However, I do agree that git as the primary software distribution > > mechanism is not a very good design. Among my many (too many) side > > projects, I'm working on some post-receive hooks to automatically > > publish the deploy-config data as timestamped tarballs (these git > > repos are a little special). I have reworked the code that manages > > the git repos, and once I get that deployed, I'll be able to introduce > > some more hooks to do creative things. > > > > The efsdeploy_config_update script, which right now just supports > > downloading via git, will be extended to support downloading the > > published releases as well. If I may fantasize about actually having > > some users some day (pause to stare the window and dream....) I would > > expect sites that are actively involved in EFS development to use the > > git based method, but sites that just want to use the published, > > published data would use the tarballs. > > > > In any event, the mechanism's evolving very rapidly, and I'm still in > > the experimental proof of concept phase with respect to managing the > > efsdeploy config rules. I think this model is going to work, but > > time will tell... > > > > On Wed, Jan 4, 2012 at 11:11 AM, Kevin Green > > <[email protected]> wrote: > >> Hi, > >> > >> Well, you could potentially work around the scale issue by using a > >> shallow clone (--depth 1). I'll use git.git as an example since you > >> have a small history repo with examples below: > >> > >> $ time git clone --depth 1 git://github.com/git/git.git > >> Cloning into 'git'... > >> remote: Counting objects: 26189, done. > >> remote: Compressing objects: 100% (13591/13591), done. > >> remote: Total 26189 (delta 21763), reused 15849 (delta 12193) > >> Receiving objects: 100% (26189/26189), 9.05 MiB | 1.13 MiB/s, done. > >> Resolving deltas: 100% (21763/21763), done. > >> > >> real 0m16.753s > >> user 0m3.931s > >> sys 0m0.609s > >> $ cd git > >> [master]$ git log > >> commit 4570aeb0d85f3b5ff274b6d5a651c2ee06d25d76 > >> Merge: 228c341 28755db > >> Author: Junio C Hamano <[email protected]> > >> Date: Tue Jan 3 14:09:28 2012 -0800 > >> > >> Merge branch 'pw/p4-docs-and-tests' > >> > >> * pw/p4-docs-and-tests: > >> git-p4: document and test submit options > >> git-p4: test and document --use-client-spec > >> git-p4: test --keep-path > >> git-p4: test --max-changes > >> git-p4: document and test --import-local > >> git-p4: honor --changesfile option and test > >> git-p4: document and test clone --branch > >> git-p4: test cloning with two dirs, clarify doc > >> git-p4: clone does not use --git-dir > >> git-p4: introduce asciidoc documentation > >> rename git-p4 tests > >> > >> commit 228c3418356d06d0596408bee1c863e53ca27d58 > >> Author: Junio C Hamano <[email protected]> > >> Date: Tue Jan 3 13:48:00 2012 -0800 > >> > >> Merge branch 'maint' > >> > >> * maint: > >> docs: describe behavior of relative submodule URLs > >> fix hang in git fetch if pointed at a 0 length bundle > >> Documentation: read-tree --prefix works with existing subtrees > >> Add MYMETA.json to perl/.gitignore > >> > >> commit 28755dbaa5213032b2da202652c214a9f94ff853 > >> Author: Pete Wyckoff <[email protected]> > >> Date: Sat Dec 24 21:07:40 2011 -0500 > >> > >> git-p4: document and test submit options > >> > >> Clarify there is a -M option, but no -C. These are both > >> configurable through variables. > >> > >> Explain that the allowSubmit variable takes a comma-separated > >> list of branch names. > >> > >> Catch earlier an invalid branch name given as an argument to > >> "git p4 clone". > >> > >> Test option --origin, variable allowSubmit, and explicit master > >> branch name. > >> > >> Signed-off-by: Pete Wyckoff <[email protected]> > >> Signed-off-by: Junio C Hamano <[email protected]> > >> > >> > >> > >> And then compare that with the time to check out the full repo: > >> > >> [master]$ cd .. > >> $ rm -rf git > >> $ time git clone git://github.com/git/git.git > >> Cloning into 'git'... > >> remote: Counting objects: 127389, done. > >> remote: Compressing objects: 100% (41918/41918), done. > >> remote: Total 127389 (delta 92731), reused 117665 (delta 83665) > >> Receiving objects: 100% (127389/127389), 27.95 MiB | 1.35 MiB/s, done. > >> Resolving deltas: 100% (92731/92731), done. > >> > >> real 0m46.661s > >> user 0m14.107s > >> sys 0m1.865s > >> > >> > >> Since you don't care about the history in your use case, you can use a > >> shallow > >> clone to pull down the least amount of data necessary... > >> > >> > >> > >> I think the idea of providing a tarball on the server side is the way to go > >> though... git really is a distributed code management tool meant for > >> keeping > >> track of change. It's not ideally suited for pure distribution. Use the > >> simple > >> git-archive (which also will do the gzip compression for you) on the > >> backend, > >> auto-generated by a git hook whenever code is updated there and just pull > >> that > >> down to the client. > >> > >> --Kevin > >> > >> On 01/04/12 10:23:42, Phillip Moore wrote: > >>> Well, "git archive" comes very close to what we want, but it only > >>> works against remote repositories using ssh, so that's not going to > >>> work for any of the real world sites that are using (or hopefully will > >>> soon be using) EFS. > >>> > >>> This really seems like a short coming in git, really. if you can > >>> anonymously clone an entire repo, it should be easy to get just a > >>> working directory for the HEAD of master anonymously, too. > >>> > >>> I think we need to come up with a mechanism for auto-generating a > >>> "latest" tarball for each of these via a commit hook, so I'll go take > >>> a look at the code Jerry wrote to implement the hooks we have today, > >>> and see how we extend that to add a new one. The creation of the > >>> tarball will end up being a VERY short script, since a one-liner with > >>> git/gzip can create it. > >>> > >>> On Wed, Jan 4, 2012 at 9:55 AM, Phillip Moore <[email protected]> > >>> wrote: > >>> > This is a great idea except that I have no clue how git works, > >>> > obviously.... > >>> > > >>> > I had confused "git checkout" with "svn export", and now that I look, > >>> > I can't find a way to accomlish this after all. What I wanted might > >>> > not be possible with git -- namely a way to download the repo, and > >>> > just get a working tree with no repo metadata. > >>> > > >>> > What I want is the equivalant result of "svn export", which gives you > >>> > HEAD of your SVN repo, without all the .svn dirs. > >>> > > >>> > Now, obviously, you can do this: > >>> > > >>> > git clone $url . > >>> > rm -r .git > >>> > > >>> > But that will NEVER scale, as the size of the git history grows. > >>> > > >>> > Maybe the better mechanism is to have a commit hook which does this, > >>> > and publishes a tarball on ftp.openefs.org with a "latest" symlink. > >>> > Then the code can use wget and tar to achieve this goal, rather than > >>> > using git directly. > >>> > > >>> > If one of you knows of a means to do this using git, directly, please > >>> > let me know. I will continue researching this... > >>> > > >>> > On Wed, Jan 4, 2012 at 8:28 AM, Phillip Moore > >>> > <[email protected]> wrote: > >>> >> I came up with an alternate way to manage deploying these > >>> >> deploy-config projects, that will make it trivial to keep them > >>> >> uptodate, AND deal with the fact that we're managing them in multiple > >>> >> repos. > >>> >> > >>> >> First of all, for flexibility, I'm still going to implement the search > >>> >> mechanism for the efsdeploy directory as I described before. However, > >>> >> based on the way I've structured the git repos, you can actually do a > >>> >> "git checkout" and drop them all into the same root directory? > >>> >> > >>> >> I'm going to try this today, since it so damn simple. > >>> >> efsdeploy_config_update will be the script that does the following: > >>> >> > >>> >> efs create autorelease efs deploy-config > >>> >> cd /efs/dev/efs/deploy-config/next/install/common > >>> >> git checkout http://git.openefs.org/deploy-config > >>> >> git checkout http://git.openefs.org/deploy-config-aix > >>> >> git checkout http://git.openefs.org/deploy-config-gnu > >>> >> .... > >>> >> efs dist autorelease efs deploy-config > >>> >> > >>> >> Now, you have /efs/dist/efs/deploy-config/current/common with ALL of > >>> >> the published git configs. > >>> >> > >>> >> Note that because ALL of these repos are structures with a > >>> >> metaproj/project structure, they can ALL co-exist in the same > >>> >> directory tree (if you use checkout, I think -- I haven't tried this > >>> >> yet, but since you don't get the .git directory, I don't see why this > >>> >> won't work -- I'll figure out how to make it work :-P) > >>> >> > >>> >> Even better, we can drop a simple file into the root of each repo, > >>> >> giving the name of the "child" repos in the obvious hierarchy here. > >>> >> For example, in the root of deploy-config, the contents of > >>> >> subrepos.txt might be: > >>> >> > >>> >> deploy-config-aix > >>> >> deploy-config-gnu > >>> >> deploy-config-rhel > >>> >> deploy-config-sunos > >>> >> > >>> >> The subrepos.txt file in deploy-config-gnu will have to live in the > >>> >> gnu subdir, to avoid clashes, but then, since the top tells us to > >>> >> checkout deploy-config-gnu, we then know to look for the next > >>> >> subrepos.txt file in ./gnu. This will then contain: > >>> >> > >>> >> deploy-config-gnu-gcc > >>> >> deploy-config-gnu-gcclib > >>> >> > >>> >> This will give us the full flexibility of an easy to use, well managed > >>> >> default (you only get the published, commited master branch), with the > >>> >> ability to create and manage your own local repos as well. For > >>> >> example, there will never be an "fsf" metaproj in the OpenEFS > >>> >> namespace, and in practice, you've going to be migrating stuff to gnu, > >>> >> I assume, but if you wanted to maintain your own deploy-config-fsf git > >>> >> repo, that works fine. You would simply manage it in: > >>> >> > >>> >> /efs/dist/fsf/deploy-config-fsf > >>> >> > >>> >> I can even support publishing this using efsdeploy_config_update via > >>> >> CLI args, if you wanted to use the same, simple mechanism. > >>> >> > >>> >> This is starting to come together very nicely, and now all we really > >>> >> need are.... > >>> >> > >>> >> Users :-( > >>> >> > >>> >> On Fri, Dec 30, 2011 at 12:57 PM, Phillip Moore > >>> >> <[email protected]> wrote: > >>> >>> On Fri, Dec 30, 2011 at 12:09 PM, Phillip Moore > >>> >>> <[email protected]> wrote: > >>> >>>> More thoughts, and some significant progress in this area.... > >>> >>>> > >>> >>>> I spent most of yesterday collecting the efsedploy rules for > >>> >>>> EVERYTHING I've built into /efs/dist over the last few months (it's a > >>> >>>> lot), by copying the src directory to: > >>> >>>> > >>> >>>> ~/dev/efs/deploy-config/$metaproj/$project > >>> >>> > >>> >>> OK, so once everything in that directory has been sanitized of ALL > >>> >>> site-specific information, then we have to figure out how to manage > >>> >>> it. Here's what I'm currently thinking, although this is going to > >>> >>> evolve, of course. > >>> >>> > >>> >>> First of all, note that efsdeploy is going to start whining at you to > >>> >>> switch from efs/deploy-config to efs/deploy-site, because I want to > >>> >>> use the name deploy-config for all of this data. Deal with it.... > >>> >>> It's *trivial* to switch, and takes about 5-10 minutes, if you type > >>> >>> slow. > >>> >>> > >>> >>> I want to create 3 types of git repo to manage this data: > >>> >>> > >>> >>> deploy-config-$metaproj-$project.git > >>> >>> deploy-config-$metaproj.git > >>> >>> deploy-config.git > >>> >>> > >>> >>> For things like gnu/gcc, we'll obviously create a project-specific git > >>> >>> repo, and for large metaprojs where we expect a lot of similarity > >>> >>> among the projects, we can create metaproj-specific ones. The > >>> >>> default, global git repo would contain all the small, simple stuff, > >>> >>> like oss/zlib. For starters, I expect to create these: > >>> >>> > >>> >>> deploy-config-gnu-gcc.git (which will be used for rhel/gcc as well) > >>> >>> deploy-config-gnu-gcclib.git (also for rhel/gcclib) > >>> >>> deploy-config-gnu.git > >>> >>> deploy-config-perl5-core.git > >>> >>> deploy-config-perl5.git > >>> >>> deploy-config-apache.git (might get it's own system, too -- we'll > >>> >>> see...) > >>> >>> > >>> >>> And of course the generic one. What I like about this is we always > >>> >>> migrate things from one to the other pretty easily. if we find that, > >>> >>> say oss/openssl has grown complex enough, we can yank it out of > >>> >>> deploy-config, and create deploy-config-oss-openssl. > >>> >>> > >>> >>> So how do we deploy this data? Having it well managed is git is > >>> >>> great, but how to we access it when building things with efsdeploy, > >>> >>> and where does it get copied/cached? > >>> >>> > >>> >>> Let's start with the generic repo first. Just as we use > >>> >>> efs/deploy-site/current to abstract the site-specific config > >>> >>> information, I think we can do the following: > >>> >>> > >>> >>> deploy-config.git => /efs/dist/efs/deploy-config/current > >>> >>> > >>> >>> The metaproj- and project-specific ones would then map to: > >>> >>> > >>> >>> deploy-config-$metaproj.git => > >>> >>> /efs/dist/$metaproj/deploy-config-$metaproj/current > >>> >>> deploy-config-$metaproj-project.git => > >>> >>> /efs/dist/$metaproj/deploy-config-$metaproj-$project/current > >>> >>> > >>> >>> This would allow us to publish, probably date-based, any of these > >>> >>> repositories with the "latest" set of efsdeploy build rules. > >>> >>> Note that the default rules go into the efs metaproj, obviously, but > >>> >>> we can still have a "deploy-config-efs.git" repo if we want, with no > >>> >>> conflict. > >>> >>> > >>> >>> It is very straight forward to code a solution that allows us to > >>> >>> automate keeping the local copies of these rules uptodate as they > >>> >>> change. I will almost certainly have a first pass at this within the > >>> >>> next month. However, what is NOT clear is just how to use this > >>> >>> information in efsdeploy when building release. > >>> >>> > >>> >>> Reproducibility concerns me. The rules are going to evolve, and when > >>> >>> we make gnu/gcc rule changes to build, say 4.7.0, we don't want to > >>> >>> break builds of 4.4.6, and yet *testing* that is extremely expensive. > >>> >>> For that reason, I think the contents of the efsdeploy directory > >>> >>> should be CACHED in the release, rather than read from these projects > >>> >>> during the build. Just as we are going to provide generic dependency > >>> >>> specs (see email from 30 minutes ago), and expanding those into > >>> >>> specific releasealiases to be used for the duration of the build, I > >>> >>> think we should do the same for the project-specific build rules, or > >>> >>> at least make it optional. > >>> >>> > >>> >>> In theory, if we just have efsdeploy search for these rules the same > >>> >>> way it searches for system-specific (i,e, gnu, perl5, etc) rules, and > >>> >>> then site-specific rules, then I could actually build EVERYTHING I > >>> >>> have in /efs/dist with EMPTY source directories!! If a project is > >>> >>> supported by one of these repos, then you can build a new release with > >>> >>> nothing more than: > >>> >>> > >>> >>> efs create project ... > >>> >>> efs create release ... > >>> >>> cd ..../src > >>> >>> efsdeploy down:up > >>> >>> > >>> >>> The contents of the src directory would contain NOTHING but the > >>> >>> changes you had to make (hooks, configs, whatever) to get the release > >>> >>> to build. Those changes should then be re-integrated with the git > >>> >>> repo in a controlled fashion, so that the next person building that > >>> >>> MPR has no pain. The specific workflow for how a new change gets > >>> >>> rolled into the published git repos will need to be worked out, but I > >>> >>> think that will be straight forward. > >>> >>> > >>> >>> Now, obviously, in order to *develop* changes to the rules, we'll need > >>> >>> a simply means of overriding the path to these published rules. > >>> >>> Maybe you want to install the latest set of gnu/gcc rules, but not > >>> >>> make them current until you've actually done a test-build of the > >>> >>> releases you care about. Maybe something in efsdeploy.conf (which > >>> >>> will now be a site/release-specific file, by definition) like this. > >>> >>> Say we wanted to test out some local changes right from the source > >>> >>> tree (I've been doing this with symlinks for now): > >>> >>> > >>> >>> [rules] > >>> >>> $metaproj/$project = /home/efsops/dev/efs/deploy-config-gnu-gcc > >>> >>> > >>> >>> or, perhaps, if we use date-based releases, you could install the > >>> >>> latest update into /efs/dist, and test it out this way: > >>> >>> > >>> >>> [rules] > >>> >>> $metaproj/$project = /efs/dist/gnu/deploy-config-gnu-gcc/20111230 > >>> >>> > >>> >>> Alternately, you could just rsync the efsdeploy directory right into a > >>> >>> release, and work with a copy. > >>> >>> > >>> >>> OK, that's enough of Phil's rantings for one day. Not that anyone's > >>> >>> paying attention, but you will see commits that implement many of > >>> >>> these features over the next few weeks. > >>> _______________________________________________ > >>> EFS-dev mailing list > >>> [email protected] > >>> http://mailman.openefs.org/mailman/listinfo/efs-dev > >> _______________________________________________ > >> EFS-dev mailing list > >> [email protected] > >> http://mailman.openefs.org/mailman/listinfo/efs-dev > _______________________________________________ > EFS-dev mailing list > [email protected] > http://mailman.openefs.org/mailman/listinfo/efs-dev _______________________________________________ EFS-dev mailing list [email protected] http://mailman.openefs.org/mailman/listinfo/efs-dev
