Re: expensive local git clone
On Mon, 4 Jul 2005, Linus Torvalds wrote: > > I'm making something based on pack-files. Ok, as of a few minutes ago there's now a pack-based "git clone" there. Of course, it hasn't percolated out to the mirrors yet, so you'll not see it for a while unless you have a master.kernel.org account like David, but basically you can do git clone /some/absolute/path new-dir and it will clone the old project into new-dir using pack-files. The need for an absolute path in the source is just because the "git clone" script ends up "cd'ing" to the new directory before the clone, so a relative path would need to be relative to the "new-dir" location, which is just confusing. Oh, well. Anyway, as mentioned, you can certainly do a local clone a lot faster with "cp -rl" (and yes, I'll apply Junio's patch if he makes it available against the new version, and adds a flag to make it conditional), but using a pack-file means that the new thing not only will be totally independent of the old one (which you may _want_ to do, especially if they are on different filesystems), but it also means that the above works over ssh too, ie git clone master.kernel.org:/pub/scm/git/git.git my-git should now do the right thing (indeed, I've even tested it - although I've not tried it with things like multiple branches etc, which _should_ all work automatically but.. I'm pulling Jeff's tree as I write this, but master.kernel.org is slow, so it will be some time..). Btw, one small note: when doing a "git clone", the newly cloned repo will not be checked out, and "master" will be the default HEAD regardless of what the other happened to be at (unless the other side was really screwed up and we can't match up any "master" at all). So use "git checkout xxx" to actually set whatever branch you want to use after a clone. Linus - To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: expensive local git clone
> "DSM" == David S Miller <[EMAIL PROTECTED]> writes: DSM> Looks interesting. Any particular reason to use "cpio" DSM> instead of "cp"? No particular reason other than I am just used to doing things that way. - To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: expensive local git clone
From: Junio C Hamano <[EMAIL PROTECTED]> Date: Mon, 04 Jul 2005 14:40:11 -0700 > > "DSM" == David S Miller <[EMAIL PROTECTED]> writes: > > DSM> I keep hoping git-clone-script is going to be a good way > DSM> to clone two local trees. Is my hope misguided? :-) > > Something along these lines? Looks interesting. Any particular reason to use "cpio" instead of "cp"? - To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: expensive local git clone
On Mon, 4 Jul 2005, David S. Miller wrote: > > I keep hoping git-clone-script is going to be a good way > to clone two local trees. Is my hope misguided? :-) Well, I'm not working on it, but tested patches... Linus - To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: expensive local git clone
> "DSM" == David S Miller <[EMAIL PROTECTED]> writes: DSM> I keep hoping git-clone-script is going to be a good way DSM> to clone two local trees. Is my hope misguided? :-) Something along these lines? Short-cut "git fetch" when cloning locally. Instead of calling local-pull to "do the right thing", just hardlink or copy the object files over when we know we are doing the clone locally. Signed-off-by: Junio C Hamano <[EMAIL PROTECTED]> --- cd /opt/packrat/playpen/public/in-place/git/git.junio/ jit-diff # - master: git-format-patch: Prepare patches for e-mail submission. # + (working tree) diff --git a/git-clone-script b/git-clone-script --- a/git-clone-script +++ b/git-clone-script @@ -1,7 +1,39 @@ #!/bin/sh repo="$1" dir="$2" -mkdir $dir || exit 1 -cd $dir -git-init-db -git fetch "$repo" && ( git-rev-parse FETCH_HEAD > .git/HEAD ) +mkdir "$dir" && +D=$( +(cd "$dir" && git-init-db && pwd) +) || exit 1 + +# See if repo is a local directory. +if ( + cd "$repo/objects" 2>/dev/null +) +then + # See if we can hardlink and drop "l" if not. + sample_file=$(cd "$repo" && find objects -type f -print | sed -e 1q) + + # objects directory should not be empty since we are cloning! + test -f "$repo/$sample_file" || exit + + if ln "$repo/$sample_file" "$D/.git/objects/sample" 2>/dev/null + then + l=l + else + l= + fi && + rm -f "$D/.git/objects/sample" && + + cd "$repo" && + find objects -type f -print | + cpio -puam$l "$D/.git" || exit 1 + + # FETCH_HEAD is always HEAD because we do not do the + # extra parameter to "git fetch". + pwd + cat "HEAD" >"$D/.git/FETCH_HEAD" + cd "$D" +else + cd "$D" && git fetch "$repo" +fi && ( git-rev-parse FETCH_HEAD > .git/HEAD ) Compilation finished at Mon Jul 4 14:37:29 - To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: expensive local git clone
From: Petr Baudis <[EMAIL PROTECTED]> Date: Mon, 4 Jul 2005 22:42:35 +0200 > Dear diary, on Mon, Jul 04, 2005 at 10:39:09PM CEST, I got a letter > where Linus Torvalds <[EMAIL PROTECTED]> told me that... > > That said, the pack-file thing I'm working on won't be perfect either, and > > the main advantage is that it should work over ssh. The fastest way to do > > a clone is really to do a recursive hardlinked tree, so you may well want > > to just do > > > > #!/bin/sh > > mkdir "$2" && cp -rl "$1/.git" "$2/.git" > > > > and it should be about a million times faster and equally effective. > > > > Untested, of course. > > I wouldn't do that on anything but .git/objects. ;-) Agreed. When I do it by hand (but I shouldn't have to do this by hand, git-clone-script should do it this way if that's the best and most efficient) I usually link the object directory then copy the rest of the stuff non-linked. I keep hoping git-clone-script is going to be a good way to clone two local trees. Is my hope misguided? :-) - To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: expensive local git clone
Dear diary, on Mon, Jul 04, 2005 at 10:39:09PM CEST, I got a letter where Linus Torvalds <[EMAIL PROTECTED]> told me that... > That said, the pack-file thing I'm working on won't be perfect either, and > the main advantage is that it should work over ssh. The fastest way to do > a clone is really to do a recursive hardlinked tree, so you may well want > to just do > > #!/bin/sh > mkdir "$2" && cp -rl "$1/.git" "$2/.git" > > and it should be about a million times faster and equally effective. > > Untested, of course. I wouldn't do that on anything but .git/objects. ;-) -- Petr "Pasky" Baudis Stuff: http://pasky.or.cz/ be careful, some twit might quote you out of context.. - To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: expensive local git clone
On Mon, 4 Jul 2005, David S. Miller wrote: > > I can't be the only person in the world seeing this :-) > It usually grows to around 88MB and runs for 8 minutes before > it finishes. I'm making something based on pack-files. git-local-pull is indeed walking the whole tree and copying one file at a time. We shouldn't use "git-fetch-script" in "git clone" (it's really designed for doing one file at a time and for small updates), That said, the pack-file thing I'm working on won't be perfect either, and the main advantage is that it should work over ssh. The fastest way to do a clone is really to do a recursive hardlinked tree, so you may well want to just do #!/bin/sh mkdir "$2" && cp -rl "$1/.git" "$2/.git" and it should be about a million times faster and equally effective. Untested, of course. Linus - To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
expensive local git clone
I'm still seeing git-local-pull going crazy when attemping local clones of kernel trees, this is from top: 22032 davem 18 0 63392 59m 920 R 76.0 5.9 5:33.15 git-local-pull and it's still chugging along. This is from a simple: bash$ git-clone-script /home/davem/src/GIT/linux-2.6/.git /home/davem/src/GIT/sparc-2.6 invocation. I can't be the only person in the world seeing this :-) It usually grows to around 88MB and runs for 8 minutes before it finishes. It must be walking the whole commit tree to the root or something silly like that. - To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
