Re: [git-users] Problem using detached worktrees with commands implemented in scripts
> From: Junio C Hamano > > ... it's not clear why GIT_WORK_TREE exists, ... > > The configuration item came _way_ later than the environment, and we > need to keep users and scripts from old world working, that is why. OK, that explains a great deal. IIRC, I first became aware that detached worktrees are possible through the documentation of core.worktree. As Git's architecture has a tight binding between the repository and the worktree, it made a great deal of sense to me that the repository points to the detached worktree. And the absence of core.worktree, a non-detached worktree, is essentially equivalent to having core.worktree specify the directory containing the .git directory. So the obvious way (to me) to invoke Git with a detached worktree is to set GIT_DIR to point to the repository, and the repository points to the root of the worktree. If the command operates on the worktree, Git can compare the cwd with the worktree root to determine the relative path of files. (And you can see that in this situation, Git doesn't have to search upward to try to determine where the worktree root is.) What you're saying is that there's an older mode of operation where the repository does not point to the worktree. Instead, the caller has to set GIT_DIR to locate the repository and GIT_WORK_TREE to locate the worktree. This would be prone to error, because the user is responsible for always pairing the repository with the correct worktree; it doesn't enforce the architectural assumption that the repository is paired with a work tree. Dale -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [git-users] Problem using detached worktrees with commands implemented in scripts
> From: Junio C Hamano > It was unclear to me which part of our documentation needs updating > and how, and that was (and still is) what I was primarily interested > in finding out. It seems to me that what is missing is a description of the circumstances under which Git can be run. With Subversion (the only other source control system I know in detail), the working tree that is operated on is at and below the cwd, and the working tree always points to the repository. (A subdirectory of a working tree is also a valid working tree.) With Git, it seems that the basic usage is that Git searches upward from the cwd to find the top of the work tree, which is distinguished by having a .git subdirectory. The rules when the worktree is detached are more complicated, and don't seem to be written in any single place. Dale -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [git-users] Problem using detached worktrees with commands implemented in scripts
> From: Junio C Hamano > Now, when you say "the cwd contains the .git directory", do you mean > > cd /repositories > git add ../working/trees/proj-wt1/file > > updates "file" in the /repositories/proj.git/index? Or do you mean > this? The pattern I use is to have this: /repository/.git /working/... then cd /repository git add /working/x/y/z works as you'd expect it to. "git rm" seems to work correctly under these circumstances as well. I seem to recall that using relative values doesn't work under some conditions involving symbolic links, but I can't recall the details right now. > you talk about starting Git command _outside_ the working tree > (whether the working tree has its repository embedded in it is not > very relevant). The above pattern is what I mean, where the cwd is not within the work tree. Dale -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [git-users] Problem using detached worktrees with commands implemented in scripts
wor...@alum.mit.edu (Dale R. Worley) writes: >> From: Junio C Hamano > >> Side note: without GIT_WORK_TREE environment (or >> core.worktree), there is no way to tell where the top level >> is, so you were limited to always be at the top level of >> your working tree if you used GIT_DIR to refer to a >> repository that is not embedded in your working tree. There >> were some changes in this area, but I do not recall the >> details offhand. > > That's not true. The core.worktree config variable tells the top of > the worktree, so once you've located the repository, you know where > the worktree is. Read the second line again, perhaps? > ... it's not clear why GIT_WORK_TREE exists, ... The configuration item came _way_ later than the environment, and we need to keep users and scripts from old world working, that is why. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [git-users] Problem using detached worktrees with commands implemented in scripts
> From: Junio C Hamano > Side note: without GIT_WORK_TREE environment (or > core.worktree), there is no way to tell where the top level > is, so you were limited to always be at the top level of > your working tree if you used GIT_DIR to refer to a > repository that is not embedded in your working tree. There > were some changes in this area, but I do not recall the > details offhand. That's not true. The core.worktree config variable tells the top of the worktree, so once you've located the repository, you know where the worktree is. Indeed, it's not clear why GIT_WORK_TREE exists, as that allows the user to set GIT_WORK_TREE inconsistently with core.worktree. (What happens if you do that?) Dale -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [git-users] Problem using detached worktrees with commands implemented in scripts
From: "Jonathan Nieder" Philip Oakley wrote: It was Dale that had the problem, I was just suggesting where he might want to look... ;-) A bit more looking gave that the cd_to_toplevel () in git-sh-setup.sh directly uses `git rev-parse --show-toplevel`, which simply returns work_tree (static char *work_tree; in environment.c, with comment /* This is set by setup_git_dir_gently() and/or git_default_config() */), apparently without a check for the GIT_WORK_TREE. Getting closer. :) The usual way to use GIT_WORK_TREE is along with GIT_DIR. When re-checking the manual's git(1) env variable section the comment that implies this didn't read well "The value will not be used in combination...". The section probably needs that to be stated explicitly ("an exported GIT_WORK_TREE is ignored if GIT_DIR is not set"). That takes you into the setup_explicit_git_dir() codepath, which does respect GIT_WORK_TREE as it should. (setup_discovered_git_dir does, too.) The strange behavior you ran into is that unlike, say, git-pull.sh and git-am.sh, filter-branch does not set SUBDIRECTORY_OK, store the prefix from 'git rev-parse --show-prefix', and then cd_to_toplevel at the top of the script. In other words, nobody bothered to make it work from anywhere other than the toplevel of the worktree to begin with, and nobody wanted it enough to fix it later. I maybe wrong, but I thought that in Dale's case he was already at the same level as the GIT_WORK_TREE he had set, so may not have expected to need a cd_to_toplevel Dale did propose a patch in http://thread.gmane.org/gmane.comp.version-control.git/236260 as the error was from later in the setup script. Hope that helps, Jonathan Philip -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [git-users] Problem using detached worktrees with commands implemented in scripts
Philip Oakley wrote: > A bit more looking gave that the cd_to_toplevel () in git-sh-setup.sh > directly uses `git rev-parse --show-toplevel`, which simply returns > work_tree (static char *work_tree; in environment.c, with comment /* > This is set by setup_git_dir_gently() and/or git_default_config() > */), apparently without a check for the GIT_WORK_TREE. Getting closer. :) The usual way to use GIT_WORK_TREE is along with GIT_DIR. That takes you into the setup_explicit_git_dir() codepath, which does respect GIT_WORK_TREE as it should. (setup_discovered_git_dir does, too.) The strange behavior you ran into is that unlike, say, git-pull.sh and git-am.sh, filter-branch does not set SUBDIRECTORY_OK, store the prefix from 'git rev-parse --show-prefix', and then cd_to_toplevel at the top of the script. In other words, nobody bothered to make it work from anywhere other than the toplevel of the worktree to begin with, and nobody wanted it enough to fix it later. Hope that helps, Jonathan -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [git-users] Problem using detached worktrees with commands implemented in scripts
From: "Philip Oakley" From: "Junio C Hamano" "Philip Oakley" writes: From: "Junio C Hamano" "Philip Oakley" writes: ... and the detection process for 'toplevel' may not work properly when in a separated work-tree environment. Without GIT_WORK_TREE exported to point at the top-level, there is nothing that lets us "detect" it, as the working tree does not have ".git" directory to tell us to stop, no? "No", but not in that way. My point (to Dale) was, as you state, that the "cd to top level" was (IIUC) the probable causes of the fault, and that a documentation update would probably be appropriate for the discussion on exporting GIT_WORK_TREE, and that it would specifically mention those git commands that needed to "cd to top level", and hence would not work in such an environment. (I wasn't sure where the appropriate "cd to top level" function was) An explanation here on the list wouldn't solve the problems for others who are yet to make the same mistake, hence the implied suggestion. I understand what you mean by these last two lines. It was unclear to me which part of our documentation needs updating and how, and that was (and still is) what I was primarily interested in finding out. I was expecting that the places would be in git(1) [git.txt] and config(1) [config.txt], in the enironment variables GIT_WORK_TREE section and core.worktree sections repectively. However what the right text would be hasn't been fully determined yet, as it should be clear about which commands don't follow the stated 'rules'. Dale's use case does appear to be stretching... Philip A bit more looking gave that the cd_to_toplevel () in git-sh-setup.sh directly uses `git rev-parse --show-toplevel`, which simply returns work_tree (static char *work_tree; in environment.c, with comment /* This is set by setup_git_dir_gently() and/or git_default_config() */), apparently without a check for the GIT_WORK_TREE. One option may be to either protect the cd_to_toplevel code with a check of `git rev-parse --local-env-vars` to see if GIT_WORK_TREE is present. Or create `git rev-parse --work-dir` to match `--git-dir`. This would be a code level fix. This makes the assumption that if a deteched GIT_WORK_TREE is set then it is the top level. In terms of command scripts that use git-sh-setup.sh we have a longish list, so a full list in the documentation is probably unreasonable (which suggests that a code fix would be more apprpriate) commands: git-am git-bisect git-filter-branch git-instaweb git-lost-found git-merge-one-file git-mergetool git-pull git-quiltimport git-rebase git-repack git-request-pull git-stash git-submodule git-web--browse git\contrib\*various* Philip -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [git-users] Problem using detached worktrees with commands implemented in scripts
From: "Junio C Hamano" "Philip Oakley" writes: From: "Junio C Hamano" "Philip Oakley" writes: ... and the detection process for 'toplevel' may not work properly when in a separated work-tree environment. Without GIT_WORK_TREE exported to point at the top-level, there is nothing that lets us "detect" it, as the working tree does not have ".git" directory to tell us to stop, no? "No", but not in that way. My point (to Dale) was, as you state, that the "cd to top level" was (IIUC) the probable causes of the fault, and that a documentation update would probably be appropriate for the discussion on exporting GIT_WORK_TREE, and that it would specifically mention those git commands that needed to "cd to top level", and hence would not work in such an environment. (I wasn't sure where the appropriate "cd to top level" function was) An explanation here on the list wouldn't solve the problems for others who are yet to make the same mistake, hence the implied suggestion. I understand what you mean by these last two lines. It was unclear to me which part of our documentation needs updating and how, and that was (and still is) what I was primarily interested in finding out. I was expecting that the places would be in git(1) [git.txt] and config(1) [config.txt], in the enironment variables GIT_WORK_TREE section and core.worktree sections repectively. However what the right text would be hasn't been fully determined yet, as it should be clear about which commands don't follow the stated 'rules'. Dale's use case does appear to be stretching... Philip -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [git-users] Problem using detached worktrees with commands implemented in scripts
"Philip Oakley" writes: > From: "Junio C Hamano" >> "Philip Oakley" writes: >> >>> ... and the detection process for 'toplevel' may not work >>> properly when in a separated work-tree environment. >> >> Without GIT_WORK_TREE exported to point at the top-level, there is >> nothing that lets us "detect" it, as the working tree does not have >> ".git" directory to tell us to stop, no? >> > > "No", but not in that way. > > My point (to Dale) was, as you state, that the "cd to top level" was > (IIUC) the probable causes of the fault, and that a documentation > update would probably be appropriate for the discussion on exporting > GIT_WORK_TREE, and that it would specifically mention those git > commands that needed to "cd to top level", and hence would not work in > such an environment. (I wasn't sure where the appropriate "cd to top > level" function was) > > An explanation here on the list wouldn't solve the problems for others > who are yet to make the same mistake, hence the implied suggestion. I understand what you mean by these last two lines. It was unclear to me which part of our documentation needs updating and how, and that was (and still is) what I was primarily interested in finding out. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [git-users] Problem using detached worktrees with commands implemented in scripts
From: "Junio C Hamano" "Philip Oakley" writes: ... and the detection process for 'toplevel' may not work properly when in a separated work-tree environment. Without GIT_WORK_TREE exported to point at the top-level, there is nothing that lets us "detect" it, as the working tree does not have ".git" directory to tell us to stop, no? "No", but not in that way. My point (to Dale) was, as you state, that the "cd to top level" was (IIUC) the probable causes of the fault, and that a documentation update would probably be appropriate for the discussion on exporting GIT_WORK_TREE, and that it would specifically mention those git commands that needed to "cd to top level", and hence would not work in such an environment. (I wasn't sure where the appropriate "cd to top level" function was) An explanation here on the list wouldn't solve the problems for others who are yet to make the same mistake, hence the implied suggestion. Philip -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [git-users] Problem using detached worktrees with commands implemented in scripts
wor...@alum.mit.edu (Dale R. Worley) writes: > I must admit I've never seen the design (and I personally doubt that > the design has ever been written down). But at least the following > commands work correctly on a detached worktree if the current > directory contains the .git directory, because I am using them in a > production manner: > > git add If you have this: /repositories/proj.git/{refs,objects,...} /working/trees/proj-wt1/ where proj-wt1 is a working tree for that proj.git repository, the idea was to set these: GIT_DIR=/repositories/proj.git GIT_WORK_TREE=/working/trees/proj-wt1 export GIT_DIR GIT_WORK_TREE and then working in /working/trees/proj-wt1 or any of its subdirectory should work as if you did not have these two environment variables and had /working/trees/proj-wt1/.git instead of /repositories/proj.git as the repository. To make that use case work was the motivation behind these environment variables. Side note: without GIT_WORK_TREE environment (or core.worktree), there is no way to tell where the top level is, so you were limited to always be at the top level of your working tree if you used GIT_DIR to refer to a repository that is not embedded in your working tree. There were some changes in this area, but I do not recall the details offhand. Now, when you say "the cwd contains the .git directory", do you mean cd /repositories git add ../working/trees/proj-wt1/file updates "file" in the /repositories/proj.git/index? Or do you mean this? cd /repositories/proj.git git add ../../working/trees/proj-wt1/file Or this? cd /repositories edit ../working/trees/proj-wt1/file git add file Most of the commands you listed do not need to look at the actual working tree files, so I would expect e.g. "git log" or "git log paths..." to work but I am wondering what your definition of "works" with respect to the pathspecs, especially when you talk about starting Git command _outside_ the working tree (whether the working tree has its repository embedded in it is not very relevant). -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [git-users] Problem using detached worktrees with commands implemented in scripts
> From: Junio C Hamano > > wor...@alum.mit.edu (Dale R. Worley) writes: > > > In general, Git commands on a repository with a detached worktree can > > be executed by cd'ing into the directory containing the .git > > directory, ... > > Eh? News to me; it might happened to have appeared to work by > accident, but that is not by design. I must admit I've never seen the design (and I personally doubt that the design has ever been written down). But at least the following commands work correctly on a detached worktree if the current directory contains the .git directory, because I am using them in a production manner: git add git cat-file git commit git commit-tree git config git gc git log git ls-tree git reset git rev-list git update-ref In my situation, the worktree is not, in my mind, dependent on the repository; the repository is intended to keep backups of the contents of the directories that are worktree. Indeed, one could establish several detached repositories to back up different subsets of the same worktree. So it is conceptually natural to execute Git in the repository directory. And, after all, the current directory identifies the repository and the repository contains a pointer to the worktree. > Not exporting GIT_DIR variable in sh-setup was done not by accident > but as a very deliberate design choice, IIRC. The intention of my change is that it appears that all of the failures of my use pattern are when the command is implemented by a shell script, and it appears that all shell scripts initially invoke git-sh-setup. The change specifically detects my use pattern and, for the remainder of the script, changes the use pattern into a pattern closely related to the one that Junio documents: - export GIT_DIR that points at the correct .git directory; - GIT_WORK_TREE is left unset - set the current directory to the top of the working tree Perhaps the change should also set GIT_WORK_TREE. I haven't noticed setting GIT_WORK_TREE to be necessary for "git filter-branch", but perhaps that is coincidental. It seems to me that this change would uniformly support the use pattern I use without affecting Git's behavior in any other case. Dale -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [git-users] Problem using detached worktrees with commands implemented in scripts
"Philip Oakley" writes: > ... and the detection process for 'toplevel' may not work > properly when in a separated work-tree environment. Without GIT_WORK_TREE exported to point at the top-level, there is nothing that lets us "detect" it, as the working tree does not have ".git" directory to tell us to stop, no? -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [git-users] Problem using detached worktrees with commands implemented in scripts
From: "Junio C Hamano" wor...@alum.mit.edu (Dale R. Worley) writes: In general, Git commands on a repository with a detached worktree can be executed by cd'ing into the directory containing the .git directory, ... Eh? News to me; it might happened to have appeared to work by accident, but that is not by design. I think it is this part in Dale's original email "However, this approach does not work with "git filter-branch", which objects with "You need to run this command from the toplevel of the working tree." that is the problem Dale has seen. IIRC there are a few commands that do require to be run from the toplevel ('git bisect' I think is another), and the detection process for 'toplevel' may not work properly when in a separated work-tree environment. Perhaps something to consider. Philip IIRC, the intended use pattern (i.e. the change that introduced GIT_DIR and GIT_WORK_TREE environment variables was designed to support) for such a working tree is to: - export GIT_DIR that points at the correct .git directory; - export GIT_WORK_TREE that points at the correct top-level of such a working tree; and then - run the commands anywhere in the working tree, as if you did not export these two environment variables and instead had the .git directory at the usual place in the working tree. It _is_ possible that we may have broken this canonical use pattern over time with more recent updates; I do not think we have extensive test coverage for "detached worktree" use case in the first place. Does anyone have any feedback on this? Not exporting GIT_DIR variable in sh-setup was done not by accident but as a very deliberate design choice, IIRC. -- -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [git-users] Problem using detached worktrees with commands implemented in scripts
wor...@alum.mit.edu (Dale R. Worley) writes: > In general, Git commands on a repository with a detached worktree can > be executed by cd'ing into the directory containing the .git > directory, ... Eh? News to me; it might happened to have appeared to work by accident, but that is not by design. IIRC, the intended use pattern (i.e. the change that introduced GIT_DIR and GIT_WORK_TREE environment variables was designed to support) for such a working tree is to: - export GIT_DIR that points at the correct .git directory; - export GIT_WORK_TREE that points at the correct top-level of such a working tree; and then - run the commands anywhere in the working tree, as if you did not export these two environment variables and instead had the .git directory at the usual place in the working tree. It _is_ possible that we may have broken this canonical use pattern over time with more recent updates; I do not think we have extensive test coverage for "detached worktree" use case in the first place. > Does anyone have any feedback on this? Not exporting GIT_DIR variable in sh-setup was done not by accident but as a very deliberate design choice, IIRC. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[git-users] Problem using detached worktrees with commands implemented in scripts
In Git, one can set up a repository with a "detached worktree", where the .git directory is not a subdirectory of the top directory of the work tree. In general, Git commands on a repository with a detached worktree can be executed by cd'ing into the directory containing the .git directory, and executing the Git command there. E.g., "git add" and "git commit" execute as one would expect. (I think they can also be executed by cd'ing to the worktree and setting GIT_DIR.) However, this approach does not work with "git filter-branch", which objects with "You need to run this command from the toplevel of the working tree." I suspect that it does not work with other Git commands that are implemented with shell scripts. The problem appears to be in the git-sh-setup script, which is called by the Git shell scripts to set up the environment and do preliminary tests. It seems to me that this inconsistency between the script commands and the binary commands can be fixed by updating git-sh-setup in this way: --- git-sh-setup.Custom.orig2013-06-20 12:59:45.0 -0400 +++ git-sh-setup2013-10-07 22:34:06.719946134 -0400 @@ -297,14 +297,18 @@ # if we require to be in a git repository. if test -z "$NONGIT_OK" then - GIT_DIR=$(git rev-parse --git-dir) || exit + export GIT_DIR=$(git rev-parse --git-dir) || exit if [ -z "$SUBDIRECTORY_OK" ] then - test -z "$(git rev-parse --show-cdup)" || { - exit=$? - echo >&2 "You need to run this command from the toplevel of the working tree." - exit $exit - } + cdup="$(git rev-parse --show-cdup)" + if [ -n "$cdup" ] + then + # Current directory is not the toplevel. + # Set GIT_DIR to the absolute path of the repository. + GIT_DIR=$(cd "$GIT_DIR" && pwd) + # cd to the toplevel. + cd $cdup + fi fi test -n "$GIT_DIR" && GIT_DIR=$(cd "$GIT_DIR" && pwd) || { echo >&2 "Unable to determine absolute path of git directory" What this change does is, when a command is invoked from a directory containing a repository with a detached worktree, is to set GIT_DIR to the directory of the repository, then cd to the top of the worktree. After that, the script command should work as expected. I am far from being an expert in Git internals, so I don't know whether this is the correct approach to take to this problem or not. Does anyone have any feedback on this? Dale -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html