[PATCH 0/4] Multiple worktrees vs. submodules fixes
Hi. These are fixes of issues with submodules with use of multiple working trees. To be applied on top of the $gmane/257559, (6b4ce012cb in current pu). Max Kirillov (4): checkout: do not fail if target is an empty directory submodule refactor: use git_path_submodule() in add_submodule_odb() git-common-dir: make "modules/" per-working-directory directory path: implement common_dir handling in git_path_submodule() Documentation/gitrepository-layout.txt | 4 +-- builtin/checkout.c | 2 +- cache.h| 1 + path.c | 26 +++--- setup.c| 17 +++--- submodule.c| 28 ++- t/t2025-checkout-to.sh | 7 +++- t/t7410-submodule-checkout-to.sh | 62 ++ 8 files changed, 114 insertions(+), 33 deletions(-) create mode 100755 t/t7410-submodule-checkout-to.sh -- 2.0.1.1697.g73c6810 -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes
On Sun, Oct 12, 2014 at 12:13 PM, Max Kirillov wrote: > These are fixes of issues with submodules with use of multiple working > trees. I think the patches look fine from the nd/multiple-work-trees writer's perspective. I know too little about submodules to judge if this is the right way and not that way.. -- Duy -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes
Am 14.10.2014 um 14:17 schrieb Duy Nguyen: On Sun, Oct 12, 2014 at 12:13 PM, Max Kirillov wrote: These are fixes of issues with submodules with use of multiple working trees. I think the patches look fine from the nd/multiple-work-trees writer's perspective. I know too little about submodules to judge if this is the right way and not that way.. Sorry, I was too busy to review this work until now, but here we go: If I understand multiple work trees correctly, everything except work tree local stuff is redirected into GIT_COMMON_DIR. This is a very cool feature I'd love to see on our CI server to reduce disk footprint and clone times, especially for submodules! But I can't see how that can work by just sharing the modules directory tree, as that contains work tree related files - e.g. the index - for each submodule. AFAICS sharing them between work trees will work only if the content of the modules directory is partly present in GIT_DIR - for work tree related files - and only the common stuff is taken from GIT_COMMON_DIR (Or did I just miss the magic that already does that?). And I didn't try to wrap my head around recursive submodules yet ... Until that problem is solved it looks wrong to pass GIT_COMMON_DIR into submodule recursion, I believe GIT_COMMON_DIR should be added to the local_repo_env array (and even if it is passed on later, we might have to append "/modules/" to make it point to the correct location). But maybe I'm missing something? -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes
Jens Lehmann writes: > But I can't see how that can work by just sharing the modules directory > tree, as that contains work tree related files - e.g. the index - for > each submodule. AFAICS sharing them between work trees will work only > if the content of the modules directory is partly present in GIT_DIR - > for work tree related files - and only the common stuff is taken from > GIT_COMMON_DIR (Or did I just miss the magic that already does that?). The first time I saw the patch 3/4 in this series, my reaction was "Huh, why should the repository data and branch tips be separated out into multiple independent copies for the same module? Do we force users to synchronise between these copies? It does not make any sense at all." But that was until I read your message ;-) You are right that the index and HEAD are dependent to a particular working tree that is checked out. There may be other things that logically are per- working tree. And multiple-worktree _is_ about keeping the same repository and history data (i.e. object database, refs, rerere database, reflogs for refs/*) only once, while allowing multiple working trees attached to that single copy. So it appears to me that to create a checkout-to copy of a superproject with a submodule, a checkout-to copy of the superproject would have a submodule, which is a checkout-to copy of the submodule in the superproject. > And I didn't try to wrap my head around recursive submodules yet ... > > Until that problem is solved it looks wrong to pass GIT_COMMON_DIR into > submodule recursion, I believe GIT_COMMON_DIR should be added to the > local_repo_env array (and even if it is passed on later, we might have > to append "/modules/" to make it point to the correct > location). -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes
On Tue, Oct 14, 2014 at 10:26:42AM -0700, Junio C Hamano wrote: > And multiple-worktree _is_ about keeping the same repository and > history data (i.e. object database, refs, rerere database, reflogs for > refs/*) only once, while allowing multiple working trees attached to > that single copy. > > So it appears to me that to create a checkout-to copy of a > superproject with a submodule, a checkout-to copy of the superproject > would have a submodule, which is a checkout-to copy of the submodule > in the superproject. That's right, this linking should be more implicit. But here are a lot of nuances. For example, it makes sense to have a superproject checkout without submodules being initialized (so that they don't waste space and machine time for working tree, which often is more than repository data). And it may happen so that this checkout is the master repository for superproject checkouts. But this should not prevent users from using initialized submodules in other checkouts. Then, a checkout copy of a submodule can be standalone (for example, git and git-html-docs are submodules of msysgit). Or, it can even belong to some other superproject. And in that cases they still should be able to be linked. Considering all above, and also the thing that I am quite new to submodules (but have to use them currently), I did not intend to create any new UI, only to make backend handle the already existing linked checkouts, which can be made manually. -- Max -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes
Am 14.10.2014 um 20:34 schrieb Max Kirillov: On Tue, Oct 14, 2014 at 10:26:42AM -0700, Junio C Hamano wrote: And multiple-worktree _is_ about keeping the same repository and history data (i.e. object database, refs, rerere database, reflogs for refs/*) only once, while allowing multiple working trees attached to that single copy. So it appears to me that to create a checkout-to copy of a superproject with a submodule, a checkout-to copy of the superproject would have a submodule, which is a checkout-to copy of the submodule in the superproject. That's right, this linking should be more implicit. Yep. And for the submodule of a submodule too ... ;-) But here are a lot of nuances. For example, it makes sense to have a superproject checkout without submodules being initialized (so that they don't waste space and machine time for working tree, which often is more than repository data). Hmm, I'm not sure if this is a problem. If the GIT_COMMON_DIR does have the submodule repo but it isn't initialized locally, we shouldn't have a problem (except for wasting some disk space if not a single checkout-to superproject initializes this submodule). And if GIT_COMMON_DIR does not have the submodule repo yet, wouldn't it be cloned the moment we init the submodule in the checkout-to? Or would that need extra functionality? > And it may happen so that this checkout is the master repository for superproject checkouts. But this should not prevent users from using initialized submodules in other checkouts. I could live with the restriction that submodule's GIT_COMMON_DIRs always live in their checkout-to superproject's GIT_COMMON_DIR. This would still be an improvement for CI servers that have multiple clones of a super- project, as they would all share their submodule common dirs at least per superproject. Then, a checkout copy of a submodule can be standalone (for example, git and git-html-docs are submodules of msysgit). Or, it can even belong to some other superproject. And in that cases they still should be able to be linked. Maybe such configurations would have to be handled manually to achieve maximum savings. At least I could live with that. Considering all above, and also the thing that I am quite new to submodules (but have to use them currently), I did not intend to create any new UI, only to make backend handle the already existing linked checkouts, which can be made manually. Maybe the way to go is to restrict GIT_COMMON_DIR to superprojects and have a distinct GIT_COMMON_MODULES_DIR? We could even set that to the same location for all submodules of different superprojects to achieve minimum disk footprint (assuming they all have different names across all superprojects and recursion levels). Hmm, so I tend towards adding GIT_COMMON_DIR to local_repo_env until we figured out how to handle this. Without that I fear bad things will happen, at least for a superproject with multiple checkout-to work trees where the same submodule is initialized more than once ... -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes
On Tue, Oct 14, 2014 at 07:09:45PM +0200, Jens Lehmann wrote: > Until that problem is solved it looks wrong to pass > GIT_COMMON_DIR into submodule recursion, I believe > GIT_COMMON_DIR should be added to the local_repo_env array > (and even if it is passed on later, we might have to > append "/modules/" to make it point to the > correct location). Actually, why there should be an _environment_ variable GIT_COMMON_DIR at all? I mean, gitdir is resolved to some directory (through link or environment), and it contains the shared data directly or referes to it with the commondir link. In which case anyone would want to override that location? I searched though tests but they don't cover this. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes
On Tue, Oct 14, 2014 at 09:51:22PM +0200, Jens Lehmann wrote: > Am 14.10.2014 um 20:34 schrieb Max Kirillov: >> But here are a lot of nuances. For example, it makes >> sense to have a superproject checkout without submodules >> being initialized (so that they don't waste space and >> machine time for working tree, which often is more than >> repository data). > > Hmm, I'm not sure if this is a problem. If the > GIT_COMMON_DIR does have the submodule repo but it isn't > initialized locally, we shouldn't have a problem (except > for wasting some disk space if not a single checkout-to > superproject initializes this submodule). If initially a repository is clone without submodules, it will not have anything in the GIT_COMMON_DIR. > And if GIT_COMMON_DIR does not have the submodule repo > yet, wouldn't it be cloned the moment we init the > submodule in the checkout-to? Or would that need extra > functionality? I cannot say I like this. Network operations should be caused only by clone and submodules. I think the logic can be simple: it a submodule is not checked-out in the repository "checkout --to" is called from, then it is not checked-out to the new one also. If it is, then checkout calls itself recursively in the submodule and works like being run in standalone repository. >> Then, a checkout copy of a submodule can be standalone >> (for example, git and git-html-docs are submodules of >> msysgit). Or, it can even belong to some other >> superproject. And in that cases they still should be able >> to be linked. > > Maybe such configurations would have to be handled > manually to achieve maximum savings. At least I could live > with that. To make manual handling of the cases, and to skip checking-out a module. I would think about the following interface: $ git checkout --to ... - does not checkout submodules, creates empty directory. $ git checkout --recursive --to ... - if a submodule is checked-out in source repository, recursed there and run "checkout --recursive" again. If a submodule is not checked-out, does not checkout it, creates an empty directory. By the way, I have found your branch recursive_submodule_checkout. Would you like to revive it? Then it can be done with the same option. > Hmm, so I tend towards adding GIT_COMMON_DIR to > local_repo_env until we figured out how to handle this. > Without that I fear bad things will happen, at least for a > superproject with multiple checkout-to work trees where > the same submodule is initialized more than once ... I learned about local_repo_env and agree it should include GIT_COMMON_DIR. Unless it is removed at all... -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes
On Wed, Oct 15, 2014 at 3:31 AM, Max Kirillov wrote: > On Tue, Oct 14, 2014 at 07:09:45PM +0200, Jens Lehmann wrote: >> Until that problem is solved it looks wrong to pass >> GIT_COMMON_DIR into submodule recursion, I believe >> GIT_COMMON_DIR should be added to the local_repo_env array >> (and even if it is passed on later, we might have to >> append "/modules/" to make it point to the >> correct location). > > Actually, why there should be an _environment_ variable > GIT_COMMON_DIR at all? I mean, gitdir is resolved to some > directory (through link or environment), and it contains the > shared data directly or referes to it with the commondir > link. In which case anyone would want to override that > location? It's how the implementation was built up. First I split the repo in two and I need something to point the new/shared part. $GIT_DIR/worktrees comes later to glue them up. Keeping it an environment variable gives us a possibility to glue things up in a different way than using $GIT_DIR/worktrees. Of course the odds of anybody doing that are probably small or even near zero. Still consuming the rest of this thread. Thanks all for working on the submodule support for this. -- Duy -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes
On Wed, Oct 15, 2014 at 5:15 AM, Max Kirillov wrote: >> Hmm, so I tend towards adding GIT_COMMON_DIR to >> local_repo_env until we figured out how to handle this. >> Without that I fear bad things will happen, at least for a >> superproject with multiple checkout-to work trees where >> the same submodule is initialized more than once ... > > I learned about local_repo_env and agree it should include > GIT_COMMON_DIR. Unless it is removed at all... It's in the same class as GIT_DIR and GIT_WORK_TREE so yeah it should be in local_repo_env. Removing it would break t0060-path-utils.sh at least. Unless we have a very good reason to remove it, I think we should keep it as is. -- Duy -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes
Duy Nguyen writes: > On Wed, Oct 15, 2014 at 3:31 AM, Max Kirillov wrote: >> On Tue, Oct 14, 2014 at 07:09:45PM +0200, Jens Lehmann wrote: >>> Until that problem is solved it looks wrong to pass >>> GIT_COMMON_DIR into submodule recursion, I believe >>> GIT_COMMON_DIR should be added to the local_repo_env array >>> (and even if it is passed on later, we might have to >>> append "/modules/" to make it point to the >>> correct location). >> >> Actually, why there should be an _environment_ variable >> GIT_COMMON_DIR at all? I mean, gitdir is resolved to some >> directory (through link or environment), and it contains the >> shared data directly or referes to it with the commondir >> link. In which case anyone would want to override that >> location? > > It's how the implementation was built up. First I split the repo in > two and I need something to point the new/shared part. > $GIT_DIR/worktrees comes later to glue them up. Keeping it an > environment variable gives us a possibility to glue things up in a > different way than using $GIT_DIR/worktrees. Of course the odds of > anybody doing that are probably small or even near zero. > > Still consuming the rest of this thread. Thanks all for working on the > submodule support for this. Hmph. I was hoping that the multiple-work-trees topic was ready for 'next' by now, but we may want to wait to see how the interaction with submodule plays out to have another chance of a clean reroll before it happens. This is a topic with large impact and is quite intrusive code-wise, even though the intrusive parts are cleanly done. So we'd want to take more time to unleash it to the general public than more usual small scale topics, anyway. Thanks. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes
Am 15.10.2014 um 00:15 schrieb Max Kirillov: On Tue, Oct 14, 2014 at 09:51:22PM +0200, Jens Lehmann wrote: Am 14.10.2014 um 20:34 schrieb Max Kirillov: But here are a lot of nuances. For example, it makes sense to have a superproject checkout without submodules being initialized (so that they don't waste space and machine time for working tree, which often is more than repository data). Hmm, I'm not sure if this is a problem. If the GIT_COMMON_DIR does have the submodule repo but it isn't initialized locally, we shouldn't have a problem (except for wasting some disk space if not a single checkout-to superproject initializes this submodule). If initially a repository is clone without submodules, it will not have anything in the GIT_COMMON_DIR. Ok. And if GIT_COMMON_DIR does not have the submodule repo yet, wouldn't it be cloned the moment we init the submodule in the checkout-to? Or would that need extra functionality? I cannot say I like this. Network operations should be caused only by clone and submodules. Sure (and please add fetch to the list ;-). Maybe I confused you by saying "init" when I meant the "submodule update" run after initializing the submodule? I think the logic can be simple: it a submodule is not checked-out in the repository "checkout --to" is called from, then it is not checked-out to the new one also. If it is, then checkout calls itself recursively in the submodule and works like being run in standalone repository. But when I later decide to populate the submodule in a "checkout --to" work tree, should it automagically also use the central storage, creating the modules/ directory there if it doesn't exist yet? I think that'd make sense to avoid having the work tree layout depend on the order commands were ran in. And imagine new submodules, they should not be handled differently from those already present. Then, a checkout copy of a submodule can be standalone (for example, git and git-html-docs are submodules of msysgit). Or, it can even belong to some other superproject. And in that cases they still should be able to be linked. Maybe such configurations would have to be handled manually to achieve maximum savings. At least I could live with that. To make manual handling of the cases, and to skip checking-out a module. I would think about the following interface: $ git checkout --to ... - does not checkout submodules, creates empty directory. This is what checkout should always do (at least until it learns --recurse-submodules, then it would populate the submodule directories). $ git checkout --recursive --to ... - if a submodule is checked-out in source repository, recursed there and run "checkout --recursive" again. If a submodule is not checked-out, does not checkout it, creates an empty directory. Hmm, I believe that when the user requests recursion explicitly it should always be checked out, no matter in what state the GIT_COMMON_DIR is in. Otherwise we'll see problems when a new submodule shows up and is populated depending on the point in time the "checkout --to" was done ... not good. By the way, I have found your branch recursive_submodule_checkout. Would you like to revive it? Then it can be done with the same option. No need to revive it, I'm currently working on that branch whenever I find some time (but I'll still need some time before I can post the next iteration). -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes
On Wed, Oct 15, 2014 at 08:57:20PM +0200, Jens Lehmann wrote: > Am 15.10.2014 um 00:15 schrieb Max Kirillov: >> I think the logic can be simple: it a submodule is not >> checked-out in the repository "checkout --to" is called >> from, then it is not checked-out to the new one also. If it >> is, then checkout calls itself recursively in the submodule >> and works like being run in standalone repository. > > But when I later decide to populate the submodule in a > "checkout --to" work tree, should it automagically also > use the central storage, creating the modules/ > directory there if it doesn't exist yet? I think that'd > make sense to avoid having the work tree layout depend > on the order commands were ran in. And imagine new > submodules, they should not be handled differently from > those already present. Like place the common directory to $MAIN_REPO/.git/modules/$SUB/ and worktree-specific part to $MAIN_REPO/.git/worktrees/$WORKTREE/modules/$SUB, rather than placing all into the socond one? It would make sense to make, but then it would be imposible to checkout a diferent repository into the same submodule in different superproject checkouts. However stupid is sounds, there could be cases if, for example, at some moment submodule is being replaced by another one, and older worktrees should work with older submodule, while newer uses the newer submodule. Maybe, there could be some options to tell the command which populates submodules (which commands that are? "submodule update" and other submodule subcommands? or there is something else?) to use the curent checkout space or the main one. But I would still leave it depend on what user explicitly calls and where the initial submodule update is executed. Also, could you clarify the usage of the /modules/ directory. I did not notice it to affect anything after the submofule is placed there. Submodule operations use the submodule repositories directly (through the git link, which can point anywhere), or in .gitmodules file, or maybe in .git/config. So there is actually no need to have that gitdir there. Is it correct? -- Max -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes
On Thu, Oct 16, 2014 at 12:09 AM, Junio C Hamano wrote: > Hmph. I was hoping that the multiple-work-trees topic was ready for > 'next' by now, but we may want to wait to see how the interaction > with submodule plays out to have another chance of a clean reroll > before it happens. This is a topic with large impact and is quite > intrusive code-wise, even though the intrusive parts are cleanly > done. So we'd want to take more time to unleash it to the general > public than more usual small scale topics, anyway. Originally I wanted to get it merged without submodule support, but I failed to spot the local_repo_env problem and could have caused a regression for submodules. So yeah delaying the series does not sound bad. Not sure about the reroll (i.e. rewriting current patches). I think putting patches on top with explanation is better. But we can keep it in 'pu' and see if we really need to reroll. -- Duy -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes
Am 16.10.2014 um 22:54 schrieb Max Kirillov: On Wed, Oct 15, 2014 at 08:57:20PM +0200, Jens Lehmann wrote: Am 15.10.2014 um 00:15 schrieb Max Kirillov: I think the logic can be simple: it a submodule is not checked-out in the repository "checkout --to" is called from, then it is not checked-out to the new one also. If it is, then checkout calls itself recursively in the submodule and works like being run in standalone repository. But when I later decide to populate the submodule in a "checkout --to" work tree, should it automagically also use the central storage, creating the modules/ directory there if it doesn't exist yet? I think that'd make sense to avoid having the work tree layout depend on the order commands were ran in. And imagine new submodules, they should not be handled differently from those already present. Like place the common directory to $MAIN_REPO/.git/modules/$SUB/ and worktree-specific part to $MAIN_REPO/.git/worktrees/$WORKTREE/modules/$SUB, rather than placing all into the socond one? It would make sense to make, but then it would be imposible to checkout a diferent repository into the same submodule in different superproject checkouts. However stupid is sounds, there could be cases if, for example, at some moment submodule is being replaced by another one, and older worktrees should work with older submodule, while newer uses the newer submodule. Yes, but I believe that the user must be careful to not reuse the same submodule name for a different repo anyways, no matter if shared or not. Currently you'll get a warning about that when trying to add a submodule whose name is already found in .git/modules to avoid such confusion. Maybe, there could be some options to tell the command which populates submodules (which commands that are? "submodule update" and other submodule subcommands? or there is something else?) to use the curent checkout space or the main one. But I would still leave it depend on what user explicitly calls and where the initial submodule update is executed. Currently only "submodule update" populates submodules, but I'm currently working hard on teaching commands like checkout (and lots of others) to do the same. I agree that the user should be able to choose and for our CI server I would also like to see a solution that could share submodules across different superprojects. So having another environment variable to decide where to put the work tree independent parts of the .git/modules directory might make sense here. Also, could you clarify the usage of the /modules/ directory. I did not notice it to affect anything after the submofule is placed there. Submodule operations use the submodule repositories directly (through the git link, which can point anywhere), or in .gitmodules file, or maybe in .git/config. So there is actually no need to have that gitdir there. Is it correct? Nope. When submodules are cloned their git directory is placed under .git/modules/, the .git file in the work tree points there and the core.worktree setting points back from there to the work tree. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes
Am 17.10.2014 um 11:14 schrieb Duy Nguyen: On Thu, Oct 16, 2014 at 12:09 AM, Junio C Hamano wrote: Hmph. I was hoping that the multiple-work-trees topic was ready for 'next' by now, but we may want to wait to see how the interaction with submodule plays out to have another chance of a clean reroll before it happens. This is a topic with large impact and is quite intrusive code-wise, even though the intrusive parts are cleanly done. So we'd want to take more time to unleash it to the general public than more usual small scale topics, anyway. Originally I wanted to get it merged without submodule support, but I failed to spot the local_repo_env problem and could have caused a regression for submodules. So yeah delaying the series does not sound bad. Not sure about the reroll (i.e. rewriting current patches). I think putting patches on top with explanation is better. But we can keep it in 'pu' and see if we really need to reroll. I didn't look into your series in detail, but it looks to my like excepting the .git/modules directory from sharing (by putting it into local_repo_env array) and adding a test for that (just to be sure that no modules directory shows up where it shouldn't) should be sufficient to get your stuff merged without submodule support. It might be better to handle submodule support in a follow up series. Does that make sense? -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes
On Sun, Oct 19, 2014 at 09:30:15PM +0200, Jens Lehmann wrote: > Am 16.10.2014 um 22:54 schrieb Max Kirillov: >> On Wed, Oct 15, 2014 at 08:57:20PM +0200, Jens Lehmann wrote: >>> Am 15.10.2014 um 00:15 schrieb Max Kirillov: I think the logic can be simple: it a submodule is not checked-out in the repository "checkout --to" is called from, then it is not checked-out to the new one also. If it is, then checkout calls itself recursively in the submodule and works like being run in standalone repository. >>> But when I later decide to populate the submodule in a >>> "checkout --to" work tree, should it automagically also >>> use the central storage, creating the modules/ >>> directory there if it doesn't exist yet? I think that'd >>> make sense to avoid having the work tree layout depend >>> on the order commands were ran in. And imagine new >>> submodules, they should not be handled differently from >>> those already present. >> Like place the common directory to >> $MAIN_REPO/.git/modules/$SUB/ and worktree-specific part to >> $MAIN_REPO/.git/worktrees/$WORKTREE/modules/$SUB, rather >> than placing all into the socond one? It would make sense to >> make, but then it would be imposible to checkout a diferent >> repository into the same submodule in different superproject >> checkouts. However stupid is sounds, there could be cases >> if, for example, at some moment submodule is being replaced >> by another one, and older worktrees should work with older >> submodule, while newer uses the newer submodule. > Yes, but I believe that the user must be careful to not > reuse the same submodule name for a different repo anyways, > no matter if shared or not. Currently you'll get a warning > about that when trying to add a submodule whose name is > already found in .git/modules to avoid such confusion. Yes, while trying to write tests for this case I discovered that there are warnings and the recommended way is to use different names for different repositories even if they are pointing to the same path. Then always placing common directory into the .git/modules/ could be a good idea, and in very special cases users could manually create repositories with custom placement. >> Also, could you clarify the usage of the /modules/ >> directory. I did not notice it to affect anything after the >> submofule is placed there. Submodule operations use the >> submodule repositories directly (through the git link, which >> can point anywhere), or in .gitmodules file, or maybe in >> .git/config. So there is actually no need to have that >> gitdir there. Is it correct? > Nope. When submodules are cloned their git directory is > placed under .git/modules/, the .git file > in the work tree points there and the core.worktree setting > points back from there to the work tree. I meant is the fact that gitdir is placed in modules, rather than in any other place, is used anywhere. There are 2 places to put the gitdir of submodule in linked copy: 1. $MAIN_REPO/.git/worktrees/$WORKTREE/modules/$SUB 2. $MAIN_REPO/.git/modules/$SUB/worktrees/$SUB_WTNAME First one is suggested by submodule way of placing gitdirs, and the second one by worrktree way. There are reasons to have the second one - garbage collection and check that 2 branch is not checked out twice. Are there resons to have the 1st one? The one is to prevent use of different repositories with the same name, anything else? -- Max -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes
Ping.. any idea how to go from here.. On Mon, Oct 20, 2014 at 11:11 AM, Max Kirillov wrote: > On Sun, Oct 19, 2014 at 09:30:15PM +0200, Jens Lehmann wrote: >> Am 16.10.2014 um 22:54 schrieb Max Kirillov: >>> On Wed, Oct 15, 2014 at 08:57:20PM +0200, Jens Lehmann wrote: Am 15.10.2014 um 00:15 schrieb Max Kirillov: > I think the logic can be simple: it a submodule is not > checked-out in the repository "checkout --to" is called > from, then it is not checked-out to the new one also. If it > is, then checkout calls itself recursively in the submodule > and works like being run in standalone repository. > But when I later decide to populate the submodule in a "checkout --to" work tree, should it automagically also use the central storage, creating the modules/ directory there if it doesn't exist yet? I think that'd make sense to avoid having the work tree layout depend on the order commands were ran in. And imagine new submodules, they should not be handled differently from those already present. > >>> Like place the common directory to >>> $MAIN_REPO/.git/modules/$SUB/ and worktree-specific part to >>> $MAIN_REPO/.git/worktrees/$WORKTREE/modules/$SUB, rather >>> than placing all into the socond one? It would make sense to >>> make, but then it would be imposible to checkout a diferent >>> repository into the same submodule in different superproject >>> checkouts. However stupid is sounds, there could be cases >>> if, for example, at some moment submodule is being replaced >>> by another one, and older worktrees should work with older >>> submodule, while newer uses the newer submodule. > >> Yes, but I believe that the user must be careful to not >> reuse the same submodule name for a different repo anyways, >> no matter if shared or not. Currently you'll get a warning >> about that when trying to add a submodule whose name is >> already found in .git/modules to avoid such confusion. > > Yes, while trying to write tests for this case I discovered > that there are warnings and the recommended way is to use > different names for different repositories even if they are > pointing to the same path. Then always placing common > directory into the .git/modules/ could be a good > idea, and in very special cases users could manually create > repositories with custom placement. > >>> Also, could you clarify the usage of the /modules/ >>> directory. I did not notice it to affect anything after the >>> submofule is placed there. Submodule operations use the >>> submodule repositories directly (through the git link, which >>> can point anywhere), or in .gitmodules file, or maybe in >>> .git/config. So there is actually no need to have that >>> gitdir there. Is it correct? > >> Nope. When submodules are cloned their git directory is >> placed under .git/modules/, the .git file >> in the work tree points there and the core.worktree setting >> points back from there to the work tree. > > I meant is the fact that gitdir is placed in modules, rather > than in any other place, is used anywhere. There are 2 > places to put the gitdir of submodule in linked copy: > 1. $MAIN_REPO/.git/worktrees/$WORKTREE/modules/$SUB > 2. $MAIN_REPO/.git/modules/$SUB/worktrees/$SUB_WTNAME > First one is suggested by submodule way of placing gitdirs, > and the second one by worrktree way. There are reasons to > have the second one - garbage collection and check that 2 > branch is not checked out twice. Are there resons to have > the 1st one? The one is to prevent use of different > repositories with the same name, anything else? > > -- > Max -- Duy -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes
Am 03.11.2014 um 13:54 schrieb Duy Nguyen: Ping.. any idea how to go from here.. I didn't dig deep enough into the multiple worktrees topic to know what "$MAIN_REPO/.git/worktrees/$WORKTREE/modules/$SUB" might mean, but a submodule whose repo lives under .git/modules/$SUBMODULE_NAME should have its worktree in $SUBMODULE_PATH of the superproject's worktree. So if we share submodule repos, the sharable stuff should live under $MAIN_REPO/.git/modules/$SUBMODULE_NAME/. But everything that can't be put there should stay under the .git/modules/$SUBMODULE_NAME directory in every worktree. The same redirection mechanism the superproject uses to redirect to $MAIN_REPO/.git should be used to redirect the sharable stuff from .git/modules/$SUBMODULE_NAME to $MAIN_REPO/.git/modules/$SUBMODULE_NAME/. I believe this should be the default. Optionally we might want to enable the user to put $MAIN_REPO/.git/modules someplace else (outside the superprojects $MAIN_REPO) to be able to reuse the object store of submodules that are shared between different superprojects. Does that make sense? On Mon, Oct 20, 2014 at 11:11 AM, Max Kirillov wrote: On Sun, Oct 19, 2014 at 09:30:15PM +0200, Jens Lehmann wrote: Am 16.10.2014 um 22:54 schrieb Max Kirillov: On Wed, Oct 15, 2014 at 08:57:20PM +0200, Jens Lehmann wrote: Am 15.10.2014 um 00:15 schrieb Max Kirillov: I think the logic can be simple: it a submodule is not checked-out in the repository "checkout --to" is called from, then it is not checked-out to the new one also. If it is, then checkout calls itself recursively in the submodule and works like being run in standalone repository. But when I later decide to populate the submodule in a "checkout --to" work tree, should it automagically also use the central storage, creating the modules/ directory there if it doesn't exist yet? I think that'd make sense to avoid having the work tree layout depend on the order commands were ran in. And imagine new submodules, they should not be handled differently from those already present. Like place the common directory to $MAIN_REPO/.git/modules/$SUB/ and worktree-specific part to $MAIN_REPO/.git/worktrees/$WORKTREE/modules/$SUB, rather than placing all into the socond one? It would make sense to make, but then it would be imposible to checkout a diferent repository into the same submodule in different superproject checkouts. However stupid is sounds, there could be cases if, for example, at some moment submodule is being replaced by another one, and older worktrees should work with older submodule, while newer uses the newer submodule. Yes, but I believe that the user must be careful to not reuse the same submodule name for a different repo anyways, no matter if shared or not. Currently you'll get a warning about that when trying to add a submodule whose name is already found in .git/modules to avoid such confusion. Yes, while trying to write tests for this case I discovered that there are warnings and the recommended way is to use different names for different repositories even if they are pointing to the same path. Then always placing common directory into the .git/modules/ could be a good idea, and in very special cases users could manually create repositories with custom placement. Also, could you clarify the usage of the /modules/ directory. I did not notice it to affect anything after the submofule is placed there. Submodule operations use the submodule repositories directly (through the git link, which can point anywhere), or in .gitmodules file, or maybe in .git/config. So there is actually no need to have that gitdir there. Is it correct? Nope. When submodules are cloned their git directory is placed under .git/modules/, the .git file in the work tree points there and the core.worktree setting points back from there to the work tree. I meant is the fact that gitdir is placed in modules, rather than in any other place, is used anywhere. There are 2 places to put the gitdir of submodule in linked copy: 1. $MAIN_REPO/.git/worktrees/$WORKTREE/modules/$SUB 2. $MAIN_REPO/.git/modules/$SUB/worktrees/$SUB_WTNAME First one is suggested by submodule way of placing gitdirs, and the second one by worrktree way. There are reasons to have the second one - garbage collection and check that 2 branch is not checked out twice. Are there resons to have the 1st one? The one is to prevent use of different repositories with the same name, anything else? -- Max -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes
On Mon, Nov 03, 2014 at 07:54:39PM +0700, Duy Nguyen wrote: > Ping.. any idea how to go from here.. I'm sorry, I happen to have little time since the last conversation. As far as I understand, my patches are correct about handling existing submodules, but they may be not enough regarding _initialization_ of the submodules. There can be desired changes to how 'git submodule update --init' behaves in the linked working trees. If others agree to leave it to some later development, and continue now with the reading part, I would keep it all unchanged, except adding the GIT_COMMON_DIR to local_repo_env (or would you like to do it?) and couple of additional tests which I have drafted during discussion. Otherwise I'll be trying but it's hard to make any estimates. PS: the change of 'git submodule update --init' would be behavior change, but git anyway must support non-linked submodules in different worktrees, so there should be no compatibility break. -- Max -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
