Re: [RFC 00/10] Make .the gitmodules file path configurable

[Antonio Ospite]

On Mon, 23 Apr 2018 10:47:09 -0700
Jonathan Nieder  wrote:

> Hi,
>

Hi Jonathan, thank you for your comment.

> Antonio Ospite wrote:
> 
> > vcsh[1] uses bare git repositories and detached work-trees to manage
> > *distinct* sets of configuration files directly into $HOME.
> 
> Cool!  I like the tooling you're creating for this, though keep in mind
> that Git has some weaknesses as a tool for deployment.
>

I am not the author BTW, just a user trying to address the remaining
corner cases.

> In particular, keep in mind that when git updates a file, there is a
> period of time while it is missing from the filesystem, which can be
> problematic for dotfiles.
>

Thanks for the reminder, it may be worth mentioning this in vcsh
documentation, however I don't have knowledge of users experiencing
problems related to that.

> [...]
> > However when multiple repositories take turns using the same directory
> > as their work-tree, and more than one of them want to use submodules,
> > there could still be conflicts about the '.gitmodules' file because git
> > hardcodes this path.
> >
> > For comparison, in case of '.gitignore' a similar conflict might arise,
> > but git has alternative ways to specify exclude files, so vcsh solves
> > this by setting core.excludesFile for each repository and track ignored
> > files somewhere else (in ~/.gitignore.d/$VCSH_REPO_NAME).
> 
> For reference:
> 
>   core.excludesFile
>   Specifies the pathname to the file that contains
>   patterns to describe paths that are not meant to be
>   tracked, in addition to .gitignore (per-directory) and
>   .git/info/exclude. Defaults to
>   $XDG_CONFIG_HOME/git/ignore. If $XDG_CONFIG_HOME is
>   either not set or empty, $HOME/.config/git/ignore is
>   used instead. See gitignore(5).
> 
> Using this as a substitute for /.gitignore is a bit of a
> hack.  It happens to work, though, so reading on. :)
> 
> [...]
> > So this series proposes a mechanism to set an alternative path for the
> > submodules configuration file (from now on "gitmodules file").
> 
> I am nervous about this.  I wonder if there is another way to
> accomplish the goal.
>
> One possibility would be to handle the case where .gitmodules is
> excluded by a sparse checkout specification and use .gitmodules from
> the index in that case.  Would that work for you?
> 

Since part of the problem is that .gitmodules *collide* between
repositories, a sparse-checkout approach make sense indeed.

As discussed[1] with Stefan Beller having git use .gitmodules from the
index without the need to have it checked out should work for us.

[1] https://www.spinics.net/lists/git/msg329153.html

Ideally git should also be able to write to that file when it's not
checked out (e.g. when running "git submodule add"), to save the
user from tedious sparse/unsparse rounds when operating with submodules.

As suggested by Stefan I'll first try to remove the hardcoded references
to .gitmodules in git-submodule.sh adding a helper sub-command to
access .gitmodules in a more robust way, and after that git could
be taught to use the file from the index, but this second part
is currently beyond my current git knowledge.

If this mechanism of using unchecked-out files from the index could be
extended to .gitignore (and .gitattributes), then vcsh might even stop
abusing core.excludesFile; sparse checkouts seem the more natural git
way to deal with colliding files in a shared-workdir scenario.

However, having users *write* to .gitignore and .gitattributes while
they are not checked out still sounds quite problematic to me, but maybe
this could be handled by vcsh itself, similarly to what is done for the
file pointed by core.excludesFile.

Ciao,
   Antonio

-- 
Antonio Ospite
https://ao2.it
https://twitter.com/ao2it

A: Because it messes up the order in which people normally read text.
   See http://en.wikipedia.org/wiki/Posting_style
Q: Why is top-posting such a bad thing?

Re: [RFC 00/10] Make .the gitmodules file path configurable

[Jonathan Nieder]

Hi,

Antonio Ospite wrote:

> vcsh[1] uses bare git repositories and detached work-trees to manage
> *distinct* sets of configuration files directly into $HOME.

Cool!  I like the tooling you're creating for this, though keep in mind
that Git has some weaknesses as a tool for deployment.

In particular, keep in mind that when git updates a file, there is a
period of time while it is missing from the filesystem, which can be
problematic for dotfiles.

[...]
> However when multiple repositories take turns using the same directory
> as their work-tree, and more than one of them want to use submodules,
> there could still be conflicts about the '.gitmodules' file because git
> hardcodes this path.
>
> For comparison, in case of '.gitignore' a similar conflict might arise,
> but git has alternative ways to specify exclude files, so vcsh solves
> this by setting core.excludesFile for each repository and track ignored
> files somewhere else (in ~/.gitignore.d/$VCSH_REPO_NAME).

For reference:

core.excludesFile
Specifies the pathname to the file that contains
patterns to describe paths that are not meant to be
tracked, in addition to .gitignore (per-directory) and
.git/info/exclude. Defaults to
$XDG_CONFIG_HOME/git/ignore. If $XDG_CONFIG_HOME is
either not set or empty, $HOME/.config/git/ignore is
used instead. See gitignore(5).

Using this as a substitute for /.gitignore is a bit of a
hack.  It happens to work, though, so reading on. :)

[...]
> So this series proposes a mechanism to set an alternative path for the
> submodules configuration file (from now on "gitmodules file").

I am nervous about this.  I wonder if there is another way to
accomplish the goal.

One possibility would be to handle the case where .gitmodules is
excluded by a sparse checkout specification and use .gitmodules from
the index in that case.  Would that work for you?

Thanks,
Jonathan

Re: [RFC 00/10] Make .the gitmodules file path configurable

[Stefan Beller]

Hi Antonio,

> I acknowledge that the two mechanisms are different, in particular
> because git *writes* the gitmodules file itself.
>
> I am not opposed to change the name, something like
> 'core.submodulesConfigFile' might highlight that the syntax of the file
> is the one of git config, but I think it's too long.

I was not speculating on a better name, but on the nature of the
configuration, gitignore is additive and errors are easy to see, but in
gitmodules, there is only one "correct" path+name, so the problem
space is not additive, rather we can have a discussion where we get
the correct config from with low odds of errors.

>> > The new configuration setting can be used to set an *alternative*
>> > location for the gitmodules file; IMHO there is no need to provide
>> > *additional* locations like in the case of exclude files.
>>
>> I think there *may* be a need for additional files.
>> Currently there is only the .gitmodules file and the configuration
>> in .git/config overriding settings from .gitmodules.
>>
>> There was some discussion on the mailing list in the past, which
>> presented a intermediate layer in between these two places, in
>> a special ref, such that:
>> base is in .gitmodules
>> overwritten via refs/meta/submodules:.gitmodules
>> overwritten via the .git/config
>>
>> The intermediate would be a config file that is tracked on another
>> ref. This (a) decouples main project history from submodule history
>> and (b) makes it easier to distribute as it is part of the repository.
>>
>> For example (a) is desired if you dig up an old project and the
>> submodules have all moved from one git hosting provider to another.
>> Another example would be when you fork a project with submodules
>> and don't want to mess with the main history but you just want to
>> adjust the submodule URLs. That is possible with such an intermediate
>> additional place.
>>
>> For (b) you can imagine the fork that you want to distribute in your
>> community and you don't want to tell everyone to change the
>> submodule URLs, but instead you can provide them with a prepared
>> .gitmodules file, that they have to place into that special ref (which
>> can be done via fetching).
>>
>> I digress as these ideas seem to be orthogonal to your patch series,
>> just FYI. prior discussion starting at:
>> https://public-inbox.org/git/1462317985-640-1-git-send-email-sbel...@google.com/
>> I recall there was a better discussion even prior to that, but have no
>> link handy.
>>
>
> Just to understand, is that 'refs/meta/submodules:.gitmodules' file
> meant to be managed manually? Or do you imagine some options to
> instruct "git submodule" to *write* to that file instead of .gitmodules?

I imagined it to be managed manually as it would enable some
workflows for downstream users of superproject repos.

But I'd think a convenient way to write to this location would be
super useful, so we ought to have that eventually.

> In the latter case your idea could cover my use case too, couldn't it?

I would think so, yes.

> But most importantly, is this realistically going to be added?

I plan on adding it eventually. It depends on the priorities and
schedule, no promises, though.

> Currently I am not that familiar with the git code base to do it
> myself, and I've never explicitly dealt with special refs in git.

I think core git only uses them for "actual refs", e.g. remote
tracking branches are used to know about the sha1. This new
special ref would be used to store content outside the main tree,
so we'd have too lookup the blob in that not-checked-out commit
and read and write there.

> The approach of this patch series is a lot simpler, and something I can
> work on in my spare time.
> Presumably (b) could even be partially supported with my changes, by
> having both '.gitmodules' and some custom '.gitmodules-alternative'
> files in the repository and tell users to clone with:
>
>   git clone --config core.submodulesFile=.gitmodules-alternative URL
>
> Not as clean as your idea but doable.

Traditionally Git had a strong stance on not transporting config
in the repo (except submodule URLs :/) itself, so I guess
this .gitmodules-alternative would not be a file in the tree, but
a URL or something?

> [...]
>> > Since the gitmodules file is meant to be checked in into the repository,
>> > the overridden file path should be relative to the work-tree; is there
>> > a way to enforce this constraint at run time (i.e. validate the config
>> > value), or is it enough to have it documented?
>>
>> I think we'd want to check at run time, if we need this constraint.
>>
>
> I'll look into it once we decide which the way to go.
>
> Thank you.
>

Thanks for bringing up this easier approach.
Stefan

Re: [RFC 00/10] Make .the gitmodules file path configurable

[Antonio Ospite]

On Thu, 12 Apr 2018 16:36:32 -0700
Stefan Beller  wrote:

> Hi Antonio,
>

Hi Stefan,

> On Thu, Apr 12, 2018 at 3:20 PM, Antonio Ospite  wrote:
> > Hi,
> >
> > vcsh[1] uses bare git repositories and detached work-trees to manage
> > *distinct* sets of configuration files directly into $HOME.
> >
> > In general, submodules have worked perfectly fine with detached
> > work-trees for some time[2,3,4].
> >
> > However when multiple repositories take turns using the same directory
> > as their work-tree, and more than one of them want to use submodules,
> > there could still be conflicts about the '.gitmodules' file because git
> > hardcodes this path.
> >
[...]
> 
> > So this series proposes a mechanism to set an alternative path for the
> > submodules configuration file (from now on "gitmodules file").
> 
[...]
> > Expanding on that change the first patch in the series makes the path
> > customizable exposing a 'core.submodulesFile' configuration setting.
> 
> I guess the similarity to core.ignoreFile is desired here. Although these
> mechanisms are very different.
>

The option name is similar to 'core.excludesFile' also because, when I
started, I first looked at how multiple ignore files were used, so I
may have been influenced by that.

I acknowledge that the two mechanisms are different, in particular
because git *writes* the gitmodules file itself.

I am not opposed to change the name, something like
'core.submodulesConfigFile' might highlight that the syntax of the file
is the one of git config, but I think it's too long.

> > The new configuration setting can be used to set an *alternative*
> > location for the gitmodules file; IMHO there is no need to provide
> > *additional* locations like in the case of exclude files.
> 
> I think there *may* be a need for additional files.
> Currently there is only the .gitmodules file and the configuration
> in .git/config overriding settings from .gitmodules.
> 
> There was some discussion on the mailing list in the past, which
> presented a intermediate layer in between these two places, in
> a special ref, such that:
> base is in .gitmodules
> overwritten via refs/meta/submodules:.gitmodules
> overwritten via the .git/config
> 
> The intermediate would be a config file that is tracked on another
> ref. This (a) decouples main project history from submodule history
> and (b) makes it easier to distribute as it is part of the repository.
> 
> For example (a) is desired if you dig up an old project and the
> submodules have all moved from one git hosting provider to another.
> Another example would be when you fork a project with submodules
> and don't want to mess with the main history but you just want to
> adjust the submodule URLs. That is possible with such an intermediate
> additional place.
> 
> For (b) you can imagine the fork that you want to distribute in your
> community and you don't want to tell everyone to change the
> submodule URLs, but instead you can provide them with a prepared
> .gitmodules file, that they have to place into that special ref (which
> can be done via fetching).
> 
> I digress as these ideas seem to be orthogonal to your patch series,
> just FYI. prior discussion starting at:
> https://public-inbox.org/git/1462317985-640-1-git-send-email-sbel...@google.com/
> I recall there was a better discussion even prior to that, but have no
> link handy.
>

Just to understand, is that 'refs/meta/submodules:.gitmodules' file
meant to be managed manually? Or do you imagine some options to
instruct "git submodule" to *write* to that file instead of .gitmodules?

In the latter case your idea could cover my use case too, couldn't it?

But most importantly, is this realistically going to be added?
Currently I am not that familiar with the git code base to do it
myself, and I've never explicitly dealt with special refs in git.

The approach of this patch series is a lot simpler, and something I can
work on in my spare time.

Presumably (b) could even be partially supported with my changes, by
having both '.gitmodules' and some custom '.gitmodules-alternative'
files in the repository and tell users to clone with:

  git clone --config core.submodulesFile=.gitmodules-alternative URL

Not as clean as your idea but doable.

[...]
> > Since the gitmodules file is meant to be checked in into the repository,
> > the overridden file path should be relative to the work-tree; is there
> > a way to enforce this constraint at run time (i.e. validate the config
> > value), or is it enough to have it documented?
> 
> I think we'd want to check at run time, if we need this constraint.
> 

I'll look into it once we decide which the way to go.

Thank you.

Ciao,
   Antonio

-- 
Antonio Ospite
https://ao2.it
https://twitter.com/ao2it

A: Because it messes up the order in which people normally read text.
   See http://en.wikipedia.org/wiki/Posting_style
Q: Why is top-posting such a bad thing?

Re: [RFC 00/10] Make .the gitmodules file path configurable

[Antonio Ospite]

On Fri, 13 Apr 2018 00:20:37 +0200
Antonio Ospite  wrote:

[...]
> Antonio Ospite (10):
>   submodule: add 'core.submodulesFile' to override the '.gitmodules'
> path
>   submodule: fix getting custom gitmodule file in fetch command
>   submodule: use the 'submodules_file' variable in output messages
>   submodule: document 'core.submodulesFile' and fix references to
> '.gitmodules'
>   submodule: adjust references to '.gitmodules' in comments
>   completion: add 'core.submodulesfile' to the git-completion.bash file
>   XXX: wrap-for-bin.sh: set 'core.submodulesFile' for each git
> invocation
>   XXX: submodule: fix t1300-repo-config.sh to take into account the new
> config
>   XXX: submodule: pass custom gitmodules file to 'test-tool
> submodule-config'
>   XXX: add a hacky script to test the changes with a patched test suite
> 

I am re-sending the last four highly experimental patches changing the
"XXX" into "FIXME" in the subject lines because vger.kernel.org refused
the original ones with the following message:

  The capital Triple-X in subject is way too often associated with junk
  email, please rephrase.

Sorry for the inconvenience.

-- 
Antonio Ospite
https://ao2.it
https://twitter.com/ao2it

A: Because it messes up the order in which people normally read text.
   See http://en.wikipedia.org/wiki/Posting_style
Q: Why is top-posting such a bad thing?

Re: [RFC 00/10] Make .the gitmodules file path configurable

[Stefan Beller]

Hi Antonio,

On Thu, Apr 12, 2018 at 3:20 PM, Antonio Ospite  wrote:
> Hi,
>
> vcsh[1] uses bare git repositories and detached work-trees to manage
> *distinct* sets of configuration files directly into $HOME.
>
> In general, submodules have worked perfectly fine with detached
> work-trees for some time[2,3,4].
>
> However when multiple repositories take turns using the same directory
> as their work-tree, and more than one of them want to use submodules,
> there could still be conflicts about the '.gitmodules' file because git
> hardcodes this path.
>
> For comparison, in case of '.gitignore' a similar conflict might arise,
> but git has alternative ways to specify exclude files, so vcsh solves
> this by setting core.excludesFile for each repository and track ignored
> files somewhere else (in ~/.gitignore.d/$VCSH_REPO_NAME).
>
> This is currently not possible with submodules configuration.

So far I agree w.r.t. Gits capabilities.

> So this series proposes a mechanism to set an alternative path for the
> submodules configuration file (from now on "gitmodules file").

That sounds interesting, so let's read on.

> Patches are based on fe0a9eaf31dd0c349ae4308498c33a5c3794b293.

... which is the current master branch by Junio.

> In commit 4c0eeafe4755 (cache.h: add GITMODULES_FILE macro)[5] the
> gitmodules file path definition was centralized, AFAIU this was done
> mainly to prevent typos, as checking a symbolic constant is something
> the compiler will do for us.

+cc Brandon author of said patch.

Digging up the discussion, this was indeed only done to prevent typos.
https://public-inbox.org/git/20170802172633.ga36...@google.com/

> Expanding on that change the first patch in the series makes the path
> customizable exposing a 'core.submodulesFile' configuration setting.

I guess the similarity to core.ignoreFile is desired here. Although these
mechanisms are very different.

> The new configuration setting can be used to set an *alternative*
> location for the gitmodules file; IMHO there is no need to provide
> *additional* locations like in the case of exclude files.

I think there *may* be a need for additional files.
Currently there is only the .gitmodules file and the configuration
in .git/config overriding settings from .gitmodules.

There was some discussion on the mailing list in the past, which
presented a intermediate layer in between these two places, in
a special ref, such that:
base is in .gitmodules
overwritten via refs/meta/submodules:.gitmodules
overwritten via the .git/config

The intermediate would be a config file that is tracked on another
ref. This (a) decouples main project history from submodule history
and (b) makes it easier to distribute as it is part of the repository.

For example (a) is desired if you dig up an old project and the
submodules have all moved from one git hosting provider to another.
Another example would be when you fork a project with submodules
and don't want to mess with the main history but you just want to
adjust the submodule URLs. That is possible with such an intermediate
additional place.

For (b) you can imagine the fork that you want to distribute in your
community and you don't want to tell everyone to change the
submodule URLs, but instead you can provide them with a prepared
.gitmodules file, that they have to place into that special ref (which
can be done via fetching).

I digress as these ideas seem to be orthogonal to your patch series,
just FYI. prior discussion starting at:
https://public-inbox.org/git/1462317985-640-1-git-send-email-sbel...@google.com/
I recall there was a better discussion even prior to that, but have no
link handy.


> For instance vcsh could set the location to
> '~/.gitmodules.d/$VCSH_REPO_NAME' to avoid conflicts.
>
> Since the gitmodules file is meant to be checked in into the repository,
> the overridden file path should be relative to the work-tree; is there
> a way to enforce this constraint at run time (i.e. validate the config
> value), or is it enough to have it documented?

I think we'd want to check at run time, if we need this constraint.

Thanks,
Stefan

[RFC 00/10] Make .the gitmodules file path configurable

[Antonio Ospite]

Hi,

vcsh[1] uses bare git repositories and detached work-trees to manage
*distinct* sets of configuration files directly into $HOME.

In general, submodules have worked perfectly fine with detached
work-trees for some time[2,3,4].

However when multiple repositories take turns using the same directory
as their work-tree, and more than one of them want to use submodules,
there could still be conflicts about the '.gitmodules' file because git
hardcodes this path.

For comparison, in case of '.gitignore' a similar conflict might arise,
but git has alternative ways to specify exclude files, so vcsh solves
this by setting core.excludesFile for each repository and track ignored
files somewhere else (in ~/.gitignore.d/$VCSH_REPO_NAME).

This is currently not possible with submodules configuration.

So this series proposes a mechanism to set an alternative path for the
submodules configuration file (from now on "gitmodules file").

Patches are based on fe0a9eaf31dd0c349ae4308498c33a5c3794b293.

In commit 4c0eeafe4755 (cache.h: add GITMODULES_FILE macro)[5] the
gitmodules file path definition was centralized, AFAIU this was done
mainly to prevent typos, as checking a symbolic constant is something
the compiler will do for us.

Expanding on that change the first patch in the series makes the path
customizable exposing a 'core.submodulesFile' configuration setting.

The new configuration setting can be used to set an *alternative*
location for the gitmodules file; IMHO there is no need to provide
*additional* locations like in the case of exclude files.

For instance vcsh could set the location to
'~/.gitmodules.d/$VCSH_REPO_NAME' to avoid conflicts.

Since the gitmodules file is meant to be checked in into the repository,
the overridden file path should be relative to the work-tree; is there
a way to enforce this constraint at run time (i.e. validate the config
value), or is it enough to have it documented?

The last patch adds a hacky 'test-custom-gitmodules-file.sh' script
which patches the test suite to fix all the hardcoded occurrences of
'.gitmodules' and runs it twice: once with the default value and once
with a custom path, passed via an environmental variable.

I guess in the final version just testing for a custom path (e.g.
'.gitmodules_custom') could be enough, as the default value can be seen
as a particular case.

The last thing I noticed is that git does not create config files
automatically if they are under a subdirectory which does not exist
already; for instance the following command would fail:

  git config -f newsubdir/test-config user.name Antonio

This means that if the user wanted to use something like:

  git -c core.submodulesFile=.gitmodules.d/repo_submodules ...

the subdirectory would have to be created beforehand. This is not a big
deal of course, but I was wondering if this is mentioned anywhere.
Fixing the current behavior to create the leading directories does not
seem hard, but I am not sure it is worth it.

Thanks,
   Antonio

[1] https://github.com/RichiH/vcsh
[2] 
http://git.661346.n2.nabble.com/git-submodule-vs-GIT-WORK-TREE-td7562165.html
[3] 
http://git.661346.n2.nabble.com/PATCH-Solve-git-submodule-issues-with-detached-work-trees-td7563377.html
[4] https://github.com/git/git/commit/be8779f7ac9a3be9aa783df008d59082f4054f67
[5] https://github.com/git/git/commit/4c0eeafe4755345b0f4636bf09904cf689703e11

Antonio Ospite (10):
  submodule: add 'core.submodulesFile' to override the '.gitmodules'
path
  submodule: fix getting custom gitmodule file in fetch command
  submodule: use the 'submodules_file' variable in output messages
  submodule: document 'core.submodulesFile' and fix references to
'.gitmodules'
  submodule: adjust references to '.gitmodules' in comments
  completion: add 'core.submodulesfile' to the git-completion.bash file
  XXX: wrap-for-bin.sh: set 'core.submodulesFile' for each git
invocation
  XXX: submodule: fix t1300-repo-config.sh to take into account the new
config
  XXX: submodule: pass custom gitmodules file to 'test-tool
submodule-config'
  XXX: add a hacky script to test the changes with a patched test suite

 Documentation/config.txt  | 18 +++--
 Documentation/git-add.txt |  4 +-
 Documentation/git-submodule.txt   | 45 +--
 Documentation/gitmodules.txt  | 15 ++--
 Documentation/gitsubmodules.txt   | 18 ++---
 .../technical/api-submodule-config.txt|  6 +-
 Makefile  |  3 +-
 builtin/fetch.c   |  2 +-
 builtin/mv.c  |  3 +-
 builtin/rm.c  |  3 +-
 builtin/submodule--helper.c   | 20 ++---
 cache.h   |  1 +
 config.c  | 13 ++--
 config.h  |  7 +-
 contrib/completion/git-completion.bash|  1