Re: [PATCH v2] fetch-pack: support negotiation tip whitelist

2018-06-28 Thread Brandon Williams
On 06/28, Jonathan Tan wrote:
> > This seems like a pretty difficult to use feature, requiring that I
> > provide the actual OIDs.  I think a much better UI would probably be to
> > accept a number of different things ranging from exact OIDs to actual
> > ref names or even better, allowing for ref-patterns which include globs.
> > That way I can do the following:
> >   
> >   git fetch --negotiation-tip=refs/remotes/my-remote/* my-remote
> > 
> > in order to easily limit the tips to all the refs I have from that
> > particular remote.
> 
> Actual ref names are already supported - it uses the same DWIM as
> others. As for globs, I thought of supporting both DWIM objects and the
> LHS of refspecs, but (1) it might be strange to support master and
> refs/heads/* but not heads/*,

I don't think that would be strange at all, and no where in git do we
handle heads/* but we do already handle refs/heads/* as well as DWIM
master.


> and (2) I can't think of anywhere in Git
> where you can provide either one - it's either SHA-1 and DWIM name, or
> SHA-1 and refspec, but not all three.

fetch is a perfect example of supporting all three.  I can do

  git fetch origin SHA1
  git fetch origin master
  git fetch origin refs/heads/*:refs/heads/*

-- 
Brandon Williams


Re: [PATCH v2] fetch-pack: support negotiation tip whitelist

2018-06-28 Thread Jonathan Tan
> This seems like a pretty difficult to use feature, requiring that I
> provide the actual OIDs.  I think a much better UI would probably be to
> accept a number of different things ranging from exact OIDs to actual
> ref names or even better, allowing for ref-patterns which include globs.
> That way I can do the following:
>   
>   git fetch --negotiation-tip=refs/remotes/my-remote/* my-remote
> 
> in order to easily limit the tips to all the refs I have from that
> particular remote.

Actual ref names are already supported - it uses the same DWIM as
others. As for globs, I thought of supporting both DWIM objects and the
LHS of refspecs, but (1) it might be strange to support master and
refs/heads/* but not heads/*, and (2) I can't think of anywhere in Git
where you can provide either one - it's either SHA-1 and DWIM name, or
SHA-1 and refspec, but not all three.


Re: [PATCH v2] fetch-pack: support negotiation tip whitelist

2018-06-28 Thread Brandon Williams
On 06/27, Jonathan Tan wrote:
> During negotiation, fetch-pack eventually reports as "have" lines all
> commits reachable from all refs. Allow the user to restrict the commits
> sent in this way by providing a whitelist of tips; only the tips
> themselves and their ancestors will be sent.
> 
> This feature is only supported for protocols that support connect or
> stateless-connect (such as HTTP with protocol v2).
> 
> This will speed up negotiation when the repository has multiple
> relatively independent branches (for example, when a repository
> interacts with multiple repositories, such as with linux-next [1] and
> torvalds/linux [2]), and the user knows which local branch is likely to
> have commits in common with the upstream branch they are fetching.
> 
> [1] https://kernel.googlesource.com/pub/scm/linux/kernel/git/next/linux-next/
> [2] https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux/
> 
> Signed-off-by: Jonathan Tan 
> ---
> v2 is exactly the same as the original, except with user-facing
> documentation in Documentation/fetch-options.txt.
> 
> > What's the plan to expose this "feature" to end-users?  There is no
> > end-user facing documentation added by this patch, and in-code
> > comments only talk about what (mechanical) effect the option has,
> > but not when a user may want to use the feature, or how the user
> > would best decide the set of commits to pass to this new option.
> 
> Jonathan Nieder also mentioned this. Lack of documentation was an
> oversight, sorry. I've added it in this version.
> 
> > Would something like this
> >
> > git fetch $(git for-each-ref \
> > --format=--nego-tip="%(objectname)" \
> > refs/remotes/linux-next/) \
> > linux-next
> >
> > be an expected typical way to pull from one remote, exposing only
> > the tips of refs we got from that remote and not the ones we
> > obtained from other places?
> 
> Yes, that is one way. Alternatively, if the user is only fetching one
> branch, they may also want to specify a single branch.
> ---
>  Documentation/fetch-options.txt | 12 +++
>  builtin/fetch.c | 21 +
>  fetch-pack.c| 19 ++--
>  fetch-pack.h|  7 +
>  t/t5510-fetch.sh| 55 +
>  transport-helper.c  |  3 ++
>  transport.c |  1 +
>  transport.h | 10 ++
>  8 files changed, 126 insertions(+), 2 deletions(-)
> 
> diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt
> index 97d3217df9..80c4c94595 100644
> --- a/Documentation/fetch-options.txt
> +++ b/Documentation/fetch-options.txt
> @@ -42,6 +42,18 @@ the current repository has the same history as the source 
> repository.
>   .git/shallow. This option updates .git/shallow and accept such
>   refs.
>  
> +--negotiation-tip::
> + By default, Git will report, to the server, commits reachable
> + from all local refs to find common commits in an attempt to
> + reduce the size of the to-be-received packfile. If specified,
> + Git will only report commits reachable from the given commit.
> + This is useful to speed up fetches when the user knows which
> + local ref is likely to have commits in common with the
> + upstream ref being fetched.

This seems like a pretty difficult to use feature, requiring that I
provide the actual OIDs.  I think a much better UI would probably be to
accept a number of different things ranging from exact OIDs to actual
ref names or even better, allowing for ref-patterns which include globs.
That way I can do the following:
  
  git fetch --negotiation-tip=refs/remotes/my-remote/* my-remote

in order to easily limit the tips to all the refs I have from that
particular remote.

> ++
> +This option may be specified more than once; if so, Git will report
> +commits reachable from any of the given commits.
> +
>  ifndef::git-pull[]
>  --dry-run::
>   Show what would be done, without making any changes.
> diff --git a/builtin/fetch.c b/builtin/fetch.c
> index ea5b9669ad..12daec0f3b 100644
> --- a/builtin/fetch.c
> +++ b/builtin/fetch.c
> @@ -63,6 +63,7 @@ static int shown_url = 0;
>  static struct refspec refmap = REFSPEC_INIT_FETCH;
>  static struct list_objects_filter_options filter_options;
>  static struct string_list server_options = STRING_LIST_INIT_DUP;
> +static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;
>  
>  static int git_fetch_config(const char *k, const char *v, void *cb)
>  {
> @@ -174,6 +175,8 @@ static struct option builtin_fetch_options[] = {
>   TRANSPORT_FAMILY_IPV4),
>   OPT_SET_INT('6', "ipv6", , N_("use IPv6 addresses only"),
>   TRANSPORT_FAMILY_IPV6),
> + OPT_STRING_LIST(0, "negotiation-tip", _tip, N_("revision"),
> + N_("report that we have only objects reachable from 
> this object")),
>