Re: [PATCH v3] fetch-pack: support negotiation tip whitelist

2018-06-29 Thread Junio C Hamano
Jonathan Tan  writes:

>> fetch is a perfect example of supporting all three.  I can do
>>
>>   git fetch origin SHA1
>>   git fetch origin master
>>   git fetch origin refs/heads/*:refs/heads/*
>
> OK, Brandon managed to convince me that this is fine. I've included glob
> support, supporting the same globs that git notes supports.

"git notes"???

As this is to be used in the context of "git fetch", using glob
e.g. "refs/heads/*", is sensible and good enough.  I was actually
wondering if we want the head-match refs/heads/, but as "git fetch
origin refs/heads/" does not work that way, I think we shouldn't.

This is a tangent, but didn't ref-in-want wanted to use head-match
refs/heads/ to match everything under refs/heads/?  If the latest
incarnation wants to do so, we may want to fix that.

> diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt
> index 97d3217df..6e4db1738 100644
> --- a/Documentation/fetch-options.txt
> +++ b/Documentation/fetch-options.txt
> @@ -42,6 +42,22 @@ the current repository has the same history as the source 
> repository.
>   .git/shallow. This option updates .git/shallow and accept such
>   refs.
>  
> +--negotiation-tip=::
> + By default, Git will report, to the server, commits reachable
> + from all local refs to find common commits in an attempt to
> + reduce the size of the to-be-received packfile. If specified,
> + Git will only report commits reachable from the given tips.
> + This is useful to speed up fetches when the user knows which
> + local ref is likely to have commits in common with the
> + upstream ref being fetched.
> ++
> +This option may be specified more than once; if so, Git will report
> +commits reachable from any of the given commits.
> ++
> +The argument to this option may be a glob on ref names, a ref, or the 
> (possibly
> +abbreviated SHA-1 of a commit. Specifying a glob is equivalent to specifying
> +this option multiple times, one for each matching ref name.
> +

> +static int add_oid(const char *refname, const struct object_id *oid, int 
> flags,
> +void *cb_data)
> +{
> + struct oid_array *oids = cb_data;
> + oid_array_append(oids, oid);
> + return 0;
> +}

This by itself is not worth a reason to reroll, but please make it a
habit to have a blank line after the run of decls before the first
statement, at least while we still forbid decl-after-stmt.  The
result is easier to read that way.

> +static void add_negotiation_tips(struct git_transport_options *smart_options)
> +{
> + struct oid_array *oids = xcalloc(1, sizeof(*oids));
> + int i;
> + for (i = 0; i < negotiation_tip.nr; i++) {
> + const char *s = negotiation_tip.items[i].string;
> + int old_nr;
> + if (!has_glob_specials(s)) {
> + struct object_id oid;
> + if (get_oid(s, ))
> + die("%s is not a valid object", s);
> + oid_array_append(oids, );
> + continue;
> + }
> + old_nr = oids->nr;
> + for_each_glob_ref(add_oid, s, oids);
> + if (old_nr == oids->nr)
> + warning("Ignoring --negotiation-tip=%s because it does 
> not match any refs",
> + s);
> + }
> + smart_options->negotiation_tips = oids;
> +}

This may insert duplicate object ids if two refs point at the same
object, or nego globs match the same ref twice, but it is OK to have
duplicate object ids in the resulting oids array is OK, because
rev_list_insert_ref() at the end will dedup them anyway.

> diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh
> index e402aee6a..8532a6faf 100755
> --- a/t/t5510-fetch.sh
> +++ b/t/t5510-fetch.sh
> @@ -865,4 +865,82 @@ test_expect_success C_LOCALE_OUTPUT 'fetch compact 
> output' '
>   test_cmp expect actual
>  '
>  
> +setup_negotiation_tip () {
> + SERVER="$1"
> + URL="$2"
> + USE_PROTOCOL_V2="$3"
> +
> + rm -rf "$SERVER" client trace &&
> + git init "$SERVER" &&
> + test_commit -C "$SERVER" alpha_1 &&
> + test_commit -C "$SERVER" alpha_2 &&
> + git -C "$SERVER" checkout --orphan beta &&
> + test_commit -C "$SERVER" beta_1 &&
> + test_commit -C "$SERVER" beta_2 &&
> +
> + git clone "$URL" client &&
> +
> + if [ "$USE_PROTOCOL_V2" -eq 1 ]

Style: "if test ..."

> + then
> + git -C "$SERVER" config protocol.version 2

broken &&-chaining?

> + git -C client config protocol.version 2
> + fi &&
> +
> + test_commit -C "$SERVER" beta_s &&
> + git -C "$SERVER" checkout master &&
> + test_commit -C "$SERVER" alpha_s &&
> + git -C "$SERVER" tag -d alpha_1 alpha_2 beta_1 beta_2
> +}


Re: [PATCH v3] fetch-pack: support negotiation tip whitelist

2018-06-28 Thread Brandon Williams
On 06/28, Jonathan Tan wrote:
> During negotiation, fetch-pack eventually reports as "have" lines all
> commits reachable from all refs. Allow the user to restrict the commits
> sent in this way by providing a whitelist of tips; only the tips
> themselves and their ancestors will be sent.
> 
> Both globs and single objects are supported.
> 
> This feature is only supported for protocols that support connect or
> stateless-connect (such as HTTP with protocol v2).
> 
> This will speed up negotiation when the repository has multiple
> relatively independent branches (for example, when a repository
> interacts with multiple repositories, such as with linux-next [1] and
> torvalds/linux [2]), and the user knows which local branch is likely to
> have commits in common with the upstream branch they are fetching.
> 
> [1] https://kernel.googlesource.com/pub/scm/linux/kernel/git/next/linux-next/
> [2] https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux/
> 
> Signed-off-by: Jonathan Tan 
> ---
> This is on jt/fetch-pack-negotiator.
> 
> > I don't think that would be strange at all, and no where in git do we
> > handle heads/* but we do already handle refs/heads/* as well as DWIM
> > master.
> >
> > > and (2) I can't think of anywhere in Git
> > > where you can provide either one - it's either SHA-1 and DWIM name, or
> > > SHA-1 and refspec, but not all three.
> >
> > fetch is a perfect example of supporting all three.  I can do
> >
> >   git fetch origin SHA1
> >   git fetch origin master
> >   git fetch origin refs/heads/*:refs/heads/*
> 
> OK, Brandon managed to convince me that this is fine. I've included glob
> support, supporting the same globs that git notes supports.
> ---
>  Documentation/fetch-options.txt | 16 +++
>  builtin/fetch.c | 41 +
>  fetch-pack.c| 19 +++-
>  fetch-pack.h|  7 +++
>  t/t5510-fetch.sh| 78 +
>  transport-helper.c  |  3 ++
>  transport.c |  1 +
>  transport.h | 10 +
>  8 files changed, 173 insertions(+), 2 deletions(-)
> 
> diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt
> index 97d3217df..6e4db1738 100644
> --- a/Documentation/fetch-options.txt
> +++ b/Documentation/fetch-options.txt
> @@ -42,6 +42,22 @@ the current repository has the same history as the source 
> repository.
>   .git/shallow. This option updates .git/shallow and accept such
>   refs.
>  
> +--negotiation-tip=::
> + By default, Git will report, to the server, commits reachable
> + from all local refs to find common commits in an attempt to
> + reduce the size of the to-be-received packfile. If specified,
> + Git will only report commits reachable from the given tips.
> + This is useful to speed up fetches when the user knows which
> + local ref is likely to have commits in common with the
> + upstream ref being fetched.
> ++
> +This option may be specified more than once; if so, Git will report
> +commits reachable from any of the given commits.
> ++
> +The argument to this option may be a glob on ref names, a ref, or the 
> (possibly
> +abbreviated SHA-1 of a commit. Specifying a glob is equivalent to specifying
> +this option multiple times, one for each matching ref name.

I think you're missing a closing ')'

Aside from that nit this patch looks good, thanks!


-- 
Brandon Williams