[PATCH v2/GSoC/RFC] fetch: git fetch --deepen

2015-03-22 Thread Dongcan Jiang
This patch is just for discusstion. An option --deepen is added to
'git fetch'. When it comes to '--deepen', git should fetch N more
commits ahead the local shallow commit, where N is indicated by
'--depth=N'. [1]

e.g.

  (upstream)
   ---o---o---o---A---B

  (you)
  A---B

After excuting git fetch --depth=1 --deepen, (you) get one more
tip and it becomes

  (you)
  o---A---B

'--deepen' is designed to be a boolean option in this patch, which
is a little different from [1]. It's designed in this way, because
it can reuse '--depth' in the program, and just costs one more bit
in some data structure, such as fetch_pack_args,
git_transport_options.

Of course, as a patch for discussion, it remains a long way to go
before being complete.

1) Documents should be completed.
2) More test cases, expecially corner cases, should be added.
3) No need to get remote refs when it comes to '--deepen' option.
4) Validity on options combination should be checked.
5) smart-http protocol remains to be supported. [2]

[1] http://article.gmane.org/gmane.comp.version-control.git/212950
[2] http://article.gmane.org/gmane.comp.version-control.git/265050

Helped-by: Duy Nguyen pclo...@gmail.com
Helped-By: Eric Sunshine sunsh...@sunshineco.com
Helped-By: Junio C Hamano gits...@pobox.com
Signed-off-by: Dongcan Jiang dongcan.ji...@gmail.com
---

-Background-

At present the command git-fetch has one option depth which deepens or 
shortens the history of a shallow repository created by git clone with 
--depth=depth option (see git-clone[1]) to the specified number of commits 
from the tip of each remote branch history [2].

The drawback is that the starting point for deepening history is limited to the 
tip of each remote branch history. However, it's not able to satisfy the needs 
of users in some cases:
1) If the user wants to know the process of how a function was created and 
developed. What he expects is that he goes to the point of latest modification 
about the function and deepens the history further back from this current 
cut-off point by N commit directly, rather than having to guess what the depth 
is from tip of remote branch.
2) What's more, if the user only cares about the revisions of this function, 
the other commits after the latest modification are redundant for him, but he 
has to fetch them concomitantly, which wastes unnecessary time and space.


For the convenience of users, a new option should be added as deepen to allow 
users to just get history commits deepened by N commits from current cut-off 
point, irrespective of location where the tips are [3].


-Goals-

1) Supports for command git fetch --deepen;
2) Conflict Detection for this option to guarantee robustness;
3) Sufficient tests;
4) Guarantee for compatibility;
5) Sufficient documents;

More Details are shown in next section.


-Details-

--1. Current Workflow of git-fetch--

 +-+-+-+-+-+-+-+-+-+  +-+-+-+-+-+-+-+-+-+-+-+
 | git-upload-pack |  | git-unpack-objects  |
 +-+-+-+-+-+-+-+-+-+  +-+-+-+-+-+-+-+-+-+-+-+
|   ^  |   ^
v   |  v   |
 fetch   +-+-+-+-+-+-+-+-+-+  +-+-+-+-+-+-+-+  +-+-+-+-+-+-+
--- | get remote refs | --- | fetch refs  | --- | get_pack  | -+
 +-+-+-+-+-+-+-+-+-+  +-+-+-+-+-+-+-+  +-+-+-+-+-+-+  |
   | ^|
   v |v
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  +-+-+-+-+-+-+-+
| git-upload-pack   |  |   update|
+  +-+-+-+-+-+-+-+-+-+-+-+-+  +-+-+-+-+-+-+-+-+-+-+ +  | local refs  |
|  | generate wanted objs  | --- | create_pack_file  | |  +-+-+-+-+-+-+-+
+  |update shallows|  +-+-+-+-+-+-+-+-+-+-+ +
|  +-+-+-+-+-+-+-+-+-+-+-+-+| ^ |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
v |
  +-+-+-+-+-+-+-+-+-+-+
  | git-pack-objects  |
  +-+-+-+-+-+-+-+-+-+-+
  Figure 1. Workflow of git-fetch (Please view it in fixed-width fonts)

Figure 1 illustrates the workflow of git-fetch. When it comes to git-fetch, a 
series of operations are performed in cooperation with the server side via 
`transport` module.

At first, it gets remote refs from git-upload-pack, and then it fetches refs, 
i.e. generating wanted objs, updating shallows and creating pack file. Finally, 
it unpacks objects from the pack file and updates the local refs.

Actually, there are several protocols running the workflow above, which is 
showed in Table 1. It is determined in 

Re: [PATCH v2/GSoC/RFC] fetch: git fetch --deepen

2015-03-22 Thread Eric Sunshine
On Sun, Mar 22, 2015 at 11:24 AM, Dongcan Jiang dongcan.ji...@gmail.com wrote:
 This patch is just for discusstion. An option --deepen is added to
 'git fetch'. When it comes to '--deepen', git should fetch N more
 commits ahead the local shallow commit, where N is indicated by
 '--depth=N'. [1]
 Signed-off-by: Dongcan Jiang dongcan.ji...@gmail.com
 ---
 diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh
 index d78f320..3b960c8 100755
 --- a/t/t5510-fetch.sh
 +++ b/t/t5510-fetch.sh
 @@ -708,4 +708,16 @@ test_expect_success 'fetching a one-level ref works' '
 )
  '

 +test_expect_success 'fetching deepen' '
 +   git clone . deepen --depth=1  (

Quoting Junio[1]: ...make sure tests serve as good examples, tests
should stick to 'options first and then arguments'...

 +   cd deepen 
 +   git fetch .. foo --depth=1

See [1].

 +   git show foo
 +   test_must_fail git show foo~
 +   git fetch .. foo --depth=1 --deepen

See [1].

 +   git show foo~
 +   test_must_fail git show foo~2

Mentioned previously[2]: Broken -chain throughout this test.

 +   )
 +'
 +
  test_done

[1]: http://article.gmane.org/gmane.comp.version-control.git/265248
[2]: http://article.gmane.org/gmane.comp.version-control.git/265419
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [PATCH v2/GSoC/RFC] fetch: git fetch --deepen

2015-03-22 Thread Duy Nguyen
On Sun, Mar 22, 2015 at 10:24 PM, Dongcan Jiang dongcan.ji...@gmail.com wrote:
 This patch is just for discusstion. An option --deepen is added to
 'git fetch'. When it comes to '--deepen', git should fetch N more
 commits ahead the local shallow commit, where N is indicated by
 '--depth=N'. [1]

 e.g.

  (upstream)
   ---o---o---o---A---B

  (you)
  A---B

 After excuting git fetch --depth=1 --deepen, (you) get one more
 tip and it becomes

  (you)
  o---A---B

 '--deepen' is designed to be a boolean option in this patch, which
 is a little different from [1]. It's designed in this way, because
 it can reuse '--depth' in the program, and just costs one more bit
 in some data structure, such as fetch_pack_args,
 git_transport_options.

 Of course, as a patch for discussion, it remains a long way to go
 before being complete.

 1) Documents should be completed.
 2) More test cases, expecially corner cases, should be added.
 3) No need to get remote refs when it comes to '--deepen' option.
 4) Validity on options combination should be checked.
 5) smart-http protocol remains to be supported. [2]

Quick notes before $DAYJOB starts. Cool pictures, perhaps they could
be part of the commit message too.

Personally i still don't think not moving the refs is worth the effort
(and it's a waste if we have to send then drop objects for these
updated refs, but I didn't check carefully). So if you we don't needs
ref updates, we probably don't need to send want lines and sort of
simplify processing at upload-pack side.

And it makes me realise, we're loosing security a bit here. We
normally don't send anything that's not reachable from the visible ref
set. But we now would accept any shallow sha-1 and send some objects
regardless if these sha-1 are connected to any refs. We may need some
more checking in place to avoid this. See check_non_sha1_tip() for a
way to do it. Pack bitmaps may help as well, but I think that's behind
the scene (i.e. behind rev-list and we already can take advantage of
it).
-- 
Duy
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html