[PATCH v4 13/23] midx: write object ids in a chunk

2018-07-12 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- Documentation/technical/pack-format.txt | 4 +++ midx.c | 47 +++-- midx.h | 1 + t/helper/test-read-midx.c | 2 ++ t/t5319-multi-pack-index.sh

[PATCH v4 16/23] config: create core.multiPackIndex setting

2018-07-12 Thread Derrick Stolee
an error in the prepare_multi_pack_index_one(), but with later commits will catch errors in object lookups, abbreviations, and approximate object counts. Signed-off-by: Derrick Stolee midx: prepare midxed_git struct Signed-off-by: Derrick Stolee --- Documentation/config.txt| 5

[PATCH v4 18/23] midx: use midx in abbreviation calculations

2018-07-12 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- midx.c | 11 + midx.h | 3 +++ packfile.c | 6 + packfile.h | 1 + sha1-name.c | 70 + 5 files changed, 91 insertions(+) diff --git a/midx.c b/midx.c index 182535933c..4e014ff6e3

[PATCH v4 08/23] packfile: generalize pack directory list

2018-07-12 Thread Derrick Stolee
In anticipation of sharing the pack directory listing with the multi-pack-index, generalize prepare_packed_git_one() into for_each_file_in_pack_dir(). Signed-off-by: Derrick Stolee --- packfile.c | 101 + packfile.h | 6 2 files changed

[PATCH v4 17/23] midx: read objects from multi-pack-index

2018-07-12 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- midx.c | 91 +- midx.h | 3 ++ packfile.c | 8 - 3 files changed, 100 insertions(+), 2 deletions(-) diff --git a/midx.c b/midx.c index 4090cf4ca4..182535933c 100644 --- a/midx.c +++ b/midx.c

[PATCH v4 12/23] midx: sort and deduplicate objects from packfiles

2018-07-12 Thread Derrick Stolee
the data, copy to a local array, then sort. Copy only the de-duplicated entries. Select the duplicate based on the most-recent modified time of a packfile containing the object. Signed-off-by: Derrick Stolee --- midx.c | 128 + packfile.c

[PATCH v4 20/23] midx: use midx in approximate_object_count

2018-07-12 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- packfile.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/packfile.c b/packfile.c index c0eb5ac885..97e7812b6b 100644 --- a/packfile.c +++ b/packfile.c @@ -861,10 +861,13 @@ unsigned long approximate_object_count(void) { if (!the_repository

[PATCH v4 15/23] midx: write object offsets

2018-07-12 Thread Derrick Stolee
does not match the actual packfile offsets. The multi-pack-index offset does match the (corrupted) pack-index offset, so a future feature will compare these offsets during a 'verify' step. Signed-off-by: Derrick Stolee --- Documentation/technical/pack-format.txt | 15 +++- midx.c

[PATCH v4 14/23] midx: write object id fanout chunk

2018-07-12 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- Documentation/technical/pack-format.txt | 5 +++ midx.c | 53 +++-- midx.h | 1 + t/helper/test-read-midx.c | 4 +- t/t5319-multi-pack-index.sh

[PATCH v4 11/23] midx: read pack names into array

2018-07-12 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- midx.c | 17 + midx.h | 1 + t/helper/test-read-midx.c | 5 + t/t5319-multi-pack-index.sh | 17 - 4 files changed, 35 insertions(+), 5 deletions(-) diff --git a/midx.c b/midx.c

[PATCH v4 10/23] multi-pack-index: write pack names in chunk

2018-07-12 Thread Derrick Stolee
-by: Derrick Stolee --- Documentation/technical/pack-format.txt | 6 + midx.c | 174 +++- midx.h | 2 + t/helper/test-read-midx.c | 7 + t/t5319-multi-pack-index.sh | 3 +- 5 files

[PATCH v4 07/23] t5319: expand test data

2018-07-12 Thread Derrick Stolee
with no "real" data. This will be expanded in future commits, along with the test expectations. Signed-off-by: Derrick Stolee --- t/t5319-multi-pack-index.sh | 84 + 1 file changed, 84 insertions(+) diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pac

[PATCH v4 05/23] midx: write header information to lockfile

2018-07-12 Thread Derrick Stolee
the return value in a later commit. Signed-off-by: Derrick Stolee --- midx.c | 50 + t/t5319-multi-pack-index.sh | 4 ++- 2 files changed, 53 insertions(+), 1 deletion(-) diff --git a/midx.c b/midx.c index 32468db1a2..f85f2d334d 100644

[PATCH v4 04/23] multi-pack-index: add 'write' verb

2018-07-12 Thread Derrick Stolee
In anticipation of writing multi-pack-indexes, add a skeleton 'git multi-pack-index write' subcommand and send the options to a write_midx_file() method. Also create a skeleton test script that tests the 'write' subcommand. Signed-off-by: Derrick Stolee --- Documentation/git-multi-pack

[PATCH v4 06/23] multi-pack-index: load into memory

2018-07-12 Thread Derrick Stolee
Create a new multi_pack_index struct for loading multi-pack-indexes into memory. Create a test-tool builtin for reading basic information about that multi-pack-index to verify the correct data is written. Signed-off-by: Derrick Stolee --- Makefile| 1 + midx.c

[PATCH v4 03/23] multi-pack-index: add builtin

2018-07-12 Thread Derrick Stolee
This new 'git multi-pack-index' builtin will be the plumbing access for writing, reading, and checking multi-pack-index files. The initial implementation is a no-op. Signed-off-by: Derrick Stolee --- .gitignore | 3 ++- Documentation/git-multi-pack-index.txt | 36

[PATCH v4 00/23] Multi-pack-index (MIDX)

2018-07-12 Thread Derrick Stolee
the latest ds/multi-pack-index and this series as part of this message. You can see the CI builds for Linux, Mac, and Windows linked from the GitHub pull request [1]. Thanks, -Stolee [1] https://github.com/gitgitgadget/git/pull/5 Derrick Stolee (23): multi-pack-index: add design document multi-pack

[PATCH v4 01/23] multi-pack-index: add design document

2018-07-12 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- Documentation/technical/multi-pack-index.txt | 109 +++ 1 file changed, 109 insertions(+) create mode 100644 Documentation/technical/multi-pack-index.txt diff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi

[PATCH v4 02/23] multi-pack-index: add format details

2018-07-12 Thread Derrick Stolee
hat writes that information. Signed-off-by: Derrick Stolee --- Documentation/technical/pack-format.txt | 49 + 1 file changed, 49 insertions(+) diff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt index 70a99fd142..e060e69

Re: [PATCH v3 07/24] multi-pack-index: expand test data

2018-07-12 Thread Derrick Stolee
On 7/12/2018 2:02 PM, Eric Sunshine wrote: On Thu, Jul 12, 2018 at 10:10 AM Derrick Stolee wrote: On 7/6/2018 12:36 AM, Eric Sunshine wrote: There seems to be a fair bit of duplication in these tests which create objects. Is it possible to factor out some of this code into a shell function

Re: [PATCH v3 0/6] Object store refactoring: commit graph

2018-07-12 Thread Derrick Stolee
/ I re-read the patch series and think you addressed all feedback. I have no more to add. Reviewed-by: Derrick Stolee

Re: [PATCH v3 15/24] midx: write object offsets

2018-07-12 Thread Derrick Stolee
On 7/6/2018 1:27 AM, Eric Sunshine wrote: On Thu, Jul 5, 2018 at 8:54 PM Derrick Stolee wrote: The final pair of chunks for the multi-pack-index file stores the object offsets. We default to using 32-bit offsets as in the pack-index version 1 format, but if there exists an offset larger than

Re: [PATCH v3 16/24] config: create core.multiPackIndex setting

2018-07-12 Thread Derrick Stolee
On 7/12/2018 9:19 AM, Derrick Stolee wrote: On 7/6/2018 1:39 AM, Eric Sunshine wrote: On Thu, Jul 5, 2018 at 8:54 PM Derrick Stolee wrote: The core.multiPackIndex config setting controls the multi-pack- index (MIDX) feature. If false, the setting will disable all reads from the multi-pack

Re: [PATCH v3 06/24] multi-pack-index: load into memory

2018-07-12 Thread Derrick Stolee
On 7/9/2018 3:08 PM, Junio C Hamano wrote: Derrick Stolee writes: diff --git a/midx.h b/midx.h index dbdbe9f873..2d83dd9ec1 100644 --- a/midx.h +++ b/midx.h @@ -1,6 +1,10 @@ #ifndef __MIDX_H__ #define __MIDX_H__ +struct multi_pack_index; I actually was quite surprised that this struct

Re: [PATCH v3 16/24] config: create core.multiPackIndex setting

2018-07-12 Thread Derrick Stolee
On 7/12/2018 9:31 AM, SZEDER Gábor wrote: On Thu, Jul 12, 2018 at 3:01 PM Derrick Stolee wrote: On 7/11/2018 5:48 AM, SZEDER Gábor wrote: diff --git a/Documentation/config.txt b/Documentation/config.txt index ab641bf5a9..ab895ebb32 100644 --- a/Documentation/config.txt +++ b/Documentation

Re: [PATCH v3 07/24] multi-pack-index: expand test data

2018-07-12 Thread Derrick Stolee
On 7/6/2018 12:36 AM, Eric Sunshine wrote: On Thu, Jul 5, 2018 at 8:54 PM Derrick Stolee wrote: + for i in $(test_seq 6 10) + do + iii=$(printf '%03i' $i) + test-tool genrandom "bar" 200 >wide_delta_$iii && + test-too

Re: [PATCH v3 16/24] config: create core.multiPackIndex setting

2018-07-12 Thread Derrick Stolee
On 7/6/2018 1:39 AM, Eric Sunshine wrote: On Thu, Jul 5, 2018 at 8:54 PM Derrick Stolee wrote: The core.multiPackIndex config setting controls the multi-pack- index (MIDX) feature. If false, the setting will disable all reads from the multi-pack-index file. Add comparison commands in t5319

Re: [PATCH v3 16/24] config: create core.multiPackIndex setting

2018-07-12 Thread Derrick Stolee
On 7/11/2018 5:48 AM, SZEDER Gábor wrote: diff --git a/Documentation/config.txt b/Documentation/config.txt index ab641bf5a9..ab895ebb32 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -908,6 +908,10 @@ core.commitGraph:: Enable git commit graph feature. Allows

Re: [PATCH v2 6/6] commit-graph: add repo arg to graph readers

2018-07-11 Thread Derrick Stolee
On 7/10/2018 1:31 PM, Jonathan Tan wrote: +static void test_get_commit_tree_in_graph(const char *gitdir, + const char *worktree, + const struct object_id *commit_oid) +{ + struct repository r; + struct

Re: [PATCH v2 6/6] commit-graph: add repo arg to graph readers

2018-07-09 Thread Derrick Stolee
On 7/9/2018 4:44 PM, Jonathan Tan wrote: Add a struct repository argument to the functions in commit-graph.h that read the commit graph. (This commit does not affect functions that write commit graphs.) Because the commit graph functions can now read the commit graph of any repository, the

Re: [PATCH v2 on ds/commit-graph-fsck 0/6] Object store refactoring: commit graph

2018-07-09 Thread Derrick Stolee
On 7/9/2018 8:30 PM, Derrick Stolee wrote: On 7/9/2018 6:27 PM, Junio C Hamano wrote: Jonathan Tan writes: This is on ds/commit-graph-fsck. I saw that ds/commit-graph-fsck has been updated to the latest version (v7, including "gc.writeCommitGraph"), so I've rebased my chang

Re: [PATCH v2 on ds/commit-graph-fsck 0/6] Object store refactoring: commit graph

2018-07-09 Thread Derrick Stolee
ex pointer inside the_repository->objects, so the equivalent of this series will not be necessary for the MIDX series. This series looks good to me, so please add "Reviewed-by: Derrick Stolee " I think we are set for another series on top of this one that lets the commit-graph f

Re: [PATCH v2 1/6] commit-graph: refactor preparing commit graph

2018-07-09 Thread Derrick Stolee
On 7/9/2018 5:41 PM, Stefan Beller wrote: Hi Jonathan, On Mon, Jul 9, 2018 at 1:44 PM Jonathan Tan wrote: Two functions in the code (1) check if the repository is configured for commit graphs, (2) call prepare_commit_graph(), and (3) check if the graph exists. Move (1) and (3) into

Re: What's (not) cooking

2018-07-09 Thread Derrick Stolee
On 7/6/2018 6:57 PM, Junio C Hamano wrote: I'll be pushing out the integration branches with some updates, but there is no change in 'next' and below. The following topics I gave a quick look and gave them topic branches, but I had trouble merging them in 'pu' and making them work correctly or

Re: [PATCH 00/17] object_id part 14

2018-07-09 Thread Derrick Stolee
On 7/8/2018 11:12 PM, Jacob Keller wrote: On Sun, Jul 8, 2018 at 4:39 PM brian m. carlson wrote: This is the fourteenth series of patches to switch to using struct object_id and the_hash_algo. This series converts several core pieces to use struct object_id, including the oid* and hex

Re: [PATCH 07/17] commit: increase commit message buffer size

2018-07-09 Thread Derrick Stolee
On 7/8/2018 7:36 PM, brian m. carlson wrote: 100 bytes is not sufficient to ensure we can write a commit message buffer when using a 32-byte hash algorithm. Increase the buffer size to ensure we have sufficient space. Signed-off-by: brian m. carlson --- refs/files-backend.c | 2 +- 1 file

[PATCH v3 23/24] packfile: skip loading index if in multi-pack-index

2018-07-05 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- packfile.c | 19 +-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/packfile.c b/packfile.c index 2c819a0ad8..e6ecf12ab5 100644 --- a/packfile.c +++ b/packfile.c @@ -469,8 +469,19 @@ static int open_packed_git_1(struct packed_git

[PATCH v3 11/24] midx: read pack names into array

2018-07-05 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- midx.c | 17 + object-store.h | 1 + t/helper/test-read-midx.c | 5 + t/t5319-multi-pack-index.sh | 7 ++- 4 files changed, 29 insertions(+), 1 deletion(-) diff --git a/midx.c b/midx.c index

[PATCH v3 22/24] midx: prevent duplicate packfile loads

2018-07-05 Thread Derrick Stolee
to the packed_git linked list or the MRU list. We still need to load the packfiles that are not tracked by the multi-pack-index. Signed-off-by: Derrick Stolee --- packfile.c | 9 + 1 file changed, 9 insertions(+) diff --git a/packfile.c b/packfile.c index 97e7812b6b..2c819a0ad8 100644

[PATCH v3 07/24] multi-pack-index: expand test data

2018-07-05 Thread Derrick Stolee
write the same file with no "real" data. This will be expanded in future commits, along with the test expectations. Signed-off-by: Derrick Stolee --- t/t5319-multi-pack-index.sh | 99 + 1 file changed, 99 insertions(+) diff --git a/t/t5319-multi-pack-

[PATCH v3 16/24] config: create core.multiPackIndex setting

2018-07-05 Thread Derrick Stolee
on and off. This currently includes 'git rev-list' and 'git log' commands to trigger several object database reads. Signed-off-by: Derrick Stolee --- Documentation/config.txt| 4 +++ cache.h | 1 + config.c| 5 environment.c | 1

[PATCH v3 17/24] midx: prepare midxed_git struct

2018-07-05 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- midx.c | 22 ++ midx.h | 3 +++ object-store.h | 9 + packfile.c | 6 +- 4 files changed, 39 insertions(+), 1 deletion(-) diff --git a/midx.c b/midx.c index cc35abe7a2..d5a61c0c53 100644 --- a/midx.c +++ b

[PATCH v3 24/24] midx: clear midx on repack

2018-07-05 Thread Derrick Stolee
If a 'git repack' command replaces existing packfiles, then we must clear the existing multi-pack-index before moving the packfiles it references. Signed-off-by: Derrick Stolee --- builtin/repack.c | 8 midx.c | 12 midx.h | 1 + 3 files changed, 21

[PATCH v3 21/24] midx: use midx in approximate_object_count

2018-07-05 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- packfile.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/packfile.c b/packfile.c index c0eb5ac885..97e7812b6b 100644 --- a/packfile.c +++ b/packfile.c @@ -861,10 +861,13 @@ unsigned long approximate_object_count(void) { if (!the_repository

[PATCH v3 20/24] midx: use existing midx when writing new one

2018-07-05 Thread Derrick Stolee
Due to how Windows handles replacing a lockfile when there is an open handle, create the close_midx() method to close the existing midx before writing the new one. Signed-off-by: Derrick Stolee --- midx.c | 116 ++--- midx.h | 1 + 2 files

[PATCH v3 18/24] midx: read objects from multi-pack-index

2018-07-05 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- midx.c | 91 +- midx.h | 2 ++ object-store.h | 1 + packfile.c | 8 - 4 files changed, 100 insertions(+), 2 deletions(-) diff --git a/midx.c b/midx.c index d5a61c0c53..84b045060a 100644

[PATCH v3 15/24] midx: write object offsets

2018-07-05 Thread Derrick Stolee
does not match the actual packfile offsets. The multi-pack-index offset does match the (corrupted) pack-index offset, so a future feature will compare these offsets during a 'verify' step. Signed-off-by: Derrick Stolee --- Documentation/technical/pack-format.txt | 15 +++- midx.c

[PATCH v3 19/24] midx: use midx in abbreviation calculations

2018-07-05 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- midx.c | 11 ++ midx.h | 3 ++ packfile.c | 6 packfile.h | 1 + sha1-name.c | 70 + t/t5319-multi-pack-index.sh | 3

[PATCH v3 14/24] midx: write object id fanout chunk

2018-07-05 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- Documentation/technical/pack-format.txt | 5 +++ midx.c | 53 +++-- object-store.h | 1 + t/helper/test-read-midx.c | 4 +- t/t5319-multi-pack-index.sh

[PATCH v3 10/24] multi-pack-index: write pack names in chunk

2018-07-05 Thread Derrick Stolee
-by: Derrick Stolee --- Documentation/technical/pack-format.txt | 6 + midx.c | 176 +++- object-store.h | 2 + t/helper/test-read-midx.c | 7 + t/t5319-multi-pack-index.sh | 3 +- 5 files

[PATCH v3 02/24] multi-pack-index: add format details

2018-07-05 Thread Derrick Stolee
hat writes that information. Signed-off-by: Derrick Stolee --- Documentation/technical/pack-format.txt | 49 + 1 file changed, 49 insertions(+) diff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt index 70a99fd142..e060e69

[PATCH v3 06/24] multi-pack-index: load into memory

2018-07-05 Thread Derrick Stolee
Create a new multi_pack_index struct for loading multi-pack-indexes into memory. Create a test-tool builtin for reading basic information about that multi-pack-index to verify the correct data is written. Signed-off-by: Derrick Stolee --- Makefile| 1 + midx.c

[PATCH v3 09/24] multi-pack-index: read packfile list

2018-07-05 Thread Derrick Stolee
When constructing a multi-pack-index file for a given object directory, read the files within the enclosed pack directory and find matches that end with ".idx" and find the correct paired packfile using add_packed_git(). Signed-off-by: Derrick Stolee --- midx.c

[PATCH v3 05/24] midx: write header information to lockfile

2018-07-05 Thread Derrick Stolee
the return value in a later commit. Signed-off-by: Derrick Stolee --- midx.c | 50 + t/t5319-multi-pack-index.sh | 3 ++- 2 files changed, 52 insertions(+), 1 deletion(-) diff --git a/midx.c b/midx.c index 32468db1a2..f85f2d334d 100644

[PATCH v3 03/24] multi-pack-index: add builtin

2018-07-05 Thread Derrick Stolee
This new 'git multi-pack-index' builtin will be the plumbing access for writing, reading, and checking multi-pack-index files. The initial implementation is a no-op. Signed-off-by: Derrick Stolee --- .gitignore | 3 +- Documentation/git-multi-pack-index.txt | 36

[PATCH v3 12/24] midx: sort and deduplicate objects from packfiles

2018-07-05 Thread Derrick Stolee
the data, copy to a local array, then sort. Copy only the de-duplicated entries. Select the duplicate based on the most-recent modified time of a packfile containing the object. Signed-off-by: Derrick Stolee --- midx.c | 131 - packfile.c

[PATCH v3 08/24] packfile: generalize pack directory list

2018-07-05 Thread Derrick Stolee
In anticipation of sharing the pack directory listing with the multi-pack-index, generalize prepare_packed_git_one() into for_each_file_in_pack_dir(). Signed-off-by: Derrick Stolee --- packfile.c | 101 + packfile.h | 6 2 files changed

[PATCH v3 04/24] multi-pack-index: add 'write' verb

2018-07-05 Thread Derrick Stolee
In anticipation of writing multi-pack-indexes, add a 'git multi-pack-index write' subcommand and send the options to a write_midx_file() method. Also create a basic test file that tests the 'write' subcommand. Signed-off-by: Derrick Stolee --- Documentation/git-multi-pack-index.txt | 22

[PATCH v3 13/24] midx: write object ids in a chunk

2018-07-05 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- Documentation/technical/pack-format.txt | 4 +++ midx.c | 47 +++-- object-store.h | 1 + t/helper/test-read-midx.c | 2 ++ t/t5319-multi-pack-index.sh

[PATCH v3 00/24] Multi-pack-index (MIDX)

2018-07-05 Thread Derrick Stolee
that as a follow-up. Thanks, -Stolee [1] https://github.com/gitgitgadget/git/pull/5 Derrick Stolee (24): multi-pack-index: add design document multi-pack-index: add format details multi-pack-index: add builtin multi-pack-index: add 'write' verb midx: write header information to lockfil

[PATCH v3 01/24] multi-pack-index: add design document

2018-07-05 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- Documentation/technical/multi-pack-index.txt | 109 +++ 1 file changed, 109 insertions(+) create mode 100644 Documentation/technical/multi-pack-index.txt diff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi

Re: [PATCH v2 05/24] midx: write header information to lockfile

2018-07-05 Thread Derrick Stolee
On 6/25/2018 3:19 PM, Junio C Hamano wrote: Derrick Stolee writes: +#define MIDX_SIGNATURE 0x4d494458 /* "MIDX" */ +#define MIDX_VERSION 1 +#define MIDX_HASH_VERSION 1 +#define MIDX_HEADER_SIZE 12 + +static char *get_midx_filename(const char *object_dir) +{ + return xstrfm

Re: [PATCH v2 06/24] multi-pack-index: load into memory

2018-07-05 Thread Derrick Stolee
On 6/25/2018 3:38 PM, Junio C Hamano wrote: Derrick Stolee writes: + cat >expect <<- EOF "<<-\EOF", i.e. make it easy for readers to spot that there is no funny substitutions happening in the here-doc body. While I don't use substitutions in this patch, I do u

Re: [PATCH 08/23] commit-graph.c: mark more strings for translation

2018-07-02 Thread Derrick Stolee
On 6/30/2018 5:08 AM, Nguyễn Thái Ngọc Duy wrote: Signed-off-by: Nguyễn Thái Ngọc Duy --- commit-graph.c | 20 ++-- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/commit-graph.c b/commit-graph.c index b63a1fc85e..c8d521923c 100644 --- a/commit-graph.c +++

Re: [RFC PATCH 07/13] test-reach

2018-06-29 Thread Derrick Stolee
On 6/29/2018 5:54 PM, Stefan Beller wrote: On Fri, Jun 29, 2018 at 9:13 AM Derrick Stolee wrote: +# Construct a grid-like commit graph with points (x,y) +# with 1 <= x <= 10, 1 <= y <= 10, where (x,y) has +# parents (x-1, y) and (x, y-1), keeping in mind that +# we d

Re: [RFC PATCH 06/13] commit-reach: move can_all_from_reach_with_flag()

2018-06-29 Thread Derrick Stolee
On 6/29/2018 5:47 PM, Stefan Beller wrote: Hi Derrick, +static int reachable(struct commit *from, int with_flag, int assign_flag, time_t min_commit_date) [...] + if (commit->date < min_commit_date) + continue; [...] +int

Re: [RFC PATCH 03/13] commit-reach: move commit_contains from ref-filter

2018-06-29 Thread Derrick Stolee
On 6/29/2018 5:38 PM, Stefan Beller wrote: Hi Derrick, +struct ref_filter_cbdata { + struct ref_array *array; + struct ref_filter *filter; + struct contains_cache contains_cache; + struct contains_cache no_contains_cache; These members also seem to be moved

Re: [RFC PATCH 00/13] Consolidate reachability logic

2018-06-29 Thread Derrick Stolee
This RFC is available as a GitHub pull request [1]. Thanks, -Stolee [1] https://github.com/derrickstolee/git/pull/8

[RFC PATCH 07/13] test-reach

2018-06-29 Thread Derrick Stolee
--- Makefile | 1 + t/helper/test-reach.c | 123 ++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/t6600-test-reach.sh | 81 5 files changed, 207 insertions(+) create mode 100644

[RFC PATCH 08/13] test-reach: test reduce_heads()

2018-06-29 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- t/t6600-test-reach.sh | 26 ++ 1 file changed, 26 insertions(+) diff --git a/t/t6600-test-reach.sh b/t/t6600-test-reach.sh index c9337b6b46..0f60db9c60 100755 --- a/t/t6600-test-reach.sh +++ b/t/t6600-test-reach.sh @@ -78,4 +78,30

[RFC PATCH 05/13] upload-pack: refactor ok_to_give_up()

2018-06-29 Thread Derrick Stolee
In anticipation of consolidating all commit reachability algorithms, refactor ok_to_give_up() in order to allow splitting its logic into an external method. Signed-off-by: Derrick Stolee --- upload-pack.c | 28 +--- 1 file changed, 17 insertions(+), 11 deletions(-) diff

[RFC PATCH 11/13] commit-reach: make can_all_from_reach... linear

2018-06-29 Thread Derrick Stolee
all Case -- Before: 1.45 s After: 0.34 s Large Case -- Before: 5.83 s After: 0.37 s Note how the time increases between the two cases in the two versions. The new code increases relative to the number of commits that need to be walked, but not directly relative to the number

[RFC PATCH 03/13] commit-reach: move commit_contains from ref-filter

2018-06-29 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- commit-reach.c | 119 +++ commit-reach.h | 44 +-- fast-import.c | 1 + ref-filter.c | 147 +++-- 4 files changed, 141 insertions(+), 170 deletions(-) diff --git

[RFC PATCH 04/13] upload-pack: make reachable() more generic

2018-06-29 Thread Derrick Stolee
In anticipation of moving the reachable() method to commit-reach.c, modify the prototype to be more generic to flags known outside of upload-pack.c. Also rename 'want' to 'from' to make the statement more clear outside of the context of haves/wants negotiation. Signed-off-by: Derrick Stolee

[RFC PATCH 09/13] commit-reach: test can_all_from_reach

2018-06-29 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- commit-reach.c| 52 +++ commit-reach.h| 4 +++- t/helper/test-reach.c | 7 -- t/t6600-test-reach.sh | 51 +++--- upload-pack.c | 2 +- 5 files changed, 105

[RFC PATCH 13/13] commit-reach: use can_all_from_reach

2018-06-29 Thread Derrick Stolee
there using 'test-tool reach ref_newer': Before: 0.12 s After: 0.11 s Signed-off-by: Derrick Stolee --- One thing I know is missing from this commit is a special-case to use the old logic when there is no commit-graph present. The can_all_from_reach() algorithm can be worse when we do not have good

[RFC PATCH 12/13] commit-reach: use is_descendant_of for ref_newer

2018-06-29 Thread Derrick Stolee
- A:v4.9 B:v3.19 Before: 0.11 s After: 0.10 s To test the negative case, add a new commit with parent v3.19, regenerate the commit-graph, and then run with B pointing at that commit. Before: 0.52 s After: 0.12 s Signed-off-by: Derrick Stolee --- commit-reach.c | 17 - commit.c

[RFC PATCH 10/13] commit-reach: test is_descendant_of

2018-06-29 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- t/helper/test-reach.c | 2 ++ t/t6600-test-reach.sh | 25 + 2 files changed, 27 insertions(+) diff --git a/t/helper/test-reach.c b/t/helper/test-reach.c index 88639a2945..14aaef5bff 100644 --- a/t/helper/test-reach.c +++ b/t/helper/test

[RFC PATCH 06/13] commit-reach: move can_all_from_reach_with_flag()

2018-06-29 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- commit-reach.c | 61 + commit-reach.h | 8 +++ upload-pack.c | 62 +++--- 3 files changed, 72 insertions(+), 59 deletions(-) diff --git a/commit-reach.c b/commit

[RFC PATCH 01/13] commit-reach: move walk methods from commit.c

2018-06-29 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- Makefile | 1 + commit-reach.c | 359 + commit-reach.h | 41 ++ commit.c | 358 4 files changed, 401 insertions(+), 358 deletions(-) create mode

[RFC PATCH 00/13] Consolidate reachability logic

2018-06-29 Thread Derrick Stolee
il the lookup_commit() and parse_commit() code is stable again. Thanks, -Stolee Derrick Stolee (13): commit-reach: move walk methods from commit.c commit-reach: move ref_newer from remote.c commit-reach: move commit_contains from ref-filter upload-pack: make reachable() more generic upload-pack: re

[RFC PATCH 02/13] commit-reach: move ref_newer from remote.c

2018-06-29 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- builtin/remote.c | 1 + commit-reach.c | 52 commit-reach.h | 2 ++ http-push.c | 1 + remote.c | 48 +--- remote.h | 1 - 6 files changed, 57

Re: [PATCH v3 00/32] object-store: lookup_commit

2018-06-29 Thread Derrick Stolee
On 6/28/2018 9:21 PM, Stefan Beller wrote: This continues the elimination of global variables in the object store and teaches lookup_commit[_reference] and alike to handle a_repository. This is also available as https://github.com/stefanbeller/git/tree/object-store-lookup-commit or applies on

Re: [PATCH v3 10/32] commit: add repository argument to parse_commit_buffer

2018-06-29 Thread Derrick Stolee
On 6/28/2018 9:22 PM, Stefan Beller wrote: Add a repository argument to allow the callers of parse_commit_buffer to be more specific about which repository to act on. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet. As

Re: [PATCH v3 09/32] commit: add repository argument to lookup_commit

2018-06-29 Thread Derrick Stolee
On 6/28/2018 9:21 PM, Stefan Beller wrote: Add a repository argument to allow callers of lookup_commit to be more specific about which repository to handle. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet. As with the

Re: What's cooking in git.git (Jun 2018, #07; Thu, 28)

2018-06-29 Thread Derrick Stolee
On 6/28/2018 6:42 PM, Stefan Beller wrote: On Thu, Jun 28, 2018 at 2:40 PM Junio C Hamano wrote: * ds/commit-graph-fsck (2018-06-27) 22 commits [...] "git fsck" learns to make sure the optional commit-graph file is in a sane state. Is this ready for 'next'? I hope so, as I plan to

Re: [PATCH v7 07/22] commit-graph: add 'verify' subcommand

2018-06-29 Thread Derrick Stolee
On 6/27/2018 5:59 PM, Jonathan Tan wrote: +int verify_commit_graph(struct repository *r, struct commit_graph *g) I haven't had the time to review this patch set, but I did rebase my object store refactoring [1] on this and wrote a test: static void test_verify_commit_graph(const char

[PATCH] commit-graph: fix documentation inconsistencies

2018-06-28 Thread Derrick Stolee
in the documentation, but unfortunately reduces the number of commits that we can store in the commit-graph. Update that estimate, too. Reported-by: Grant Welch Signed-off-by: Derrick Stolee --- Documentation/technical/commit-graph-format.txt | 10 +- 1 file changed, 5 insertions(+),

Re: Inconsistencies in commit-graph technical docs.

2018-06-28 Thread Derrick Stolee
On 6/28/2018 1:11 AM, Grant Welch wrote: I recently read the "Supercharging the Git Commit Graph blog by Derrick Stolee. I found the article interesting and wanted to verify the performance numbers for myself. Then that led me to want to know more about the implementation, so I

Re: [PATCH v7 21/22] gc: automatically write commit-graph files

2018-06-27 Thread Derrick Stolee
On 6/27/2018 2:09 PM, Junio C Hamano wrote: Derrick Stolee writes: @@ -40,6 +41,7 @@ static int aggressive_depth = 50; static int aggressive_window = 250; static int gc_auto_threshold = 6700; static int gc_auto_pack_limit = 50; +static int gc_write_commit_graph = 0; Please avoid

[PATCH v7 20/22] commit-graph: add '--reachable' option

2018-06-27 Thread Derrick Stolee
When writing commit-graph files, it can be convenient to ask for all reachable commits (starting at the ref set) in the resulting file. This is particularly helpful when writing to stdin is complicated, such as a future integration with 'git gc'. Signed-off-by: Derrick Stolee --- Documentation

[PATCH v7 19/22] commit-graph: use string-list API for input

2018-06-27 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- builtin/commit-graph.c | 39 +-- commit-graph.c | 15 +++ commit-graph.h | 7 +++ 3 files changed, 23 insertions(+), 38 deletions(-) diff --git a/builtin/commit-graph.c b/builtin/commit

[PATCH v7 09/22] commit-graph: verify required chunks are present

2018-06-27 Thread Derrick Stolee
The commit-graph file requires the following three chunks: * OID Fanout * OID Lookup * Commit Data If any of these are missing, then the 'verify' subcommand should report a failure. This includes the chunk IDs malformed or the chunk count is truncated. Signed-off-by: Derrick Stolee --- commit

[PATCH v7 18/22] fsck: verify commit-graph

2018-06-27 Thread Derrick Stolee
-graph file from the other fsck details. 2. The commit-graph verification requires the commits to be loaded in a specific order to guarantee we parse from the commit-graph file for some objects and from the object database for others. Signed-off-by: Derrick Stolee --- Documentation/git

[PATCH v7 17/22] commit-graph: verify contents match checksum

2018-06-27 Thread Derrick Stolee
our tests, as we break the checksum as we modify bytes of the commit-graph file. Signed-off-by: Derrick Stolee --- commit-graph.c | 16 ++-- t/t5318-commit-graph.sh | 6 ++ 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/commit-graph.c b/commit-gra

[PATCH v7 11/22] commit-graph: verify objects exist

2018-06-27 Thread Derrick Stolee
In the 'verify' subcommand, load commits directly from the object database to ensure they exist. Parse by skipping the commit-graph. Signed-off-by: Derrick Stolee --- commit-graph.c | 18 ++ t/t5318-commit-graph.sh | 7 +++ 2 files changed, 25 insertions(+) diff

[PATCH v7 22/22] commit-graph: update design document

2018-06-27 Thread Derrick Stolee
The commit-graph feature is now integrated with 'fsck' and 'gc', so remove those items from the "Future Work" section of the commit-graph design document. Also remove the section on lazy-loading trees, as that was completed in an earlier patch series. Signed-off-by: Derr

[PATCH v7 15/22] commit-graph: verify commit date

2018-06-27 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- commit-graph.c | 6 ++ t/t5318-commit-graph.sh | 6 ++ 2 files changed, 12 insertions(+) diff --git a/commit-graph.c b/commit-graph.c index e0f71658da..6d6c6beff9 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -986,6 +986,12 @@ int

[PATCH v7 21/22] gc: automatically write commit-graph files

2018-06-27 Thread Derrick Stolee
-by: Derrick Stolee --- Documentation/config.txt | 9 ++--- Documentation/git-gc.txt | 4 builtin/gc.c | 6 ++ t/t5318-commit-graph.sh | 14 ++ 4 files changed, 30 insertions(+), 3 deletions(-) diff --git a/Documentation/config.txt b/Documentation/config.txt

[PATCH v7 10/22] commit-graph: verify corrupt OID fanout and lookup

2018-06-27 Thread Derrick Stolee
In the commit-graph file, the OID fanout chunk provides an index into the OID lookup. The 'verify' subcommand should find incorrect values in the fanout. Similarly, the 'verify' subcommand should find out-of-order values in the OID lookup. Signed-off-by: Derrick Stolee --- commit-graph.c

<    1   2   3   4   5   6   7   8   9   10   >