[PATCH v2 09/12] fsck: verify commit-graph

2018-05-11 Thread Derrick Stolee
-graph file from the other fsck details. 2. The commit-graph verification requires the commits to be loaded in a specific order to guarantee we parse from the commit-graph file for some objects and from the object database for others. Signed-off-by: Derrick Stolee <dsto...@microsoft.

[PATCH v2 12/12] commit-graph: update design document

2018-05-11 Thread Derrick Stolee
The commit-graph feature is now integrated with 'fsck' and 'gc', so remove those items from the "Future Work" section of the commit-graph design document. Also remove the section on lazy-loading trees, as that was completed in an earlier patch series. Signed-off-by: Derrick St

[PATCH v2 11/12] gc: automatically write commit-graph files

2018-05-11 Thread Derrick Stolee
-trivial 'git gc' command. Defaults to false while the commit-graph feature matures. We specifically do not want to turn this on by default until the commit-graph feature is fully integrated with history-modifying features like shallow clones. Signed-off-by: Derrick Stolee <dsto...@microsoft.

[PATCH v2 10/12] commit-graph: add '--reachable' option

2018-05-11 Thread Derrick Stolee
' after performing cleanup of the object database. Signed-off-by: Derrick Stolee <dsto...@microsoft.com> --- Documentation/git-commit-graph.txt | 8 ++-- builtin/commit-graph.c | 41 ++ 2 files changed, 43 insertions(+), 6 deletions(-)

[PATCH v2 08/12] commit-graph: verify commit contents against odb

2018-05-11 Thread Derrick Stolee
range causes a segmentation fault. Add a new check in insert_parent_or_die() that prevents this fault. Check for that error during the test, both in the typical parents and in the list of parents for octopus merges. Signed-off-by: Derrick Stolee <dsto...@microsoft.com> --- commit-g

[PATCH v2 07/12] commit-graph: load a root tree from specific graph

2018-05-11 Thread Derrick Stolee
When lazy-loading a tree for a commit, it will be important to select the tree from a specific struct commit_graph. Create a new method that specifies the commit-graph file and use that in get_commit_tree_in_graph(). Signed-off-by: Derrick Stolee <dsto...@microsoft.com> --- commit-graph.

Re: What's cooking in git.git (May 2018, #02; Thu, 17)

2018-05-17 Thread Derrick Stolee
On 5/17/2018 2:01 AM, Junio C Hamano wrote: * ds/generation-numbers (2018-05-02) 11 commits - commit-graph.txt: update design document - merge: check config before loading commits - commit: use generation number in remove_redundant() - commit: add short-circuit to paint_down_to_common()

Re: worktrees vs. alternates

2018-05-16 Thread Derrick Stolee
On 5/16/2018 6:33 AM, Ævar Arnfjörð Bjarmason wrote: [big snip] And here's where this isn't at all like "worktree", each of those 100 will have their own "master" branch, and they can all create 100 different branches called "topic" that can be different. This is the biggest difference. You

Re: Git log range reverse bug

2018-05-16 Thread Derrick Stolee
Hi Mendi, On 5/16/2018 2:19 PM, Mehdi Zeinali wrote: Git Version: Version: 2.14.2 When reversing a range in git log, it does not start from the expected commit: $ git show 8e11b4a41ec21e47fb0bf8b76e1edba739f57a9b commit 8e11b4a41ec21e47fb0bf8b76e1edba739f57a9b Author: Some Name

commit-graph: change in "best" merge-base when ambiguous

2018-05-21 Thread Derrick Stolee
Hello all, While working on the commit-graph feature, I made a test commit that sets core.commitGraph and gc.commitGraph to true by default AND runs 'git commit-graph write --reachable' after each 'git commit' command. This helped me find instances in the test suite where the commit-graph

Re: commit-graph: change in "best" merge-base when ambiguous

2018-05-22 Thread Derrick Stolee
On 5/22/2018 1:39 AM, Michael Haggerty wrote: On 05/21/2018 08:10 PM, Derrick Stolee wrote: [...] In the Discussion section of the `git merge-base` docs [1], we have the following:     When the history involves criss-cross merges, there can be more than one best common ancestor for two

[PATCH v3 00/20] Integrate commit-graph into 'fsck' and 'gc'

2018-05-24 Thread Derrick Stolee
mentioning: in "commit-graph: add '--reachable' option" I put the ref-iteration into a new external 'write_commit_graph_reachable()' method inside commit-graph.c. This makes the 'gc: automatically write commit-graph files' a simpler change. Thanks, -Stolee Derrick Stolee (20): commit-gra

[PATCH v3 06/20] commit-graph: add 'verify' subcommand

2018-05-24 Thread Derrick Stolee
state. Do not report any errors. Signed-off-by: Derrick Stolee <dsto...@microsoft.com> --- Documentation/git-commit-graph.txt | 6 ++ builtin/commit-graph.c | 38 ++ commit-graph.c | 26 ++ commit-g

[PATCH v3 11/20] commit-graph: verify root tree OIDs

2018-05-24 Thread Derrick Stolee
directly from the object database. Add checks for the root tree OID. Signed-off-by: Derrick Stolee <dsto...@microsoft.com> --- commit-graph.c | 17 - t/t5318-commit-graph.sh | 7 +++ 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/commit-gra

[PATCH v3 10/20] commit-graph: verify objects exist

2018-05-24 Thread Derrick Stolee
In the 'verify' subcommand, load commits directly from the object database to ensure they exist. Parse by skipping the commit-graph. Signed-off-by: Derrick Stolee <dsto...@microsoft.com> --- commit-graph.c | 20 t/t5318-commit-graph.sh | 7 +++ 2 files c

[PATCH v3 19/20] gc: automatically write commit-graph files

2018-05-24 Thread Derrick Stolee
-trivial 'git gc' command. Defaults to false while the commit-graph feature matures. We specifically do not want to turn this on by default until the commit-graph feature is fully integrated with history-modifying features like shallow clones. Signed-off-by: Derrick Stolee <dsto...@microsoft.

[PATCH v3 16/20] commit-graph: verify contents match checksum

2018-05-24 Thread Derrick Stolee
our tests, as we break the checksum as we modify bytes of the commit-graph file. Signed-off-by: Derrick Stolee <dsto...@microsoft.com> --- commit-graph.c | 16 ++-- t/t5318-commit-graph.sh | 6 ++ 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/commit-

[PATCH v3 02/20] commit-graph: fix GRAPH_MIN_SIZE

2018-05-24 Thread Derrick Stolee
the magic constants. Signed-off-by: Derrick Stolee <dsto...@microsoft.com> --- commit-graph.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/commit-graph.c b/commit-graph.c index a8c337dd77..82295f0975 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -33,10

[PATCH v3 05/20] commit-graph: load a root tree from specific graph

2018-05-24 Thread Derrick Stolee
When lazy-loading a tree for a commit, it will be important to select the tree from a specific struct commit_graph. Create a new method that specifies the commit-graph file and use that in get_commit_tree_in_graph(). Signed-off-by: Derrick Stolee <dsto...@microsoft.com> --- commit-graph.

[PATCH v3 12/20] commit-graph: verify parent list

2018-05-24 Thread Derrick Stolee
from the object database. Test these checks for corrupt parents, too many parents, and wrong parents. The octopus merge will be tested in a later commit. Signed-off-by: Derrick Stolee <dsto...@microsoft.com> --- commit-graph.c | 25 + t/t5318-commit-graph.s

[PATCH v3 01/20] commit-graph: UNLEAK before die()

2018-05-24 Thread Derrick Stolee
Signed-off-by: Derrick Stolee <dsto...@microsoft.com> --- builtin/commit-graph.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c index 37420ae0fd..f0875b8bf3 100644 --- a/builtin/commit-graph.c +++ b/builtin/commit-g

[PATCH v3 04/20] commit: force commit to parse from object database

2018-05-24 Thread Derrick Stolee
is explicit in avoiding commits from the commit-graph file. Signed-off-by: Derrick Stolee <dsto...@microsoft.com> --- commit.c | 9 +++-- commit.h | 1 + 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/commit.c b/commit.c index 1d28677dfb..6eaed0174c 100644 --- a/commit.c

[PATCH v3 14/20] commit-graph: verify commit date

2018-05-24 Thread Derrick Stolee
Signed-off-by: Derrick Stolee <dsto...@microsoft.com> --- commit-graph.c | 6 ++ t/t5318-commit-graph.sh | 6 ++ 2 files changed, 12 insertions(+) diff --git a/commit-graph.c b/commit-graph.c index ead92460c1..d2b291aca2 100644 --- a/commit-graph.c +++ b/commit-graph.c @@

[PATCH v3 09/20] commit-graph: verify corrupt OID fanout and lookup

2018-05-24 Thread Derrick Stolee
In the commit-graph file, the OID fanout chunk provides an index into the OID lookup. The 'verify' subcommand should find incorrect values in the fanout. Similarly, the 'verify' subcommand should find out-of-order values in the OID lookup. Signed-off-by: Derrick Stolee <dsto...@microsoft.

[PATCH v3 13/20] commit-graph: verify generation number

2018-05-24 Thread Derrick Stolee
, we drop the single-line condition on the output. Signed-off-by: Derrick Stolee <dsto...@microsoft.com> --- commit-graph.c | 18 ++ t/t5318-commit-graph.sh | 6 ++ 2 files changed, 24 insertions(+) diff --git a/commit-graph.c b/commit-graph.c index fff2

Re: [PATCH v2 03/12] commit-graph: test that 'verify' finds corruption

2018-05-24 Thread Derrick Stolee
On 5/21/2018 2:53 PM, Jakub Narebski wrote: +corrupt_data() { + file=$1 + pos=$2 + data="${3:-\0}" + printf "$data" | dd of="$file" bs=1 seek="$pos" conv=notrunc +} First, if we do this that way (and not by adding a test helper), the use of this function should be, I

[PATCH v3 15/20] commit-graph: test for corrupted octopus edge

2018-05-24 Thread Derrick Stolee
The commit-graph file has an extra chunk to store the parent int-ids for parents beyond the first parent for octopus merges. Our test repo has a single octopus merge that we can manipulate to demonstrate the 'verify' subcommand detects incorrect values in that chunk. Signed-off-by: Derrick Stolee

[PATCH v3 17/20] fsck: verify commit-graph

2018-05-24 Thread Derrick Stolee
-graph file from the other fsck details. 2. The commit-graph verification requires the commits to be loaded in a specific order to guarantee we parse from the commit-graph file for some objects and from the object database for others. Signed-off-by: Derrick Stolee <dsto...@microsoft.

[PATCH v3 08/20] commit-graph: verify required chunks are present

2018-05-24 Thread Derrick Stolee
The commit-graph file requires the following three chunks: * OID Fanout * OID Lookup * Commit Data If any of these are missing, then the 'verify' subcommand should report a failure. This includes the chunk IDs malformed or the chunk count is truncated. Signed-off-by: Derrick Stolee <d

[PATCH v3 20/20] commit-graph: update design document

2018-05-24 Thread Derrick Stolee
The commit-graph feature is now integrated with 'fsck' and 'gc', so remove those items from the "Future Work" section of the commit-graph design document. Also remove the section on lazy-loading trees, as that was completed in an earlier patch series. Signed-off-by: Derrick St

[PATCH v3 18/20] commit-graph: add '--reachable' option

2018-05-24 Thread Derrick Stolee
performing cleanup of the object database. Signed-off-by: Derrick Stolee <dsto...@microsoft.com> --- Documentation/git-commit-graph.txt | 8 ++-- builtin/commit-graph.c | 16 commit-graph.c | 32 commit-g

[PATCH v3 07/20] commit-graph: verify catches corrupt signature

2018-05-24 Thread Derrick Stolee
the test is run verbosely. Then, the output is checked to contain a specific error message. Signed-off-by: Derrick Stolee <dsto...@microsoft.com> --- t/t5318-commit-graph.sh | 43 +++ 1 file changed, 43 insertions(+) diff --git a/t/t5318-commit-graph.sh b/t

[PATCH v3 03/20] commit-graph: parse commit from chosen graph

2018-05-24 Thread Derrick Stolee
Before verifying a commit-graph file against the object database, we need to parse all commits from the given commit-graph file. Create parse_commit_in_graph_one() to target a given struct commit_graph. Signed-off-by: Derrick Stolee <dsto...@microsoft.com> --- commit-graph.

Re: [PATCH v3 01/20] commit-graph: UNLEAK before die()

2018-05-24 Thread Derrick Stolee
On 5/24/2018 6:47 PM, Stefan Beller wrote: On Thu, May 24, 2018 at 9:25 AM, Derrick Stolee <dsto...@microsoft.com> wrote: Signed-off-by: Derrick Stolee <dsto...@microsoft.com> --- builtin/commit-graph.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/bu

Re: [RFC PATCH 00/18] Multi-pack index (MIDX)

2018-06-06 Thread Derrick Stolee
On 6/6/2018 4:13 AM, Ævar Arnfjörð Bjarmason wrote: On Mon, Jan 08 2018, Derrick Stolee wrote: On 1/7/2018 5:42 PM, Ævar Arnfjörð Bjarmason wrote: On Sun, Jan 07 2018, Derrick Stolee jotted: git log --oneline --raw --parents Num Packs | Before MIDX | After MIDX | Rel % | 1 pack

[PATCH v5 12/21] commit-graph: verify parent list

2018-06-06 Thread Derrick Stolee
from the object database. Test these checks for corrupt parents, too many parents, and wrong parents. Add a boundary check to insert_parent_or_die() for when the parent position value is out of range. The octopus merge will be tested in a later commit. Signed-off-by: Derrick Stolee --- commit

[PATCH v5 11/21] commit-graph: verify root tree OIDs

2018-06-06 Thread Derrick Stolee
the object database. Add checks for the root tree OID. Signed-off-by: Derrick Stolee --- commit-graph.c | 17 - t/t5318-commit-graph.sh | 7 +++ 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/commit-graph.c b/commit-graph.c index 893cc2f346

[PATCH v5 02/21] commit-graph: fix GRAPH_MIN_SIZE

2018-06-06 Thread Derrick Stolee
the magic constants. Signed-off-by: Derrick Stolee --- commit-graph.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/commit-graph.c b/commit-graph.c index b63a1fc85e..f83f6d2373 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -35,10 +35,11 @@ #define GRAPH_LAST_EDGE

[PATCH v5 08/21] commit-graph: verify required chunks are present

2018-06-06 Thread Derrick Stolee
The commit-graph file requires the following three chunks: * OID Fanout * OID Lookup * Commit Data If any of these are missing, then the 'verify' subcommand should report a failure. This includes the chunk IDs malformed or the chunk count is truncated. Signed-off-by: Derrick Stolee --- commit

[PATCH v5 01/21] commit-graph: UNLEAK before die()

2018-06-06 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- builtin/commit-graph.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c index 37420ae0fd..f0875b8bf3 100644 --- a/builtin/commit-graph.c +++ b/builtin/commit-graph.c @@ -51,8 +51,11 @@ static

[PATCH v5 06/21] commit-graph: add 'verify' subcommand

2018-06-06 Thread Derrick Stolee
state. Do not report any errors. Helped-by: Ramsay Jones Signed-off-by: Derrick Stolee --- Documentation/git-commit-graph.txt | 6 + builtin/commit-graph.c | 38 ++ commit-graph.c | 23 ++ commit-graph.h

[PATCH v5 04/21] commit: force commit to parse from object database

2018-06-06 Thread Derrick Stolee
is explicit in avoiding commits from the commit-graph file. Signed-off-by: Derrick Stolee --- commit.c | 9 +++-- commit.h | 1 + 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/commit.c b/commit.c index 298ad747c6..922bb68741 100644 --- a/commit.c +++ b/commit.c @@ -405,7

Re: [PATCH v5 18/21] commit-graph: use string-list API for input

2018-06-06 Thread Derrick Stolee
On 6/6/2018 8:11 AM, Ævar Arnfjörð Bjarmason wrote: On Wed, Jun 06 2018, Derrick Stolee wrote: Signed-off-by: Derrick Stolee --- builtin/commit-graph.c | 39 +-- commit-graph.c | 15 +++ commit-graph.h | 7 +++ 3 files

Re: [PATCH v4 17/21] fsck: verify commit-graph

2018-06-06 Thread Derrick Stolee
On 6/6/2018 7:08 AM, Ævar Arnfjörð Bjarmason wrote: On Mon, Jun 04 2018, Derrick Stolee wrote: + prepare_alt_odb(); + for (alt = alt_odb_list; alt; alt = alt->next) { + verify_argv[2] = "--object-dir"; +

Re: [PATCH v5 18/21] commit-graph: use string-list API for input

2018-06-06 Thread Derrick Stolee
On 6/6/2018 8:26 AM, Ævar Arnfjörð Bjarmason wrote: On Wed, Jun 06 2018, Derrick Stolee wrote: On 6/6/2018 8:11 AM, Ævar Arnfjörð Bjarmason wrote: On Wed, Jun 06 2018, Derrick Stolee wrote: Signed-off-by: Derrick Stolee --- builtin/commit-graph.c | 39

[PATCH v5 20/21] gc: automatically write commit-graph files

2018-06-06 Thread Derrick Stolee
-by: Derrick Stolee --- Documentation/config.txt | 10 +- Documentation/git-gc.txt | 4 builtin/gc.c | 6 ++ t/t5318-commit-graph.sh | 14 ++ 4 files changed, 33 insertions(+), 1 deletion(-) diff --git a/Documentation/config.txt b/Documentation/config.txt

[PATCH v5 13/21] commit-graph: verify generation number

2018-06-06 Thread Derrick Stolee
have a mix of zero and non-zero generation numbers. Create a test that sets one commit to generation zero and all following commits report a failure as they have non-zero generation in a file that contains generation number zero. Signed-off-by: Derrick Stolee --- commit-graph.c | 34

[PATCH v5 03/21] commit-graph: parse commit from chosen graph

2018-06-06 Thread Derrick Stolee
Before verifying a commit-graph file against the object database, we need to parse all commits from the given commit-graph file. Create parse_commit_in_graph_one() to target a given struct commit_graph. Signed-off-by: Derrick Stolee --- commit-graph.c | 18 +++--- 1 file changed, 15

[PATCH v5 10/21] commit-graph: verify objects exist

2018-06-06 Thread Derrick Stolee
In the 'verify' subcommand, load commits directly from the object database to ensure they exist. Parse by skipping the commit-graph. Signed-off-by: Derrick Stolee --- commit-graph.c | 17 + t/t5318-commit-graph.sh | 7 +++ 2 files changed, 24 insertions(+) diff

[PATCH v5 19/21] commit-graph: add '--reachable' option

2018-06-06 Thread Derrick Stolee
When writing commit-graph files, it can be convenient to ask for all reachable commits (starting at the ref set) in the resulting file. This is particularly helpful when writing to stdin is complicated, such as a future integration with 'git gc'. Signed-off-by: Derrick Stolee --- Documentation

[PATCH v5 18/21] commit-graph: use string-list API for input

2018-06-06 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- builtin/commit-graph.c | 39 +-- commit-graph.c | 15 +++ commit-graph.h | 7 +++ 3 files changed, 23 insertions(+), 38 deletions(-) diff --git a/builtin/commit-graph.c b/builtin/commit

[PATCH v5 14/21] commit-graph: verify commit date

2018-06-06 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- commit-graph.c | 6 ++ t/t5318-commit-graph.sh | 6 ++ 2 files changed, 12 insertions(+) diff --git a/commit-graph.c b/commit-graph.c index b97fa05ec9..d83f0ce5d5 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -985,6 +985,12 @@ int

[PATCH v5 16/21] commit-graph: verify contents match checksum

2018-06-06 Thread Derrick Stolee
our tests, as we break the checksum as we modify bytes of the commit-graph file. Signed-off-by: Derrick Stolee --- commit-graph.c | 16 ++-- t/t5318-commit-graph.sh | 6 ++ 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/commit-graph.c b/commit-gra

[PATCH v5 07/21] commit-graph: verify catches corrupt signature

2018-06-06 Thread Derrick Stolee
that is intended for a typical user. Helped-by: Szeder Gábor Signed-off-by: Derrick Stolee --- t/t5318-commit-graph.sh | 43 + 1 file changed, 43 insertions(+) diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh index 0830ef9fdd..c0c1ff09b9

[PATCH v5 09/21] commit-graph: verify corrupt OID fanout and lookup

2018-06-06 Thread Derrick Stolee
In the commit-graph file, the OID fanout chunk provides an index into the OID lookup. The 'verify' subcommand should find incorrect values in the fanout. Similarly, the 'verify' subcommand should find out-of-order values in the OID lookup. Signed-off-by: Derrick Stolee --- commit-graph.c

[PATCH v5 00/21] Integrate commit-graph into 'fsck' and 'gc'

2018-06-06 Thread Derrick Stolee
. Thanks, -Stolee Derrick Stolee (21): commit-graph: UNLEAK before die() commit-graph: fix GRAPH_MIN_SIZE commit-graph: parse commit from chosen graph commit: force commit to parse from object database commit-graph: load a root tree from specific graph commit-graph: add 'verify

[PATCH v5 05/21] commit-graph: load a root tree from specific graph

2018-06-06 Thread Derrick Stolee
When lazy-loading a tree for a commit, it will be important to select the tree from a specific struct commit_graph. Create a new method that specifies the commit-graph file and use that in get_commit_tree_in_graph(). Signed-off-by: Derrick Stolee --- commit-graph.c | 12 +--- 1 file

[PATCH v5 15/21] commit-graph: test for corrupted octopus edge

2018-06-06 Thread Derrick Stolee
The commit-graph file has an extra chunk to store the parent int-ids for parents beyond the first parent for octopus merges. Our test repo has a single octopus merge that we can manipulate to demonstrate the 'verify' subcommand detects incorrect values in that chunk. Signed-off-by: Derrick Stolee

[PATCH v5 21/21] commit-graph: update design document

2018-06-06 Thread Derrick Stolee
The commit-graph feature is now integrated with 'fsck' and 'gc', so remove those items from the "Future Work" section of the commit-graph design document. Also remove the section on lazy-loading trees, as that was completed in an earlier patch series. Signed-off-by: Derr

[PATCH v5 17/21] fsck: verify commit-graph

2018-06-06 Thread Derrick Stolee
-graph file from the other fsck details. 2. The commit-graph verification requires the commits to be loaded in a specific order to guarantee we parse from the commit-graph file for some objects and from the object database for others. Signed-off-by: Derrick Stolee --- Documentation/git

Re: [PATCH v6 00/21] Integrate commit-graph into 'fsck' and 'gc'

2018-06-08 Thread Derrick Stolee
On 6/8/2018 11:05 AM, Jakub Narębski wrote: On Fri, 8 Jun 2018 at 15:56, Derrick Stolee wrote: [..], the following diff occurs from the previous patch: [...] diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh index b24e8b6689..9a0661983c 100755 --- a/t/t5318-commit-graph.sh

Re: is there a canonical doc about how to deal with whitespace issues?

2018-06-08 Thread Derrick Stolee
On 6/8/2018 9:18 AM, Robert P. J. Day wrote: for one of my courses, i wanted to write a section about the various techniques for dealing with whitespace issues in git, so i started making a list, things like: - running "git diff --check" - "git commit --cleanup=" possibilities -

[PATCH 11/23] midx: sort and deduplicate objects from packfiles

2018-06-07 Thread Derrick Stolee
to group the data, copy to a local array, then sort. Copy only the de-duplicated entries. Select the duplicate based on the most-recent modified time of a packfile containing the object. Signed-off-by: Derrick Stolee --- midx.c | 138 + 1 file

[PATCH 18/23] midx: use midx in abbreviation calculations

2018-06-07 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- midx.c | 11 midx.h | 3 +++ packfile.c | 6 + packfile.h | 1 + sha1-name.c | 70 + t/t5319-midx.sh | 3 ++- 6 files changed, 93 insertions(+), 1 deletion(-) diff

[PATCH 22/23] midx: use midx to find ref-deltas

2018-06-07 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- midx.c | 2 +- midx.h | 1 + packfile.c | 15 +++ 3 files changed, 17 insertions(+), 1 deletion(-) diff --git a/midx.c b/midx.c index 3242646fe0..e46f392fa4 100644 --- a/midx.c +++ b/midx.c @@ -214,7 +214,7 @@ struct object_id

[PATCH 21/23] midx: prevent duplicate packfile loads

2018-06-07 Thread Derrick Stolee
If the multi-pack-index contains a packfile, then we do not need to add that packfile to the packed_git linked list or the MRU list. Signed-off-by: Derrick Stolee --- midx.c | 23 +++ midx.h | 1 + packfile.c | 7 +++ 3 files changed, 31 insertions(+) diff

[PATCH 15/23] midx: create core.midx config setting

2018-06-07 Thread Derrick Stolee
. This currently includes 'git rev-list' and 'git log' commands to trigger several object database reads. Signed-off-by: Derrick Stolee --- Documentation/config.txt | 4 +++ cache.h | 1 + config.c | 5 environment.c| 1 + t/t5319-midx.sh | 57

[PATCH 19/23] midx: use existing midx when writing new one

2018-06-07 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- midx.c | 68 +- 1 file changed, 63 insertions(+), 5 deletions(-) diff --git a/midx.c b/midx.c index 25d8142c2a..388d79b7d9 100644 --- a/midx.c +++ b/midx.c @@ -389,6 +389,23 @@ static int midx_oid_compare

[PATCH 14/23] midx: write object offsets

2018-06-07 Thread Derrick Stolee
, but the data does not match the actual packfile offsets. The MIDX offset does match the (corrupted) pack-index offset, so a later commit will compare these offsets during a 'verify' step. Signed-off-by: Derrick Stolee --- Documentation/technical/pack-format.txt | 15 +++- builtin/midx.c

[PATCH 20/23] midx: use midx in approximate_object_count

2018-06-07 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- packfile.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/packfile.c b/packfile.c index 638e113972..059b2aa097 100644 --- a/packfile.c +++ b/packfile.c @@ -819,11 +819,14 @@ unsigned long approximate_object_count(void

[PATCH 23/23] midx: clear midx on repack

2018-06-07 Thread Derrick Stolee
If a 'git repack' command replaces existing packfiles, then we must clear the existing multi-pack-index before moving the packfiles it references. Signed-off-by: Derrick Stolee --- builtin/repack.c | 8 midx.c | 8 midx.h | 1 + 3 files changed, 17

[PATCH 13/23] midx: write object id fanout chunk

2018-06-07 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- Documentation/technical/pack-format.txt | 5 +++ builtin/midx.c | 4 +- midx.c | 53 +++-- object-store.h | 1 + t/t5319-midx.sh

[PATCH 05/23] midx: write header information to lockfile

2018-06-07 Thread Derrick Stolee
the return value in a later commit. Signed-off-by: Derrick Stolee --- midx.c | 53 + t/t5319-midx.sh | 5 +++-- 2 files changed, 56 insertions(+), 2 deletions(-) diff --git a/midx.c b/midx.c index 616af66b13..3e55422a21 100644 --- a/midx.c +++ b

[PATCH 07/23] midx: expand test data

2018-06-07 Thread Derrick Stolee
with no "real" data. This will be expanded in future commits, along with the test expectations. Signed-off-by: Derrick Stolee --- t/t5319-midx.sh | 101 1 file changed, 101 insertions(+) diff --git a/t/t5319-midx.sh b/t/t5319-midx.sh index

[PATCH 06/23] midx: struct midxed_git and 'read' subcommand

2018-06-07 Thread Derrick Stolee
(object_dir). Create the 'git midx read' subcommand to output basic information about the multi-pack-index file. This will be expanded as more information is written to the file. Signed-off-by: Derrick Stolee --- Documentation/git-midx.txt | 11 +++ builtin/midx.c | 23

[PATCH 08/23] midx: read packfiles from pack directory

2018-06-07 Thread Derrick Stolee
When constructing a multi-pack-index file for a given object directory, read the files within the enclosed pack directory and find matches that end with ".idx" and find the correct paired packfile using add_packed_git(). Signed-off-by: Derrick Stolee --- midx.c

[PATCH 17/23] midx: read objects from multi-pack-index

2018-06-07 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- midx.c | 96 -- midx.h | 2 ++ object-store.h | 1 + packfile.c | 8 - 4 files changed, 104 insertions(+), 3 deletions(-) diff --git a/midx.c b/midx.c index 5e9290ca8f..6eca8f1b12 100644

[PATCH 02/23] midx: add midx format details to pack-format.txt

2018-06-07 Thread Derrick Stolee
e the code that writes that information. Signed-off-by: Derrick Stolee --- Documentation/technical/pack-format.txt | 49 + 1 file changed, 49 insertions(+) diff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt index 70a99fd142..17666b4

[PATCH 10/23] midx: write a lookup into the pack names chunk

2018-06-07 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- Documentation/technical/pack-format.txt | 5 +++ builtin/midx.c | 7 midx.c | 56 +++-- object-store.h | 2 + t/t5319-midx.sh

[PATCH 04/23] midx: add 'write' subcommand and basic wiring

2018-06-07 Thread Derrick Stolee
In anticipation of writing multi-pack-indexes (MIDX files), add a 'git midx write' subcommand and send the options to a write_midx_file() method. Also create a basic test file that tests the 'write' subcommand. Signed-off-by: Derrick Stolee --- Documentation/git-midx.txt | 22

[PATCH 12/23] midx: write object ids in a chunk

2018-06-07 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- Documentation/technical/pack-format.txt | 4 ++ builtin/midx.c | 2 + midx.c | 50 +++-- object-store.h | 1 + t/t5319-midx.sh | 4

[PATCH 00/23] Multi-pack-index (MIDX)

2018-06-07 Thread Derrick Stolee
area, too. Thanks, -Stolee [1] https://public-inbox.org/git/20180107181459.222909-1-dsto...@microsoft.com/T/#u Previous MIDX RFC. [2] https://public-inbox.org/git/20171012120220.226427-1-dsto...@microsoft.com/ A patch series on abbreviation speedups Derrick Stolee (23): midx: a

[PATCH 03/23] midx: add midx builtin

2018-06-07 Thread Derrick Stolee
This new 'git midx' builtin will be the plumbing access for writing, reading, and checking multi-pack-index (MIDX) files. The initial implementation is a no-op. Signed-off-by: Derrick Stolee --- .gitignore | 1 + Documentation/git-midx.txt | 29

[PATCH 09/23] midx: write pack names in chunk

2018-06-07 Thread Derrick Stolee
when dealing with very large data sets. These large sets are hard to encode into test cases. Signed-off-by: Derrick Stolee --- Documentation/technical/pack-format.txt | 6 + builtin/midx.c | 7 + midx.c | 176

[PATCH 16/23] midx: prepare midxed_git struct

2018-06-07 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- midx.c | 22 ++ midx.h | 2 ++ object-store.h | 7 +++ packfile.c | 6 +- 4 files changed, 36 insertions(+), 1 deletion(-) diff --git a/midx.c b/midx.c index a49300bf75..5e9290ca8f 100644 --- a/midx.c +++ b

[PATCH 01/23] midx: add design document

2018-06-07 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- Documentation/technical/midx.txt | 109 +++ 1 file changed, 109 insertions(+) create mode 100644 Documentation/technical/midx.txt diff --git a/Documentation/technical/midx.txt b/Documentation/technical/midx.txt new file mode 100644

Re: [PATCH 00/23] Multi-pack-index (MIDX)

2018-06-07 Thread Derrick Stolee
On 6/7/2018 10:03 AM, Derrick Stolee wrote: This patch series includes a rewrite of the previous multi-pack-index RFC [1] using the feedback from the commit-graph feature. Sorry to everyone who got a duplicate copy of this series. I misspelled 'kernel.org' and it didn't go to the list. I

[PATCH v6 04/21] commit: force commit to parse from object database

2018-06-08 Thread Derrick Stolee
is explicit in avoiding commits from the commit-graph file. Signed-off-by: Derrick Stolee --- commit.c | 9 +++-- commit.h | 1 + 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/commit.c b/commit.c index d53dc16d72..720c6acddf 100644 --- a/commit.c +++ b/commit.c @@ -418,7

[PATCH v6 01/21] commit-graph: UNLEAK before die()

2018-06-08 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- builtin/commit-graph.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c index 37420ae0fd..f0875b8bf3 100644 --- a/builtin/commit-graph.c +++ b/builtin/commit-graph.c @@ -51,8 +51,11 @@ static

[PATCH v6 11/21] commit-graph: verify root tree OIDs

2018-06-08 Thread Derrick Stolee
the object database. Add checks for the root tree OID. Signed-off-by: Derrick Stolee --- commit-graph.c | 17 - t/t5318-commit-graph.sh | 7 +++ 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/commit-graph.c b/commit-graph.c index 00e89b71e9

[PATCH v6 14/21] commit-graph: verify commit date

2018-06-08 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- commit-graph.c | 6 ++ t/t5318-commit-graph.sh | 6 ++ 2 files changed, 12 insertions(+) diff --git a/commit-graph.c b/commit-graph.c index e0f71658da..6d6c6beff9 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -986,6 +986,12 @@ int

[PATCH v6 07/21] commit-graph: verify catches corrupt signature

2018-06-08 Thread Derrick Stolee
that is intended for a typical user. Helped-by: Szeder Gábor Signed-off-by: Derrick Stolee --- t/t5318-commit-graph.sh | 43 + 1 file changed, 43 insertions(+) diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh index 6ca451dfd2..8f96e2636c

[PATCH v6 08/21] commit-graph: verify required chunks are present

2018-06-08 Thread Derrick Stolee
The commit-graph file requires the following three chunks: * OID Fanout * OID Lookup * Commit Data If any of these are missing, then the 'verify' subcommand should report a failure. This includes the chunk IDs malformed or the chunk count is truncated. Signed-off-by: Derrick Stolee --- commit

[PATCH v6 13/21] commit-graph: verify generation number

2018-06-08 Thread Derrick Stolee
have a mix of zero and non-zero generation numbers. Create a test that sets one commit to generation zero and all following commits report a failure as they have non-zero generation in a file that contains generation number zero. Signed-off-by: Derrick Stolee --- commit-graph.c | 34

[PATCH v6 17/21] fsck: verify commit-graph

2018-06-08 Thread Derrick Stolee
-graph file from the other fsck details. 2. The commit-graph verification requires the commits to be loaded in a specific order to guarantee we parse from the commit-graph file for some objects and from the object database for others. Signed-off-by: Derrick Stolee --- Documentation/git

[PATCH v6 12/21] commit-graph: verify parent list

2018-06-08 Thread Derrick Stolee
from the object database. Test these checks for corrupt parents, too many parents, and wrong parents. Add a boundary check to insert_parent_or_die() for when the parent position value is out of range. The octopus merge will be tested in a later commit. Signed-off-by: Derrick Stolee --- commit

[PATCH v6 09/21] commit-graph: verify corrupt OID fanout and lookup

2018-06-08 Thread Derrick Stolee
In the commit-graph file, the OID fanout chunk provides an index into the OID lookup. The 'verify' subcommand should find incorrect values in the fanout. Similarly, the 'verify' subcommand should find out-of-order values in the OID lookup. Signed-off-by: Derrick Stolee --- commit-graph.c

[PATCH v6 16/21] commit-graph: verify contents match checksum

2018-06-08 Thread Derrick Stolee
our tests, as we break the checksum as we modify bytes of the commit-graph file. Signed-off-by: Derrick Stolee --- commit-graph.c | 16 ++-- t/t5318-commit-graph.sh | 6 ++ 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/commit-graph.c b/commit-gra

[PATCH v6 21/21] commit-graph: update design document

2018-06-08 Thread Derrick Stolee
The commit-graph feature is now integrated with 'fsck' and 'gc', so remove those items from the "Future Work" section of the commit-graph design document. Also remove the section on lazy-loading trees, as that was completed in an earlier patch series. Signed-off-by: Derr

[PATCH v6 10/21] commit-graph: verify objects exist

2018-06-08 Thread Derrick Stolee
In the 'verify' subcommand, load commits directly from the object database to ensure they exist. Parse by skipping the commit-graph. Signed-off-by: Derrick Stolee --- commit-graph.c | 18 ++ t/t5318-commit-graph.sh | 7 +++ 2 files changed, 25 insertions(+) diff

<    1   2   3   4   5   6   7   8   9   10   >