[PATCH v6 13/21] commit-graph: verify generation number

have a mix of zero and non-zero generation numbers. Create a test that sets one commit to generation zero and all following commits report a failure as they have non-zero generation in a file that contains generation number zero. Signed-off-by: Derrick Stolee --- commit-graph.c | 34

[PATCH v6 04/21] commit: force commit to parse from object database

is explicit in avoiding commits from the commit-graph file. Signed-off-by: Derrick Stolee --- commit.c | 9 +++-- commit.h | 1 + 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/commit.c b/commit.c index d53dc16d72..720c6acddf 100644 --- a/commit.c +++ b/commit.c @@ -418,7

[PATCH v6 01/21] commit-graph: UNLEAK before die()

Signed-off-by: Derrick Stolee --- builtin/commit-graph.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c index 37420ae0fd..f0875b8bf3 100644 --- a/builtin/commit-graph.c +++ b/builtin/commit-graph.c @@ -51,8 +51,11 @@ static

[PATCH v6 11/21] commit-graph: verify root tree OIDs

the object database. Add checks for the root tree OID. Signed-off-by: Derrick Stolee --- commit-graph.c | 17 - t/t5318-commit-graph.sh | 7 +++ 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/commit-graph.c b/commit-graph.c index 00e89b71e9

[PATCH v6 14/21] commit-graph: verify commit date

Signed-off-by: Derrick Stolee --- commit-graph.c | 6 ++ t/t5318-commit-graph.sh | 6 ++ 2 files changed, 12 insertions(+) diff --git a/commit-graph.c b/commit-graph.c index e0f71658da..6d6c6beff9 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -986,6 +986,12 @@ int

[PATCH v6 07/21] commit-graph: verify catches corrupt signature

that is intended for a typical user. Helped-by: Szeder Gábor Signed-off-by: Derrick Stolee --- t/t5318-commit-graph.sh | 43 + 1 file changed, 43 insertions(+) diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh index 6ca451dfd2..8f96e2636c

[PATCH v6 08/21] commit-graph: verify required chunks are present

The commit-graph file requires the following three chunks: * OID Fanout * OID Lookup * Commit Data If any of these are missing, then the 'verify' subcommand should report a failure. This includes the chunk IDs malformed or the chunk count is truncated. Signed-off-by: Derrick Stolee --- commit

[PATCH v6 19/21] commit-graph: add '--reachable' option

When writing commit-graph files, it can be convenient to ask for all reachable commits (starting at the ref set) in the resulting file. This is particularly helpful when writing to stdin is complicated, such as a future integration with 'git gc'. Signed-off-by: Derrick Stolee --- Documentation

[PATCH v6 20/21] gc: automatically write commit-graph files

-by: Derrick Stolee --- Documentation/config.txt | 10 +- Documentation/git-gc.txt | 4 builtin/gc.c | 6 ++ t/t5318-commit-graph.sh | 14 ++ 4 files changed, 33 insertions(+), 1 deletion(-) diff --git a/Documentation/config.txt b/Documentation/config.txt

[PATCH v6 15/21] commit-graph: test for corrupted octopus edge

The commit-graph file has an extra chunk to store the parent int-ids for parents beyond the first parent for octopus merges. Our test repo has a single octopus merge that we can manipulate to demonstrate the 'verify' subcommand detects incorrect values in that chunk. Signed-off-by: Derrick Stolee

[PATCH v6 16/21] commit-graph: verify contents match checksum

our tests, as we break the checksum as we modify bytes of the commit-graph file. Signed-off-by: Derrick Stolee --- commit-graph.c | 16 ++-- t/t5318-commit-graph.sh | 6 ++ 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/commit-graph.c b/commit-gra

[PATCH v6 21/21] commit-graph: update design document

The commit-graph feature is now integrated with 'fsck' and 'gc', so remove those items from the "Future Work" section of the commit-graph design document. Also remove the section on lazy-loading trees, as that was completed in an earlier patch series. Signed-off-by: Derr

[PATCH v6 18/21] commit-graph: use string-list API for input

Signed-off-by: Derrick Stolee --- builtin/commit-graph.c | 39 +-- commit-graph.c | 15 +++ commit-graph.h | 7 +++ 3 files changed, 23 insertions(+), 38 deletions(-) diff --git a/builtin/commit-graph.c b/builtin/commit

[PATCH v6 06/21] commit-graph: add 'verify' subcommand

state. Do not report any errors. Helped-by: Ramsay Jones Signed-off-by: Derrick Stolee --- Documentation/git-commit-graph.txt | 6 + builtin/commit-graph.c | 39 ++ commit-graph.c | 23 ++ commit-graph.h

[PATCH v6 03/21] commit-graph: parse commit from chosen graph

Before verifying a commit-graph file against the object database, we need to parse all commits from the given commit-graph file. Create parse_commit_in_graph_one() to target a given struct commit_graph. Signed-off-by: Derrick Stolee --- commit-graph.c | 18 +++--- 1 file changed, 15

[PATCH v6 05/21] commit-graph: load a root tree from specific graph

When lazy-loading a tree for a commit, it will be important to select the tree from a specific struct commit_graph. Create a new method that specifies the commit-graph file and use that in get_commit_tree_in_graph(). Signed-off-by: Derrick Stolee --- commit-graph.c | 12 +--- 1 file

[PATCH v6 17/21] fsck: verify commit-graph

-graph file from the other fsck details. 2. The commit-graph verification requires the commits to be loaded in a specific order to guarantee we parse from the commit-graph file for some objects and from the object database for others. Signed-off-by: Derrick Stolee --- Documentation/git

[PATCH v6 12/21] commit-graph: verify parent list

from the object database. Test these checks for corrupt parents, too many parents, and wrong parents. Add a boundary check to insert_parent_or_die() for when the parent position value is out of range. The octopus merge will be tested in a later commit. Signed-off-by: Derrick Stolee --- commit

[PATCH v6 09/21] commit-graph: verify corrupt OID fanout and lookup

In the commit-graph file, the OID fanout chunk provides an index into the OID lookup. The 'verify' subcommand should find incorrect values in the fanout. Similarly, the 'verify' subcommand should find out-of-order values in the OID lookup. Signed-off-by: Derrick Stolee --- commit-graph.c

[PATCH v6 10/21] commit-graph: verify objects exist

In the 'verify' subcommand, load commits directly from the object database to ensure they exist. Parse by skipping the commit-graph. Signed-off-by: Derrick Stolee --- commit-graph.c | 18 ++ t/t5318-commit-graph.sh | 7 +++ 2 files changed, 25 insertions(+) diff

[PATCH v6 02/21] commit-graph: fix GRAPH_MIN_SIZE

the magic constants. Signed-off-by: Derrick Stolee --- commit-graph.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/commit-graph.c b/commit-graph.c index b63a1fc85e..f83f6d2373 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -35,10 +35,11 @@ #define GRAPH_LAST_EDGE

[PATCH v6 00/21] Integrate commit-graph into 'fsck' and 'gc'

_OCTOPUS_EDGES)) +GRAPH_BYTE_FOOTER=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4 * $NUM_OCTOPUS_EDGES)) # usage: corrupt_graph_and_verify # Manipulates the commit-graph file at the position --- Derrick Stolee (21): commit-graph: UNLEAK before die() commit-graph: fix GRAPH_MIN_SIZE commit-gra

Re: [PATCH v5 18/21] commit-graph: use string-list API for input

On 6/6/2018 8:45 AM, Derrick Stolee wrote: On 6/6/2018 8:26 AM, Ævar Arnfjörð Bjarmason wrote: On Wed, Jun 06 2018, Derrick Stolee wrote: On 6/6/2018 8:11 AM, Ævar Arnfjörð Bjarmason wrote: On Wed, Jun 06 2018, Derrick Stolee wrote: Signed-off-by: Derrick Stolee +    string_list_clear

Re: [PATCH 00/23] Multi-pack-index (MIDX)

On 6/7/2018 10:45 AM, Ævar Arnfjörð Bjarmason wrote: On Thu, Jun 07 2018, Derrick Stolee wrote: To test the performance in this situation, I created a script that organizes the Linux repository in a similar fashion. I split the commit history into 50 parts by creating branches on every 10,000

Re: [PATCH 00/23] Multi-pack-index (MIDX)

On 6/7/2018 10:03 AM, Derrick Stolee wrote: This patch series includes a rewrite of the previous multi-pack-index RFC [1] using the feedback from the commit-graph feature. Sorry to everyone who got a duplicate copy of this series. I misspelled 'kernel.org' and it didn't go to the list. I

[PATCH 07/23] midx: expand test data

with no "real" data. This will be expanded in future commits, along with the test expectations. Signed-off-by: Derrick Stolee --- t/t5319-midx.sh | 101 1 file changed, 101 insertions(+) diff --git a/t/t5319-midx.sh b/t/t5319-midx.sh index

[PATCH 06/23] midx: struct midxed_git and 'read' subcommand

(object_dir). Create the 'git midx read' subcommand to output basic information about the multi-pack-index file. This will be expanded as more information is written to the file. Signed-off-by: Derrick Stolee --- Documentation/git-midx.txt | 11 +++ builtin/midx.c | 23

[PATCH 08/23] midx: read packfiles from pack directory

When constructing a multi-pack-index file for a given object directory, read the files within the enclosed pack directory and find matches that end with ".idx" and find the correct paired packfile using add_packed_git(). Signed-off-by: Derrick Stolee --- midx.c

[PATCH 17/23] midx: read objects from multi-pack-index

Signed-off-by: Derrick Stolee --- midx.c | 96 -- midx.h | 2 ++ object-store.h | 1 + packfile.c | 8 - 4 files changed, 104 insertions(+), 3 deletions(-) diff --git a/midx.c b/midx.c index 5e9290ca8f..6eca8f1b12 100644

[PATCH 02/23] midx: add midx format details to pack-format.txt

e the code that writes that information. Signed-off-by: Derrick Stolee --- Documentation/technical/pack-format.txt | 49 + 1 file changed, 49 insertions(+) diff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt index 70a99fd142..17666b4

[PATCH 05/23] midx: write header information to lockfile

the return value in a later commit. Signed-off-by: Derrick Stolee --- midx.c | 53 + t/t5319-midx.sh | 5 +++-- 2 files changed, 56 insertions(+), 2 deletions(-) diff --git a/midx.c b/midx.c index 616af66b13..3e55422a21 100644 --- a/midx.c +++ b

[PATCH 21/23] midx: prevent duplicate packfile loads

If the multi-pack-index contains a packfile, then we do not need to add that packfile to the packed_git linked list or the MRU list. Signed-off-by: Derrick Stolee --- midx.c | 23 +++ midx.h | 1 + packfile.c | 7 +++ 3 files changed, 31 insertions(+) diff

[PATCH 15/23] midx: create core.midx config setting

. This currently includes 'git rev-list' and 'git log' commands to trigger several object database reads. Signed-off-by: Derrick Stolee --- Documentation/config.txt | 4 +++ cache.h | 1 + config.c | 5 environment.c| 1 + t/t5319-midx.sh | 57

[PATCH 19/23] midx: use existing midx when writing new one

Signed-off-by: Derrick Stolee --- midx.c | 68 +- 1 file changed, 63 insertions(+), 5 deletions(-) diff --git a/midx.c b/midx.c index 25d8142c2a..388d79b7d9 100644 --- a/midx.c +++ b/midx.c @@ -389,6 +389,23 @@ static int midx_oid_compare

[PATCH 14/23] midx: write object offsets

, but the data does not match the actual packfile offsets. The MIDX offset does match the (corrupted) pack-index offset, so a later commit will compare these offsets during a 'verify' step. Signed-off-by: Derrick Stolee --- Documentation/technical/pack-format.txt | 15 +++- builtin/midx.c

[PATCH 20/23] midx: use midx in approximate_object_count

Signed-off-by: Derrick Stolee --- packfile.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/packfile.c b/packfile.c index 638e113972..059b2aa097 100644 --- a/packfile.c +++ b/packfile.c @@ -819,11 +819,14 @@ unsigned long approximate_object_count(void

[PATCH 23/23] midx: clear midx on repack

If a 'git repack' command replaces existing packfiles, then we must clear the existing multi-pack-index before moving the packfiles it references. Signed-off-by: Derrick Stolee --- builtin/repack.c | 8 midx.c | 8 midx.h | 1 + 3 files changed, 17

[PATCH 13/23] midx: write object id fanout chunk

Signed-off-by: Derrick Stolee --- Documentation/technical/pack-format.txt | 5 +++ builtin/midx.c | 4 +- midx.c | 53 +++-- object-store.h | 1 + t/t5319-midx.sh

[PATCH 18/23] midx: use midx in abbreviation calculations

Signed-off-by: Derrick Stolee --- midx.c | 11 midx.h | 3 +++ packfile.c | 6 + packfile.h | 1 + sha1-name.c | 70 + t/t5319-midx.sh | 3 ++- 6 files changed, 93 insertions(+), 1 deletion(-) diff

[PATCH 22/23] midx: use midx to find ref-deltas

Signed-off-by: Derrick Stolee --- midx.c | 2 +- midx.h | 1 + packfile.c | 15 +++ 3 files changed, 17 insertions(+), 1 deletion(-) diff --git a/midx.c b/midx.c index 3242646fe0..e46f392fa4 100644 --- a/midx.c +++ b/midx.c @@ -214,7 +214,7 @@ struct object_id

[PATCH 11/23] midx: sort and deduplicate objects from packfiles

to group the data, copy to a local array, then sort. Copy only the de-duplicated entries. Select the duplicate based on the most-recent modified time of a packfile containing the object. Signed-off-by: Derrick Stolee --- midx.c | 138 + 1 file

[PATCH 10/23] midx: write a lookup into the pack names chunk

Signed-off-by: Derrick Stolee --- Documentation/technical/pack-format.txt | 5 +++ builtin/midx.c | 7 midx.c | 56 +++-- object-store.h | 2 + t/t5319-midx.sh

[PATCH 04/23] midx: add 'write' subcommand and basic wiring

In anticipation of writing multi-pack-indexes (MIDX files), add a 'git midx write' subcommand and send the options to a write_midx_file() method. Also create a basic test file that tests the 'write' subcommand. Signed-off-by: Derrick Stolee --- Documentation/git-midx.txt | 22

[PATCH 12/23] midx: write object ids in a chunk

Signed-off-by: Derrick Stolee --- Documentation/technical/pack-format.txt | 4 ++ builtin/midx.c | 2 + midx.c | 50 +++-- object-store.h | 1 + t/t5319-midx.sh | 4

[PATCH 00/23] Multi-pack-index (MIDX)

area, too. Thanks, -Stolee [1] https://public-inbox.org/git/20180107181459.222909-1-dsto...@microsoft.com/T/#u Previous MIDX RFC. [2] https://public-inbox.org/git/20171012120220.226427-1-dsto...@microsoft.com/ A patch series on abbreviation speedups Derrick Stolee (23): midx: a

[PATCH 03/23] midx: add midx builtin

This new 'git midx' builtin will be the plumbing access for writing, reading, and checking multi-pack-index (MIDX) files. The initial implementation is a no-op. Signed-off-by: Derrick Stolee --- .gitignore | 1 + Documentation/git-midx.txt | 29

[PATCH 01/23] midx: add design document

Signed-off-by: Derrick Stolee --- Documentation/technical/midx.txt | 109 +++ 1 file changed, 109 insertions(+) create mode 100644 Documentation/technical/midx.txt diff --git a/Documentation/technical/midx.txt b/Documentation/technical/midx.txt new file mode 100644

[PATCH 09/23] midx: write pack names in chunk

when dealing with very large data sets. These large sets are hard to encode into test cases. Signed-off-by: Derrick Stolee --- Documentation/technical/pack-format.txt | 6 + builtin/midx.c | 7 + midx.c | 176

[PATCH 16/23] midx: prepare midxed_git struct

Signed-off-by: Derrick Stolee --- midx.c | 22 ++ midx.h | 2 ++ object-store.h | 7 +++ packfile.c | 6 +- 4 files changed, 36 insertions(+), 1 deletion(-) diff --git a/midx.c b/midx.c index a49300bf75..5e9290ca8f 100644 --- a/midx.c +++ b

Re: [PATCH v5 18/21] commit-graph: use string-list API for input

On 6/6/2018 8:26 AM, Ævar Arnfjörð Bjarmason wrote: On Wed, Jun 06 2018, Derrick Stolee wrote: On 6/6/2018 8:11 AM, Ævar Arnfjörð Bjarmason wrote: On Wed, Jun 06 2018, Derrick Stolee wrote: Signed-off-by: Derrick Stolee --- builtin/commit-graph.c | 39

Re: [PATCH v5 18/21] commit-graph: use string-list API for input

On 6/6/2018 8:11 AM, Ævar Arnfjörð Bjarmason wrote: On Wed, Jun 06 2018, Derrick Stolee wrote: Signed-off-by: Derrick Stolee --- builtin/commit-graph.c | 39 +-- commit-graph.c | 15 +++ commit-graph.h | 7 +++ 3 files

[PATCH v5 12/21] commit-graph: verify parent list

from the object database. Test these checks for corrupt parents, too many parents, and wrong parents. Add a boundary check to insert_parent_or_die() for when the parent position value is out of range. The octopus merge will be tested in a later commit. Signed-off-by: Derrick Stolee --- commit

[PATCH v5 08/21] commit-graph: verify required chunks are present

The commit-graph file requires the following three chunks: * OID Fanout * OID Lookup * Commit Data If any of these are missing, then the 'verify' subcommand should report a failure. This includes the chunk IDs malformed or the chunk count is truncated. Signed-off-by: Derrick Stolee --- commit

[PATCH v5 11/21] commit-graph: verify root tree OIDs

the object database. Add checks for the root tree OID. Signed-off-by: Derrick Stolee --- commit-graph.c | 17 - t/t5318-commit-graph.sh | 7 +++ 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/commit-graph.c b/commit-graph.c index 893cc2f346

[PATCH v5 02/21] commit-graph: fix GRAPH_MIN_SIZE

the magic constants. Signed-off-by: Derrick Stolee --- commit-graph.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/commit-graph.c b/commit-graph.c index b63a1fc85e..f83f6d2373 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -35,10 +35,11 @@ #define GRAPH_LAST_EDGE

[PATCH v5 01/21] commit-graph: UNLEAK before die()

Signed-off-by: Derrick Stolee --- builtin/commit-graph.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c index 37420ae0fd..f0875b8bf3 100644 --- a/builtin/commit-graph.c +++ b/builtin/commit-graph.c @@ -51,8 +51,11 @@ static

[PATCH v5 06/21] commit-graph: add 'verify' subcommand

state. Do not report any errors. Helped-by: Ramsay Jones Signed-off-by: Derrick Stolee --- Documentation/git-commit-graph.txt | 6 + builtin/commit-graph.c | 38 ++ commit-graph.c | 23 ++ commit-graph.h

[PATCH v5 04/21] commit: force commit to parse from object database

is explicit in avoiding commits from the commit-graph file. Signed-off-by: Derrick Stolee --- commit.c | 9 +++-- commit.h | 1 + 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/commit.c b/commit.c index 298ad747c6..922bb68741 100644 --- a/commit.c +++ b/commit.c @@ -405,7

[PATCH v5 14/21] commit-graph: verify commit date

Signed-off-by: Derrick Stolee --- commit-graph.c | 6 ++ t/t5318-commit-graph.sh | 6 ++ 2 files changed, 12 insertions(+) diff --git a/commit-graph.c b/commit-graph.c index b97fa05ec9..d83f0ce5d5 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -985,6 +985,12 @@ int

[PATCH v5 16/21] commit-graph: verify contents match checksum

our tests, as we break the checksum as we modify bytes of the commit-graph file. Signed-off-by: Derrick Stolee --- commit-graph.c | 16 ++-- t/t5318-commit-graph.sh | 6 ++ 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/commit-graph.c b/commit-gra

[PATCH v5 07/21] commit-graph: verify catches corrupt signature

that is intended for a typical user. Helped-by: Szeder Gábor Signed-off-by: Derrick Stolee --- t/t5318-commit-graph.sh | 43 + 1 file changed, 43 insertions(+) diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh index 0830ef9fdd..c0c1ff09b9

[PATCH v5 09/21] commit-graph: verify corrupt OID fanout and lookup

In the commit-graph file, the OID fanout chunk provides an index into the OID lookup. The 'verify' subcommand should find incorrect values in the fanout. Similarly, the 'verify' subcommand should find out-of-order values in the OID lookup. Signed-off-by: Derrick Stolee --- commit-graph.c

[PATCH v5 10/21] commit-graph: verify objects exist

In the 'verify' subcommand, load commits directly from the object database to ensure they exist. Parse by skipping the commit-graph. Signed-off-by: Derrick Stolee --- commit-graph.c | 17 + t/t5318-commit-graph.sh | 7 +++ 2 files changed, 24 insertions(+) diff

[PATCH v5 19/21] commit-graph: add '--reachable' option

When writing commit-graph files, it can be convenient to ask for all reachable commits (starting at the ref set) in the resulting file. This is particularly helpful when writing to stdin is complicated, such as a future integration with 'git gc'. Signed-off-by: Derrick Stolee --- Documentation

[PATCH v5 18/21] commit-graph: use string-list API for input

Signed-off-by: Derrick Stolee --- builtin/commit-graph.c | 39 +-- commit-graph.c | 15 +++ commit-graph.h | 7 +++ 3 files changed, 23 insertions(+), 38 deletions(-) diff --git a/builtin/commit-graph.c b/builtin/commit

[PATCH v5 20/21] gc: automatically write commit-graph files

-by: Derrick Stolee --- Documentation/config.txt | 10 +- Documentation/git-gc.txt | 4 builtin/gc.c | 6 ++ t/t5318-commit-graph.sh | 14 ++ 4 files changed, 33 insertions(+), 1 deletion(-) diff --git a/Documentation/config.txt b/Documentation/config.txt

[PATCH v5 13/21] commit-graph: verify generation number

have a mix of zero and non-zero generation numbers. Create a test that sets one commit to generation zero and all following commits report a failure as they have non-zero generation in a file that contains generation number zero. Signed-off-by: Derrick Stolee --- commit-graph.c | 34

[PATCH v5 03/21] commit-graph: parse commit from chosen graph

Before verifying a commit-graph file against the object database, we need to parse all commits from the given commit-graph file. Create parse_commit_in_graph_one() to target a given struct commit_graph. Signed-off-by: Derrick Stolee --- commit-graph.c | 18 +++--- 1 file changed, 15

[PATCH v5 21/21] commit-graph: update design document

The commit-graph feature is now integrated with 'fsck' and 'gc', so remove those items from the "Future Work" section of the commit-graph design document. Also remove the section on lazy-loading trees, as that was completed in an earlier patch series. Signed-off-by: Derr

[PATCH v5 15/21] commit-graph: test for corrupted octopus edge

The commit-graph file has an extra chunk to store the parent int-ids for parents beyond the first parent for octopus merges. Our test repo has a single octopus merge that we can manipulate to demonstrate the 'verify' subcommand detects incorrect values in that chunk. Signed-off-by: Derrick Stolee

[PATCH v5 00/21] Integrate commit-graph into 'fsck' and 'gc'

. Thanks, -Stolee Derrick Stolee (21): commit-graph: UNLEAK before die() commit-graph: fix GRAPH_MIN_SIZE commit-graph: parse commit from chosen graph commit: force commit to parse from object database commit-graph: load a root tree from specific graph commit-graph: add 'verify

[PATCH v5 05/21] commit-graph: load a root tree from specific graph

When lazy-loading a tree for a commit, it will be important to select the tree from a specific struct commit_graph. Create a new method that specifies the commit-graph file and use that in get_commit_tree_in_graph(). Signed-off-by: Derrick Stolee --- commit-graph.c | 12 +--- 1 file

[PATCH v5 17/21] fsck: verify commit-graph

-graph file from the other fsck details. 2. The commit-graph verification requires the commits to be loaded in a specific order to guarantee we parse from the commit-graph file for some objects and from the object database for others. Signed-off-by: Derrick Stolee --- Documentation/git

Re: [PATCH v4 17/21] fsck: verify commit-graph

On 6/6/2018 7:08 AM, Ævar Arnfjörð Bjarmason wrote: On Mon, Jun 04 2018, Derrick Stolee wrote: + prepare_alt_odb(); + for (alt = alt_odb_list; alt; alt = alt->next) { + verify_argv[2] = "--object-dir"; +

Re: [RFC PATCH 00/18] Multi-pack index (MIDX)

On 6/6/2018 4:13 AM, Ævar Arnfjörð Bjarmason wrote: On Mon, Jan 08 2018, Derrick Stolee wrote: On 1/7/2018 5:42 PM, Ævar Arnfjörð Bjarmason wrote: On Sun, Jan 07 2018, Derrick Stolee jotted: git log --oneline --raw --parents Num Packs | Before MIDX | After MIDX | Rel % | 1 pack

Re: [PATCH v4 00/21] Integrate commit-graph into 'fsck' and 'gc'

On 6/5/2018 10:51 AM, Ævar Arnfjörð Bjarmason wrote: On Mon, Jun 4, 2018 at 6:52 PM, Derrick Stolee wrote: Thanks for the feedback on v3. There were several small cleanups, but perhaps the biggest change is the addition of "commit-graph: use string-list API for input" which makes &qu

Re: [PATCH v4 00/21] Integrate commit-graph into 'fsck' and 'gc'

Sorry I forgot to --in-reply-to the previous version [1] [1] https://public-inbox.org/git/20180524162504.158394-1-dsto...@microsoft.com/T/#u On 6/4/2018 12:52 PM, Derrick Stolee wrote: Thanks for the feedback on v3. There were several small cleanups, but perhaps the biggest change

[PATCH v4 04/21] commit: force commit to parse from object database

is explicit in avoiding commits from the commit-graph file. Signed-off-by: Derrick Stolee --- commit.c | 9 +++-- commit.h | 1 + 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/commit.c b/commit.c index 1d28677dfb..6eaed0174c 100644 --- a/commit.c +++ b/commit.c @@ -392,7

[PATCH v4 15/21] commit-graph: test for corrupted octopus edge

The commit-graph file has an extra chunk to store the parent int-ids for parents beyond the first parent for octopus merges. Our test repo has a single octopus merge that we can manipulate to demonstrate the 'verify' subcommand detects incorrect values in that chunk. Signed-off-by: Derrick Stolee

[PATCH v4 14/21] commit-graph: verify commit date

Signed-off-by: Derrick Stolee --- commit-graph.c | 6 ++ t/t5318-commit-graph.sh | 6 ++ 2 files changed, 12 insertions(+) diff --git a/commit-graph.c b/commit-graph.c index 5faecae2a7..47fdd62e88 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -982,6 +982,12 @@ int

[PATCH v4 16/21] commit-graph: verify contents match checksum

our tests, as we break the checksum as we modify bytes of the commit-graph file. Signed-off-by: Derrick Stolee --- commit-graph.c | 16 ++-- t/t5318-commit-graph.sh | 6 ++ 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/commit-graph.c b/commit-gra

[PATCH v4 17/21] fsck: verify commit-graph

-graph file from the other fsck details. 2. The commit-graph verification requires the commits to be loaded in a specific order to guarantee we parse from the commit-graph file for some objects and from the object database for others. Signed-off-by: Derrick Stolee --- Documentation/git

[PATCH v4 21/21] commit-graph: update design document

The commit-graph feature is now integrated with 'fsck' and 'gc', so remove those items from the "Future Work" section of the commit-graph design document. Also remove the section on lazy-loading trees, as that was completed in an earlier patch series. Signed-off-by: Derr

[PATCH v4 18/21] commit-graph: use string-list API for input

Signed-off-by: Derrick Stolee --- builtin/commit-graph.c | 39 +-- commit-graph.c | 15 +++ commit-graph.h | 7 +++ 3 files changed, 23 insertions(+), 38 deletions(-) diff --git a/builtin/commit-graph.c b/builtin/commit

[PATCH v4 13/21] commit-graph: verify generation number

have a mix of zero and non-zero generation numbers. Create a test that sets one commit to generation zero and all following commits report a failure as they have non-zero generation in a file that contains generation number zero. Signed-off-by: Derrick Stolee --- commit-graph.c | 34

[PATCH v4 20/21] gc: automatically write commit-graph files

-by: Derrick Stolee --- Documentation/config.txt | 10 +- Documentation/git-gc.txt | 4 builtin/gc.c | 6 ++ t/t5318-commit-graph.sh | 14 ++ 4 files changed, 33 insertions(+), 1 deletion(-) diff --git a/Documentation/config.txt b/Documentation/config.txt

[PATCH v4 19/21] commit-graph: add '--reachable' option

When writing commit-graph files, it can be convenient to ask for all reachable commits (starting at the ref set) in the resulting file. This is particularly helpful when writing to stdin is complicated, such as a future integration with 'git gc'. Signed-off-by: Derrick Stolee --- Documentation

[PATCH v4 10/21] commit-graph: verify objects exist

In the 'verify' subcommand, load commits directly from the object database to ensure they exist. Parse by skipping the commit-graph. Signed-off-by: Derrick Stolee --- commit-graph.c | 17 + t/t5318-commit-graph.sh | 7 +++ 2 files changed, 24 insertions(+) diff

[PATCH v4 09/21] commit-graph: verify corrupt OID fanout and lookup

In the commit-graph file, the OID fanout chunk provides an index into the OID lookup. The 'verify' subcommand should find incorrect values in the fanout. Similarly, the 'verify' subcommand should find out-of-order values in the OID lookup. Signed-off-by: Derrick Stolee --- commit-graph.c

[PATCH v4 12/21] commit-graph: verify parent list

from the object database. Test these checks for corrupt parents, too many parents, and wrong parents. Add a boundary check to insert_parent_or_die() for when the parent position value is out of range. The octopus merge will be tested in a later commit. Signed-off-by: Derrick Stolee --- commit

[PATCH v4 11/21] commit-graph: verify root tree OIDs

the object database. Add checks for the root tree OID. Signed-off-by: Derrick Stolee --- commit-graph.c | 17 - t/t5318-commit-graph.sh | 7 +++ 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/commit-graph.c b/commit-graph.c index 0cf1b61d80

[PATCH v4 03/21] commit-graph: parse commit from chosen graph

Before verifying a commit-graph file against the object database, we need to parse all commits from the given commit-graph file. Create parse_commit_in_graph_one() to target a given struct commit_graph. Signed-off-by: Derrick Stolee --- commit-graph.c | 18 +++--- 1 file changed, 15

[PATCH v4 00/21] Integrate commit-graph into 'fsck' and 'gc'

large, but I'll send it in a follow-up PR. Thanks, -Stolee Derrick Stolee (21): commit-graph: UNLEAK before die() commit-graph: fix GRAPH_MIN_SIZE commit-graph: parse commit from chosen graph commit: force commit to parse from object database commit-graph: load a root tree from specific graph commi

[PATCH v4 06/21] commit-graph: add 'verify' subcommand

state. Do not report any errors. Helped-by: Ramsay Jones Signed-off-by: Derrick Stolee --- Documentation/git-commit-graph.txt | 6 + builtin/commit-graph.c | 38 ++ commit-graph.c | 23 ++ commit-graph.h

[PATCH v4 07/21] commit-graph: verify catches corrupt signature

that is intended for a typical user. Helped-by: Szeder Gábor Signed-off-by: Derrick Stolee --- t/t5318-commit-graph.sh | 43 + 1 file changed, 43 insertions(+) diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh index 0830ef9fdd..c0c1ff09b9

[PATCH v4 08/21] commit-graph: verify required chunks are present

The commit-graph file requires the following three chunks: * OID Fanout * OID Lookup * Commit Data If any of these are missing, then the 'verify' subcommand should report a failure. This includes the chunk IDs malformed or the chunk count is truncated. Signed-off-by: Derrick Stolee --- commit

[PATCH v4 05/21] commit-graph: load a root tree from specific graph

When lazy-loading a tree for a commit, it will be important to select the tree from a specific struct commit_graph. Create a new method that specifies the commit-graph file and use that in get_commit_tree_in_graph(). Signed-off-by: Derrick Stolee --- commit-graph.c | 12 +--- 1 file

[PATCH v4 02/21] commit-graph: fix GRAPH_MIN_SIZE

the magic constants. Signed-off-by: Derrick Stolee --- commit-graph.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/commit-graph.c b/commit-graph.c index bb54c1214c..c09e87c3c2 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -34,10 +34,11 @@ #define GRAPH_LAST_EDGE

[PATCH v4 01/21] commit-graph: UNLEAK before die()

Signed-off-by: Derrick Stolee --- builtin/commit-graph.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c index 37420ae0fd..f0875b8bf3 100644 --- a/builtin/commit-graph.c +++ b/builtin/commit-graph.c @@ -51,8 +51,11 @@ static

Re: [PATCH v3 15/20] commit-graph: test for corrupted octopus edge

On 6/2/2018 8:39 AM, Jakub Narebski wrote: Derrick Stolee writes: The commit-graph file has an extra chunk to store the parent int-ids for parents beyond the first parent for octopus merges. Our test repo has a single octopus merge that we can manipulate to demonstrate the 'verify' subcommand

<    3   4   5   6   7   8   9   10   11   12   >