[PATCH v3 1/2] commit-reach: properly peel tags

2018-09-21 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee The can_all_from_reach_with_flag() algorithm was refactored in 4fbcca4e "commit-reach: make can_all_from_reach... linear" but incorrectly assumed that all objects provided were commits. During a fetch negotiation, ok_to_give_up() in upload-pack.c may provide unp

[PATCH v3 2/2] commit-reach: fix memory and flag leaks

2018-09-21 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee The can_all_from_reach_with_flag() method uses 'assign_flag' as a value we can use to mark objects temporarily during our commit walk. The intent is that these flags are removed from all objects before returning. However, this is not the case. The 'from' array could also

[PATCH v3 0/1] contrib: Add script to show uncovered "new" lines

2018-09-21 Thread Derrick Stolee via GitGitGadget
197) ctx->show_object(obj, base->buf, ctx->show_data); ll-merge.c d64324cb60e (Torsten Bögershausen 2018-09-12 21:32:02 +0200 379) marker_size = DEFAULT_CONFLICT_MARKER_SIZE; midx.c 56ee7ff1565 (Derrick Stolee 2018-09-13 11:02:13 -

[PATCH v3 1/1] contrib: add coverage-diff script

2018-09-21 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee We have coverage targets in our Makefile for using gcov to display line coverage based on our test suite. The way I like to do it is to run: make coverage-test make coverage-report This leaves the repo in a state where every X.c file that was covered has

[PATCH v3 0/7] Use generation numbers for --topo-order

2018-09-21 Thread Derrick Stolee via GitGitGadget
ed.blob.core.windows.net/media/2018/06/commit-graph-topo-order-b-a.png Animation showing three-part walk [3] https://github.com/derrickstolee/git/tree/topo-order/testA branch containing this series along with commits to compute commit-graph in entire test suite. Cc: avarab@gmail.comCc: szeder...

[PATCH v3 5/7] commit/revisions: bookkeeping before refactoring

2018-09-21 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee There are a few things that need to move around a little before making a big refactoring in the topo-order logic: 1. We need access to record_author_date() and compare_commits_by_author_date() in revision.c. These are used currently by sort_in_topological_order

[PATCH v3 7/7] revision.c: refactor basic topo-order logic

2018-09-21 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee When running a command like 'git rev-list --topo-order HEAD', Git performed the following steps: 1. Run limit_list(), which parses all reachable commits, adds them to a linked list, and distributes UNINTERESTING flags. If all unprocessed commits are UNINTERESTING

[PATCH v3 1/7] prio-queue: add 'peek' operation

2018-09-21 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee When consuming a priority queue, it can be convenient to inspect the next object that will be dequeued without actually dequeueing it. Our existing library did not have such a 'peek' operation, so add it as prio_queue_peek(). Add a reference-level comparison in t/helper

[PATCH v3 3/7] test-reach: add rev-list tests

2018-09-21 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee The rev-list command is critical to Git's functionality. Ensure it works in the three commit-graph environments constructed in t6600-test-reach.sh. Here are a few important types of rev-list operations: * Basic: git rev-list --topo-order HEAD * Range: git rev-list --topo

[PATCH v3 6/7] revision.h: add whitespace in flag definitions

2018-09-21 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee In anticipation of adding longer flag names in the next change, add an extra tab to each flag definition in revision.h. Signed-off-by: Derrick Stolee --- revision.h | 28 ++-- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git

[PATCH v3 4/7] revision.c: begin refactoring --topo-order logic

2018-09-21 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee When running 'git rev-list --topo-order' and its kin, the topo_order setting in struct rev_info implies the limited setting. This means that the following things happen during prepare_revision_walk(): * revs->limited implies we run limit_list() to walk the ent

[PATCH v3 2/7] test-reach: add run_three_modes method

2018-09-21 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee The 'test_three_modes' method assumes we are using the 'test-tool reach' command for our test. However, we may want to use the data shape of our commit graph and the three modes (no commit-graph, full commit-graph, partial commit-graph) for other git commands. Split

[PATCH v4 0/1] Properly peel tags in can_all_from_reach_with_flags()

2018-09-24 Thread Derrick Stolee via GitGitGadget
n-commit objects. [1] https://public-inbox.org/git/0bf9103c-9377-506b-7ad7-e5273d8e9...@gmail.com/T/#u Derrick Stolee (1): commit-reach: properly peel tags and clear flags commit-reach.c| 44 +-- t/helper/test-reach.c | 22 +- t/

[PATCH v4 1/1] commit-reach: properly peel tags and clear flags

2018-09-24 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee The can_all_from_reach_with_flag() algorithm was refactored in 4fbcca4e "commit-reach: make can_all_from_reach... linear" but incorrectly assumed that all objects provided were commits. During a fetch negotiation, ok_to_give_up() in upload-pack.c may provide unp

[PATCH 0/1] read-cache: update index format default to v4

2018-09-24 Thread Derrick Stolee via GitGitGadget
of Git. Personally, I'm not an expert here, but I am happy to start the conversation. Thanks, -Stolee Derrick Stolee (1): read-cache: update index format default to v4 read-cache.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) base-commit: 53f9a3e157dbbc901a02ac2c73346d375e24978c

[PATCH 1/1] read-cache: update index format default to v4

2018-09-24 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee The index v4 format has been available since 2012 with 9d22778 "reach-cache.c: write prefix-compressed names in the index". Since the format has been stable for so long, almost all versions of Git in use today understand version 4, removing one barrier

[PATCH v2 1/6] revision: add mark_tree_uninteresting_sparse

2018-11-29 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee In preparation for a new algorithm that walks fewer trees when creating a pack from a set of revisions, create a method that takes an oidset of tree oids and marks reachable objects as UNINTERESTING. The current implementation uses the existing mark_tree_uninteresting

[PATCH v2 2/6] list-objects: consume sparse tree walk

2018-11-29 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee When creating a pack-file using 'git pack-objects --revs' we provide a list of interesting and uninteresting commits. For example, a push operation would make the local topic branch be interesting and the known remote refs as uninteresting. We want to discover the set of new

[PATCH v2 5/6] pack-objects: create pack.useSparse setting

2018-11-29 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee The '--sparse' flag in 'git pack-objects' changes the algorithm used to enumerate objects to one that is faster for individual users pushing new objects that change only a small cone of the working directory. The sparse algorithm is not recommended for a server, which likely

[PATCH v2 0/6] Add a new "sparse" tree walk algorithm

2018-11-29 Thread Derrick Stolee via GitGitGadget
V2: * Added GIT_TEST_PACK_SPARSE test option. * Fixed test breakages when GIT_TEST_PACK_SPARSE is enabled by adding null checks. Derrick Stolee (6): revision: add mark_tree_uninteresting_sparse list-objects: consume sparse tree walk pack-objects: add --sparse option revision: i

[PATCH v2 6/6] pack-objects: create GIT_TEST_PACK_SPARSE

2018-11-29 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee Create a test variable GIT_TEST_PACK_SPARSE to enable the sparse object walk algorithm by default during the test suite. Enabling this variable ensures coverage in many interesting cases, such as shallow clones, partial clones, and missing objects. Signed-off-by: Derrick

[PATCH v2 4/6] revision: implement sparse algorithm

2018-11-29 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee When enumerating objects to place in a pack-file during 'git pack-objects --revs', we discover the "frontier" of commits that we care about and the boundary with commit we find uninteresting. From that point, we walk trees to discover which trees and blobs are uni

[PATCH v2 3/6] pack-objects: add --sparse option

2018-11-29 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee Add a '--sparse' option flag to the pack-objects builtin. This allows the user to specify that they want to use the new logic for walking trees. This logic currently does not differ from the existing output, but will in a later change. Create a new test script, t5322-pack

[PATCH 4/5] revision: implement sparse algorithm

2018-11-28 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee When enumerating objects to place in a pack-file during 'git pack-objects --revs', we discover the "frontier" of commits that we care about and the boundary with commit we find uninteresting. From that point, we walk trees to discover which trees and blobs are uni

[PATCH 0/5] Add a new "sparse" tree walk algorithm

2018-11-28 Thread Derrick Stolee via GitGitGadget
py" because a rename would (usually) change the original folder and trigger a walk into that path, discovering the objects. In order to benefit from this approach, the user can opt-in using the pack.useSparse config setting. This setting can be overridden using the '--no-sparse' opti

[PATCH 3/5] pack-objects: add --sparse option

2018-11-28 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee Add a '--sparse' option flag to the pack-objects builtin. This allows the user to specify that they want to use the new logic for walking trees. This logic currently does not differ from the existing output, but will in a later change. Create a new test script, t5322-pack

[PATCH 5/5] pack-objects: create pack.useSparse setting

2018-11-28 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee The '--sparse' flag in 'git pack-objects' changes the algorithm used to enumerate objects to one that is faster for individual users pushing new objects that change only a small cone of the working directory. The sparse algorithm is not recommended for a server, which likely

[PATCH 2/5] list-objects: consume sparse tree walk

2018-11-28 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee When creating a pack-file using 'git pack-objects --revs' we provide a list of interesting and uninteresting commits. For example, a push operation would make the local topic branch be interesting and the known remote refs as uninteresting. We want to discover the set of new

[PATCH 1/5] revision: add mark_tree_uninteresting_sparse

2018-11-28 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee In preparation for a new algorithm that walks fewer trees when creating a pack from a set of revisions, create a method that takes an oidset of tree oids and marks reachable objects as UNINTERESTING. The current implementation uses the existing mark_tree_uninteresting

[PATCH 1/1] revision.c: use new topo-order logic in tests

2018-11-19 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee The revision-walk machinery is being rewritten to use generation numbers in the commit-graph when availble. Due to some problematic commit histories, the new logic can be slower than the previous method due to how commit dates and generation numbers interact. Thus, the new

[PATCH 0/1] Use new topo-order logic with GIT_TEST_COMMIT_GRAPH

2018-11-19 Thread Derrick Stolee via GitGitGadget
, but that doesn't seem necessary. Thanks, -Stolee Derrick Stolee (1): revision.c: use new topo-order logic in tests revision.c | 4 1 file changed, 4 insertions(+) base-commit: 561b583749b7428f1790f03164d0d0e75be71d7b Published-As: https://github.com/gitgitgadget/git/releases/tags/pr-83

[PATCH 0/1] contrib: Add script to show uncovered "new" lines

2018-09-12 Thread Derrick Stolee via GitGitGadget
t: builtin/commit.c 859fdc0c3cf (Derrick Stolee 2018-08-29 05:49:04 -0700 1657) write_commit_graph_reachable(get_object_directory(), 0); builtin/rev-list.c 250edfa8c87 (Harald Nordgren2018-04-18 23:05:35 +0200 431) bisect_flags |= BISECT_FIND_ALL

[PATCH 1/1] contrib: add coverage-diff script

2018-09-12 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee We have coverage targets in our Makefile for using gcov to display line coverage based on our test suite. The way I like to do it is to run: make coverage-test make coverage-report This leaves the repo in a state where every X.c file that was covered has

[PATCH v2 0/1] contrib: Add script to show uncovered "new" lines

2018-09-13 Thread Derrick Stolee via GitGitGadget
t: builtin/commit.c 859fdc0c3cf (Derrick Stolee 2018-08-29 05:49:04 -0700 1657) write_commit_graph_reachable(get_object_directory(), 0); builtin/rev-list.c 250edfa8c87 (Harald Nordgren2018-04-18 23:05:35 +0200 431) bisect_flags |= BISECT_FIND_ALL

[PATCH v2 1/1] contrib: add coverage-diff script

2018-09-13 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee We have coverage targets in our Makefile for using gcov to display line coverage based on our test suite. The way I like to do it is to run: make coverage-test make coverage-report This leaves the repo in a state where every X.c file that was covered has

[PATCH v2 6/6] revision.c: refactor basic topo-order logic

2018-09-17 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee When running a command like 'git rev-list --topo-order HEAD', Git performed the following steps: 1. Run limit_list(), which parses all reachable commits, adds them to a linked list, and distributes UNINTERESTING flags. If all unprocessed commits are UNINTERESTING

[PATCH v2 4/6] revision.c: begin refactoring --topo-order logic

2018-09-17 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee When running 'git rev-list --topo-order' and its kin, the topo_order setting in struct rev_info implies the limited setting. This means that the following things happen during prepare_revision_walk(): * revs->limited implies we run limit_list() to walk the ent

[PATCH v2 2/6] test-reach: add run_three_modes method

2018-09-17 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee The 'test_three_modes' method assumes we are using the 'test-tool reach' command for our test. However, we may want to use the data shape of our commit graph and the three modes (no commit-graph, full commit-graph, partial commit-graph) for other git commands. Split

[PATCH v2 5/6] commit/revisions: bookkeeping before refactoring

2018-09-17 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee There are a few things that need to move around a little before making a big refactoring in the topo-order logic: 1. We need access to record_author_date() and compare_commits_by_author_date() in revision.c. These are used currently by sort_in_topological_order

[PATCH v2 0/6] Use generation numbers for --topo-order

2018-09-17 Thread Derrick Stolee via GitGitGadget
Graph III: Generations and Graph Algorithms [2] https://msdnshared.blob.core.windows.net/media/2018/06/commit-graph-topo-order-b-a.png Animation showing three-part walk [3] https://github.com/derrickstolee/git/tree/topo-order/testA branch containing this series along with commits to compute commit-graph in e

[PATCH v2 3/6] test-reach: add rev-list tests

2018-09-17 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee The rev-list command is critical to Git's functionality. Ensure it works in the three commit-graph environments constructed in t6600-test-reach.sh. Here are a few important types of rev-list operations: * Basic: git rev-list --topo-order HEAD * Range: git rev-list --topo

[PATCH 0/1] Properly peel tags in can_all_from_reach_with_flags()

2018-09-12 Thread Derrick Stolee via GitGitGadget
...@gmail.com/T/#u Derrick Stolee (1): commit-reach: properly peel tags commit-reach.c| 25 ++--- t/helper/test-reach.c | 22 +- t/t6600-test-reach.sh | 30 -- 3 files changed, 63 insertions(+), 14 deletions(-) base-commit

[PATCH 1/1] commit-reach: properly peel tags

2018-09-12 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee The can_all_from_reach_with_flag() algorithm was refactored in 4fbcca4e "commit-reach: make can_all_from_reach... linear" but incorrectly assumed that all objects provided were commits. During a fetch negotiation, ok_to_give_up() in upload-pack.c may provide unp

[PATCH v2 1/1] t3206-range-diff.sh: cover single-patch case

2018-09-12 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee The commit 40ce4160 "format-patch: allow --range-diff to apply to a lone-patch" added the ability to see a range-diff as commentary after the commit message of a single patch series (i.e. [PATCH] instead of [PATCH X/N]). However, this functionality was not covered

[PATCH v2 0/1] Add coverage for 'git format-patch --range-diff' single-patch case

2018-09-12 Thread Derrick Stolee via GitGitGadget
...@sunshine.co Cc: p...@peff.net [1] https://github.com/derrickstolee/git/blob/coverage/contrib/coverage-diff.shA bash script to report uncovered lines that were added in a diff. Derrick Stolee (1): t3206-range-diff.sh: cover single-patch case t/t3206-range-diff.sh | 5 + 1 file changed, 5

[PATCH 1/1] t3206-range-diff.sh: cover single-patch case

2018-09-11 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee The commit 40ce4160 "format-patch: allow --range-diff to apply to a lone-patch" added the ability to see a range-diff as commentary after the commit message of a single patch series (i.e. [PATCH] instead of [PATCH X/N]). However, this functionality was not covered

[PATCH 0/1] Add coverage for 'git format-patch --range-diff' single-patch case

2018-09-11 Thread Derrick Stolee via GitGitGadget
...@sunshine.co Cc: p...@peff.net [1] https://github.com/derrickstolee/git/blob/coverage/contrib/coverage-diff.shA bash script to report uncovered lines that were added in a diff. Derrick Stolee (1): t3206-range-diff.sh: cover single-patch case t/t3206-range-diff.sh | 5 + 1 file changed, 5

[PATCH v2 1/6] prio-queue: add 'peek' operation

2018-09-17 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee When consuming a priority queue, it can be convenient to inspect the next object that will be dequeued without actually dequeueing it. Our existing library did not have such a 'peek' operation, so add it as prio_queue_peek(). Add a reference-level comparison in t/helper

[PATCH v2 1/1] commit-reach: properly peel tags

2018-09-13 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee The can_all_from_reach_with_flag() algorithm was refactored in 4fbcca4e "commit-reach: make can_all_from_reach... linear" but incorrectly assumed that all objects provided were commits. During a fetch negotiation, ok_to_give_up() in upload-pack.c may provide unp

[PATCH v2 0/1] Properly peel tags in can_all_from_reach_with_flags()

2018-09-13 Thread Derrick Stolee via GitGitGadget
n-commit objects. [1] https://public-inbox.org/git/0bf9103c-9377-506b-7ad7-e5273d8e9...@gmail.com/T/#u Derrick Stolee (1): commit-reach: properly peel tags commit-reach.c| 33 ++--- t/helper/test-reach.c | 22 +- t/t6600-test-reach.sh | 30 +++

[PATCH v2 10/11] multi-pack-index: report progress during 'verify'

2018-09-13 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee When verifying a multi-pack-index, the only action that takes significant time is checking the object offsets. For example, to verify a multi-pack-index containing 6.2 million objects in the Linux kernel repository takes 1.3 seconds on my machine. 99% of that time is spent

[PATCH v2 00/11] Add 'git multi-pack-index verify' command

2018-09-13 Thread Derrick Stolee via GitGitGadget
d a commit that provides progress updates when checking object offsets. Based on ds/multi-pack-index [1] https://public-inbox.org/git/20180904202729.13900-1-ava...@gmail.com/T/#u Derrick Stolee (11): multi-pack-index: add 'verify' verb multi-pack-index: verify bad header multi-pack-ind

[PATCH v2 04/11] multi-pack-index: verify packname order

2018-09-13 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee The final check we make while loading a multi-pack-index is that the packfile names are in lexicographical order. Make this error be a die() instead. In order to test this condition, we need multiple packfiles. Earlier in t5319-multi-pack-index.sh, we tested the interaction

[PATCH v2 09/11] multi-pack-index: verify object offsets

2018-09-13 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee The 'git multi-pack-index verify' command must verify the object offsets stored in the multi-pack-index are correct. There are two ways the offset chunk can be incorrect: the pack-int-id and the object offset. Replace the BUG() statement with a die() statement, now that we

[PATCH v2 08/11] multi-pack-index: fix 32-bit vs 64-bit size check

2018-09-13 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee When loading a 64-bit offset, we intend to check that off_t can store the resulting offset. However, the condition accidentally checks the 32-bit offset to see if it is smaller than a 64-bit value. Fix it, and this will be covered by a test in the 'git multi-pack-index

[PATCH v2 05/11] multi-pack-index: verify missing pack

2018-09-13 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee Signed-off-by: Derrick Stolee --- midx.c | 16 t/t5319-multi-pack-index.sh | 5 + 2 files changed, 21 insertions(+) diff --git a/midx.c b/midx.c index e655a15aed..a02b19efc1 100644 --- a/midx.c +++ b/midx.c @@ -926,13 +926,29

[PATCH v2 11/11] fsck: verify multi-pack-index

2018-09-13 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee When core.multiPackIndex is true, we may have a multi-pack-index in our object directory. Add calls to 'git multi-pack-index verify' at the end of 'git fsck' if so. Signed-off-by: Derrick Stolee --- builtin/fsck.c | 18 ++ t/t5319-multi-pack

[PATCH v2 07/11] multi-pack-index: verify oid lookup order

2018-09-13 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee Signed-off-by: Derrick Stolee --- midx.c | 11 +++ t/t5319-multi-pack-index.sh | 8 2 files changed, 19 insertions(+) diff --git a/midx.c b/midx.c index dfd26b4d74..06d5cfc826 100644 --- a/midx.c +++ b/midx.c @@ -959,5 +959,16 @@ int

[PATCH v2 03/11] multi-pack-index: verify corrupt chunk lookup table

2018-09-13 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee Signed-off-by: Derrick Stolee --- midx.c | 3 +++ t/t5319-multi-pack-index.sh | 13 + 2 files changed, 16 insertions(+) diff --git a/midx.c b/midx.c index ec78254bb6..8b054b39ab 100644 --- a/midx.c +++ b/midx.c @@ -100,6 +100,9 @@ struct

[PATCH v2 02/11] multi-pack-index: verify bad header

2018-09-13 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee When verifying if a multi-pack-index file is valid, we want the command to fail to signal an invalid file. Previously, we wrote an error to stderr and continued as if we had no multi-pack-index. Now, die() instead of error(). Add tests that check corrupted headers in a few

[PATCH v2 06/11] multi-pack-index: verify oid fanout order

2018-09-13 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee Signed-off-by: Derrick Stolee --- midx.c | 9 + t/t5319-multi-pack-index.sh | 8 2 files changed, 17 insertions(+) diff --git a/midx.c b/midx.c index a02b19efc1..dfd26b4d74 100644 --- a/midx.c +++ b/midx.c @@ -950,5 +950,14 @@ int

[PATCH v2 01/11] multi-pack-index: add 'verify' verb

2018-09-13 Thread Derrick Stolee via GitGitGadget
From: Derrick Stolee The multi-pack-index builtin writes multi-pack-index files, and uses a 'write' verb to do so. Add a 'verify' verb that checks this file matches the contents of the pack-indexes it replaces. The current implementation is a no-op, but will be extended in small increments

<    9   10   11   12   13   14