Re: [PATCH v6 10/12] fsmonitor: add test cases for fsmonitor extension

2017-09-18 Thread Ben Peart
On 9/16/2017 11:27 AM, Torsten Bögershausen wrote: On 2017-09-15 21:20, Ben Peart wrote: +if [ "$1" != 1 ] +then + echo -e "Unsupported core.fsmonitor hook version.\n" >&2 + exit 1 +fi The echo -e not portable (It was detected by a tighter version of

Re: [PATCH v6 10/12] fsmonitor: add test cases for fsmonitor extension

2017-09-18 Thread Ben Peart
On 9/17/2017 12:47 AM, Junio C Hamano wrote: Ben Peart writes: +write_integration_script() { + write_script .git/hooks/fsmonitor-test<<-\EOF + if [ "$#" -ne 2 ]; then + echo "$0: exactly 2 arguments expected" + exit 2 +

RE: [PATCH v6 08/12] fsmonitor: add a test tool to dump the index extension

2017-09-18 Thread Ben Peart
> -Original Message- > From: Torsten Bögershausen [mailto:tbo...@web.de] > Sent: Monday, September 18, 2017 11:43 AM > To: Ben Peart ; Junio C Hamano > ; Ben Peart > Cc: david.tur...@twosigma.com; ava...@gmail.com; > christian.cou...@gmail.com; git@vger.kernel.or

RE: [PATCH v6 12/12] fsmonitor: add a performance test

2017-09-18 Thread Ben Peart
> -Original Message- > From: Johannes Schindelin [mailto:johannes.schinde...@gmx.de] > Sent: Monday, September 18, 2017 10:25 AM > To: Ben Peart > Cc: david.tur...@twosigma.com; ava...@gmail.com; > christian.cou...@gmail.com; git@vger.kernel.org; gits...@pobox.com; >

RE: [PATCH v6 08/12] fsmonitor: add a test tool to dump the index extension

2017-09-19 Thread Ben Peart
> -Original Message- > From: Torsten Bögershausen [mailto:tbo...@web.de] > Sent: Tuesday, September 19, 2017 10:16 AM > To: Ben Peart > Cc: Ben Peart ; Junio C Hamano > ; david.tur...@twosigma.com; ava...@gmail.com; > christian.cou...@gmail.com; git@vger.kernel.or

[PATCH v7 07/12] update-index: add fsmonitor support to update-index

2017-09-19 Thread Ben Peart
Add support in update-index to manually add/remove the fsmonitor extension via --[no-]fsmonitor flags. Add support in update-index to manually set/clear the fsmonitor valid bit via --[no-]fsmonitor-valid flags. Signed-off-by: Ben Peart --- builtin/update-index.c | 33

[PATCH v7 12/12] fsmonitor: add a performance test

2017-09-19 Thread Ben Peart
tests Signed-off-by: Ben Peart Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile| 1 + t/helper/.gitignore | 1 + t/helper/test-drop-caches.c | 162 ++ t/perf/p7519-fsmonitor.sh | 184

[PATCH v7 00/12] Fast git status via a file system watcher

2017-09-19 Thread Ben Peart
"Falling back to scanning...\n" if $o->{error}; - binmode STDOUT, ":utf8"; - local $, = "\0"; - print @{$o->{files}}; - open ($fh, ">", ".git/watchman-output.out"); print $fh @{$o->{files}};

[PATCH v7 02/12] preload-index: add override to enable testing preload-index

2017-09-19 Thread Ben Peart
Preload index doesn't run unless it has a minimum number of 1000 files. To enable running tests with fewer files, add an environment variable (GIT_FORCE_PRELOAD_TEST) which will override that minimum and set it to 2. Signed-off-by: Ben Peart --- preload-index.c | 2 ++ 1 file chang

[PATCH v7 11/12] fsmonitor: add a sample integration script for Watchman

2017-09-19 Thread Ben Peart
. Configure git to use the extension: git config core.fsmonitor .git/hooks/fsmonitor-watchman Optionally turn on the untracked cache for optimal performance. Signed-off-by: Ben Peart Signed-off-by: Johannes Schindelin Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Christian Couder

[PATCH v7 03/12] update-index: add a new --force-write-index option

2017-09-19 Thread Ben Peart
write-index) to update-index that will ensure the index is written out even if the cache_changed flag is not set. Signed-off-by: Ben Peart --- builtin/update-index.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/builtin/update-index.c b/builtin/update-index.c index d562f

[PATCH v7 05/12] fsmonitor: add documentation for the fsmonitor extension.

2017-09-19 Thread Ben Peart
This includes the core.fsmonitor setting, the query-fsmonitor hook, and the fsmonitor index extension. Signed-off-by: Ben Peart --- Documentation/config.txt | 7 + Documentation/git-ls-files.txt | 7 - Documentation/git-update-index.txt | 45

[PATCH v7 10/12] fsmonitor: add test cases for fsmonitor extension

2017-09-19 Thread Ben Peart
n Watchman where it holds on to handles for directories and files which prevents the test directory from being cleaned up properly. Signed-off-by: Ben Peart --- t/t7519-status-fsmonitor.sh | 304 t/t7519/fsmonitor-all | 24 t/t7519/fsmo

[PATCH v7 09/12] split-index: disable the fsmonitor extension when running the split index test

2017-09-19 Thread Ben Peart
which causes the test to fail. The potential matrix of index extensions and index versions can is quite large so instead disable the extension before attempting to run the test. Signed-off-by: Ben Peart --- t/t1700-split-index.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/t/t1700-split

[PATCH v7 08/12] fsmonitor: add a test tool to dump the index extension

2017-09-19 Thread Ben Peart
Add a test utility (test-dump-fsmonitor) that will dump the fsmonitor index extension. Signed-off-by: Ben Peart --- Makefile | 1 + t/helper/test-dump-fsmonitor.c | 21 + 2 files changed, 22 insertions(+) create mode 100644 t/helper/test-dump

[PATCH v7 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.

2017-09-19 Thread Ben Peart
; when git updates a cache entry to match the current state on disk, it will now set the CE_FSMONITOR_VALID bit. Inversely, anytime git changes a cache entry, the CE_FSMONITOR_VALID bit is cleared and the corresponding untracked cache directory is marked invalid. Signed-off-by: Ben Peart --- Makefile

[PATCH v7 01/12] bswap: add 64 bit endianness helper get_be64

2017-09-19 Thread Ben Peart
Add a new get_be64 macro to enable 64 bit endian conversions on memory that may or may not be aligned. Signed-off-by: Ben Peart --- compat/bswap.h | 22 ++ 1 file changed, 22 insertions(+) diff --git a/compat/bswap.h b/compat/bswap.h index 7d063e9e40..6b22c46214 100644

[PATCH v7 06/12] ls-files: Add support in ls-files to display the fsmonitor valid bit

2017-09-19 Thread Ben Peart
Add a new command line option (-f) to ls-files to have it use lowercase letters for 'fsmonitor valid' files Signed-off-by: Ben Peart --- builtin/ls-files.c | 8 ++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/builtin/ls-files.c b/builtin/ls-files.c index

Re: [PATCH v6 10/12] fsmonitor: add test cases for fsmonitor extension

2017-09-19 Thread Ben Peart
7 @@ do git add new && git add dir1/new && git add dir2/new && + git add dir3/new && git status >actual &&

Re: [PATCH v7 06/12] ls-files: Add support in ls-files to display the fsmonitor valid bit

2017-09-19 Thread Ben Peart
On 9/19/2017 3:46 PM, David Turner wrote: -Original Message- From: Ben Peart [mailto:benpe...@microsoft.com] Sent: Tuesday, September 19, 2017 3:28 PM To: benpe...@microsoft.com Cc: David Turner ; ava...@gmail.com; christian.cou...@gmail.com; git@vger.kernel.org; gits...@pobox.com

RE: [PATCH v7 06/12] ls-files: Add support in ls-files to display the fsmonitor valid bit

2017-09-19 Thread Ben Peart
> -Original Message- > From: David Turner [mailto:david.tur...@twosigma.com] > Sent: Tuesday, September 19, 2017 5:27 PM > To: 'Ben Peart' ; Ben Peart > > Cc: ava...@gmail.com; christian.cou...@gmail.com; git@vger.kernel.org; > gits...@pobox.com; j

Re: [PATCH v7 03/12] update-index: add a new --force-write-index option

2017-09-20 Thread Ben Peart
On 9/20/2017 1:47 AM, Junio C Hamano wrote: Ben Peart writes: + OPT_SET_INT(0, "force-write-index", &force_write, + N_("write out the index even if is not flagged as changed"), 1), Hmph. The only time this makes difference is

Re: [PATCH v7 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.

2017-09-20 Thread Ben Peart
On 9/19/2017 10:28 PM, Junio C Hamano wrote: Ben Peart writes: +/* do stat comparison even if CE_FSMONITOR_VALID is true */ +#define CE_MATCH_IGNORE_FSMONITOR 0X20 Hmm, when should a programmer use this flag? Pretty much the same places you would also use CE_MATCH_IGNORE_VALID and

Re: [PATCH v7 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.

2017-09-20 Thread Ben Peart
On 9/20/2017 2:23 AM, Junio C Hamano wrote: Ben Peart writes: @@ -344,6 +346,7 @@ struct index_state { struct hashmap dir_hash; unsigned char sha1[20]; struct untracked_cache *untracked; + uint64_t fsmonitor_last_update; This field being zero has more

Re: [PATCH v7 05/12] fsmonitor: add documentation for the fsmonitor extension.

2017-09-20 Thread Ben Peart
Thanks for the review. I'm not an English major so appreciate the feedback on my attempts to document the feature. On 9/20/2017 6:00 AM, Martin Ågren wrote: On 19 September 2017 at 21:27, Ben Peart wrote: diff --git a/Documentation/git-update-index.txt b/Documentation/git-update-inde

Re: [PATCH v6 09/12] split-index: disable the fsmonitor extension when running the split index test

2017-09-20 Thread Ben Peart
On 9/19/2017 4:43 PM, Jonathan Nieder wrote: Hi, Ben Peart wrote: The split index test t1700-split-index.sh has hard coded SHA values for the index. Currently it supports index V4 and V3 but assumes there are no index extensions loaded. When manually forcing the fsmonitor extension to be

Re: [PATCH v7 02/12] preload-index: add override to enable testing preload-index

2017-09-20 Thread Ben Peart
On 9/20/2017 6:06 PM, Stefan Beller wrote: On Tue, Sep 19, 2017 at 12:27 PM, Ben Peart wrote: Preload index doesn't run unless it has a minimum number of 1000 files. To enable running tests with fewer files, add an environment variable (GIT_FORCE_PRELOAD_TEST) which will override

Re: [PATCH v6 09/12] split-index: disable the fsmonitor extension when running the split index test

2017-09-20 Thread Ben Peart
On 9/20/2017 1:46 PM, Jonathan Nieder wrote: Hi, Ben Peart wrote: On 9/19/2017 4:43 PM, Jonathan Nieder wrote: This feels to me like the wrong fix. Wouldn't it be better for the test not to depend on the precise object ids? See the "Tips for Writing Tests" section i

Re: [PATCH v7 03/12] update-index: add a new --force-write-index option

2017-09-20 Thread Ben Peart
On 9/20/2017 9:46 PM, Junio C Hamano wrote: Ben Peart writes: Lets see how my ascii art skills do at describing this: Your ascii art is fine. If you said upfront that the capital letters signify points in time, lower letters are file-touching events, and time flows from left to right, it

Re: [PATCH v7 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.

2017-09-20 Thread Ben Peart
On 9/20/2017 10:00 PM, Junio C Hamano wrote: Ben Peart writes: Pretty much the same places you would also use CE_MATCH_IGNORE_VALID and CE_MATCH_IGNORE_SKIP_WORKTREE which serve the same role for those features. That is generally when you are about to overwrite data so want to be *really

Re: [PATCH v7 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.

2017-09-21 Thread Ben Peart
On 9/20/2017 10:24 PM, Ben Peart wrote: On 9/20/2017 10:00 PM, Junio C Hamano wrote: Ben Peart writes: Pretty much the same places you would also use CE_MATCH_IGNORE_VALID and CE_MATCH_IGNORE_SKIP_WORKTREE which serve the same role for those features.  That is generally when you are

[PATCH v8 01/12] bswap: add 64 bit endianness helper get_be64

2017-09-22 Thread Ben Peart
Add a new get_be64 macro to enable 64 bit endian conversions on memory that may or may not be aligned. Signed-off-by: Ben Peart --- compat/bswap.h | 22 ++ 1 file changed, 22 insertions(+) diff --git a/compat/bswap.h b/compat/bswap.h index 7d063e9e40..6b22c46214 100644

[PATCH v8 02/12] preload-index: add override to enable testing preload-index

2017-09-22 Thread Ben Peart
u run existing tests and have the core.preloadindex code path execute as long as the test has at least 2 files by setting GIT_FORCE_PRELOAD_TEXT=1 before running the test. Signed-off-by: Ben Peart --- preload-index.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/preload-index.c b/preload-in

[PATCH v8 00/12] Fast git status via a file system watcher

2017-09-22 Thread Ben Peart
TCH_IGNORE_FSMONITOR)) + ie_match_stat(o->src_index, old, &st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE)) update |= CE_UPDATE; } add_entry(o, old, update, 0); Ben Peart (12): bswap: a

[PATCH v8 03/12] update-index: add a new --force-write-index option

2017-09-22 Thread Ben Peart
write-index) to update-index that will ensure the index is written out even if the cache_changed flag is not set. Signed-off-by: Ben Peart --- builtin/update-index.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/builtin/update-index.c b/builtin/update-index.c index d562f

[PATCH v8 12/12] fsmonitor: add a performance test

2017-09-22 Thread Ben Peart
. Signed-off-by: Ben Peart Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile| 1 + t/helper/.gitignore | 1 + t/helper/test-drop-caches.c | 164 +++ t/perf/p7519-fsmonitor.sh | 184

[PATCH v8 05/12] fsmonitor: add documentation for the fsmonitor extension.

2017-09-22 Thread Ben Peart
This includes the core.fsmonitor setting, the fsmonitor integration hook, and the fsmonitor index extension. Also add documentation for the new fsmonitor options to ls-files and update-index. Signed-off-by: Ben Peart --- Documentation/config.txt | 7 + Documentation/git-ls

[PATCH v8 06/12] ls-files: Add support in ls-files to display the fsmonitor valid bit

2017-09-22 Thread Ben Peart
Add a new command line option (-f) to ls-files to have it use lowercase letters for 'fsmonitor valid' files Signed-off-by: Ben Peart --- builtin/ls-files.c | 8 ++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/builtin/ls-files.c b/builtin/ls-files.c index

[PATCH v8 10/12] fsmonitor: add test cases for fsmonitor extension

2017-09-22 Thread Ben Peart
n Watchman where it holds on to handles for directories and files which prevents the test directory from being cleaned up properly. Signed-off-by: Ben Peart --- t/t7519-status-fsmonitor.sh | 304 t/t7519/fsmonitor-all | 24 t/t7519/fsmo

[PATCH v8 09/12] split-index: disable the fsmonitor extension when running the split index test

2017-09-22 Thread Ben Peart
which causes the test to fail. The potential matrix of index extensions and index versions can is quite large so instead temporarily disable the extension before attempting to run the test until the underlying problem of hard coded SHA values is fixed. Signed-off-by: Ben Peart --- t/t1700-split

[PATCH v8 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.

2017-09-22 Thread Ben Peart
; when git updates a cache entry to match the current state on disk, it will now set the CE_FSMONITOR_VALID bit. Inversely, anytime git changes a cache entry, the CE_FSMONITOR_VALID bit is cleared and the corresponding untracked cache directory is marked invalid. Signed-off-by: Ben Peart --- Makefile

[PATCH v8 11/12] fsmonitor: add a sample integration script for Watchman

2017-09-22 Thread Ben Peart
. Configure git to use the extension: git config core.fsmonitor .git/hooks/fsmonitor-watchman Optionally turn on the untracked cache for optimal performance. Signed-off-by: Ben Peart Signed-off-by: Johannes Schindelin Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Christian Couder

[PATCH v8 07/12] update-index: add fsmonitor support to update-index

2017-09-22 Thread Ben Peart
Add support in update-index to manually add/remove the fsmonitor extension via --[no-]fsmonitor flags. Add support in update-index to manually set/clear the fsmonitor valid bit via --[no-]fsmonitor-valid flags. Signed-off-by: Ben Peart --- builtin/update-index.c | 33

[PATCH v8 08/12] fsmonitor: add a test tool to dump the index extension

2017-09-22 Thread Ben Peart
Add a test utility (test-dump-fsmonitor) that will dump the fsmonitor index extension. Signed-off-by: Ben Peart --- Makefile | 1 + t/helper/test-dump-fsmonitor.c | 21 + 2 files changed, 22 insertions(+) create mode 100644 t/helper/test-dump

RE: [PATCH v8 01/12] bswap: add 64 bit endianness helper get_be64

2017-09-23 Thread Ben Peart
Thanks, Ben > -Original Message- > From: Martin Ågren [mailto:martin.ag...@gmail.com] > Sent: Friday, September 22, 2017 7:37 PM > To: Ben Peart > Cc: David Turner ; Ævar Arnfjörð Bjarmason > ; Christian Couder ; > git@vger.kernel.org; Junio C Hamano ; Johannes

RE: [PATCH v8 08/12] fsmonitor: add a test tool to dump the index extension

2017-09-23 Thread Ben Peart
> -Original Message- > From: Martin Ågren [mailto:martin.ag...@gmail.com] > Sent: Friday, September 22, 2017 7:37 PM > To: Ben Peart > Cc: David Turner ; Ævar Arnfjörð Bjarmason > ; Christian Couder ; > git@vger.kernel.org; Junio C Hamano ; Johannes > Schindeli

RE: [PATCH v8 00/12] Fast git status via a file system watcher

2017-09-29 Thread Ben Peart
> -Original Message- > From: Junio C Hamano [mailto:gits...@pobox.com] > Sent: Thursday, September 28, 2017 10:21 PM > To: Ben Peart > Cc: david.tur...@twosigma.com; ava...@gmail.com; > christian.cou...@gmail.com; git@vger.kernel.org; > johannes.schinde...@gmx.de;

Re: [PATCH v6 00/40] Add initial experimental external ODB support

2017-10-02 Thread Ben Peart
tial-refspec to git clone along with tests. - Patch 40/40 adds documentation about transfering objects and metadata when using the external odb mechanism. This patch is new since v5. Future work ~~~ There are still things that could be cleaned or improved. I t

Re: [PATCH v6 09/40] Add initial external odb support

2017-10-02 Thread Ben Peart
On 9/29/2017 4:36 PM, Jonathan Tan wrote: On Wed, 27 Sep 2017 18:46:30 +0200 Christian Couder wrote: I am ok to split the patch series, but I am not sure that 01/40 to 09/40 is the right range for the first patch series. I would say that 01/40 to 07/40 is better as it can be seen as a separa

Re: [PATCH v8 00/12] Fast git status via a file system watcher

2017-10-03 Thread Ben Peart
On 10/1/2017 4:24 AM, Junio C Hamano wrote: Ben Peart writes: I had accumulated the same set of changes with one addition of removing a duplicate "the" from a comment in the fsmonitor.h file: diff --git a/fsmonitor.h b/fsmonitor.h index 8eb6163455..0de644e01a 100644 --- a/fsmonit

Re: [PATCH v8 00/12] Fast git status via a file system watcher

2017-10-04 Thread Ben Peart
On 10/3/2017 10:09 PM, Junio C Hamano wrote: Ben Peart writes: Well, rats. I found one more issue that applies to two of the commits. Can you squash this in as well or do you want it in some other form? Rats indeed. Let's go incremental as promised, perhaps like this (but please sup

Re: [PATCH v8 00/12] Fast git status via a file system watcher

2017-10-04 Thread Ben Peart
n my perl capabilities and is fixed with the following patch: -- >8 -- From 3e3b983a4208a62d166c233a7de3bf045322f6c7 Mon Sep 17 00:00:00 2001 From: Ben Peart Date: Wed, 4 Oct 2017 08:33:39 -0400 Subject: [PATCH] fsmonitor: preserve utf8 filenames in fsmonitor-watchman log Update the test fs

[PATCH v1] read_index_from(): Skip verification of the cache entry order to speed index loading

2017-10-18 Thread Ben Peart
1000 times 0.41(0.04+0.04) 0.50(0.00+0.10) +22.0% Signed-off-by: Ben Peart --- Notes: Base Ref: Web-Diff: https://github.com/benpeart/git/commit/54fa9cf954 Checkout: git fetch https://github.com/benpeart/git verify_ce_order-v1 && git checkout 54fa9cf954 builtin/fsc

Re: [PATCH v1] read_index_from(): Skip verification of the cache entry order to speed index loading

2017-10-19 Thread Ben Peart
On 10/19/2017 1:22 AM, Junio C Hamano wrote: Ben Peart writes: There is code in post_read_index_from() to catch out of order entries when reading an index file. This order verification is ~13% of the cost of every call to read_index_from(). I find this a bit over-generalized claim

Re: [PATCH 2/4] fsmonitor: Don't bother pretty-printing JSON from watchman

2017-10-20 Thread Ben Peart
On 10/20/2017 9:00 AM, Johannes Schindelin wrote: Hi Alex, On Thu, 19 Oct 2017, Alex Vandiver wrote: This provides small performance savings. diff --git a/t/t7519/fsmonitor-watchman b/t/t7519/fsmonitor-watchman index 377edc7be..eba46c78b 100755 --- a/t/t7519/fsmonitor-watchman +++ b/t/t7519

Re: [PATCH 3/4] fsmonitor: Document GIT_TRACE_FSMONITOR

2017-10-20 Thread Ben Peart
On 10/19/2017 9:11 PM, Alex Vandiver wrote: Signed-off-by: Alex Vandiver --- Documentation/git.txt | 4 1 file changed, 4 insertions(+) diff --git a/Documentation/git.txt b/Documentation/git.txt index 1fca63634..720db196e 100644 --- a/Documentation/git.txt +++ b/Documentation/git.txt

Re: [PATCH 4/4] fsmonitor: Delay updating state until after split index is merged

2017-10-20 Thread Ben Peart
On 10/20/2017 9:16 AM, Johannes Schindelin wrote: Hi Alex, On Thu, 19 Oct 2017, Alex Vandiver wrote: extern struct index_state the_index; diff --git a/fsmonitor.c b/fsmonitor.c index 7c1540c05..4c2668f57 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -49,20 +49,7 @@ int read_fsmonitor_exten

Re: [PATCH 4/4] fsmonitor: Delay updating state until after split index is merged

2017-10-23 Thread Ben Peart
On 10/23/2017 5:57 AM, Johannes Schindelin wrote: Hi Peff, On Fri, 20 Oct 2017, Jeff King wrote: On Fri, Oct 20, 2017 at 03:16:20PM +0200, Johannes Schindelin wrote: void tweak_fsmonitor(struct index_state *istate) { + int i; + + if (istate->fsmonitor_dirty) { +

[PATCH v2] read_index_from(): Skip verification of the cache entry order to speed index loading

2017-10-24 Thread Ben Peart
/discard_cache 1000 times 201.64(0.01+0.09) 217.89(0.03+0.07) +8.1% Signed-off-by: Ben Peart Signed-off-by: Johannes Schindelin --- Notes: Base Ref: master Web-Diff: https://github.com/benpeart/git/commit/95e20f17ff Checkout: git fetch https://github.com/benpeart/git no_ce_order-v2

Re: [PATCH v2 1/4] fsmonitor: Set the PWD to the top of the working tree

2017-10-26 Thread Ben Peart
On 10/25/2017 9:31 PM, Alex Vandiver wrote: The fsmonitor command inherits the PWD of its caller, which may be anywhere in the working copy. This makes is difficult for the fsmonitor command to operate on the whole repository. Specifically, for the watchman integration, this causes each subdi

Re: [PATCH v2 2/4] fsmonitor: Don't bother pretty-printing JSON from watchman

2017-10-26 Thread Ben Peart
On 10/25/2017 9:31 PM, Alex Vandiver wrote: This provides modest performance savings. Benchmarking with the following program, with and without `--no-pretty`, we find savings of 23% (0.316s -> 0.242s) in the git repository, and savings of 8% (5.24s -> 4.86s) on a large repository with 580k fil

Re: [PATCH v2 4/4] fsmonitor: Delay updating state until after split index is merged

2017-10-26 Thread Ben Peart
On 10/25/2017 9:31 PM, Alex Vandiver wrote: If the fsmonitor extension is used in conjunction with the split index extension, the set of entries in the index when it is first loaded is only a subset of the real index. This leads to only the non-"base" index being marked as CE_FSMONITOR_VALID.

Re: [PATCH v2 2/4] fsmonitor: Don't bother pretty-printing JSON from watchman

2017-10-26 Thread Ben Peart
On 10/26/2017 4:05 PM, Ben Peart wrote: On 10/25/2017 9:31 PM, Alex Vandiver wrote: This provides modest performance savings.  Benchmarking with the following program, with and without `--no-pretty`, we find savings of 23% (0.316s -> 0.242s) in the git repository, and savings of 8% (5.

Re: [PATCH v2 2/4] fsmonitor: Don't bother pretty-printing JSON from watchman

2017-10-26 Thread Ben Peart
On 10/26/2017 5:29 PM, Alex Vandiver wrote: On Thu, 26 Oct 2017, Ben Peart wrote: On 10/25/2017 9:31 PM, Alex Vandiver wrote: diff --git a/t/t7519/fsmonitor-watchman b/t/t7519/fsmonitor-watchman index a3e30bf54..79f24325c 100755 --- a/t/t7519/fsmonitor-watchman +++ b/t/t7519/fsmonitor

Re: [PATCH v2 1/4] fsmonitor: Set the PWD to the top of the working tree

2017-10-26 Thread Ben Peart
On 10/26/2017 5:27 PM, Alex Vandiver wrote: On Thu, 26 Oct 2017, Ben Peart wrote: On 10/25/2017 9:31 PM, Alex Vandiver wrote: diff --git a/fsmonitor.c b/fsmonitor.c index 7c1540c05..0d26ff34f 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -121,6 +121,7 @@ static int query_fsmonitor(int

Re: [PATCH v3] 0/4 fsmonitor fixes

2017-10-30 Thread Ben Peart
On 10/27/2017 7:26 PM, Alex Vandiver wrote: Updates since v2: - Fix tab which crept into 1/4 - Fixed the benchmarking code in the commit message in 2/4 to just always load JSON::XS -- the previous version was the version where I'd broken that to force loading of JSON::PP. - Rem

Re: What's cooking in git.git (Oct 2017, #07; Mon, 30)

2017-10-30 Thread Ben Peart
* av/fsmonitor (2017-10-30) 5 commits - SQUASH??? - fsmonitor: delay updating state until after split index is merged - fsmonitor: document GIT_TRACE_FSMONITOR - fsmonitor: don't bother pretty-printing JSON from watchman - fsmonitor: set the PWD to the top of the working tree (this br

Re: [PATCH v2] read_index_from(): Skip verification of the cache entry order to speed index loading

2017-10-30 Thread Ben Peart
things up. On 10/24/2017 10:45 AM, Ben Peart wrote: There is code in post_read_index_from() to detect out of order cache entries when reading an index file. This order verification adds cost to read_index_from() that grows with the size of the index. Put this on-disk data-structure validation

Re: [PATCH v2] read_index_from(): Skip verification of the cache entry order to speed index loading

2017-10-31 Thread Ben Peart
On 10/30/2017 9:49 PM, Junio C Hamano wrote: Ben Peart writes: Any updates or thoughts on this one? While the patch has become quite trivial, it does results in a savings of 5%-15% in index load time. I thought the compromise of having this test only run when DEBUG is defined should limit

Re: [PATCH v2] read_index_from(): Skip verification of the cache entry order to speed index loading

2017-10-31 Thread Ben Peart
On 10/30/2017 8:33 PM, Alex Vandiver wrote: On Mon, 30 Oct 2017, Jeff King wrote: On Mon, Oct 30, 2017 at 08:48:48AM -0400, Ben Peart wrote: Any updates or thoughts on this one? While the patch has become quite trivial, it does results in a savings of 5%-15% in index load time. I like

[RFC] fastindex: parallelize index load

2017-11-08 Thread Ben Peart
signature bytes SHA1 Signed-off-by: Ben Peart --- Notes: Base Ref: v2.14.3.windows.1 Web-Diff: https://github.com/benpeart/git/commit/1e818c7835 Checkout: git fetch https://github.com/benpeart/git fastindex-v1 && git checkout 1e818c7835 Makefile

Re: [RFC] fastindex: parallelize index load

2017-11-08 Thread Ben Peart
LSE Savings test-read-cache 500 100K6.398.33 23.36% test-read-cache 100 1M 12.49 18.68 33.12% On 11/8/2017 9:42 AM, Ben Peart wrote: This patch will address the CPU cost of loading the index by adding additional data to the index that will allow us to m

Re: [RFC] fastindex: parallelize index load

2017-11-08 Thread Ben Peart
LSE Savings test-read-cache 500 100K6.398.33 23.36% test-read-cache 100 1M 12.49 18.68 33.12% On 11/8/2017 9:42 AM, Ben Peart wrote: This patch will address the CPU cost of loading the index by adding additional data to the index that will allow us to m

[PATCH v1 0/4] Speed up index load through parallelization

2017-11-09 Thread Ben Peart
2-bit size 32-bit version 32-bit Cache Entry Offset 1 32-bit Cache Entry count 32-bit Cache Entry Offset 2 32-bit Cache Entry count . . . 32-bit version 32-bit size IEOT signature bytes SHA1 Signed-off-by: Ben Peart Base Ref: master Web-Diff: https://github.com/benpeart/git/commit/1146d38932 Chec

[PATCH v1 2/4] update-index: add fastindex support to update-index

2017-11-09 Thread Ben Peart
Add support in update-index to manually add/remove the fastindex extension via --[no-]fastindex flags. Signed-off-by: Ben Peart --- builtin/update-index.c | 22 ++ 1 file changed, 22 insertions(+) diff --git a/builtin/update-index.c b/builtin/update-index.c index fefbe60167

[PATCH v1 3/4] fastindex: add test tools and a test script

2017-11-09 Thread Ben Peart
extension return the same results with both V2 and V4 indexes. Signed-off-by: Ben Peart --- Makefile| 2 + t/helper/test-dump-fast-index.c | 68 + t/helper/test-fast-index.c | 84 + t/t1800-fast

[PATCH v1 1/4] fastindex: speed up index load through parallelization

2017-11-09 Thread Ben Peart
extensions can be loaded and processed normally (skipping the IEOT entry as it has already been processed). If the IEOT extension is not available then parsing the index will proceed as usual with a single thread. Signed-off-by: Ben Peart --- cache.h | 25 + config.c | 20

[PATCH v1 4/4] fastindex: add documentation for the fastindex extension

2017-11-09 Thread Ben Peart
This includes the core.fastindex setting, the update-index additions, and the fastindex index extension. Signed-off-by: Ben Peart --- Documentation/config.txt | 8 Documentation/git-update-index.txt | 11 +++ Documentation/technical/index-format.txt | 26

[PATCH v1] fsmonitor: simplify determining the git worktree under Windows

2017-11-10 Thread Ben Peart
ssary. It also spawns multiple processes (uname and cygpath) which slows things down. Simplify and speed up the process of finding the git worktree when running on Windows by keeping it in perl and avoiding spawning helper processes. Signed-off-by: Ben Peart Signed-off-by: Johannes Schin

Re: [PATCH 2/2] fsmonitor: Store fsmonitor bitmap before splitting index

2017-11-13 Thread Ben Peart
On 11/9/2017 2:58 PM, Alex Vandiver wrote: ba1b9caca6 resolved the problem of the fsmonitor data being applied to the non-base index when reading; however, a similar problem exists when writing the index. Specifically, writing of the fsmonitor extension happens only after the work to split the

Re: [PATCH v1 1/4] fastindex: speed up index load through parallelization

2017-11-13 Thread Ben Peart
On 11/9/2017 11:46 PM, Junio C Hamano wrote: Ben Peart writes: To make this work for V4 indexes, when writing the index, it periodically "resets" the prefix-compression by encoding the current entry as if the path name for the previous entry is completely different and saves the

Re: [PATCH v1 1/4] fastindex: speed up index load through parallelization

2017-11-14 Thread Ben Peart
On 11/13/2017 8:10 PM, Junio C Hamano wrote: Ben Peart writes: The proposed format for extensions (ie having both a header and a footer with name and size) in the current patch already enables having multiple extensions that can be parsed forward or backward. Any extensions that would want

Re: [PATCH v1 1/4] fastindex: speed up index load through parallelization

2017-11-14 Thread Ben Peart
On 11/14/2017 10:04 AM, Junio C Hamano wrote: Ben Peart writes: How about I add the logic to write out a special extension right before the SHA1 that contains an offset to the beginning of the extensions section. I will also add the logic in do_read_index() to search for and load this

Re: [PATCH v1 1/4] fastindex: speed up index load through parallelization

2017-11-14 Thread Ben Peart
On 11/14/2017 8:12 PM, Junio C Hamano wrote: Ben Peart writes: I have no thoughts or plans for changes in the future of IEOT (which I plan to rename ITOC). At this point in time, I can't even imagine what else we'd want as the index only contains cache entries, ... Yeah, but th

<    2   3   4   5   6   7