[PATCH v2 1/2] fsexcludes: add a programmatic way to exclude files from git's working directory traversal logic

logic is also checked as it could further reduce the set of files that should be included. Signed-off-by: Ben Peart <benpe...@microsoft.com> --- Makefile | 1 + dir.c| 23 +- fsexcludes.c | 211 +++ fsexcludes.h | 29

Re: [PATCH v1 1/2] fsexcludes: add a programmatic way to exclude files from git's working directory traversal logic

On 4/10/2018 6:09 PM, Martin Ågren wrote: On 10 April 2018 at 23:04, Ben Peart <ben.pe...@microsoft.com> wrote: The File System Excludes module is a new programmatic way to exclude files and folders from git's traversal of the working directory. fsexcludes_init() should be

[PATCH v1 1/2] fsexcludes: add a programmatic way to exclude files from git's working directory traversal logic

logic is also checked as it could further reduce the set of files that should be included. Signed-off-by: Ben Peart <benpe...@microsoft.com> --- Makefile | 1 + fsexcludes.c | 210 +++ fsexcludes.h | 27 +++ 3 files change

[PATCH v1 0/2] fsexcludes: Add programmatic way to exclude files

to these programmatic applications. Base Ref: master Web-Diff: https://github.com/benpeart/git/commit/2ccbcd6360 Checkout: git fetch https://github.com/benpeart/git fsexcludes-v1 && git checkout 2ccbcd6360 Ben Peart (2): fsexcludes: add a programmatic way to exclude files from git's

[PATCH v1 2/2] fsmonitor: switch to use new fsexcludes logic and remove unused untracked cache based logic

Update fsmonitor to utilize the new fsexcludes based logic for excluding paths that do not need to be scaned for new or modified files. Remove the old logic in dir.c that utilized the untracked cache (if enabled) to accomplish the same goal. Signed-off-by: Ben Peart <benpe...@microsoft.

[PATCH v1] fsmonitor: fix incorrect buffer size when printing version number

This is a trivial bug fix for passing the incorrect size to snprintf() when outputing the version. It should be passing the size of the destination buffer rather than the size of the value being printed. Signed-off-by: Ben Peart <benpe...@microsoft.com> --- Notes: Base Ref: v2.17.0

[PATCH v1] fsmonitor: force index write after full scan

if we end up having to stat() all files and scan the entire working directory. The assumption being that must be expensive or you would not have turned on the feature. Signed-off-by: Ben Peart <benpe...@microsoft.com> --- Notes: Base Ref: Web-Diff: https://github.com/benpeart/git/

Re: core.fsmonitor always perform rescans

On 3/26/2018 12:41 AM, Tatsuyuki Ishi wrote: Hello, I'm facing issue with core.fsmonitor. I'm currently using the provided watchman hook, but it doesn't seem to record the fact that it has queried the fsmonitor backend, and as a result the timestamp passed to the hook doesn't seem to change.

Re: [PATCH v1] Fix bugs preventing adding updated cache entries to the name hash

On 3/15/2018 1:58 PM, Junio C Hamano wrote: Ben Peart <benpe...@microsoft.com> writes: Update replace_index_entry() to clear the CE_HASHED flag from the new cache entry so that it can add it to the name hash in set_index_entry() OK. diff --git a/read-cache.c b/read-cache.c

[PATCH v1] Fix bugs preventing adding updated cache entries to the name hash

. Signed-off-by: Ben Peart <benpe...@microsoft.com> --- Notes: Base Ref: master Web-Diff: https://github.com/benpeart/git/commit/079be4ac07 Checkout: git fetch https://github.com/benpeart/git ce_hashed-v1 && git checkout 079be4ac07 read-cache.c | 4 +++- 1 file changed, 3 in

Re: [PATCH v1] dir.c: don't flag the index as dirty for changes to the untracked cache

On 3/1/2018 2:42 AM, Jeff King wrote: On Wed, Feb 28, 2018 at 01:27:03PM -0800, Junio C Hamano wrote: Somehow this topic has been hanging without getting further attention for too long. It's time to wrap it up and moving it forward. How about this? -- >8 -- From: Junio C Hamano

[PATCH v1] fsmonitor: update documentation to remove reference to invalid config settings

Remove the reference to setting core.fsmonitor to `true` (or `false`) as those are not valid settings. Signed-off-by: Ben Peart <benpe...@microsoft.com> --- Notes: Base Ref: master Web-Diff: https://github.com/benpeart/git/commit/4b7ec2c11e Checkout: git fetch https://gith

Re: [PATCH v1] dir.c: don't flag the index as dirty for changes to the untracked cache

On 2/12/2018 5:20 AM, Duy Nguyen wrote: On Wed, Feb 7, 2018 at 9:13 PM, Ben Peart <peart...@gmail.com> wrote: On 2/6/2018 7:27 AM, Duy Nguyen wrote: This is another thing that bugs me. I know you're talking about huge index files, but at what size should we start this sort of optimi

[PATCH v2] name-hash: properly fold directory names in adjust_dirname_case()

ent memcpy that would fold the name to the version with the correct case never executed. Add a test to validate the corrected behavior with name folding of directories. Signed-off-by: Ben Peart <benpe...@microsoft.com> --- Notes: Base Ref: v2.16.1 Web-Diff: https://github.com/ben

Re: [PATCH v1] name-hash: properly fold directory names in adjust_dirname_case()

On 2/8/2018 12:21 PM, Torsten Bögershausen wrote: On Wed, Feb 07, 2018 at 07:41:56PM -0500, Ben Peart wrote: [] diff --git a/t/t0050-filesystem.sh b/t/t0050-filesystem.sh index b29d749bb7..219c96594c 100755 --- a/t/t0050-filesystem.sh +++ b/t/t0050-filesystem.sh @@ -80,7 +80,17

[PATCH v1] name-hash: properly fold directory names in adjust_dirname_case()

ent memcpy that would fold the name to the version with the correct case never executed. Add a test to validate the corrected behavior with name folding of directories. Signed-off-by: Ben Peart <benpe...@microsoft.com> --- Notes: Base Ref: v2.16.1 Web-Diff: https://github.com/ben

Re: [PATCH v2] dir.c: ignore paths containing .git when invalidating untracked cache

ty (because UNTR extension is updated). Depending on the index size, writing it down could also be slow. Noticed-by: Ævar Arnfjörð Bjarmason <ava...@gmail.com> Signed-off-by: Nguyễn Thái Ngọc Duy <pclo...@gmail.com> Signed-off-by: Ben Peart <benpe...@microsoft.com> --- Notes:

Re: [PATCH v1] dir.c: don't flag the index as dirty for changes to the untracked cache

On 2/6/2018 7:27 AM, Duy Nguyen wrote: This is another thing that bugs me. I know you're talking about huge index files, but at what size should we start this sort of optimization? Writing down a few MBs on linux is cheap enough that I won't bother optimizing (and I get my UNTR extension

Re: [PATCH v1] dir.c: don't flag the index as dirty for changes to the untracked cache

On 2/7/2018 5:59 AM, Duy Nguyen wrote: On Tue, Feb 6, 2018 at 7:55 PM, Duy Nguyen <pclo...@gmail.com> wrote: On Tue, Feb 6, 2018 at 7:27 PM, Duy Nguyen <pclo...@gmail.com> wrote: On Tue, Feb 6, 2018 at 8:48 AM, Ben Peart <peart...@gmail.com> wrote: With the new behavio

Re: [PATCH v1] dir.c: don't flag the index as dirty for changes to the untracked cache

On 2/5/2018 4:58 PM, Brandon Williams wrote: On 02/05, Ben Peart wrote: The untracked cache saves its current state in the UNTR index extension. Currently, _any_ change to that state causes the index to be flagged as dirty and written out to disk. Unfortunately, the cost to write out

[PATCH v1] dir.c: don't flag the index as dirty for changes to the untracked cache

such as a status followed by add, commit, etc). After this patch, all the logic to track statistics for the untracked cache could be removed as it is only used by debug tracing used to debug the untracked cache. Signed-off-by: Ben Peart <benpe...@microsoft.com> --- Notes: Base Ref: master We

Re: [PATCH] dir.c: ignore paths containing .git when invalidating untracked cache

On 2/4/2018 4:38 AM, Nguyễn Thái Ngọc Duy wrote: read_directory() code ignores all paths named ".git" even if it's not a valid git repository. See treat_path() for details. Since ".git" is basically invisible to read_directory(), when we are asked to invalidate a path that contains ".git", we

Re: Some rough edges of core.fsmonitor

On 1/30/2018 6:16 PM, Ævar Arnfjörð Bjarmason wrote: On Tue, Jan 30 2018, Ben Peart jotted: While some of these issues have been discussed in other threads, I thought I'd summarize my thoughts here. Thanks for this & your other reply. I'm going to get to testing some of Duy's patches

Re: Some rough edges of core.fsmonitor

While some of these issues have been discussed in other threads, I thought I'd summarize my thoughts here. On 1/26/2018 7:28 PM, Ævar Arnfjörð Bjarmason wrote: I just got around to testing this since it landed, for context some previous poking of mine in [1]. Issues / stuff I've noticed: 1)

Re: Some rough edges of core.fsmonitor

On 1/27/2018 2:01 PM, Ævar Arnfjörð Bjarmason wrote: On Sat, Jan 27 2018, Duy Nguyen jotted: On Sat, Jan 27, 2018 at 07:39:27PM +0700, Duy Nguyen wrote: On Sat, Jan 27, 2018 at 6:43 PM, Ævar Arnfjörð Bjarmason wrote: a) no fsmonitor $ time GIT_TRACE_PERFORMANCE=1

Re: Some rough edges of core.fsmonitor

On 1/28/2018 5:28 PM, Ævar Arnfjörð Bjarmason wrote: On Sun, Jan 28, 2018 at 9:44 PM, Johannes Schindelin wrote: Hi, On Sat, 27 Jan 2018, Ævar Arnfjörð Bjarmason wrote: I just got around to testing this since it landed, for context some previous poking of mine

Re: Some rough edges of core.fsmonitor

On 1/29/2018 4:40 AM, Duy Nguyen wrote: On Sat, Jan 27, 2018 at 12:43:41PM +0100, Ævar Arnfjörð Bjarmason wrote: b) with fsmonitor $ time GIT_TRACE_PERFORMANCE=1 ~/g/git/git-status 12:34:23.833625 read-cache.c:1890 performance: 0.049485685 s: read cache .git/index This is

RE: [PATCH] RelNotes: fsmonitor: add a pointer to man page and the word itself

> -Original Message- > From: Yasushi SHOJI [mailto:ya...@atmark-techno.com] > Sent: Wednesday, January 17, 2018 12:09 AM > To: Ben Peart <ben.pe...@microsoft.com>; gits...@pobox.com > Cc: git@vger.kernel.org > Subject: [PATCH] RelNotes: fsmonitor: add a pointer to

Re: [PATCH 6/6] fsmonitor: Use fsmonitor data in `git diff`

On 1/5/2018 5:22 PM, Junio C Hamano wrote: Johannes Schindelin writes: diff --git a/diff-lib.c b/diff-lib.c index 8104603a3..13ff00d81 100644 --- a/diff-lib.c +++ b/diff-lib.c @@ -95,6 +95,9 @@ int run_diff_files(struct rev_info *revs, unsigned int option)

Re: [PATCH 4/6] fsmonitor: Make output of test-dump-fsmonitor more concise

On 1/4/2018 5:33 PM, Johannes Schindelin wrote: Hi Alex, On Tue, 2 Jan 2018, Alex Vandiver wrote: Rather than display one very long line, summarize the contents of that line. The tests do not currently rely on any content except the first line ("no fsmonitor" / "fsmonitor last update").

Re: [PATCH 2/6] fsmonitor: Stop inline'ing mark_fsmonitor_valid / _invalid

On 1/4/2018 5:27 PM, Johannes Schindelin wrote: Hi Alex, On Tue, 2 Jan 2018, Alex Vandiver wrote: These were inline'd when they were first introduced, presumably as an optimization for cases when they were called in tight loops. This complicates using these functions, as

Re: [PATCH v4 0/4] Add --no-ahead-behind to status

On 1/8/2018 10:48 AM, Jeff Hostetler wrote: From: Jeff Hostetler This is version 4 of my patch series to avoid expensive ahead/behind calculations in status. This version removes the last commit containing the experimental config setting. And removes the undefined

RE: git-p4 + watchman - watching the p4 repo?

Ben > -Original Message- > From: Luke Diamand [mailto:l...@diamand.org] > Sent: Monday, January 8, 2018 12:15 PM > To: Git Users <git@vger.kernel.org> > Cc: Alex Vandiver <ale...@dropbox.com>; Ben Peart > <ben.pe...@microsoft.com> > Subject: git-p4 + w

RE: [PATCH] p7519: improve check for prerequisite WATCHMAN

> -Original Message- > From: René Scharfe [mailto:l@web.de] > Sent: Saturday, December 16, 2017 7:12 AM > To: Git List <git@vger.kernel.org> > Cc: Junio C Hamano <gits...@pobox.com>; Ben Peart > <ben.pe...@microsoft.com>; Ævar Arnfjörð Bjarmason &g

Re: [PATCH v1 1/4] fastindex: speed up index load through parallelization

On 11/20/2017 6:51 PM, Ramsay Jones wrote: On 20/11/17 14:01, Ben Peart wrote: Further testing has revealed that switching from the regular heap to a refactored version of the mem_pool in fast-import.c produces similar gains as parallelizing do_index_load().  This appears to be a much

Re: [PATCH v1 1/4] fastindex: speed up index load through parallelization

Further testing has revealed that switching from the regular heap to a refactored version of the mem_pool in fast-import.c produces similar gains as parallelizing do_index_load(). This appears to be a much simpler patch for similar gains so we will be pursuing that path. Combining the two

RE: What's cooking in git.git (Nov 2017, #05; Fri, 17)

> -Original Message- > From: Junio C Hamano [mailto:gits...@pobox.com] > Sent: Friday, November 17, 2017 1:35 AM > To: Ben Peart <ben.pe...@microsoft.com>; Alex Vandiver > <ale...@dropbox.com> > Cc: git@vger.kernel.org > Subject: Re: What's cooking i

Re: [PATCH v1 1/4] fastindex: speed up index load through parallelization

On 11/14/2017 8:12 PM, Junio C Hamano wrote: Ben Peart <peart...@gmail.com> writes: I have no thoughts or plans for changes in the future of IEOT (which I plan to rename ITOC). At this point in time, I can't even imagine what else we'd want as the index only contains cache e

Re: [PATCH v1 1/4] fastindex: speed up index load through parallelization

On 11/14/2017 10:04 AM, Junio C Hamano wrote: Ben Peart <peart...@gmail.com> writes: How about I add the logic to write out a special extension right before the SHA1 that contains an offset to the beginning of the extensions section. I will also add the logic in do_read_index() to

Re: [PATCH v1 1/4] fastindex: speed up index load through parallelization

On 11/13/2017 8:10 PM, Junio C Hamano wrote: Ben Peart <peart...@gmail.com> writes: The proposed format for extensions (ie having both a header and a footer with name and size) in the current patch already enables having multiple extensions that can be parsed forward or backward

Re: [PATCH v1 1/4] fastindex: speed up index load through parallelization

On 11/9/2017 11:46 PM, Junio C Hamano wrote: Ben Peart <benpe...@microsoft.com> writes: To make this work for V4 indexes, when writing the index, it periodically "resets" the prefix-compression by encoding the current entry as if the path name for the previous entry is comp

Re: [PATCH 2/2] fsmonitor: Store fsmonitor bitmap before splitting index

On 11/9/2017 2:58 PM, Alex Vandiver wrote: ba1b9caca6 resolved the problem of the fsmonitor data being applied to the non-base index when reading; however, a similar problem exists when writing the index. Specifically, writing of the fsmonitor extension happens only after the work to split

[PATCH v1] fsmonitor: simplify determining the git worktree under Windows

. It also spawns multiple processes (uname and cygpath) which slows things down. Simplify and speed up the process of finding the git worktree when running on Windows by keeping it in perl and avoiding spawning helper processes. Signed-off-by: Ben Peart <benpe...@microsoft.com> Sign

[PATCH v1 4/4] fastindex: add documentation for the fastindex extension

This includes the core.fastindex setting, the update-index additions, and the fastindex index extension. Signed-off-by: Ben Peart <benpe...@microsoft.com> --- Documentation/config.txt | 8 Documentation/git-update-index.txt | 11 +++ Documen

[PATCH v1 1/4] fastindex: speed up index load through parallelization

extensions can be loaded and processed normally (skipping the IEOT entry as it has already been processed). If the IEOT extension is not available then parsing the index will proceed as usual with a single thread. Signed-off-by: Ben Peart <benpe...@microsoft.com> --- cache.h | 25

[PATCH v1 2/4] update-index: add fastindex support to update-index

Add support in update-index to manually add/remove the fastindex extension via --[no-]fastindex flags. Signed-off-by: Ben Peart <benpe...@microsoft.com> --- builtin/update-index.c | 22 ++ 1 file changed, 22 insertions(+) diff --git a/builtin/update-index.c b/builtin/

[PATCH v1 3/4] fastindex: add test tools and a test script

extension return the same results with both V2 and V4 indexes. Signed-off-by: Ben Peart <benpe...@microsoft.com> --- Makefile| 2 + t/helper/test-dump-fast-index.c | 68 + t/helper/test-fast-index.c

[PATCH v1 0/4] Speed up index load through parallelization

rsion 32-bit Cache Entry Offset 1 32-bit Cache Entry count 32-bit Cache Entry Offset 2 32-bit Cache Entry count . . . 32-bit version 32-bit size IEOT signature bytes SHA1 Signed-off-by: Ben Peart <benpe...@microsoft.com> Base Ref: master Web-Diff: https://github.com/benpeart/git/commit/114

Re: [RFC] fastindex: parallelize index load

test-read-cache 500 100K6.398.33 23.36% test-read-cache 100 1M 12.49 18.68 33.12% On 11/8/2017 9:42 AM, Ben Peart wrote: This patch will address the CPU cost of loading the index by adding additional data to the index that will allow us to multi-thread

Re: [RFC] fastindex: parallelize index load

test-read-cache 500 100K6.398.33 23.36% test-read-cache 100 1M 12.49 18.68 33.12% On 11/8/2017 9:42 AM, Ben Peart wrote: This patch will address the CPU cost of loading the index by adding additional data to the index that will allow us to multi-thread

[RFC] fastindex: parallelize index load

signature bytes SHA1 Signed-off-by: Ben Peart <benpe...@microsoft.com> --- Notes: Base Ref: v2.14.3.windows.1 Web-Diff: https://github.com/benpeart/git/commit/1e818c7835 Checkout: git fetch https://github.com/benpeart/git fastindex-v1 && git checko

Re: [PATCH v2] read_index_from(): Skip verification of the cache entry order to speed index loading

On 10/30/2017 8:33 PM, Alex Vandiver wrote: On Mon, 30 Oct 2017, Jeff King wrote: On Mon, Oct 30, 2017 at 08:48:48AM -0400, Ben Peart wrote: Any updates or thoughts on this one? While the patch has become quite trivial, it does results in a savings of 5%-15% in index load time. I like

Re: [PATCH v2] read_index_from(): Skip verification of the cache entry order to speed index loading

On 10/30/2017 9:49 PM, Junio C Hamano wrote: Ben Peart <peart...@gmail.com> writes: Any updates or thoughts on this one? While the patch has become quite trivial, it does results in a savings of 5%-15% in index load time. I thought the compromise of having this test only run when

Re: [PATCH v2] read_index_from(): Skip verification of the cache entry order to speed index loading

things up. On 10/24/2017 10:45 AM, Ben Peart wrote: There is code in post_read_index_from() to detect out of order cache entries when reading an index file. This order verification adds cost to read_index_from() that grows with the size of the index. Put this on-disk data-structure validation

Re: What's cooking in git.git (Oct 2017, #07; Mon, 30)

* av/fsmonitor (2017-10-30) 5 commits - SQUASH??? - fsmonitor: delay updating state until after split index is merged - fsmonitor: document GIT_TRACE_FSMONITOR - fsmonitor: don't bother pretty-printing JSON from watchman - fsmonitor: set the PWD to the top of the working tree (this

Re: [PATCH v3] 0/4 fsmonitor fixes

On 10/27/2017 7:26 PM, Alex Vandiver wrote: Updates since v2: - Fix tab which crept into 1/4 - Fixed the benchmarking code in the commit message in 2/4 to just always load JSON::XS -- the previous version was the version where I'd broken that to force loading of JSON::PP. -

Re: [PATCH v2 1/4] fsmonitor: Set the PWD to the top of the working tree

On 10/26/2017 5:27 PM, Alex Vandiver wrote: On Thu, 26 Oct 2017, Ben Peart wrote: On 10/25/2017 9:31 PM, Alex Vandiver wrote: diff --git a/fsmonitor.c b/fsmonitor.c index 7c1540c05..0d26ff34f 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -121,6 +121,7 @@ static int query_fsmonitor(int

Re: [PATCH v2 2/4] fsmonitor: Don't bother pretty-printing JSON from watchman

On 10/26/2017 5:29 PM, Alex Vandiver wrote: On Thu, 26 Oct 2017, Ben Peart wrote: On 10/25/2017 9:31 PM, Alex Vandiver wrote: diff --git a/t/t7519/fsmonitor-watchman b/t/t7519/fsmonitor-watchman index a3e30bf54..79f24325c 100755 --- a/t/t7519/fsmonitor-watchman +++ b/t/t7519/fsmonitor

Re: [PATCH v2 2/4] fsmonitor: Don't bother pretty-printing JSON from watchman

On 10/26/2017 4:05 PM, Ben Peart wrote: On 10/25/2017 9:31 PM, Alex Vandiver wrote: This provides modest performance savings.  Benchmarking with the following program, with and without `--no-pretty`, we find savings of 23% (0.316s -> 0.242s) in the git repository, and savings of 8% (5.

Re: [PATCH v2 4/4] fsmonitor: Delay updating state until after split index is merged

On 10/25/2017 9:31 PM, Alex Vandiver wrote: If the fsmonitor extension is used in conjunction with the split index extension, the set of entries in the index when it is first loaded is only a subset of the real index. This leads to only the non-"base" index being marked as CE_FSMONITOR_VALID.

Re: [PATCH v2 2/4] fsmonitor: Don't bother pretty-printing JSON from watchman

On 10/25/2017 9:31 PM, Alex Vandiver wrote: This provides modest performance savings. Benchmarking with the following program, with and without `--no-pretty`, we find savings of 23% (0.316s -> 0.242s) in the git repository, and savings of 8% (5.24s -> 4.86s) on a large repository with 580k

Re: [PATCH v2 1/4] fsmonitor: Set the PWD to the top of the working tree

On 10/25/2017 9:31 PM, Alex Vandiver wrote: The fsmonitor command inherits the PWD of its caller, which may be anywhere in the working copy. This makes is difficult for the fsmonitor command to operate on the whole repository. Specifically, for the watchman integration, this causes each

[PATCH v2] read_index_from(): Skip verification of the cache entry order to speed index loading

/discard_cache 1000 times 201.64(0.01+0.09) 217.89(0.03+0.07) +8.1% Signed-off-by: Ben Peart <benpe...@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schinde...@gmx.de> --- Notes: Base Ref: master Web-Diff: https://github.com/benpeart/git/commit/95e20f17ff C

Re: [PATCH 4/4] fsmonitor: Delay updating state until after split index is merged

On 10/23/2017 5:57 AM, Johannes Schindelin wrote: Hi Peff, On Fri, 20 Oct 2017, Jeff King wrote: On Fri, Oct 20, 2017 at 03:16:20PM +0200, Johannes Schindelin wrote: void tweak_fsmonitor(struct index_state *istate) { + int i; + + if (istate->fsmonitor_dirty) { +

Re: [PATCH 4/4] fsmonitor: Delay updating state until after split index is merged

On 10/20/2017 9:16 AM, Johannes Schindelin wrote: Hi Alex, On Thu, 19 Oct 2017, Alex Vandiver wrote: extern struct index_state the_index; diff --git a/fsmonitor.c b/fsmonitor.c index 7c1540c05..4c2668f57 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -49,20 +49,7 @@ int

Re: [PATCH 3/4] fsmonitor: Document GIT_TRACE_FSMONITOR

On 10/19/2017 9:11 PM, Alex Vandiver wrote: Signed-off-by: Alex Vandiver --- Documentation/git.txt | 4 1 file changed, 4 insertions(+) diff --git a/Documentation/git.txt b/Documentation/git.txt index 1fca63634..720db196e 100644 --- a/Documentation/git.txt +++

Re: [PATCH 2/4] fsmonitor: Don't bother pretty-printing JSON from watchman

On 10/20/2017 9:00 AM, Johannes Schindelin wrote: Hi Alex, On Thu, 19 Oct 2017, Alex Vandiver wrote: This provides small performance savings. diff --git a/t/t7519/fsmonitor-watchman b/t/t7519/fsmonitor-watchman index 377edc7be..eba46c78b 100755 --- a/t/t7519/fsmonitor-watchman +++

Re: [PATCH v1] read_index_from(): Skip verification of the cache entry order to speed index loading

On 10/19/2017 1:22 AM, Junio C Hamano wrote: Ben Peart <benpe...@microsoft.com> writes: There is code in post_read_index_from() to catch out of order entries when reading an index file. This order verification is ~13% of the cost of every call to read_index_from(). I find this a bi

[PATCH v1] read_index_from(): Skip verification of the cache entry order to speed index loading

1000 times 0.41(0.04+0.04) 0.50(0.00+0.10) +22.0% Signed-off-by: Ben Peart <benpe...@microsoft.com> --- Notes: Base Ref: Web-Diff: https://github.com/benpeart/git/commit/54fa9cf954 Checkout: git fetch https://github.com/benpeart/git verify_ce_order-v1 && git checkout 54f

Re: [PATCH v8 00/12] Fast git status via a file system watcher

that is within my perl capabilities and is fixed with the following patch: -- >8 -- From 3e3b983a4208a62d166c233a7de3bf045322f6c7 Mon Sep 17 00:00:00 2001 From: Ben Peart <benpe...@microsoft.com> Date: Wed, 4 Oct 2017 08:33:39 -0400 Subject: [PATCH] fsmonitor: preserve utf8 filenames

Re: [PATCH v8 00/12] Fast git status via a file system watcher

On 10/3/2017 10:09 PM, Junio C Hamano wrote: Ben Peart <peart...@gmail.com> writes: Well, rats. I found one more issue that applies to two of the commits. Can you squash this in as well or do you want it in some other form? Rats indeed. Let's go incremental as promised, perhap

Re: [PATCH v8 00/12] Fast git status via a file system watcher

On 10/1/2017 4:24 AM, Junio C Hamano wrote: Ben Peart <ben.pe...@microsoft.com> writes: I had accumulated the same set of changes with one addition of removing a duplicate "the" from a comment in the fsmonitor.h file: diff --git a/fsmonitor.h b/fsmonitor.h index 8eb616

Re: [PATCH v6 09/40] Add initial external odb support

On 9/29/2017 4:36 PM, Jonathan Tan wrote: On Wed, 27 Sep 2017 18:46:30 +0200 Christian Couder wrote: I am ok to split the patch series, but I am not sure that 01/40 to 09/40 is the right range for the first patch series. I would say that 01/40 to 07/40 is better

Re: [PATCH v6 00/40] Add initial experimental external ODB support

ith tests. - Patch 40/40 adds documentation about transfering objects and metadata when using the external odb mechanism. This patch is new since v5. Future work ~~~ There are still things that could be cleaned or improved. I think I may work on: - Integra

RE: [PATCH v8 00/12] Fast git status via a file system watcher

> -Original Message- > From: Junio C Hamano [mailto:gits...@pobox.com] > Sent: Thursday, September 28, 2017 10:21 PM > To: Ben Peart <ben.pe...@microsoft.com> > Cc: david.tur...@twosigma.com; ava...@gmail.com; > christian.cou...@gmail.com; git@vger.kernel.org; >

RE: [PATCH v8 08/12] fsmonitor: add a test tool to dump the index extension

> -Original Message- > From: Martin Ågren [mailto:martin.ag...@gmail.com] > Sent: Friday, September 22, 2017 7:37 PM > To: Ben Peart <ben.pe...@microsoft.com> > Cc: David Turner <david.tur...@twosigma.com>; Ævar Arnfjörð Bjarmason > <ava...@gmail.com&

RE: [PATCH v8 01/12] bswap: add 64 bit endianness helper get_be64

Thanks, Ben > -Original Message- > From: Martin Ågren [mailto:martin.ag...@gmail.com] > Sent: Friday, September 22, 2017 7:37 PM > To: Ben Peart <ben.pe...@microsoft.com> > Cc: David Turner <david.tur...@twosigma.com>; Ævar Arnfjörð Bjarmason > <av

[PATCH v8 07/12] update-index: add fsmonitor support to update-index

Add support in update-index to manually add/remove the fsmonitor extension via --[no-]fsmonitor flags. Add support in update-index to manually set/clear the fsmonitor valid bit via --[no-]fsmonitor-valid flags. Signed-off-by: Ben Peart <benpe...@microsoft.com> --- builtin/update-index.

[PATCH v8 08/12] fsmonitor: add a test tool to dump the index extension

Add a test utility (test-dump-fsmonitor) that will dump the fsmonitor index extension. Signed-off-by: Ben Peart <benpe...@microsoft.com> --- Makefile | 1 + t/helper/test-dump-fsmonitor.c | 21 + 2 files changed, 22 insertions(+) create mode 10

[PATCH v8 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.

; when git updates a cache entry to match the current state on disk, it will now set the CE_FSMONITOR_VALID bit. Inversely, anytime git changes a cache entry, the CE_FSMONITOR_VALID bit is cleared and the corresponding untracked cache directory is marked invalid. Signed-off-by: Ben Peart <be

[PATCH v8 11/12] fsmonitor: add a sample integration script for Watchman

. Configure git to use the extension: git config core.fsmonitor .git/hooks/fsmonitor-watchman Optionally turn on the untracked cache for optimal performance. Signed-off-by: Ben Peart <benpe...@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schinde...@gmx.de> Signed-

[PATCH v8 09/12] split-index: disable the fsmonitor extension when running the split index test

which causes the test to fail. The potential matrix of index extensions and index versions can is quite large so instead temporarily disable the extension before attempting to run the test until the underlying problem of hard coded SHA values is fixed. Signed-off-by: Ben Peart <benpe...@microsoft.

[PATCH v8 06/12] ls-files: Add support in ls-files to display the fsmonitor valid bit

Add a new command line option (-f) to ls-files to have it use lowercase letters for 'fsmonitor valid' files Signed-off-by: Ben Peart <benpe...@microsoft.com> --- builtin/ls-files.c | 8 ++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/builtin/ls-files.c b/builtin/ls-f

[PATCH v8 10/12] fsmonitor: add test cases for fsmonitor extension

where it holds on to handles for directories and files which prevents the test directory from being cleaned up properly. Signed-off-by: Ben Peart <benpe...@microsoft.com> --- t/t7519-status-fsmonitor.sh | 304 t/t7519/fsmonitor-all | 24 +

[PATCH v8 05/12] fsmonitor: add documentation for the fsmonitor extension.

This includes the core.fsmonitor setting, the fsmonitor integration hook, and the fsmonitor index extension. Also add documentation for the new fsmonitor options to ls-files and update-index. Signed-off-by: Ben Peart <benpe...@microsoft.com> --- Documentation/config.txt

[PATCH v8 12/12] fsmonitor: add a performance test

. Signed-off-by: Ben Peart <benpe...@microsoft.com> Signed-off-by: Ævar Arnfjörð Bjarmason <ava...@gmail.com> --- Makefile| 1 + t/helper/.gitignore | 1 + t/helper/test-drop-caches.c | 164 +++ t/perf/p7519-fsmonito

[PATCH v8 03/12] update-index: add a new --force-write-index option

-index) to update-index that will ensure the index is written out even if the cache_changed flag is not set. Signed-off-by: Ben Peart <benpe...@microsoft.com> --- builtin/update-index.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/builtin/update-index.c b/builtin/

[PATCH v8 00/12] Fast git status via a file system watcher

if (lstat(old->name, ) || - ie_match_stat(o->src_index, old, , CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR)) + ie_match_stat(o->src_index, old, , CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORK

[PATCH v8 01/12] bswap: add 64 bit endianness helper get_be64

Add a new get_be64 macro to enable 64 bit endian conversions on memory that may or may not be aligned. Signed-off-by: Ben Peart <benpe...@microsoft.com> --- compat/bswap.h | 22 ++ 1 file changed, 22 insertions(+) diff --git a/compat/bswap.h b/compat/bswap.h index 7d06

[PATCH v8 02/12] preload-index: add override to enable testing preload-index

existing tests and have the core.preloadindex code path execute as long as the test has at least 2 files by setting GIT_FORCE_PRELOAD_TEXT=1 before running the test. Signed-off-by: Ben Peart <benpe...@microsoft.com> --- preload-index.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/p

Re: [PATCH v7 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.

On 9/20/2017 10:24 PM, Ben Peart wrote: On 9/20/2017 10:00 PM, Junio C Hamano wrote: Ben Peart <peart...@gmail.com> writes: Pretty much the same places you would also use CE_MATCH_IGNORE_VALID and CE_MATCH_IGNORE_SKIP_WORKTREE which serve the same role for those fe

Re: [PATCH v7 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.

On 9/20/2017 10:00 PM, Junio C Hamano wrote: Ben Peart <peart...@gmail.com> writes: Pretty much the same places you would also use CE_MATCH_IGNORE_VALID and CE_MATCH_IGNORE_SKIP_WORKTREE which serve the same role for those features. That is generally when you are about to overwrite d

Re: [PATCH v7 03/12] update-index: add a new --force-write-index option

On 9/20/2017 9:46 PM, Junio C Hamano wrote: Ben Peart <peart...@gmail.com> writes: Lets see how my ascii art skills do at describing this: Your ascii art is fine. If you said upfront that the capital letters signify points in time, lower letters are file-touching events, and time

Re: [PATCH v6 09/12] split-index: disable the fsmonitor extension when running the split index test

On 9/20/2017 1:46 PM, Jonathan Nieder wrote: Hi, Ben Peart wrote: On 9/19/2017 4:43 PM, Jonathan Nieder wrote: This feels to me like the wrong fix. Wouldn't it be better for the test not to depend on the precise object ids? See the "Tips for Writing Tests" section in t/R

Re: [PATCH v7 02/12] preload-index: add override to enable testing preload-index

On 9/20/2017 6:06 PM, Stefan Beller wrote: On Tue, Sep 19, 2017 at 12:27 PM, Ben Peart <benpe...@microsoft.com> wrote: Preload index doesn't run unless it has a minimum number of 1000 files. To enable running tests with fewer files, add an environment variable (GIT_FORCE_PRELOAD_TEST)

Re: [PATCH v6 09/12] split-index: disable the fsmonitor extension when running the split index test

On 9/19/2017 4:43 PM, Jonathan Nieder wrote: Hi, Ben Peart wrote: The split index test t1700-split-index.sh has hard coded SHA values for the index. Currently it supports index V4 and V3 but assumes there are no index extensions loaded. When manually forcing the fsmonitor extension

Re: [PATCH v7 05/12] fsmonitor: add documentation for the fsmonitor extension.

Thanks for the review. I'm not an English major so appreciate the feedback on my attempts to document the feature. On 9/20/2017 6:00 AM, Martin Ågren wrote: On 19 September 2017 at 21:27, Ben Peart <benpe...@microsoft.com> wrote: diff --git a/Documentation/git-update-index.

Re: [PATCH v7 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.

On 9/20/2017 2:23 AM, Junio C Hamano wrote: Ben Peart <benpe...@microsoft.com> writes: @@ -344,6 +346,7 @@ struct index_state { struct hashmap dir_hash; unsigned char sha1[20]; struct untracked_cache *untracked; + uint64_t fsmonitor_last_update; This

Re: [PATCH v7 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.

On 9/19/2017 10:28 PM, Junio C Hamano wrote: Ben Peart <benpe...@microsoft.com> writes: +/* do stat comparison even if CE_FSMONITOR_VALID is true */ +#define CE_MATCH_IGNORE_FSMONITOR 0X20 Hmm, when should a programmer use this flag? Pretty much the same places you would al

Re: [PATCH v7 03/12] update-index: add a new --force-write-index option

On 9/20/2017 1:47 AM, Junio C Hamano wrote: Ben Peart <benpe...@microsoft.com> writes: + OPT_SET_INT(0, "force-write-index", _write, + N_("write out the index even if is not flagged as changed"), 1), Hmph. The only time thi

<    1   2   3   4   5   6   7   >