Thanks for all the feedback. The biggest change since the last version is how this patch series interacts with the split-index feature. With a split index, most of the cache entries are stored in the shared index so would benefit from multi-threaded parsing. To enable that, the EOIE and IEOT extensions are now written into the shared index (rather than being stripped out like the other extensions).
Because of this, I can now update the tests in t1700-split-index.sh to have updated SHA values that include the EOIE extension instead of disabling the extension. Using p0002-read-cache.sh to generate some performance numbers shows how each of the various patches contribute to the overall performance win on a particularly large repo. Repo w/3M files Baseline Optimize V4 Extensions Entries -------------------------------------------------------------------------- 0002.1: read_cache 693.29 655.65 -5.4% 470.71 -32.1% 399.62 -42.4% Note how this cuts nearly 300ms off the index load time! Base Ref: master Web-Diff: https://github.com/benpeart/git/commit/c1125a5d9a Checkout: git fetch https://github.com/benpeart/git read-index-multithread-v7 && git checkout c1125a5d9a ### Patches Ben Peart (6): read-cache: clean up casting and byte decoding eoie: add End of Index Entry (EOIE) extension config: add new index.threads config setting read-cache: load cache extensions on a worker thread ieot: add Index Entry Offset Table (IEOT) extension read-cache: load cache entries on worker threads Nguyễn Thái Ngọc Duy (1): read-cache.c: optimize reading index format v4 Documentation/config.txt | 7 + Documentation/technical/index-format.txt | 41 ++ config.c | 18 + config.h | 1 + read-cache.c | 749 +++++++++++++++++++---- t/README | 5 + t/t1700-split-index.sh | 13 +- 7 files changed, 715 insertions(+), 119 deletions(-) base-commit: fe8321ec057f9231c26c29b364721568e58040f7 -- 2.18.0.windows.1
