On Thu, Mar 10, 2016 at 6:21 AM, Junio C Hamano <gits...@pobox.com> wrote: > Junio C Hamano <gits...@pobox.com> writes: > >> David Turner <dtur...@twopensource.com> writes: >> >>> From: Nguyễn Thái Ngọc Duy <pclo...@gmail.com> >>> >>> Instead of reading the index from disk and worrying about disk >>> corruption, the index is cached in memory (memory bit-flips happen >>> too, but hopefully less often). The result is faster read. Read time >>> is reduced by 70%. >>> >>> The biggest gain is not having to verify the trailing SHA-1, which >>> takes lots of time especially on large index files. > > Come to think of it, wouldn't it be far less intrusive change to > just put the index on a ramdisk and skip the trailing SHA-1 > verification, to obtain a similar result with the same trade off > (i.e. blindly trusting memory instead of being paranoid)?
I'm straying off-topic again because this reminded me about lmdb :) For the record when I looked at lmdb I thought of using it as an index format too. It has everything we wanted in index-v5. Good enough that we would not need index-helper and split-index to work around current index format. Though I finally decided it didn't fit well. On the good side, lmdb is b+ trees, we can update directly on disk (without rewriting whole file), read directly from mmap'd file and not worry about corrupting it. There's no SHA-1 verification either (and we can do crc32 per entry if we want too). It's good. But, it does not fit well the our locking model (but we can change this). And we sometimes want to create temporary throw-away indexes (e.g. partial commits) which I don't think is easy to do. And the reading directly from mmap does not give us much because we have to do byte endian conversion anyway. Hmm.. on second thought, even though lmdb can't be the default format (for a bunch of other limitations), it can still be an option for super big worktrees, just like index-helper being an optional optimization. Hm.. hm.. if we can support lmdb as index format in addition to the current format, bringing back some work from Thomas.. We may still need a daemon or something to deal with watchman, but the underlying mechanism is exactly the same... David, Junio, what do you think? -- Duy -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
