On Fri, May 31, 2024 at 03:03:47PM -0700, Felix Lechner wrote: > Hi Efraim, > > On Tue, May 28 2024, Efraim Flashner wrote: > > > As your store grows larger the inherent deduplication from the > > guix-daemon approaches a 3:1 file deduplication ratio. > > Thank you for your explanations and your data about btrfs! Btrfs > compression is a well-understood feature, although even its developers > acknowledge that the benefit is hard to quantify. > > It probably makes more sense to focus on the Guix daemon here. I hope > you don't mind a few clarifying questions. > > Why, please, does the benefit of de-duplication approach a fixed ratio > of 3:1? Does the benefit not depend on the number of copies in the > store, which can vary by any number? (It sounds like the answer may > have something to do with store size.)
It would seem that this is just my experience and I'm not sure of an actual reason why this is the case. I believe that with the hardlinks only files which are identical would share a link, as opposed to a block based deduplication, where there could be more granular deduplication, so it's quite likely that multiple copies of the same package at the same version would share the majority of their files with the other copies of the package. > Further, why is the removal of hardlinks counted as saving space even > when their inode reference count, which is widely available [1] is > greater than one? I suspect that this part of the code is in the C++ daemon, which no one really wants to hack on. AFAIK Nix turned off deduplication by default years ago to speed up store operations, so I wouldn't be surprised if they also haven't worked on that part of the code. > Finally, barring a better solution should our output numbers be divided > by three to being them closer to the expected result for users? > > [1] https://en.wikipedia.org/wiki/Hard_link#Reference_counting (ins)efraim@3900XT ~$ sudo compsize -x /gnu Processed 39994797 files, 12867013 regular extents (28475611 refs), 20558307 inline. Type Perc Disk Usage Uncompressed Referenced TOTAL 56% 437G 776G 2.1T none 100% 275G 275G 723G zstd 32% 161G 500G 1.4T It looks like right now my store is physically using 437GB of space. Looking only at the total the Uncompressed -> Referenced ratio being about 2.77:1 and Disk Usage -> Uncompressed being about 1.78:1, I'm netting a total of 4.92:1. Numbers on Berlin are a bit different: (ins)efraim@berlin ~$ time guix shell compsize -- sudo compsize -x /gnu Processed 41030472 files, 14521470 regular extents (37470325 refs), 17429255 inline. Type Perc Disk Usage Uncompressed Referenced TOTAL 59% 578G 970G 3.2T none 100% 402G 402G 1.1T zstd 31% 176G 567G 2.1T real 45m9.762s user 1m53.984s sys 24m37.338s Uncompressed -> Referenced: 3.4:1 Disk Usage -> Uncompressed: 1.68:1 Total: 5.67:1 Looking at it another way, the bits that are compressible with zstd together move from 3.79:1 to 12.22:1, with no change (2.8:1) for the uncompressible bits. -- Efraim Flashner <efr...@flashner.co.il> רנשלפ םירפא GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted
signature.asc
Description: PGP signature
