> On Jul 7, 2019, at 11:09 AM, Stilez <stil...@gmail.com> wrote: > > I feel that, while that's true and valid, it also kind of misses the point? > > What I'm wondering is, are there simple enhancements that would be beneficial > in that area, or provide useful internal data. > > It seems plausible that if a configurable in space map block size helps, > perhaps a configurable DDT block size could as well, and that if DDT contents > are needed on every file load/save for a deduped pool, then a way to preload > them could be beneficial in the same.way as a way to preload spacemap data.
Both spacemaps and DDT are AVL trees. But there is one DDT vs hundreds (or more) of spacemaps. But spacemaps are only needed for writes, so if we aren't allocating space from a metaslab, the spacemap for that metaslab can be evicted from RAM. Or, to look at it another way, spacemaps are constrained to space (LBA range) but DDT covers the whole pool. > > Clearly allocation devices will help, but gains are always layered. "Faster > storage will fix it" isnt really an answer, any more than its an answer for > any greatly bottlenecked critical pathway in a file system - and for deduped > pools, access to DDT records is *the* critical element, no user data can be > read from disk or put to txgs without them. Fundamentally there are a few > useful configurables available for spacemaps that could be potential wins if > also available for DDTs. Since analogs for these settings already exist in > the code, perhaps no great work is involved in them, and perhaps they would > give cheap but significant IO gains for pools with dedup enabled. In the past, some folks have proposed a different structure for DDT, such as using bloom filters or other fast-lookup techniques. But as Garrett points out, the real wins are hard to come by. Meanwhile, PRs are welcome. -- richard > > Hence I think the question is valid, and remains valid both before allocation > classes and after them, and might be worth considering deeper. > > > > On 7 July 2019 16:03:59 Richard Elling <richard.ell...@richardelling.com> > wrote: > >> Yes, from the ZoL zpool man page: >> A device dedicated solely for deduplication tables. >> >> -- richard >> >> >> >> On Jul 7, 2019, at 5:41 AM, Stilez <stil...@gmail.com >> <mailto:stil...@gmail.com>> wrote: >> >>> "Dedup special class"? >>> >>> On 6 July 2019 16:24:27 Richard Elling <richard.ell...@richardelling.com >>> <mailto:richard.ell...@richardelling.com>> wrote: >>> >>>> >>>> On Jul 5, 2019, at 9:11 PM, Stilez <stil...@gmail.com >>>> <mailto:stil...@gmail.com>> wrote: >>>> >>>>> I'm one of many end-users with highly dedupable pools held back by DDT >>>>> and spacemap RW inefficiencies. There's been discussion and presentations >>>>> - Matt Ahrens' talk at BSDCan 2016 ("Dedup doesn't have to suck") was >>>>> especially useful, and allocation classes from the ZoL/ZoF work will >>>>> allow metadata-specific offload to SSD. But briad discussion of this >>>>> general area is not on the roadmap atm, probably bc so much else is a >>>>> priority and seems nobody's stepped up. >>>> >>>> In part because dedup will always be slower than non-dedup while the cost >>>> of storage continues to plummet (flash SSDs down 40% in the past year and >>>> there is currently an oversupply of NAND). A good starting point for >>>> experiments is to use the dedup special class and report back to the >>>> community how well it works for you. >>>> >>>> -- richard >>>> >>>> openzfs <https://openzfs.topicbox.com/latest> / openzfs-developer / see >>>> discussions <https://openzfs.topicbox.com/groups/developer> + participants >>>> <https://openzfs.topicbox.com/groups/developer/members> + delivery options >>>> <https://openzfs.topicbox.com/groups/developer/subscription>Permalink >>>> <https://openzfs.topicbox.com/groups/developer/Td9c7189186fd24f2-Mab4823ef7eaa494c26bfa3d8> > > ------------------------------------------ openzfs: openzfs-developer Permalink: https://openzfs.topicbox.com/groups/developer/Td9c7189186fd24f2-M3d9e7f78fbe5012e53b176a8 Delivery options: https://openzfs.topicbox.com/groups/developer/subscription
