Dan and Frank are the chunking experts, but iiuc you're right, dir_max has no effect when chunking is enabled, and iiuc also yes, the plan is to retire the old dirent cache in favor of chuking. Frank also has new commits for pruning the dirent cache via it's LRU list.
regards, Matt On Wed, Jul 12, 2017 at 11:13 PM, gui mark <guiggl...@gmail.com> wrote: > IIUC, the dirent chunking is a complete replacement of the old-style dirent > cache. As I tried, when I have 'Dir_Chunk' on, then 'Dir_Max' will not limit > the number of cached dirents, so are we going to kick out the old direct > cache completely ? > > On Tue, Jul 11, 2017 at 8:40 PM, Daniel Gryniewicz <d...@redhat.com> wrote: >> >> On 07/11/2017 06:00 AM, gui mark wrote: >>> >>> Hi Frank (cc list) >>> >>> I started to grow interested with your dirent chunking these days, and >>> I've got a few questions below: >>> >>> 1. Why we need chunking ? only for lru-style management for dirents ? >> >> >> The big driver for it was the way that dirents were cached before. When a >> directory was cached, the entire thing needed to be cached before any >> dirents were returned to the client. This had 2 problems for large >> directories: 1) It took a lot of time, and clients could time out before >> receiving a reply; 2) It took a lot of memory, since we couldn't reap any >> dirents, because we needed them all to return any. >> >> The chunking breaks the readdir up into chunks, so that only one chunk >> must be read before results can be returned to clients, and so that chunks >> can be reaped, freeing memory without breaking readdir. >> >>> 2. Is this big feature completed? If not, could you please share some >>> blueprints on that, so we may be able to contribute some effort ? >> >> >> The feature itself is complete; the last bit of the memory management is >> still outstanding here: https://review.gerrithub.io/#/c/367446/ >> >>> 3. Do the underlying FSALs have to add impl for any new interfaces (if >>> any) to support chunking ? >> >> >> Yes and no. Everything works correctly without any changes. However, if >> the FSAL can compute dirent cookies directly from the name (ie, no >> round-trip to the cluster) then it can implement compute_readdir_cookie(), >> and be more efficient when handling dirents created from calls other than >> readdir() (such as lookup() and rename()). This is purely an optimization, >> but one that can theoretically make a difference to come workloads. >> >> >>> 4. Does the new-style dirent cache borrows ideas for the kernel dcache ? >> >> >> Not really, no. It mostly borrows from the Ganesha's object cache. >> >> Daniel >> >> >> ------------------------------------------------------------------------------ >> Check out the vibrant tech community on one of the world's most >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >> _______________________________________________ >> Nfs-ganesha-devel mailing list >> Nfs-ganesha-devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > > ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
