Here is some additional information I believe to be relevant.
1. I believe the adjusted directory entries are correctly cached before
nfs4_readdir_callback() calls fsal_status = obj->obj_ops.test_access().
2. mdcache_is_attrs_valid() returns false because "flags |=
MDCACHE_TRUST_ATTRS;" is executed and that causes
"((entry->mde_flags & flags) != flags)" to be false.
3. mdcache_test_access() calls fsal_test_access() and the
fsal_test_access() obj_hdl->obj_ops.getattrs() call skips over the
NULL derived layer and goes to the PROXY getattrs().
4. I've also attached a gdb debug log that might provide additional
useful info
1. Near the top of the log, it shows that pxy_readdir() was called
by my NULL derived layer.
2. Below mdcache_test_access() is where the NULL derived layer is
being ignored and unadjusted attributes get saved in the cache.
3. I haven't yet traced why the NULL derived layer is ignored when
reading (if the directory is listed/viewed before the read).
I believe it would be easy to stack over VFS but I'm trying to keep my
testing as realistic as possible so I'm stacking my NULL derived layer
over PROXY.
I suggest you populate your bottom (e.g. PROXY or VFS) file system with the attached 1.txt file so that you don't need to do any writes (and afterwards flush caches or restart). Through the NULL derived layer, the file size is shown as 19 bytes. The NULL derived test layer is inserting 32 bytes every 256 bytes. If you "cat" 1.txt before listing/viewing the directory, the NULL derived layer is not bypassed and the read data is as expected. If you ls 1.txt after reading the file but before listing the directory or "cat *", the adjusted size is listed.
This is prototype (proof of concept) code so it isn't yet optimized or realistic processing.
If I truly understood the layering, I might already understand how to stop ignoring the NULL derived layer below the mdcache_test_access() calls. I'll continue to learn as I get more time to work with NFS-Ganesha.
Thanks for taking a look. Kevin On 12/21/2016 10:30 AM, Daniel Gryniewicz wrote:
Okay, that callpath calls through getattrs(), so it should get back whatever the modified getattrs() call returns. If you'd like to post code, I'll take a look when I can. I'm off over the holidays, so there's no hurry from my end. Daniel On 12/21/2016 09:58 AM, Kevin C. wrote:I have meetings starting soon and lasting for several hours so I cannot be as detailed as I'd like in this reply. I believe I see the processing that you describe. I had modified the NULL derived layer's *_readdir_cb() function. I enabled full debug logging for all components and started reviewing that information. I'm not yet done reviewing the information but it looks like mdcache_refresh_atttributes() processing is setting the file size to the unadjusted size. If you desire, I can provide patches so you could see the NULL derived layer. If you have access to very large transfer locations, I could even provide VirtualBox virtual PCs with my test configuration. On 12/21/2016 08:13 AM, Daniel Gryniewicz wrote:Hi, Kevin. Welcome! Glad someone's getting use out of stacking. First, your stack. It sounds like you're stacking like this: MDCACHE | NULL (modified) | PROXY Correct? The readdir() path is somewhat different than the lookup/open path. In a normal lookup (say), the call comes into the protocol layer, and is passed to MDCACHE, and so down the stack, to the bottom FSAL, which creates a handle. The handle is then passed back up the stack, each layer wrapping it as it goes, until MDCACHE creates it's handle and caches it. Future attempts to get the metadata (getattr() calls) are handled directly by MDCACHE until the cache entry times out, or until an event causes cache invalidation on the entry. For readdir(), however, no lookup is ever made. The Protocol layer calls readdir(), which calls into MDCACHE. MDCACHE then caches the *entire* directory worth of dirents for the directory (except for large directories, but we'll skip that for a second). This involves calling readdir() down the stack starting at 0 and going until there are no more entries. However, a readdir() call doesn't just create directory entries, it also materializes handles for each object in the directory, and passes them back up. This includes attributes, which are cached at the MDCACHE layer as stated before. This means that any modification of size or other attributes that you do on create(), lookup(), or getattr() must also be done during readdir(), or the full original attributes will be cached, causing the incorrect values to be returned until the entry times out. Does this help? Daniel On 12/20/2016 02:49 PM, Kevin C. wrote:I'm trying to simplify migration away from a legacy non-standard NFS server by creating a stackable FSAL. For my tests, I’m using a slightly modified copy of FSAL NULL above FSAL PROXY. Reads from the legacy non-standard NFS server includes data not originally provided by user application writes. Writes to the legacy non-standard NFS server also requires insertion of data not provided by the user application. I am unable to change the legacy system (anytime soon). For all tests I’ve run, my FSAL NULL derived layer is successfully inserting test data. When file system operations are done in a certain order (e.g. reading files before listing directory contents), my FSAL NULL derived layer successfully removes the inserted test data from the read data stream presented to user applications and FSAL adjusted file sizes are shown by file manager GUI or "ls". If the PROXY NFS server directory contents are listed first (via "ls" or file manager GUI) or output via "cat //mnt/nfstst//*", it seems like the FSAL NULL derived layer is bypassed. The user applications receive the extra/unwanted data as well as the expected user application data and directory listings show file sizes that are NOT adjusted to subtract the extra/unwanted data portions from the size. I hope that someone more familiar with NFS-Ganesha's architecture can help identify the processing path(s) that I have failed to identify (i.e. adjust via the stackableFSAL) so that I can eliminate this operation order dependency. My original work was done on the 2.4 stable branch. The development process indicates all patches should be made to the current development branch. I've recreated these results with the 2.5 development branch. If your input helps me find a bug, I'll submit a fix. I’m requesting input from more experienced NFS-Ganesha developers. Please provide any input that helps me eliminate this processing order dependency. Thanks in advance, Kevin ------------------------------------------------------------------------------ Developer Access Program for Intel Xeon Phi Processors Access to Intel Xeon Phi processor-based developer platforms. With one year of Intel Parallel Studio XE. Training and support from Colfax. Order your platform today.http://sdm.link/intel _______________________________________________ Nfs-ganesha-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel------------------------------------------------------------------------------ Developer Access Program for Intel Xeon Phi Processors Access to Intel Xeon Phi processor-based developer platforms. With one year of Intel Parallel Studio XE. Training and support from Colfax. Order your platform today.http://sdm.link/intel _______________________________________________ Nfs-ganesha-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel------------------------------------------------------------------------------ Developer Access Program for Intel Xeon Phi Processors Access to Intel Xeon Phi processor-based developer platforms. With one year of Intel Parallel Studio XE. Training and support from Colfax. Order your platform today.http://sdm.link/intel _______________________________________________ Nfs-ganesha-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel------------------------------------------------------------------------------ Developer Access Program for Intel Xeon Phi Processors Access to Intel Xeon Phi processor-based developer platforms. With one year of Intel Parallel Studio XE. Training and support from Colfax. Order your platform today.http://sdm.link/intel _______________________________________________ Nfs-ganesha-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
skippedlayer.tgz
Description: application/compressed-tar
------------------------------------------------------------------------------ Developer Access Program for Intel Xeon Phi Processors Access to Intel Xeon Phi processor-based developer platforms. With one year of Intel Parallel Studio XE. Training and support from Colfax. Order your platform today.http://sdm.link/intel
_______________________________________________ Nfs-ganesha-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
