Re: [GIT PULL] afs fixes
The pull request you sent on Thu, 29 Oct 2020 14:07:26 +: > git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git > tags/afs-fixes-20201029 has been merged into torvalds/linux.git: https://git.kernel.org/torvalds/c/598a597636f8618a0520fd3ccefedaed9e4709b0 Thank you! -- Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/prtracker.html
[GIT PULL] afs fixes
Hi Linus, Could you pull these afs fixes, please? They include the following: (1) Fix copy_file_range() to an afs file now returning EINVAL if the splice_write file op isn't supplied. (2) Fix a deref-before-check in afs_unuse_cell(). (3) Fix a use-after-free in afs_xattr_get_acl(). (4) Fix afs to not try to clear PG_writeback when laundering a page. (5) Fix afs to take a ref on a page that it sets PG_private on and to drop that ref when clearing PG_private. This is done through recently added helpers. (6) Fix a page leak if write_begin() fails. (7) Fix afs_write_begin() to not alter the dirty region info stored in page->private, but rather do this in afs_write_end() instead when we know what we actually changed. (8) Fix afs_invalidatepage() to alter the dirty region info on a page when partial page invalidation occurs so that we don't inadvertantly include a span of zeros that will get written back if a page gets laundered due to a remote 3rd-party induced invalidation. We mustn't, however, reduce the dirty region if the page has been seen to be mapped (ie. we got called through the page_mkwrite vector) as the page might still be mapped and we might lose data if the file is extended again. (9) Fix the dirty region info to have a lower resolution if the size of the page is too large for this to be encoded (e.g. powerpc32 with 64K pages). Note that this might not be the ideal way to handle this, since it may allow some leakage of undirtied zero bytes to the server's copy in the case of a 3rd-party conflict. To aid (8) and (9), two additional patches are included: (*) Wrap the manipulations of the dirty region info stored in page->private into helper functions. (*) Alter the encoding of the dirty region so that the region bounds can be stored with one fewer bit, making a bit available for the indication of mappedness. Thanks, David --- The following changes since commit 3650b228f83adda7e5ee532e2b90429c03f7b9ec: Linux 5.10-rc1 (2020-10-25 15:14:11 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git tags/afs-fixes-20201029 for you to fetch changes up to 2d9900f26ad61e63a34f239bc76c80d2f8a6ff41: afs: Fix dirty-region encoding on ppc32 with 64K pages (2020-10-29 13:53:04 +) AFS fixes Dan Carpenter (1): afs: Fix a use after free in afs_xattr_get_acl() David Howells (10): afs: Fix copy_file_range() afs: Fix tracing deref-before-check afs: Fix afs_launder_page to not clear PG_writeback afs: Fix to take ref on page when PG_private is set afs: Fix page leak on afs_write_begin() failure afs: Fix where page->private is set during write afs: Wrap page->private manipulations in inline functions afs: Alter dirty range encoding in page->private afs: Fix afs_invalidatepage to adjust the dirty region afs: Fix dirty-region encoding on ppc32 with 64K pages fs/afs/cell.c | 3 +- fs/afs/dir.c | 12 ++ fs/afs/dir_edit.c | 6 +-- fs/afs/file.c | 78 ++--- fs/afs/internal.h | 57 fs/afs/write.c | 105 - fs/afs/xattr.c | 2 +- include/trace/events/afs.h | 20 ++--- 8 files changed, 188 insertions(+), 95 deletions(-)
Re: [GIT PULL] afs: Fixes for bugs found by xfstests
The pull request you sent on Tue, 16 Jun 2020 22:51:46 +0100: > git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git > tags/afs-fixes-20200616 has been merged into torvalds/linux.git: https://git.kernel.org/torvalds/c/26c20ffcb5c86eb6a052e0d528396de5600c9710 Thank you! -- Deet-doot-dot, I am a bot. https://korg.wiki.kernel.org/userdoc/prtracker
[GIT PULL] afs: Fixes for bugs found by xfstests
Hi Linus, I've managed to get xfstests kind of working with afs. Here are a set of patches that fix most of the bugs found. There are a number of primary issues: (1) Incorrect handling of mtime and non-handling of ctime. It might be argued, that the latter isn't a bug since the AFS protocol doesn't support ctime, but I should probably still update it locally. (2) Shared-write mmap, truncate and writeback bugs. This includes not changing i_size under the callback lock, overwriting local i_size with the reply from the server after a partial writeback, not limiting the writeback from an mmapped page to EOF. (3) Checks for an abort code indicating that the primary vnode in an operation was deleted by a third-party are done in the wrong place. (4) Silly rename bugs. This includes an incomplete conversion to the new operation handling, duplicate nlink handling, nlink changing not being done inside the callback lock and insufficient handling of third-party conflicting directory changes. And some secondary ones: (1) The UAEOVERFLOW abort code should map to EOVERFLOW not EREMOTEIO. (2) Remove a couple of unused or incompletely used bits. (3) Remove a couple of redundant success checks. These seem to fix all the data-corruption bugs found by "./check -afs -g quick", along with the obvious silly rename bugs and time bugs. There are still some test failures, but they seem to fall into two classes: firstly, the authentication/security model is different to the standard UNIX model and permission is arbitrated by the server and cached locally; and secondly, there are a number of features that AFS does not support (such as mknod). But in these cases, the tests themselves need to be adapted or skipped. Using the in-kernel afs client with xfstests also found a bug in the AuriStor AFS server that has been fixed for a future release. David --- The following changes since commit b3a9e3b9622ae10064826dccb4f7a52bd88c7407: Linux 5.8-rc1 (2020-06-14 12:45:04 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git tags/afs-fixes-20200616 for you to fetch changes up to b6489a49f7b71964e37978d6f89bbdbdb263f6f5: afs: Fix silly rename (2020-06-16 22:00:28 +0100) AFS fixes David Howells (12): afs: Fix non-setting of mtime when writing into mmap afs: afs_write_end() should change i_size under the right lock afs: Fix EOF corruption afs: Concoct ctimes afs: Fix truncation issues and mmap writeback size afs: Fix the mapping of the UAEOVERFLOW abort code afs: Remove yfs_fs_fetch_file_status() as it's not used afs: Fix yfs_fs_fetch_status() to honour vnode selector afs: Remove afs_operation::abort_code afs: Fix use of afs_check_for_remote_deletion() afs: afs_vnode_commit_status() doesn't need to check the RPC error afs: Fix silly rename fs/afs/dir.c | 62 ++ fs/afs/dir_silly.c| 38 +++-- fs/afs/file.c | 2 +- fs/afs/flock.c| 4 +-- fs/afs/fs_operation.c | 10 +- fs/afs/inode.c| 91 ++--- fs/afs/internal.h | 36 +--- fs/afs/misc.c | 1 + fs/afs/write.c| 12 +-- fs/afs/yfsclient.c| 93 ++- 10 files changed, 225 insertions(+), 124 deletions(-)
Re: [GIT PULL] afs: Fixes
The pull request you sent on Thu, 22 Aug 2019 14:10:39 +0100: > git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git > tags/afs-fixes-20190822 has been merged into torvalds/linux.git: https://git.kernel.org/torvalds/c/e8c3fa9f4d3b9c56ee9436c310318a8165d695c1 Thank you! -- Deet-doot-dot, I am a bot. https://korg.wiki.kernel.org/userdoc/prtracker
[GIT PULL] afs: Fixes
Hi Linus, Here are three fixes for afs: (1) Fix a cell record leak due to the default error not being cleared. (2) Fix an oops in tracepoint due to a pointer that may contain an error. (3) Fix the ACL storage op for YFS where the wrong op definition is being used. By luck, this only actually affects the information appearing in traces. David --- The following changes since commit d1abaeb3be7b5fa6d7a1fbbd2e14e3310005c4c1: Linux 5.3-rc5 (2019-08-18 14:31:08 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git tags/afs-fixes-20190822 for you to fetch changes up to 7533be858f5b9a036b9f91556a3ed70786abca8e: afs: use correct afs_call_type in yfs_fs_store_opaque_acl2 (2019-08-22 13:33:27 +0100) AFS fixes David Howells (1): afs: Fix leak in afs_lookup_cell_rcu() Marc Dionne (1): afs: Fix possible oops in afs_lookup trace event YueHaibing (1): afs: use correct afs_call_type in yfs_fs_store_opaque_acl2 fs/afs/cell.c | 4 fs/afs/dir.c | 3 ++- fs/afs/yfsclient.c | 2 +- 3 files changed, 7 insertions(+), 2 deletions(-)
Re: [GIT PULL] afs: Fixes
The pull request you sent on Wed, 14 Aug 2019 15:18:40 +0100: > git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git > tags/afs-fixes-20190814 has been merged into torvalds/linux.git: https://git.kernel.org/torvalds/c/e22a97a2a85d2a0bdfb134cbbc7ff856ae67edba Thank you! -- Deet-doot-dot, I am a bot. https://korg.wiki.kernel.org/userdoc/prtracker
[GIT PULL] afs: Fixes
Hi Linus, Can you pull these afs fixes please? (1) Fix the CB.ProbeUuid handler to generate its reply correctly. (2) Fix a mix up in indices when parsing a Volume Location entry record. (3) Fix a potential NULL-pointer deref when cleaning up a read request. (4) Fix the expected data version of the destination directory in afs_rename(). (5) Fix afs_d_revalidate() to only update d_fsdata if it's not the same as the directory data version to reduce the likelihood of overwriting the result of a competing operation. (d_fsdata carries the directory DV or the least-significant word thereof). (6) Fix the tracking of the data-version on a directory and make sure that dentry objects get properly initialised, updated and revalidated. Also fix rename to update d_fsdata to match the new directory's DV if the dentry gets moved over and unhash the dentry to stop afs_d_revalidate() from interfering. David --- The following changes since commit 2a11c76e5301dddefcb618dac04f74e6314df6bc: Merge tag 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost (2019-07-29 11:34:12 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git tags/afs-fixes-20190814 for you to fetch changes up to 9dd0b82ef530cdfe805c9f7079c99e104be59a14: afs: Fix missing dentry data version updating (2019-07-30 14:38:52 +0100) AFS Fixes Reviewed-by: Marc Dionne David Howells (4): afs: Fix the CB.ProbeUuid service handler to reply correctly afs: Fix off-by-one in afs_rename() expected data version calculation afs: Only update d_fsdata if different in afs_d_revalidate() afs: Fix missing dentry data version updating Jia-Ju Bai (1): fs: afs: Fix a possible null-pointer dereference in afs_put_read() Marc Dionne (1): afs: Fix loop index mixup in afs_deliver_vl_get_entry_by_name_u() fs/afs/cmservice.c | 10 ++ fs/afs/dir.c | 89 -- fs/afs/file.c | 12 +--- fs/afs/vlclient.c | 11 --- 4 files changed, 89 insertions(+), 33 deletions(-)
[GIT PULL] afs: Fixes
Hi Linus, Can you pull these afs fixes please? (1) Fix the CB.ProbeUuid handler to generate its reply correctly. (2) Fix a mix up in indices when parsing a Volume Location entry record. (3) Fix a potential NULL-pointer deref when cleaning up a read request. (4) Fix the expected data version of the destination directory in afs_rename(). (5) Fix afs_d_revalidate() to only update d_fsdata if it's not the same as the directory data version to reduce the likelihood of overwriting the result of a competing operation. (d_fsdata carries the directory DV or the least-significant word thereof). (6) Fix the tracking of the data-version on a directory and make sure that dentry objects get properly initialised, updated and revalidated. Also fix rename to update d_fsdata to match the new directory's DV if the dentry gets moved over and unhash the dentry to stop afs_d_revalidate() from interfering. David --- The following changes since commit 2a11c76e5301dddefcb618dac04f74e6314df6bc: Merge tag 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost (2019-07-29 11:34:12 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git tags/afs-fixes-20190814 for you to fetch changes up to 9dd0b82ef530cdfe805c9f7079c99e104be59a14: afs: Fix missing dentry data version updating (2019-07-30 14:38:52 +0100) AFS Fixes Reviewed-by: Marc Dionne David Howells (4): afs: Fix the CB.ProbeUuid service handler to reply correctly afs: Fix off-by-one in afs_rename() expected data version calculation afs: Only update d_fsdata if different in afs_d_revalidate() afs: Fix missing dentry data version updating Jia-Ju Bai (1): fs: afs: Fix a possible null-pointer dereference in afs_put_read() Marc Dionne (1): afs: Fix loop index mixup in afs_deliver_vl_get_entry_by_name_u() fs/afs/cmservice.c | 10 ++ fs/afs/dir.c | 89 -- fs/afs/file.c | 12 +--- fs/afs/vlclient.c | 11 --- 4 files changed, 89 insertions(+), 33 deletions(-)
Re: [GIT PULL] AFS fixes
The pull request you sent on Wed, 26 Jun 2019 14:39:53 +0100: > git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git > tags/afs-fixes-20190620 has been merged into torvalds/linux.git: https://git.kernel.org/torvalds/c/cd0f3aaebc5b17e0ccb1b9ef9ae43042d075d767 Thank you! -- Deet-doot-dot, I am a bot. https://korg.wiki.kernel.org/userdoc/prtracker
[GIT PULL] AFS fixes
Hi Linus, Could you pull this please? There are four patches: (1) Fix the printing of the "vnode modified" warning to exclude checks on files for which we don't have a callback promise from the server (and so don't expect the server to tell us when it changes). Without this, for every file or directory for which we still have an in-core inode that gets changed on the server, we may get a message logged when we next look at it. This can happen in bulk if, for instance, someone does "vos release" to update a R/O volume from a R/W volume and a whole set of files are all changed together. We only really want to log a message if the file changed and the server didn't tell us about it or we failed to track the state internally. (2) Fix accidental corruption of either afs_vlserver struct objects or the the following memory locations (which could hold anything). The issue is caused by a union that points to two different structs in struct afs_call (to save space in the struct). The call cleanup code assumes that it can simply call the cleanup for one of those structs if not NULL - when it might be actually pointing to the other struct. This means that every Volume Location RPC op is going to corrupt something. (3) Fix an uninitialised spinlock. This isn't too bad, it just causes a one-off warning if lockdep is enabled when "vos release" is called, but the spinlock still behaves correctly. (4) Fix the setting of i_block in the inode. This causes du, for example, to produce incorrect results, but otherwise should not be dangerous to the kernel. The in-kernel AFS client has been undergoing testing on opendev.org on one of their mirror machines. They are using AFS to hold data that is then served via apache, and Ian Wienand had reported seeing oopses, spontaneous machine reboots and updates to volumes going missing. This patch series appears to have fixed the problem, very probably due to patch (2), but it's not 100% certain. Reviewed-by: Jeffrey Altman Tested-by: Marc Dionne Tested-by: Ian Wienand --- The following changes since commit a188339ca5a396acc588e5851ed7e19f66b0ebd9: Linux 5.2-rc1 (2019-05-19 15:47:09 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git tags/afs-fixes-20190620 for you to fetch changes up to 2cd42d19cffa0ec3dfb57b1b3e1a07a9bf4ed80a: afs: Fix setting of i_blocks (2019-06-20 18:12:02 +0100) AFS fixes David Howells (4): afs: Fix over zealous "vnode modified" warnings afs: Fix vlserver record corruption afs: Fix uninitialised spinlock afs_volume::cb_break_lock afs: Fix setting of i_blocks fs/afs/callback.c | 4 ++-- fs/afs/inode.c| 31 +++ fs/afs/internal.h | 8 +++- fs/afs/volume.c | 1 + 4 files changed, 25 insertions(+), 19 deletions(-)
Re: [GIT PULL] AFS fixes
Oops... I forgot to include the pull request bit. Will resend. David
[GIT PULL] AFS fixes
Hi Linus, Could you pull this please? There are four patches: (1) Fix the printing of the "vnode modified" warning to exclude checks on files for which we don't have a callback promise from the server (and so don't expect the server to tell us when it changes). Without this, for every file or directory for which we still have an in-core inode that gets changed on the server, we may get a message logged when we next look at it. This can happen in bulk if, for instance, someone does "vos release" to update a R/O volume from a R/W volume and a whole set of files are all changed together. We only really want to log a message if the file changed and the server didn't tell us about it or we failed to track the state internally. (2) Fix accidental corruption of either afs_vlserver struct objects or the the following memory locations (which could hold anything). The issue is caused by a union that points to two different structs in struct afs_call (to save space in the struct). The call cleanup code assumes that it can simply call the cleanup for one of those structs if not NULL - when it might be actually pointing to the other struct. This means that every Volume Location RPC op is going to corrupt something. (3) Fix an uninitialised spinlock. This isn't too bad, it just causes a one-off warning if lockdep is enabled when "vos release" is called, but the spinlock still behaves correctly. (4) Fix the setting of i_block in the inode. This causes du, for example, to produce incorrect results, but otherwise should not be dangerous to the kernel. The in-kernel AFS client has been undergoing testing on opendev.org on one of their mirror machines. They are using AFS to hold data that is then served via apache, and Ian Wienand had reported seeing oopses, spontaneous machine reboots and updates to volumes going missing. This patch series appears to have fixed the problem, very probably due to patch (2), but it's not 100% certain. Reviewed-by: Jeffrey Altman Tested-by: Marc Dionne Tested-by: Ian Wienand
Re: [GIT PULL] AFS fixes and development
The pull request you sent on Tue, 07 May 2019 21:11:17 +0100: > git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git > tags/afs-next-20190507 has been merged into torvalds/linux.git: https://git.kernel.org/torvalds/c/e5fef2a9732580c5bd30c0097f5e9091a3d58ce5 Thank you! -- Deet-doot-dot, I am a bot. https://korg.wiki.kernel.org/userdoc/prtracker
[GIT PULL] AFS fixes and development
Hi Linus, The following are a set of fix and development patches for AFS for 5.2. To summarise: (1) Fix the AFS file locking so that sqlite can run on an AFS mount and also so that firefox and gnome can use a homedir that's mounted through AFS. This required emulation of fine-grained locking when the server will only support whole-file locks and no upgrade/downgrade. Four modes are provided, settable by mount parameter: "flock=local" - No reference to the server "flock=openafs" - Fine-grained locks are local-only, whole-file locks require sufficient server locks "flock=strict" - All locks require sufficient server locks "flock=write" - Always get an exclusive server lock If the volume is a read-only or backup volume, then flock=local for that volume. (2) Log extra information for a couple of cases where the client mucks up somehow: AFS vnode with undefined type and dir check failure - in both cases we seem to end up with unfilled data, but the issues happen infrequently and are difficult to reproduce at will. (3) Implement silly rename for unlink() and rename(). (4) Set i_blocks so that du can get some information about usage. (5) Fix xattr handlers to return the right amount of data and to not overflow buffers. (6) Implement getting/setting raw AFS and YFS ACLs as xattrs. Tested-by: Jonathan Billings Signed-off-by: David Howells --- The following changes since commit cd8dead0c39457e58ec1d36db93aedca811d48f1: Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net (2019-04-24 16:18:59 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git tags/afs-next-20190507 for you to fetch changes up to f5e4546347bc847be30b3cf904db5fc874b3c5dc: afs: Implement YFS ACL setting (2019-05-07 16:48:44 +0100) AFS Development David Howells (17): afs: Split wait from afs_make_call() afs: Calculate lock extend timer from set/extend reply reception afs: Fix AFS file locking to allow fine grained locks afs: Further fix file locking afs: Add file locking tracepoints afs: Improve dir check failure reports afs: Handle lock rpc ops failing on a file that got deleted afs: Add directory reload tracepoint afs: Implement sillyrename for unlink and rename afs: Add more tracepoints afs: Provide mount-time configurable byte-range file locking emulation afs: Log more information for "kAFS: AFS vnode with undefined type\n" afs: Fix the afs.cell and afs.volume xattr handlers afs: Fix getting the afs.fid xattr afs: Get an AFS3 ACL as an xattr afs: Get YFS ACLs and information through xattrs afs: Implement YFS ACL setting Joe Gorse (1): afs: implement acl setting Marc Dionne (1): afs: Calculate i_blocks based on file size fs/afs/Makefile| 1 + fs/afs/afs_fs.h| 2 + fs/afs/dir.c | 167 ++-- fs/afs/dir_silly.c | 239 ++ fs/afs/flock.c | 616 - fs/afs/fs_probe.c | 13 +- fs/afs/fsclient.c | 277 +--- fs/afs/inode.c | 43 +++- fs/afs/internal.h | 64 - fs/afs/protocol_yfs.h | 6 +- fs/afs/rxrpc.c | 33 ++- fs/afs/super.c | 34 ++- fs/afs/vl_probe.c | 14 +- fs/afs/vlclient.c | 26 +- fs/afs/xattr.c | 270 +++- fs/afs/yfsclient.c | 329 +--- include/linux/fs.h | 1 + include/trace/events/afs.h | 385 +++- 18 files changed, 2142 insertions(+), 378 deletions(-) create mode 100644 fs/afs/dir_silly.c
Re: [GIT PULL] afs: Fixes
The pull request you sent on Thu, 18 Apr 2019 10:17:32 +0100: > git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git > tags/afs-fixes-20190413 has been merged into torvalds/linux.git: https://git.kernel.org/torvalds/c/2a852fd1ac893d75879923025306f146b7e0747e Thank you! -- Deet-doot-dot, I am a bot. https://korg.wiki.kernel.org/userdoc/prtracker
[GIT PULL] afs: Fixes
Hi Linus, Could you pull these afs fixes please? There are five: (1) Stop using the deprecated get_seconds(). (2) Don't make tracepoint strings const as the section they go in isn't read-only. (3) Differentiate failure due to unmarshalling from other failure cases - we shouldn't abort with RXGEN_CC/SS_UNMARSHAL if it's not due to unmarshalling. (4) Add a missing unlock_page(). (5) Fix the interaction between receiving a notification from a server that it has invalidated all outstanding callback promises and a client call that we're in the middle of making that will get a new promise. Tested-by: Jonathan Billings Thanks, David --- The following changes since commit ed0de45a1008991fdaa27a0152befcb74d126a8b: ipv4: recompile ip options in ipv4_link_failure (2019-04-12 17:23:46 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git tags/afs-fixes-20190413 for you to fetch changes up to eeba1e9cf31d064284dd1fa7bd6cfe01395bd03d: afs: Fix in-progess ops to ignore server-level callback invalidation (2019-04-13 08:37:37 +0100) AFS fixes Andi Kleen (1): afs: Avoid section confusion in CM_NAME Arnd Bergmann (1): afs: avoid deprecated get_seconds() David Howells (2): afs: Differentiate abort due to unmarshalling from other errors afs: Fix in-progess ops to ignore server-level callback invalidation Marc Dionne (1): afs: Unlock pages for __pagevec_release() fs/afs/callback.c | 3 +-- fs/afs/cmservice.c | 2 +- fs/afs/inode.c | 4 +--- fs/afs/internal.h | 4 +--- fs/afs/rxrpc.c | 6 +- fs/afs/server.c| 1 - fs/afs/write.c | 1 + 7 files changed, 10 insertions(+), 11 deletions(-)
Re: [GIT PULL] AFS fixes and other bits
Linus Torvalds wrote: > The thing hasn't even seen a compiler, and when you *do* show the code > to a compiler, said compiler correctly warns about No, it doesn't - at least not gcc-8.3.1 on Fedora 29. What compiler are you using? > afs_do_silly_unlink() potentially returning an uninitialized variable. Not only has it seen a compiler, it's also been tested numerous times, with firefox, sqlite and some commands that tests file locking manually. But please hold off for now. I've just tried the system flock program - and that causes a lock leakage message for some reason that needs investigating. David
Re: [GIT PULL] AFS fixes and other bits
On Fri, Mar 15, 2019 at 4:41 PM David Howells wrote: > > Here's a set of fixes and other bits for AFS to improve the life of desktop > applications such as firefox. I pulled, and immediately unpulled. The thing hasn't even seen a compiler, and when you *do* show the code to a compiler, said compiler correctly warns about afs_do_silly_unlink() potentially returning an uninitialized variable. And yes, it's _trivially_ and obviously uninitialized. Feel free to submit this for 5.2 after it has actually seen testing. But this late in the 5.1 merge window, I'm no longer interested in totally untested new crap. It clearly wasn't ready before the merge window, and it clearly isn't ready *now*. Much too late to try to fix this up, Linus
[GIT PULL] AFS fixes and other bits
Hi Linus, Here's a set of fixes and other bits for AFS to improve the life of desktop applications such as firefox. It makes the following improvements/fixes: (1) Fix file locking to allow fine-grained locking, as required by firefox and sqlite, with the caveat that you can't get a partial write lock on a file if you first get a partial read lock on it as the AFS protocol only supports whole-file locks. [At some point I need to look at how best to emulate OpenAFS's behaviour whereby all partial locks are just granted without reference to the kernel lock - maybe with some sort of mount parameter to enable it.] (2) Fix the way the server lock, once obtained, is distributed to the local processes waiting for it, thereby making sure they wait if they can't be immediately granted a local lock. (3) Fix the nonappearance of afs locks in /proc/locks. (4) Fix the handling of asynchronous file lock RPC operation failure on files that get deleted whilst the lock is being extended or released. (5) Implement silly-rename. (6) Fix the file lock expiry calculation to be based on the time the set- or extend-lock reply is seen (packet timestamp) rather than by the time at which we've done processing the operation (wallclock time) to allow for delays in processing. (7) Split the synchronous wait out of the client call dispatcher to make it possible to convert synchronous calls into async calls in the event of a signal. The series also: (1) Adds/modifies a number of tracepoints, mostly related to file locking and silly rename. (2) Provides a more comprehensive data dump in the event that a directory content check fails. David --- The following changes since commit ebc551f2b8f905eca0e25c476c1e5c098cd92103: Merge tag 'nfsd-5.1' of git://linux-nfs.org/~bfields/linux (2019-03-12 15:06:54 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git tags/afs-fixes-20190315 for you to fetch changes up to 78cf74c367eb608e8a15167da5aeae2c8abed902: afs: Add more tracepoints (2019-03-15 22:06:31 +) AFS fixes David Howells (10): afs: Split wait from afs_make_call() afs: Calculate lock extend timer from set/extend reply reception afs: Fix AFS file locking to allow fine grained locks afs: Further fix file locking afs: Add file locking tracepoints afs: Improve dir check failure reports afs: Handle lock rpc ops failing on a file that got deleted afs: Add directory reload tracepoint afs: Implement sillyrename for unlink and rename afs: Add more tracepoints fs/afs/Makefile| 1 + fs/afs/dir.c | 161 +++-- fs/afs/dir_silly.c | 239 +++ fs/afs/flock.c | 569 +++-- fs/afs/fs_probe.c | 13 +- fs/afs/fsclient.c | 92 +--- fs/afs/inode.c | 2 + fs/afs/internal.h | 25 +- fs/afs/rxrpc.c | 33 ++- fs/afs/super.c | 5 +- fs/afs/vl_probe.c | 14 +- fs/afs/vlclient.c | 26 ++- fs/afs/yfsclient.c | 72 -- include/linux/fs.h | 1 + include/trace/events/afs.h | 348 ++- 15 files changed, 1258 insertions(+), 343 deletions(-) create mode 100644 fs/afs/dir_silly.c
[GIT PULL] AFS fixes and other bits
Hi Linus, Here's a set of fixes and other bits for AFS to improve the life of desktop applications such as firefox. It makes the following improvements/fixes: (1) Fix file locking to allow fine-grained locking, as required by firefox and sqlite, with the caveat that you can't get a partial write lock on a file if you first get a partial read lock on it as the AFS protocol only supports whole-file locks. [At some point I need to look at how best to emulate OpenAFS's behaviour whereby all partial locks are just granted without reference to the kernel lock - maybe with some sort of mount parameter to enable it.] (2) Fix the way the server lock, once obtained, is distributed to the local processes waiting for it, thereby making sure they wait if they can't be immediately granted a local lock. (3) Fix the nonappearance of afs locks in /proc/locks. (4) Fix the handling of asynchronous file lock RPC operation failure on files that get deleted whilst the lock is being extended or released. (5) Implement silly-rename. (6) Fix the file lock expiry calculation to be based on the time the set- or extend-lock reply is seen (packet timestamp) rather than by the time at which we've done processing the operation (wallclock time) to allow for delays in processing. (7) Split the synchronous wait out of the client call dispatcher to make it possible to convert synchronous calls into async calls in the event of a signal. The series also: (1) Adds/modifies a number of tracepoints, mostly related to file locking and silly rename. (2) Provides a more comprehensive data dump in the event that a directory content check fails. David --- The following changes since commit ebc551f2b8f905eca0e25c476c1e5c098cd92103: Merge tag 'nfsd-5.1' of git://linux-nfs.org/~bfields/linux (2019-03-12 15:06:54 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git tags/afs-fixes-20190315 for you to fetch changes up to 78cf74c367eb608e8a15167da5aeae2c8abed902: afs: Add more tracepoints (2019-03-15 22:06:31 +) AFS fixes David Howells (10): afs: Split wait from afs_make_call() afs: Calculate lock extend timer from set/extend reply reception afs: Fix AFS file locking to allow fine grained locks afs: Further fix file locking afs: Add file locking tracepoints afs: Improve dir check failure reports afs: Handle lock rpc ops failing on a file that got deleted afs: Add directory reload tracepoint afs: Implement sillyrename for unlink and rename afs: Add more tracepoints fs/afs/Makefile| 1 + fs/afs/dir.c | 161 +++-- fs/afs/dir_silly.c | 239 +++ fs/afs/flock.c | 569 +++-- fs/afs/fs_probe.c | 13 +- fs/afs/fsclient.c | 92 +--- fs/afs/inode.c | 2 + fs/afs/internal.h | 25 +- fs/afs/rxrpc.c | 33 ++- fs/afs/super.c | 5 +- fs/afs/vl_probe.c | 14 +- fs/afs/vlclient.c | 26 ++- fs/afs/yfsclient.c | 72 -- include/linux/fs.h | 1 + include/trace/events/afs.h | 348 ++- 15 files changed, 1258 insertions(+), 343 deletions(-) create mode 100644 fs/afs/dir_silly.c
[GIT PULL] afs: Fixes
Hi Linus, Here's a set of patches that fix a number of bugs in the in-kernel AFS client, including: (1) Fix directory locking to not use individual page locks for directory reading/scanning but rather to use a semaphore on the afs_vnode struct as the directory contents must be read in a single blob and data from different reads must not be mixed as the entire contents may be shuffled about between reads. (2) Fix address list parsing to handle port specifiers correctly. (3) Only give up callback records on a server if we actually talked to that server (we might not be able to access a server). (4) Fix some callback handling bugs, including refcounting, whole-volume callbacks and when callbacks actually get broken in response to a CB.CallBack op. (5) Fix some server/address rotation bugs, including giving up if we can't probe a server; giving up if a server says it doesn't have a volume, but there are more servers to try. (6) Fix the decoding of fetched statuses to be OpenAFS compatible. (7) Fix the handling of server lookups in Cache Manager ops (such as CB.InitCallBackState3) to use a UUID if possible and to handle no server being found. (8) Fix a bug in server lookup where not all addresses are compared. (9) Fix the non-encryption of calls that prevents some servers from being accessed (this also requires an AF_RXRPC patch that has already gone in through the net tree). There's also a patch that adds tracepoints to log Cache Manager ops that don't find a matching server, either by UUID or by address. Thanks, David --- The following changes since commit f0ab773f5c96c29a5227234c4b5a820f5591b74d: Merge branch 'akpm' (patches from Andrew) (2018-05-11 18:04:12 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git tags/afs-fixes-20180514 for you to fetch changes up to 4776cab43fd3111618112737a257dc3ef368eddd: afs: Fix the non-encryption of calls (2018-05-14 15:15:19 +0100) AFS fixes David Howells (13): afs: Fix directory page locking afs: Fix address list parsing afs: Fix giving up callbacks on server destruction afs: Fix refcounting in callback registration afs: Fix server rotation's handling of fileserver probe failure afs: Fix AFSFetchStatus decoder to provide OpenAFS compatibility afs: Fix VNOVOL handling in address rotation afs: Fix the handling of CB.InitCallBackState3 to find the server by UUID afs: Add a tracepoint to record callbacks from unlisted servers afs: Fix the handling of an unfound server in CM operations afs: Fix whole-volume callback handling afs: Fix CB.CallBack handling afs: Fix the non-encryption of calls Marc Dionne (1): afs: Fix afs_find_server search loop fs/afs/addr_list.c | 25 -- fs/afs/callback.c | 84 +- fs/afs/cmservice.c | 67 fs/afs/dir.c | 54 ++--- fs/afs/file.c | 2 +- fs/afs/flock.c | 6 ++-- fs/afs/fsclient.c | 31 - fs/afs/inode.c | 19 ++- fs/afs/internal.h | 25 -- fs/afs/rotate.c| 20 +++ fs/afs/rxrpc.c | 18 ++ fs/afs/security.c | 7 ++-- fs/afs/server.c| 21 +++- fs/afs/server_list.c | 7 ++-- fs/afs/super.c | 4 +-- fs/afs/write.c | 2 +- include/trace/events/afs.h | 42 +++ 17 files changed, 266 insertions(+), 168 deletions(-)
Re: [GIT PULL] afs: Fixes
Linus Torvalds wrote: > However, even when you do that, the page can be writable in other > mappings. At least fork(), for example, only clears the dirty bit, > doesn't mark it write-protected. I assumed the rmap walk done by page_mkclean() would take care of that but I'm not really clear on what the code does. > I just hope that the inconsistency isn't fatal to the afs client or > server code. For example, if you retry writes forever when a checksum > were to not match the data, that would be bad. Shouldn't be a problem for the the in-Linux client. Data is copied into sk_bufs preparatory to doing further things to it like checksumming, encryption or transmission (actually, in future, I would like to use the encryption process to save on the copy, but this shouldn't bother that either). AFAIK, the servers are all userspace jobs that don't let anyone else touch their storage so that they can maintain correctness on the data version number of each vnode. > so I just wanted to bring this up as a potential issue, not > necessarily as a big problem. Thanks. David
Re: [GIT PULL] afs: Fixes
On Sat, Nov 25, 2017 at 12:35 PM, David Howells wrote: > > Doesn't clear_page_dirty_for_io() write-protect the PTE for the page to be > written out, in which case ->page_mkwrite() will get called again before the > page is redirtied? No, it literally just sets the dirty bit (and does accounting). But I think you may be right that we always write-protect he page when we move the dirty bit from the page tables to the 'struct page' (page_mkclean_one()). However, even when you do that, the page can be writable in other mappings. At least fork(), for example, only clears the dirty bit, doesn't mark it write-protected. So there is some rate-limiting of dirty pages, but I do not believe that we've ever really *serialized* writes. >> (b) can cause some really nasty latency issues > > True, but I think the most common case is a file being opened, written start > to finish and then closed. Actually, the worst-handled thing I've seen is a > shell script appending a bunch of things to a file because ->flush() syncs the > file each time it is closed:-/ > > What would you recommend instead? I'm currently trying and keep track of what > needs to be written so that I only write what's changed to the server, rather > than writing only whole pages. I don't think that what you are doing is necessarily wrong, I'm just pointing out that you may still see mmap'ed pages being modified concurrently with the actual IO, and that this will potentially mean (for example) that things like checksums won't be reliably unless you do the checksum as you copy the data to a network packet or something. Of course, if that kind of inconsistency happens, a later write-back will also happen, and eventually fix it. So the server may see temporarily 'wrong' data, but it won't be final. I just hope that the inconsistency isn't fatal to the afs client or server code. For example, if you retry writes forever when a checksum were to not match the data, that would be bad. And yes, this can be (a) really hard to trigger in practice (b) very hard to debug due to a combination of very specific timing and behavior. so I just wanted to bring this up as a potential issue, not necessarily as a big problem. Linus
Re: [GIT PULL] afs: Fixes
On Sat, Nov 25, 2017 at 10:35:43PM +, David Howells wrote: > Linus Torvalds wrote: > > > So I see in the commit message why afs needs to do this, but it's > > worth pointing out that it's > > > > (a) impossible to avoid the "inconsistent data" case for writable mmap'ed > > pages > > Doesn't clear_page_dirty_for_io() write-protect the PTE for the page to be > written out, in which case ->page_mkwrite() will get called again before the > page is redirtied? Yes, but page_mkwrite will only block on writeback in progress is if the backing device says it needs stable pages. See wait_for_stable_page(). e.g. stable pages are required if RAID is in use, otherwise modification during IO can result in broken on-disk parity/mirroring Cheers, Dave. -- Dave Chinner da...@fromorbit.com
Re: [GIT PULL] afs: Fixes
Linus Torvalds wrote: > So I see in the commit message why afs needs to do this, but it's > worth pointing out that it's > > (a) impossible to avoid the "inconsistent data" case for writable mmap'ed > pages Doesn't clear_page_dirty_for_io() write-protect the PTE for the page to be written out, in which case ->page_mkwrite() will get called again before the page is redirtied? > (b) can cause some really nasty latency issues True, but I think the most common case is a file being opened, written start to finish and then closed. Actually, the worst-handled thing I've seen is a shell script appending a bunch of things to a file because ->flush() syncs the file each time it is closed:-/ What would you recommend instead? I'm currently trying and keep track of what needs to be written so that I only write what's changed to the server, rather than writing only whole pages. David
Re: [GIT PULL] afs: Fixes
On Fri, Nov 24, 2017 at 4:22 AM, David Howells wrote: > > (2) Don't write to a page that's being written out, but wait for it to > complete. So I see in the commit message why afs needs to do this, but it's worth pointing out that it's (a) impossible to avoid the "inconsistent data" case for writable mmap'ed pages (b) can cause some really nasty latency issues So the "page->private" issue does mean that write-vs-writeback clearly needs that serialization (and that's not an issue for the mapped page being changed at the same time), but in general filesystem people should be aware that data under writeback may still be further dirtied while the writeback is active (and the page marked dirty again). It can obviously screw with things like checksums etc, and make for temporarily inconsistent state if the filesystem does those kinds of things. Linus
[GIT PULL] afs: Fixes
Hi Linus, Can you pull these fixes for the AFS filesystem? They include: (1) Make AFS file locking work again. (2) Don't write to a page that's being written out, but wait for it to complete. (3) Do d_drop() and d_add() in the right places. (4) Put keys on error paths. (5) Remove some redundant code. Thanks, David --- The following changes since commit cf9b0772f2e410645fece13b749bd56505b998b8: Merge tag 'armsoc-drivers' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc (2017-11-16 16:05:01 -0800) are available in the git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git tags/afs-fixes-20171124 for you to fetch changes up to 43dd388b21c722616413781bd434522376e117cc: afs: remove redundant assignment of dvnode to itself (2017-11-24 13:55:46 +) AFS fixes Colin Ian King (1): afs: remove redundant assignment of dvnode to itself David Howells (4): afs: Fix file locking afs: Make afs_write_begin() avoid writing to a page that's being stored afs: Fix some dentry handling in dir ops and missing key_puts afs: Fix signal handling in some file ops Gustavo A. R. Silva (1): afs: cell: Remove unnecessary code in afs_lookup_cell fs/afs/cell.c| 7 +- fs/afs/dir.c | 25 +-- fs/afs/flock.c | 548 +-- fs/afs/internal.h| 23 ++- fs/afs/rotate.c | 70 +-- fs/afs/security.c| 4 +- fs/afs/server_list.c | 2 +- fs/afs/write.c | 5 + 8 files changed, 405 insertions(+), 279 deletions(-)
Re: [GIT PULL] AFS fixes
On Fri, Mar 17, 2017 at 8:29 AM, David Howells wrote: > > Could you pull these fixes to the AFS filesystem in the kernel please? I took this, but honestly, most of the patches look like "merge window" patches rather than -rc patches. Exactly _one_ of the patches was marked for stable. The rest look like "yes, they are technically bugs, but this is general afs development". Linus
[GIT PULL] AFS fixes
Hi Linus, Could you pull these fixes to the AFS filesystem in the kernel please? They fix a variety of bugs. These include some issues fixed for consistency with other AFS implementations: (*) Handle AFS mode bits better. (*) Use the client mtime rather than the server mtime in the protocol. (*) Handle the server returning more or less data than was requested in a FetchData call. (*) Distinguish mountpoints from symlinks based on the mode bits rather than preemptively reading every symlink to find out what it actually represents. One other notable change for the user is that files are now flushed on close analogously with other network filesystems. Thanks, David --- The following changes since commit 69eea5a4ab9c705496e912b55a9d312325de19e6: Merge branch 'for-linus' of git://git.kernel.dk/linux-block (2017-03-15 16:54:58 -0700) are available in the git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git tags/afs-20170316 for you to fetch changes up to c5051c7bc777dffa5661569dec5997f432b9a34a: afs: Don't wait for page writeback with the page lock held (2017-03-16 16:29:30 +) kAFS fixes Andreea-Cristina Bernat (2): afs: inode: Replace rcu_assign_pointer() with RCU_INIT_POINTER() afs: security: Replace rcu_assign_pointer() with RCU_INIT_POINTER() David Howells (20): afs: Fix missing put_page() afs: Fix page overput in afs_fill_page() afs: Handle better the server returning excess or short data afs: Kill struct afs_read::pg_offset afs: Handle a short write to an AFS page afs: Flush outstanding writes when an fd is closed afs: Distinguish mountpoints from symlinks by file mode alone afs: Fix AFS read bug afs: Make struct afs_read::remain 64-bit afs: Use a bvec rather than a kvec in afs_send_pages() afs: Fix the maths in afs_fs_store_data() afs: Invalid op ID should abort with RXGEN_OPCODE afs: Better abort and net error handling afs: Don't set PG_error on local EINTR or ENOMEM when filling a page afs: Fix page leak in afs_write_begin() afs: Fix afs_kill_pages() afs: Fix an off-by-one error in afs_send_pages() afs: Fix abort on signal while waiting for call completion afs: ->writepage() shouldn't call clear_page_dirty_for_io() afs: Don't wait for page writeback with the page lock held Marc Dionne (4): afs: Populate group ID from vnode status afs: Adjust mode bits processing afs: Deal with an empty callback array afs: Populate and use client modification time Tina Ruchandani (2): afs: Migrate vlocation fields to 64-bit afs: Prevent callback expiry timer overflow fs/afs/callback.c | 7 +-- fs/afs/cmservice.c | 11 ++-- fs/afs/file.c | 20 +-- fs/afs/fsclient.c | 77 +-- fs/afs/inode.c | 42 --- fs/afs/internal.h | 23 + fs/afs/misc.c | 2 + fs/afs/mntpt.c | 53 --- fs/afs/rxrpc.c | 149 ++--- fs/afs/security.c | 9 +++- fs/afs/server.c| 6 +-- fs/afs/vlocation.c | 16 +++--- fs/afs/write.c | 76 +++ 13 files changed, 269 insertions(+), 222 deletions(-)
