Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)
On 2021-May-20, at 22:19, Rick Macklem wrote: > Ok, so it isn't related to "soft". > I am wondering if it is something specific to what > "diff -r" does? > > Could you try: > # cd /usr/ports > # ls -R > /tmp/x > # cd /mnt > # ls -R > /tmp/y > # cd /tmp > # diff -u -p x y > --> To see if "ls -R" finds any difference? > # diff -u -p x y --- x 2021-05-20 22:35:48.021663000 -0700 +++ y 2021-05-20 22:39:03.691936000 -0700 @@ -227209,10 +227209,10 @@ patch-chrome_browser_background_background__mode__mana patch-chrome_browser_background_background__mode__optimizer.cc patch-chrome_browser_browser__resources.grd patch-chrome_browser_browsing__data_chrome__browsing__data__remover__delegate.cc +patch-chrome_browser_chrome__browser patch-chrome_browser_chrome__browser__interface__binders.cc patch-chrome_browser_chrome__browser__main.cc patch-chrome_browser_chrome__browser__main__linux.cc -patch-chrome_browser_chrome__browser__main__posix.cc patch-chrome_browser_chrome__content__browser__client.cc patch-chrome_browser_chrome__content__browser__client.h patch-chrome_browser_crash__upload__list_crash__upload__list.cc # find /usr/ports/ -name 'patch-chrome_browser_chrome__browser*' -print | more /usr/ports/devel/electron12/files/patch-chrome_browser_chrome__browser__main__linux.cc /usr/ports/devel/electron12/files/patch-chrome_browser_chrome__browser__main.cc /usr/ports/devel/electron12/files/patch-chrome_browser_chrome__browser__main__posix.cc /usr/ports/devel/electron12/files/patch-chrome_browser_chrome__browser__interface__binders.cc /usr/ports/www/chromium/files/patch-chrome_browser_chrome__browser__main__posix.cc /usr/ports/www/chromium/files/patch-chrome_browser_chrome__browser__main.cc /usr/ports/www/chromium/files/patch-chrome_browser_chrome__browser__main__linux.cc /usr/ports/www/chromium/files/patch-chrome_browser_chrome__browser__interface__binders.cc find /mnt/ -name 'patch-chrome_browser_chrome__browser*' -print | more /mnt/devel/electron12/files/patch-chrome_browser_chrome__browser__main__linux.cc /mnt/devel/electron12/files/patch-chrome_browser_chrome__browser__main.cc /mnt/devel/electron12/files/patch-chrome_browser_chrome__browser__main__posix.cc /mnt/devel/electron12/files/patch-chrome_browser_chrome__browser__interface__binders.cc /mnt/www/chromium/files/patch-chrome_browser_chrome__browser /mnt/www/chromium/files/patch-chrome_browser_chrome__browser__main.cc /mnt/www/chromium/files/patch-chrome_browser_chrome__browser__main__linux.cc /mnt/www/chromium/files/patch-chrome_browser_chrome__browser__interface__binders.cc So: patch-chrome_browser_chrome__browser appears to be a truncated: patch-chrome_browser_chrome__browser__main__posix.cc file name and find also gets the same oddity. (Note: This had /usr/ports in a main context and /mnt/ referring to a release/13.0.0 context.) > ps: I do not think that r367492 could cause this, but it would be > nice if you try a kernel with the r367492 patch reverted. > It is currently in all of releng13, stable13 and main, although > the patch to fix this is was just reviewed and may hit main soon. Do you want a debug kernel to be used? Do you have a preference for main vs. stable/13 vs. release/13.0.0 based? Is it okay to stick to the base version things are now based on --or do you want me to update to more recent? (That last only applies if main or stable/13 is to be put to use.) > . . . old history deleted . . . === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)
Ok, so it isn't related to "soft". I am wondering if it is something specific to what "diff -r" does? Could you try: # cd /usr/ports # ls -R > /tmp/x # cd /mnt # ls -R > /tmp/y # cd /tmp # diff -u -p x y --> To see if "ls -R" finds any difference? rick ps: I do not think that r367492 could cause this, but it would be nice if you try a kernel with the r367492 patch reverted. It is currently in all of releng13, stable13 and main, although the patch to fix this is was just reviewed and may hit main soon. From: Mark Millard Sent: Friday, May 21, 2021 12:40 AM To: Rick Macklem Cc: FreeBSD-STABLE Mailing List Subject: Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context) CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to ith...@uoguelph.ca [main test example and main/releng/13 mixed example] On 2021-May-20, at 20:36, Mark Millard wrote: > [stable/13 test: example ends up being odder. That might > allow eliminating some potential alternatives.] > > On 2021-May-20, at 19:38, Mark Millard wrote: >> >> On 2021-May-20, at 18:09, Rick Macklem wrote: >>> >>> Oh, one additional thing that I'll dare to top post... >>> r367492 broke the TCP upcalls that the NFS server uses, such >>> that intermittent hangs of NFS mounts to FreeBSD13 servers can occur. >>> This has not yet been resolved in "main" etc and could explain >>> why an RPC could time out for a soft mount. >> >> See later notes that I added: soft mount is not required >> to see the problem. >> >>> You can revert the patch in r367492 to avoid the problem. >> >> If I understand right, you are indicating that this would >> not apply to the non-soft mount case that I got. >> >>> Disabling TSO, LRO are also de-facto standard things to do when >>> you observe weird NFS behaviour, because they are often broken >>> in various network device drivers. >> >> I'll have to figure out how to experiment with such. Things >> are at defaults rather generally on the systems. I'm not >> literate in the subject areas. >> >> I'm the only user of the machines and network. It is not >> outward facing. It is a rather small EtherNet network. >> >>> rick >>> >>> >>> From: owner-freebsd-sta...@freebsd.org >>> on behalf of Rick Macklem >>> Sent: Thursday, May 20, 2021 8:55 PM >>> To: FreeBSD-STABLE Mailing List; Mark Millard >>> Subject: Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs >>> (in a zfs file systems context) >>> >>> Mark Millard wrote: [I warn that I'm a fairly minimal user of NFS mounts, not knowing all that much. I'm mostly reporting this in case it ends up as evidence via eventually matching up with others observing possibly related oddities.] I got the following odd sequence (that I've mixed notes into). It involved a diff -r over NFS showing differences (files missing) and then a later diff finding matches for the same files, no file system changes made on either machine. I'm unable to reproduce the oddity on demand. Note: A larger scope diff -r originally returned the below as well, but doing the narrower diff -r did repeat the result and that is what I show. (I make no use of devel/ice .) # diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more Only in /usr/ports/devel/ice/files: Make.rules.FreeBSD >> . . . Only in /usr/ports/devel/ice/files: patch-scripts-TestUtil.py Note: The above was not expected. So I tried: # ls -Tld /mnt/devel/ice/files/* -rw-r--r-- 1 root wheel 755 Apr 21 21:07:54 2021 /mnt/devel/ice/files/Make.rules.FreeBSD >> . . . -rw-r--r-- 1 root wheel 2588 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-scripts-TestUtil.py Note: So that indicated that the files were there on the machine that /mnt references. So attempting the original diff -r again: # diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more # (Empty difference.) Note: So after the explicit "ls -Tld /mnt/devel/ice/files/*" the odd result of the diff -r no longer happened: no differences reported. For reference (both machines reported): . . . The original mount command was on CA72_16Gp_ZFS: # mount -onoatime,soft 192.168.1.170:/usr/ports/ /mnt/ >>> The likely explanation for this is your use of a "soft" mount. >>> - If the NFS server is slow to respond or there is a temporary network >>> issue, >>> the RPC request can time out and then the >>> syscall can fail with EINT/ETIMEDOUT. Since almost nothing, including the >>> readdir(3) libc functions expect syscalls to fail this way...
Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)
[Direct drive connection to machine: no problem.] On 2021-May-20, at 21:40, Mark Millard wrote: > [main test example and main/releng/13 mixed example] > > On 2021-May-20, at 20:36, Mark Millard wrote: > >> [stable/13 test: example ends up being odder. That might >> allow eliminating some potential alternatives.] >> >> On 2021-May-20, at 19:38, Mark Millard wrote: >>> >>> On 2021-May-20, at 18:09, Rick Macklem wrote: Oh, one additional thing that I'll dare to top post... r367492 broke the TCP upcalls that the NFS server uses, such that intermittent hangs of NFS mounts to FreeBSD13 servers can occur. This has not yet been resolved in "main" etc and could explain why an RPC could time out for a soft mount. >>> >>> See later notes that I added: soft mount is not required >>> to see the problem. >>> You can revert the patch in r367492 to avoid the problem. >>> >>> If I understand right, you are indicating that this would >>> not apply to the non-soft mount case that I got. >>> Disabling TSO, LRO are also de-facto standard things to do when you observe weird NFS behaviour, because they are often broken in various network device drivers. >>> >>> I'll have to figure out how to experiment with such. Things >>> are at defaults rather generally on the systems. I'm not >>> literate in the subject areas. >>> >>> I'm the only user of the machines and network. It is not >>> outward facing. It is a rather small EtherNet network. >>> rick From: owner-freebsd-sta...@freebsd.org on behalf of Rick Macklem Sent: Thursday, May 20, 2021 8:55 PM To: FreeBSD-STABLE Mailing List; Mark Millard Subject: Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context) Mark Millard wrote: > [I warn that I'm a fairly minimal user of NFS > mounts, not knowing all that much. I'm mostly > reporting this in case it ends up as evidence > via eventually matching up with others observing > possibly related oddities.] > > I got the following odd sequence (that I've > mixed notes into). It involved a diff -r over NFS > showing differences (files missing) and then a > later diff finding matches for the same files, > no file system changes made on either machine. > I'm unable to reproduce the oddity on demand. > > Note: A larger scope diff -r originally returned the > below as well, but doing the narrower diff -r did > repeat the result and that is what I show. (I > make no use of devel/ice .) > > # diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more > Only in /usr/ports/devel/ice/files: Make.rules.FreeBSD >>> . . . > Only in /usr/ports/devel/ice/files: patch-scripts-TestUtil.py > > Note: The above was not expected. So I tried: > > # ls -Tld /mnt/devel/ice/files/* > -rw-r--r-- 1 root wheel 755 Apr 21 21:07:54 2021 > /mnt/devel/ice/files/Make.rules.FreeBSD >>> . . . > -rw-r--r-- 1 root wheel 2588 Apr 21 21:07:54 2021 > /mnt/devel/ice/files/patch-scripts-TestUtil.py > > Note: So that indicated that the files were there on the > machine that /mnt references. So attempting the original > diff -r again: > > # diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more > # > > (Empty difference.) > > Note: So after the explicit "ls -Tld /mnt/devel/ice/files/*" > the odd result of the diff -r no longer happened: no > differences reported. > > > > For reference (both machines reported): > > . . . > The original mount command was on CA72_16Gp_ZFS: > > # mount -onoatime,soft 192.168.1.170:/usr/ports/ /mnt/ The likely explanation for this is your use of a "soft" mount. - If the NFS server is slow to respond or there is a temporary network issue, the RPC request can time out and then the syscall can fail with EINT/ETIMEDOUT. Since almost nothing, including the readdir(3) libc functions expect syscalls to fail this way... Then the cached directory is messed up. Doing the "ls" read the directory again and fixed the problem. Try to reproduce it for a mount without the "soft" option. (If a mount point is hung, due to an unresponsive server "umount -N /mnt" can usually get rid of it.) Personally, I thought "soft" was a bad idea when Sun introduced it in NFS in 1985 and I still feel that way. --> If you can reproduce it without "soft" then I can't explain it. To be honest, the directory reading/caching code in the NFSv3 client hasn't changed significantly in literally decades, as far as I can remember. >>> >>> Well . . . trying an even wider scope diff than >>> the original . . . >>> >>> # umount /mnt/ >>> # mount -onoatime
Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)
[main test example and main/releng/13 mixed example] On 2021-May-20, at 20:36, Mark Millard wrote: > [stable/13 test: example ends up being odder. That might > allow eliminating some potential alternatives.] > > On 2021-May-20, at 19:38, Mark Millard wrote: >> >> On 2021-May-20, at 18:09, Rick Macklem wrote: >>> >>> Oh, one additional thing that I'll dare to top post... >>> r367492 broke the TCP upcalls that the NFS server uses, such >>> that intermittent hangs of NFS mounts to FreeBSD13 servers can occur. >>> This has not yet been resolved in "main" etc and could explain >>> why an RPC could time out for a soft mount. >> >> See later notes that I added: soft mount is not required >> to see the problem. >> >>> You can revert the patch in r367492 to avoid the problem. >> >> If I understand right, you are indicating that this would >> not apply to the non-soft mount case that I got. >> >>> Disabling TSO, LRO are also de-facto standard things to do when >>> you observe weird NFS behaviour, because they are often broken >>> in various network device drivers. >> >> I'll have to figure out how to experiment with such. Things >> are at defaults rather generally on the systems. I'm not >> literate in the subject areas. >> >> I'm the only user of the machines and network. It is not >> outward facing. It is a rather small EtherNet network. >> >>> rick >>> >>> >>> From: owner-freebsd-sta...@freebsd.org >>> on behalf of Rick Macklem >>> Sent: Thursday, May 20, 2021 8:55 PM >>> To: FreeBSD-STABLE Mailing List; Mark Millard >>> Subject: Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs >>> (in a zfs file systems context) >>> >>> Mark Millard wrote: [I warn that I'm a fairly minimal user of NFS mounts, not knowing all that much. I'm mostly reporting this in case it ends up as evidence via eventually matching up with others observing possibly related oddities.] I got the following odd sequence (that I've mixed notes into). It involved a diff -r over NFS showing differences (files missing) and then a later diff finding matches for the same files, no file system changes made on either machine. I'm unable to reproduce the oddity on demand. Note: A larger scope diff -r originally returned the below as well, but doing the narrower diff -r did repeat the result and that is what I show. (I make no use of devel/ice .) # diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more Only in /usr/ports/devel/ice/files: Make.rules.FreeBSD >> . . . Only in /usr/ports/devel/ice/files: patch-scripts-TestUtil.py Note: The above was not expected. So I tried: # ls -Tld /mnt/devel/ice/files/* -rw-r--r-- 1 root wheel 755 Apr 21 21:07:54 2021 /mnt/devel/ice/files/Make.rules.FreeBSD >> . . . -rw-r--r-- 1 root wheel 2588 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-scripts-TestUtil.py Note: So that indicated that the files were there on the machine that /mnt references. So attempting the original diff -r again: # diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more # (Empty difference.) Note: So after the explicit "ls -Tld /mnt/devel/ice/files/*" the odd result of the diff -r no longer happened: no differences reported. For reference (both machines reported): . . . The original mount command was on CA72_16Gp_ZFS: # mount -onoatime,soft 192.168.1.170:/usr/ports/ /mnt/ >>> The likely explanation for this is your use of a "soft" mount. >>> - If the NFS server is slow to respond or there is a temporary network >>> issue, >>> the RPC request can time out and then the >>> syscall can fail with EINT/ETIMEDOUT. Since almost nothing, including the >>> readdir(3) libc functions expect syscalls to fail this way... >>> Then the cached directory is messed up. >>> Doing the "ls" read the directory again and fixed the problem. >>> >>> Try to reproduce it for a mount without the "soft" option. >>> (If a mount point is hung, due to an unresponsive server "umount -N /mnt" >>> can usually get rid of it.) >>> Personally, I thought "soft" was a bad idea when Sun introduced it in NFS >>> in 1985 >>> and I still feel that way. >>> --> If you can reproduce it without "soft" then I can't explain it. >>>To be honest, the directory reading/caching code in the NFSv3 client >>>hasn't changed significantly in literally decades, as far as I can >>> remember. >> >> Well . . . trying an even wider scope diff than >> the original . . . >> >> # umount /mnt/ >> # mount -onoatime 192.168.1.170:/usr/ports/ /mnt/ >> # diff -r /usr/ports/ /mnt/ | more >> Only in /mnt/databases/mongodb42/files/aarch64: >> patch-src_third__party_mozjs-60_ >> Only in /usr/ports/databases/mongodb42/files/aarch64: >>
Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)
[stable/13 test: example ends up being odder. That might allow eliminating some potential alternatives.] On 2021-May-20, at 19:38, Mark Millard wrote: > > On 2021-May-20, at 18:09, Rick Macklem wrote: >> >> Oh, one additional thing that I'll dare to top post... >> r367492 broke the TCP upcalls that the NFS server uses, such >> that intermittent hangs of NFS mounts to FreeBSD13 servers can occur. >> This has not yet been resolved in "main" etc and could explain >> why an RPC could time out for a soft mount. > > See later notes that I added: soft mount is not required > to see the problem. > >> You can revert the patch in r367492 to avoid the problem. > > If I understand right, you are indicating that this would > not apply to the non-soft mount case that I got. > >> Disabling TSO, LRO are also de-facto standard things to do when >> you observe weird NFS behaviour, because they are often broken >> in various network device drivers. > > I'll have to figure out how to experiment with such. Things > are at defaults rather generally on the systems. I'm not > literate in the subject areas. > > I'm the only user of the machines and network. It is not > outward facing. It is a rather small EtherNet network. > >> rick >> >> >> From: owner-freebsd-sta...@freebsd.org on >> behalf of Rick Macklem >> Sent: Thursday, May 20, 2021 8:55 PM >> To: FreeBSD-STABLE Mailing List; Mark Millard >> Subject: Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs >> (in a zfs file systems context) >> >> Mark Millard wrote: >>> [I warn that I'm a fairly minimal user of NFS >>> mounts, not knowing all that much. I'm mostly >>> reporting this in case it ends up as evidence >>> via eventually matching up with others observing >>> possibly related oddities.] >>> >>> I got the following odd sequence (that I've >>> mixed notes into). It involved a diff -r over NFS >>> showing differences (files missing) and then a >>> later diff finding matches for the same files, >>> no file system changes made on either machine. >>> I'm unable to reproduce the oddity on demand. >>> >>> Note: A larger scope diff -r originally returned the >>> below as well, but doing the narrower diff -r did >>> repeat the result and that is what I show. (I >>> make no use of devel/ice .) >>> >>> # diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more >>> Only in /usr/ports/devel/ice/files: Make.rules.FreeBSD > . . . >>> Only in /usr/ports/devel/ice/files: patch-scripts-TestUtil.py >>> >>> Note: The above was not expected. So I tried: >>> >>> # ls -Tld /mnt/devel/ice/files/* >>> -rw-r--r-- 1 root wheel 755 Apr 21 21:07:54 2021 >>> /mnt/devel/ice/files/Make.rules.FreeBSD > . . . >>> -rw-r--r-- 1 root wheel 2588 Apr 21 21:07:54 2021 >>> /mnt/devel/ice/files/patch-scripts-TestUtil.py >>> >>> Note: So that indicated that the files were there on the >>> machine that /mnt references. So attempting the original >>> diff -r again: >>> >>> # diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more >>> # >>> >>> (Empty difference.) >>> >>> Note: So after the explicit "ls -Tld /mnt/devel/ice/files/*" >>> the odd result of the diff -r no longer happened: no >>> differences reported. >>> >>> >>> >>> For reference (both machines reported): >>> >>> . . . >>> The original mount command was on CA72_16Gp_ZFS: >>> >>> # mount -onoatime,soft 192.168.1.170:/usr/ports/ /mnt/ >> The likely explanation for this is your use of a "soft" mount. >> - If the NFS server is slow to respond or there is a temporary network issue, >> the RPC request can time out and then the >> syscall can fail with EINT/ETIMEDOUT. Since almost nothing, including the >> readdir(3) libc functions expect syscalls to fail this way... >> Then the cached directory is messed up. >> Doing the "ls" read the directory again and fixed the problem. >> >> Try to reproduce it for a mount without the "soft" option. >> (If a mount point is hung, due to an unresponsive server "umount -N /mnt" >> can usually get rid of it.) >> Personally, I thought "soft" was a bad idea when Sun introduced it in NFS in >> 1985 >> and I still feel that way. >> --> If you can reproduce it without "soft" then I can't explain it. >> To be honest, the directory reading/caching code in the NFSv3 client >> hasn't changed significantly in literally decades, as far as I can >> remember. > > Well . . . trying an even wider scope diff than > the original . . . > > # umount /mnt/ > # mount -onoatime 192.168.1.170:/usr/ports/ /mnt/ > # diff -r /usr/ports/ /mnt/ | more > Only in /mnt/databases/mongodb42/files/aarch64: > patch-src_third__party_mozjs-60_ > Only in /usr/ports/databases/mongodb42/files/aarch64: > patch-src_third__party_mozjs-60_platform_aarch64_freebsd_build_Unified__cpp__js__src25.cpp > Only in /usr/ports/devel/ice/files: Make.rules.FreeBSD > Only in /usr/ports/devel/ice/files: patch-config-Make.common.rules > Only in
Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)
> On 2021-May-20, at 18:09, Rick Macklem wrote: > > Oh, one additional thing that I'll dare to top post... > r367492 broke the TCP upcalls that the NFS server uses, such > that intermittent hangs of NFS mounts to FreeBSD13 servers can occur. > This has not yet been resolved in "main" etc and could explain > why an RPC could time out for a soft mount. See later notes that I added: soft mount is not required to see the problem. > You can revert the patch in r367492 to avoid the problem. If I understand right, you are indicating that this would not apply to the non-soft mount case that I got. > Disabling TSO, LRO are also de-facto standard things to do when > you observe weird NFS behaviour, because they are often broken > in various network device drivers. I'll have to figure out how to experiment with such. Things are at defaults rather generally on the systems. I'm not literate in the subject areas. I'm the only user of the machines and network. It is not outward facing. It is a rather small EtherNet network. > rick > > > From: owner-freebsd-sta...@freebsd.org on > behalf of Rick Macklem > Sent: Thursday, May 20, 2021 8:55 PM > To: FreeBSD-STABLE Mailing List; Mark Millard > Subject: Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs > (in a zfs file systems context) > > Mark Millard wrote: >> [I warn that I'm a fairly minimal user of NFS >> mounts, not knowing all that much. I'm mostly >> reporting this in case it ends up as evidence >> via eventually matching up with others observing >> possibly related oddities.] >> >> I got the following odd sequence (that I've >> mixed notes into). It involved a diff -r over NFS >> showing differences (files missing) and then a >> later diff finding matches for the same files, >> no file system changes made on either machine. >> I'm unable to reproduce the oddity on demand. >> >> Note: A larger scope diff -r originally returned the >> below as well, but doing the narrower diff -r did >> repeat the result and that is what I show. (I >> make no use of devel/ice .) >> >> # diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more >> Only in /usr/ports/devel/ice/files: Make.rules.FreeBSD . . . >> Only in /usr/ports/devel/ice/files: patch-scripts-TestUtil.py >> >> Note: The above was not expected. So I tried: >> >> # ls -Tld /mnt/devel/ice/files/* >> -rw-r--r-- 1 root wheel 755 Apr 21 21:07:54 2021 >> /mnt/devel/ice/files/Make.rules.FreeBSD . . . >> -rw-r--r-- 1 root wheel 2588 Apr 21 21:07:54 2021 >> /mnt/devel/ice/files/patch-scripts-TestUtil.py >> >> Note: So that indicated that the files were there on the >> machine that /mnt references. So attempting the original >> diff -r again: >> >> # diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more >> # >> >> (Empty difference.) >> >> Note: So after the explicit "ls -Tld /mnt/devel/ice/files/*" >> the odd result of the diff -r no longer happened: no >> differences reported. >> >> >> >> For reference (both machines reported): >> >> . . . >> The original mount command was on CA72_16Gp_ZFS: >> >> # mount -onoatime,soft 192.168.1.170:/usr/ports/ /mnt/ > The likely explanation for this is your use of a "soft" mount. > - If the NFS server is slow to respond or there is a temporary network issue, > the RPC request can time out and then the > syscall can fail with EINT/ETIMEDOUT. Since almost nothing, including the >readdir(3) libc functions expect syscalls to fail this way... >Then the cached directory is messed up. >Doing the "ls" read the directory again and fixed the problem. > > Try to reproduce it for a mount without the "soft" option. > (If a mount point is hung, due to an unresponsive server "umount -N /mnt" > can usually get rid of it.) > Personally, I thought "soft" was a bad idea when Sun introduced it in NFS in > 1985 > and I still feel that way. > --> If you can reproduce it without "soft" then I can't explain it. > To be honest, the directory reading/caching code in the NFSv3 client > hasn't changed significantly in literally decades, as far as I can > remember. Well . . . trying an even wider scope diff than the original . . . # umount /mnt/ # mount -onoatime 192.168.1.170:/usr/ports/ /mnt/ # diff -r /usr/ports/ /mnt/ | more Only in /mnt/databases/mongodb42/files/aarch64: patch-src_third__party_mozjs-60_ Only in /usr/ports/databases/mongodb42/files/aarch64: patch-src_third__party_mozjs-60_platform_aarch64_freebsd_build_Unified__cpp__js__src25.cpp Only in /usr/ports/devel/ice/files: Make.rules.FreeBSD Only in /usr/ports/devel/ice/files: patch-config-Make.common.rules Only in /usr/ports/devel/ice/files: patch-cpp-Makefile . . . Only in /usr/ports/devel/ice/files: patch-python-test-Slice-unicodePaths-run.py Only in /usr/ports/devel/ice/files: patch-scripts-Expect.py Only in /usr/ports/devel/ice/files: patch-scripts-IceGridAdmin.py Only in /usr/ports/devel/ice/files:
Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)
Oh, one additional thing that I'll dare to top post... r367492 broke the TCP upcalls that the NFS server uses, such that intermittent hangs of NFS mounts to FreeBSD13 servers can occur. This has not yet been resolved in "main" etc and could explain why an RPC could time out for a soft mount. You can revert the patch in r367492 to avoid the problem. Disabling TSO, LRO are also de-facto standard things to do when you observe weird NFS behaviour, because they are often broken in various network device drivers. rick From: owner-freebsd-sta...@freebsd.org on behalf of Rick Macklem Sent: Thursday, May 20, 2021 8:55 PM To: FreeBSD-STABLE Mailing List; Mark Millard Subject: Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context) Mark Millard wrote: >[I warn that I'm a fairly minimal user of NFS >mounts, not knowing all that much. I'm mostly >reporting this in case it ends up as evidence >via eventually matching up with others observing >possibly related oddities.] > >I got the following odd sequence (that I've >mixed notes into). It involved a diff -r over NFS >showing differences (files missing) and then a >later diff finding matches for the same files, >no file system changes made on either machine. >I'm unable to reproduce the oddity on demand. > >Note: A larger scope diff -r originally returned the >below as well, but doing the narrower diff -r did >repeat the result and that is what I show. (I >make no use of devel/ice .) > ># diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more >Only in /usr/ports/devel/ice/files: Make.rules.FreeBSD >Only in /usr/ports/devel/ice/files: patch-config-Make.common.rules >Only in /usr/ports/devel/ice/files: patch-cpp-Makefile >Only in /usr/ports/devel/ice/files: patch-cpp-allTests.py >Only in /usr/ports/devel/ice/files: patch-cpp-config-Make.rules >Only in /usr/ports/devel/ice/files: patch-cpp-include-Ice-FactoryTableInit.h >Only in /usr/ports/devel/ice/files: patch-cpp-include-IceUtil-Config.h >Only in /usr/ports/devel/ice/files: patch-cpp-include-IceUtil-ScannerConfig.h >Only in /usr/ports/devel/ice/files: >patch-cpp-src-Glacier2CryptPermissionsVerifier-CryptPermissionsVerifierI.cpp >Only in /usr/ports/devel/ice/files: patch-cpp-src-Ice-ProxyFactory.cpp >Only in /usr/ports/devel/ice/files: patch-cpp-src-IceGrid-PluginFacadeI.cpp >Only in /usr/ports/devel/ice/files: patch-cpp-src-IceGrid-RegistryI.cpp >Only in /usr/ports/devel/ice/files: patch-cpp-src-IceSSL-Makefile >Only in /usr/ports/devel/ice/files: patch-cpp-test-Glacier2-ssl-Server.cpp >Only in /usr/ports/devel/ice/files: >patch-cpp-test-Glacier2-staticFiltering-run.py >Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-info-AllTests.cpp >Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-metrics-AllTests.cpp >Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-objects-Makefile >Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-properties-run.py >Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-admin-run.py >Only in /usr/ports/devel/ice/files: >patch-cpp-test-IceGrid-deployer-AllTests.cpp >Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-deployer-Makefile >Only in /usr/ports/devel/ice/files: >patch-cpp-test-IceGrid-deployer-application.xml >Only in /usr/ports/devel/ice/files: >patch-cpp-test-IceGrid-distribution-AllTests.cpp >Only in /usr/ports/devel/ice/files: >patch-cpp-test-IceGrid-distribution-application.xml >Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-distribution-run.py >Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-session-run.py >Only in /usr/ports/devel/ice/files: >patch-cpp-test-IceSSL-configuration-AllTests.cpp >Only in /usr/ports/devel/ice/files: patch-cpp-test-IceSSL-configuration-run.py >Only in /usr/ports/devel/ice/files: patch-cpp-test-Slice-headers-run.py >Only in /usr/ports/devel/ice/files: patch-cpp-test-Slice-unicodePaths-run.py >Only in /usr/ports/devel/ice/files: patch-cpp-test-include-TestCommon.h >Only in /usr/ports/devel/ice/files: patch-php-Makefile >Only in /usr/ports/devel/ice/files: patch-php-config-Make.rules.php >Only in /usr/ports/devel/ice/files: patch-php-lib-Makefile >Only in /usr/ports/devel/ice/files: patch-python-Makefile >Only in /usr/ports/devel/ice/files: patch-python-config-Make.rules >Only in /usr/ports/devel/ice/files: patch-python-modules-IcePy-Types.cpp >Only in /usr/ports/devel/ice/files: patch-python-modules-IcePy-Types.h >Only in /usr/ports/devel/ice/files: patch-python-python-Makefile >Only in /usr/ports/devel/ice/files: patch-python-test-Ice-info-AllTests.py >Only in /usr/ports/devel/ice/files: patch-python-test-Ice-properties-run.py >Only in /usr/ports/devel/ice/files: patch-python-test-Slice-unicodePaths-run.py >Only in /usr/ports/devel/ice/files: patch-scripts-Expect.py >Only in /usr/ports/devel/ice/files: patch-scripts-IceGridAdmin.py >Only in /usr/ports/devel/ice/files: patch-scripts-TestUtil.py
Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)
Mark Millard wrote: >[I warn that I'm a fairly minimal user of NFS >mounts, not knowing all that much. I'm mostly >reporting this in case it ends up as evidence >via eventually matching up with others observing >possibly related oddities.] > >I got the following odd sequence (that I've >mixed notes into). It involved a diff -r over NFS >showing differences (files missing) and then a >later diff finding matches for the same files, >no file system changes made on either machine. >I'm unable to reproduce the oddity on demand. > >Note: A larger scope diff -r originally returned the >below as well, but doing the narrower diff -r did >repeat the result and that is what I show. (I >make no use of devel/ice .) > ># diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more >Only in /usr/ports/devel/ice/files: Make.rules.FreeBSD >Only in /usr/ports/devel/ice/files: patch-config-Make.common.rules >Only in /usr/ports/devel/ice/files: patch-cpp-Makefile >Only in /usr/ports/devel/ice/files: patch-cpp-allTests.py >Only in /usr/ports/devel/ice/files: patch-cpp-config-Make.rules >Only in /usr/ports/devel/ice/files: patch-cpp-include-Ice-FactoryTableInit.h >Only in /usr/ports/devel/ice/files: patch-cpp-include-IceUtil-Config.h >Only in /usr/ports/devel/ice/files: patch-cpp-include-IceUtil-ScannerConfig.h >Only in /usr/ports/devel/ice/files: >patch-cpp-src-Glacier2CryptPermissionsVerifier-CryptPermissionsVerifierI.cpp >Only in /usr/ports/devel/ice/files: patch-cpp-src-Ice-ProxyFactory.cpp >Only in /usr/ports/devel/ice/files: patch-cpp-src-IceGrid-PluginFacadeI.cpp >Only in /usr/ports/devel/ice/files: patch-cpp-src-IceGrid-RegistryI.cpp >Only in /usr/ports/devel/ice/files: patch-cpp-src-IceSSL-Makefile >Only in /usr/ports/devel/ice/files: patch-cpp-test-Glacier2-ssl-Server.cpp >Only in /usr/ports/devel/ice/files: >patch-cpp-test-Glacier2-staticFiltering-run.py >Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-info-AllTests.cpp >Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-metrics-AllTests.cpp >Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-objects-Makefile >Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-properties-run.py >Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-admin-run.py >Only in /usr/ports/devel/ice/files: >patch-cpp-test-IceGrid-deployer-AllTests.cpp >Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-deployer-Makefile >Only in /usr/ports/devel/ice/files: >patch-cpp-test-IceGrid-deployer-application.xml >Only in /usr/ports/devel/ice/files: >patch-cpp-test-IceGrid-distribution-AllTests.cpp >Only in /usr/ports/devel/ice/files: >patch-cpp-test-IceGrid-distribution-application.xml >Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-distribution-run.py >Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-session-run.py >Only in /usr/ports/devel/ice/files: >patch-cpp-test-IceSSL-configuration-AllTests.cpp >Only in /usr/ports/devel/ice/files: patch-cpp-test-IceSSL-configuration-run.py >Only in /usr/ports/devel/ice/files: patch-cpp-test-Slice-headers-run.py >Only in /usr/ports/devel/ice/files: patch-cpp-test-Slice-unicodePaths-run.py >Only in /usr/ports/devel/ice/files: patch-cpp-test-include-TestCommon.h >Only in /usr/ports/devel/ice/files: patch-php-Makefile >Only in /usr/ports/devel/ice/files: patch-php-config-Make.rules.php >Only in /usr/ports/devel/ice/files: patch-php-lib-Makefile >Only in /usr/ports/devel/ice/files: patch-python-Makefile >Only in /usr/ports/devel/ice/files: patch-python-config-Make.rules >Only in /usr/ports/devel/ice/files: patch-python-modules-IcePy-Types.cpp >Only in /usr/ports/devel/ice/files: patch-python-modules-IcePy-Types.h >Only in /usr/ports/devel/ice/files: patch-python-python-Makefile >Only in /usr/ports/devel/ice/files: patch-python-test-Ice-info-AllTests.py >Only in /usr/ports/devel/ice/files: patch-python-test-Ice-properties-run.py >Only in /usr/ports/devel/ice/files: patch-python-test-Slice-unicodePaths-run.py >Only in /usr/ports/devel/ice/files: patch-scripts-Expect.py >Only in /usr/ports/devel/ice/files: patch-scripts-IceGridAdmin.py >Only in /usr/ports/devel/ice/files: patch-scripts-TestUtil.py > >Note: The above was not expected. So I tried: > ># ls -Tld /mnt/devel/ice/files/* >-rw-r--r-- 1 root wheel 755 Apr 21 21:07:54 2021 >/mnt/devel/ice/files/Make.rules.FreeBSD >-rw-r--r-- 1 root wheel 1542 Apr 21 21:07:54 2021 >/mnt/devel/ice/files/patch-config-Make.common.rules >-rw-r--r-- 1 root wheel 388 Apr 21 21:07:54 2021 >/mnt/devel/ice/files/patch-cpp-Makefile >-rw-r--r-- 1 root wheel 1695 Apr 21 21:07:54 2021 >/mnt/devel/ice/files/patch-cpp-allTests.py >-rw-r--r-- 1 root wheel 1112 Apr 21 21:07:54 2021 >/mnt/devel/ice/files/patch-cpp-config-Make.rules >-rw-r--r-- 1 root wheel 1512 Apr 21 21:07:54 2021 >/mnt/devel/ice/files/patch-cpp-include-Ice-FactoryTableInit.h >-rw-r--r-- 1 root wheel 1496 Apr 21 21:07:54 2021 >/mnt/devel/ice/files/patch-cpp-include-IceUtil-Config.h
releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)
[I warn that I'm a fairly minimal user of NFS mounts, not knowing all that much. I'm mostly reporting this in case it ends up as evidence via eventually matching up with others observing possibly related oddities.] I got the following odd sequence (that I've mixed notes into). It involved a diff -r over NFS showing differences (files missing) and then a later diff finding matches for the same files, no file system changes made on either machine. I'm unable to reproduce the oddity on demand. Note: A larger scope diff -r originally returned the below as well, but doing the narrower diff -r did repeat the result and that is what I show. (I make no use of devel/ice .) # diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more Only in /usr/ports/devel/ice/files: Make.rules.FreeBSD Only in /usr/ports/devel/ice/files: patch-config-Make.common.rules Only in /usr/ports/devel/ice/files: patch-cpp-Makefile Only in /usr/ports/devel/ice/files: patch-cpp-allTests.py Only in /usr/ports/devel/ice/files: patch-cpp-config-Make.rules Only in /usr/ports/devel/ice/files: patch-cpp-include-Ice-FactoryTableInit.h Only in /usr/ports/devel/ice/files: patch-cpp-include-IceUtil-Config.h Only in /usr/ports/devel/ice/files: patch-cpp-include-IceUtil-ScannerConfig.h Only in /usr/ports/devel/ice/files: patch-cpp-src-Glacier2CryptPermissionsVerifier-CryptPermissionsVerifierI.cpp Only in /usr/ports/devel/ice/files: patch-cpp-src-Ice-ProxyFactory.cpp Only in /usr/ports/devel/ice/files: patch-cpp-src-IceGrid-PluginFacadeI.cpp Only in /usr/ports/devel/ice/files: patch-cpp-src-IceGrid-RegistryI.cpp Only in /usr/ports/devel/ice/files: patch-cpp-src-IceSSL-Makefile Only in /usr/ports/devel/ice/files: patch-cpp-test-Glacier2-ssl-Server.cpp Only in /usr/ports/devel/ice/files: patch-cpp-test-Glacier2-staticFiltering-run.py Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-info-AllTests.cpp Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-metrics-AllTests.cpp Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-objects-Makefile Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-properties-run.py Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-admin-run.py Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-deployer-AllTests.cpp Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-deployer-Makefile Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-deployer-application.xml Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-distribution-AllTests.cpp Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-distribution-application.xml Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-distribution-run.py Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-session-run.py Only in /usr/ports/devel/ice/files: patch-cpp-test-IceSSL-configuration-AllTests.cpp Only in /usr/ports/devel/ice/files: patch-cpp-test-IceSSL-configuration-run.py Only in /usr/ports/devel/ice/files: patch-cpp-test-Slice-headers-run.py Only in /usr/ports/devel/ice/files: patch-cpp-test-Slice-unicodePaths-run.py Only in /usr/ports/devel/ice/files: patch-cpp-test-include-TestCommon.h Only in /usr/ports/devel/ice/files: patch-php-Makefile Only in /usr/ports/devel/ice/files: patch-php-config-Make.rules.php Only in /usr/ports/devel/ice/files: patch-php-lib-Makefile Only in /usr/ports/devel/ice/files: patch-python-Makefile Only in /usr/ports/devel/ice/files: patch-python-config-Make.rules Only in /usr/ports/devel/ice/files: patch-python-modules-IcePy-Types.cpp Only in /usr/ports/devel/ice/files: patch-python-modules-IcePy-Types.h Only in /usr/ports/devel/ice/files: patch-python-python-Makefile Only in /usr/ports/devel/ice/files: patch-python-test-Ice-info-AllTests.py Only in /usr/ports/devel/ice/files: patch-python-test-Ice-properties-run.py Only in /usr/ports/devel/ice/files: patch-python-test-Slice-unicodePaths-run.py Only in /usr/ports/devel/ice/files: patch-scripts-Expect.py Only in /usr/ports/devel/ice/files: patch-scripts-IceGridAdmin.py Only in /usr/ports/devel/ice/files: patch-scripts-TestUtil.py Note: The above was not expected. So I tried: # ls -Tld /mnt/devel/ice/files/* -rw-r--r-- 1 root wheel 755 Apr 21 21:07:54 2021 /mnt/devel/ice/files/Make.rules.FreeBSD -rw-r--r-- 1 root wheel 1542 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-config-Make.common.rules -rw-r--r-- 1 root wheel 388 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-Makefile -rw-r--r-- 1 root wheel 1695 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-allTests.py -rw-r--r-- 1 root wheel 1112 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-config-Make.rules -rw-r--r-- 1 root wheel 1512 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-include-Ice-FactoryTableInit.h -rw-r--r-- 1 root wheel 1496 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-include-IceUtil-Config.h -rw-r--r-- 1 root wheel 447 Apr 21 21:07:54 2021
