Agreed. I had made the changes on the FreeBSD Server side and was suggesting
that a new TCP connection needed to be established between the client and
server for the settings to take effect.
I rebooted all of my Debian clients on Sunday to achieve that goal,
establishing a new NFSv4 TCP connection with the file server, and will let the
group know if I see another hang.
Jason Breitman
On Mar 22, 2021, at 7:27 AM, Youssef GHORBAL <youssef.ghor...@pasteur.fr> wrote:
> On 21 Mar 2021, at 14:41, Jason Breitman <jbreit...@tildenparkcapital.com>
> wrote:
>
> Thanks for sharing as this sounds exactly like my issue.
>
> I had implemented the change below on 3/8/2021 and have experienced the NFS
> hang after that.
> Do I need to reboot or umount / mount all of the clients and then I will be
> ok?
>
> I had not rebooted the clients, but would to get out of this situation.
> It is logical that a new TCP session over 2049 needs to be reestablished for
> the changes to take effect.
>
> net.inet.tcp.fast_finwait2_recycle=1
> net.inet.tcp.finwait2_timeout=1000
In my case, those were implemented on the server (FreeBSD side) since the BSD
box that was closing the connection and the FIN_WAIT_2 state was on its side.
In your cas the FIN_WAIT_2 is on the client side. I don’t know if these sysctl
are even availale on Linux.
> I can also confirm that the iptables solution that you use on the client to
> get out of the hung mount without a reboot work for me.
> #!/bin/sh
>
> progName="nfsClientFix"
> delay=15
> nfs_ip=NFS.Server.IP.X
>
> nfs_fin_wait2_state() {
> /usr/bin/netstat -an | /usr/bin/grep ${nfs_ip}:2049 | /usr/bin/grep
> FIN_WAIT2 > /dev/null 2>&1
> return $?
> }
>
>
> nfs_fin_wait2_state
> result=$?
> if [ ${result} -eq 0 ] ; then
> /usr/bin/logger -s -i -p local7.error -t ${progName} "NFS Connection is in
> FIN_WAIT2!"
> /usr/bin/logger -s -i -p local7.error -t ${progName} "Enabling firewall to
> block ${nfs_ip}!"
> /usr/sbin/iptables -A INPUT -s ${nfs_ip} -j DROP
>
> while true
> do
> /usr/bin/sleep ${delay}
> nfs_fin_wait2_state
> result=$?
> if [ ${result} -ne 0 ] ; then
> /usr/bin/logger -s -i -p local7.notice -t ${progName} "NFS
> Connection is OK."
> /usr/bin/logger -s -i -p local7.error -t ${progName} "Disabling
> firewall to allow access to ${nfs_ip}!"
> /usr/sbin/iptables -D INPUT -s ${nfs_ip} -j DROP
> break
> fi
> done
> fi
>
>
> Jason Breitman
>
>
> On Mar 19, 2021, at 8:40 PM, Youssef GHORBAL <youssef.ghor...@pasteur.fr>
> wrote:
>
> Hi Jason,
>
>> On 17 Mar 2021, at 18:17, Jason Breitman <jbreit...@tildenparkcapital.com>
>> wrote:
>>
>> Please review the details below and let me know if there is a setting that I
>> should apply to my FreeBSD NFS Server or if there is a bug fix that I can
>> apply to resolve my issue.
>> I shared this information with the linux-nfs mailing list and they believe
>> the issue is on the server side.
>>
>> Issue
>> NFSv4 mounts periodically hang on the NFS Client.
>>
>> During this time, it is possible to manually mount from another NFS Server
>> on the NFS Client having issues.
>> Also, other NFS Clients are successfully mounting from the NFS Server in
>> question.
>> Rebooting the NFS Client appears to be the only solution.
>
> I had experienced a similar weird situation with periodically stuck Linux NFS
> clients mounting Isilon NFS servers (Isilon is FreeBSD based but they seem to
> have there own nfsd)
> We’ve had better luck and we did manage to have packet captures on both sides
> during the issue. The gist of it goes like follows:
>
> - Data flows correctly between SERVER and the CLIENT
> - At some point SERVER starts decreasing it's TCP Receive Window until it
> reachs 0
> - The client (eager to send data) can only ack data sent by SERVER.
> - When SERVER was done sending data, the client starts sending TCP Window
> Probes hoping that the TCP Window opens again so he can flush its buffers.
> - SERVER responds with a TCP Zero Window to those probes.
> - After 6 minutes (the NFS server default Idle timeout) SERVER racefully
> closes the TCP connection sending a FIN Packet (and still a TCP Window at 0)
> - CLIENT ACK that FIN.
> - SERVER goes in FIN_WAIT_2 state
> - CLIENT closes its half part part of the socket and goes in LAST_ACK state.
> - FIN is never sent by the client since there still data in its SendQ and
> receiver TCP Window is still 0. At this stage the client starts sending TCP
> Window Probes again and again hoping that the server opens its TCP Window so
> it can flush it's buffers and terminate its side of the socket.
> - SERVER keeps responding with a TCP Zero Window to those probes.
> => The last two steps goes on and on for hours/days freezing the NFS mount
> bound to that TCP session.
>
> If we had a situation where CLIENT was responsible for closing the TCP Window
> (and initiating the TCP FIN first) and server wanting to send data we’ll end
> up in the same state as you I think.
>
> We’ve never had the root cause of why the SERVER decided to close the TCP
> Window and no more acccept data, the fix on the Isilon part was to recycle
> more aggressively the FIN_WAIT_2 sockets
> (net.inet.tcp.fast_finwait2_recycle=1 & net.inet.tcp.finwait2_timeout=5000).
> Once the socket recycled and at the next occurence of CLIENT TCP Window
> probe, SERVER sends a RST, triggering the teardown of the session on the
> client side, a new TCP handchake, etc and traffic flows again (NFS starts
> responding)
>
> To avoid rebooting the client (and before the aggressive FIN_WAIT_2 was
> implemented on the Isilon side) we’ve added a check script on the client that
> detects LAST_ACK sockets on the client and through iptables rule enforces a
> TCP RST, Something like: -A OUTPUT -p tcp -d $nfs_server_addr --sport
> $local_port -j REJECT --reject-with tcp-reset (the script removes this
> iptables rule as soon as the LAST_ACK disappears)
>
> The bottom line would be to have a packet capture during the outage (client
> and/or server side), it will show you at least the shape of the TCP exchange
> when NFS is stuck.
>
> Youssef
>
>
_______________________________________________
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
