Note I am NOT a TCP expert, but know enough about it to add a comment... > Alan Somers wrote: > [stuff snipped] > >Is the 128K limit related to MAXPHYS? If so, it should be greater in 13.0. > For the client, yes. For the server, no. > For the server, it is just a compile time constant NFS_SRVMAXIO. > > It's mainly related to the fact that I haven't gotten around to testing larger > sizes yet. > - kern.ipc.maxsockbuf needs to be several times the limit, which means it > would > have to increase for 1Mbyte. > - The session code must negotiate a maximum RPC size > 1 Mbyte. > (I think the server code does do this, but it needs to be tested.) > And, yes, the client is limited to MAXPHYS. > > Doing this is on my todo list, rick > > The client should acquire the attributes that indicate that and set > rsize/wsize > to that. "# nfsstat -m" on the client should show you what the client > is actually using. If it is larger than 128K, set both rsize and wsize to > 128K. > > >Output from the NFS Client when the issue occurs > ># netstat -an | grep NFS.Server.IP.X > >tcp 0 0 NFS.Client.IP.X:46896 NFS.Server.IP.X:2049 > >FIN_WAIT2 > I'm no TCP guy. Hopefully others might know why the client would be > stuck in FIN_WAIT2 (I vaguely recall this means it is waiting for a fin/ack, > but could be wrong?)
The most common way to get stuck in FIN_WAIT2 is to call shutdown(2) on a socket, but never following up with a close(2) after some timeout period. The "client" is still connected to the socket and can stay in this shutdown state for ever, the kernel well not reap the socket as it is associated with a processes, aka not orphaned. I suspect that the Linux client has a corner condition that is leading to this socket leak. If on the Linux client you can look at the sockets to see if these are still associated with a process, ala fstat or what ever Linux tool does this that would be helpfull. If they are infact connected to a process it is that process that must call close(2) to clean these up. IIRC the server side socket would be gone at this point and there is nothing the server can do that would allow a FIN_WAIT2 to close down. The real TCP experts can now correct my 30 year old TCP stack understanding... > > ># cat /sys/kernel/debug/sunrpc/rpc_xprt/*/info > >netid: tcp > >addr: NFS.Server.IP.X > >port: 2049 > >state: 0x51 > > > >syslog > >Mar 4 10:29:27 hostname kernel: [437414.131978] -pid- flgs status -client- > >--rqstp- ->timeout ---ops-- > >Mar 4 10:29:27 hostname kernel: [437414.133158] 57419 40a1 0 9b723c73 > >>143cfadf 30000 4ca953b5 nfsv4 OPEN_NOATTR a:call_connect_status [sunrpc] > >>q:xprt_pending > I don't know what OPEN_NOATTR means, but I assume it is some variant > of NFSv4 Open operation. > [stuff snipped] > >Mar 4 10:29:30 hostname kernel: [437417.110517] RPC: 57419 > >xprt_connect_status: >connect attempt timed out > >Mar 4 10:29:30 hostname kernel: [437417.112172] RPC: 57419 > >call_connect_status > >(status -110) > I have no idea what status -110 means? > >Mar 4 10:29:30 hostname kernel: [437417.113337] RPC: 57419 call_timeout > >(major) > >Mar 4 10:29:30 hostname kernel: [437417.114385] RPC: 57419 call_bind > >(status 0) > >Mar 4 10:29:30 hostname kernel: [437417.115402] RPC: 57419 call_connect > >xprt >00000000e061831b is not connected > >Mar 4 10:29:30 hostname kernel: [437417.116547] RPC: 57419 xprt_connect > >xprt >00000000e061831b is not connected > >Mar 4 10:30:31 hostname kernel: [437478.551090] RPC: 57419 > >xprt_connect_status: >connect attempt timed out > >Mar 4 10:30:31 hostname kernel: [437478.552396] RPC: 57419 > >call_connect_status >(status -110) > >Mar 4 10:30:31 hostname kernel: [437478.553417] RPC: 57419 call_timeout > >(minor) > >Mar 4 10:30:31 hostname kernel: [437478.554327] RPC: 57419 call_bind > >(status 0) > >Mar 4 10:30:31 hostname kernel: [437478.555220] RPC: 57419 call_connect > >xprt >00000000e061831b is not connected > >Mar 4 10:30:31 hostname kernel: [437478.556254] RPC: 57419 xprt_connect > >xprt >00000000e061831b is not connected > Is it possible that the client is trying to (re)connect using the same client > port#? > I would normally expect the client to create a new TCP connection using a > different client port# and then retry the outstanding RPCs. > --> Capturing packets when this happens would show us what is going on. > > If there is a problem on the FreeBSD end, it is most likely a broken > network device driver. > --> Try disabling TSO , LRO. > --> Try a different driver for the net hardware on the server. > --> Try a different net chip on the server. > If you can capture packets when (not after) the hang > occurs, then you can look at them in wireshark and see > what is actually happening. (Ideally on both client and > server, to check that your network hasn't dropped anything.) > --> I know, if the hangs aren't easily reproducible, this isn't > easily done. > --> Try a newer Linux kernel and see if the problem persists. > The Linux folk will get more interested if you can reproduce > the problem on 5.12. (Recent bakeathon testing of the 5.12 > kernel against the FreeBSD server did not find any issues.) > > Hopefully the network folk have some insight w.r.t. why > the TCP connection is sitting in FIN_WAIT2. > > rick > > > > Jason Breitman > > > > > > > _______________________________________________ > freebsd-net@freebsd.org<mailto:freebsd-net@freebsd.org> mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscr...@freebsd.org<mailto:freebsd-net-unsubscr...@freebsd.org>" > > _______________________________________________ > freebsd-net@freebsd.org<mailto:freebsd-net@freebsd.org> mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscr...@freebsd.org<mailto:freebsd-net-unsubscr...@freebsd.org>" > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org" > -- Rod Grimes rgri...@freebsd.org _______________________________________________ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
