OK, so I'm still bashing my head against a brick wall with this, and so far the brick wall is holding out better than my head. With school hols started I now have much greater access to the system.
The problem is severe in the following situation: running VNC client on a school PC controlling a Pi on the other side of the firewall. The Pi runs a face follower program which continuously displays image captures from the camera, causing continual screen refreshes to be sent to the VNC client. The firewall NIC on the school network side repeatedly goes DOWN for around 30 secs at a time. ip -s link ls shows it's getting large numbers of dropped packets. In this situation, control of the session from the VNC client is almost impossible. Yesterday I tried 2 things, with interesting results: 1. I completely rebuilt the system from scratch, installing Shorewall 5 instead of Shorewall 4, and with kernel 4.8.0-53. The problem remains. 2. I dug out an old hard disk with a version of the system I built last Summer (if not before) and kernel 4.4.0-34. All other hardware was unchanged. The problem disappeared! This seems to indicate software, not hardware. No clues that I can spot in /var/log/messages. Comparing the outputs of sysctl -a on the 2 systems shows various parameters changed, but nearly all increased. (My best guess had been that a kernel buffer needed to be larger.) See blueskylark.org/stuff/sysctl-diffs.txt for an sdiff - old system on the left, new on the right. /etc/shorewall/interfaces is identical between systems (except for commented lines) and the only differences in shorewall.conf are in logging and verbosity. Any suggestions? Regards - Philip On 15/06/2017 15:10, Philip Le Riche wrote: > We have Shorewall 4 protecting the school network from a group of > Raspberry Pis, which we operate from PCs on the school network using VNC > running through Shorewall. For some weeks we've had frequent problems > with VNC sessions hanging for around 30 seconds. I've been trying to > track it down with increasingly focussed Wireshark captures, and this is > what seems to be happening on one fairly typical hang: > > Two Pis are being controlled from separate PCs. I have ping running from > the firewall to one of the Pis and also from the firewall to the default > gateway on the school network. > > Hundreds of packets are passing through the firewall from one of the Pis > to the PC controlling it, containing VNC screen update data. These are > interspersed every second by a ping/reply to one of the Pis and a > ping/reply to the default gateway. > > Suddenly TCP retransmissions of VNC traffic start appearing. Often at > this point you see one or two other packets, such as an ntp or a VNC > from the other Pi, but this may only be because they're no longer being > hidden amongst a mass of VNC. > > More retransmissions from the Pi(s) but nothing on the school network > NIC, and in particular, no pings to the default gateway. > > After around 10 seconds, the Pi network NIC sends ICMP network > unreachable to both Pis. > > Sometimes I've seen ICMP host unreachable, I think from the school > network NIC back to a Pi. Other times I've seen RST, ACK packets from > one of the VNC client PCs - I don't see RST, ACK in the standard TCP > state diagram. > > After a total of around 30 seconds, everything seems to recover, and > pings reappear on the school network, though VNC generally has to open > a new TCP connection. > > Only fairly recently have we regularly run more than one Pi at the same > time. Maybe we're just running out of kernel buffers? Or we need a more > powerful machine to run Shorewall? (It's an unremarkable desktop machine > maybe 5 years old.) Or maybe I've just got something misconfigured. > Ideas please? > > Regards - Philip > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users > ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Shorewall-users mailing list Shorewall-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/shorewall-users
