Re: [Shorewall-users] Firewall hangs
Solved - or rather, workaround found. For the record, the problem seems to be a regression in the e1000e driver or firmware between Mint 17 and 18, or a difference between the 32 and 64 bit versions. I had the school network on the motherboard NIC (this is the one that was causing the trouble) and the Pi network on a PCI NIC. Having rebuilt the system yet again to no avail, I swapped over the two interfaces, and lo and behold - problem vanished! Actually, I've just checked that the driver is the same (v3.2.6-k) but the firmware has gone from 0.13-4 (works) to 1.8-0 (troublesome), and I've moved from a 32 bit system (works) to a 64 bit system (troublesome). The motherboard NIC on the system I have at home, which I believe is the same or similar, is Intel 82579LM. Regards - Philip On 26/07/2017 02:20, Tom Eastep wrote: > On 07/25/2017 11:52 AM, Philip Le Riche wrote: >> OK, so I'm still bashing my head against a brick wall with this, and so >> far the brick wall is holding out better than my head. With school hols >> started I now have much greater access to the system. >> >> The problem is severe in the following situation: running VNC client on >> a school PC controlling a Pi on the other side of the firewall. The Pi >> runs a face follower program which continuously displays image captures >> from the camera, causing continual screen refreshes to be sent to the >> VNC client. The firewall NIC on the school network side repeatedly goes >> DOWN for around 30 secs at a time. ip -s link ls shows it's getting >> large numbers of dropped packets. In this situation, control of the >> session from the VNC client is almost impossible. >> >> Yesterday I tried 2 things, with interesting results: >> 1. I completely rebuilt the system from scratch, installing Shorewall 5 >> instead of Shorewall 4, and with kernel 4.8.0-53. The problem remains. >> 2. I dug out an old hard disk with a version of the system I built last >> Summer (if not before) and kernel 4.4.0-34. All other hardware was >> unchanged. The problem disappeared! >> >> This seems to indicate software, not hardware. No clues that I can spot >> in /var/log/messages. >> >> Comparing the outputs of sysctl -a on the 2 systems shows various >> parameters changed, but nearly all increased. (My best guess had been >> that a kernel buffer needed to be larger.) See >> blueskylark.org/stuff/sysctl-diffs.txt for an sdiff - old system on the >> left, new on the right. >> >> /etc/shorewall/interfaces is identical between systems (except for >> commented lines) and the only differences in shorewall.conf are in >> logging and verbosity. >> >> Any suggestions? >> > Not really. I know of nothing in a Shorewall configuration that could > produce these symptoms. One thing I noticed in the diff you posted was > that the 'new' output showed nothing from net.netfilter, but I don't > know if that is significant. > > -Tom > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > > ___ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ Shorewall-users mailing list Shorewall-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/shorewall-users
Re: [Shorewall-users] Firewall hangs
On 07/25/2017 11:52 AM, Philip Le Riche wrote: > OK, so I'm still bashing my head against a brick wall with this, and so > far the brick wall is holding out better than my head. With school hols > started I now have much greater access to the system. > > The problem is severe in the following situation: running VNC client on > a school PC controlling a Pi on the other side of the firewall. The Pi > runs a face follower program which continuously displays image captures > from the camera, causing continual screen refreshes to be sent to the > VNC client. The firewall NIC on the school network side repeatedly goes > DOWN for around 30 secs at a time. ip -s link ls shows it's getting > large numbers of dropped packets. In this situation, control of the > session from the VNC client is almost impossible. > > Yesterday I tried 2 things, with interesting results: > 1. I completely rebuilt the system from scratch, installing Shorewall 5 > instead of Shorewall 4, and with kernel 4.8.0-53. The problem remains. > 2. I dug out an old hard disk with a version of the system I built last > Summer (if not before) and kernel 4.4.0-34. All other hardware was > unchanged. The problem disappeared! > > This seems to indicate software, not hardware. No clues that I can spot > in /var/log/messages. > > Comparing the outputs of sysctl -a on the 2 systems shows various > parameters changed, but nearly all increased. (My best guess had been > that a kernel buffer needed to be larger.) See > blueskylark.org/stuff/sysctl-diffs.txt for an sdiff - old system on the > left, new on the right. > > /etc/shorewall/interfaces is identical between systems (except for > commented lines) and the only differences in shorewall.conf are in > logging and verbosity. > > Any suggestions? > Not really. I know of nothing in a Shorewall configuration that could produce these symptoms. One thing I noticed in the diff you posted was that the 'new' output showed nothing from net.netfilter, but I don't know if that is significant. -Tom -- Tom Eastep\ Q: What do you get when you cross a mobster with Shoreline, \ an international standard? Washington, USA \ A: Someone who makes you an offer you can't http://shorewall.org \ understand \___ signature.asc Description: OpenPGP digital signature -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ Shorewall-users mailing list Shorewall-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/shorewall-users
Re: [Shorewall-users] Firewall hangs
OK, so I'm still bashing my head against a brick wall with this, and so far the brick wall is holding out better than my head. With school hols started I now have much greater access to the system. The problem is severe in the following situation: running VNC client on a school PC controlling a Pi on the other side of the firewall. The Pi runs a face follower program which continuously displays image captures from the camera, causing continual screen refreshes to be sent to the VNC client. The firewall NIC on the school network side repeatedly goes DOWN for around 30 secs at a time. ip -s link ls shows it's getting large numbers of dropped packets. In this situation, control of the session from the VNC client is almost impossible. Yesterday I tried 2 things, with interesting results: 1. I completely rebuilt the system from scratch, installing Shorewall 5 instead of Shorewall 4, and with kernel 4.8.0-53. The problem remains. 2. I dug out an old hard disk with a version of the system I built last Summer (if not before) and kernel 4.4.0-34. All other hardware was unchanged. The problem disappeared! This seems to indicate software, not hardware. No clues that I can spot in /var/log/messages. Comparing the outputs of sysctl -a on the 2 systems shows various parameters changed, but nearly all increased. (My best guess had been that a kernel buffer needed to be larger.) See blueskylark.org/stuff/sysctl-diffs.txt for an sdiff - old system on the left, new on the right. /etc/shorewall/interfaces is identical between systems (except for commented lines) and the only differences in shorewall.conf are in logging and verbosity. Any suggestions? Regards - Philip On 15/06/2017 15:10, Philip Le Riche wrote: > We have Shorewall 4 protecting the school network from a group of > Raspberry Pis, which we operate from PCs on the school network using VNC > running through Shorewall. For some weeks we've had frequent problems > with VNC sessions hanging for around 30 seconds. I've been trying to > track it down with increasingly focussed Wireshark captures, and this is > what seems to be happening on one fairly typical hang: > > Two Pis are being controlled from separate PCs. I have ping running from > the firewall to one of the Pis and also from the firewall to the default > gateway on the school network. > > Hundreds of packets are passing through the firewall from one of the Pis > to the PC controlling it, containing VNC screen update data. These are > interspersed every second by a ping/reply to one of the Pis and a > ping/reply to the default gateway. > > Suddenly TCP retransmissions of VNC traffic start appearing. Often at > this point you see one or two other packets, such as an ntp or a VNC > from the other Pi, but this may only be because they're no longer being > hidden amongst a mass of VNC. > > More retransmissions from the Pi(s) but nothing on the school network > NIC, and in particular, no pings to the default gateway. > > After around 10 seconds, the Pi network NIC sends ICMP network > unreachable to both Pis. > > Sometimes I've seen ICMP host unreachable, I think from the school > network NIC back to a Pi. Other times I've seen RST, ACK packets from > one of the VNC client PCs - I don't see RST, ACK in the standard TCP > state diagram. > > After a total of around 30 seconds, everything seems to recover, and > pings reappear on the school network, though VNC generally has to open > a new TCP connection. > > Only fairly recently have we regularly run more than one Pi at the same > time. Maybe we're just running out of kernel buffers? Or we need a more > powerful machine to run Shorewall? (It's an unremarkable desktop machine > maybe 5 years old.) Or maybe I've just got something misconfigured. > Ideas please? > > Regards - Philip > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users > -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Shorewall-users mailing list Shorewall-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/shorewall-users
Re: [Shorewall-users] Firewall hangs
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 06/15/2017 03:00 PM, Philip Le Riche wrote: > > > On 15/06/2017 21:20, Tom Eastep wrote: >> On 06/15/2017 07:10 AM, Philip Le Riche wrote: >>> We have Shorewall 4 protecting the school network from a group >>> of Raspberry Pis, which we operate from PCs on the school >>> network using VNC running through Shorewall. For some weeks >>> we've had frequent problems with VNC sessions hanging for >>> around 30 seconds. I've been trying to track it down with >>> increasingly focussed Wireshark captures, and this is what >>> seems to be happening on one fairly typical hang: >> > ... snip... >> >> Are you monitoring ARP traffic between the Shorewall box and the >> School network? >> >> -Tom > ARP was about the first thing that I filtered out in my capture > filter as there was so much of it on the school network. Is this > significant? > It is the first thing that I would want to look at when traffic suddenly stops then later starts again. Also, are you seeing any errors (ip -s link ls) on the school network NIC? - -Tom - -- Tom Eastep\ Q: What do you get when you cross a mobster with Shoreline, \ an international standard? Washington, USA \ A: Someone who makes you an offer you can't http://shorewall.org \ understand \___ -BEGIN PGP SIGNATURE- Version: GnuPG v2 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJZQ/sSAAoJEJbms/JCOk0QbXsQAJBTobmvPRVbrquxeUTOuS6H 5u3ZQSSKBbus3Mf8ASTzZ1gswBUX+7jHZsz+VtfjpfDfbUGJF7QJ4AfPVB1gpfmh FtdXHdoRs/A3GpUzjThYKgU+jARmlXnDdYADom5s8TojizS+8C+BezMd6M7c43ii 5ofuOeDSVTD1JBGjlxZNO5+H1EXzEMYy2t2BzuCFlh0m4HcdUHvb6SqHQi1sDYaU 1pxH9UTvngdQAk35IvgB1ahmeLsMAgslvsQViL9pl9x0WdNSVqs4G2l3K4QEoSXu Blx7RsIIxxrKifksfiRxCwkJ17xCgN0ij7rU/Qun97uJDYlnYV/xpYL9BeIOeYpf 0aXtAieEUSyohclEpUQUo9AeCiOurrOB/2gJwzEAr2L+MDPY7UdGQXcUV81xA8zu 5c8xV7QEzAlVHqksXUL0aDYgDdlGVszfdxiKYcnAPpLPF6q9G7cNVkUBcO6YKhtw YrZw0t/TyLwitRGNu+ZwGnfJzOstSVKnqhs3z9I/cHkLbE1hGQ2HNN8R7KxUL55e uPNSYrEyfdHvvUsrtpEZrPAIEi4OXq3nqszCECtZE6z0jnMPSUpj0dkHBsZMnfyp J5QzFtiO0mU88zj3nkGJuu8yYd+t+8NhNHvvW5eFD/owHjur8ep+kFuT+OTHRzhf z5urE4pp/dLo0s+BVDhS =64yR -END PGP SIGNATURE- -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Shorewall-users mailing list Shorewall-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/shorewall-users
Re: [Shorewall-users] Firewall hangs
On 15/06/2017 21:20, Tom Eastep wrote: > On 06/15/2017 07:10 AM, Philip Le Riche wrote: > > We have Shorewall 4 protecting the school network from a group of > > Raspberry Pis, which we operate from PCs on the school network > > using VNC running through Shorewall. For some weeks we've had > > frequent problems with VNC sessions hanging for around 30 seconds. > > I've been trying to track it down with increasingly focussed > > Wireshark captures, and this is what seems to be happening on one > > fairly typical hang: > ... snip... > > Are you monitoring ARP traffic between the Shorewall box and the > School network? > > -Tom ARP was about the first thing that I filtered out in my capture filter as there was so much of it on the school network. Is this significant? Regards - Philip > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users > -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ Shorewall-users mailing list Shorewall-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/shorewall-users
Re: [Shorewall-users] Firewall hangs
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 06/15/2017 07:10 AM, Philip Le Riche wrote: > We have Shorewall 4 protecting the school network from a group of > Raspberry Pis, which we operate from PCs on the school network > using VNC running through Shorewall. For some weeks we've had > frequent problems with VNC sessions hanging for around 30 seconds. > I've been trying to track it down with increasingly focussed > Wireshark captures, and this is what seems to be happening on one > fairly typical hang: > > Two Pis are being controlled from separate PCs. I have ping running > from the firewall to one of the Pis and also from the firewall to > the default gateway on the school network. > > Hundreds of packets are passing through the firewall from one of > the Pis to the PC controlling it, containing VNC screen update > data. These are interspersed every second by a ping/reply to one of > the Pis and a ping/reply to the default gateway. > > Suddenly TCP retransmissions of VNC traffic start appearing. Often > at this point you see one or two other packets, such as an ntp or a > VNC from the other Pi, but this may only be because they're no > longer being hidden amongst a mass of VNC. > > More retransmissions from the Pi(s) but nothing on the school > network NIC, and in particular, no pings to the default gateway. > > After around 10 seconds, the Pi network NIC sends ICMP network > unreachable to both Pis. > > Sometimes I've seen ICMP host unreachable, I think from the school > network NIC back to a Pi. Other times I've seen RST, ACK packets > from one of the VNC client PCs - I don't see RST, ACK in the > standard TCP state diagram. > > After a total of around 30 seconds, everything seems to recover, > and pings reappear on the school network, though VNC generally has > to open a new TCP connection. > > Only fairly recently have we regularly run more than one Pi at the > same time. Maybe we're just running out of kernel buffers? Or we > need a more powerful machine to run Shorewall? (It's an > unremarkable desktop machine maybe 5 years old.) Or maybe I've just > got something misconfigured. Ideas please? > Are you monitoring ARP traffic between the Shorewall box and the School network? - -Tom - -- Tom Eastep\ Q: What do you get when you cross a mobster with Shoreline, \ an international standard? Washington, USA \ A: Someone who makes you an offer you can't http://shorewall.org \ understand \___ -BEGIN PGP SIGNATURE- Version: GnuPG v2 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJZQuwVAAoJEJbms/JCOk0QLLAQAIebQp/lFW8prkbDxSqPQnju kWxyczgs+Am6jS9BsfVKWh4WJNhomdBVJIjutVtich4yxK5pG5yyuHIly593YWy4 qK4y6ytVo625H+wyPMd1b8TQ1NzvUmNoKK/rmTY6bgtcbSR5oK9jFFKzmoxu3jN2 I2y5e0vaHL95nyfNuJM9BsoIUG1/gsyzbU9JGqEn+OHbbiCFunugLzeqN5WxUYrK hZZkLSuK6YryWAfMegf3f2Fe7q8to7CuicpmhECHM4qE76Yz5EVyGbxXpMx+ETBn klL/Pbip1nDlvlMuYXEANBjFq7zou2EAqo0DTOK0igcP7CupFesiS52aoLzt41Li MTlqSfSzGrM22XD90S/8fJYvNrPymjnrlrDiatxYUrDfhQ7IvGU1CWIs0+2sb+JE C0z9/sqc0V1/ONAm0xZSrf4+8BQgvLAZLBDbsxS3YLciVkTNmQ7/5crMYmhPv4Iz PEG6r+fIfdykiIS9gC3lsjE9UU1jD7bIQmxLaD7vbD1IOoJOxMu+h/Ij0+AwEpe4 NfZyHfhiVjOZwxVdlaCUNRpNgeh15CX7/u8V5YEkL2LBaPQ63g1B/1bQkWkV2h68 lYfk/8zHusO1WyxmmCcxunwU77xXTD2oNvkpqRyP1D4Mz9weW/xPwGErF6W2cj7U gHDPH30a1MpQKQ3pxDLo =WHhS -END PGP SIGNATURE- -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Shorewall-users mailing list Shorewall-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/shorewall-users
[Shorewall-users] Firewall hangs
We have Shorewall 4 protecting the school network from a group of Raspberry Pis, which we operate from PCs on the school network using VNC running through Shorewall. For some weeks we've had frequent problems with VNC sessions hanging for around 30 seconds. I've been trying to track it down with increasingly focussed Wireshark captures, and this is what seems to be happening on one fairly typical hang: Two Pis are being controlled from separate PCs. I have ping running from the firewall to one of the Pis and also from the firewall to the default gateway on the school network. Hundreds of packets are passing through the firewall from one of the Pis to the PC controlling it, containing VNC screen update data. These are interspersed every second by a ping/reply to one of the Pis and a ping/reply to the default gateway. Suddenly TCP retransmissions of VNC traffic start appearing. Often at this point you see one or two other packets, such as an ntp or a VNC from the other Pi, but this may only be because they're no longer being hidden amongst a mass of VNC. More retransmissions from the Pi(s) but nothing on the school network NIC, and in particular, no pings to the default gateway. After around 10 seconds, the Pi network NIC sends ICMP network unreachable to both Pis. Sometimes I've seen ICMP host unreachable, I think from the school network NIC back to a Pi. Other times I've seen RST, ACK packets from one of the VNC client PCs - I don't see RST, ACK in the standard TCP state diagram. After a total of around 30 seconds, everything seems to recover, and pings reappear on the school network, though VNC generally has to open a new TCP connection. Only fairly recently have we regularly run more than one Pi at the same time. Maybe we're just running out of kernel buffers? Or we need a more powerful machine to run Shorewall? (It's an unremarkable desktop machine maybe 5 years old.) Or maybe I've just got something misconfigured. Ideas please? Regards - Philip -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Shorewall-users mailing list Shorewall-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/shorewall-users
