On Wed, 24 Apr 2019 at 10:31, likun wrote:
> Hello, Stephen, thank you for input.
>
> Yes, these servers have the same firewall rules, and both of them have the
> same problem from time to time, most of time they are good.
>
> Actually, these servers are newly installed to be used as the Glusterfs
> storage server, so not much data flowing at this time.
> From the sysctl output, I suppose it can't be a conntrack table overflow :
>
> net.netfilter.nf_conntrack_count = 1116
> net.netfilter.nf_conntrack_max = 262144
>
> And another tcpdump ouput of a successful ssh connection between these two
> servers for reference:
>
> 21:41:53.225977 IP (tos 0x0, ttl 64, id 30083, offset 0, flags [DF], proto
> TCP (6), length 60)
> 10.3.3.3.49221 > 10.3.3.4.22: Flags [S], cksum 0x1ab0 (incorrect ->
> 0x62bc), seq 320483660, win 29200, options [mss 1460,sackOK,TS val
> 253838543 ecr 0,nop,wscale 7], length 0
> 21:41:53.226083 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP
> (6), length 60)
> 10.3.3.4.22 > 10.3.3.3.49221: Flags [S.], cksum 0x15da (correct), seq
> 3543971827, ack 320483661, win 28960, options [mss 1460,sackOK,TS val
> 94551278 ecr 253838543,nop,wscale 7], length 0
> 21:41:53.226116 IP (tos 0x0, ttl 64, id 30084, offset 0, flags [DF], proto
> TCP (6), length 52)
> 10.3.3.3.49221 > 10.3.3.4.22: Flags [.], cksum 0x1aa8 (incorrect ->
> 0xb4e1), seq 1, ack 1, win 229, options [nop,nop,TS val 253838543 ecr
> 94551278], length 0
>
>
I notice these have checksum's in them but the first set didn't. Was there
an option changed in the tcpdump? It is clear that I am wrong about the
[SA] being needed since this set does have an [S.] Sorry I don't have any
clues.. as far as I can tell it should have worked. Could you put a log
before the REJECT to see if something else stands out?
> Likun
>
> -邮件原件-
> 发件人: CentOS [mailto:centos-boun...@centos.org] 代表 Stephen John Smoogen
> 发送时间: 2019年4月24日 18:35
> 收件人: CentOS mailing list
> 主题: Re: [CentOS] Iptables blocks out going connetion some times
>
> On Wed, 24 Apr 2019 at 06:01, likun wrote:
>
> > Hi guys.
> >
> > There is a wierd problem with iptables recently, hopes somebody can
> > help me.
> >
> > I have installed Centos 7.2.1511 on a bare metal Dell server these
> > days, disabled firewalld and enabled iptables.services, and setup a
> > group of very simple rules, as the following:
> >
> >
> I believe you are saying both 10.3.3.3 and 10.3.3.4 have this same
> firewall but I am not sure.
>
>
>
> > # iptables-save
> >
> > # Generated by iptables-save v1.4.21 on Tue Apr 23 09:15:14 2019
> >
> > *filter
> >
> > :INPUT ACCEPT [0:0]
> >
> > :FORWARD ACCEPT [0:0]
> >
> > :OUTPUT ACCEPT [2449555:327804572]
> >
> > -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
> >
> > -A INPUT -p icmp -j ACCEPT
> >
> > -A INPUT -i lo -j ACCEPT
> >
> > -A INPUT -s 172.22.0.0/16 -p tcp -m tcp --dport 49152:49664 -m
> > conntrack --ctstate NEW -j ACCEPT
> >
> > -A INPUT -s 10.3.3.0/25 -p tcp -m tcp --dport 49152:49664 -m conntrack
> > --ctstate NEW -j ACCEPT
> >
> > -A INPUT -s 172.22.0.0/16 -p tcp -m tcp --dport 24007 -m conntrack
> > --ctstate NEW -j ACCEPT
> >
> > -A INPUT -s 10.3.3.0/25 -p tcp -m tcp --dport 24007 -m conntrack
> > --ctstate NEW -j ACCEPT
> >
> > -A INPUT -s 10.3.3.0/25 -p tcp -m tcp --dport 22 -m conntrack
> > --ctstate NEW -j ACCEPT
> >
> > -A INPUT -s 10.3.7.0/25 -p tcp -m tcp --dport 22 -m conntrack
> > --ctstate NEW -j ACCEPT
> >
> > -A INPUT -j REJECT --reject-with icmp-host-prohibited
> >
> > -A FORWARD -j REJECT --reject-with icmp-host-prohibited
> >
> > COMMIT
> >
> > # Completed on Tue Apr 23 09:15:14 2019
> >
> >
> >
> > From time to time, when this server, say 10.3.3.3, trying to connect
> > to port
> > 24007 of another server 10.3.3.4, it will fail sometimes. from tcpdump
> > output, you can see these packages:
> >
> > 22:49:05.992737 IP 10.3.3.3.49149 > 10.3.3.4.24007: Flags [S], seq
> > 2454712274, win 29200, options [mss 1460,sackOK,TS val 24055648 ecr
> > 0,nop,wscale 7], length 0
> >
> > 22:49:05.992847 IP 10.3.3.4.24007 > 10.3.3.3.49149: Flags [S.], seq
> > 3127562073, ack 2454712275, win 28960, options [mss 1460,sackOK,TS val
> > 17803660 ecr 24055648,nop,wscale 7], length 0
> >
> > 22:49:05.992872 IP 10.3.3.3 > 10.3.3.4: ICMP host 10.3.3.3 unreachable
> > - admin prohibited, length 68
> >
> >
> >
> So it looks like the packet getting rejected wasn't just a SYN packet but
> had something extra tagged NONE tagged in it [S.]. Do the packets which
> don't trigger an ICMP return contain a [SA] or a [S.] ?
>
> My guess is that something in that return packet is making the cstate not
> be NEW or ESTABLISHED and so it is failing down to the drop. If the packets
> which are accepted look the same then I would look to see if the conntrack
> table is overflowing.. if the memory is tight and the traffic is high,
> streams will get pushed out and you will get similar failures. That however
> usally puts something in dmesg when
