On Thu, Oct 21, 2010 at 12:12, Jack Neely <[email protected]> wrote: > Folks, > > I've recently upgraded our kerberos servers to RHEL 5. The four new > servers live in two data centers. Each pair behind a cisco firewall > context. Of course, I used IPTables to model what rules I needed in the > hardware firewalls. When we went production, I saw no reason to disable > the local firewalls even with the hardware firewalls in place. > > That is...until iptables started getting things wrong. > > The kerberos master, like all masters, uses kprop to propagate the > database to the slaves in a TCP connection. Kprop connections to the > kerberos slaves in the other data center would some times get stuck and > the slave would accept no new connections and the old connection would > not terminate. I would kill the child kpropd process on the slave and > things would work. At least until this happened again. (So not with > every connection.) > > Turns out, well into the TCP session the kerberos master's iptables > firewall started rejecting ACKs sent from the slave. Depending on how > badly it rejected these packets it looks like a duplicate ACK/resend > storm could arise and basically hang the TCP session. > > My iptables rules are fairly basic and fairly standard. > > -A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT > > and the kprop is a connection initiated on the master to the slaves. > There are no rules on OUTPUT besides a general ACCEPT policy.
I would need to see more of the rules to get a better idea. Most of the times I have seen issues like this it comes down to a packet coming across as INVALID versus NEW due to something in an upstream firewall handling packets. EG when I was seeing similar issues with SSH I needed to change my -A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT to -A INPUT -m state --state NEW,INVALID -m tcp -p tcp --dport 22 -j ACCEPT then ssh quit having problems like what you are listing. > Why would iptables start rejecting packets for an established TCP > session? When I added a rule to explicitly allow traffic from a source > port of 754 (kpropd) the problem went away. I've checked my connection > tracking table and its no where near full. Usually a packet comes in wrong or resent and so it looks 'INVALID' . Of course because it falls off the end and gets a ICMP resend or some similar thing your session storm occurs. > This problem never occurred talking to the kerberos slave in the same > data center and behind the same cisco firewall. It only occurs when > the packets traverse the cisco firewalls. Can anyone shed some light on > what is happening here? > > Jack Neely > > -- > Jack Neely <[email protected]> > Linux Czar, OIT Campus Linux Services > Office of Information Technology, NC State University > GPG Fingerprint: 1917 5AC1 E828 9337 7AA4 EA6B 213B 765F 3B6A 5B89 > > _______________________________________________ > rhelv5-list mailing list > [email protected] > https://www.redhat.com/mailman/listinfo/rhelv5-list > -- Stephen J Smoogen. “The core skill of innovators is error recovery, not failure avoidance.” Randy Nelson, President of Pixar University. "We have a strategic plan. It's called doing things."" — Herb Kelleher, founder Southwest Airlines _______________________________________________ rhelv5-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/rhelv5-list
