I'm having an issue, maybe someone has seen before or can help me with. Scenario: I have 2 firewall boxes with carp on the outer and inner interfaces of our network and pfsync running between them. On the inner side of the firewalls they drop into 2 cisco 3750G switches that are stacked using stackwise. There is a cluster of web servers sitting behind the firewalls running Micosoft IIS and NLB in Multicast mode with IGMP. When packets come in destined for the web cluster they are broadcast across all ports on the switch due to the MAC being sent out multiple ports. The cisco's don't like this and spit out the packet on all ports and igmp snooping doesnt work due to the ms implementation. Cisco wont help us because they say that Microsoft isnt following the RFC correctly and Microsoft says there is a patch for this in the works but its been like this for years so I'm not holding my breath. I'm not too concerned with this. We know how to deal with it by mapping the multicast mac address to the static ports the webservers are on.
Situation: The problem came into play when we needed to replace some of our cisco switches and had to delete the static mac addresses on the ciscos in order not to blackhole webservers during the transition. After we deleted the mac addresses on the cisco's all ports were once again flooded with inbound web traffic during the maintenance. This we expected. The Problem: However what we didn't expect was our carp devices to go haywire. They were flapping back and forth and we had intermittent connectivity issues until we unplugged one of the boxes and our connection was stable again. It didnt matter witch one we unplugged. As soon as we unplugged the opposite device the connection was stable again. At the time there may have been about 25mb of traffic to our webservers. The only thing that makes sense to me is some sort of race condition with the broadcast messages. Does this make sense to anyone? Currently we have an advbase of 1. Now I havent attempted to bump that up. Should I? I just wanted to get some opinions on this before I make any changes. Has anyone seen this behavior before? and know how to solve it correctly? Thanks.
