Heres a watered down and cleansed version of my pf.conf and a relevant
packet capture. pf.conf file is the same on both boxes. Traffic originated
externally(10.0.0.5) hitting the webserver (192.168.0.100) will be broadcast
by the switch, hit the secondary firewalls internal interface, and should be
blocked at this point due to the ruleset as shown. However I do not see it
being blocked in pflog0 but I do see it on the internal interface...
...
07:21:20.011003 10.0.0.5.4665 > 192.168.0.100.80: P 0:524(524) ack 1 win
65003 (DF)
07:21:20.011031 10.0.0.5.4665 > 192.168.0.100.80: P 0:524(524) ack 1 win
65003 (DF)
07:21:20.011060 10.0.0.5.4665 > 192.168.0.100.80: P 0:524(524) ack 1 win
65003 (DF)
07:21:20.011079 10.0.0.5.4665 > 192.168.0.100.80: P 0:524(524) ack 1 win
65003 (DF)
07:21:25.999875 10.0.0.5.4665 > 192.168.0.100.80: P 0:524(524) ack 1 win
65003 (DF) [ttl 1]
07:21:25.999886 10.0.0.5.4665 > 192.168.0.100.80: P 0:524(524) ack 1 win
65003 (DF) [ttl 1]
07:21:25.999932 10.0.0.5.4665 > 192.168.0.100.80: P 0:524(524) ack 1 win
65003 (DF) [ttl 1]
07:21:25.999982 10.0.0.5.4665 > 192.168.0.100.80: P 0:524(524) ack 1 win
65003 (DF) [ttl 1]
......
/etc/pf.conf
#MACROS & TABLES
#INTERFACES
ext_if="em0"
int_if="em1"
xover_if="em2"
vpn_if="enc0"
pfsync_if="pfsync0"
table <routed_net> { 192.168.0.0/24 }
table <web> { 192.168.0.100 }
#OPTIONS
set limit { states 1000000, src-nodes 1000000, frags 500000, tables
1000, table-entries 300000 }
set skip on { lo }
scrub in all
#PACKET FILTER
#Default Deny
block in log all
block out log all
#Pass Traffic allowed that was allowed in to interface outward
pass out quick on { $ext_if $int_if $xover_if }
#CARP
pass quick on { $int_if $ext_if } proto carp
#PFSYNC
pass quick on $xover_if proto pfsync
#OUTBOUND POLICY
pass in quick on { $int_if $vpn_if } proto tcp from <routed_net> to
any
pass in quick on { $int_if $vpn_if } proto udp from <routed_net> to
any
#INBOUND POLICY
pass in log quick on { $ext_if $vpn_if } proto tcp from any to <web>
port { 80 443 }
-----
On Dec 26, 2007 1:26 PM, Fridiric Pli <[EMAIL PROTECTED]> wrote:
>
>
> On 26/12/2007, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
> >
> > Did you try to tcpdump on the interface that support CARP interface too
> > > ?
> > >
> >
> > I did as well as the external and xover interfaces to see if it was
> > forwarding it to another interface and back in for some reason. It was
not.
> >
>
> Stricly the same problem
>
> [...]
> >
> >
> > This makes sense as to what I am seeing but it is my understanding that
> > traffic recieved on one interface cannot be forwarded back thru the same
> > interface. (At least on cisco hardware), although I do remember
accomplising
> > this thru a GRE tunnel between two Openbsd 4.0 boxes when we moved
> > datacenters and had to renumber ips.
> >
> > >
> > > The dirty workaround I found is to filter with pf incoming traffic
> > > going to networks behind the firewalls on my both openbsd routers (this
> > > traffic should be received only by the firewall boxes).
> > > I thought about modify openbsd Ethernet layer to drop incoming packets
> > > with the firewall mac multicast as destination address but that is a
really
> > > silly way to do.
> > >
> >
> > I came to a similar conclusion and workaround as I attempted to deny
> > traffic to the internal interface destined to the network behind the
> > firewalls. However this did not seem to mitigate the problem.
> >
>
> I does for me.
> I don t know if that can help you but here are the PF rules I set :
> (default policy is block)
>
> # Traffic from internet
> pass in quick on { vlan0 vlan1 } inet from $internet to $our_nets
> pass out quick on dc3 inet from $internet to $our_nets
> # Traffic to internet
> pass out quick on { vlan0 vlan1 } inet from $our_nets to $internet
> pass in quick on dc3 inet from $our_nets to $internet
>
> vlan0 and vlan1 are interfaces to my 2 Internet transit providers.
> dc3 is the interface to the (stonegate) firewalls. $our_nets are routed to
> those firewalls.
>
> Let me know if you fix the problem
>
> Regards
>
> Fred
>
> I would be interested in any clue to apply a proper fix to this problem.
> > >
> > > Fred
> > >
> > >
> > > On 23/12/2007, [EMAIL PROTECTED] <[EMAIL PROTECTED] > wrote:
> > > >
> > > > I'm having an issue, maybe someone has seen before or can help me
> > > > with.
> > > >
> > > > Scenario:
> > > > I have 2 firewall boxes with carp on the outer and inner interfaces
> > > > of our
> > > > network and pfsync running between them. On the inner side of the
> > > > firewalls
> > > > they drop into 2 cisco 3750G switches that are stacked using
> > > > stackwise.
> > > > There is a cluster of web servers sitting behind the firewalls
> > > > running
> > > > Micosoft IIS and NLB in Multicast mode with IGMP. When packets come
> > > > in
> > > > destined for the web cluster they are broadcast across all ports on
> > > > the
> > > > switch due to the MAC being sent out multiple ports. The cisco's
> > > > don't like
> > > > this and spit out the packet on all ports and igmp snooping doesnt
> > > > work due
> > > > to the ms implementation. Cisco wont help us because they say that
> > > > Microsoft
> > > > isnt following the RFC correctly and Microsoft says there is a patch
> > > > for
> > > > this in the works but its been like this for years so I'm not
> > > > holding my
> > > > breath. I'm not too concerned with this. We know how to deal with it
> > > > by
> > > > mapping the multicast mac address to the static ports the webservers
> > > > are on.
> > > >
> > > >
> > > > Situation:
> > > > The problem came into play when we needed to replace some of our
> > > > cisco
> > > > switches and had to delete the static mac addresses on the ciscos in
> > > > order
> > > > not to blackhole webservers during the transition. After we deleted
> > > > the mac
> > > > addresses on the cisco's all ports were once again flooded with
> > > > inbound web
> > > > traffic during the maintenance. This we expected.
> > > >
> > > > The Problem:
> > > > However what we didn't expect was our carp devices to go haywire.
> > > > They were
> > > > flapping back and forth and we had intermittent connectivity issues
> > > > until we
> > > > unplugged one of the boxes and our connection was stable again. It
> > > > didnt
> > > > matter witch one we unplugged. As soon as we unplugged the opposite
> > > > device
> > > > the connection was stable again. At the time there may have been
> > > > about 25mb
> > > > of traffic to our webservers.
> > > >
> > > > The only thing that makes sense to me is some sort of race condition
> > > > with
> > > > the broadcast messages. Does this make sense to anyone? Currently we
> > > > have an
> > > > advbase of 1. Now I havent attempted to bump that up. Should I? I
> > > > just
> > > > wanted to get some opinions on this before I make any changes.
> > > >
> > > > Has anyone seen this behavior before? and know how to solve it
> > > > correctly?
> > > > Thanks.
