On 2/10/2011 7:58 PM, Vaughn L. Reid III wrote:


On 2/10/2011 7:30 PM, Moshe Katz wrote:
Is your ISP Verizon? We have had many ARP issues with Verizon FIOS. For our pfSense box to get all of our IPs, we have to manually set each of the IPs as the WAN IP (one by one), then set up the Virtual IP settings after we do that.

Moshe

------------------------------
Moshe Katz
-- mo...@ymkatz.net <mailto:mo...@ymkatz.net>
-- +1(301)867-3732



On Thu, Feb 10, 2011 at 7:19 PM, Vaughn L. Reid III <vaughn_reid_...@elitemail.org <mailto:vaughn_reid_...@elitemail.org>> wrote:



    On 2/10/2011 12:57 PM, Evgeny Yurchenko wrote:

        On 11-02-10 11:07 AM, Vaughn L. Reid III wrote:



            On 2/10/2011 10:42 AM, Vaughn L. Reid III wrote:



                On 2/10/2011 9:32 AM, Vaughn L. Reid III wrote:

                    On 2/10/2011 2:43 AM, Seth Mos wrote:

                        Op 10-2-2011 4:18, Vaughn L. Reid III schreef:




                            1. All the Master and backup status
                            notifications in the web interface
                            on both PFSense boxes show the correct status
                            2. I'll do a packet capture tomorrow and
                            see if the carp-heartbeat shows up

                            I was unaware that any Carp related
                            traffic passed between any of the
                            interfaces except the one designated as
                            the synchronization interface. I
                            need to double-check the multi-cast
                            configuration on the switch tomorrow
                            also ( I think I have multi-cast enabled
                            on the switch, but need to
                            confirm that).


                        Yes, some switch support multicast filtering,
                        I know from experience with HP switches that
                        it works with the setting on. So I know they
                        have it implemented correctly. This way not
                        all switch ports get the carp traffic unless
                        they participate in the multicast group. This
                        cuts down on broadcast a lot.

                        I recommend the HP switches, they have never
                        given me any grief as long as I've worked
                        with them. I even have a carp cluster
                        spanning 2 building across the street over a
                        fiber connection. It just works.

                        If you need a managed switch on a budget I
                        can confirm that the HP Procurve 1810-8G
                        works well. It's web managed, supports vlans
                        and basic traffic counters. It is also fanless.

                        The smallest I have in use on a carp cluster
                        is a Procurcve 2650 in combination with a
                        2900-48G. The biggest I have is a 8212zl. Do
                        note that the software in the 1810 differs a
                        lot from the other managed switches.

                        Regards,

                        Seth

                        
---------------------------------------------------------------------
                        To unsubscribe, e-mail:
                        support-unsubscr...@pfsense.com
                        <mailto:support-unsubscr...@pfsense.com>
                        For additional commands, e-mail:
                        support-h...@pfsense.com
                        <mailto:support-h...@pfsense.com>

                        Commercial support available -
                        https://portal.pfsense.org



                    I've run a packet capture and here are the results:

                    1.  Capture shows a bunch of VRRP announcements
                    from the primary firewall to destination
                    224.0.0.18.  The destination confirms this is a
                    multicast address I  believe.  According to
                    Wikipedia, VRRP and CARP share the same protocol
                    number.  So, I believe that these are CARP
                    announcements.

                    2.  All the VRRP requests had a vrrp.prio value
                    of 0 with a description of "Priority: 0 (Current
                    Master has stopped participating in VRRP)

                    3.  Over a 114 second capture, there were no VRRP
                    announcements from the secondary firewall.

                    4.  There were lots of ARP broadcast requests
                    from the secondary firewall asking for who has
                    the IP of the default gateway.  There were 0 ARP
                    requests from the primary firewall during the
                    capture period.

                    5.  There were lots of ICMP pings from both the
                    primary and secondary Pfsense firewalls to the
                    default gateway on this WAN interface.  I assume
                    this is from the Load Balance Fail-Over
                    configuration I have enabled for the cluster on
                    this interface.

                    I confirmed that the Master firewall shows itself
                    as Master for all interfaces.  I confirmed that
                    the Secondary firewall shows itself as Backup for
                    all interfaces.



                    
---------------------------------------------------------------------
                    To unsubscribe, e-mail:
                    support-unsubscr...@pfsense.com
                    <mailto:support-unsubscr...@pfsense.com>
                    For additional commands, e-mail:
                    support-h...@pfsense.com
                    <mailto:support-h...@pfsense.com>

                    Commercial support available -
                    https://portal.pfsense.org


                I performed a second capture of 3 minutes on
                malfunctioning WAN and noted identical results for
                the VRRP/CARP packets.  On the second capture,
                however, I did see ARP requests from both firewalls
                asking for the MAC of the IP of the Default Gateway
                -- this was different from my item number 4 in the
                previous post.

                I also performed a 3 minute packet capture from one
                of the known working WAN connections on the cluster.
                 The VRRP packets on that connection showed an
                origination address of the "Real" IP on
                primary/Master firewall and a multi-cast destination,
                just like the results from the problem WAN
                connection.  I also noted that the vrrp.prio value
                and description was the same on the working WAN as on
                the not-working WAN.

                Both the working WAN connection packet capture and
                the non-Working WAN packet captures show IGMP packets
                noting the entering and leaving of multi-cast groups.



                
---------------------------------------------------------------------
                To unsubscribe, e-mail:
                support-unsubscr...@pfsense.com
                <mailto:support-unsubscr...@pfsense.com>
                For additional commands, e-mail:
                support-h...@pfsense.com
                <mailto:support-h...@pfsense.com>

                Commercial support available - https://portal.pfsense.org


            One more thing.  If I unplug the connection that leads to
            the ISP's black box  from the switch and leave everything
            else in place, pings from the secondary/backup firewall
            to the CARP start working as expected.

            I'm not sure I understand this behavior.  With 2 IP
            addresses on the same subnet that can communicate with
            each other on the same VLAN of a switch, it seems to me
            that it shouldn't matter what else I plug into that
            switch (as long as it has a different IP and as long as
            it is not doing some sort of ARP cache spoofing) that
            pings should still work.

            I've asked the ISP twice to confirm that the IP's in
            question aren't being used elsewhere, and I've been
            assured both times that they are free for our use.

        You can easily check. Plug the cable from your ISP into your
        laptop, ping all your IPs in question and look at arp table,
        if you see anything then these IPs are in use.
        Your set up seems to be correct so it is either ISP or switch.
        Evgeny

        ---------------------------------------------------------------------
        To unsubscribe, e-mail: support-unsubscr...@pfsense.com
        <mailto:support-unsubscr...@pfsense.com>
        For additional commands, e-mail: support-h...@pfsense.com
        <mailto:support-h...@pfsense.com>

        Commercial support available - https://portal.pfsense.org

    I want to thank everyone for their help.  After getting into
    contact with the ISP's senior network engineer, it appears that
    the issue I'm experiencing is related to the ISP's network
    topology and passive optical fiber infrastructure.

    Thanks for everyone's assistance and advice.  I learned some new
    things about how CARP works in the process.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: support-unsubscr...@pfsense.com
    <mailto:support-unsubscr...@pfsense.com>
    For additional commands, e-mail: support-h...@pfsense.com
    <mailto:support-h...@pfsense.com>

    Commercial support available - https://portal.pfsense.org


I won't name the ISP (it's not Verizon), but basically, here's the setup:

It's a passive optical network. The engineer explained it this way. The ONT box that terminates the fiber at the customer premise is sort of like a smart fiber to ethernet converter. The switching and routing takes place inside the ISP's core network. He explained that they assigned the pfsense boxes' static IP's inside their core network and assigned MAC addresses to them. Then they do some sort of fancy MAC address translation/mapping on the ONT. So, this was causing havoc with CARP. This ISP, which is a small regional type ISP, is going to reconfigure their network so that we have a /29 or /28 between the pfsense boxes and their core, which will eliminate the MAC mappings, etc. that they normally perform in the core network.

I'm sure my understanding of this passive optical fiber network is probably not technically accurate. If anyone wants to clarify this type of topology please feel free to do that.


Here's an update on this issue that may prove helpful for other customers that reside on a passive optical network and want to use CARP. The service provider, service provider hardware vendor, and I have performed extensive troubleshooting on why CARP was not working over the passive optical network, and we have gotten it working. Basically, there were 2 issues on the Service Provider side that was preventing the CARP IP's from being accessible from beyond the ONT.

First the typology. The redundant set of pfsense boxes were plugged into a managed layer 2 switch. The ONT was plugged into this switch also. The ONT terminated at the service provider's local point of presence into a managed layer 2 switch. Up-links from the service provider's switch led to a router in their core network.

2. The Service provider, in order to make its standard residential-type configuration work efficiently, had proxy-arp enabled on its router. This being enabled was causing the original problem of the secondary firewall not being able to ping the CARP IP's. Disabling proxy-arp for the Pfsense and CARP IP's on the service provider's router fixed this initial issue, but we could still not communicate with the CARP IP's from a remote network or from inside the Service Provider's network.

3. The Service provider, as a security item, had its local point of presence switch configured to not allow communication between switch ports (even on the same VLAN). Any local traffic bound for other local traffic on the switch had to first travel to the provider's core network. Unfortunately, this broke CARP by preventing the core router from asking for which MAC's belonged to the CARP IP's. The service provider removed this restriction for the IP's in use by the PFsense box and CARP, and things started working correctly.

I hope this helps others who may have Internet Service over a Passive Optical Network and are having trouble getting CARP working.

Reply via email to