/bsd: carpXX: ip_output failed: 64
Hello all, Since I added another physical and carp interface to our firewalls, I get strange error messages, and strange behaviour for carp failover. Jul 25 15:00:03 fw2 /bsd: carp32: ip_output failed: 64 Jul 25 15:00:03 fw2 /bsd: carp34: ip_output failed: 64 Jul 25 15:00:03 fw2 /bsd: carp40: ip_output failed: 64 Jul 25 15:00:03 fw2 /bsd: carp5: ip_output failed: 64 Jul 25 15:00:03 fw2 /bsd: carp11: ip_output failed: 64 Jul 25 15:00:03 fw2 /bsd: carp31: ip_output failed: 64 Jul 25 15:00:04 fw2 /bsd: carp: carp31 demoted group carp by 1 to 2\ (> snderrors) I'm pretty sure that I had no of that "64" errors before I added "carp 40" - the old logfiles show none (though I have only those from a few days). I googled for the error but didn't find anything helpful. Any quick pointers what may be going wrong? Marcus
Re: CARP compatibility
Router 2 carp1: flags=8803 mtu 1500 lladdr 00:00:00:00:00:00 priority: 0 carp: INIT carpdev none vhid 2 advbase 1 advskew 0 groups: carp This mightily looks like some other interface is trying to use the same IP-address (the 00:00:00:00:00:00 hints at that). In that case the carp interface naturally remains at INIT. Marcus
pf: Load Balancing Outgoing traffic over multiple WAN-connections with something like "sticky address"
Hello list,
is it possible to make outgoing traffic load-balance in a way that
connections from the same internal IP to the same external IP always use
the same WAN-connection (at least until the
The example under
> http://www.openbsd.org/faq/pf/pools.html#outexample
circumvents it by using only one connection. It would be nice if I could
use something like:
pass in on $int_if from $lan_net \
route-to { ($ext_if1 $ext_gw1), ($ext_if2 $ext_gw2) } \
round-robin \
"target-hash"
using "round-robin sticky-address" or "source-hash" obviously does not
work, since there is not a single redirection address but *two*.
Is there any other workaround than hardwiring only one connection?
Marcus
Just curious: what happens on "tcpdump -nei pfsync0"?
Hello all, just curious on a strange behaviour of an active/active firewall configuration: when I do a "tcpdump -nei pfsync0" to watch what sort of state updates are passed and do a "tail -f /var/log/messages" I see that in the moment the command is issued the VHID CARP interfaces on that firewall are demoted from MASTER to BACKUP, just to go back from BACKUP to MASTER a second later. When I stop the tcpdump the same behaviour happens. Doing the tcpdump on the physical interface does not have such an effect. What am I missing? Marcus
Can't get multipath working correctly
Hello all, please forgive if my question turns out to be very hazy and unclear. If I could myself clearer I could probably understand what happens ;) I have set up a pf firewall with two external NICs and CARP on that external IPs. (I think) I followed http://www.openbsd.org/faq/faq6.html#Multipath Now, for example if I sent a ping to 8.8.8.8, I see that the pf rule makes a pass out on em1: *WAN_IP_1* > 8.8.8.8: icmp: echo request however, the first request sometimes goes out on em0, while the echo replies and all other echo requests use em1. Sometimes something similar happens when a connection comes in through an external interface: the first return packet goes out through the wrong interface, and is thus blocked (duh!) - though I'm pretty sure (and can see it through tcpdump) that I've set a "pass in" rule with reply-to *ROUTER_IP*@em1". Now I see that "netstat -rn" shows me Destination Gateway Flags Refs Use Mtu Prio Iface default ROUTER_IP_0 UGSP 2 83 - 8 em0 default ROUTER_IP_1 UGSP 2 92 - 8 em1 *WAN_NET_0*/29 link#1 UC 20 - 4 em0 *WAN_NET_1*/29 link#4 UC 30 - 4 em2 If I understand correctly, something for WAN_NET_1 is pointing wrong. After a reboot I have even seen once that *both* links pointed wrong, the *WAN_NET_0* on em0 to the *ROUTER_IP* on em1. Now I have three questions: 1) Is this really the error? 2) What can I do to correct it manually? 3) What mistake did I do in the first place in my hostname.em and hostname.carp files? Marcus
Re: Wildest Africa Tour
Am 06.04.2011 11:52, schrieb Robert:
On Wed, 06 Apr 2011 11:34:23 +0200
Marcus M|lb|sch wrote:
# sudo pkg_add -v lion
results in
"Can't find lion"
What am i doing wrong?
They are in ports, not packages:
$ grep -R lion /usr/ports/
/usr/ports/games/falconseye/pkg/PLIST:${GAMEDIR}/sound/lion.raw
/usr/ports/net/gajim/pkg/PLIST:share/gajim/data/emoticons/static/lion.png
...
So, to actually increase physical security I also need a speaker
connected to my pf-Firewall?
Okay, a really big speaker with a corresponding big amplifier will
probably do the trick!
Re: Wildest Africa Tour
Am 04.04.2011 19:09, schrieb Stuart VanZee: Don't be silly. While Lions do provide excelent physical security they don't provide any data security at all. # sudo pkg_add -v lion results in "Can't find lion" What am i doing wrong?
Re: Firewall sends wrong MAC address per ARP?
Am 22.03.2011 14:42, schrieb Claudio Jeker: The lladdr is not wrong. It just happens to be the one for the second vhid. Since you do arp balancing the two lladdrs are split among the various hosts on the lan. Your carp setup runs with two MACs 00:00:5e:00:01:21 for vid 33 and 00:00:5e:00:01:85 for vid 133. So the MAC addr your linux box got is not wrong. Does the traffic from the linux box end up on the FW or is the traffic lost somewhere in between? Thanks, that helped a lot. I didn't realize that arp balancing with two vhids necessarily creates two MACs. Switching between ARP and IP balancing and back again I'm now back at ARP balancing. The fw advertises now at 00:00:5e:00:01:85 and reacts to pings at 192.168.3.1 Changing the arp table on the linux host to 00:00:5e:00:01:21 with "arp -s 192.168.3.1 00:00:5e:00:01:21" results in the fw reacting to the pings correctly, too. I should have watched the traffic with "tcpdump -e" before, however I forgot about the usefulness of that switch when watching physical interfaces. Dumb, but these things happen. Now I see that pings arrive at the fw and are replied to correctly. All other traffic through the fw is also routed correctly. Why it did not work before I cannot say. Something changed, and probably it was me who did it, but I cannot say what, how and when. diffing the pf.conf files before and afterwards showed nothing. Thanks to all, Marcus
Re: (solved?) Firewall sends wrong MAC address per ARP?
Am 22.03.2011 13:57, schrieb Marcus M|lb|sch: Duh! Now the machine replies with the correct MAC-address: 14:19:01.314759 arp who-has 192.168.3.1 tell 192.168.3.205 14:19:01.314785 arp reply 192.168.3.1 (01:00:5e:00:01:21) is-at 01:00:5e:00:01:21 I did restart networking again and did restart pf again. However, I feel that was unrelated, since the pings started to work sometimes afterwards. Now I wonder: Did the FW change its virtual MAC-address sometimes today? Maybe after the last netstart, and I didn't notice at first, since I was connected via the physical address? How could that happen? Will it happen again? Which cache did serve the wrong MAC-Address? Was it the switch? Is there any way in which I can tell the FW to use a specific virtual MAC-address when carping? Marcus
Re: Firewall sends wrong MAC address per ARP?
More Info: - Neither rebooting the FW nor the linux machine did change anything - changing the load balancing from "arp balancing" to "ip balancing" did not change anything. - At first I thought it might be a problem of the switch and it has an "old" virtual IP address cached. However, the log on the FW does show that the machgine itself replies to to the arp-request, does it not? - it happened "suddenly". I did change a pf-rule and restarted pf; however I did not restart networking (AFAIK) - unfortunately I cannot determine whether the "wrong" lladdress was used as virtual address before. I did not note it down, before this happened. Marcus
Re: Firewall sends wrong MAC address per ARP?
Am 22.03.2011 13:27, schrieb Patrick Lamaiziere: Le Tue, 22 Mar 2011 13:01:48 +0100, Marcus M|lb|sch a icrit : hello, carp3: flags=8843 mtu 1500 lladdr 00:00:5e:00:01:21 priority: 0 carp: carpdev bge0 advbase 1 balancing arp carppeer 192.168.3.3 state MASTER vhid 33 advskew 0 state MASTER vhid 133 advskew 100 Why do you have two vhid and with different advskew values? To set up a second FW with active/active configuration as shown here: http://www.kernel-panic.it/openbsd/carp/carp4.html#carp-4.2.2 That does work when the second FW is up; however for testing purposes this machine is now down. Marcus
Firewall sends wrong MAC address per ARP?
Hello all, a linux machine connected to the OpenBSD-FW (which uses CARP) cannot connect to the Firewall. Somehow it got the worng Mac-Address in its ARP-table. I removed it manually, but when pinging again the OpenBSD firewall send the worng address: "tcpdump -nvi bge0 arp" on the OpenBSD machine shows: > tcpdump: listening on bge0, link-type EN10MB > 12:47:16.467905 arp who-has 192.168.3.1 tell 192.168.3.200 > 12:47:16.467939 arp reply 192.168.3.1 is-at 00:00:5e:00:01:85 Whereas "ifconfig carp3" shows the Virtual MAC-address to be: carp3: flags=8843 mtu 1500 lladdr 00:00:5e:00:01:21 priority: 0 carp: carpdev bge0 advbase 1 balancing arp carppeer 192.168.3.3 state MASTER vhid 33 advskew 0 state MASTER vhid 133 advskew 100 groups: carp dmz status: master inet6 fe80::200:5eff:fe00:121%carp3 prefixlen 64 scopeid 0x12 inet 192.168.3.1 netmask 0xff00 broadcast 192.168.3.255 No wonder the linux machine again cannot connect. arp -n Address HWtype HWaddress Flags MaskIface 192.168.3.1 ether 00:00:5e:00:01:85 C eth0 Can anybody tell me what went on here? Marcus
DNS queries through pf firewall takes "forever"
Hello all, I'm struggling with my pf configuration again. Problem is: pinging to an IP is as fast as I suspect it to be; pinging to a FQDN is slow. From a computer in the DMZ I try to ping to "heise.de" (which resolves to 193.99.144.80) When looking at pflog I see something like this on port 53 (somewhat shortened to avoid line breaks). r 419 pass in on bge0: .37749 > .53: 40380+[|domain] (DF) r 380 match in on bge0: .37749 > .53: 40380+[|domain] (DF) r 345 pass out on em0: .37749 > .53: 40380+[|domain] r 329 match out on em0: .37749 > .53: 40380+[|domain] (DF) with bge0 being the DMZ interface and em0 being the external interface pfctl -vvsr shows the rules as: @419 pass in log quick on bge0 inet all flags S/SA keep state tagged dmz-ok route-to @em0 @380 match in log on bge0 inet proto udp from to port = domain tag dmz-ok @349 pass out log quick on em0 inet proto udp all keep state tagged dmz-ok @329 match out log on em0 inet proto udp from to any nat-to tcpdumping at the external interface I see: .64578 > .53: [udp sum ok] 7399+ A? heise.de. (26) (ttl 64, id 14713, len 54) .53 > .64578: [udp sum ok] 7399 1/0/0 heise.de. A 193.99.144.80 (42) (DF) (ttl 56, id 0, len 70) .54038 > .53: [udp sum ok] 21898+ PTR? 80.144.99.193.in-addr.arpa. (44) (ttl 64, id 25755, len 72) .53 > .54038: 21898 1/0/0 80.144.99.193.in-addr.arpa. PTR[|domain] (DF) (ttl 56, id 0, len 105) I freely admit that I am at my wits end. Everything looks fine to me, but still something is wrong. Any pointer is highly appreciated. Marcus M|lb|sch
Re: CARP between 4.7 and 48 possible?
Am 19.01.2011 12:42, schrieb Alastair Johnson: We also have a very similar question. I have a pair of CARP'd& pfsync'd firewalls of embarrasing vintage and would like to be able to swap them for new 4.8 machines (new hardware) one at a time. Is this possible? Note that beginning with 4.7 the syntax has been changed for the "match", "rdr-to" and "nat-to". Prepare to make (and test) major changes to your pf.conf Marcus
Re: CARP between 4.7 and 48 possible?
So far both machines claim to be master on an active/active configuration on both assigned VHIDs. Quick Update: After changing from sending CARP per multicast to ip-unicast the newer machine is backup on both VHIDs. If the older machine goes down, the newer machine gets master on both VHIDs, until the older machine gets up again. tcpdump shows that CARP traffic flows freely on the 4.8 machine: "hostname.carp3" inet 192.168.3.1 255.255.255.0 192.168.3.255 balancing ip-unicast carppeer 192.168.3.4 carpdev bge0 pass somepass carpnodes 33:0,233:100 and on the 4.7 machine: "hostname.carp3" inet 192.168.3.1 255.255.255.0 192.168.3.255 balancing ip-unicast carppeer 192.168.3.2 carpdev bge0 pass somepass carpnodes 33:100,233:0 Marcus
CARP between 4.7 and 48 possible?
A quick question: Is it possible to make a carp interface between two machines if one is running on openBSD 4.7 and the other on openBSD 4.8? Background: Finally a got a second machine for our Firewall. Since the working one runs on 4.7 and I am reluctant to upgrade it before I got the second one working I try to CARP the interfaces first, then get the new one working, then take the first one of the net and upgrade it to. So far both machines claim to be master on an active/active configuration on both assigned VHIDs. It's entirely possible that I misconfigured something, but before I check everything a third (fourth, nth) time I'd rather be sure that that has nothing to do with the machines being on a different version. Marcus
An idea for a very simple port knocking with pf
Hello all, it occured to me that with a combination of some pass rules and adding the address via overload to a sort of "whitelist" tables you can implement a simple portknocking; using nothing but pf. The rules would look like this: pass in on $ext_if inet proto tcp from any to any port $knock1 synproxy state (max-src-conn 1 overload ) pass in on $ext_if inet proto tcp from to any port $knock2 synproxy state (max-src-conn 1 overload ) pass in on $ext_if inet proto tcp from to any port $knock3 synproxy state (max-src-conn 1 overload ) pass in on $ext_if inet proto tcp from to any port $knock3 synproxy state (max-src-conn 1 overload ) pass in on $ext_if inet proto tcp from to any port ssh No port knocking daemeon is needed, and with an appropriate blocking rule the ssh port is closed to all. This works; all you have to do is to try to connect to each port $knock in order twice (since the max-src-conn is set to 1). I have two questions: 1) Is there any problem with that setup? I don't see any, but then again, it seems so simple and I didn't find any howtos on the web. Either nobody else did think of it before, or there is something wrong with my reasoning. If so, I'm happy if you tell me :-) 2) I would like to knock on each port only once. However, setting "max-src-conn 0" does not change anything. I would expect that the first connect will fill the appropriate table, but it doesn't. Is there something I do not understand, or must the that is allowed be equal or greater to one? Thanks for any pointers, Marcus
relayd and smpp?
Hello all, Is there any way how relayd can loadbalance two several SMPP-Servers? As far as I can see it only speaks http or https? Has anybody done so? Thanks, Marcus
Re: Problems with Carp, Multi-WAN and pf syntax.
Stuart Henderson schrieb: you're probably looking for "reply-to", something along these lines: pass in quick on gif1 inet to (gif1) reply-to 10.33@gif1 pass in quick on pppoe0 inet to (pppoe0) reply-to 0.0@pppoe0 Yes I was. Except that the syntax was not exactly clear to me if you want a packet both to redirect-to an internal interface and then reply-to an external interface. Now I found out that the following does work: # Redirect WWW traffic pass in log quick on $if_wan1 inet proto tcp from any to any \ reply-to ( $if_wan1 $gw_wan1 ) rdr-to $srv_www round-robin (And similar lines for the other interfaces) My only "problem" is that the rule resolves to: > pass in log quick on em0 inet proto tcp from any to any flags S/SA keep state reply-to @em0 if shown with "pfctl -sr" In fact pfctl -sr does not show a single redirection, nor does it show that it does redirect to several servers in a round-robin-manner; though obviously it does. While I'm not perfectly happy with that, at least I'm now in a state of "works for me". Thank you all. Marcus
Re: Problems with Carp, Multi-WAN and pf syntax.
Marcus M|lb|sch schrieb:
How do I configure a pf in a way that traffic that comes in one one
CARP-Interface goes out to the same CARP-Interface? The syntax in
-current has changed from the FAQ (which assumes OpenBSD-4.6).
After some help from a friendly soul, and reducinge my pf.conf to
the bare minimum it still does not work as intended. Either I have hit a
bug, or still have a wrong conf.
NICS are configured so:
# /etc/hostname.bge0
inet 192.168.3.1 255.255.255.0 192.168.3.255
# /etc/hostname.em0 (WAN-1)
inet 255.255.255.248
!route add -mpath default
# /etc/hostname.em1 (WAN-2)
inet 255.255.255.248
!route add -mpath default
sysctl is configured for multipath and forwarding:
# /etc/syctl.conf
net.inet.ip.forwarding=1
net.inet.ip.multipath=1
pf.conf looks like this:
# /etc/pf.conf
# Macros
if_wan1 = "em0"
if_wan2 = "em1"
if_wan = "{" $if_wan1 $if_wan2 "}"
if_dmz = "bge0"
gw_wan1 =
gw_wan2 =
# Allow ICMP
passin log quick on $if_wan inet proto icmp from any to any
# Redirect WWW traffic
passin log quick on $if_wan inet proto tcp from any to any
rdr-to round-robin
# NAT for outgoing connections on each internet interface
passout logon $if_wan1from any to any nat-to ($if_wan1)
passout logon $if_wan2from any to any nat-to ($if_wan2)
# route packets from any IPs on $if_wan1 to $gw_wan2 and the same for
$if_versa and $gw_versa
passout log quick on $if_wan1 from $if_wan2
route-to ($if_wan2 $gw_wan2)
passout log quick on $if_wan2 from $if_wan1
route-to ($if_wan1 $gw_wan1)
At first everything seems to be fine:
Accessing the www-servers from outside per the wan2 interface works as
intended: The traffic goes in through the wan2 interface, gets
redirected to the www-servers via round robin (if one of them goes down
that doesn't matter, as is the whole idea), and gets back through wan2.
However. If I access the www-servers from outside via wan-1 ip, 50% of
the time the traffic tries to go back through the wan-2 interface, and
that is something I don't understand.
Same for ICMP.
Any help?
Marcus M|lb|sch
Problems with Carp, Multi-WAN and pf syntax.
Hello all, How do I configure a pf in a way that traffic that comes in one one CARP-Interface goes out to the same CARP-Interface? The syntax in -current has changed from the FAQ (which assumes OpenBSD-4.6). http://www.openbsd.org/faq/pf/pools.html#outgoing On a HP ProLiant with BCM5703X NICS I had to go with -current, because the NICS do not work with 4.6 (see here: http://old.nabble.com/ProLiant-DL360-G3---bge-won't-work-td26746681.html and here: http://marc.info/?l=openbsd-cvs&m=12492713264&w=2 ) I can make neither head nor tails from the manpage in this regard, so can anybody help? Marcus M|lb|sch
