Re: IO fencing question
On Fri, Apr 07, 2006 at 12:26:45PM -0400, Barry, Christopher wrote: Thanks much for your answers. By 'soft', I mean a controlled reboot/shutdown where the power remains on even though the OS has obviously stopped running. I have not experienced any actual failures of anything, so I do not the outcome of that. Induced 'Hard' failure (e.g. pulling the plug) works perfectly. The more I look at it, and think about it, I'm guessing the problem is more related to the redundant fibre ports on the 350-24T switch, actually holding onto information about the directly connect interface, and stubbornly sticking to it if it detects any kind of signal whatsoever. I experienced this same sort of weirdness when setting up a pair of redundant routers. The two upstreams, which I had no control over, ran OSPF. If I powered off the machine, all was well. If I simply halted the machine, or there was power to it at all, their OSPF daemon would detect a link and continue to route in the direction of our downed router. The problem, in the end, was that the Dell 1850s primary onboard ethernet controller will exhibit link when there is power to the board. The secondary, and any PCI/PCI-X cards that we added on afterward, did not exhibit this behavior. -jon
Re: OpenBSD has bad security
On Mon, Mar 06, 2006 at 09:09:35PM +0100, RedShift wrote: [EMAIL PROTECTED] ~]$ nslookup www.wideopenbsd.org www.wideopenbsd.org A 129.128.5.191 129.128.5.191 Name: openbsd.sunsite.ualberta.ca Address: 129.128.5.191 www.openbsd.org www.openbsd.org A 129.128.5.191 *** insert conspiracy theory here *** Right... If I control a domain, I can point it wherever I want. This is clearly a part of their smear campaign against OpenBSD: $ host www.openbsd.org www.openbsd.org A 129.128.5.191 $ host www.wideopenbsd.org www.wideopenbsd.org A 129.128.5.191 $ host wideopenbsd.org wideopenbsd.org A 194.145.249.6 www.wideopenbsd.org points to the machine that truly runs openbsd.org, whereas wideopenbsd.org points to a machine in Sweden. Conspiracy? No. You've been had. -jon
Re: firewall (pf): where to view current scrub settings
On Mon, Feb 06, 2006 at 05:04:09PM +0100, mgEDV.net wrote: hi, if i, for example setup scrub max-mss 1462 in my pf.conf, where can i see these values have been set? is there any command that views the current scrub rules/states? btw., anybody had a look on my other posting regarding the macros for filenames in table-statements? Yes. See pfctl(8), specifically the '-s' option. -jon
Re: ifstated.conf documentation problem?
On Sun, Jan 01, 2006 at 01:50:58AM +, Karl O. Pinc wrote:
man 5 ifstated.conf says:
The init block is used to initialise the state and is executed each
time the state is entered.
But this does not seem to be true if you use 'init-state' to enter the
state. Or maybe there's something else wrong with my config below, or
with ifstated when there's no body. Or something.
Odd things happen with the following ifstated.conf. It just hangs out
in the state starting. (Starting the daemon by hand after booting.)
Things go back in sync after a state change on carp0. (carp0 starts
out in MASTER. Unpugging the cable sends it into BACKUP. At that
time ifstated first changes state to 'master', and then immediately to
'not_master'.)
OTOH, the given config works just fine if you remove the init { and
} from the starting state, leaving the action as the body of the
starting. I would expect the opposite, that without init the
state would stay in starting and have to wait for a state change
before the body was evaluated.
So, it seems that ifstated with init-state executes the body of the
inital state on startup, and ignores the init block.
The BNF seems to indicate that what you are trying to do is legal
syntax-wise. At one point I had an ifstated.conf that did something
similiar with a master switch state that was the target of init-state
-- it would help determine what the correct initial state of ifstated
was. I believe I tried to use 'set-state' in the init block there but
don't recall it working.
I've since stopped trying to do set-state in init as this does not seem
like what init was intended for. Typically all I do in init states is
execute something -- send an email or make a call to ifconfig (as the
example ifstated.conf shows).
my $.02,
-jon
Re: crash with 3.8 GENERIC.MP#353 on Dell PE 1850
On Wed, Dec 28, 2005 at 10:31:56PM +0100, Srebrenko Sehic wrote: Try to increase kern.maxclusters. It is possible that the box crashed due to network buffer shortage. pedro@ commited a fix to both -currentand 3.8-STABLE which increases buffer on-the-fly without panic. Also, run a netstat -m and look at peak usage. If that number is reaching max, increase kern.maxclusters with sysctl. Great, I'll give that a shot today after I see if I can repeat the crash. Do you happen to know what date the commit was done? I checked MARC and couldn't seem to find the commit from Pedro after 3.8-stable. Thanks! -jon
Re: crash with 3.8 GENERIC.MP#353 on Dell PE 1850
On Wed, Dec 28, 2005 at 10:31:56PM +0100, Srebrenko Sehic wrote: Try to increase kern.maxclusters. It is possible that the box crashed due to network buffer shortage. pedro@ commited a fix to both -currentand 3.8-STABLE which increases buffer on-the-fly without panic. Also, run a netstat -m and look at peak usage. If that number is reaching max, increase kern.maxclusters with sysctl. $ netstat -m 1543 mbufs in use: 1539 mbufs allocated to data 1 mbuf allocated to packet headers 3 mbufs allocated to socket names and addresses 1538/6152/6144 mbuf clusters in use (current/peak/max) 12732 Kbytes allocated to network (27% in use) 0 requests for memory denied 0 requests for memory delayed 4896 calls to protocol drain routines The peak to 6152 happened when the connectivity through the fw seemed to hang. We got the following error messages in the logs: Dec 29 16:28:54 mcu-fw-1 /bsd: WARNING: mclpool limit reached; increase kern.maxclusters Dec 29 16:28:54 mcu-fw-1 /bsd: bmc_io_wait fails : v=40 m=03 b=01 read_data Increasing kern.maxclusters does alleviate the problem, but presumably I could run into this problem again if maxclusters isn't high enough. This build is from Oct 18 so it is likely that the aformentioned commit isn't in my build. -jon
crash with 3.8 GENERIC.MP#353 on Dell PE 1850
One of my carp masters crashed last night. Fortunately, the slave immediately picked up and nobody was the wiser. Hardware: Dell Poweredge 1850 (2.8Ghz Xeon, 1G RAM, 1x 36G U360 15K RPM SCSI, 2x onboard em, 1x dual intel em card, 1x quad intel em card, attached 16x usb serial console device) Not too sure what the machine was doing network-wise at the time, but chances are it was pushing quite a bit of traffic (maybe 10Mb/s or more -- reportedly, this was right about the time 12 machines all tried to read the same very large file on a NFS server that this firewall protects). Two clues I have are that another pair of carp'd firewalls that were helping push all this traffic flopped back and forth as master/slave and the following last log message from the firewall in this pair that failed: Dec 27 18:07:47 xxx-fw-1 /bsd: WARNING: mclpool limit reached; increase kern.maxclusters Below is a trace, limited ps (machine got booted prior to me finishing getting what I needed) and a dmesg. I'm not sure if this condition is repeatable or not but will likely know later today after I do more testing. Any insight would be much appreciated! -jon trace: kernel: page fualt trap, code=0 stopped at ether_input+0x3d: testb $0x1,0(%ecx) trace: ether_input(d18a1830,0,e9806400,be,bf) at ether_input+0x3d ether_input_mbuf(d18a1830,e9806400,e9806400,d1a29800) at ether_input_mbuf+0x23 em_process_receive_interrupts(d18a1800,fffc,e9037cbc,d0346b55,b0) at em_process_receive_interrrupts+0x190 em_inter(d18a1800) at em_intr+0x93 Xintr_ioapic5() at Xintr_ioapic5+0x74 --- interrupt --- bcopy(d18a3030,d7002700,d1967b50,d7ade220,30) at bcopy+0x20 ip_output(d7031400,0,d05d37e4,1,0,0,0,0) at ip_output+0x87a ip_forward(d7031400,0,0,50,e9037f00) at ip_forward+0x159 ipv4_input(d7031400,30,3,0) at ipv4_input+0x25a ipintr(d18a0058,10,e9030010,e9030010,e9036000) at ipintr+0x67 Bad frame pointer: 0xe9037f20 ps 8123 1 12702 0 3 0x284 select mrouted 17362 1 17362 0 3 0x284 kqread ifstated 16250 3631 3631 74 3 0x2000184 bpf pflogd 3631 1 3631 0 3 0x284 netio pflogd poll syslogd netio syslogd crypt_wa crypto aidoned aidoned syncer update cleaner cleaner reaper repaper pgdaemon pagedaemon pftm pfpurge usbevt usb3 usbevt usb2 usbevt usb1 usbtsk usbtsk usbevt usb0 kmalloc kmthread wait init scheduler swapper dmesg: OpenBSD 3.8-current (GENERIC.MP) #353: Tue Oct 18 05:16:18 MDT 2005 [EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC.MP cpu0: Intel(R) Xeon(TM) CPU 2.80GHz (GenuineIntel 686-class) 2.80 GHz cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,CNXT-ID real mem = 1073065984 (1047916K) avail mem = 972525568 (949732K) using 4278 buffers containing 53755904 bytes (52496K) of memory mainbus0 (root) bios0 at mainbus0: AT/286+(00) BIOS, date 01/19/05, BIOS32 rev. 0 @ 0xffe90 pcibios0 at bios0: rev 2.1 @ 0xf/0x1 pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfb460/256 (14 entries) pcibios0: PCI Interrupt Router at 000:31:0 (Intel 82801EB/ER LPC rev 0x00) pcibios0: PCI bus #10 is the last bus bios0: ROM list: 0xc/0xb000! 0xcb000/0x4000 0xec000/0x4000! ipmi0 at mainbus0: version 1.5 interface kcs ibase 0xca8/8 spacing 4 irq -1 mainbus0: Intel MP Specification (Version 1.4) (DELL PE 016C ) cpu0 at mainbus0: apid 0 (boot processor) cpu0: apic clock running at 199 MHz mainbus0: bus 0 is type PCI mainbus0: bus 1 is type PCI mainbus0: bus 2 is type PCI mainbus0: bus 3 is type PCI mainbus0: bus 4 is type PCI mainbus0: bus 5 is type PCI mainbus0: bus 6 is type PCI mainbus0: bus 7 is type PCI mainbus0: bus 8 is type PCI mainbus0: bus 9 is type PCI mainbus0: bus 10 is type PCI mainbus0: bus 11 is type ISA ioapic0 at mainbus0: apid 2 pa 0xfec0, version 20, 24 pins ioapic0: misconfigured as apic 0, remapped to apic 2 ioapic1 at mainbus0: apid 3 pa 0xfec8, version 20, 24 pins ioapic1: misconfigured as apic 0, remapped to apic 3 ioapic2 at mainbus0: apid 4 pa 0xfec83000, version 20, 24 pins ioapic2: misconfigured as apic 0, remapped to apic 4 pci0 at mainbus0 bus 0: configuration mode 1 (no bios) pchb0 at pci0 dev 0 function 0 Intel E7710 SMCH rev 0x09 ppb0 at pci0 dev 2 function 0 Intel E7710 MCH PCIE rev 0x09 pci1 at ppb0 bus 1 ppb1 at pci1 dev 0 function 0 Intel PCIE-PCIE rev 0x09 pci2 at ppb1 bus 2 ppb2 at pci2 dev 11 function 0 IBM PCIX-PCIX rev 0x02 pci3 at ppb2 bus 3 em0 at pci3 dev 4 function 0 Intel PRO/1000MT QP (82546EB) rev 0x01: apic 3 int 5 (irq 3), address 00:04:23:ba:9a:ac em1 at pci3 dev 4 function 1 Intel PRO/1000MT QP (82546EB) rev 0x01: apic 3 int 6 (irq 7), address 00:04:23:ba:9a:ad em2 at pci3 dev 6 function 0 Intel PRO/1000MT QP (82546EB) rev 0x01: apic 3 int 7 (irq 10), address 00:04:23:ba:9a:ae em3 at pci3 dev 6 function 1 Intel PRO/1000MT QP (82546EB) rev 0x01: apic 3 int 4 (irq 11), address 00:04:23:ba:9a:af ppb3 at pci1 dev 0 function 2 Intel
Re: carp + no ip address on iface (only master can receive acks)
On Thu, Nov 17, 2005 at 10:02:46PM +1100, Alex Strawman wrote:
Traffic shouldn't even be getting OUT on the backup in this situation.
i agree - there is no correct solution without using an ip addr for
each real interface.
would be nice to for example use an external ntp server to sync with,
but unless it uses another route (rather than ip-less carp'd
interface), it cannot (without dodgy work-arounds).
Assuming you are also using pfsync, that means you've got another
interface that directly connects both firewalls. Assign normal,
non-multicast addresses to those physical pfsync interfaces and ensure
that you can pass traffic between the two. Configure pf on both boxes
to NAT traffic out over its external carp'd IP address when it is coming
in on $pfsync_if from $pfsync_net.
This allows your carp backup to still have outbound net so things like
NTP, mail and external DNS lookups still work. Yes, there are ways you
could run these and other various services internally but there may be
reasons you cannot do this. In the end, having outbound connectivity
for an otherwise unreachable host is a good thing, IMO.
The catch here is that when the backup carp host is a backup, its
routing table must be aware that its route is no longer out over the
carp'd interface, but rather over your pfsync interface and the
receiving end is going to nat for you. There may be other ways to
accomplish this (ospf, perhaps), but I went with ifstated.
In my case, carp5/em5 on each box points upstream and each box has
a single external IP address assigned to it. The two boxes each have
the same two failover upstream gateways (not under my control -- a.b.c.d
and w.x.y.z). Yes, this setup is different than yours, but it should
give you enough to help you figure out the routing bit.
The config below is for the backup carp host. A similar one exists for
the master (all that is different is that the primary/secondary routes
are different and the pfsync IP to route to is different (.3 vs .2)).
Pretty? Depends who you ask. The right solution? Likely not, but it
got me out of a hole that I needed a way out of quickly, and it may
help you too.
-jon
#
init-state wan_master
wan_carp_up = carp5.link.up
wan_carp_init = carp5.link.unknown
wan_iface_up = em5.link.up
wan_primary_route_up = 'ping -q -c1 -w1 a.b.c.d 2 /dev/null 1 /dev/null
every 2'
wan_secondary_route_up = 'ping -q -c1 -w1 w.x.y.z 2 /dev/null 1 /dev/null
every 2'
state wan_master {
init {
run echo WAN master at `date` | mail -s 'WAN master change' \
[EMAIL PROTECTED]
}
# probably just came up. give things a chance to sync
if ($wan_carp_init)
run sleep 5
if (! $wan_carp_up)
set-state wan_slave
if ($wan_primary_route_up)
set-state primary_route
if (! $wan_primary_route_up) ($wan_secondary_route_up)
set-state secondary_route
if (! $wan_primary_route_up) (! $wan_secondary_route_up)
run echo THIS SHOULD NEVER HAPPEN!
}
state wan_slave {
init {
# simple. drop the default route and route over $SYNC_IF
run route change default 192.168.0.2
run echo WAN slave at `date` | mail -s 'WAN slave change' \
[EMAIL PROTECTED]
}
# if our link(s) come up, become the master
if ($wan_carp_up)
set-state wan_master
}
state primary_route {
init {
run route change default a.b.c.d
run echo Using primary route at `date` | mail -s 'Primary route change'
[EMAIL PROTECTED]
}
# if our link(s) go down, become the slave
if (! $wan_carp_up)
set-state wan_slave
if (! $wan_primary_route_up)
set-state secondary_route
}
state secondary_route {
init {
run route change default w.x.y.z
run echo Using secondary route at `date` | mail -s 'Secondary route
change' [EMAIL PROTECTED]
}
# if our link(s) go down, become the slave
if (! $wan_carp_up)
set-state wan_slave
if (! $wan_secondary_route_up)
set-state primary_route
}
Re: PF Tables Issue
On Tue, Nov 15, 2005 at 02:39:59PM -0800, Christian Petro wrote:
OpenBSD 3.6
/etc/pf.conf
When a table, and corresponding rule is defined using:
table LimitedAccess persist { 192.168.1.16, 192.168.1.17 }
block out quick on $ExtIf inet proto { tcp, udp } from LimitedAccess
to any port $OutIm
OR EVEN
block out quick on $ExtIf inet proto { icmp, udp, tcp } from
LimitedAccess to any
The result is both IP addresses are allowed to pass through the firewall.
Can anyone comment?
Yes.
There can be many reasons that either of your rules will result in those
two hosts being allowed through the firewall.
What is the rest of the pf.conf? Without that, I can only guess.
-jon
Re: PF Tables Issue
On Tue, Nov 15, 2005 at 04:59:50PM -0800, Christian Petro wrote:
What is the rest of the pf.conf? Without that, I can only guess.
-jon
set loginterface fxp1
set limit { states 9, frags 9 }
set optimization conservative
set block-policy drop
scrub in all
###
# DEFINE MACROS
###
# list of interfaces
LoIf=lo0
IntIf=fxp0
ExtIf=fxp1
OutTcp= some ports
OutIcmp= some types
OutUdp= some ports
OutIm= some ports
table AllAccess persist { some ips }
table LimitedAccess persist { 192.168.1.16, 192.168.1.17 }
table private persist { some ips }
table SitesAllowed persist { some sites }
##
# RULES IN/OUT for lo0, fxp0, fxp1
##
# default policy
block in log all
block out log all
# trusted interfaces
pass in quick on $LoIf all
pass out quick on $LoIf all
pass in quick on $IntIf all
pass out quick on $IntIf all
# anti-spoofing rool
block drop in quick on $ExtIf inet from private to any
# outbound traffic
pass out quick on $ExtIf inet proto icmp from AllAccess to any
icmp-type $OutIcmp keep state
pass out quick on $ExtIf inet proto udp from AllAccess to any port
$OutUdp keep state
pass out quick on $ExtIf inet proto tcp from AllAccess to any keep state
block out quick on $ExtIf inet proto { tcp, udp } from LimitedAccess
to any port $OutIm
pass out quick on $ExtIf inet proto tcp from LimitedAccess to
SitesAllowed port $OutWeb keep state
block out on $ExtIf inet proto { icmp, udp, tcp } from LimitedAccess
to any
#pass out on $ExtIf inet proto tcp from any to any port $OutTcp keep state
Your problem is likely your use of 'quick' on most of your rules,
specifically the section labeled trusted interfaces. Read pf.conf(5)
and search the archives
(http://marc.theaimsgroup.com/?l=openbsd-miscr=1w=2) for more
information about quick.
What is likely happening, assuming a normal routed setup, is those first
2 quick rules that pass in/out on $IntIf are allowing all traffic on
$IntIf in and out. Any traffic that comes in or out on $IntIf matches
that rule and that is the last rule that is evaluated, thanks to quick.
Your packets from LimitedAccess come in on $IntIf and are allowed.
Your state-policy is (by default) floating so the packets are in turn
allowed to pass out on $ExtIf. The bulk of your rules after the
4 trusted interfaces rules are likely rarely (if ever) evaluated.
I say likely because I am not 100% sure about your mixture of quick
and your first two default block in/out rules.
My suggestion? Get rid of quick, get a solid default block stance
in/out on all interfaces, and then selectively allow traffic. You are
close.
-jon
multicast routing problems with 3.8 and -current
Prior to the official 3.8 release I had been running a modified version of GENERIC that simply had MROUTING turned on. Everything worked fine -- the firewall would route multicast packets between interfaces. There were the occassional errors that I chalked up to the fact that perhaps MROUTING hadn't seen much use. I was happy to see within the past month or two that MROUTING was now on by default and can be tweaked via a sysctl, namely net.inet.ip.mforwarding. That error still exists and I'm not sure it is an error per-se, or rather a known condition that I'll run into when doing multicast routing. The error message is: Nov 14 23:07:06 fw-1 mrouted[3111]: warning - age_table_entry: SIOCGETSGCNT failing for (10.2.2.111 239.0.0.100): Invalid argument I get these messages every 5 minutes on the dot and only for the hosts in the 239.0.0.0/24 multicast group. My mrouted.conf is below: name C1 239.0.0.0/24 name E1 239.2.2.0/24 name C2 239.8.0.0/24 name R1 239.3.1.0/24 name M1 239.7.0.0/24 phyint em0 boundary C2 boundary E1 boundary M1 boundary R1 phyint em1 boundary C2 boundary E1 boundary M1 boundary R1 phyint em5 boundary C2 boundary E1 boundary M1 boundary R1 phyint em2 disable phyint em3 disable multicast_host is set to NO and multicast_router is set to YES and like I said multicast routing works. I don't believe this particular error message is a sign of something being broken but I am hunting down other bugs and wanted to eliminate this one in particular. So, that said... What causes this error and is it a condition that I can safely ignore? I've checked the archives but have not found anything. I found a few entries on NetBSD lists from the 90s and the remaining references where mrouted binaries that got indexed by google. Any insight would be much appreciated! -jon
Re: Slower http/s access with Pf enabled
On Mon, Nov 14, 2005 at 06:27:35PM -0700, Joe Barnett wrote:
The machine is running 3.8 from the CDs, GENERIC kernel, etc.
pf.conf follows (any critique of the rules and is welcome...):
#
# pf.conf -- Pf ruleset
#
# set up some variables
#
nic=rl0
spoofed={ 127.0.0.1/8, 172.16.0.0/12, 10.0.0.0/8, 255.255.255.255/32 }
local=192.168.0.0/16
#
# Scrub all packets by default
scrub in all
#
# Block all in by default, pass out all by default
#
block in all
pass out all
#
# Block spoofers (this might be redundant...)
#
block in quick on $nic from $spoofed
#
# open http/s
pass in quick proto tcp from $local to $nic port = 80
pass in quick proto tcp from any to $nic port = 443
#
# And SSH only, no ftp or telnet
#
pass in quick proto tcp from any to $nic port = 22
#
# MySQL
pass in quick proto tcp from $local to $nic port = 3306
#
# SMB shares
pass in quick proto tcp from $local to $nic port = 139
pass in quick proto tcp from $local to $nic port = 445
#
# Allow loopback traffic
#
pass in quick on lo0 all
#
# Allow local machines to ping
#
pass in quick proto icmp from $local to $nic
#
# Allow out all TCP, UDP, and ICMP traffic keep state on it
# so that it's allowed back in.
#
# tcp
pass out quick proto tcp from $nic to any keep state
# udp
pass out quick proto udp from $nic to any keep state
# icmp
pass out quick inet proto icmp from $nic to any keep state
#
# Block office generated SMB broadcast traffic without logging (very
noisy) (this might also be redundant...)
#
block in quick proto udp from any to $nic port = 137
block in quick proto udp from any to $nic port = 138
#
# just to be safe, end by blocking anything that is left
#
block in all
I can't recall a time on this or the pf@benzedrine.cx list where running
with the pass quick first, then block all at the end mindset was
worthwhile. If you've got a real good reason to be doing it, fine.
Otherwise, I suggest avoiding it. You are doing a little of both --
You've got a default drop policy inbound yet a default allow outbound,
and you tacked on an extra default block in at the end for good measure.
I'm not sure what your intention was.
My suggestions:
1) Turn on logging so you can see what (if anything) is going wrong
2) For TCP connections, only allow in packets that are part of a valid
connection. That is, use flags S/SA to match the initial syn and
follow #3 to match the rest.
3) Use the various state options (keep, synproxy, modulate) everywhere
unless you've got a good reason not too. Especially for those
incoming 80/443 connections.
4) Using $nic will only work when you are referring to an interface.
That means rules like pass out quick inet proto icmp from $nic blah
blah won't work. They'll actually try to lookup rl0 which you
definitely don't want. Use the proper address or address block
there.
5) Consider set skip on lo0. You've got a rule that allows traffic in
on lo0, but never out.
6) See #1
I have another pf.conf I use for testing, which allows all packets
in and out, only scrubbing them, and performance is significantly
better with this:
When you say significantly, do you mean you are back to acceptable
speeds?
-jon
Re: multicast routing problems with 3.8 and -current
On Mon, Nov 14, 2005 at 10:38:21PM -0500, Mathieu Sauve-Frankel wrote: On Mon, Nov 14, 2005 at 07:23:35PM -0500, Jon Hart wrote: Prior to the official 3.8 release I had been running a modified version There is no net.inet.ip.mforwarding in 3.8-release, only in -current. If you are running stock 3.8 you will still need to build a kernel with option MROUTING. If by any chance you are running -current, then you still will need to enable multicast routing by issuing sysctl net.inet.ip.mforwarding=1 Yes, sorry for the confusion. Before 3.8 came out I was running a custom kernel with MROUTING enabled. Since 3.8 came out, I've been running -current which has MROUTING on. So really, we can just ignore the 3.8 point and say I've run both a custom kernel and -current with the same results. MROUTING and net.inet.ip.mforwarding are both on and like I said, multicast routing *does* work. Its just that every 5 minutes or so when some timer goes off I get those error messages. -jon
Re: Strange behavior with carp and preemption
On Thu, Nov 10, 2005 at 09:31:15PM -0500, Nick Holland wrote: I'd have prefered that a more experienced person answer this one, but they don't seem to have, so be forewarned: everything I say here might be wrong. However, through the glory of mail lists, if I say something wrong, fifty people will jump all over me, and Google will put it at the top of the list when people google for my name. :) Consider it done! I set up two OpenBSD 3.7 -stable firewalls using carp. Everything works except preemption. When only one interface on the master side fails (pull the Cable) the regarding carp0 interface on the backup side becomes master. But not carp1. Right. Nothing's wrong with the master carp1, why should it demote itself and have the backup take over? Because that is what preemption is supposed to do. When one interface on the carp master goes into BACKUP state (or is it any state that is not MASTER?), the others should become BACKUPs too. My experience is *sometimes* this is not instantaneous. At a minimum, the advskew should change and they should become BACKUPs in short order. I waited some minutes, but carp1 keeps being backup until I do a simple ifconfig(8) on the master side. Then it changes immediately. yep. (though I'm not entirely sure I know what command you are typing by simple ifconfig(8).) I can reproduce it, waiting some minutes, or only a fiew seconds. Once I do an ifconfig on the master side, the backup side becomes master on all carp's. Strange...? not really, if you understand the modular approach here. My config: ... Can anybody reproduce it, and has a solution for this problem? Any help would be very nice! :-) Look at the pieces here: * CARP gives you redunancy on your INTERFACES...not your entire firewall. * pfsync keeps your firewall state tables in sync, so either machine can take over. If you lose a box completely, your system is fine. If you lose one cable or one NIC or so on, you have a problem. That is definitely not true. Preemption is the answer here. If one carp interface fails, they all fail. Without preemption you either have a really good reason to be not using it or have a way to deal with such a situation. Imagine the typical situation: $wan_if, $lan_if, and $sync_if. Your run of the mill two legged failover setup. With preemption, if one or more of $wan_if/$lan_if fails, all other carp interfaces fail. Without preemption, if $wan_if fails, $lan_if is still the master and you've got a situation on your hands -- if all of $lan_if:network is using the current LAN master as their gateway, how is that host going to get out? Unless you play some tricks with ospf, bgpd or heck, even ifstated like I've done in the past, routing will fail. This is why preemption is a good choice in many cases. What you need is something that will watch all interfaces and shut down ALL (forcing a COMPLETE fail-over) if something goes wrong with any. That's a third part of the CARP toolset: ifstated(8) and ifstated(5). Yes, that's missing from the PF FAQ, though I just tossed a couple links in faq/pf/carp.html. More will get added when I get more knowledge of the topic (or Joel writes it :) Yes, you *can* do this with ifstated, but I'm not sure how recommended it is. I think the stock example that comes with ifstated is going down this path, but I'm not 100% sure. My suggestion would be to see that the advskew changes on the other carp interfaces when carp0 becomes a backup. If they do, that means preemption is definitely turned on and should work. -jon
Re: carp incorrect hash debugging
On Fri, Nov 04, 2005 at 02:57:35AM +, Ryan McBride wrote: On Thu, Nov 03, 2005 at 06:11:20PM -0500, Jon Hart wrote: 1) used to determine that a particular carp packet is intended for you carp host? carp(4) does a number of validity checks before treating the packet a real carp packet: - was the device recieved on a interface that has a carp device on it? - is the ttl 255 (prevents routed carp packets from being accepted) - packet length - crc32 checksum - VHID - Is the carp interface UP and RUNNING? - version - SHA-1 HMAC 2) given that a carp host knows that a particular carp packet is one that it cares about, how does it verify that all of the parameters contained within are legit? It checks the HMAC, which contains the password, version, counter, type, and the addresses. [snip] Great, thank you! If the answer to all this is to just ensure that if I ever have more than one carp pair on the same network to ensure that I have different vhids, Yes, you MUST use a different vhid for different carp clusters on the same link-local network; the MAC address for the carp interface is generated from the vhid, and if you don't keep this unique your switch will likely get confused. Great, this confirms what I learned. I also checked out the pf/carp faq and the description of vhid there adequately describes the purpose of a vhid and makes it fairly clear that different carp clusters on the same network must use different vhids. The carp manpage eludes to this but could perhaps be made more clear. Something along the lines of: To use carp, the administrator needs to configure at minimum a common virtual host ID (vhid) and virtual host IP address on each machine which is to take part in the virtual group. The vhid is used to uniquely identify all members of a virtual group and hosts on the same link-local network must different vhids. does anyone have a vhid numbering scheme that they've found workable? I had been using interface number +1 (so the carp for em0 would be vhid 1, etc). In many situations, I use the last octet of the first virtual IP address. (If your virtual IP is 192.168.0.23, use 23 as your vhid) Genius! Thanks again, -jon
carp incorrect hash debugging
Greetings, We've all probably had or seen the carp error similar to: carp0: incorrect hash In most cases that I've seen on this and other lists it was because of something obvious like a mismatched pass or two supposed carp partners using different vhid's. I've taken a look at the code but wanted to verify. What pieces of information are: 1) used to determine that a particular carp packet is intended for you carp host? 2) given that a carp host knows that a particular carp packet is one that it cares about, how does it verify that all of the parameters contained within are legit? I believe the answer to 1 is the version, type and vhid from the carp packet. 2 I'm not so sure about, but I'm assuming that at least part of this decision is based on the pass. I had a situation earlier today that I could not explain. Put simply, I had hosts A, B, C and D all on the same /24. Hosts A and B where a carp pair for 192.168.0.1 and hosts C and D were a carp pair for 192.168.0.4. If A and B were using the same vhid as C and D, both ends would complain about an incorrect hash. Having never been in that situation before, I figured the vhid's were clashing since the pass happened to be the same on all 4 machines. I destroyed carp0 and did a 'sh /etc/netstart carp0'. I was still getting the messages but they seemed less frequent. I worked on other things which required a reboot and from then on, the messages were gone. The two carp pairs have functioned as expected ever since. Was my fix (prior to rebooting) the correct one? If so, why did I continue to get the incorrect hash messages? Gremlins or operator error? If the answer to all this is to just ensure that if I ever have more than one carp pair on the same network to ensure that I have different vhids, does anyone have a vhid numbering scheme that they've found workable? I had been using interface number +1 (so the carp for em0 would be vhid 1, etc). Any input would be much appreciated! -jon
Re: Carp scp loosing connection
On Mon, Oct 24, 2005 at 10:48:03AM -0400, Monah Baki wrote: Solved it, had to switch pass in quick on $int_if all pass out quick on $int_if all to pass in quick on $int_if all keep state pass out quick on $int_if all keep state Is there any particular reason you are using 'quick' on most of your rules? There are certain situations that quick is needed or recommended, but I'm of the school that using quick on all of your rules just leads to unnecessary confusion. Also, I'm not too sure what your intention was surrounding the ordering of your rules. The most common way is to put all your 'default block' rules at the top of your ruleset and all the specific allow rules following those. When you've got default block rules peppered throughout your ruleset, it'll quickly become fault prone and difficult to manage. IMO, of course. There was a thread some time ago that (I believe) discussed using 'quick' in large/complicated rulesets to speed up processing. I'm not 100% sure what the consensus was, but I think what part of it boiled down to was that the benefits that you gain by using quick are far outweighed by those of having a tight and easy to manage ruleset. http://marc.theaimsgroup.com/?l=openbsd-pfm=111522051104764w=2 -jon
Re: em(4) problems with -current
On Wed, Oct 19, 2005 at 12:56:44PM -0400, Jon Hart wrote: On Wed, Oct 19, 2005 at 12:10:35PM -0400, Brian A. Seklecki wrote: The Intel IPMI on the motherboard may be to blame. It's always up/on and listening. Also, see my thread in freebsd-questions@ about Dells with Intel em(4) and Dell PowerEdge switches w/ NIC Teaming, 802.3ad, ng_many2_one, etc. For example, traffic sent from the IPMI IP/MAC of the interface is visible from the OS via tcpdump(8), which is kind of spooky. This was something I had thought of and I believe I disabled all traces of it. Console redirection, BMC/IPMI, etc, all disabled. Perhaps disabled simply means don't accept connections to IPMI but keep the link up. This does appear to be the case. BMC is disabled and the card still exhibits the same behavior. I'll double check this today and verify. Will the IPMI on the motherboard only work with the onboard ethernet controllers, or will it get its grubby little hands on any/all controllers it finds? If it only works with the onboard, then maybe switching to the PCI card ports will be a sufficient workaround.Z Testing these machines shows that only the primary onboard controller acts dumb. My secondary onboard controller (em5, in this case) works as expected. The only question that remains is... is the fact that 'ifconfig em4 down' put the interface into a DOWN state but keeps the link up a bug? I guess em did its best to down the interface, but IPMI brought it back up. I wonder if its worthwhile for the driver to detect this? Anyway, thanks for your insight! -jon
em(4) problems with -current
I've got a snapshot from October 6, 2005 running on a Dell PE 1850. Nothing overly special. 3.2Ghz Xeon, 2G RAM, dual onboard Intel PRO/1000MT, Intel PRO/1000QP in the 64-bit/133mhz PCI-X slot, and a 36G U320/15K RPM SCSI disk. dmesg at the end of the email. The most relevant bits from the dmesg are as follows: em0 at pci3 dev 4 function 0 Intel PRO/1000MT QP (82546EB) rev 0x01: em1 at pci3 dev 4 function 1 Intel PRO/1000MT QP (82546EB) rev 0x01: em2 at pci3 dev 6 function 0 Intel PRO/1000MT QP (82546EB) rev 0x01: em3 at pci3 dev 6 function 1 Intel PRO/1000MT QP (82546EB) rev 0x01: em4 at pci7 dev 7 function 0 Intel PRO/1000MT (82541GI) rev 0x05: em5 at pci8 dev 8 function 0 Intel PRO/1000MT (82541GI) rev 0x05: On em4 and em5, if I 'ifconfig em4 down' the interface looks like it is down to the OS -- tcpdump shows no packets coming in or going out. As expected. However, the link light is still on and any device connected to em4 (a SMC switch, in this case), sees the interface as UP. I have not done the same test with em0-3 (different chipset. see above), but I suspect I'll see the same problem. I'll be downloading a snapshot overnight while I sleep in hopes that the changes checked in on Oct 7 and beyond from FreeBSD's em driver help, but I was curious if anyone else has seen or can explain this behavior of the em driver or others for that matter. In attempting to debug this, I was told that the xl driver seems to work as expected. 'ifconfig xl0 down' drops the link. Any input would be much appreciated. Thanks! -jon dmesg: OpenBSD 3.8-current (GENERIC) #179: Thu Oct 6 11:32:36 MDT 2005 [EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC cpu0: Intel(R) Xeon(TM) CPU 3.20GHz (GenuineIntel 686-class) 3.20 GHz cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,CNXT-ID real mem = 2146807808 (2096492K) avail mem = 1952960512 (1907188K) using 4278 buffers containing 107442176 bytes (104924K) of memory mainbus0 (root) bios0 at mainbus0: AT/286+(00) BIOS, date 01/19/05, BIOS32 rev. 0 @ 0xffe90 pcibios0 at bios0: rev 2.1 @ 0xf/0x1 pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfb460/256 (14 entries) pcibios0: PCI Interrupt Router at 000:31:0 (Intel 82801EB/ER LPC rev 0x00) pcibios0: PCI bus #10 is the last bus bios0: ROM list: 0xc/0xb000! 0xcb000/0x1000 0xcc000/0x4000 0xd/0x1000 0xec000/0x4000! ipmi0 at mainbus0: version 1.5 interface kcs ibase 0xca8/8 spacing 4 irq -1 cpu0 at mainbus0 pci0 at mainbus0 bus 0: configuration mode 1 (no bios) pchb0 at pci0 dev 0 function 0 Intel E7710 SMCH rev 0x09 ppb0 at pci0 dev 2 function 0 Intel E7710 MCH PCIE rev 0x09 pci1 at ppb0 bus 1 ppb1 at pci1 dev 0 function 0 Intel PCIE-PCIE rev 0x09 pci2 at ppb1 bus 2 ppb2 at pci2 dev 11 function 0 IBM PCIX-PCIX rev 0x02 pci3 at ppb2 bus 3 em0 at pci3 dev 4 function 0 Intel PRO/1000MT QP (82546EB) rev 0x01: irq 3, address: 00:04:23:ba:94:88 em1 at pci3 dev 4 function 1 Intel PRO/1000MT QP (82546EB) rev 0x01: irq 7, address: 00:04:23:ba:94:89 em2 at pci3 dev 6 function 0 Intel PRO/1000MT QP (82546EB) rev 0x01: irq 10, address: 00:04:23:ba:94:8a em3 at pci3 dev 6 function 1 Intel PRO/1000MT QP (82546EB) rev 0x01: irq 11, address: 00:04:23:ba:94:8b ppb3 at pci1 dev 0 function 2 Intel PCIE-PCIE rev 0x09 pci4 at ppb3 bus 4 mpt0 at pci4 dev 5 function 0 Symbios Logic 53c1030 rev 0x08: irq 7 mpt0: sending FW Upload request to IOC (size: 36, img size: 40048) mpt0: IM support: 0 scsibus0 at mpt0: 16 targets sd0 at scsibus0 targ 0 lun 0: SEAGATE, ST336754LC, D402 SCSI3 0/direct fixed sd0: 34732MB, 50824 cyl, 2 head, 699 sec, 512 bytes/sec, 71132959 sec total sd1 at scsibus0 targ 6 lun 0: , , SCSI0 0/direct fixed sd1(mpt0:6:0): could not get size sd1: drive offline mpt0: target 0 Synchronous at 160MHz width 16bit offset 63 QAS 0 DT 1 IU 1 mpt0: target 6 Asynchronous at 0MHz width 8bit offset 0 QAS 0 DT 0 IU 0 ppb4 at pci0 dev 4 function 0 Intel E7710 MCH PCIE rev 0x09 pci5 at ppb4 bus 5 ppb5 at pci0 dev 5 function 0 Intel E7710 MCH PCIE rev 0x09 pci6 at ppb5 bus 6 ppb6 at pci6 dev 0 function 0 Intel PCIE-PCIE rev 0x09 pci7 at ppb6 bus 7 em4 at pci7 dev 7 function 0 Intel PRO/1000MT (82541GI) rev 0x05: irq 11, address: 00:14:22:16:3b:c8 ppb7 at pci6 dev 0 function 2 Intel PCIE-PCIE rev 0x09 pci8 at ppb7 bus 8 em5 at pci8 dev 8 function 0 Intel PRO/1000MT (82541GI) rev 0x05: irq 3, address: 00:14:22:16:3b:c9 ppb8 at pci0 dev 6 function 0 Intel E7710 MCH PCIE rev 0x09 pci9 at ppb8 bus 9 uhci0 at pci0 dev 29 function 0 Intel 82801EB/ER USB rev 0x02: irq 11 usb0 at uhci0: USB revision 1.0 uhub0 at usb0 uhub0: Intel UHCI root hub, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered uhci1 at pci0 dev 29 function 1 Intel 82801EB/ER USB rev 0x02: irq 10 usb1 at uhci1: USB revision 1.0 uhub1 at usb1 uhub1: Intel UHCI root hub, rev 1.00/1.00, addr 1 uhub1: 2 ports with 2 removable, self powered uhci2 at pci0 dev
Re: Stupid Carp question
On Thu, Aug 04, 2005 at 08:28:49AM -0400, Monah Baki wrote:
Hi all,
Implementing carp, I have 2 net4801's that seem to be synchronizing, when I do
a ifconfig -a on the secondary I see carp0 on the slave becomes Master when
the primary goes down.
The internal machines are working fine accessing the internet and all.
The pf.conf rule has the 2 rules:
pass quick on { sis2 } proto pfsync
pass on { sis0 sis1 } proto carp keep state
However when I physiclly remove the ethernet cable from sis0 on the master,
the internal machine cannot access the net anymore.
Do I need to copy the pf.conf from the master to the scondary unit, have them
both identical
The way I understand it (someone correct me if I'm wrong), is that if
the slave has no ruleset (or a non identical ruleset), when the master
goes down, the slave will have all the states that the master did, but
packets will not pass unless there are rules that explicitly allow them.
-jon
Re: Dell PowerEdge 750 SATA
On Mon, Jul 11, 2005 at 06:54:28AM +0200, Matteo Mancini wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi *, I've to build an high performance openbsd firewall with ids included. I think to use the server in subject, does any one have tried it?? For a price that is in the same ballpark, you should go with the 1850s. They've got faster PCI buses that'll be a big help: PE750: 1x 64-bit/66Mhz PCI-X 1x 32-bit/33Mhz PCI PE1850 1x 64-bit/133Mhz PCI-X 1x 64-bit/100Mhz PCI-X Especially if you are slapping dual or quad port cards in there, the faster PCI will be a big help. -jon
Re: Need Quad Ethernet for router box
On Thu, Jul 21, 2005 at 03:19:48PM -0400, Brad wrote: Note, there are cards that are supported that are not listed in the man page. It's hard to have an exact list when there are so many cards out there and sometimes even different revisions with the same name and different chipsets. The chipset revision is what really matters. I've been looking at the SK-9S22 and SK-9E22 for a project but will pass on the 9S22 if its not supported. I was trying to get prices and found this: http://www.zones.com/cgi-bin/zones/site/product/index.html?id=000194725 Can't find the 9822 on the syskonnect site, but the 98xx series look to have automatic failover in the event of link failure. Anyone used these (in OpenBSD or otherwise) and have news to share? The syskonnect online store seems to be down. Does anyone know of a syskonnect reseller that they've had a good experience with? Thanks, -jon
Re: Need Quad Ethernet for router box
On Fri, Jul 22, 2005 at 04:06:33PM +0200, Henning Brauer wrote: can sombody just GET one and stuff it in a machine? chances are good supporting them is as easy as adding the IDs to pcidevs. I tried contacting syskonnect about an evaluation unit which they mention on their site but the mail bounced. The US reseller I'm dealing with (incat) has what looks like great prices (SK-9S22 for $144, SK-9E22 for $154), so I'll get one and get the PCI IDs. If the existing sk driver works (*crosses fingers*), I'll order the rest through this same reseller. -jon
Re: Make OpenBSD 3.7 bootable ISO image
On Sat, Jun 11, 2005 at 10:13:10PM +1000, Z L wrote: I downloaded /3.7/i386 and want to create a bootable CD. So first I need to make a bootable ISO image and then burn the image in a CD. I did mkisofs -r -b ~/openbsd/3.7/i386/cdrom37.fs -c boot.catalog -o openbsd.iso. It doesn't seem to work. Along with ~/../3/7 I also have ~/openbsd/openssh and ~/openbsd/3.7/changelog directory. I was wondering if I could burn them altogether in a bootable CD. Any help would be appreciated. I've been using mkisofs for quite some time to make bootable CDs of snapshots. First I make a directory to hold the CD layout, lets call it OpenBSD3.7-snap060705. Then create the dirs needed for the layout. `mkdir OpenBSD3.7-snap060705/3.7/i386/` Within that directory, download all the necessary files -- the tgzs that you want/need, cdrom37.fs, bsd.*, CKSUM, MD5, INSTALL.*, index.txt, etc. Now, from the parent directory of OpenBSD3.7-snap060705, run something like the following: `mkisofs -vrTJV OpenBSD3.7-snap060705 -b 3.7/i386/cdrom37.fs -c boot.cat -o /tmp/OpenBSD3.7-snap060705.iso OpenBSD3.7-snap060705` The important thing to note is that the argument to -b is a relative path within the last argument. -jon
