Re: OpenBSD fw freezing with ps/trace.
On Tue, Oct 18, 2011 at 09:58:39AM +, Stuart Henderson wrote: > On 2011-10-18, Leon Me?ner wrote: > > On Mon, Oct 17, 2011 at 08:43:50PM +, Stuart Henderson wrote: > >> This is what a BREAK on a serial console looks like. > >> > > On Mon, Oct 17, 2011 at 07:02:07PM +0200, Leon Me_ner wrote: > > Thats the way i got the ddb output. What's the interesting output then > > in this case? As suggested i'll try to get another machine going and > > test with that. This one did unfreeze after an undetermined amount of > > time after i unplugged all network devices in the internal 192.x lan > > segment. > > Depending on the cause, besides ps/trace, these are some of the more > useful things you can type: > > sh reg > sh malloc > sh all pools > sh uvmexp > > Not sure what else to suggest from ddb, with this type of problem > it's sadly really difficult to get useful information. If you can't > take diagnosis further, giving as much information as possible about > the setup and what you're doing at the time in the hope that somebody > has the time/ability to reproduce and debug it is often all you can do. Yes I see. I will try to minimize the setup and see if i can create some nice description of the problem. > On 2011-10-17, Leon Me?ner wrote: > > we are running a backup firewall machine which regularly freezes since > > OpenBSD 4.6. The configuration also changed at this time. When frozen no > > input is accepted by serial or keyboard console. Breaking to ddb works > > though. The output of ps and trace are below. The machine is primarily > > working as a transparent firewalling bridge but also runs NAT, pf and > > dhcpd for a 192.168.x/24. The freeze can often be provoked by obtaining > > an IP in the 192.168.x/24 and immediately sshing from this network into > > a Host on the bridged network part. > > Is the natted network, 192.168.x, also involved in the bridge? If so, and > if that can be changed, that might be worth investigating. I think many of > us are generally trying to avoid bridges, so it's not going to be the best- > tested part of the network stack... The natted network has a dedicated NIC on the firewall (incoming). The IP that the network gets natted to is an alias on the management NIC (outgoing). The bridge is between two seperate NIC's that are not used for anything else and one of these NIC's is disabled by spanning-tree on the switch side. Thanks with all the help, Leon
Re: OpenBSD fw freezing with ps/trace.
On Mon, Oct 17, 2011 at 08:43:50PM +, Stuart Henderson wrote: > This is what a BREAK on a serial console looks like. > > > On 2011-10-17, Leon Me?ner wrote: > > On Mon, Oct 17, 2011 at 07:02:07PM +0200, Leon Me_ner wrote: > >> On Mon, Oct 17, 2011 at 09:37:37AM -0700, Bryan Irvine wrote: > >> > On Mon, Oct 17, 2011 at 8:20 AM, Chris Cappuccio > >> > wrote: > >> > > Time to upgrade to 5.0. Report any failures after you do that. > >> > > >> > I think he's saying it's been doing this since 4.6. I parsed that as > >> > him being on at least the current release. > >> > > >> > Leon, can you send a dmesg? > >> > >> > >> The machine is just beeing updated to a 5.0 snapshot. I had this dmesg > >> still in my scrollback buffer which i took when i was doing the trace > >> and ps. > >> Sorry for the truncated lines. > > > > > > > > Upgrading to 5.0 changed nothing. After dhcping and invoking ssh the > > machine froze. Trace of this freeze is below. Actually i forgot to > > mention that sometimes the machine manages to unfreeze again after some > > minutes. Thats the way i got the ddb output. What's the interesting output then in this case? As suggested i'll try to get another machine going and test with that. This one did unfreeze after an undetermined amount of time after i unplugged all network devices in the internal 192.x lan segment. My resolv.conf is fine i think (attached). Thanks, Leon # cat /etc/resolv.conf domain physik-pool.tu-berlin.de search physik-pool.tu-berlin.de physik.tu-berlin.de # emmi.physik-pool.tu-berlin.de nameserver 130.149.58.146 # ns.tu-berlin.de nameserver 130.149.7.7 # ws-ber1.win-ip.dfn.de nameserver 193.174.75.142 options timeout:2 lookup file bind
Re: OpenBSD fw freezing with ps/trace.
On Mon, Oct 17, 2011 at 07:02:07PM +0200, Leon Me_ner wrote:
> On Mon, Oct 17, 2011 at 09:37:37AM -0700, Bryan Irvine wrote:
> > On Mon, Oct 17, 2011 at 8:20 AM, Chris Cappuccio wrote:
> > > Time to upgrade to 5.0. Report any failures after you do that.
> >
> > I think he's saying it's been doing this since 4.6. I parsed that as
> > him being on at least the current release.
> >
> > Leon, can you send a dmesg?
>
>
> The machine is just beeing updated to a 5.0 snapshot. I had this dmesg
> still in my scrollback buffer which i took when i was doing the trace
> and ps.
> Sorry for the truncated lines.
Upgrading to 5.0 changed nothing. After dhcping and invoking ssh the
machine froze. Trace of this freeze is below. Actually i forgot to
mention that sometimes the machine manages to unfreeze again after some
minutes.
ddb{0}> trace
Debugger() at Debugger+0x5
comintr() at comintr+0x268
Xintr_ioapic_edge4() at Xintr_ioapic_edge4+0xe8
--- interrupt ---
Bad frame pointer: 0x8000213dfcf0
end trace frame: 0x8000213dfcf0, count: -3
ip_output+0x5ee:
Re: OpenBSD fw freezing with ps/trace.
On Mon, Oct 17, 2011 at 09:37:37AM -0700, Bryan Irvine wrote:
> On Mon, Oct 17, 2011 at 8:20 AM, Chris Cappuccio wrote:
> > Time to upgrade to 5.0. Report any failures after you do that.
>
> I think he's saying it's been doing this since 4.6. I parsed that as
> him being on at least the current release.
>
> Leon, can you send a dmesg?
The machine is just beeing updated to a 5.0 snapshot. I had this dmesg
still in my scrollback buffer which i took when i was doing the trace
and ps.
Sorry for the truncated lines.
ddb{0}> dmesg
OpenBSD 4.9 (GENERIC.MP) #819: Wed Mar 2 06:57:49 MST 2011
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 2146172928 (2046MB)
avail mem = 2075017216 (1978MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xfc5b0 (58 entries)
bios0: vendor American Megatrends Inc. version "080014" date 10/22/2009
bios0: Supermicro H8DM8-2
acpi0 at bios0: rev 0
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP APIC MCFG OEMB SRAT
acpi0: wakeup devices PS2K(S4) PS2M(S4) NSMB(S4) USB0(S4) USB2(S1) NMAC(S5) NMA
D(S5) P0P1(S4) HDAC(S4) BR10(S4) BR11(S4) BR12(S4) BR15(S4) SLPB(S4) PWRB(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Dual-Core AMD Opteron(tm) Processor 2214 HE, 2211.65 MHz
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,C
FLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 1MB 64b/line 16
-way L2 cache
cpu0: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu0: DTLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu0: apic clock running at 201MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Dual-Core AMD Opteron(tm) Processor 2214 HE, 2211.33 MHz
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,C
FLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW
cpu1: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 1MB 64b/line 16
-way L2 cache
cpu1: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu1: DTLB 32 4KB entries fully associative, 8 4MB entries fully associative
ioapic0 at mainbus0: apid 2 pa 0xfec0, version 11, 24 pins
acpimcfg0 at acpi0 addr 0xe000, bus 0-255
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (P0P1)
acpiprt2 at acpi0: bus 7 (BR10)
acpiprt3 at acpi0: bus 6 (BR11)
acpiprt4 at acpi0: bus 5 (BR12)
acpiprt5 at acpi0: bus 3 (BR30)
acpiprt6 at acpi0: bus 4 (BR31)
acpicpu0 at acpi0
acpicpu1 at acpi0
acpibtn0 at acpi0: SLPB
acpibtn1 at acpi0: PWRB
ipmi at mainbus0 not configured
pci0 at mainbus0 bus 0
"NVIDIA MCP55 Memory" rev 0xa2 at pci0 dev 0 function 0 not configured
pcib0 at pci0 dev 1 function 0 "NVIDIA MCP55 ISA" rev 0xa3
nviic0 at pci0 dev 1 function 1 "NVIDIA MCP55 SMBus" rev 0xa3
iic0 at nviic0
iic1 at nviic0
wbng0 at iic1 addr 0x2c: w83793g
lm1 at iic1 addr 0x2d: W83627HF
iic1: addr 0x4c 00=80 03=48 07=48 0a=43 0f=48 11=00 12=43 17=48 1a=43 1f=48 21=
00 22=43 27=48 29=00 2a=43 2f=48 31=00 32=43 37=48 39=00 3a=43 3f=48 41=00 42=4
3 48=80 49=00 4a=43 50=80 51=00 52=43 58=80 5a=43 5f=48 61=00 62=43 67=48 69=00
6a=43 6f=48 71=00 72=43 77=48 79=00 7a=43 7f=48 81=00 82=43 87=48 89=00 8a=43 8
f=48 91=00 92=43 97=48 99=00 9a=43 9f=48 a1=00 a2=43 a7=48 aa=43 b0=80 b1=00 b2
=43 b8=80 b9=00 ba=43 c0=80 c1=00 c2=43 c8=80 c9=00 ca=43 d0=80 d2=43 d8=80 d9=
00 da=43 e0=80 e1=00 e2=43 e8=80 e9=00 ea=43 f0=80 f1=00 f2=43 f8=80 f9=00 fa=4
3 words 00=8000 01=00ff 02= 03= 04= 05= 06= 07=4800
ohci0 at pci0 dev 2 function 0 "NVIDIA MCP55 USB" rev 0xa1: apic 2 int 7 (irq 7
), version 1.0, legacy support
ehci0 at pci0 dev 2 function 1 "NVIDIA MCP55 USB" rev 0xa2: apic 2 int 10 (irq 1
0)
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "NVIDIA EHCI root hub" rev 2.00/1.00 addr 1
pciide0 at pci0 dev 4 function 0 "NVIDIA MCP55 IDE" rev 0xa1: DMA, channel 0 co
nfigured to compatibility, channel 1 configured to compatibility
pciide0: channel 0 disabled (no drives)
pciide0: channel 1 ignored (disabled)
pciide1 at pci0 dev 5 function 0 "NVIDIA MCP55 SATA" rev 0xa3: DMA
pciide1: using apic 2 int 11 (irq 11) for native-PCI interrupt
atapiscsi0 at pciide1 channel 0 drive 0
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: ATAPI 5/cdrom r
emovable
cd0(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 5
pciide2 at pci0 dev 5 function 1 "NVIDIA MCP55 SATA" rev 0xa3: DMA
pciide2: using apic 2 int 5 (irq 5) for native-PCI interrupt
pciide3 at pci0 dev 5 function 2 "NVIDIA MCP55 SATA" rev 0xa3: DMA
pciide3: using apic 2 int 10 (irq 10) for native-PCI interrupt
ppb0 at pci0 dev 6 function 0 "NVIDIA MCP55 PCI-PCI" rev 0xa2
pci1 at ppb0 bus 1
vga1 at pci1 dev 5 function 0 "ATI ES1000" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80
Re: OpenBSD fw freezing with ps/trace.
On Mon, Oct 17, 2011 at 08:20:31AM -0700, Chris Cappuccio wrote:
> Time to upgrade to 5.0. Report any failures after you do that.
The CD set did not yet arrive at my local retailer. So i probably have
to wait till 1st of Nov. ;(
> Leon Me?ner [l.mess...@physik.tu-berlin.de] wrote:
> > Hi all,
> >
> > we are running a backup firewall machine which regularly freezes since
> > OpenBSD 4.6. The configuration also changed at this time. When frozen no
> > input is accepted by serial or keyboard console. Breaking to ddb works
> > though. The output of ps and trace are below. The machine is primarily
> > working as a transparent firewalling bridge but also runs NAT, pf and
> > dhcpd for a 192.168.x/24. The freeze can often be provoked by obtaining
> > an IP in the 192.168.x/24 and immediately sshing from this network into
> > a Host on the bridged network part.
> > The machine is currently still sitting there, so if additional info is
> > needed please ask.
> >
> > thanks,
> > Leon
> >
> > Stopped at Debugger+0x5: leave
> > ddb{0}> ps
> >PID PPID PGRPUID S FLAGS WAIT COMMAND
> > 8693 1 1 0 3 0x2004080 ttyopngetty
> > 1066 28162 1066 0 3 0x2004080 ttyin zsh
> > 28162 16328 28162 1001 3 0x2004080 pause zsh
> > 16328 27349 16328 1000 3 0x2004080 pause ksh
> > 27349 15191 15191 1000 3 0x2000180 selectsshd
> > 15191 19218 15191 0 3 0x2004180 netio sshd
> > 27916 1 27916 0 3 0x2040180 selectsendmail
> > 18379 1 18379 0 2 0x2004080getty
> > 22394 1 22394 0 3 0x2004080 ttyin getty
> > 24536 1 24536 0 3 0x2004080 ttyin getty
> > 17228 1 17228 0 3 0x2004080 ttyin getty
> > 23114 1 23114 0 3 0x2004080 ttyin getty
> > 14383 1 14383 0 3 0x2004080 ttyin getty
> > 16906 1 16906 0 3 0x280 selectcron
> > 24123 1 24123 0 3 0x2000180 selectinetd
> > 9001 1 9001 77 2 0x2000180dhcpd
> > 19218 1 19218 0 7 0x280sshd
> > 32707 31588 26440 83 3 0x2000180 poll ntpd
> > 31588 26440 26440 83 3 0x2000180 poll ntpd
> > 26440 1 26440 0 3 0x280 poll ntpd
> > 29250 32685 32685 74 2 0x2000180pflogd
> > 32685 1 32685 0 3 0x280 netio pflogd
> > 8367 29197 29197 73 3 0x2000180 poll syslogd
> > 29197 1 29197 0 3 0x288 netio syslogd
> > 14 0 0 0 3 0x2100200 aiodoned aiodoned
> > 13 0 0 0 3 0x2100200 syncerupdate
> > 12 0 0 0 3 0x2100200 cleaner cleaner
> > 11 0 0 0 30x100200 reaperreaper
> > 10 0 0 0 3 0x2100200 pgdaemon pagedaemon
> > 9 0 0 0 3 0x2100200 bored crypto
> > 8 0 0 0 3 0x2100200 pftm pfpurge
> > 7 0 0 0 3 0x2100200 usbtskusbtask
> > 6 0 0 0 3 0x2100200 usbatsk usbatsk
> > 5 0 0 0 3 0x2100200 acpi0 acpi0
> > 4 0 0 0 3 0x40100200idle1
> > 3 0 0 0 3 0x2100200 bored syswq
> > *2 0 0 0 7 0x40100200idle0
> > 1 0 1 0 3 0x2004080 wait init
> > 0 -1 0 0 3 0x2080200 scheduler swapper
> > ddb{0}> trace
> > Debugger() at Debugger+0x5
> > comintr() at comintr+0x268
> > Xintr_ioapic_edge4() at Xintr_ioapic_edge4+0xe8
> > --- interrupt ---
> > Bad frame pointer: 0x8000155b1b30
> > end trace frame: 0x8000155b1b30, count: -3
> > bcopy+0x16:
> > ddb{0}>
OpenBSD fw freezing with ps/trace.
Hi all,
we are running a backup firewall machine which regularly freezes since
OpenBSD 4.6. The configuration also changed at this time. When frozen no
input is accepted by serial or keyboard console. Breaking to ddb works
though. The output of ps and trace are below. The machine is primarily
working as a transparent firewalling bridge but also runs NAT, pf and
dhcpd for a 192.168.x/24. The freeze can often be provoked by obtaining
an IP in the 192.168.x/24 and immediately sshing from this network into
a Host on the bridged network part.
The machine is currently still sitting there, so if additional info is
needed please ask.
thanks,
Leon
Stopped at Debugger+0x5: leave
ddb{0}> ps
PID PPID PGRPUID S FLAGS WAIT COMMAND
8693 1 1 0 3 0x2004080 ttyopngetty
1066 28162 1066 0 3 0x2004080 ttyin zsh
28162 16328 28162 1001 3 0x2004080 pause zsh
16328 27349 16328 1000 3 0x2004080 pause ksh
27349 15191 15191 1000 3 0x2000180 selectsshd
15191 19218 15191 0 3 0x2004180 netio sshd
27916 1 27916 0 3 0x2040180 selectsendmail
18379 1 18379 0 2 0x2004080getty
22394 1 22394 0 3 0x2004080 ttyin getty
24536 1 24536 0 3 0x2004080 ttyin getty
17228 1 17228 0 3 0x2004080 ttyin getty
23114 1 23114 0 3 0x2004080 ttyin getty
14383 1 14383 0 3 0x2004080 ttyin getty
16906 1 16906 0 3 0x280 selectcron
24123 1 24123 0 3 0x2000180 selectinetd
9001 1 9001 77 2 0x2000180dhcpd
19218 1 19218 0 7 0x280sshd
32707 31588 26440 83 3 0x2000180 poll ntpd
31588 26440 26440 83 3 0x2000180 poll ntpd
26440 1 26440 0 3 0x280 poll ntpd
29250 32685 32685 74 2 0x2000180pflogd
32685 1 32685 0 3 0x280 netio pflogd
8367 29197 29197 73 3 0x2000180 poll syslogd
29197 1 29197 0 3 0x288 netio syslogd
14 0 0 0 3 0x2100200 aiodoned aiodoned
13 0 0 0 3 0x2100200 syncerupdate
12 0 0 0 3 0x2100200 cleaner cleaner
11 0 0 0 30x100200 reaperreaper
10 0 0 0 3 0x2100200 pgdaemon pagedaemon
9 0 0 0 3 0x2100200 bored crypto
8 0 0 0 3 0x2100200 pftm pfpurge
7 0 0 0 3 0x2100200 usbtskusbtask
6 0 0 0 3 0x2100200 usbatsk usbatsk
5 0 0 0 3 0x2100200 acpi0 acpi0
4 0 0 0 3 0x40100200idle1
3 0 0 0 3 0x2100200 bored syswq
*2 0 0 0 7 0x40100200idle0
1 0 1 0 3 0x2004080 wait init
0 -1 0 0 3 0x2080200 scheduler swapper
ddb{0}> trace
Debugger() at Debugger+0x5
comintr() at comintr+0x268
Xintr_ioapic_edge4() at Xintr_ioapic_edge4+0xe8
--- interrupt ---
Bad frame pointer: 0x8000155b1b30
end trace frame: 0x8000155b1b30, count: -3
bcopy+0x16:
ddb{0}>
Re: relayd redirection not changing dst-mac (bridge),should it?
Hello, On Wed, Oct 27, 2010 at 01:59:29PM +0200, Reyk Floeter wrote: > hi, > > On Tue, Oct 26, 2010 at 10:54:59PM +0200, Leon Me?ner wrote: > > As you can see in below tcpdump the dst-mac does not change with the > > redirection. So the packet gets routed to the wrong switch port. > > > > it is an "unsupported mode of operation". > > rdr-to will not update the dstmac on a bridge and it doesn't do a > route lookup to get it because we cannot guarantee that there is an > arp entry for the updated dstip address. this would even be impossible > on a fully transparent bridge without configured ip addresses where we > don't have a way to resolve the dstmac/dstip at all. > > rdr-to could theoretically try to do a lookup for this ip and only > update the dstmac if an entry is found but this is tricky and somewhat > whacky and not intended. This bridge actually got a regular interface for management but i wouldnt want any non-standard solution. We get our public IP segment switched to us by the university. So if i would want to use relayd i need to put up a router in-front of our bridges which then does rdr-to? Thanks for your reply, Leon [demime 1.01d removed an attachment of type application/pgp-signature]
relayd redirection not changing dst-mac (bridge),should it?
Hi,
i'm new here so please excuse if this is the wrong list or so.
I do have a problem with getting my relayd to work on an OpenBSD 4.7
bridge thats using pf as a firewall. My configuration is the following:
Internet <--> em2 <--> bridge (pf/relayd) <--> em1 <--> (two
testservers)
Here's the relevant part of relayd.conf i want to debug:
table { $commhost1 $commhost2 }
table { $commhost2 }
redirect test {
listen on $commhost1 port 3 interface em2
tag RELAYD
forward to check tcp
}
As you can see in below tcpdump the dst-mac does not change with the
redirection. So the packet gets routed to the wrong switch port.
First inside if, then outside (lines truncated, sry):
22:38_r...@backdoor:/etc# tcpdump -e -i em1 port 3
tcpdump: listening on em1, link-type EN10MB
22:38:49.909273 00:19:a9:93:c5:80 00:15:17:0e:83:c9 ip 74:
pD9587F1A.dip.t-dialin.net.51864 > comm2.3:
S 1827691053:1827691053(0) win 5840
22:38:52.919782 00:19:a9:93:c5:80 00:15:17:0e:83:c9 ip 74:
pD9587F1A.dip.t-dialin.net.51864 > comm2.3:
S 1827691053:1827691053(0) win 5840
^C
2584 packets received by filter
0 packets dropped by kernel
22:39_r...@backdoor:/etc# tcpdump -e -i em2 port 3
tcpdump: listening on em2, link-type EN10MB
22:39:53.753698 00:19:a9:93:c5:80 00:15:17:0e:83:c9 ip 74:
pD9587F1A.dip.t-dialin.net.51866 > comm.3:
S 2830743421:2830743421(0) win 5840 (DF)
22:39:56.754475 00:19:a9:93:c5:80 00:15:17:0e:83:c9 ip 74:
pD9587F1A.dip.t-dialin.net.51866 > comm.3:
S 2830743421:2830743421(0) win 5840 (DF)
^C
1679 packets received by filter
0 packets dropped by kernel
What am i doing wrong? Why is the dst-mac not changing? If you need more
information please tell. Below is the pf rule that gets generated by
relayd. I will try some "match in on em2 xxx rdr-to other.ip" type rules
later and tell if they work.
Thanks,
Leon
pf rules created by relayd:
# pfctl -a "relayd/test" -s r
pass in quick on em2 inet proto tcp from any to COMMHOST1'sIP port =
3 flags S/SA keep state (tcp.established 600) tag RELAYD rdr-to
port 3 round-robin
With the followin in the Table :
# pfctl -a "relayd/test" -t test -T show
130.149.58.168
