Re: Is PFSync over IPSec still broken?
Hi, Pfsync + ipsec setup IS broken. Links: http://marc.info/?l=openbsd-misc&m=143463803906528&w=2 Patch to manual page has been applied: http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/share/man/man4/pfsync.4.diff?r1=1.32&r2=1.33 Please remove example of this setup: "2. Use the ifconfig(8) syncpeer option (see below) so that updates are unicast directly to the peer, then configure ipsec(4) between the hosts to secure the pfsync(4) traffic." from webpage: http://www.openbsd.org/faq/pf/carp.html Thanks Lukasz W dniu 26.06.2015 o 09:45, Jason McIntyre pisze: > On Fri, Jun 26, 2015 at 09:05:08AM +0200, ??ukasz Czarniecki wrote: >> W dniu 25.06.2015 o 12:19, Jason McIntyre pisze: >> > Please fix this bug or remove this example from documentation. > For me this setup is broken since 2011. > http://marc.info/?l=openbsd-misc&m=130624207811609&w=2 > > Nobody cares or nobody uses? >>> >>> i've just committed something similar to the diff below, though i >>> commented out text rather than removing it. >>> >>> thanks for the diff, >>> jmc >> >> >> Thank you. >> Please also remove this line: >> >> 2. Use the ifconfig(8) syncpeer option (see below) so that updates are >> unicast directly to the peer, then configure ipsec(4) between the hosts >> to secure the pfsync(4) traffic. >> >> from http://www.openbsd.org/faq/pf/carp.html >> > > i'm in less well known territory here... > > cc'ing dlg again to ok, and nick to please make the change if he feels > it's right - www pages have their own logic. > > jmc
Re: Is PFSync over IPSec still broken?
W dniu 25.06.2015 o 12:19, Jason McIntyre pisze: >>> Please fix this bug or remove this example from documentation. >>> For me this setup is broken since 2011. >>> http://marc.info/?l=openbsd-misc&m=130624207811609&w=2 >>> >>> Nobody cares or nobody uses? >> > > i've just committed something similar to the diff below, though i > commented out text rather than removing it. > > thanks for the diff, > jmc Thank you. Please also remove this line: 2. Use the ifconfig(8) syncpeer option (see below) so that updates are unicast directly to the peer, then configure ipsec(4) between the hosts to secure the pfsync(4) traffic. from http://www.openbsd.org/faq/pf/carp.html
Re: Is PFSync over IPSec still broken?
W dniu 2015-06-18 o 17:30, Łukasz Czarniecki pisze: >> It's still broken because as mentioned at the end of the thread you >> linked IPsec state gets replicated to the peer and this is causing >> the "replayed" packets you're seeing. The peer already has IPsec state >> in memory (created by pfsync replication) which matches incoming IPsec >> packets directed at it. So the peer's IPsec stack ends up believing it's >> seen the incoming packet already (while it actually hasn't seen the packet, >> it just copied the IPsec state from the sender) and drops the packet. >> >> No good fix is known as of yet. I've given up on it for now. >> > > Please fix this bug or remove this example from documentation. > For me this setup is broken since 2011. > http://marc.info/?l=openbsd-misc&m=130624207811609&w=2 > > Nobody cares or nobody uses? # diff -u -p /usr/src/share/man/man4/pfsync.4 ./pfsync.4 --- /usr/src/share/man/man4/pfsync.4Sun Feb 1 09:33:48 2015 +++ ./pfsync.4 Sun Jun 21 15:14:00 2015 @@ -112,24 +112,13 @@ An alternative destination address for packets can be specified using the .Ic syncpeer keyword. -This can be used in combination with -.Xr ipsec 4 -to protect the synchronisation traffic. -In such a configuration, the syncdev should be set to the -.Xr enc 4 -interface, as this is where the traffic arrives when it is decapsulated, -e.g.: -.Bd -literal -offset indent -# ifconfig pfsync0 syncpeer 10.0.0.2 syncdev enc0 .Ed .Pp It is important that the pfsync traffic be well secured as there is no authentication on the protocol and it would be trivial to spoof packets which create states, bypassing the pf ruleset. -Either run the pfsync protocol on a trusted network \- ideally a network -dedicated to pfsync messages such as a crossover cable between two firewalls, -or specify a peer address and protect the traffic with -.Xr ipsec 4 . +Run the pfsync protocol on a trusted network \- ideally a network +dedicated to pfsync messages such as a crossover cable between two firewalls. .Sh EXAMPLES .Nm and @@ -219,10 +208,8 @@ net.inet.carp.preempt=1 .Sh SEE ALSO .Xr bpf 4 , .Xr carp 4 , -.Xr enc 4 , .Xr inet 4 , .Xr inet6 4 , -.Xr ipsec 4 , .Xr netintro 4 , .Xr pf 4 , .Xr hostname.if 5 , @@ -244,3 +231,8 @@ protocol and kernel implementation were significantly and .Ox 4.5 . The two protocols are incompatible and will not interoperate. +.Sh BUGS +The +.Nm +protocol does not work over IPsec tunnels. +
Re: Is PFSync over IPSec still broken?
> It's still broken because as mentioned at the end of the thread you > linked IPsec state gets replicated to the peer and this is causing > the "replayed" packets you're seeing. The peer already has IPsec state > in memory (created by pfsync replication) which matches incoming IPsec > packets directed at it. So the peer's IPsec stack ends up believing it's > seen the incoming packet already (while it actually hasn't seen the packet, > it just copied the IPsec state from the sender) and drops the packet. > > No good fix is known as of yet. I've given up on it for now. > Please fix this bug or remove this example from documentation. For me this setup is broken since 2011. http://marc.info/?l=openbsd-misc&m=130624207811609&w=2 Nobody cares or nobody uses? http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man4/pfsync.4?query=pfsync This can be used in combination with ipsec(4) to protect the synchronisation traffic. In such a configuration, the syncdev should be set to the enc(4) interface, as this is where the traffic arrives when it is decapsulated, e.g.: # ifconfig pfsync0 syncpeer 10.0.0.2 syncdev enc0 Lukasz
Is PFSync over IPSec still broken?
Hi, I have the same problem described here: http://openbsd-archive.7691.n7.nabble.com/pfsync-over-ipsec-is-broken-td257496.html#a257681 My system is 5.7 i386 I have keep state (no-sync) on all local terminated traffic (including ipsec udp/esp) and set skip on enc in pf.conf. I can see only outgoing PFSync traffic (no incoming) with increasing replayed packets received on both firewalls. netstat -p esp -s | grep replay 304 possibly replayed packets received Does anyone have working PFSync over IPsec Setup? Lukasz
FTP inspection / ftp-proxy transparent mode
Hi,
I am protecting IPv6 FTP server in my LAN with PF firewall.
I have two options:
1.
pass out inet6 proto tcp to {XXX:XXX::XXX:XX } port 21
pass out inet6 proto tcp to {XXX:XXX::XXX:XX } port > 1024
2.
anchor "ftp-proxy/*"
pass in inet6 proto tcp to XXX:XXX::XXX:XX port 21 divert-to ::1 port 8021
Option 1 leaves open ports > 1024.
Option 2 rewrites client address to my firewall address.
I know that FTP is dead, but does anyone know a solution for that?
I have found post from 2011 about ftp-proxy being ready for transparent
mode:
http://marc.info/?l=openbsd-tech&m=130219958807458&w=1
Any progress?
Lukasz
Re: Dell R310 - does SAS6/iR work ok?
W dniu 2012-01-09 18:58, Graham Allan pisze: > Prepurchase check... I know the SAS 6/iR disk controller has been > supported since OpenBSD 4.3 or but I saw some reports of write > performance issues (due to disabling cache). Does it work ok in 5.0? > > Seems like my choices on the R310 are: > onboard SATA - not available as option with hotswap drives (not > essential but handy) > PERC H200 - uses one of the PCIe slots, want these for other devices > SAS 6/iR - uses custom internal slot, should work with mpi driver > > Thanks, > > Graham It works great now. Write Cache is enabled by default on H200. Big thanks to all OpenBSD devs. OpenBSD 5.0 (GENERIC.MP) #63: Wed Aug 17 10:14:30 MDT 2011 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 8579026944 (8181MB) avail mem = 8336535552 (7950MB) mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.6 @ 0xbf79c000 (66 entries) bios0: vendor Dell Inc. version "1.5.3" date 12/08/2010 bios0: Dell Inc. PowerEdge R310 acpi0 at bios0: rev 2 acpi0: sleep states S0 S4 S5 acpi0: tables DSDT FACP APIC SPCR HPET DM__ MCFG WD__ SLIC ERST HEST BERT EINJ TCPA SSDT acpi0: wakeup devices PCI0(S5) USBA(S0) USBB(S0) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU X3460 @ 2.80GHz, 2793.36 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,NXE,LONG cpu0: 256KB 64b/line 8-way L2 cache cpu0: apic clock running at 132MHz cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Xeon(R) CPU X3460 @ 2.80GHz, 2792.98 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,NXE,LONG cpu1: 256KB 64b/line 8-way L2 cache cpu2 at mainbus0: apid 4 (application processor) cpu2: Intel(R) Xeon(R) CPU X3460 @ 2.80GHz, 2792.98 MHz cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,NXE,LONG cpu2: 256KB 64b/line 8-way L2 cache cpu3 at mainbus0: apid 6 (application processor) cpu3: Intel(R) Xeon(R) CPU X3460 @ 2.80GHz, 2792.98 MHz cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,NXE,LONG cpu3: 256KB 64b/line 8-way L2 cache cpu4 at mainbus0: apid 1 (application processor) cpu4: Intel(R) Xeon(R) CPU X3460 @ 2.80GHz, 2792.98 MHz cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,NXE,LONG cpu4: 256KB 64b/line 8-way L2 cache cpu5 at mainbus0: apid 3 (application processor) cpu5: Intel(R) Xeon(R) CPU X3460 @ 2.80GHz, 2792.98 MHz cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,NXE,LONG cpu5: 256KB 64b/line 8-way L2 cache cpu6 at mainbus0: apid 5 (application processor) cpu6: Intel(R) Xeon(R) CPU X3460 @ 2.80GHz, 2792.98 MHz cpu6: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,NXE,LONG cpu6: 256KB 64b/line 8-way L2 cache cpu7 at mainbus0: apid 7 (application processor) cpu7: Intel(R) Xeon(R) CPU X3460 @ 2.80GHz, 2792.98 MHz cpu7: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,NXE,LONG cpu7: 256KB 64b/line 8-way L2 cache ioapic0 at mainbus0: apid 0 pa 0xfec0, version 20, 24 pins acpihpet0 at acpi0: 14318179 Hz acpimcfg0 at acpi0 addr 0xe000, bus 0-255 acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 4 (LYD0) acpiprt2 at acpi0: bus 5 (LYD2) acpiprt3 at acpi0: bus -1 (HVD0) acpiprt4 at acpi0: bus -1 (HVD2) acpiprt5 at acpi0: bus 3 (PEX0) acpiprt6 at acpi0: bus -1 (PEX2) acpiprt7 at acpi0: bus -1 (PEX3) acpiprt8 at acpi0: bus 2 (PEX4) acpiprt9 at acpi0: bus 1 (COMP) acpicpu0 at acpi0: C3, C1 acpicpu1 at acpi0: C3, C1 acpicpu2 at acpi0: C3, C1 acpicpu3 at acpi0: C3, C1 acpicpu4 at acpi0: C3, C1 acpicpu5 at acpi0: C3, C1 acpicpu6 at acpi0: C3, C1 acpicpu7 at acpi0: C3, C1 ipmi at mainbus0 not configured pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel Core DMI" rev 0x11 ppb0 at pci0 dev 3 function 0 "Intel Core PCIE" rev 0x11: msi pci1 at ppb0 bus 4 bnx0 at pci1 d
Re: dhclient, resolv.conf
W dniu 2011-10-20 20:11, sophia.ort...@googlemail.com pisze: > But again, I insist in my first question: how I get that > dhclient respect my resolv.conf and do not touch it? chflags uchg /etc/resolv.conf
Re: OpenBSD 5.0 PF and Syn attak
W dniu 2011-09-25 17:50, Pui Edylie pisze: > Hi Everyone, > > I am trying to put a pair of OBSD box together to provide Syn, UDP and > ICMP flood protection with pretty graphs. > > May I know if anyone has accomplished this? Check this out: Bakeca.it DDoS: How Evil Forces Have Been Defeated http://data.proidea.org.pl/confidence/5edycja/materialy/prezentacje/alessio_pennasilico_bakeca_ddos_confidence_2009.pdf http://data.proidea.org.pl/confidence/5edycja/materialy/video/alessio_pennasilico.avi
pfsync over IPsec kernel panic when syncdev is not enc0
Hi
While testing pfsync over IPsec I have spotted a bug. While it is
documented in man pfsync that enc0 should be used as syncdev when using
pfsync over ipsec IMHO the system should not crash when the physical
interface is used.
This bug can be spotted on 4.8/i386, 4.9/i386 and Current/i386. I have
not tested any other architectures.
I have noticed it when I tried to secure (with IPSEC) already configured
pfsync environment.
It is very easy to reproduce. You need to have two physical interfaces.
One with some traffic and on the second you need to configure pfsync
over ipsec like this:
FW1:
ifconfig xl0 10.0.0.2/24
ping 10.0.0.1 (in background)
ifconfig bge0 192.168.1.1/24 up
/etc/ipsec.conf:
ike esp from 192.168.1.1 to 192.168.1.2 peer 192.168.1.2 psk test
isakmpd -4 -v -K
ipsecctl -f /etc/ipsec.conf
ifconfig pfsync0 create
ifconfig pfsync0 syncpeer 192.168.1.2 syncdev bge0 up
ifconfig pfsync0 down
ifconfig pfsync0 up
crash within seconds
On the second machine you need:
ifconfig em0 192.168.1.2/24 up
/etc/ipsec.conf:
ike esp from 192.168.1.2 to 192.168.1.1 peer 192.168.1.1 psk test
isakmpd -4 -v -K
ipsecctl -f /etc/ipsec.conf
Sometimes my system freezes sometimes it crashes sometimes kernel panics.
uvm_fault(0xd0a34340, 0x100, 0,1) -> e
kernel: page fault trap, code=0
Stopped at m_cluncount+0x1a: movzwl 0x12(%edx),%eax
ddb> trace
m_cluncount(d6d75600,1,d6d59d00,dc006e74,d0400252) at m_cluncount_0x1a
ether_input(d1dc904c,0,d6d75600,200,a) at ether_input+0x2b
xl_rxeof(d1dc9000,9000,e,6c01,d0202490) at xl_rxeof+0x121
xl_intr(d1dc9000) at xl_intr+0xd5
Xintr_ioapic0() at Xintr_ioapic0+0x70
--- interrupt ---
cpu_idle_cycle(d0af0e20) at cpu_idle_cycle+0xf
Bad frame pointer: 0xd0ba8e48
Dmesg below:
OpenBSD 4.9-current (GENERIC) #2: Sun May 22 22:31:55 MDT 2011
dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Pentium(R) D CPU 2.80GHz ("GenuineIntel" 686-class) 2.82 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,CNXT-ID,CX16,xTPR
real mem = 1064824832 (1015MB)
avail mem = 1036578816 (988MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 09/26/05, BIOS32 rev. 0 @ 0xfb390,
SMBIOS rev. 2.3 @ 0xf0100 (39 entries)
bios0: vendor Award Software International, Inc. version "F8" date
09/26/2005
bios0: Gigabyte Technology Co., Ltd. 8I945G Pro
acpi0 at bios0: rev 0
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP MCFG APIC
acpi0: wakeup devices PEX0(S5) PEX1(S5) PEX2(S5) PEX3(S5) PEX4(S5)
PEX5(S5) HUB0(S5) USB0(S1) USB1(S1) USB2(S1) USB3(S1) USBE(S1) AC97(S5)
MC97(S5) AZAL(S5) PCI0(S5)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimcfg0 at acpi0 addr 0xf000, bus 0-255
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 200MHz
cpu at mainbus0: not configured
ioapic0 at mainbus0: apid 2 pa 0xfec0, version 20, 24 pins
ioapic0: misconfigured as apic 0, remapped to apid 2
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (PEX0)
acpiprt2 at acpi0: bus -1 (PEX1)
acpiprt3 at acpi0: bus 2 (PEX2)
acpiprt4 at acpi0: bus -1 (PEX3)
acpiprt5 at acpi0: bus -1 (PEX4)
acpiprt6 at acpi0: bus -1 (PEX5)
acpiprt7 at acpi0: bus 3 (HUB0)
acpicpu0 at acpi0
acpibtn0 at acpi0: PWRB
bios0: ROM list: 0xc/0xa800! 0xcc000/0x8000!
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
pchb0 at pci0 dev 0 function 0 "Intel 82945G Host" rev 0x02
vga1 at pci0 dev 2 function 0 "Intel 82945G Video" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
intagp0 at vga1
agp0 at intagp0: aperture at 0xd000, size 0x1000
inteldrm0 at vga1: apic 2 int 16
drm0 at inteldrm0
ppb0 at pci0 dev 28 function 0 "Intel 82801GB PCIE" rev 0x01: apic 2 int 16
pci1 at ppb0 bus 1
ppb1 at pci0 dev 28 function 2 "Intel 82801GB PCIE" rev 0x01: apic 2 int 18
pci2 at ppb1 bus 2
bge0 at pci2 dev 0 function 0 "Broadcom BCM5789" rev 0x11, BCM5750 B1
(0x4101): apic 2 int 18, address 00:14:85:e1:d7:51
brgphy0 at bge0 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
uhci0 at pci0 dev 29 function 0 "Intel 82801GB USB" rev 0x01: apic 2 int 23
uhci1 at pci0 dev 29 function 1 "Intel 82801GB USB" rev 0x01: apic 2 int 19
uhci2 at pci0 dev 29 function 2 "Intel 82801GB USB" rev 0x01: apic 2 int 18
uhci3 at pci0 dev 29 function 3 "Intel 82801GB USB" rev 0x01: apic 2 int 16
ehci0 at pci0 dev 29 function 7 "Intel 82801GB USB" rev 0x01: apic 2 int 23
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb2 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xe1
pci3 at ppb2 bus 3
xl0 at pci3 dev 0 function 0 "3Com 3c905C 100Base-TX" rev 0x78: apic 2
int 20, address 00:04:75:dc:57:48
exphy0 at xl0 phy 24: 3Com internal media interface
"Creative Labs SoundBlaster Audigy LS" rev 0x00 at pci3 dev 1 function 0
not configured
pciide0 at pci3 dev 6 function
Is it a hardware problem?
Trying to compile 4.8-stable:
*1st attempt:*
uvm_fault(0xd0a05960, 0x8000, 0, 1) -> e
kernel: page fault trap, code=0
Stopped at pmap_page_remove+0xad: movl0x4(%esi),%eax
ddb>
More details in dmesg below.
*2nd attempt:*
cc -Werror -Wall -Wstrict-prototypes -Wmissing-prototypes -Wno-main
-Wno-uninitialized -Wno-format -Wstack-larger-than-2047
-fno-builtin-printf -fno-builtin-snprintf -fno-builtin-vsnprintf
-fno-builtin-log -fno-builtin-log2 -fno-builtin-malloc -O2 -pipe
-nostdinc -I. -I../../../.. -I../../../../arch -DDDB -DDIAGNOSTIC
-DKTRACE -DACCOUNTING -DKMEMSTATS -DPTRACE -DCRYPTO -DSYSVMSG -DSYSVSEM
-DSYSVSHM -DUVM_SWAP_ENCRYPT -DCOMPAT_43 -DLKM -DFFS -DFFS2
-DFFS_SOFTUPDATES -DUFS_DIRHASH -DQUOTA -DEXT2FS -DMFS -DNNPFS
-DTCP_SACK -DTCP_ECN -DTCP_SIGNATURE -DNFSCLIENT -DNFSSERVER -DCD9660
-DUDF -DMSDOSFS -DFIFO -DINET -DALTQ -DINET6 -DIPSEC -DPPP_BSDCOMP
-DPPP_DEFLATE -DMROUTING -DBOOT_CONFIG -DUSER_PCICONF -DKVM86 -DUSER_LDT
-DAPERTURE -DCOMPAT_SVR4 -DCOMPAT_LINUX -DCOMPAT_FREEBSD -DCOMPAT_AOUT
-DPROCFS -DPCIVERBOSE -DEISAVERBOSE -DUSBVERBOSE -DWSDISPLAY_COMPAT_USL
-DWSDISPLAY_COMPAT_RAWKBD -DWSDISPLAY_DEFAULTSCREENS="6"
-DWSDISPLAY_COMPAT_PCVT -DX86EMU -DONEWIREVERBOSE -DMAXUSERS=80
-D_KERNEL -c ../../../../dev/pci/if_wpi.c
../../../../dev/pci/if_wpi.c: In function 'wpi_read_firmware':
../../../../dev/pci/if_wpi.c:3019: error: stray '\301' in program
../../../../dev/pci/if_wpi.c:3019: error: 'WPI_FW_TEXT_M' undeclared
(first use in this function)
../../../../dev/pci/if_wpi.c:3019: error: (Each undeclared identifier is
reported only once
../../../../dev/pci/if_wpi.c:3019: error: for each function it appears in.)
../../../../dev/pci/if_wpi.c:3019: error: expected ')' before 'XSZ'
*** Error code 1
Stop in /usr/src/sys/arch/i386/compile/GENERIC (line 92 of
/usr/share/mk/sys.mk).
*3rd attempt:*
cc -Werror -Wall -Wstrict-prototypes -Wmissing-prototypes -Wno-main
-Wno-uninitialized -Wno-format -Wstack-larger-than-2047
-fno-builtin-printf -fno-builtin-snprintf -fno-builtin-vsnprintf
-fno-builtin-log -fno-builtin-log2 -fno-builtin-malloc -O2 -pipe
-nostdinc -I. -I../../../.. -I../../../../arch -DDDB -DDIAGNOSTIC
-DKTRACE -DACCOUNTING -DKMEMSTATS -DPTRACE -DCRYPTO -DSYSVMSG -DSYSVSEM
-DSYSVSHM -DUVM_SWAP_ENCRYPT -DCOMPAT_43 -DLKM -DFFS -DFFS2
-DFFS_SOFTUPDATES -DUFS_DIRHASH -DQUOTA -DEXT2FS -DMFS -DNNPFS
-DTCP_SACK -DTCP_ECN -DTCP_SIGNATURE -DNFSCLIENT -DNFSSERVER -DCD9660
-DUDF -DMSDOSFS -DFIFO -DINET -DALTQ -DINET6 -DIPSEC -DPPP_BSDCOMP
-DPPP_DEFLATE -DMROUTING -DBOOT_CONFIG -DUSER_PCICONF -DKVM86 -DUSER_LDT
-DAPERTURE -DCOMPAT_SVR4 -DCOMPAT_LINUX -DCOMPAT_FREEBSD -DCOMPAT_AOUT
-DPROCFS -DPCIVERBOSE -DEISAVERBOSE -DUSBVERBOSE -DWSDISPLAY_COMPAT_USL
-DWSDISPLAY_COMPAT_RAWKBD -DWSDISPLAY_DEFAULTSCREENS="6"
-DWSDISPLAY_COMPAT_PCVT -DX86EMU -DONEWIREVERBOSE -DMAXUSERS=80
-D_KERNEL -c ../../../../dev/ic/if_wi.c
../../../../dev/ic/if_wi.c: In function 'wi_setmulti':
../../../../dev/ic/if_wi.c:1396: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See http://gcc.gnu.org/bugs.html> for instructions.
*** Error code 1
Stop in /usr/src/sys/arch/i386/compile/GENERIC (line 92 of
/usr/share/mk/sys.mk).
*4th attempt:*
Successfull.
I assume that it is a hardware problem. Am I right?
Does successfull compilation mean that compiled kernel is ok? Can you
share your experience?
Dmesg:
OpenBSD 4.8 (GENERIC) #0: Thu Feb 10 16:36:16 CET 2011
root@xxx:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Celeron(R) CPU 2.40GHz ("GenuineIntel" 686-class) 2.40 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL
,CNXT-ID,xTPR
real mem = 534540288 (509MB)
avail mem = 515837952 (491MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 04/02/04, BIOS32 rev. 0 @ 0xf0010,
SMBIOS rev. 2.3 @ 0xfba10 (80 entries)
bios0: vendor Intel Corp. version "RL86510A.86A.0075.P15.0404021333"
date 04/02/2004
bios0: Intel Corporation D865PERL
acpi0 at bios0: rev 0
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP APIC ASF! WDDT
acpi0: wakeup devices TANA(S4) P0P3(S4) AC97(S4) USB0(S4) USB1(S4)
USB2(S4) USB3(S4) USB7(S4) UAR1(S4) SLPB(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 133MHz
ioapic0 at mainbus0: apid 1 pa 0xfec0, version 20, 24 pins
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (P0P1)
acpiprt2 at acpi0: bus 2 (P0P2)
acpiprt3 at acpi0: bus 3 (P0P3)
acpicpu0 at acpi0
acpipwrres0 at acpi0: URP1
acpipwrres1 at acpi0: FDDP
acpipwrres2 at acpi0: LPTP
acpibtn0 at acpi0: SLPB
bios0: ROM list: 0xc/0xd000
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
pchb0 at pci0 dev 0 function 0 "Intel 82865G Host" rev 0x02
intelagp0 at pchb0
agp0 at intelagp0: aperture at 0xf800, size 0x400
ppb0 at pci0 dev 1
Re: Dell R310 - H200 Raid performance problem
With following Mike's suggestions it worked.
could you please change this line
if (mpii_req_cfg_page(sc, addr, 0, &hdr, 1, vpg, pagelen) != 0) {
to
if (mpii_req_cfg_page(sc, addr, MPII_PG_POLL, &hdr, 1, vpg,
pagelen) != 0) {
>>> and one more:
>>>
>>> this:
>>> if (mpii_req_cfg_header(sc, MPII_CONFIG_REQ_PAGE_TYPE_RAID_VOL, 0,
>>>addr, 0, &hdr) != 0)
>>> to:
>>> if (mpii_req_cfg_header(sc, MPII_CONFIG_REQ_PAGE_TYPE_RAID_VOL, 0,
>>>addr, MPII_PG_POLL, &hdr) != 0)
mpii0 at pci2 dev 0 function 0 "Symbios Logic SAS2008" rev 0x02: apic 0
int 16 (irq 15)
scsibus0 at mpii0: 42 targets
sd0 at scsibus0 targ 1 lun 0: SCSI4 0/direct
fixed
sd0: 237824MB, 512 bytes/sec, 487063552 sec total
ses0 at scsibus0 targ 10 lun 0: SCSI3 13/enclosure
services fixed
ses0: unable to read enclosure configuration
# scsi -f /dev/rsd0c -m 8
IC: 0
ABPF: 0
CAP: 0
DISC: 0
SIZE: 0
WCE: 1
MF: 0
RCD: 0
Demand Retention Priority: 0
Write Retention Priority: 0
Disable Pre-fetch Transfer Length: 65535
Minimum Pre-fetch: 0
Maximum Pre-fetch: 65280
Maximum Pre-fetch Ceiling: 65535
FSW: 0
LBCSS: 0
DRA: 0
Vendor-specific: 0
NV_DIS: 0
Number of Cache Segments: 15
Cache Segment Size: 0
how to manipulate write cache policy?
Lukasz
Re: Dell R310 - H200 Raid performance problem
On 18.02.2011 07:57, David Gwynne wrote: > this diff implements the disk cache ioctl handling in mpii so sd(4) > can drive the change rather than have mpii(4) whack everything. > modelled on the same functionality in mpi(4) and mikeb's code... > > could someone test this please? It freezes on my system. Last lines from dmesg are: mpii0 at pci2 dev 0 function 0 "Symbios Logic SAS2008" rev 0x02: apic 0 int 16 (irq 15) scsibus0 at mpii0: 42 targets sd0 at scsibus0 targ 1 lun0: SCSI4 0/direct fixed sd0: 237824MB, 512 bytes/sec, 487063772 sec total Lukasz [demime 1.01d removed an attachment of type image/jpeg which had a name of 20110218281.jpg]
Re: Dell R310 - H200 Raid performance problem
On 17.02.2011 16:22, Mike Belopuhov wrote: > Lukasz has tested the patch below and it works fine for him. I don't > have the hardware myself, so I'm not going to push it for the release, > but if someone thinks it's worth it, please speak up. Here are some numbers: 4.8 # time tar xzf ./sys.tar.gz 0m11.06s real 0m0.80s user 0m0.86s system w/softdeps 0m4.97s real 0m0.68s user 0m0.58s system Current 0m7.13s real 0m0.75s user 0m0.83s system w/softdeps 0m3.72s real 0m0.60s user 0m0.37s system It seems that 4.9 has a lot of improvements. Big thanks for Mike and all developers. Lukasz
Re: Dell R310 - H200 Raid performance problem
W dniu 2/11/2011 8:20 PM, Rodolfo Gouveia pisze: On Thu, Feb 10, 2011 at 09:49:43PM -0500, Nick Holland wrote: Also, check to see if your RAID card has a battery for its cache, if it doesn't, a lot of RAID controllers drop to non-cached writes, and often seem to slow down way beyond what you'd expect just to make you buy the dang battery :). I believe most of the current crop of Dell RAID controllers have an option buried in the RAID setup screens to cache writes even without a battery. Don't blame me (or Dell, or anyone else) if you trip over the power cord and blow away your array. I second this too. Check if you have that write cache enabled. Hi Machine Bios, DRAC and RAID firmware are fully updated. There is no option in RAID firmware to enable write cache. According to this: http://www.dell.com/content/topics/topic.aspx/global/products/pvaul/topics/en/us/raid_controller?c=us&dgc=CJ&cid=24471&lid=566643&acd=10550055-4166869-1hyp1szsf4our and this: http://support.dell.com/support/edocs/storage/storlink/h200/en/ug/html/features.htm#wp1062398 Controller doesn't have internal cache and "On a PERC H200 card, caching is forced to be disabled for all physical disks configured into a virtual disk, regardless of the drive type and default drive settings." I guess that Linux somehow reenables disc drive cache. I've tried to do that with OpenBSD by writing WCE:1 to this command: scsi -f /dev/rsd0c -m 8 -e But it fails. Does anyone know how to reenable disks cache? # scsi -f /dev/rsd0c -m 8 -e SCIOCCOMMAND ioctl: Command accepted. return status 3 (Sense Returned) host adapter status 2 Command out (6 of 6): 15 00 00 00 18 00 Data out (0 of 24): Error code is "current errors" Segment number is 07 Sense key is "Illegal request" The Information field is not valid but contains ff02 (-16777214). The Command Specific Information field is 05000380 (83886976). Additional sense code: 20 Additional sense code qualifier: 00 Illegal value in the parameter list. Byte 0 (value 00) is illegal. sense (18 of 48): 70 07 05 ff 00 00 02 00 05 00 03 80 20 00 04 80 00 00 # scsi -f /dev/rsd0c -m 8 IC: 0 ABPF: 0 CAP: 0 DISC: 0 SIZE: 0 WCE: 0 MF: 0 RCD: 0 Demand Retention Priority: 0 Write Retention Priority: 0 Disable Pre-fetch Transfer Length: 65535 Minimum Pre-fetch: 0 Maximum Pre-fetch: 65280 Maximum Pre-fetch Ceiling: 65535 FSW: 0 LBCSS: 0 DRA: 0 Vendor-specific: 0 NV_DIS: 0 Number of Cache Segments: 15 Cache Segment Size: 0 Lukasz
Re: Dell R310 - H200 Raid performance problem
On 11.02.2011 03:49, Nick Holland wrote: > tip: use OpenBSD's resident ftp app, save a package: > /tmp $ ftp http://ftp.spline.de/pub/OpenBSD/4.8/sys.tar.gz :) > i.e., "basically the same for all" Therefore, I'm ignoring all but the > 4.9 GENERIC. I almost never complain about dmesgs being included, but > including four different dmesgs that show the same result wasn't overly > interesting and 57k emails are a bit big... :) Sorry, just trying to be helpfull :). > > Sounds like you don't have softdeps running on your system. Use 'em > (FAQ 14). > A lot slower, but still a lot better than you are getting, so, I suspect > you have both issues going on. > > There are about 10,000 files in that file, so that's a lot of file > creations, that's the stuff that Softdeps shines on. Enabling softdeps made some improvement but performance is still unacceptable. # mount /dev/sd0a on / type ffs (local) /dev/sd0m on /home type ffs (local, nodev, nosuid, softdep) /dev/sd0d on /tmp type ffs (local, nodev, nosuid) /dev/sd0f on /usr type ffs (local, nodev) /dev/sd0g on /usr/X11R6 type ffs (local, nodev) /dev/sd0h on /usr/local type ffs (local, nodev) /dev/sd0l on /usr/obj type ffs (local, nodev, nosuid) /dev/sd0k on /usr/src type ffs (local, nodev, nosuid) /dev/sd0e on /var type ffs (local, nodev, nosuid) # pwd /home/test/6 # time tar xzf ./sys.tar.gz 1m2.53s real 0m0.58s user 0m0.62s system I think my problem is somehow related to this: http://old.nabble.com/Dell-R310-with-SAS-drives-very-slow-td28659314.html And this: http://support.dell.com/support/edocs/storage/storlink/h200/en/ug/html/features.htm#wp1062398 Thank you. Lukasz
Dell R310 - H200 Raid performance problem
Hi I've bought a Dell R310 with H200 raid controller reported in dmesg as: Symbios Logic SAS2008. It uses mpii driver and has two hard drives configured in RAID 1. I had a kernel crash while booting amd64-stable kernel. System Event Log: E171F PCIe Fatal Error on Bus 0 Device 5 Function 0 - it is RAID Controller. It happened only once, I cant reproduce it. Now it seems to work fine but i still have a problem with its performance. Raid is fully initialized. How can I help to resolve this problem? I'm doing simple benchmark: wget ftp.spline.de/pub/OpenBSD/4.8/sys.tar.gz time tar xzf ./sys.tar.gz On the same hardware Linux unpacks it in less then two seconds. Numbers for OpenBSD: 4.8 amd64 sp: 3m40.95s real 0m0.65s user 0m0.71s system 4.8 amd64 mp-stable: 3m43.36s real 0m0.48s user 0m0.98s system 4.9 amd64 sp: 3m47.72s real 0m0.51s user 0m0.69s system 4.9 i386 rd : 3m45.11s real 0m1.03s user 0m1.19s system While unpacking iostat -w 1 -c 10 4.8 amd64 sp: ttysd0 cd0 cpu tin tout KB/t t/s MB/s KB/t t/s MB/s us ni sy in id 2 470 16.79 105 1.72 0.00 0 0.00 0 0 1 0 99 00 17.26 153 2.59 0.00 0 0.00 0 0 0 0100 00 17.69 154 2.67 0.00 0 0.00 0 0 0 0100 00 17.46 157 2.68 0.00 0 0.00 0 0 1 0 99 00 18.66 150 2.74 0.00 0 0.00 0 0 1 0 99 00 18.53 153 2.78 0.00 0 0.00 0 0 1 0 99 00 20.07 166 3.26 0.00 0 0.00 1 0 0 0 99 00 16.56 161 2.61 0.00 0 0.00 0 0 0 0100 00 18.29 164 2.94 0.00 0 0.00 0 0 0 0100 00 16.34 163 2.61 0.00 0 0.00 0 0 1 0 99 4.8 amd64 mp-stable: ttysd0 cd0 sd1 cpu tin tout KB/t t/s MB/s KB/t t/s MB/s KB/t t/s MB/s us ni sy in id 04 16.42 1 0.02 0.00 0 0.00 1.17 0 0.00 0 0 0 0100 0 55 18.35 15 0.27 0.00 0 0.00 0.00 0 0.00 0 0 0 0100 0 18 18.05 14 0.26 0.00 0 0.00 0.00 0 0.00 0 0 0 0100 0 18 17.88 15 0.26 0.00 0 0.00 0.00 0 0.00 0 0 0 0100 0 18 17.98 14 0.25 0.00 0 0.00 0.00 0 0.00 0 0 0 0100 0 18 18.12 15 0.26 0.00 0 0.00 0.00 0 0.00 0 0 0 0100 0 18 18.12 15 0.26 0.00 0 0.00 0.00 0 0.00 0 0 0 0100 0 18 18.70 15 0.27 0.00 0 0.00 0.00 0 0.00 0 0 0 0100 0 18 17.88 15 0.26 0.00 0 0.00 0.00 0 0.00 0 0 0 0100 0 18 18.23 15 0.26 0.00 0 0.00 0.00 0 0.00 0 0 0 0100 4.9 amd64 sp: tty sd0:a335ec15bdcd0: cpu tin tout KB/t t/s MB/s KB/t t/s MB/s us ni sy in id 2 169 16.59 71 1.15 0.00 0 0.00 0 0 1 0 99 00 15.01 132 1.93 0.00 0 0.00 0 0 0 0100 00 12.66 126 1.55 0.00 0 0.00 0 0 0 0100 00 15.70 134 2.05 0.00 0 0.00 0 0 0 0100 00 21.10 209 4.30 0.00 0 0.00 2 0 3 0 95 00 14.12 135 1.86 0.00 0 0.00 0 0 0 0100 00 14.45 131 1.84 0.00 0 0.00 0 0 0 1 99 00 14.67 137 1.96 0.00 0 0.00 0 0 0 1 99 00 15.83 117 1.81 0.00 0 0.00 0 0 1 0 99 00 14.79 153 2.22 0.00 0 0.00 0 0 0 0100 # bioctl -ivh mpii0 Volume Status Size Device mpii0 0 Online 232G RAID1 0 Online 233G 0:1.0 noencl 'WD-WCAT1H125700' 1 Online 233G 0:0.0 noencl 'WD-WCAT1H123678' # pcidump -v Domain /dev/pci0: 0:0:0: Intel Core DMI 0x: Vendor ID: 8086 Product ID: d130 0x0004: Command: Status ID: 0010 0x0008: Class: 06 Subclass: 00 Interface: 00 Revision: 11 0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 10 0x0010: BAR empty () 0x0014: BAR empty () 0x0018: BAR empty () 0x001c: BAR empty () 0x0020: BAR empty () 0x0024: BAR empty () 0x0028: Cardbus CIS: 0x002c: Subsystem Vendor ID: 1028 Product ID: 02a3 0x0030: Expansion ROM Base Address: 0x0038: 0x003c: Interrupt Pin: 00 Line: 00 Min Gnt: 00 Max Lat: 00 0x0060: Capability 0x05: Message Signaled Interrupts (MSI) 0x0090: Capability 0x10: PCI Express Link Speed: 2.5 / 2.5 Gb/s Link Width: x4 / x4 0x00e0: Capability 0x01: Power Management 0:3:0: Intel Core PCIE 0x: Vendor ID: 8086 Product ID: d138 0x0004: Command: 0147 Status ID: 0010 0x0008: Class: 06 Subclass: 04 Interface: 00 Revision: 11 0x000c: BIST: 00 Header Type: 01 Latency Timer: 00 Cache Line Size: 10 0x0010: 0x0014: 0x0018: Primary Bus: 0 Secondary Bus: 4 Subordinate Bus: 4 Secondary Latency T
