Re: Rspamd with smtpd
On 2015-11-11, Joerg Jung wrote: >> Am 11.11.2015 um 05:44 schrieb Daniel Ouellet : >> >> Does anyone use this port yet Rspamd. >> >> I saw Stuart + a few helpers making a port of Rspamd. Only on current >> now, so I install current on a server and try to run it. >> >> But anyone have any clue stick to provide on how to actually plug it >> with smtpd? > > I do not use it, but I guess you can use it in LDA mode > with "... deliver to mda rspamc..." in smtpd.conf, > as described here https://rspamd.com/doc/integration.html I don't use it either (I'm seeing timeouts on a significant proportion of mails), but assuming you're only wanting to use it for locally delivered mail, this is correct. There is also rmilter which connects between milter-capable MTAs and rspamd, this hasn't been ported yet, but smtpd doesn't support milter anyway.. A better way to connect it to smtpd would be via a filter. A filter would need to listen for SMTP connections, extract the message and feed to rspamd, and send the modified message back to smtpd via another port. >> Looks like Rspamd accept only input via the http standard. spamc just accepts mail on standard input and feeds it to rspamd for you.
Re: Rspamd with smtpd
On 11/11/15 5:30 AM, Stuart Henderson wrote: > On 2015-11-11, Joerg Jung wrote: >>> Am 11.11.2015 um 05:44 schrieb Daniel Ouellet : >>> >>> Does anyone use this port yet Rspamd. >>> >>> I saw Stuart + a few helpers making a port of Rspamd. Only on current >>> now, so I install current on a server and try to run it. >>> >>> But anyone have any clue stick to provide on how to actually plug it >>> with smtpd? >> >> I do not use it, but I guess you can use it in LDA mode >> with "... deliver to mda rspamc..." in smtpd.conf, >> as described here https://rspamd.com/doc/integration.html > > I don't use it either (I'm seeing timeouts on a significant proportion of > mails), but assuming you're only wanting to use it for locally delivered mail, > this is correct. There is also rmilter which connects between milter-capable > MTAs and rspamd, this hasn't been ported yet, but smtpd doesn't support > milter anyway.. But do you have a choice to send it back to smtpd for local delivery or you need to use Dovecot? rspamc will not go as daemon, so you restart it for every emails, not making it better really... I could go back and use postfix with it, but I sure didn't want too. > A better way to connect it to smtpd would be via a filter. A filter would > need to listen for SMTP connections, extract the message and feed to rspamd, > and send the modified message back to smtpd via another port. > >>> Looks like Rspamd accept only input via the http standard. > > spamc just accepts mail on standard input and feeds it to rspamd for you. > I did read a lot on it. I even looked at sptmd-extras and all. But it is not stable yet or recommended for production for sure. It's all easier set then done. But I keep trying. The main reason is that I have way to many issues with spamassassin when there is more then one relay with tagged in the spamd.conf I was hoping to have something faster may be using less resources and definitely more stable. But so far, I can't even get this going. Oh well I will keep digging then... So far it's more an exercise in frustration and increase the beer consumption! ):> I can't pull my hairs, I have not much left...
Re: 5.8 freezes on Shuttle DS87, anybody else?
Hi folks,
below you can find the trace and ps for the frozen system,
as well as the output of dmesg.
Hope this helps. Please mail if I can help to track down this
problem.
Many thanx
Harri
-
OpenBSD/amd64 (redgate.red.aixigo.de) (tty00)
login: Stopped at Debugger+0x9: leave
ddb{0}> trace
Debugger() at Debugger+0x9
comintr() at comintr+0x253
intr_handler() at intr_handler+0x67
Xintr_ioapic_edge4() at Xintr_ioapic_edge4+0xc9
--- interrupt ---
x86_bus_space_io_write_2() at x86_bus_space_io_write_2+0xf
re_intr() at re_intr+0xbd
intr_handler() at intr_handler+0x67
Xintr_ioapic_edge22() at Xintr_ioapic_edge22+0xc9
--- interrupt ---
Xsoftclock() at Xsoftclock+0x15
--- interrupt ---
end trace frame: 0x0, count: -9
0x8:
ddb{0}> ps
PID PPID PGRPUID S FLAGS WAIT COMMAND
30454 1 30454 0 30x83 ttyin getty
10874 1 1 0 30x82 ttyopngetty
28447 1 28447 0 30x83 ttyin getty
5725 1 5725 0 30x83 ttyin getty
3173 1 3173 0 30x83 ttyin getty
29633 1 29633 0 30x83 ttyin getty
19476 1 19476 0 30x83 ttyin getty
24969 1 24969 0 30x80 poll cron
14110 1 14110 0 30x80 kqreadapmd
8681 1 4009631 30x90 poll dnsmasq
26045 1 26045 99 30x90 poll sndiod
18870522522 95 30x90 kqreadsmtpd
473522522 95 30x90 kqreadsmtpd
6553522522 95 30x90 kqreadsmtpd
14104522522 95 30x90 kqreadsmtpd
29084522522 95 30x90 kqreadsmtpd
24592522522103 30x90 kqreadsmtpd
522 1522 0 30x80 kqreadsmtpd
14125 1 14125 0 30x80 selectsshd
10581 32102 19391 83 30x90 poll ntpd
32102 19391 19391 83 30x90 poll ntpd
19391 1 19391 0 30x80 poll ntpd
127 5292 5292 74 30x90 bpf pflogd
5292 1 5292 0 30x80 netio pflogd
1933 1100 1100 73 30x90 kqreadsyslogd
1100 1 1100 0 30x80 netio syslogd
172 0 0 0 3 0x14200 pgzerozerothread
23588 0 0 0 3 0x14200 aiodoned aiodoned
4889 0 0 0 3 0x14200 syncerupdate
23401 0 0 0 3 0x14200 cleaner cleaner
7897 0 0 0 3 0x14200 reaperreaper
32283 0 0 0 3 0x14200 pgdaemon pagedaemon
319 0 0 0 3 0x14200 bored crypto
2315 0 0 0 3 0x14200 pftm pfpurge
24227 0 0 0 3 0x14200 usbtskusbtask
31176 0 0 0 3 0x14200 usbatsk usbatsk
22851 0 0 0 3 0x14200 bored intelrel
9237 0 0 0 3 0x40014200 acpi0 acpi0
859 0 0 0 7 0x40014200idle3
8843 0 0 0 7 0x40014200idle2
22722 0 0 0 7 0x40014200idle1
21595 0 0 0 3 0x14200 bored sensors
32532 0 0 0 2 0x14200softnet
17651 0 0 0 3 0x14200 bored systqmp
26185 0 0 0 3 0x14200 bored systq
*32437 0 0 0 7 0x40014200idle0
1 0 1 0 30x82 wait init
0 -1 0 0 3 0x10200 scheduler swapper
-
OpenBSD 5.8-stable (GENERIC.DEBUG) #0: Thu Oct 29 08:21:11 CET 2015
r...@redgate.red.aixigo.de:/usr/src/sys/arch/amd64/compile/GENERIC.DEBUG
real mem = 4161720320 (3968MB)
avail mem = 4031696896 (3844MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xec1e0 (76 entries)
bios0: vendor American Megatrends Inc. version "1.00" date 08/21/2014
bios0: Shuttle Inc. DS87D
acpi0 at bios0: rev 2
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP APIC FPDT SLIC SSDT SSDT MCFG HPET SSDT SSDT
acpi0: wakeup devices PXSX(S4) RP01(S4) PXSX(S4) RP02(S4) PXSX(S4) RP03(S4)
PXSX(S4) RP04(S4) PXSX(S4) RP05(S4) PXSX(S4) RP06(S4) PXSX(S4) RP07(S4)
PXSX(S4) RP08(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Core(TM) i3-4150 CPU @ 3.50GHz, 3492.38 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MC
MacBook2,1 resuming
Previously, resuming my old amd64/current MacBook2,1 (dmesg below) resulted in a garbled screen. I just want to report that this problem has disappeared; apparently, the recent inteldrm thing got improved. Resume from X now works: I get my screen back and everything just works, including the resumed wifi in the middle of this very sentence. Thank you. However, there is a problem when suspending form the console. The machine does resume back, but the keyboard is unusable: only garbage comes out, whatever I type. In particular, I cannot ctrl+alt+f5 to get back to X, so I reboot with the power button. The /var/log/messages of the session is below. Is this related to all the USB stuff being detached and re-attached? Nov 11 14:04:05 mac /bsd: ukbd0: was console keyboard Jan OpenBSD 5.8-current (GENERIC.MP) #1579: Mon Nov 9 11:04:11 MST 2015 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 3171901440 (3024MB) avail mem = 3071705088 (2929MB) mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xe (37 entries) bios0: vendor Apple Inc. version "MB21.88Z.00A5.B07.0706270922" date 06/27/07 bios0: Apple Inc. MacBook2,1 acpi0 at bios0: rev 2 acpi0: sleep states S0 S3 S4 S5 acpi0: tables DSDT FACP HPET APIC MCFG ASF! SBST ECDT SSDT SSDT SSDT acpi0: wakeup devices ADP1(S3) LID0(S3) PXS1(S4) PXS2(S4) USB1(S3) USB2(S3) USB3(S3) USB4(S3) USB7(S3) EC__(S3) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpihpet0 at acpi0: 14318179 Hz acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Core(TM)2 CPU T7400 @ 2.16GHz, 2161.56 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,NXE,LONG,LAHF,PERF,SENSOR cpu0: 4MB 64b/line 16-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 166MHz cpu0: mwait min=64, max=64, C-substates=0.2.2.2.2, IBE cpu1 at mainbus0: apid 1 (application processor) cpu1: Intel(R) Core(TM)2 CPU T7400 @ 2.16GHz, 2161.25 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,NXE,LONG,LAHF,PERF,SENSOR cpu1: 4MB 64b/line 16-way L2 cache cpu1: smt 0, core 1, package 0 ioapic0 at mainbus0: apid 1 pa 0xfec0, version 20, 24 pins ioapic0: misconfigured as apic 0, remapped to apid 1 acpimcfg0 at acpi0 addr 0xf000, bus 0-255 acpiec0 at acpi0 acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 1 (RP01) acpiprt2 at acpi0: bus 2 (RP02) acpiprt3 at acpi0: bus 3 (PCIB) acpicpu0 at acpi0: !C3(100@55 mwait@0x31), !C2(500@1 mwait@0x10), C1(1000@1 mwait), PSS acpicpu1 at acpi0: !C3(100@55 mwait@0x31), !C2(500@1 mwait@0x10), C1(1000@1 mwait), PSS acpiac0 at acpi0: AC unit online acpibtn0 at acpi0: LID0 acpibtn1 at acpi0: PWRB acpibtn2 at acpi0: SLPB acpibat0 at acpi0: BAT0 model "15253732082930497" type 15253732284385612 oem "15253732284387396" acpivideo0 at acpi0: GFX0 cpu0: Enhanced SpeedStep 2161 MHz: speeds: 2167, 2000, 1833, 1667, 1500, 1333, 1000 MHz memory map conflict 0xbef0/0x10 memory map conflict 0xbf00/0x100 memory map conflict 0xf00f8000/0x1000 memory map conflict 0xfed1c000/0x4000 memory map conflict 0xfffb/0x3 pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel 82945GM Host" rev 0x03 inteldrm0 at pci0 dev 2 function 0 "Intel 82945GM Video" rev 0x03 drm0 at inteldrm0 intagp0 at inteldrm0 agp0 at intagp0: aperture at 0xc000, size 0x1000 inteldrm0: apic 1 int 16 inteldrm0: 1280x800 wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation) wsdisplay0: screen 1-5 added (std, vt100 emulation) "Intel 82945GM Video" rev 0x03 at pci0 dev 2 function 1 not configured vendor "Intel", unknown product 0x27a3 (class DASP subclass Time and Frequency, rev 0x03) at pci0 dev 7 function 0 not configured azalia0 at pci0 dev 27 function 0 "Intel 82801GB HD Audio" rev 0x02: msi azalia0: codecs: Sigmatel STAC9220/1 audio0 at azalia0 ppb0 at pci0 dev 28 function 0 "Intel 82801GB PCIE" rev 0x02: msi pci1 at ppb0 bus 1 mskc0 at pci1 dev 0 function 0 "Marvell Yukon 88E8053" rev 0x22, Yukon-2 EC rev. A3 (0x2): apic 1 int 16 msk0 at mskc0 port A: address 00:1b:63:36:2b:5d eephy0 at msk0 phy 0: 88E Gigabit PHY, rev. 2 ppb1 at pci0 dev 28 function 1 "Intel 82801GB PCIE" rev 0x02: msi pci2 at ppb1 bus 2 athn0 at pci2 dev 0 function 0 "Atheros AR5418" rev 0x01: apic 1 int 17 athn0: MAC AR5418 rev 2, RF AR5133 (2T3R), ROM rev 4, address 00:1c:b3:c4:b2:ae uhci0 at pci0 dev 29 function 0 "Intel 82801GB USB" rev 0x02: apic 1 int 21 uhci1 at pci0 dev 29 function 1 "Intel 82801GB USB" rev 0x02: apic 1 int 19 uhci2 at pci0 dev 29 function 2 "Intel 82801GB USB" rev 0x02: apic 1 int 18 uhci3 at pci0
athn0/AR9287. Having to switch PCI-E slot after every reboot. Weird wireless issues!
Hi, Tp-Link TL-WN881ND PCI-E ( Atheros / AR9287 / athn0 ) This adapter works fantastic when it is plugged in to my motherboard, can even set it up with hostap for an access point - great!! ..Until I try to reboot the machine, then it goes crazy and ifconfig says the device doesn't exist etc.. and dmesg is saying not configured! ..But if I place the AR9287 in to a different PCI-E slot and boot up the system, everything is back to normal until I reboot! This happens in a loop, after every reboot I have to open up the computer case and move the wifi adapter in to another PCI-E slot. The man pages state that the AR9287 adapter is supported. Have I stumbled across a bug or am I doing something wrong? Can someone help bring reason or logic to this madness? Here are the outputs: $ifconfig athn0 athn0: no such interface $ifconfig athn0 up ifconfig: SIOCGIFFLAGS: Device not configured. $cat /etc/hostname.athn0 inet 10.0.0.1 hostap mediaopt hostap nwid OpenBSD wpakey MYPASSWORD $dmesg | grep athn0 athn0 at pci1 dev 0 function 0 "Atheros AR9287" rev 0x01: apic 6 int 4 athn0: AR9287 rev 2 (2TR2), ROM rev 4, address c4:a4:42:8d:eb:4d $dmesg | grep Atheros athn0 at pci1 dev 0 function 0 "Atheros AR9287" rev 0x01: apic 6 int 4 vendor "Atheros", unknown product 0xff1c (class network subclass ethernet, rev 0 x01) at pci1 dev 0 function 0 not configured -=-=-=-=-=-= AFTER switching the PCI-E slot and powering up again! Everything just works the way it should: -=-=-=-=-=-= $ifconfig athn0 athn0: flags=8843 mtu 1500 lladdr c4:a4:42:8d:eb:4d priority: 4 groups: wlan media: IEEE802.11 autoselect hostap status: active i80211: nwid OpenBSD chan 1 bssid c4:a4:42:8d:eb:4d wpakey 0xf53a736 273ab364c83b836e73273ab364c83b836e73273ab36 wpaprotos wpa1, wpa2 wpaakms psk wpaciphers tkip,ccmp wpagroupcipher tkip inet 10.0.0.1 netmask 0xff00 broadcast 10.0.0.255 $dmesg | grep athn0 athn0 at pci1 dev 0 function 0 "Atheros AR9287" rev 0x01: apic 6 int 0 athn0: AR9287 rev 2 (2TR2), ROM rev 4, address c4:a4:42:8d:eb:4d $dmesg | grep Atheros athn0 at pci1 dev 0 function 0 "Atheros AR9287" rev 0x01: apic 6 int 0
Re: em(4) watchdog timeouts
I've done some further testing and I think I've narrowed it down to the "Unlocking em(4) a bit further"-patch [0]. With the patch reverted, I haven't seen any watchdog timeouts yet. I'm currently running the router with the patch reverted to make sure the timeouts don't happen again. [0]: https://www.marc.info/?l=openbsd-tech&m=144347723907388&w=4 -- Gregor
Re: pfctl -f /etc/pf.conf fails on boot when DNS-resolved symbolic names are used
Em 11-11-2015 00:06, Nick Holland escreveu: > The point is...if you put in a DNS name, odds are you are going to end > up thinking you are blocking/passing/redirecting a DNS name..when in > reality, you are whatevering JUST the IP address that it resolves to at > the time the firewall rules were loaded. You may have missed a lot, or > it may move. > > IF you are really in a situation where the only things you are trying to > manage with DNS names are simple 1:1 name:ip mappings, an easy solution > would be to have your pf.conf file a "stub" with enough to let the > system come up, then a post boot and periodic (re)load of the "real" > rules in a separate file. I tried to help the OP by suggesting he use macros or anchors; I'd like to take it back. Don't ever use dns names on pf.conf. The only safe way to properly deal with this is using a proxy. Relayd can work quite well for simple cases. Cheers, Giancarlo Razzolini
Re: athn0/AR9287. Having to switch PCI-E slot after every reboot. Weird wireless issues!
On Wed, Nov 11, 2015 at 08:58:37AM -0500, szs wrote: > $dmesg | grep Atheros > athn0 at pci1 dev 0 function 0 "Atheros AR9287" rev 0x01: apic 6 int 4 > vendor "Atheros", unknown product 0xff1c (class network subclass ethernet, > rev 0 > x01) at pci1 dev 0 function 0 not configured > > -=-=-=-=-=-= > AFTER switching the PCI-E slot and powering up again! Everything just works > the way it should: Known issue. Some analysis was done here: http://marc.info/?l=openbsd-tech&m=140307967032079&w=2 and here: http://marc.info/?l=openbsd-tech&m=140308213300749&w=2 Another report: http://marc.info/?l=openbsd-misc&m=143138337405221&w=2
EFI: Booting from other (not the first) GPT partition possible? How? It's an Apple :-O
Hello! My computer is a MacBook Pro 8,2. There is a GPT on the HD (big surprise!) with four partitions, the last one being of type OpenBSD. I managed to put a recent OpenBSD 5.8 snapshot there by booting and installing from an USB stick via EFI created like that (in OSX): dd if=~/install58.fs of=/dev/rdisk2 bs=1m After installing rEFInd 0.9.2 and putting OpenBSD 5.8 snapshot's BOOTX64.EFI file to the MacBook's EFI partition the rEFInd boot manager shows the OpenBSD EFI option. Selecting that OpenBSD entry starts the boot programm showing hd0 hd1 hd2 and hd3. Is it possible to boot my "EFI OpenBSD installation" from here? If so, how to proceed? I already played with set device hd0d etc. - but it did not work. I will gladly share more details, if of any help. Thanks in advance! Marcel
GUI is for wimps second the currently opinion of hardcore OpenBSD user community?
Is good idea to create a user-friendly and easy-to-use variant of OpenBSD second the hardcore OpenBSD user community? If no, because? GUI is for wimps second the currently opinion of hardcore OpenBSD user community? If yes because? -- View this message in context: http://openbsd-archive.7691.n7.nabble.com/GUI-is-for-wimps-second-the-currently-opinion-of-hardcore-OpenBSD-user-community-tp282595.html Sent from the openbsd user - misc mailing list archive at Nabble.com.
Re: [OpenBSD and GUIs]
français wrote: > Is good idea to create a user-friendly and easy-to-use variant of > OpenBSD second the hardcore OpenBSD user community? > > If no, because? My opinion: this would be useful to a lot of people, but don't expect help from upstream. Things like the lack of Linux capabilities would make it harder to gracefully manage WiFi etc. I'm not sure whether these are solved problems. We already have stable GNOME and XFCE ports, so there's an easy entry point. Alternatively, you could make a custom environment like Crunchbang. As little modification as possible (maybe just a shell script run after install?) would be ideal. > GUI is for wimps second the currently opinion of hardcore OpenBSD user > community? > > If yes because? I think most people use some kind of GUI, albeit usually minimalist ones.
Re: iked & nat-t problem (no keep alive)
From: jtub...@gmail.com [mailto:jtub...@gmail.com] On Behalf Of Jason Tubnor Sent: Tuesday, November 03, 2015 1:44 AM To: igyht Cc: OPENBSD forum Subject: Re: iked & nat-t problem (no keep alive) On 19 October 2015 at 21:49, igyht wrote: I am testing iked on OpenBSD phobos 5.7 GENERIC#738 i386, I think there is keep-alive problem when use with NAT-T, detailed configurations are: http://daemonforums.org/showthread.php?t=9446 I think, iked & nat-t need some kind of "keep alive", but I can't find it in iked.conf configuration. Any idea? Keep alive is not really the best way to do it and from memory, the RFC states it shouldn't be done (though I think the NAT-T RFC stated it could, but might be getting a whole lot of them mixed up). To keep your link alive, it is easier to be done at the firewall level. Depending on your firewall or NAT device you can configure to use what is known as 'static-port'. Good firewalls like PF use upper port ransomisation for NAT connections, as you have found out, the state expires and when you try to send data again, the firewall sets up a new UDP 4500 link but return responses will be back to the original client port, not the new one that is setup - thus dropped packets and you need to re-initialise the link. The RFC states that the client (active) is not allowed to send the server (passive) the new return port to prevent DDoS occurring on the server. The best way I have found to solve this problem is configuring a match rule specifically for NAT outbound to remote [any] 4500 static-port (see man 5 pf.conf), leaving good port ransomisation in place for all other traffic. Seems the network admins leave static-port matching on on Cisco routers so I haven't had any issues when connecting through these. Cheers, Jason. -- "If my calculations are correct, when this baby hits 88MPH, you're gonna to see some serious shit" - Emmett "Doc" Brown Yes, I understand your idea, but âroad warrirorâ ie âactive ike deviceâ is not behind pf. Roadwarriror is not behind MY firewall at all, It is somewhere out there, behind some other company/hotel/ISP firewall. Igor
cron log in /var/log
As cron got a quite interested recently, isn't right time to move its log to /var/log? Or does having /var/cron/log have any specific reason? j.
Re: Will mmap and the read buffer cache be unified, anyone working with it?
On 2015-11-10 14:10, Tinker wrote: ... A "safe" approach to file access would be to read data using mmap() but write data using fwrite() only. Mmap does have a read-only mode. This does NOT work in OpenBSD currently though because of the absence of unified caching. I find this conversation puzzling, since even back in BSD 4.3, read() was actually implemented by memory mapping the underlying file. The nice thing about reading files from memory using mmap instead of using fread(), is that you offload Lots of work to the OS kernel. Suddenly file reading is free of mallocs for instance. And the program doesn't need internal file caching, so the extent of the OS' disk caching is increased. And I guess maybe the OS disk cache can prioritize better what to keep in RAM. In a way, mmap() is a way to "zero-copy file access", which is just awesome. A database that uses this technique is LMDB (OpenLDAP's default DB backend). A key feature of LMDB is that it's only 9600 locs, https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblmdb/mdb.c : LMDB is interesting in how lowlevel it is, as it's written in C and the "loaded" database entries are simply pointers into the mmap():ed space. I saw some fantastic-looking benchmarks, I think at http://symas.com/mdb/#bench , where LMDB goes light-speed where others remain on the ground. (LMDB has a limit in usability in that it never shrinks a DB file, however, that is in no way because of its use of mmap() and could really be overcome by working more on it. That is not a limit in usability. Any active database undergoes insert and delete operations; returning space to the OS on a delete would be foolish since the DB will just request the space back again on the next insert operation. High performance malloc libraries generally operate the same way - once they have acquired memory from the OS they seldom/never return it. There's no good reason to incur the cost of allocation more than once. Also, LMDB serializes its DB writes, which also is an architecture decision specific to it and which has severe performance implications - and that is unspecific to mmap() also, and could be overcome.) LMDB's write performance is pretty mediocre, by design - we emphasized durability/reliability over performance here. But in practice, it is always faster than e.g. BerkeleyDB, which supports multiple concurrent writers. With multiple writer concurrency, we found that BDB spends much of its time in contended locks and deadlock resolution. In most applications, lock acquisition/release, deadlock detection, and resolution will consume a huge amount of CPU time, completely erasing any potential throughput gains from allowing multiple concurrent writers. If you want to do writes thru mmap() then you need to be extremely careful, so yes, how LMDB does writes is actually highly specific to its use of mmap. Transactional integrity requires that certain writes are persisted to disk in a particular order, otherwise you get corrupted data structures. You can use mlock() to prevent pages from being flushed before you intend, but then you're invoking a number of system calls per write, and so you haven't gained anything in the performance department. Or you can do what LMDB does, and write arbitrarily to the map until the end of the transaction (using no syscalls), and then do carefully sequenced final updates and msyncs. Note that LMDB works perfectly well on OpenBSD even without a unified buffer cache; it just requires you to perform writes thru the mmap as well as reads to sidestep the cache coherency issue. (Of course, using a writable mmap means you lose LMDB's default immunity to stray writes thru wild pointers.) -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
OpenMP
Is there any implementation of the OpenMP API contained in one of the GNU compiler packages?
Re: Will mmap and the read buffer cache be unified, anyone working with it?
> I find this conversation puzzling, since even back in BSD 4.3, read() was > actually implemented by memory mapping the underlying file. That is an wildly incorrect description of the 4.3 BSD buffer cache, furthermore 4.3 lacked a working mmap() to hit the coherency issue.
Re: Will mmap and the read buffer cache be unified, anyone working with it?
On 2015-11-12 01:30, Theo de Raadt wrote: I find this conversation puzzling, since even back in BSD 4.3, read() was actually implemented by memory mapping the underlying file. That is an wildly incorrect description of the 4.3 BSD buffer cache, furthermore 4.3 lacked a working mmap() to hit the coherency issue. Wait, so if you make a read-only mmap(), will the buffer in OpenBSD be "unified" in the sense that fwrite():s always will be immediately visible in the memory mapped version? (This is what I understood that Howard suggested would apply from BSD 4.3 and on and Theo now said not was correct, so, my question is if that behavior was introduced later in OpenBSD.)
Re: Will mmap and the read buffer cache be unified, anyone working with it?
... LMDB's write performance is pretty mediocre, by design - we emphasized durability/reliability over performance here. But in practice, it is always faster than e.g. BerkeleyDB, which supports multiple concurrent writers. With multiple writer concurrency, we found that BDB spends much of its time in contended locks and deadlock resolution. In most applications, lock acquisition/release, deadlock detection, and resolution will consume a huge amount of CPU time, completely erasing any potential throughput gains from allowing multiple concurrent writers. If you want to do writes thru mmap() then you need to be extremely careful, so yes, how LMDB does writes is actually highly specific to its use of mmap. Transactional integrity requires that certain writes are persisted to disk in a particular order, otherwise you get corrupted data structures. You can use mlock() to prevent pages from being flushed before you intend, but then you're invoking a number of system calls per write, and so you haven't gained anything in the performance department. Or you can do what LMDB does, and write arbitrarily to the map until the end of the transaction (using no syscalls), and then do carefully sequenced final updates and msyncs. Note that LMDB works perfectly well on OpenBSD even without a unified buffer cache; it just requires you to perform writes thru the mmap as well as reads to sidestep the cache coherency issue. (Of course, using a writable mmap means you lose LMDB's default immunity to stray writes thru wild pointers.) But LMDB doesn't even compile on OpenBSD in mmap mode, does it, or did you fix it last months? Having some kind of vacuum/GC feature on the database file would be nice, even if it's only rare and sporadic. I am principally against database files that are hardcoded to always be ever-growing. Am aware that some kind of "hotswap" logic can be implemented, maybe quite easily, atop LMDB though. Perhaps such a "hotswap-vacuum" wrapper could be provided as a standard wrapper atop LMDB just because it is so useful to everyone though. *A lot* would be needed for me to accept an ever-growing database file.
Re: Will mmap and the read buffer cache be unified, anyone working with it?
> On 2015-11-12 01:30, Theo de Raadt wrote: > >> I find this conversation puzzling, since even back in BSD 4.3, read() > >> was > >> actually implemented by memory mapping the underlying file. > > > > That is an wildly incorrect description of the 4.3 BSD buffer cache, > > furthermore 4.3 lacked a working mmap() to hit the coherency issue. > > Wait, so if you make a read-only mmap(), will the buffer in OpenBSD be > "unified" in the sense that fwrite():s always will be immediately > visible in the memory mapped version? My reply is specifically about what Howard said. > (This is what I understood that Howard suggested would apply from BSD > 4.3 and on and Theo now said not was correct, so, my question is if that > behavior was introduced later in OpenBSD.) I made none of the claims you are composing.
Re: OpenMP
On Wed, Nov 11, 2015 at 05:11:52PM +0100, Jens A. Griepentrog wrote: > Is there any implementation of the OpenMP API > contained in one of the GNU compiler packages? > A working patch was discussed here: https://marc.info/?t=14332549291&r=1&w=2 but as far as I know nothing has been committed, yet.
Re: cron log in /var/log
On Wed, 11 Nov 2015 12:29:30 -0500, Jiri B wrote: > As cron got a quite interested recently, isn't > right time to move its log to /var/log? > Or does having /var/cron/log have any specific reason? Since it is just another syslog file /var/log makes sense. I worry a bit about people's log watching scripts, though. - todd
Re: cron log in /var/log
On Wed, Nov 11, 2015 at 10:47:00AM -0700, Todd C. Miller wrote: > On Wed, 11 Nov 2015 12:29:30 -0500, Jiri B wrote: > > > As cron got a quite interested recently, isn't > > right time to move its log to /var/log? > > Or does having /var/cron/log have any specific reason? > > Since it is just another syslog file /var/log makes sense. > I worry a bit about people's log watching scripts, though. Other thing, when I was playing with most filesystems r/o I also found having '.sock' in /var/cron/tabs little annoying, as we usually use /var/run and I was already having /var/run as mfs. Since like piece of cake to move it to /var/run. j.
Re: Will mmap and the read buffer cache be unified, anyone working with it?
On 2015-11-12 01:42, Theo de Raadt wrote: On 2015-11-12 01:30, Theo de Raadt wrote: >> I find this conversation puzzling, since even back in BSD 4.3, read() >> was >> actually implemented by memory mapping the underlying file. > > That is an wildly incorrect description of the 4.3 BSD buffer cache, > furthermore 4.3 lacked a working mmap() to hit the coherency issue. Wait, so if you make a read-only mmap(), will the buffer in OpenBSD be "unified" in the sense that fwrite():s always will be immediately visible in the memory mapped version? My reply is specifically about what Howard said. (This is what I understood that Howard suggested would apply from BSD 4.3 and on and Theo now said not was correct, so, my question is if that behavior was introduced later in OpenBSD.) I made none of the claims you are composing. Aha noted, so unifying the buffer cache then is still altogether in the future. Thanks for clarifyign.
Re: cron log in /var/log
On Wed, 11 Nov 2015 12:52:51 -0500, Jiri B wrote: > Other thing, when I was playing with most filesystems r/o I also > found having '.sock' in /var/cron/tabs little annoying, > as we usually use /var/run and I was already having /var/run > as mfs. Since like piece of cake to move it to /var/run. Funny you should mention that. I was considering moving that to /var/run/cron.sock. The only reason for it to be in the cron dir is for older systems that don't respect the file modes on Unix domain sockets. That's not an issue for us... - todd
Re: GUI is for wimps second the currently opinion of hardcore OpenBSD user community?
> Is good idea to create a [...] variant of OpenBSD No. If anything can be improved, submit patches.
cron daily insecurity output
Hi misc@ cron started to be recently reported in my insecurity output after upgrading to snapshot from Nov 6: Checking special files and directories. Output format is: filename: criteria (shouldbe, reallyis) var/cron/atjobs: permissions (01770, 0770) var/cron/tabs: permissions (01730, 0730) mtree special: exit code 2 Last known snapshot known to not complain about those issues was from Oct 7th. Reports started on the snapshot upgrade & continue till now. Did anyone else notice this? Regards, Adam
Re: em(4) watchdog timeouts
Hi Alexis, On Wed, Nov 11, 2015 at 08:11:15PM +, Alexis VACHETTE wrote: > [...] > Even with heavy network load ? > [...] So far, yes. I've saturated the device for about 45 Minutes with something like this (the other end is my laptop): ## on the router $ dd if=/dev/zero bs=8k | nc 172.31.64.174 55000 ## on my laptop $ nc -l 55000 | dd of=/dev/null bs=8k (with two or three streams in parallel). There were about 6k interrupts per second and bandwidth was about 250Mbps, which seems to be the maximum the tiny CPU in this router can do. No watchdog timeouts appeared, where previously something relatively low bandwidth (the SSDs in router and laptop suck) like this caused one every 20 or 30 seconds: ## on the router $ pax -w /home | nc 172.31.64.174 55000 I'll keep an eye on things, but so far it looks good. Regular usage works out so far as well. If you need me to run some special workload for you, I'd be more than happy to do that. -- Gregor
UEFI boot-looping on Asus M5A97 LE R2.0 motherboard
Hello, I recently installed a UEFI-capable Asus M5A97 LE R2.0 motherboard in one of my systems and tried to boot the November 11th amd64 miniroot58.fs image to test UEFI booting. I get to the bootloader, but it appears to fail while loading the kernel and goes into a reboot loop. Here's everything I see on screen before it reboots: probing: pc0 mem[640K 2984M 4M 48K 5103M] disk: hd0 hd1 hd2* >> OpenBSD/amd64 EFIBOOT 3.29 boot> cannot open hd0a:/etc/random.seed: No such file or directory booting hd0a:/bsd:3273216+1394144+2409472+0+569344=0x74d238 This is with the latest available UEFI version for this board (version 2601). All BIOS/UEFI options are default values except for setting the Secure Boot option to "Other OS". I am able to boot normally in BIOS mode. Full dmesg of a regular BIOS-mode install follows at the end of this email. I understand that UEFI support is still a work in progress, so if there's any way I can provide further info or test new code, please let me know. OpenBSD 5.8-current (GENERIC.MP) #1591: Wed Nov 11 09:38:33 MST 2015 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 8470441984 (8078MB) avail mem = 8209600512 (7829MB) mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xbc41b018 (58 entries) bios0: vendor American Megatrends Inc. version "2601" date 03/24/2015 bios0: ASUSTeK COMPUTER INC. M5A97 LE R2.0 acpi0 at bios0: rev 2 acpi0: sleep states S0 S3 S4 S5 acpi0: tables DSDT FACP APIC FPDT MCFG HPET SSDT BGRT acpi0: wakeup devices SBAZ(S4) PS2K(S3) PS2M(S3) UAR1(S4) P0PC(S4) UHC1(S4) UHC2(S4) UHC4(S4) UHC6(S4) UHC7(S4) PC02(S4) PC03(S4) PC04(S4) PC05(S4) PC06(S4) PC07(S4) [...] acpitimer0 at acpi0: 3579545 Hz, 32 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 16 (boot processor) cpu0: AMD Opteron(tm) Processor 3350 HE, 2809.75 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,POPCN T,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR 8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,XOP,SKINIT,WDT,FMA4,NODEID,TBM,TOPEXT,ITSC, BMI1 cpu0: 64KB 64b/line 2-way I-cache, 16KB 64b/line 4-way D-cache, 2MB 64b/line 16-way L2 cache, 8MB 64b/line 64-way L3 cache cpu0: ITLB 48 4KB entries fully associative, 24 4MB entries fully associative cpu0: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 200MHz cpu0: mwait min=64, max=64, IBE cpu1 at mainbus0: apid 17 (application processor) cpu1: AMD Opteron(tm) Processor 3350 HE, 2809.37 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,POPCN T,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR 8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,XOP,SKINIT,WDT,FMA4,NODEID,TBM,TOPEXT,ITSC, BMI1 cpu1: 64KB 64b/line 2-way I-cache, 16KB 64b/line 4-way D-cache, 2MB 64b/line 16-way L2 cache, 8MB 64b/line 64-way L3 cache cpu1: ITLB 48 4KB entries fully associative, 24 4MB entries fully associative cpu1: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 18 (application processor) cpu2: AMD Opteron(tm) Processor 3350 HE, 2809.37 MHz cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,POPCN T,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR 8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,XOP,SKINIT,WDT,FMA4,NODEID,TBM,TOPEXT,ITSC, BMI1 cpu2: 64KB 64b/line 2-way I-cache, 16KB 64b/line 4-way D-cache, 2MB 64b/line 16-way L2 cache, 8MB 64b/line 64-way L3 cache cpu2: ITLB 48 4KB entries fully associative, 24 4MB entries fully associative cpu2: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative cpu2: smt 0, core 2, package 0 cpu3 at mainbus0: apid 19 (application processor) cpu3: AMD Opteron(tm) Processor 3350 HE, 2809.37 MHz cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,POPCN T,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR 8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,XOP,SKINIT,WDT,FMA4,NODEID,TBM,TOPEXT,ITSC, BMI1 cpu3: 64KB 64b/line 2-way I-cache, 16KB 64b/line 4-way D-cache, 2MB 64b/line 16-way L2 cache, 8MB 64b/line 64-way L3 cache cpu3: ITLB 48 4KB entries fully associative, 24 4MB entries fully associative cpu3: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative cpu3: smt 0, core 3, package 0 ioapic0 at mainbus0: apid 5 pa 0xfec0, version 21, 24 pins ioapic1 at mainbus0: apid 6 pa 0xfec2, version 21, 32 pi
Re: em(4) watchdog timeouts
Hi Gregor, Even with heavy network load ? Regards, Alexis. De : owner-t...@openbsd.org de la part de Gregor Best Envoyé : mercredi 11 novembre 2015 15:20 À : Mark Kettenis Cc : t...@openbsd.org; misc@openbsd.org Objet : Re: em(4) watchdog timeouts I've done some further testing and I think I've narrowed it down to the "Unlocking em(4) a bit further"-patch [0]. With the patch reverted, I haven't seen any watchdog timeouts yet. I'm currently running the router with the patch reverted to make sure the timeouts don't happen again. [0]: https://www.marc.info/?l=openbsd-tech&m=144347723907388&w=4 -- Gregor
Re: cron daily insecurity output
On Wed, 11 Nov 2015 20:31:03 +0100, Adam Wolk wrote: > cron started to be recently reported in my insecurity output after > upgrading to snapshot from Nov 6: > > Checking special files and directories. > Output format is: > filename: > criteria (shouldbe, reallyis) > var/cron/atjobs: > permissions (01770, 0770) > var/cron/tabs: > permissions (01730, 0730) > mtree special: exit code 2 This is a side effect of pledge(2) restrictions in cron coupled with a minor bug in the code that caused it to change the mode when it doesn't actually need to. I committed a fix for the bug earlier today so the next snapshot containing that fix will not strip the sticky bit from those directories. However, you'll need to fix up the directory permissions manuall. E.g. # chmod chmod a+t /var/cron/atjobs /var/cron/tabs - todd
