Re: OpenBSD (memory management) performance issues
> On Wed, Jan 27, 2021 at 08:39:46AM +0100, Otto Moerbeek wrote:
>
> > On Tue, Jan 26, 2021 at 04:08:40PM +, Marek Klein wrote:
> >
> > > Hi,
> > >
> > > We are working on an appliance like product that is based on OpenBSD.
> > > Recently we found out that our performance critical C++ program is
> > > ~2.5 times slower on OpenBSD compared to Ubuntu 20.04.
> > >
> > > The program basically just reads data from stdin, does some
> > > transformation of the data, and returns the result on stdout, thus
> > > the program does not perform any further I/O operations nor interacts
> > > with other programs. We extensively use the C++ standard library string
> > > class for manipulation of data.
> > >
> > > We started searching for the reason, and eliminated I/O as a factor.
> > > During some experiments we found out that one, perhaps not the only
> > > one, factor is OpenBSD's memory management. To test this assumption we
> > > wrote a simple program that allocates and frees memory in a loop.
> > > Something like:
> > >
> > > for (...) {
> > > void *buffer = malloc(...);
> > > ...
> > > free(buffer);
> > > }
> > >
> > > We compiled it on OpenBSD with clang
> > > $ /usr/bin/c++ --version
> > > OpenBSD clang version 10.0.1
> > > Target: amd64-unknown-openbsd6.8
> > > Thread model: posix
> > > InstalledDir: /usr/bin
> > >
> > > using options '-O3 -DNDEBUG -std=gnu++11' and ran it without memory
> > > junking.
> > >
> > > $ time MALLOC_OPTIONS=jj ./memory_allocs --cycles 123456789 --size
> 1024
> > >
> > > real 0m27.218s
> > > user 0m27.220s
> > > sys 0m0.020s
> > >
> > > We compiled the same program on Ubuntu 20.04 with g++
> > > $ /usr/bin/c++ --version
> > > c++ (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
> > >
> > > using the same options '-O3 -DNDEBUG -std=gnu++11'
> > >
> > > $ time ./memory_allocs --cycles 123456789 --size 1024
> > >
> > > real 0m1,920s
> > > user 0m1,915s
> > > sys 0m0,004s
> > >
> > > Both systems were tested in the same virtualized environment (VSphere),
> > > thus we can assume the "hardware" is the same.
> > >
> > > Given the virtual environment, the tests might not be scientifically
> > > the best choice, but they serve the observation well enough. We
> > > actually ruled out virtualization as a cause in other tests.
> >
> > Short story: the slowness is because you get more security.
> >
> > Somewhat longer story: depending on the size if the allocation actual
> > unmaps take place on free. This will catch use-after-free always. For
> > smaller allocations, caching takes place, sadly you did not tell us
> > how big the total of your allocations are. So I cannot predict if
> > enlargering the cache will help you.
> >
> > Now the differnence is quite big so I like to know what you are doing
> > exactly in your test program. Please provide the full test porogram
> > so I can take a look.
> >
> > >
> > > What other options are there we could try in order to speed the memory
> > > management up?
> >
> > Some hintss: allocate/free less, use better algorithms that do not
> > allocate as much. With C++ make sure your code uses moves of objects
> > instead of copies whenever possible. Use reserve() wisely. If all else
> > fails you might go for custom allocaters, but you will loose security
> > features.
> >
> > -Otto
> >
> > >
> > > Also are there any other known areas, for CPU bound processing, where
> > > OpenBSD performs worse than other "common" platforms?
> > >
> > > Cheers,
> > > Marek
> > >
> >
>
> To reply to myself.
>
> Be VERY careful when drawing conclusions from these kinds of test
> programs. To demonstrate, the loop in the test program below gets
> compiled out by some compilers with some settings.
>
> So again, please provide your test program.
>
> -Otto
>
> #include
> #include
> #include
> #include
>
> int
> main(int argc, char *argv[])
> {
> size_t count, sz, i;
> char *p;
> const char *errstr;
>
> count = strtonum(argv[1], 0, LONG_MAX, &errstr);
> if (errstr)
> errx(1, "%s: %s", argv[1], errstr);
> sz = strtonum(argv[2], 0, LONG_MAX, &errstr);
> if (errstr)
> errx(1, "%s: %s", argv[2], errstr);
>
> printf("Run with %zu %zu\n", count, sz);
>
> for (i = 0; i < count; i++) {
> p = malloc(sz);
> if (p == NULL)
> err(1, NULL);
> *p = 1;
> free(p);
> }
> }
>
>
Hi Otto,
My test program does something very similar.
As stated before I compile with
1. OpenBSD: clang version 10.0.1 and
2. Ubuntu: g++ (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
with the same options '-O3 -DNDEBUG -std=gnu++11'.
The execution time grows with the number of cycles and also with
the size of allocated memory on both platforms, thus I think the loop
is not optimized out.
OpenBSD needs consistently ~10x longer to finish the test compared to
Ubuntu. Regarding the size of allocations, we operat
HID reference doc - broken URL
URL seems broken, patch attached. Index: uhid.c === RCS file: /cvs/src/sys/dev/usb/uhid.c,v retrieving revision 1.82 diff -u -p -u -p -r1.82 uhid.c --- uhid.c 23 Jan 2021 05:08:36 - 1.82 +++ uhid.c 29 Jan 2021 12:15:58 - @@ -32,7 +32,7 @@ */ /* - * HID spec: http://www.usb.org/developers/devclass_docs/HID1_11.pdf + * HID spec: http://www.usb.org/sites/default/files/hid1_11.pdf */ #include "fido.h" -- Alessandro
Re: www.stare.cz daily insecurity output
This is an output from the nightly security(8). This can only mean my /var/backup storage is corrupted, right? The /var/backups/var_yp_Makefile.yp.current is apparently subtly corrupted at places. (I don't use YP at all. Is it still a thing?) It's an old ALIX living on an old CF card: $ dmesg | grep wd0 wd0 at pciide0 channel 0 drive 0: wd0: 1-sector PIO, LBA, 7279MB, 14909328 sectors wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2 root on wd0a (bf940e6c7aaf2c50.a) swap on wd0b dump on wd0b Jan On Jan 29 01:45:51, r...@stare.cz wrote: > > Running security(8): > > == > /var/yp/Makefile.yp diffs (-OLD +NEW) > == > --- /var/backups/var_yp_Makefile.yp.current Sun Nov 24 02:25:02 2019 > +++ /var/yp/Makefile.yp Thu Jan 14 18:53:28 2021 > @@ -297,16 +297,17 @@ > > aliases.time: $(DIR)/mail/aliases > -@if [ -f $(>) ]; then \ > - $(BP) -p $(>) $(DOMAIN)-aliases; \ > - $ SENDMAIL) -bi -oA$(PWD)/$(DOMAIN)-aliases; \ > - $(MAKEDBM) mU $(DOMAIN)-aliases | $(MAKEDM) - mail.aliases; \ > + $(CP) -p $(>) $(DOMAIN)-aliases; \ > + $(SENDMAIL) -bi -oA$(PWD)/$(DOMAIN)-aliases; \ > + $(MAKEDBM) -U $(DOMAIN)-aliases | $(MAKEDBM) - mail.aliases; \ > $(MKALIAS) mail.aliases mail.byaddr; \ > $(TOUCH) $(@); \ > - $(RM) &(DOMAIN)-aliasesdb $(DOMAIN)-almases; \ > - $(ECHG) "updated aliases"; \ if [ ! $(NOPUSH) ]; > then \ > + $(RM) $(DOMAIN)-aliases.db $(DOMAIN)-aliases; \ > + $(ECHO) "updated aliases"; \ > + if [ ! $(NOPUSH) ]; then \ > $(YPPUSH) -d $(DOMAIN) mail.aliases; \ > $(YPPUSH) -d $(DOMAIN) mail.byaddr; \ > - �$(ECHO) "pushed aliase�"; \ > + $(ECHO) "pushed aliases"; \ > else \ > : ; \ > fi \ > >
Re: Managed to mess up the system encrypted disk. I can no longer boot.
On Thu, Jan 28, 2021 at 10:27:47AM +0200, Samarul Meu wrote: Thank you so much! You made my day! So I used FuguIta (6.8 - stable) attached the encrypted partition (accessible as sd1 now) and 'installboot sd1', reboot and surprise - everything is working. I still have no idea why detaching the softraid determined this kind of behavior. Best regards, Samarul P.S. To tetrahedra --- maybe this solution will solve your problem also. Unfortunately not... I will explain what I tried over in that thread.
Re: www.stare.cz daily insecurity output
On 2021-01-29, Jan Stary wrote: > This is an output from the nightly security(8). > This can only mean my /var/backup storage is corrupted, right? Yes those are all bit flips. That card wants replacing and destroying so it doesn't get accidentally used again :) > The /var/backups/var_yp_Makefile.yp.current is apparently > subtly corrupted at places. > > (I don't use YP at all. Is it still a thing?) > > It's an old ALIX living on an old CF card: > > $ dmesg | grep wd0 > wd0 at pciide0 channel 0 drive 0: > wd0: 1-sector PIO, LBA, 7279MB, 14909328 sectors > wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2 > root on wd0a (bf940e6c7aaf2c50.a) swap on wd0b dump on wd0b > > Jan > > > On Jan 29 01:45:51, r...@stare.cz wrote: >> >> Running security(8): >> >> == >> /var/yp/Makefile.yp diffs (-OLD +NEW) >> == >> --- /var/backups/var_yp_Makefile.yp.current Sun Nov 24 02:25:02 2019 >> +++ /var/yp/Makefile.yp Thu Jan 14 18:53:28 2021 >> @@ -297,16 +297,17 @@ >> >> aliases.time: $(DIR)/mail/aliases >> -@if [ -f $(>) ]; then \ >> -$(BP) -p $(>) $(DOMAIN)-aliases; \ >> -$ SENDMAIL) -bi -oA$(PWD)/$(DOMAIN)-aliases; \ >> -$(MAKEDBM) mU $(DOMAIN)-aliases | $(MAKEDM) - mail.aliases; \ >> +$(CP) -p $(>) $(DOMAIN)-aliases; \ >> +$(SENDMAIL) -bi -oA$(PWD)/$(DOMAIN)-aliases; \ >> +$(MAKEDBM) -U $(DOMAIN)-aliases | $(MAKEDBM) - mail.aliases; \ >> $(MKALIAS) mail.aliases mail.byaddr; \ >> $(TOUCH) $(@); \ >> -$(RM) &(DOMAIN)-aliasesdb $(DOMAIN)-almases; \ >> -$(ECHG) "updated aliases"; \ if [ ! $(NOPUSH) ]; >> then \ >> +$(RM) $(DOMAIN)-aliases.db $(DOMAIN)-aliases; \ >> +$(ECHO) "updated aliases"; \ >> +if [ ! $(NOPUSH) ]; then \ >> $(YPPUSH) -d $(DOMAIN) mail.aliases; \ >> $(YPPUSH) -d $(DOMAIN) mail.byaddr; \ >> -�$(ECHO) "pushed aliase�"; \ >> +$(ECHO) "pushed aliases"; \ >> else \ >> : ; \ >> fi \ >> >> > >
Re: HID reference doc - broken URL
On 2021-01-29, Alessandro Ricci wrote: > URL seems broken, patch attached. Thanks, I've fixed that, instances of the same URL in other files in the dir, and the other spec URLs. > Index: uhid.c >=== > RCS file: /cvs/src/sys/dev/usb/uhid.c,v > retrieving revision 1.82 > diff -u -p -u -p -r1.82 uhid.c > --- uhid.c 23 Jan 2021 05:08:36 - 1.82 > +++ uhid.c 29 Jan 2021 12:15:58 - > @@ -32,7 +32,7 @@ > */ > > /* > - * HID spec: http://www.usb.org/developers/devclass_docs/HID1_11.pdf > + * HID spec: http://www.usb.org/sites/default/files/hid1_11.pdf > */ > > #include "fido.h" >
ospf on wg(4)
Hi, I'm trying to setup OSPF on a working Wireguard VPN using 6.8 amd64 machines. This is what I get: # ospfd -dvvv id = "172.26.1.1" startup kr_init: priority filter enabled orig_rtr_lsa: area 0.0.0.0 orig_rtr_lsa: stub net, interface wg0 if_fsm: event UP resulted in action START and changing state for interface wg0 from DOWN to P2P send_packet: error sending packet to 224.0.0.5 on interface wg0: Network is unreachable send_hello: Network is unreachable [...] # ifconfig wg0 wg0: flags=80c3 mtu 1420 index 23 priority 0 llprio 3 wgport 33222 wgpubkey XXX wgpeer YYY wgpka 23 (sec) wgendpoint A.B.C.D 31502 tx: 4317366604, rx: 382870060 last handshake: 47 seconds ago wgaip 192.168.1.0/24 wgaip 172.26.1.3/32 wgpeer WWW wgpka 23 (sec) wgendpoint E.F.G.H 15776 tx: 609183380, rx: 1523684 last handshake: 1 seconds ago wgaip 172.26.0.0/24 wgaip 172.26.1.2/32 groups: wg inet 172.26.1.1 netmask 0xff00 broadcast 172.26.1.255 Is it possible to use a wg(4) interface for ospfd(8)? Thank you, Best. -- Olivier Cherrier Phone: +352691570680 mailto:o...@symacx.com
Re: Bootloader on USB stick fails with "root device not found"
On Tue, Jan 26, 2021 at 03:41:23PM +, tetrahe...@danwin1210.me wrote: Any ideas why the kernel isn't seeing sr0a as a root device? As suggested in the "Managed to mess up the system encrypted disk. I can no longer boot" thread, I tried running installboot from the working (booted from the internal disk bootloader) system against all the relevant disks: - the USB drive - sd0 (the physical disk, with the bootloader) - sd1 (aka sr0 or softraid0, the now-decrypted encrypted volume) Unfortunately this did not fix the problem, I am still geting kernel panics. Looking more closely at the error message: panic: root device (e8) not found the code given (e8) is the disklabel disk ID of the root volume (sd1 / softraid0) as given in /etc/fstab (when the system is booted from the internal bootloader). It's the ID of the (decrypted) softraid volume. Therefore: - when booting from the bootloader on the internal HD, and after decrypting the encrypted volume, the system is able to find the disk e8 without trouble, but - when booting from the bootloader on the USB stick, and after decrypting the same encrypted volume (with the same password, etc), the system is NOT able to find the disk e8. What the heck is going on? Why can't the system find softraid volume that it just decrypted?
Program getting stuck at sqlite3_exec()
Hello! I am porting a stupid program to OpenBSD and found a roadblock. The program is a terminal game launcher. It is intended to serve roguelike games over telnet or SSH.The main project site is http://github.com/paxed/dgamelaunch. So far I have sanitized the autoconf config file to use OpenBSD and replaced the program's crypt() calls with OpenBSD's cryptools alternatives. The problem I am fazing now is that when I attempt to create a new user, the program gets stuck in an sqlite3_exec() call until I hit Enter. This is specially puzzling because the program is not supposed to be accepting user input at that point. And once Enter is hit, the sqlite3_exec() action is actually performed properly(!). If anybody is aware of any reason why sqlite_exec() would get stuck, I'd be glad to hear. How to reproduce (I recommend using a spare system, because this program is kinda intrusive): The program is intended to be suid, chroot into some directory and drop privileges. Prepare a directory for runnign a chroot in it. Download the current code: gopher://gopher.operationalsecurity.es/5/dgamelaunch-openbsd.tar Untar it somewhere. Configure a build environment (export CPATH=/usr/local/include, export LIBRARY_PATH=/usr/lical/lib, etc) Configure the build with ./autogen.sh --enable-sqlite --with-config-file=/path/of/chroot/etc/dgamelaunch.conf Make it with gmake. Edit example/dgamelaunch.conf in the source code folder to suit your needs. For a quick test, you only need to change the chroot=, shed_uid= and shed_guid= options. Edit dgl-create-chroot in the source code folder. The only important value to change is chroot=. Execute the dgl-create-chroot script as root, which will populate the chroot directory you have selected. Set the suid bit of the dgamelaunch binary and execute it. It will chroot into the directory and offer you a bunch of options. Registrate a new user (r) and go with the instructions. The bug is triggered after you enter your email address. The ncurses interface gets frozen and nothing gets written to the database. Then you hit enter while the interface is frozen, and it returns back to normal. I think the problem is triggered in the writefile function in dgamelaunch.c, where sqlite_exec() gets called. Help and suggestions are appreciated. AFAIK the main project is dead. -- OpenPGP Key Fingerprint: BB5A C2A2 2CAD ACB7 D50D C081 1DB9 6FC4 5AB7 92FA
Re: ospf on wg(4)
Le 29/01/2021 à 17:44, Olivier Cherrier a écrit : Hi, I'm trying to setup OSPF on a working Wireguard VPN using 6.8 amd64 machines. This is what I get: # ospfd -dvvv id = "172.26.1.1" startup kr_init: priority filter enabled orig_rtr_lsa: area 0.0.0.0 orig_rtr_lsa: stub net, interface wg0 if_fsm: event UP resulted in action START and changing state for interface wg0 from DOWN to P2P send_packet: error sending packet to 224.0.0.5 on interface wg0: Network is unreachable send_hello: Network is unreachable [...] # ifconfig wg0 wg0: flags=80c3 mtu 1420 index 23 priority 0 llprio 3 wgport 33222 wgpubkey XXX wgpeer YYY wgpka 23 (sec) wgendpoint A.B.C.D 31502 tx: 4317366604, rx: 382870060 last handshake: 47 seconds ago wgaip 192.168.1.0/24 wgaip 172.26.1.3/32 wgpeer WWW wgpka 23 (sec) wgendpoint E.F.G.H 15776 tx: 609183380, rx: 1523684 last handshake: 1 seconds ago wgaip 172.26.0.0/24 wgaip 172.26.1.2/32 groups: wg inet 172.26.1.1 netmask 0xff00 broadcast 172.26.1.255 Is it possible to use a wg(4) interface for ospfd(8)? Thank you, Best. Hello. It is possible, I use it myself. You have to allow multicast address on wg(4) interface(s): 225.0.0.5 for all OSPF routers 224.0.0.6 for all DR/BDR (I use wgaip 0.0.0.0/0, so my config is not relavant for you) Regards, -- Bastien
SIOCSIFPARENT SIOCAIFADDR SIOCSIFFLAGS in bsd.rd
Hi,
I upgraded my APU2 on 2021-01-16 and I have this in the upgrade log
email:
Terminal type? [vt220] vt220
Available disks are: sd0.
Which disk is the root disk? ('?' for details) [sd0] sd0
Checking root filesystem (fsck -fp /dev/sd0a)... OK.
Mounting root filesystem (mount -o ro /dev/sd0a /mnt)... OK.
ifconfig: SIOCSIFPARENT: Invalid argument
ifconfig: SIOCSIFPARENT: Invalid argument
ifconfig: SIOCAIFADDR: Device not configured
ifconfig: SIOCSIFFLAGS: Device not configured
ifconfig: SIOCSIFPARENT: Invalid argument
ifconfig: SIOCAIFADDR: Device not configured
ifconfig: SIOCSIFFLAGS: Device not configured
ifconfig: SIOCSIFPARENT: Invalid argument
ifconfig: SIOCAIFADDR: Device not configured
ifconfig: SIOCSIFFLAGS: Device not configured
ifconfig: SIOCSIFPARENT: Invalid argument
ifconfig: SIOCAIFADDR: Device not configured
ifconfig: SIOCSIFFLAGS: Device not configured
ifconfig: SIOCSIFPARENT: Invalid argument
ifconfig: SIOCAIFADDR: Device not configured
ifconfig: SIOCSIFFLAGS: Device not configured
Force checking of clean non-root filesystems? [no] no
[...]
The upgrade log before (2020-12-10) was just
Terminal type? [vt220] vt220
Available disks are: sd0.
Which disk is the root disk? ('?' for details) [sd0] sd0
Checking root filesystem (fsck -fp /dev/sd0a)... OK.
Mounting root filesystem (mount -o ro /dev/sd0a /mnt)... OK.
Force checking of clean non-root filesystems? [no] no
[...]
I guess this comes from me switching from trunk(4) to aggr(4).
Is it normal/expected?
It doesn't cause me any trouble but I would have expected the same
'behavior' from trunk(4) and aggr(4) in this regard. Or is it to keep
bsd.rd on a diet?
Cheers,
Daniel
Re: Installing across two SSDs, encrypted
On 1/29/21 9:37 PM, Joe Nelson wrote: I'd like to install obsd on a laptop that has one built-in 128GB SSD, and a 1TB SATA SSD added in a separate bay. Was thinking of putting the system files on the small drive, and /home, /var, /tmp, and /usr/local on the big one. I'd like to use full-disk encryption for the big drive. Two questions. First, will the installer set up the partitions across the drives for me, or do I need to do a custom procedure with disklabel? Second, how do I get the OS to prompt me during startup for a passphrase, and mount the encrypted drive? (It's not the primary drive with the OS on it, which seems nonstandard.) Thanks for any help. It takes less than 15 minutes to do an OpenBSD install, far less time than it takes to give detailed answers to questions like this. So, please, do your own testing, and do it over and over until you understand how it works. The installer won't magically read your mind and set things up on multiple drives as you want them, but you can easily guide it along the way, or you can come back and adjust things later as you wish. And really, you should spend an hour or two finding out how this is done -- it's really easy. As for how the secondary encrypted drive is handled, again, JUST TRY IT. If you have questions about what you see and how it happens, THEN come back and ask those specific questions. For what it's worth, 120G is still huge to OpenBSD, so I'd put everything on that, and only use the second drive for whatever it that's so big that you need a second drive (i.e., /var/www, maybe /home/music, etc.). And don't allocate all of that 1T disk unless you actually need most of it. Biggest mistake I see people making is allocating all of their disk, "because I have it, I must allocate it, right?" WRONG. Nick.
Chromium with WebAssembly flavor?
Hi, I wasn't sure if this was worthy of ports or not, so I wanted to throw this out here first. I don't like Zoom, and I understand WebAssembly has some inherent issues in it, but I imagine a good number of people don't have a choice and have to use it as well in light of the current state of things. On top of this, Citrix apps need it enabled as well. Wouldn't it be reasonable to make a flavor with ENABLE_WASM set for chromium then? Compiling chrome manually with the flag is a beast, and my laptop will usually throw a kernel panic before it'll finish compiling, plus as a package it's updated quite regularly which means it needs to be recompiled quite regularly. Best regards, Charlie
Installing across two SSDs, encrypted
I'd like to install obsd on a laptop that has one built-in 128GB SSD, and a 1TB SATA SSD added in a separate bay. Was thinking of putting the system files on the small drive, and /home, /var, /tmp, and /usr/local on the big one. I'd like to use full-disk encryption for the big drive. Two questions. First, will the installer set up the partitions across the drives for me, or do I need to do a custom procedure with disklabel? Second, how do I get the OS to prompt me during startup for a passphrase, and mount the encrypted drive? (It's not the primary drive with the OS on it, which seems nonstandard.) Thanks for any help.
