Re: uvm_obj_wirepages() undocumented?

[Stephan]

As I mentioned here

https://mail-index.netbsd.org/netbsd-users/2023/11/27/msg030366.html

I am implementing some BeOS/Haiku APIs so it should potentially be
possible to get parts of Haiku to run on NetBSD.

There is a facility called Areas which is pretty much a shared memory
system and described here:

https://www.haiku-os.org/legacy-docs/bebook/TheKernelKit_Areas.html

Areas can be locked into RAM. I am creating memory objects using
uao_create() and I want to use uvm_obj_wirepages() to apply locking,
if requested.

Stephan

Am Do., 13. Juni 2024 um 21:19 Uhr schrieb Jason Thorpe :
>
>
>
> > On Jun 13, 2024, at 5:31 AM, Stephan  wrote:
> >
> > Hello,
> >
> > is uvm_obj_wirepages() (and uvm_obj_unwirepages()) as used in
> > sys/kern/sysv_shm.c undocumented?
>
> What are you trying to do?
>
> -- thorpej
>

Re: uvm_obj_wirepages() undocumented?

[Jason Thorpe]

> On Jun 13, 2024, at 5:31 AM, Stephan  wrote:
> 
> Hello,
> 
> is uvm_obj_wirepages() (and uvm_obj_unwirepages()) as used in
> sys/kern/sysv_shm.c undocumented?

What are you trying to do?

-- thorpej

uvm_obj_wirepages() undocumented?

[Stephan]

Hello,

is uvm_obj_wirepages() (and uvm_obj_unwirepages()) as used in
sys/kern/sysv_shm.c undocumented?

I didn´t find it in any man page nor by searching the internet.


Thanks,

Stephan

Re: NetBSD-10.0/i386 spurious SIGSEGV

[Mouse]

>> pid 853.853 (nagios): signal 11 code=1 (trap 6) @eip 0xbb61057a addr 
>> 0x7513f5a9 error=14
> Anyone has ideas of things to investigate?

I don't have anything specific to suggest; I don't work with either
10.* or Xen myself.  But...

> I am about to upgrade the offending domU to amd64, in order to work
> around the problem.  If that works (and I hope it will), I will have
> no way left to test for this bug.

...if you have the space and can set up a test environment you don't
mind giving a copy of to others, it might be useful to help someone
else track it down.  Ideally, that would be a copy of the whole dom0
and domU in question, but I recognize that might involve a prohibitive
amount of space - though if you can strip the test setup down to
essentials, it'll both save on space and reduce what people have to
look at.

But, of course, that depends on you having the resources (disk space,
time, and energy/motivation) to do that.

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Re: NetBSD-10.0/i386 spurious SIGSEGV

[Emmanuel Dreyfus]

On Mon, Jun 10, 2024 at 01:23:38AM +, Emmanuel Dreyfus wrote:
> pid 853.853 (nagios): signal 11 code=1 (trap 6) @eip 0xbb61057a addr 
> 0x7513f5a9 error=14

Anyone has ideas of things to investigate? I am about to upgrade the 
offending domU to amd64, in order to work around the problem. If that
works (and I hope it will), I will have no way left to test for this
bug.

-- 
Emmanuel Dreyfus
m...@netbsd.org

Re: NetBSD-10.0/i386 spurious SIGSEGV

[Emmanuel Dreyfus]

On Sun, Jun 09, 2024 at 08:44:19AM -0400, Mouse wrote:
> Okay, this is strictly a debugging workaround: how about building a
> kernel with code added so that, whenever a SIGSEGV is delivered, the
> siginfo is printed on the console?

Latest SIGSEGV on nagios:
   0xbb610570 <__gettimeofday50>:   mov$0x1a2,%eax
   0xbb610575 <__gettimeofday50+5>: int$0x80
   0xbb610577 <__gettimeofday50+7>: jb 0xbb61057a <__gettimeofday50+10>
   0xbb610579 <__gettimeofday50+9>: ret
=> 0xbb61057a <__gettimeofday50+10>:push   %ebx

I enabled TRAP_SIGDEBUG in src/sys/arch/i386/i386/trap.c and here is what
it says:

pid 853.853 (nagios): signal 11 code=1 (trap 6) @eip 0xbb61057a addr 0x7513f5a9 
error=14
trapframe 0xdd351fa8
eip 0xbb61057a  esp 0xbf7fdf34  efl 0x00010217
edi 0xb9a56b80  esi 0xb9be2380  edx 0x
ecx 0xb9a56b8b
ebp 0xbf7fdf50  ebx 0xbf7fdfa0  eax 0x
cs 0x0017  ds 0x001f  es 0x001f  fs 0x00ab  gs 0x00b3  ss 0x001f
fsbase 0x00cff300 gsbase 0xbbcff36e89b4

Stack dump: 256 bytes @ 0xdd351fa8
b3 00 00 00 ab 00 00 00  1f 00 00 00 1f 00 7f bf | 
80 6b a5 b9 80 23 be b9  50 df 7f bf a0 df 7f bf | .k...#..P...
00 00 00 00 8b 6b a5 b9  00 00 00 00 06 00 00 00 | .k..
04 00 00 00 7a 05 61 bb  17 00 00 00 17 02 01 00 | z.a.
34 df 7f bf 1f 00 00 00  00 00 00 00 00 00 00 00 | 4...
00 00 00 00 00 00 00 00  f0 3f 35 dd b0 3c 35 dd | .?5..<5.
f4 3c 35 dd 00 00 00 00  f0 e9 c5 ba 00 20 a7 00 | .<5.. ..
01 00 00 00 ff ff 00 00  00 f3 cf 00 ff ff b4 39 | ...9
e6 f3 cf ba 00 00 00 00  00 00 00 00 00 00 00 00 | 
7f 03 00 00 00 00 00 00  00 00 00 00 00 00 00 00 | 
00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 | 
00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 | 
00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 | 
00 00 00 00 00 00 00 00  7f 03 00 00 00 00 00 00 | 
00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 | 
80 1f 00 00 ff ff 00 00  00 00 00 00 00 00 00 00 | 


-- 
Emmanuel Dreyfus
m...@netbsd.org

Re: tty process group use-after-free fix RFC

[Jonathan A. Kollasch]

On Tue, Apr 23, 2024 at 09:08:13PM -0500, Jonathan A. Kollasch wrote:
> 

This is now also at http://gnats.netbsd.org/58327

Re: NetBSD-10.0/i386 spurious SIGSEGV

[Mouse]

>> I have seen many crashes on system call returns.  [...]

> I would suggest printing the siginfo, but apparently our gdb doesn't
> support it (so I filed PR 58325): [...]

Okay, this is strictly a debugging workaround: how about building a
kernel with code added so that, whenever a SIGSEGV is delivered, the
siginfo is printed on the console?  It would at least let you get the
information, and I suspect SEGVs are rare enough you wouldn't have to
sift through too many false positives.

It does, though, assume you're comfortable adding code to your kernel
and rebuilding it.  (If trying to build a new kernel SEGVs, maybe
cross-build it?)

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Re: NetBSD-10.0/i386 spurious SIGSEGV

[Taylor R Campbell]

> Date: Sun, 9 Jun 2024 05:28:49 +
> From: Emmanuel Dreyfus 
> 
> I have seen many crashes on system call returns. Another one on
> __gettimeofday50:
> 
>0xbb610570 <__gettimeofday50>: mov$0x1a2,%eax
>0xbb610575 <__gettimeofday50+5>:   int$0x80
>0xbb610577 <__gettimeofday50+7>:   jb 0xbb61057a <__gettimeofday50+10>
>0xbb610579 <__gettimeofday50+9>:   ret
> => 0xbb61057a <__gettimeofday50+10>:  push   %ebx
> 
> Another one:
>0xbb610570 <__gettimeofday50>: mov$0x1a2,%eax
>0xbb610575 <__gettimeofday50+5>:   int$0x80
> => 0xbb610577 <__gettimeofday50+7>:   jb 0xbb61057a <__gettimeofday50+10>
>0xbb610579 <__gettimeofday50+9>:   ret  

I would suggest printing the siginfo, but apparently our gdb doesn't
support it (so I filed PR 58325):

(gdb) print $_siginfo
$1 = void
(gdb) ptype $_siginfo
type = void

If we had the siginfo, it might provide a clue about why something
trapped to the kernel.

Re: NetBSD-10.0/i386 spurious SIGSEGV

[Emmanuel Dreyfus]

On Sat, Jun 08, 2024 at 10:10:58PM -0400, Mouse wrote:
> Are all the failures in __gettimeofday50?  All in trap-to-the-kernel
> calls?

Here is an example with syslogd

Core was generated by `syslogd'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0xbb0384d7 in writev () from /lib/libc.so.12
(gdb) bt
#0  0xbb0384d7 in writev () from /lib/libc.so.12
#1  0xa252 in fprintlog (f=0xbaf44c00, passedbuffer=, 
qentry=0x0) at /usr/src/usr.sbin/syslogd/syslogd.c:2474
#2  0xb2b6 in logmsg (buffer=0xbaf295c0)
at /usr/src/usr.sbin/syslogd/syslogd.c:2012
(...)

   0xbb0384d0 : mov$0x79,%eax
   0xbb0384d5 :   int$0x80
=> 0xbb0384d7 :   jb 0xbb0384da 
   0xbb0384d9 :   ret


-- 
Emmanuel Dreyfus
m...@netbsd.org

Re: NetBSD-10.0/i386 spurious SIGSEGV

[Emmanuel Dreyfus]

On Sat, Jun 08, 2024 at 10:10:58PM -0400, Mouse wrote:
> First thing I'd look at is the userland instruction(s) around the crash
> point, maybe look at instructions starting at 0xbb610480 or something
> and then disassemble forwards looking for 0xbb610579.  In particular,
> I'd be interested in whether it's a store instruction that failed or
> whether this happened during a syscall trap.

   0xbb610570 <__gettimeofday50>:   mov$0x1a2,%eax
   0xbb610575 <__gettimeofday50+5>: int$0x80
   0xbb610577 <__gettimeofday50+7>: jb 0xbb61057a <__gettimeofday50+10>
=> 0xbb610579 <__gettimeofday50+9>: ret  

> Are all the failures in __gettimeofday50?  All in trap-to-the-kernel
> calls?

I have seen many crashes on system call returns. Another one on
__gettimeofday50:

   0xbb610570 <__gettimeofday50>:   mov$0x1a2,%eax
   0xbb610575 <__gettimeofday50+5>: int$0x80
   0xbb610577 <__gettimeofday50+7>: jb 0xbb61057a <__gettimeofday50+10>
   0xbb610579 <__gettimeofday50+9>: ret
=> 0xbb61057a <__gettimeofday50+10>:push   %ebx

Another one:
   0xbb610570 <__gettimeofday50>:   mov$0x1a2,%eax
   0xbb610575 <__gettimeofday50+5>: int$0x80
=> 0xbb610577 <__gettimeofday50+7>: jb 0xbb61057a <__gettimeofday50+10>
   0xbb610579 <__gettimeofday50+9>: ret  

At once I thought about a stack problem, but I think the last one proves
this is not the case. This one involves no memory access.

> You say "multiple machines"; are those multiple domUs on a single dom0,
> or are they spread across multiple underlying hardware machines? 

It happens on multiple hardware machines and starts on upgrading the 
domU. I even tested moving a domU from one machine to another one 
and the bug folllowed. Other netbsd-9 domU on the same dom0 have
no problem, or at least it is rare enough that I did not notice
for years.

> If the latter, how similar are those underlying machines? 

Same model:
vcpu3: Intel(R) Xeon(R) CPU E3-1220 v6 @ 3.00GHz, id 0x906e9


-- 
Emmanuel Dreyfus
m...@netbsd.org

Re: NetBSD-10.0/i386 spurious SIGSEGV

[Mouse]

> After upgrading i386 XEN3PAE_DOMU to NetBSD 10.0, various daemons on
> multuple machines get SIGSEGV at places I could not figure any reason
> why it happens.  [...]

> Program terminated with signal SIGSEGV, Segmentation fault.
> #0  0xbb610579 in __gettimeofday50 () from /lib/libc.so.12
> (gdb) bt
> #0  0xbb610579 in __gettimeofday50 () from /lib/libc.so.12
> #1  0xbb60ca82 in __time50 (t=t@entry=0xbf7fde88)
> at /usr/src/lib/libc/gen/time.c:52
> #2  0x0808afdd in update_check_stats (check_type=3, check_time=1717878817)
> at utils.c:3015

First thing I'd look at is the userland instruction(s) around the crash
point, maybe look at instructions starting at 0xbb610480 or something
and then disassemble forwards looking for 0xbb610579.  In particular,
I'd be interested in whether it's a store instruction that failed or
whether this happened during a syscall trap.

Are all the failures in __gettimeofday50?  All in trap-to-the-kernel
calls?

You say "multiple machines"; are those multiple domUs on a single dom0,
or are they spread across multiple underlying hardware machines?  If
the latter, how similar are those underlying machines?  I'm wondering
if perhaps something is broken in a subtle way such that it manifests
on only certain hardware (I'm talking about something along the lines
of "this tickles erratum #2188 in stepping 478 of Intel CPUs from the
Forest Lawn family").

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

NetBSD-10.0/i386 spurious SIGSEGV

[Emmanuel Dreyfus]

Hello

After upgrading i386 XEN3PAE_DOMU to NetBSD 10.0, various daemons
on multuple machines get SIGSEGV at places I could not figure any 
reason why it happens. Here is an exemple with nagios, but I see 
similar problems with apache httpd, sendmail, slapd, and even 
built-in syslogd and ping. This is a rare even that happens a 
few times a day.

Any hint how I could track this problem down?

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0xbb610579 in __gettimeofday50 () from /lib/libc.so.12
(gdb) bt
#0  0xbb610579 in __gettimeofday50 () from /lib/libc.so.12
#1  0xbb60ca82 in __time50 (t=t@entry=0xbf7fde88)
at /usr/src/lib/libc/gen/time.c:52
#2  0x0808afdd in update_check_stats (check_type=3, check_time=1717878817)
at utils.c:3015
#3  0x0806275b in run_async_host_check (hst=hst@entry=0xbb412300, 
check_options=check_options@entry=0, 
latency=latency@entry=0.010999, 
scheduled_check=scheduled_check@entry=1, 
reschedule_check=reschedule_check@entry=1, 
time_is_valid=time_is_valid@entry=0xbf7fe2dc, 
preferred_time=preferred_time@entry=0xbf7fe2e8) at checks.c:3257
#4  0x08062b41 in run_scheduled_host_check (hst=hst@entry=0xbb412300, 
check_options=0, latency=latency@entry=0.010999)
at checks.c:3023
#5  0x08078882 in handle_timed_event (event=event@entry=0xb968e4f0)
at events.c:1235
#6  0x080792f8 in event_execution_loop () at events.c:1164
#7  0x080583d5 in main (argc=3, argv=0xbf7fe590) at nagios.c:846
(gdb) frame 1
#1  0xbb60ca82 in __time50 (t=t@entry=0xbf7fde88)
at /usr/src/lib/libc/gen/time.c:52
52  if (gettimeofday(, NULL) == -1)
(gdb) print tt 
$1 = {tv_sec = 1717878817, tv_usec = 592767}



-- 
Emmanuel Dreyfus
m...@netbsd.org

Re: hang in vcache_vget()

[Emmanuel Dreyfus]

On Thu, Jun 06, 2024 at 06:28:51AM -0400, Greg Troxel wrote:
> > Is it worth a PR? I have no clue if it can be reproduced.
> I would say yes it's worth it.

kern/58317

-- 
Emmanuel Dreyfus
m...@netbsd.org

Re: hang in vcache_vget()

[Greg Troxel]

Emmanuel Dreyfus  writes:

> Hello
>
> I experienced a system freeze on NetBSD-10.0/i386. Many processes 
> waiting on tstile, and one waiting on vnode, with this backtrace:
> sleepq_block
> cv_wait
> vcache_vget
> vcache_get
> ufs_lookup
> VOP_LOOKUP
> lookup_once
> namei_tryemulroot.constprop.0
> namei
> vn_open
> do_open
> do_sys_openat
>
> I regret I did not took the time to show vnode. 
>
> Is it worth a PR? I have no clue if it can be reproduced.

I would say yes it's worth it.

I have had hangs on 10/amd64, on a system with 32G of ram.  I have been
blaming zfs, but my "never hangs" experience has been on 9/ufs.  But,
others say zfs is fine.

I just came across the "threads leak memory" problem pointed
out by Brian Marcotte, and found a 17G gpg-agent.  I now wonder if
whatever is hanging is being provoked by running out of memory.  Still a
bug, but I no longer feel my "new problem" can be pointed at zfs.

Do you think your system had high memory pressure at the time of your
crash?

(Sort of off topic, you should know that because 32-bit computers are no
longer manufactured, the rust project thinks you shouldn't be using
them.  Take it to the ewaste center right away!)

Re: poll(): IN/OUT vs {RD,WR}NORM

[Brian Buhrow]

Hello.  I just did a quick scan of the telnetd sources from NetBSD-5 
and there are
interesting notes in there about all of this and how urgent data is used, or 
not used, in
different cases.  A check of -current sources still have the same notes and 
code regarding all
of this in telnetd.


-Brian

Re: TCP vs urgent data [was Re: poll(): IN/OUT vs {RD,WR}NORM]

[Mouse]

>> I might rip out the OOB stuff just to find and fix anything trying
>> to use it, though.
> I think ripping it out would be the right thing.  And I suspect very
> little would notice.

I just did a first pass, doing

find . -type f -print0 | xargs -0 mcgrep -H -l MSG_OOB

in /usr/src.

Most of it is no surprise, or is irrelevant here: telnet and ftp,
documentation, the kernel support that backends it, sys/compat, those
are expected.

But I got one surprise: rlogin{,d}.  And I had a quick look at the
code - it actually _uses_ it.  It is not a case where completely
ignoring URG will work.  (It actually uses it as out-of-band data, too.
You'd almost think it came from the same people who tried to turn the
urgent pointer into out-of-band data in the first place.)

Fortunately or unfortunately, I don't care about rlogin.  I would ditch
it when eliminating MSG_OOB.  In theory, eliminating MSG_OOB is wrong,
because TCP may not be the only protocol that uses it.  My sweep found
sys/netiso/tp_usrreq.c, and searching for SSD_RCVATMARK under sys/
finds hits in netiso as well as netinet.  But I care about netiso about
as much as I care about rlogin; I certainly don't mind losing it for
long enough to find everything using TCP "OOB".

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Re: TCP vs urgent data [was Re: poll(): IN/OUT vs {RD,WR}NORM]

[Johnny Billquist]

On 2024-05-29 16:54, Mouse wrote:

But reading RFC 959, there is no mention of using urgent data under
any circumstances.


No _explicit_ mention.  It's there by reference.


What that RFC says about aborting is:



[...]
2. User system sends the Telnet "Synch" signal.


This involves setting URG.  Read the telnet spec's description of the
SYNCH operation (the top of page 9 of RFC854 is a good starting point).


Ah. Didn't bother digging deep enough this time. Fair enough. :-)


See also SO_OOBINLINE, SIOCATMARK, and SS_RCVATMARK.  I am sorely
tempted to try to rip out the OOB botch and design a socket interface
to the urgent pointer that isn't so badly broken, but I doubt I'd
actually find any use for the latter.  I might rip out the OOB stuff
just to find and fix anything trying to use it, though.


I think ripping it out would be the right thing. And I suspect very 
little would notice. And just pretend the sun is shining and continue 
running would probably work for those who even try using it.


  Johnny

--
Johnny Billquist  || "I'm on a bus
  ||  on a psychedelic trip
email: b...@softjar.se ||  Reading murder books
pdp is alive! ||  tryin' to stay hip" - B. Idol

Re: TCP vs urgent data [was Re: poll(): IN/OUT vs {RD,WR}NORM]

[Mouse]

> But reading RFC 959, there is no mention of using urgent data under
> any circumstances.

No _explicit_ mention.  It's there by reference.

> What that RFC says about aborting is:

> [...]
>2. User system sends the Telnet "Synch" signal.

This involves setting URG.  Read the telnet spec's description of the
SYNCH operation (the top of page 9 of RFC854 is a good starting point).

See also SO_OOBINLINE, SIOCATMARK, and SS_RCVATMARK.  I am sorely
tempted to try to rip out the OOB botch and design a socket interface
to the urgent pointer that isn't so badly broken, but I doubt I'd
actually find any use for the latter.  I might rip out the OOB stuff
just to find and fix anything trying to use it, though.

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Re: TCP vs urgent data [was Re: poll(): IN/OUT vs {RD,WR}NORM]

[Johnny Billquist]

On 2024-05-29 14:45, Mouse wrote:

Should we maybe move this to tech-net?  It's no longer about poll().


I question whether it actually works except by accident; see RFC
6093.

I hadn't seen that one before,


Neither had I until Johnny Billquist mentioned it upthread.  (I tend to
share your reaction to the modern IETF, though I have additional
reasons.)


I became aware of it a bunch of years ago when I was doing my TCP 
implementation for RSX and needed to figure what, if anything I should 
do with urgent data. That RFC was nice to discover at that point. :-)



But the facility it provides is of little-to-no use.  I can't recall
anything other than TELNET that actually uses it,

TELNET and those protocols based upon it (SMTP and FTP command at
least).


FTP command, yes.  SMTP I'm moderately sure doesn't do TELNET in any
meaningful sense; for example, I'm fairly sure octet 0xff is not
special.  I find no mention of TELNET in 5321.


I also have not seen anything suggesting that SMTP use the NVT model.
It's basically just the CRLF as end of line thing, and nothing more.
FTP on the other hand do use NVT (or if you want to call that telnet).
The only situation I know of when FTP uses urgent is when trying to 
abort an ongoing transfer. But it's primarily used in this case to just 
potentially wake the server up, as it might be ignoring the command 
channel while transferring data.


But I've also found this to be inconsistent, and behaving in odd ways in 
different implementations, so I would in general say that while it might 
not hurt to sent the ABOR with urgent, it's not something anyone should 
expect.


But reading RFC 959, there is no mention of using urgent data under any 
circumstances. What that RFC says about aborting is:


"Some servers may not be able to monitor the control and data connections
   simultaneously, in which case some special action will be necessary
   to get the server's attention.  The following ordered format is
   tentatively recommended:

  1. User system inserts the Telnet "Interrupt Process" (IP) signal
  in the Telnet stream.

  2. User system sends the Telnet "Synch" signal.

  3. User system inserts the command (e.g., ABOR) in the Telnet
  stream.

  4. Server PI, after receiving "IP", scans the Telnet stream for
  EXACTLY ONE FTP command.

   (For other servers this may not be necessary but the actions listed
   above should have no unusual effect.)"

Which is definitely different than for telnet, which suppsedly scans 
until a DM is found. Here instead you should start searching for the 
first command after IP. Exactly how you trigger the ftp server to even 
start this searching is left undefined, although urgent would seem the 
logical choice, if it worked.


But here again, it becomes the case that the actual urgent pointer 
information isn't really used. It appears that, just as with telnet, 
urgent is actually just used as a signal for the process to start 
reading through the stream of incoming data looking for something.


If you have an implementation that anyway is all the time accepting 
data, then urgent becomes superfluous.


So I would still argue that urgent, as described, isn't actually that 
useful as described. And I'm still left wondering what they actually 
intended with it, as it supposedly is used to mark which data the 
process receives (or sends) is intended to be urgent. But there is no 
strict correlation between what the sender then considered urgent data, 
and what the receiver reports as urgent data.


I do look at both telnet and ftp as implementations that use urgent in a 
way that isn't matching what appears to have been the intent, and is 
actually an implementation doing a workaround for a deficit in the tcp 
spec. In the end, a simpler solution would have sufficed. All that was 
needed was just a way to send a signal to the other end to wake the 
process up, informing it that it should start scanning through data that 
has been received.



SMTP has no actual use for urgent data, and never sends any, but FTP
can in some circumstances I believe (very ancient unreliable memory).


Yes.  It should, according to the spec, be done when sending an ABOR to
abort a transfer in progress.  But, unlike TELNET's specification that
data is to be dropped while looking for IAC DM, the urgent bit can be
completely ignored by an FTP server which is capable of paying
attention to the control channel while a data transfer is in progress.


I don't even see any actual mention of urgent in the ftp spec. But there 
is an implication that the server needs to be notified somehow if it 
isn't usually servicing the command channel while a transfer is in progress.



then botched it further by pointing the urgent sequence number to
the wrong place,

In fairness, when that was done, it wasn't clear it was wrong - that
all long predated anyone even being aware that there were two
different meanings in the TCP spec, people just used 

TCP vs urgent data [was Re: poll(): IN/OUT vs {RD,WR}NORM]

[Mouse]

Should we maybe move this to tech-net?  It's no longer about poll().

>> I question whether it actually works except by accident; see RFC
>> 6093.
> I hadn't seen that one before,

Neither had I until Johnny Billquist mentioned it upthread.  (I tend to
share your reaction to the modern IETF, though I have additional
reasons.)

>> But the facility it provides is of little-to-no use.  I can't recall
>> anything other than TELNET that actually uses it,
> TELNET and those protocols based upon it (SMTP and FTP command at
> least).

FTP command, yes.  SMTP I'm moderately sure doesn't do TELNET in any
meaningful sense; for example, I'm fairly sure octet 0xff is not
special.  I find no mention of TELNET in 5321.

> SMTP has no actual use for urgent data, and never sends any, but FTP
> can in some circumstances I believe (very ancient unreliable memory).

Yes.  It should, according to the spec, be done when sending an ABOR to
abort a transfer in progress.  But, unlike TELNET's specification that
data is to be dropped while looking for IAC DM, the urgent bit can be
completely ignored by an FTP server which is capable of paying
attention to the control channel while a data transfer is in progress.

>> then botched it further by pointing the urgent sequence number to
>> the wrong place,
> In fairness, when that was done, it wasn't clear it was wrong - that
> all long predated anyone even being aware that there were two
> different meanings in the TCP spec, people just used whichever of
> them was most convenient (in terms of how it was expressed, not which
> is easier to implement) and ignored the other completely.   That's
> why it took decades to get fixed - no-one knew that the spec was
> broken for a long time.

So...I guess next to nothing depended on it even then, or someone would
have noticed the interoperability fail sooner than decades.

> Further, if used properly, it really doesn't matter much, the
> application is intended to recognise the urgent data by its content
> in the data stream, all the U bit (& urgent pointer) should be doing
> is giving it a boot up the read stream to suggest that it should
> consume more quickly than it otherwise would.

Right.  But...

> Whether than indication stops one byte earlier or later should not
> really matter.

That depends.  Consider TELNET, which is defined to drop data while
searching for IAC DM.  If the sender consider the urgent pointer to
point _after_ the last urgent octet but the receiver considers it to
point _to_ the last urgent octet, the receiver will get the IAC DM and
notice the urgent pointer points past it and continue reading and
dropping, looking for another IAC DM, dropping at least one data octet
the sender didn't expect.

> The text in that RFC about multiple urgent sequences also misses that
> I think -

I thought that was probably there for clarity, clarifying what
logically follows from the rest.

> all that matters is that as long as there is urgent data coming, the
> application should be aware of that and modify its behaviour to read
> more rapidly than it otherwise might (if it never delays reading from
> the network, always receives & processes packets as soon as they
> arrive, which for example, systems which do remote end echo need to
> do) then it doesn't need to pay attention to the U bit at all).

Well, there are correctness issues in some cases.  For example, in
TELNET's case, it is defined to drop data while sarching for the IAC DM
that makes up part of a synch; ignoring the urgent bit means that
dropping won't happen.  (Does that matter in practice?  Probably not,
especially given how little TELNET is used outside walled gardens.  But
it still is a correctness issue.)

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Driver for PCI-X SAS controller

[SungWon Chung]

Hi,

I'm trying to bring up an old 21264 Alphaserver motherboard, which has two
66MHz 64bit PCI-X slots. To use a SAS SSD, I'm wondering if anyone has used
a PCI-X SAS disk controller with NetBSD?

There are a few available PCI-X SAS controllers (e.g., LSI SAS3080X, LSI
SAS3041X-R, LSI SAS3442X-R) on eBay but not sure whether they're supported
by NetBSD. Although there are many PCIe SAS controllers supported, since
their max data rate will be used by a factor of 2 (because PCIe-to-PCI
bridge card only supports 32bit PCI), I was looking for PCI-X SAS
controllers.

sungwon

Re: poll(): IN/OUT vs {RD,WR}NORM

[Robert Elz]

Date:Tue, 28 May 2024 22:46:09 -0400 (EDT)
From:Mouse 
Message-ID:  <202405290246.waa17...@stone.rodents-montreal.org>

  | I question whether it actually works except by accident; see RFC 6093.

I hadn't seen that one before, I stopped seriously following the IETF
around the end of the last millennium, it was becoming way too commercially
based (decisions no longer based purely on technical merit) with way way
too much bureaucracy.

Aside from the middlebox problem (of which I wasn't aware -- and IMO
anything in the middle of the internet which touches anything above the
IP layer is simply broken - I know NAT forces force a little of that,
but NAT is also simply broken) there is nothing new in there, except
that their change from the Hosts Requirement solution for the off-by-one
issue was the wrong way to go.   The HR group discussed that at length,
using the "last byte of the urgent data" is safe, using that + 1 is not,
in that a system receiving urgent data which believes it should be +0
will be waiting for one more byte to arrive, which might never be sent,
if the transmitter is using +1.   On the other hand, if the transmitter
uses +0 and the receiver is expecting it to be +1, all that happens is
that the U bit turns off one byte sooner, all the urgent data is still
there and available to be read (of course, if anything is pretending this
is one byte of out of bound data they fail either way - but that, as that RFC
says, is simply broken).   (The uninteresting cases when both sender and
transmitter use the same concept aren't worthy of mention).

Of course, if essentially the whole internet has settled on the +1 version
(the original specification, instead of the example code) then perhaps
that change may have been warranted - I certainly haven't surveyed anything
to see which way various systems actually do it, and I expect a lot of
the original systems are long gone by now.

  | But only a few implementors paid any attention, it appears.

Does the BSD stack not do this the way that HR defined things?   I thought
that was changed way way back, before CSRG stopped generating the code.

  | But the facility it provides is of little-to-no use.  I can't recall
  | anything other than TELNET that actually uses it,

TELNET and those protocols based upon it (SMTP and FTP command at least).
SMTP has no actual use for urgent data, and never sends any, but FTP can
in some circumstances I believe (very ancient unreliable memory).

  | Furthermore, given that probably the most popular API to TCP, sockets,
  | botched it by trying to turn it into an out-of-band data stream,

Yes, that was broken.

  | then botched it further by pointing the urgent sequence number to
  | the wrong place,

In fairness, when that was done, it wasn't clear it was wrong - that
all long predated anyone even being aware that there were two different
meanings in the TCP spec, people just used whichever of them was most
convenient (in terms of how it was expressed, not which is easier to
implement) and ignored the other completely.   That's why it took
decades to get fixed - no-one knew that the spec was broken for a long
time.

Further, if used properly, it really doesn't matter much, the application
is intended to recognise the urgent data by its content in the data stream,
all the U bit (& urgent pointer) should be doing is giving it a boot up
the read stream to suggest that it should consume more quickly than it
otherwise would.  Whether than indication stops one byte earlier or later
should not really matter.

The text in that RFC about multiple urgent sequences also misses that I
think - all that matters is that as long as there is urgent data coming,
the application should be aware of that and modify its behaviour to read
more rapidly than it otherwise might (if it never delays reading from the
network, always receives & processes packets as soon as they arrive, which
for example, systems which do remote end echo need to do) then it doesn't
need to pay attention to the U bit at all).

If there are multiple sequences that demand speedy processing, each should
be processed when it is encountered, and if that affects what is done
with other, "normal" data that is also being read quickly, that's just an
aspect of the application protocol.

kre

ps: I am not suggesting that anyone go design new protocols to use urgent
data, just that the system isn't nearly as broken as some people like to
claim.

Re: poll(): IN/OUT vs {RD,WR}NORM

[Mouse]

>>> However, the urgent pointer is close to useless in today's network,
>>> in that there are few-to-no use cases that it is actually useful
>>> for.
> That's probably correct too.  It is however still used (and still
> works) in telnet - though that is not a frequently used application
> any more.

I question whether it actually works except by accident; see RFC 6093.

> [That's where the off by one occurred, there were two references to
> it, one suggested that the urgent pointer would reference the final
> byte of what is considered urgent, the other that it would reference
> one beyond that, that is, the first byte beyond the urgent data.
> This was corrected in the Hosts Requirements RFCs, somewhere in the
> mid 80's if I remember them, roughly.]

But only a few implementors paid any attention, it appears.

> That is all very simple, and works very well, particularly on high
> latency or lossy networks, as long as you're not expecting "urgent"
> to mean "out of band" or "arrive quickly" or anything else like that.

But the facility it provides is of little-to-no use.  I can't recall
anything other than TELNET that actually uses it, though I am by no
stretch fmailiar with more than some of the commonest protocols out
there.

Furthermore, given that probably the most popular API to TCP, sockets,
botched it by trying to turn it into an out-of-band data stream, then
botched it further by pointing the urgent sequence number to the wrong
place, I'd say it is questionable whether it is good for _anything_ any
longer.

> If an application needs a mechanism like this, it works well.

That's a bit like saying that car hand crank starter handles are useful
if you need them: strictly true, but to a first and even second
approximation both the statement and the thing stated about are
irrelevant to everyone.

Also, it's true only provided you don't use sockets for your API (or
fixed sockets - has anyone done a TCP socket interface that exposes the
urgent popinter _properly_?), and provided your and the peer's
implementations agree on which sequence number goes in the urgent
field.

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Re: poll(): IN/OUT vs {RD,WR}NORM

[Robert Elz]

Date:Tue, 28 May 2024 11:03:02 +0200
From:Johnny Billquist 
Message-ID:  <3853e930-4e77-4f6d-8a73-ec826a067...@softjar.se>

  | This is a bit offtopic, but anyway...

So it is, but anyway...

[Quoting Mouse:]
  | > TCP's urgent pointer is well defined.  It is not, however, an
  | > out-of-band data stream,

That's correct.

  | > However, the urgent pointer is close to useless in today's network, in
  | > that there are few-to-no use cases that it is actually useful for.

That's probably correct too.  It is however still used (and still works)
in telnet - though that is not a frequently used application any more.

[end Mouse quotes]

  | It was always useless. The original design clearly had an idea that they 
  | wanted to get something, but it was never clear exactly what that 
  | something was, and even less clear how the urgent pointer would provide it.

That's incorrect.   It is quite clear what was wanted, and aside from a
possible off by one in the original wording, was quite clear in how it
worked, and it did work.

The U bit in the header simply tells the receiver that there is some
data in the data stream (which is not sent out of band) that it probably
should see as soon as it can, and (perhaps, this depends upon the application)
that temporarily suspending any time consuming processing of the intervening
data (such as passing commands to a shell to be executed) would be a good
idea, until the "urgent" data has been processed.

The urgent pointer simply indicates where in the data stream the receiver
needs to have processed to have encountered the urgent data.  It does not
(and never did) "point to" the urgent data.   [That's where the off by one
occurred, there were two references to it, one suggested that the urgent
pointer would reference the final byte of what is considered urgent, the
other that it would reference one beyond that, that is, the first byte beyond
the urgent data.   This was corrected in the Hosts Requirements RFCs, somewhere
in the mid 80's if I remember them, roughly.]   The actual data considered
as urgent could be any number of bytes leading up to that, depending upon
the application protocol.   The application was expected to be able to
detect that, provided it actually saw it in the stream - the U bit (which
would remain set in every packet until one was sent containing no data
that included or preceded any of the urgent data) just allows the receiver
to know that something is coming which it might want to look for - but it
is entirely up to the application protocol design to decide how it is to
be recognised, and what should be done because of it ("nothing" could be
a reasonable answer in some cases).


That is all very simple, and works very well, particularly on high
latency or lossy networks, as long as you're not expecting "urgent"
to mean "out of band" or "arrive quickly" or anything else like that.

It is (was) mostly use with telnet to handle things like interrupts,
where the telnet server would have received a command line, sent that
to the shell (command interpreter) to be processed, and is now waiting
for that to be complete before reading the next command - essentially
using the network, and the sender, as buffering so that it does not need
to grow indefinitely big buffers if the sender just keeps on sending
more and more.

In this situation, if the sender tries to abort a command, when someone
or something realises that it will never finish by itself, then (given that
TCP has no out of band data, which vastly decreases its complexity, and
by so doing increases its reliability) there's no way for the sender to
communicate with the server to convey a "stop that now" message.   And do
remember that all this was designed before unix existed (before RFC's existed,
you need to go back to the original IEN's) when operating systems didn't
work like unix does - it was possible that only one telnet connection
could be made to a destination host (not a TCP or telnet restriction, but
imposed by the OS not providing any kind of parallel processing or 
multi-tasking), so simply connecting again and killing the errant process
wasn't necessarily possible.   Character echo was often done by the
client, not by sending the echoed characters back from the server.
A very different world to the one we're used to.

The U bit (and the urgent pointer which is just a necessary accessory,
not the principle feature) allowed this to be handled.   When the client
had something that needed attention to send, it would send that as "urgent"
data.  But that would just go in sequence with previously sent data (which
in the case of telnet, where the receive window doesn't often fill, was
probably already in the network somewhere) - however the U bit can be set
in the header of every packet transmitted, including retransmits of earlier
data, or even in an in sequence, no data, packet, and will be - with the
sender sending a duplicate, or empty, packet if needed to 

Re: poll(): IN/OUT vs {RD,WR}NORM

[Valery Ushakov]

On Tue, May 28, 2024 at 02:33:48 +, David Holland wrote:

> anything other than the same set of vague descriptions we had in the
> older poll(2).

poll(2) is ... ok, I'm not even sure what adjective to use here.  I
had to write some async TCP poll code that needed to work on Linux,
Solaris and MacOS, and I also tested it on NetBSD - the behavior was
fairly noticably different (half-close was half the fun).  Yet all
behaviors were conforming to what the vaguue descriptions in (various)
poll(2) manpages said.

-uwe

Re: poll(): IN/OUT vs {RD,WR}NORM

[Johnny Billquist]

This is a bit offtopic, but anyway...

On 2024-05-28 05:02, Mouse wrote:

Where do we attach 3 priority levels to data?

[I]n the context of poll itself, it's undefined.  But it's easy to
think that the TCP urgent data would be something usable in this
context.  But as you note, the urgent data is a somewhat broken thing
that noone ever really figured out how it was meant to be used or
anything about it at all.


TCP's urgent pointer is well defined.  It is not, however, an
out-of-band data stream, nor, despite Berkeley's attempts, can it
really be twisted and bent into one, unless you are on a network which
is high bandwidth, low latency, and low loss (as compared to the
"out-of-band" data rate).  Even then, the receiving process has to
handle data promptly.  Which, probably not coincidentally, describes
Berkeley's network and most of their network programs at the time.

However, the urgent pointer is close to useless in today's network, in
that there are few-to-no use cases that it is actually useful for.


It was always useless. The original design clearly had an idea that they 
wanted to get something, but it was never clear exactly what that 
something was, and even less clear how the urgent pointer would provide it.


It's not a BSD problem, but a problem with the whole design. Which is 
why there is an RFC that basically says that it should all go away (RFC 
6093). But then BSD (and others) did try to make it somehow OOB, which 
makes it even more strange.


Oh well. As for poll, it tries to be generic, but at the same time, why 
did it then divide incoming data into three priority brackets? SysV 
stuff was never something that I got into.


  Johnny

--
Johnny Billquist  || "I'm on a bus
  ||  on a psychedelic trip
email: b...@softjar.se ||  Reading murder books
pdp is alive! ||  tryin' to stay hip" - B. Idol

Re: Possibility of basing a QNX-like OS on NetBSD?

[Andrew Warkentin]

On Mon, May 27, 2024 at 9:13 PM David Holland  wrote:
>
> Barrelfish? Also, compared to the cost of the rest of the system,
> dusting off a microkernel that's been on the shelf a bit is cheap.
> But IDK, I don't follow these in any detail.
>

Even the inactive L4-like kernels I'm aware of aren't any closer to
QNX Neutrino than seL4 is. Most of them lack capabilities entirely and
use threads as IPC destinations rather than having thread-independent
endpoints/channels (even the IPC gates in Fiasco.OC and NOVA are bound
to threads, whereas the only connection in MCS seL4 is that its
endpoints maintain queues of waiting threads). I guess VSTa's kernel
would be closer to QNX, but its IPC model isn't fully
capability-oriented (QNX's isn't either, which makes security more
complex than necessary; this is one of the issues with QNX I want to
fix in my own OS) and it only supports x86-32.

> I don't see the difference between what you're describing and the
> rfork/clone model. But in any event, the problems with pthreads
> remain; it's difficult to implement the pthreads behavior of
> fork/exec/wait in a model where threads aren't tied to processes.
>
In the traditional variant of the rfork()/clone() model, there is
nothing to visibly group related threads together, and the state
containers are anonymous and can't be manipulated directly, only
cloned. With the model I'm going to use, a process will be a group of
threads (rather than threads being a type of process), but will have
basically no state other than a list of threads and a command
line/environment, and state containers will be directly visible in
procfs. Every thread will belong to one and only one process. fork()
(which will be a library function that uses various state manipulation
APIs) will only fork a single thread. exec*() will replace an entire
process and all threads in it regardless of what state containers are
bound to them. wait() will specifically wait for an an entire process,
although it will be built on top of a more general API that can wait
for individual threads, entire processes, or entire cgroups.
>
> It only looks like Unix from the outside, and not all that much even
> then...
>
I guess it depends on which definition of "Unix" you consider the most
important. Usually I tend to consider "functional Unix" (which is a
matter of APIs and commands) more important than "conventional
functional Unix" (which is a matter of kernel internals). QNX 4 and
Neutrino certainly qualify as "functional Unix" the way I define it,
but definitely do not qualify as "conventional functional Unix".

Re: poll(): IN/OUT vs {RD,WR}NORM

[David Holland]

On Mon, May 27, 2024 at 11:02:06PM -0400, Mouse wrote:
 > > I have no idea if any such device or file ever have had such a
 > > distinction.
 > 
 > Possibly nobody except System V (or possibly its direct ancestors) ever
 > did.  My impression is that it's a STREAMS thing, but that's a fuzzy
 > impression, mostly coming from manpage notes ("The distinction between
 > some of the fields in the events and revents bitmasks is really not
 > useful without STREAMS.").

Yes, all this stuff is from STREAMS originally, that's why it's poorly
documented with vague semantics...

-- 
David A. Holland
dholl...@netbsd.org

Re: Possibility of basing a QNX-like OS on NetBSD?

[David Holland]

On Sat, May 25, 2024 at 07:40:39PM -0600, Andrew Warkentin wrote:
 > > I think the question you should be asking is what your goal is -- are
 > > you using seL4 because you specifically want to leverage seL4's
 > > properties? If so, launching off in another direction seems like the
 > > wrong move. If not, there are other L4-style microkernels you can use
 > > that don't have as many restrictions as seL4, and there's a largish
 > > community of advocates that will each be eager to help you decide to
 > > use theirs.
 > 
 > AFAIK the only other active L4-like kernels are NOVA and Fiasco.OC,

Barrelfish? Also, compared to the cost of the rest of the system,
dusting off a microkernel that's been on the shelf a bit is cheap.
But IDK, I don't follow these in any detail.

 > My OS is going to be multi-server, although it will mostly have a
 > process-per-subsystem architecture like QNX does, and not a
 > process-per-component architecture like Genode or L4Re. Protection
 > domains correspond to subsystems more often than they do to
 > components, so there's not much point in splitting up components in a
 > lot of cases.

Well, that depends. The whole premise of a microkernel is that you
gain something by being in a separate address space and that it makes
sense to split up the (traditionally unitary) kernel protection domain
to gain that something. How finely you split it seems like a matter of
taste. (Or, sometimes, religious doctrine...)

 > > The primary reason the world's gradually moved from that model of
 > > threads to a model where threads are second-class within processes
 > > is... design stupidity in pthreads. If you want to support pthreads,
 > > especially if you have any concerns about it being fast, don't go in
 > > this direction.
 > 
 > I don't think I've seen any Unix-like OS that uses the thread model
 > I'm planning to use. They pretty much all either use a Plan
 > 9/Linux-like rfork()/clone() "threads are processes sharing state"
 > model, or a Mach-like "processes consist of state and threads" model,
 > rather than a "state is independent of processes and explicitly bound
 > to specific threads" model. The model I'm planning to use is a close
 > match to that of seL4, where capability spaces and virtual address
 > spaces exist independently of each other and threads.

I don't see the difference between what you're describing and the
rfork/clone model. But in any event, the problems with pthreads
remain; it's difficult to implement the pthreads behavior of
fork/exec/wait in a model where threads aren't tied to processes.

 > >  > Another thing that I'm not sure about is the real-time performance. In
 > >  > addition to desktop and server use, embedded systems with hard
 > >  > real-time constraints are also an important use case for this system.
 > >
 > > In that case you want to stay a long way away from anything that looks
 > > like Unix.
 >
 > QNX is Unix-like and has reasonable performance for real-time systems,
 > although of course it is quite different from conventional Unix in its
 > architecture.

It only looks like Unix from the outside, and not all that much even
then...

-- 
David A. Holland
dholl...@netbsd.org

Re: poll(): IN/OUT vs {RD,WR}NORM

[Mouse]

>> Where do we attach 3 priority levels to data?
> [I]n the context of poll itself, it's undefined.  But it's easy to
> think that the TCP urgent data would be something usable in this
> context.  But as you note, the urgent data is a somewhat broken thing
> that noone ever really figured out how it was meant to be used or
> anything about it at all.

TCP's urgent pointer is well defined.  It is not, however, an
out-of-band data stream, nor, despite Berkeley's attempts, can it
really be twisted and bent into one, unless you are on a network which
is high bandwidth, low latency, and low loss (as compared to the
"out-of-band" data rate).  Even then, the receiving process has to
handle data promptly.  Which, probably not coincidentally, describes
Berkeley's network and most of their network programs at the time.

However, the urgent pointer is close to useless in today's network, in
that there are few-to-no use cases that it is actually useful for.

> It's not really even oob.

No, and it never has been, despite Berkeley's hallucinations.

> But poll isn't really getting into any details about this.  Just that
> if you have some sort of [data stream], where you can assign multiple
> priorities to the data, then poll can make use of it.

For a few priorities, yes.  It's a poor design in that it doesn't
provide any good way to handle more than two or maybe three different
priorities.

> I have no idea if any such device or file ever have had such a
> distinction.

Possibly nobody except System V (or possibly its direct ancestors) ever
did.  My impression is that it's a STREAMS thing, but that's a fuzzy
impression, mostly coming from manpage notes ("The distinction between
some of the fields in the events and revents bitmasks is really not
useful without STREAMS.").

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Re: poll(): IN/OUT vs {RD,WR}NORM

[Mouse]

>>  POLLPRIHigh priority data may be read without blocking.
>>  POLLRDBAND Priority data may be read without blocking.
>>  POLLRDNORM Normal data may be read without blocking.

> Is this related to the "oob data" scheme in TCP (which is a hack that
> doesn't work)?

I really wish BSD hadn't tried to turn the urgent pointer into an
out-of-band data stream, because, as you say, it doesn't work for that.

It doesn't really work very well even as an API to TCP's urgent
pointer.

> Where do we attach 3 priority levels to data?

That's part of what I was wondering.

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Re: poll(): IN/OUT vs {RD,WR}NORM

[Mouse]

> Also, I suspect mouse was thinking of the TCP URG concept, and not
> PUSH when he wrote what he did, but I don't know for sure.

Ouch.  Yes, you are entirely correct in that regard.  Total braino on
my part.  My apologies.

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Re: poll(): IN/OUT vs {RD,WR}NORM

[Mouse]

>> I can understand treating POLLWRNORM as identical to POLLOUT.  But
>> why the distinction between POLLRDNORM and POLLIN?  Might anyone
>> know the reason and be willing to explain?
> I'd hazard a guess that you'll likely not get an explanation.

Quite possibly not, but I figured it was still workt asking.

>> ...  I'd still be curious where it came from.
> Those answers are CVS:

Not where it came from in the sense of who committed it or what it
looked like at the time, but rather where that person got the various
distinctions from.

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Re: poll(): IN/OUT vs {RD,WR}NORM

[David Holland]

On Mon, May 27, 2024 at 08:27:28PM -0400, Mouse wrote:
 > In sys/sys/poll.h, I see (in 1.4T, -current according to cvsweb, and
 > everything I've checked in between):
 > 
 > #define POLLIN  0x0001
 > #define POLLPRI 0x0002
 > #define POLLOUT 0x0004
 > #define POLLRDNORM  0x0040
 > #define POLLWRNORM  POLLOUT
 > #define POLLRDBAND  0x0080
 > #define POLLWRBAND  0x0100
 > 
 > I can understand treating POLLWRNORM as identical to POLLOUT.  But why
 > the distinction between POLLRDNORM and POLLIN?  Might anyone know the
 > reason and be willing to explain?
 >
 > [...]
 >
 > -current's manpage's BUGS entry implies, to me, that this distinction
 > is something of a historical accident that should be fixed, but, even
 > if that reading is correct, I'd still be curious where it came from.

In NetBSD's implementation, the distinction is a bug and there are (or
were as of when I put that text in the man page) only a handful of
cases where they're different, all of which could clearly be
interpreted as mistakes. (I don't remember why cleaning them up wasn't
on the agenda at the time, probably lack of immediate resources for
tackling invasive patches.)

The research I did when I rewrote the page indicated that it was
correct for POLLRDNORM and POLLIN to be the same, but I don't remember
the details. I can't remember or not if that involved finding and
discounting docs that claimed POLLIN was the same as
POLLRDNORM|POLLRDBAND, like the one cited upthread. I think it may
have, because I definitely recall having to dig a fair amount to find
anything other than the same set of vague descriptions we had in the
older poll(2).

I'm not sure if I have any notes from then, probably not.

There are a whole bunch of remaining unanswered questions, though,
which are sitting in PR 55983.

-- 
David A. Holland
dholl...@netbsd.org

Re: poll(): IN/OUT vs {RD,WR}NORM

[Johnny Billquist]

On 2024-05-28 03:05, Greg Troxel wrote:

Johnny Billquist  writes:


  POLLPRIHigh priority data may be read without blocking.

  POLLRDBAND Priority data may be read without blocking.

  POLLRDNORM Normal data may be read without blocking.


Is this related to the "oob data" scheme in TCP (which is a hack that
doesn't work)?   Where do we attach 3 priority levels to data?


At the end of my quoting there were:

 The distinction between normal, priority, and high-priority data is
 specific to particular file types or devices.


So in the context of poll itself, it's undefined. But it's easy to think 
that the TCP urgent data would be something usable in this context. But 
as you note, the urgent data is a somewhat broken thing that noone ever 
really figured out how it was meant to be used or anything about it at 
all. It's not really even oob. And it should basically just go away.


But poll isn't really getting into any details about this. Just that if 
you have some sort of device of file, where you can assign multiple 
priorities to the data, then poll can make use of it.

I have no idea if any such device or file ever have had such a distinction.

  Johnny

--
Johnny Billquist  || "I'm on a bus
  ||  on a psychedelic trip
email: b...@softjar.se ||  Reading murder books
pdp is alive! ||  tryin' to stay hip" - B. Idol

Re: poll(): IN/OUT vs {RD,WR}NORM

[Taylor R Campbell]

> Date: Mon, 27 May 2024 20:27:28 -0400 (EDT)
> From: Mouse 
> 
> In sys/sys/poll.h, I see (in 1.4T, -current according to cvsweb, and
> everything I've checked in between):
> 
> #define POLLIN  0x0001
> #define POLLPRI 0x0002
> #define POLLOUT 0x0004
> #define POLLRDNORM  0x0040
> #define POLLWRNORM  POLLOUT
> #define POLLRDBAND  0x0080
> #define POLLWRBAND  0x0100
> 
> I can understand treating POLLWRNORM as identical to POLLOUT.  But why
> the distinction between POLLRDNORM and POLLIN?  Might anyone know the
> reason and be willing to explain?

It's been this way since sysvr4, according to the archive at

(sysvr4.tar.bz2, svr4/uts/i386/sys/poll.h):

/*
 * Testable select events 
 */
#define POLLIN  0x0001  /* fd is readable */
#define POLLPRI 0x0002  /* high priority info at fd */
#define POLLOUT 0x0004  /* fd is writeable (won't block) */
#define POLLRDNORM  0x0040  /* normal data is readable */
#define POLLWRNORM  POLLOUT
#define POLLRDBAND  0x0080  /* out-of-band data is readable */
#define POLLWRBAND  0x0100  /* out-of-band data is writeable */

sysvr3 didn't have the NORM/BAND ones.  Perhaps you'll find more clues
by digging through the changes from sysvr3 to sysvr4.

Re: poll(): IN/OUT vs {RD,WR}NORM

[Greg Troxel]

Johnny Billquist  writes:

>  POLLPRIHigh priority data may be read without blocking.
>
>  POLLRDBAND Priority data may be read without blocking.
>
>  POLLRDNORM Normal data may be read without blocking.

Is this related to the "oob data" scheme in TCP (which is a hack that
doesn't work)?   Where do we attach 3 priority levels to data?

Re: poll(): IN/OUT vs {RD,WR}NORM

[Johnny Billquist]

Maybe not an explanation, but looking at the doc on OSX is maybe a bit 
of an explanation:


 POLLIN Data other than high priority data may be read without
blocking.  This is equivalent to ( POLLRDNORM | 
POLLRDBAND

).

 POLLNVAL   The file descriptor is not open.  This flag is output
only, and ignored if present in the input events 
bitmask.


 POLLOUTNormal data may be written without blocking.  This is
equivalent to POLLWRNORM.

 POLLPRIHigh priority data may be read without blocking.

 POLLRDBAND Priority data may be read without blocking.

 POLLRDNORM Normal data may be read without blocking.

 POLLWRBAND Priority data may be written without blocking.

 POLLWRNORM Normal data may be written without blocking.

 The distinction between normal, priority, and high-priority data is
 specific to particular file types or devices.


Which could/would explain why POLLIN isn't equivalent to POLLRDNORM.

Also, I suspect mouse was thinking of the TCP URG concept, and not PUSH 
when he wrote what he did, but I don't know for sure.


  Johnny


On 2024-05-28 02:42, Simon Burge wrote:

Mouse wrote:


In sys/sys/poll.h, I see (in 1.4T, -current according to cvsweb, and
everything I've checked in between):

#define POLLIN  0x0001
#define POLLPRI 0x0002
#define POLLOUT 0x0004
#define POLLRDNORM  0x0040
#define POLLWRNORM  POLLOUT
#define POLLRDBAND  0x0080
#define POLLWRBAND  0x0100

I can understand treating POLLWRNORM as identical to POLLOUT.  But why
the distinction between POLLRDNORM and POLLIN?  Might anyone know the
reason and be willing to explain?


I'd hazard a guess that you'll likely not get an explanation.


...  I'd still be curious where it came from.


Those answers are CVS:

   thoreau 55002> cvs annotate -r1.1 sys/sys/poll.h | grep define.POLL
   Annotations for sys/sys/poll.h
   ***
   1.1  (mycroft  07-Sep-96): #definePOLLIN  0x0001
   1.1  (mycroft  07-Sep-96): #definePOLLPRI 0x0002
   1.1  (mycroft  07-Sep-96): #definePOLLOUT 0x0004
   1.1  (mycroft  07-Sep-96): #definePOLLRDNORM  0x0040
   1.1  (mycroft  07-Sep-96): #definePOLLWRNORM  POLLOUT
   1.1  (mycroft  07-Sep-96): #definePOLLRDBAND  0x0080
   1.1  (mycroft  07-Sep-96): #definePOLLWRBAND  0x0100
   1.1  (mycroft  07-Sep-96): #definePOLLERR 0x0008
   1.1  (mycroft  07-Sep-96): #definePOLLHUP 0x0010
   1.1  (mycroft  07-Sep-96): #definePOLLNVAL0x0020
   thoreau 55003> cvs log -N -r1.1 sys/sys/poll.h
   ...
   
   revision 1.1
   date: 1996-09-08 03:42:49 +1000;  author: mycroft;  state: Exp;
   Definitions for poll(2).  Prototype it here.
   =

Cheers,
Simon.


--
Johnny Billquist  || "I'm on a bus
  ||  on a psychedelic trip
email: b...@softjar.se ||  Reading murder books
pdp is alive! ||  tryin' to stay hip" - B. Idol

Re: poll(): IN/OUT vs {RD,WR}NORM

[Simon Burge]

Mouse wrote:

> In sys/sys/poll.h, I see (in 1.4T, -current according to cvsweb, and
> everything I've checked in between):
>
> #define POLLIN  0x0001
> #define POLLPRI 0x0002
> #define POLLOUT 0x0004
> #define POLLRDNORM  0x0040
> #define POLLWRNORM  POLLOUT
> #define POLLRDBAND  0x0080
> #define POLLWRBAND  0x0100
>
> I can understand treating POLLWRNORM as identical to POLLOUT.  But why
> the distinction between POLLRDNORM and POLLIN?  Might anyone know the
> reason and be willing to explain?

I'd hazard a guess that you'll likely not get an explanation.

> ...  I'd still be curious where it came from.

Those answers are CVS:

  thoreau 55002> cvs annotate -r1.1 sys/sys/poll.h | grep define.POLL
  Annotations for sys/sys/poll.h
  ***
  1.1  (mycroft  07-Sep-96): #definePOLLIN  0x0001
  1.1  (mycroft  07-Sep-96): #definePOLLPRI 0x0002
  1.1  (mycroft  07-Sep-96): #definePOLLOUT 0x0004
  1.1  (mycroft  07-Sep-96): #definePOLLRDNORM  0x0040
  1.1  (mycroft  07-Sep-96): #definePOLLWRNORM  POLLOUT
  1.1  (mycroft  07-Sep-96): #definePOLLRDBAND  0x0080
  1.1  (mycroft  07-Sep-96): #definePOLLWRBAND  0x0100
  1.1  (mycroft  07-Sep-96): #definePOLLERR 0x0008
  1.1  (mycroft  07-Sep-96): #definePOLLHUP 0x0010
  1.1  (mycroft  07-Sep-96): #definePOLLNVAL0x0020
  thoreau 55003> cvs log -N -r1.1 sys/sys/poll.h
  ...
  
  revision 1.1
  date: 1996-09-08 03:42:49 +1000;  author: mycroft;  state: Exp;
  Definitions for poll(2).  Prototype it here.
  =

Cheers,
Simon.

poll(): IN/OUT vs {RD,WR}NORM

[Mouse]

In sys/sys/poll.h, I see (in 1.4T, -current according to cvsweb, and
everything I've checked in between):

#define POLLIN  0x0001
#define POLLPRI 0x0002
#define POLLOUT 0x0004
#define POLLRDNORM  0x0040
#define POLLWRNORM  POLLOUT
#define POLLRDBAND  0x0080
#define POLLWRBAND  0x0100

I can understand treating POLLWRNORM as identical to POLLOUT.  But why
the distinction between POLLRDNORM and POLLIN?  Might anyone know the
reason and be willing to explain?

Not that it matters tremendously.  But I'm curious, because it
indicates there's something I don't understand somewhere there.  The
wording is a bit confusing.  In -current's poll(2), POLLIN is said to
be synonymous with POLLRDNORM (and POLLOUT with POLLWRNORM); in 1.4T
and 5.2, there is confusing wording about "high priority data" and
"data with a non-zero priority", which may or may not be talking about
the same thing and may or may not map onto other concepts such as TCP's
push semantics, and the wording is different in the two directions.
-current's manpage's BUGS entry implies, to me, that this distinction
is something of a historical accident that should be fixed, but, even
if that reading is correct, I'd still be curious where it came from.

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Re: Possibility of basing a QNX-like OS on NetBSD?

[Andrew Warkentin]

On Sat, May 25, 2024 at 3:04 PM David Holland  wrote:
>
>
> I think the question you should be asking is what your goal is -- are
> you using seL4 because you specifically want to leverage seL4's
> properties? If so, launching off in another direction seems like the
> wrong move. If not, there are other L4-style microkernels you can use
> that don't have as many restrictions as seL4, and there's a largish
> community of advocates that will each be eager to help you decide to
> use theirs.
>

AFAIK the only other active L4-like kernels are NOVA and Fiasco.OC,
and neither of them have QNX-style long message support. seL4's memory
model is probably better for the system I'm working on; NOVA and
Fiasco.OC are more designed with strongly hierarchical systems in mind
where there are a few subsystem servers that each get their own memory
quota and hand out parts of that to child processes, rather than a
more traditional Unix-like model (I do plan to support memory quotas
and resource trading, although they won't follow a strict hierarchy).
There are several things I'd like to add to my fork of seL4, but most
of them should be simpler to add than long message support. The only
changes I could see myself making on a similar scale to long IPC would
be if there are issues with SMP scalability.

>
> If on the other hand you're specifically interested in making
> something QNX-like, you'll probably need to write most of it from
> scratch, since the only way to get that level of small and fast is
> careful integration and careful design balancing goals and costs.
>
> In any event, NetBSD isn't a microkernel and turning it into one would
> pretty much be a 100% rewrite, the existence of rump notwithstanding.
> It would be expensive and probably not come out all that well.
>

That was pretty much my conclusion as well.

>
> If you want a single-server Unix on top of a microkernel, rump is
> maybe not a bad place to start... but this has been done many times
> before and you're probably better off using one of the existing
> examples.
>

My OS is going to be multi-server, although it will mostly have a
process-per-subsystem architecture like QNX does, and not a
process-per-component architecture like Genode or L4Re. Protection
domains correspond to subsystems more often than they do to
components, so there's not much point in splitting up components in a
lot of cases.

>
>  > I also wish to disaggregate the usual Unix process model into a far
>  > more thread-centric one where a process is nothing more than a
>  > collection of threads that share a command line and get replaced on
>  > exec, with all of the usual process state like virtual address space,
>  > open file descriptors, and filesystem namespace being separate context
>  > objects that have to be explicitly shared between threads, and the
>  > basic process creating primitive just creating a completely blank
>  > process that the parent explicitly initializes with all the necessary
>  > state using various APIs (of course, there will be a library that
>  > implements fork(), spawn(), and pthreads on top of this).
>
> The primary reason the world's gradually moved from that model of
> threads to a model where threads are second-class within processes
> is... design stupidity in pthreads. If you want to support pthreads,
> especially if you have any concerns about it being fast, don't go in
> this direction.
>

I don't think I've seen any Unix-like OS that uses the thread model
I'm planning to use. They pretty much all either use a Plan
9/Linux-like rfork()/clone() "threads are processes sharing state"
model, or a Mach-like "processes consist of state and threads" model,
rather than a "state is independent of processes and explicitly bound
to specific threads" model. The model I'm planning to use is a close
match to that of seL4, where capability spaces and virtual address
spaces exist independently of each other and threads.

>
>  > Another thing that I'm not sure about is the real-time performance. In
>  > addition to desktop and server use, embedded systems with hard
>  > real-time constraints are also an important use case for this system.
>
> In that case you want to stay a long way away from anything that looks
> like Unix.
>
QNX is Unix-like and has reasonable performance for real-time systems,
although of course it is quite different from conventional Unix in its
architecture.

Re: Possibility of basing a QNX-like OS on NetBSD?

[David Holland]

On Sat, May 25, 2024 at 05:48:56AM -0600, Andrew Warkentin wrote:
 > I'm currently working on a QNX-like microkernel OS based on a fork of
 > seL4 and an original root server. Recently I had someone suggest that
 > I should look at trying to turn NetBSD into a QNX-like microkernel
 > because seL4's focus is more on static non-Unix-like systems.

That's a curious proposition, kind of like suggesting you should
harness a dump truck to your horses because chariots are for racing.

I think the question you should be asking is what your goal is -- are
you using seL4 because you specifically want to leverage seL4's
properties? If so, launching off in another direction seems like the
wrong move. If not, there are other L4-style microkernels you can use
that don't have as many restrictions as seL4, and there's a largish
community of advocates that will each be eager to help you decide to
use theirs.

If on the other hand you're specifically interested in making
something QNX-like, you'll probably need to write most of it from
scratch, since the only way to get that level of small and fast is
careful integration and careful design balancing goals and costs.

In any event, NetBSD isn't a microkernel and turning it into one would
pretty much be a 100% rewrite, the existence of rump notwithstanding.
It would be expensive and probably not come out all that well.

If you want a single-server Unix on top of a microkernel, rump is
maybe not a bad place to start... but this has been done many times
before and you're probably better off using one of the existing
examples.

 > I also wish to disaggregate the usual Unix process model into a far
 > more thread-centric one where a process is nothing more than a
 > collection of threads that share a command line and get replaced on
 > exec, with all of the usual process state like virtual address space,
 > open file descriptors, and filesystem namespace being separate context
 > objects that have to be explicitly shared between threads, and the
 > basic process creating primitive just creating a completely blank
 > process that the parent explicitly initializes with all the necessary
 > state using various APIs (of course, there will be a library that
 > implements fork(), spawn(), and pthreads on top of this).

The primary reason the world's gradually moved from that model of
threads to a model where threads are second-class within processes
is... design stupidity in pthreads. If you want to support pthreads,
especially if you have any concerns about it being fast, don't go in
this direction.

 > Another thing that I'm not sure about is the real-time performance. In
 > addition to desktop and server use, embedded systems with hard
 > real-time constraints are also an important use case for this system.

In that case you want to stay a long way away from anything that looks
like Unix.

-- 
David A. Holland
dholl...@netbsd.org

Possibility of basing a QNX-like OS on NetBSD?

[Andrew Warkentin]

I'm currently working on a QNX-like microkernel OS based on a fork of
seL4 and an original root server. Recently I had someone suggest that
I should look at trying to turn NetBSD into a QNX-like microkernel
because seL4's focus is more on static non-Unix-like systems. However,
I think that would be more difficult than it seems at first glance
despite NetBSD being a Unix-like general-purpose OS like QNX. Even
though QNX looks superficially similar to conventional Unix-like OSes
in a lot of ways and is quite compatible with them, it is really quite
different from them in some pretty important ways.

One of the biggest of these is the IPC model. QNX's IPC essentially
acts as a cross-address space function call, and not a one-way message
queue. When a client process sends a message to a server over a
channel, the remainder of its timeslice is transferred directly to the
server process and a context switch occurs immediately, entirely
bypassing the scheduler queue in most cases. The client process is
blocked while the server processes the message. Once the server is
done, it sends a reply (rather than specifying the channel, it instead
specifies the message ID that it got when it received the message),
and the same direct context switch happens again in the opposite
direction, and the client is unblocked. It is possible for a server to
receive further messages even if it has previous messages that it
hasn't replied to. A collection of data buffers of arbitrary size and
location may be transferred in both directions; these buffers are
copied directly from the address space of the sender to that of the
receiver with no intermediary buffers in kernel space (specified by a
readv()/writev()-style vector). seL4's IPC already has basically
identical call/receive/reply with direct context switch semantics to
QNX, although it is limited to copying between per-thread single-page
buffers rather than arbitrary vectors. In my (currently unnamed) hard
fork of seL4 I already have a working preliminary implementation of
long IPC with arbitrary vectors, and while dealing with seL4's
unconventional preemption semantics was a little tricky, it wasn't
especially difficult to add long copying to the existing IPC layer. On
the other hand, from what I've seen from looking at the NetBSD
sources, implementing QNX-style IPC would require writing a complete
IPC layer from scratch and making several modifications to the
scheduler and virtual memory manager (trying to port a QNX/L4-style
IPC layer from a kernel that already has one probably wouldn't work,
since they're pretty tightly integrated).

Also, the general architecture is quite different. In order to get
something that is QNX-like enough for my liking, the vast majority of
subsystems would have to be removed from the kernel and moved into
separate user processes. Really it would have to be sort of like an
inverse rump kernel where just the inner kernel with the scheduler,
IPC, virtual memory management and some parts of the VFS would be
left. By the time I'm done, I'm not sure there would be much code left
untouched, and I think it might be easier to just continue on my
current path of using an seL4 fork and an original root server. I may
incorporate more code from other OSes into my root server, since I've
already done a little bit of that, but trying to wholesale convert
something like NetBSD into the kind of OS I want seems like it might
not be worth it. There have been monolithic-to-microkernel conversions
done in the past, although all of the ones I'm aware of are either
done with monolithic kernels that were designed from the start to be
converted (e.g. early Mach kernels), are "serverizations" where the
inner parts of the monolithic kernel like scheduling and basic memory
management are removed and the remainder is ported to a purpose-built
microkernel (e.g. LP49, Lites, and possibly MkLinux), or are just
treating the microkernel as a hypervisor (e.g. L4Linux). I'm not aware
of any monolithic-to-microkernel conversions where the outer
subsystems of the kernel are removed and the inner kernel is retained
as a microkernel happening, and I've read a lot on OS history and
played around with most of the historical OSes I can get my hands on
going back to ones from the 50s.

The VFS architecture specifically is rather different between
something like QNX and more conventional Unix-like systems. QNX's
central VFS or pathname manager (part of the process server) is much
simpler than a conventional Unix VFS; it pretty much just matches path
prefixes and hands out a connection to the channel ID of the server
handling the one that fits the path best; the open request to the
server contains the remainder of the path rather than an inode number.
Server calls generally bypass the VFS entirely and go straight from
the client through the microkernel to the server. The VFS I'm planning
to implement will be a little more conventional in order to allow for
stronger security (QNX's channel 

Re: NetBSD-10: write to rw null mount on top of LFS fs hangs up

[Jose Luis Rodriguez Garcia]

>Some diagnostics from crash(8) are below and in attachments.
>
>crash> show all tstiles
>  PID   LID  COMMAND  WAITING-FOR WAIT-CHANNEL
> 4999  4999 find 882851694480 88290bf87680
>crash>
>.

May be, it could be useful the output from dumplfs until the ouptut of
semgent 0. (I don't have knowledge for debug it).



dumplfs /dev/ld2a

Re: NetBSD-10: write to rw null mount on top of LFS fs hangs up

[Aleksey Cheusov]

Some diagnostics from crash(8) are below and in attachments.

crash> show all tstiles
  PID   LID  COMMAND  WAITING-FOR WAIT-CHANNEL
 4999  4999 find 882851694480 88290bf87680
crash>

sleepq_block() at sleepq_block+0x13a
mtsleep() at mtsleep+0x146
lfs_check() at lfs_check+0x309
lfs_write() at lfs_write+0xfd
layer_bypass() at layer_bypass+0xf0
VOP_WRITE() at VOP_WRITE+0x12f
vn_write() at vn_write+0x10e
dofilewrite() at dofilewrite+0x80
sys_write() at sys_write+0x49
syscall() at syscall+0x1fc
--- syscall (number 4) ---
syscall+0x1fc:

sleepq_block() at sleepq_block+0x13a
mtsleep() at mtsleep+0x146
lfs_writer_enter() at lfs_writer_enter+0x87
lfs_segwrite() at lfs_segwrite+0x29d
lfs_vflush() at lfs_vflush+0x6cf
lfs_update() at lfs_update+0x1ba
lfs_fsync() at lfs_fsync+0x24d
layer_bypass() at layer_bypass+0xf0
VOP_FSYNC() at VOP_FSYNC+0x6f
sys_fsync() at sys_fsync+0x5f
syscall() at syscall+0x1fc
--- syscall (number 95) ---
syscall+0x1fc:

sleepq_block() at sleepq_block+0x13a
mtsleep() at mtsleep+0x146
lfs_seglock() at lfs_seglock+0x79
lfs_valloc() at lfs_valloc+0x70
lfs_newvnode() at lfs_newvnode+0x6a
vcache_new() at vcache_new+0x97
lfs_makeinode() at lfs_makeinode+0x3a
lfs_create() at lfs_create+0xb8
layer_bypass() at layer_bypass+0xf0
VOP_CREATE() at VOP_CREATE+0x9e
vn_open() at vn_open+0x442
do_open() at do_open+0xc3
do_sys_openat() at do_sys_openat+0x74
sys_open() at sys_open+0x24
syscall() at syscall+0x1fc
--- syscall (number 5) ---
syscall+0x1fc:

sleepq_block() at sleepq_block+0x13a
cv_wait_sig() at cv_wait_sig+0xbd
lfs_set_dirop() at lfs_set_dirop+0xcd
lfs_create() at lfs_create+0x84
layer_bypass() at layer_bypass+0xf0
VOP_CREATE() at VOP_CREATE+0x9e
vn_open() at vn_open+0x442
do_open() at do_open+0xc3
do_sys_openat() at do_sys_openat+0x74
sys_open() at sys_open+0x24
syscall() at syscall+0x1fc
--- syscall (number 5) ---
syscall+0x1fc:

sleepq_block() at sleepq_block+0x13a
cv_wait_sig() at cv_wait_sig+0xbd
lfs_set_dirop() at lfs_set_dirop+0xcd
lfs_mkdir() at lfs_mkdir+0xa3
layer_bypass() at layer_bypass+0xf0
VOP_MKDIR() at VOP_MKDIR+0xa4
do_sys_mkdirat.isra.0() at do_sys_mkdirat.isra.0+0x145
syscall() at syscall+0x1fc
--- syscall (number 136) ---
syscall+0x1fc:


PIDLID S CPU FLAGS   STRUCT LWP *   NAME WAIT
15841>15841 7  10   100   8828991ed580  crash
9368  9368 3   4   180   882900d0e040 sh wait
12114 13781 3   1   1000180   8828e9dae200 gdb worker parked
12114 5297 3   0   1000180   8828e46a7700 gdb worker parked
12114   75 3  11   1000180   8828e9daea80 gdb worker parked
12114 11640 3  10   1000180   88290bd4b5c0 gdb worker parked
12114 5711 3   9   1000180   88290d6c5980 gdb worker parked
12114 22813 3   8   1000180   882906f32a80 gdb worker parked
12114 2094 3   7   1000180   88276fd305c0 gdb worker parked
12114 16537 3   6   1000180   882893969b40 gdb worker parked
12114 22079 3   5   1000180   88290dce68c0 gdb worker parked
12114 20383 3   2   1000180   8828feeafa00 gdb worker parked
12114 29023 3   4   1000180   882871957480 gdb worker parked
12114 16684 3   2   1000180   882882788580 gdb worker parked
12114>12114 7   3 40100   8828258c7900gdb
14007 14007 3   7   180   88290c7f3340 pickup kqueue
4999  4999 3   0 0   8829017235c0   find tstile
20718 20718 3   0   180   8827ed4d7540   postdrop netio
3828  3828 3   0   180   88287226fbc0   sendmail pipe_rd
9746  9746 3   7   180   88290633ea80tee pipe_rd
1006  1006 3  11   180   88268d644540 sh wait
17276 17276 3   4   180   882906c10300 sh wait
4803  4803 3   9   180   882901c36040   cron pipe_rd
800800 3   2   180   882908a12680 sh ttyraw
2303  2303 3   3   180   882900d0e480   sshd poll
15511 15511 3   3   180   8829098f31c0 sh ttyraw
10621 10621 3   3   180   8828a9338300   sshd poll
26393 26393 3   5   180   882851694480 mktemp lfs_dirop
7605  7605 3   2   180   8828827889c0 sh pipe_rd
19611 19611 3   5   180   8829098f3600 sh wait
2644  2644 3   4   180   88290633e640pkg_add wait
25464 25464 3  11   180   8828fc982280  mkdir lfs_dirop
11159 11159 3   1   180   8828ff805a80   make wait
451451 3   3   180   8827a6871200 sh wait
2520  2520 3  11   180   882890341140 sh wait
27462 27462 3   5   180   88290dce6040 sh wait
14573 14573 3   0   180   882902b8f8c0 sh wait
19318 19318 3   7   180   

NetBSD-10: write to rw null mount on top of LFS fs hangs up

[Aleksey Cheusov]

Hi. I configured a pkgsrc bulk build on NetBSD-10 virtual machine and
selected LFS file system for temporary directory where object files are
created.

I know the status of LFS is "unknown" but this is not a production
machine. So, I just wanted to play with LFS a little bit.

Virtual machine was run on Linux/KVM as the following:

qemu-system-x86_64 -accel kvm \
-smp cpus=12 \
\
-drive file=netbsd-10.qcow2,if=none,id=hd0 \
-device virtio-blk-pci,drive=hd0 \
\
-drive file=netbsd-10_srv.qcow2,if=none,id=hd1 \
-device virtio-blk-pci,drive=hd1 \
\
-drive file=netbsd-10_newsrv.qcow2,if=none,id=hd2 \
-device virtio-blk-pci,drive=hd2 \
\
-object rng-random,filename=/dev/urandom,id=viornd0 \
-device virtio-rng-pci,rng=viornd0 \
\
-netdev tap,id=vioif0 \
-device virtio-net-pci,netdev=vioif0,mac=00:06:70:A3:85:FC \
\
-m 12288

Packages are built in 18 parallel subprocesses in chroots where /tmp/
directory is null mounted on top of LFS. So, IO was relatively high.
After a few hours of work writing to file system is stuck.

File system was created by newfs_lfs(8) without options.

Currently ps looks like the following

- /usr/bin/make -f Makefile -f /home/cheusov/pkg_distbb/etc/distbb.local.mk -f 
/home/cheusov/pkg_distbb/share/distbb/distbb.mk DEPENDS_TARGET=nonexistant 
BATCH=yes PKG_VERBOSE=1 build
`-- /bin/sh -c set -e;\t\t\t\t\t if test -n "" &&  /usr/sbin/pkg_info -K 
/usr/pkg/pkgdb -qe mariadb-server-10.11.7; then  echo ===\\> "Skipping 
installation of already handled package";  else
  `-- /usr/bin/make _MAKE=/usr/bin/make NATIVE_OPSYS=NetBSD 
NATIVE_OS_VERSION=10.0 NATIVE_OPSYS_VERSION=10 HOST_MACHINE_ARCH=amd64 
NATIVE_LOWER_OPSYS=netbsd _PKGSRCDIR=/srv/pkgsrc/current P
`-- /usr/bin/make -f Makefile all
  `-- /usr/bin/make -s -f CMakeFiles/Makefile2 all
`-- /usr/bin/make -s -f storage/maria/CMakeFiles/aria.dir/build.make 
storage/maria/CMakeFiles/aria.dir/build
  `-- /tmp/obj_pkgsrc/databases/mariadb1011-server/work/.gcc/bin/gcc 
-fcommon -fstack-protector-strong -D_FORTIFY_SOURCE=2 -Wl,-zrelro -fPIC 
-DHAVE_CONFIG_H -DWITH_S3_STORAGE_ENGINE -I/
`-- /usr/libexec/cc1 -quiet -I 
/tmp/obj_pkgsrc/databases/mariadb1011-server/work/mariadb-10.11.7/wsrep-lib/include
 -I /tmp/obj_pkgsrc/databases/mariadb1011-server/work/mariadb-10.11


That's it. It seems that writing to LFS file system hangs up. System itself
is fully functional. Kernel didn't panic.

Is there anyone interested in debugging? I can provide all required
information.

mount:
/dev/ld2a on /srv2 type lfs (local)
/srv2/tmps/tmp1 on /srv/sandboxes/sandbox1-nb10/tmp type null (local)

df:
Filesystem  1K-blocks UsedAvail %Cap Mounted on
/dev/ld2a   300911577 45415971225401031  16% /srv2

uname:
NetBSD builder_nb10 10.0 NetBSD 10.0 (GENERIC) #0: Thu Mar 28 08:33:33
UTC 2024
mkre...@mkrepro.netbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC amd64

(key, value) lookup algos in src/common/lib/libc/gen

[Mathew, Cherry G.]

Hello Kernel people, et. al.,

I've been researching range querying over multi-dimensional keys,
(mainly related to its application in searching for ("delta","epsilon")
tuple timestamp range queries over a database of timers, events.)

This led me to the current key,value pair lookup code in the NetBSD
source so far. Some of what I found seems fragmented, sometimes
specialised, partially documented, possibly in disrepair.

Although daunting, it seems to be a worthy task to consolidate and
document the bits that need some attention/love.

I'm including a set of sources that I have identified - I'm sure I've
left some out. If so, please point me at them.

Network route matching related (key would be IP/prefix address):

- src/sys/net/radix.[ch]
- src/sbin/routed/radix.c
- src/sys/net/npf/lpm.[ch]

string matching:
Focussed on parallelism:
- src/sys/sys/thmap.h
- src/sys/kern/subr_thmap.c

Generic:
N-way Trie:
- src/common/lib/libc/gen/ptree.c
Radix Tr[ie]e
- src/common/lib/libc/gen/radixtree.c
RB Tree:
- src/common/lib/libc/gen/rb.c
Radix Priority Search Tree:
- src/common/lib/libc/gen/rpst.c

I'd like to understand if anyone here has looked
organising/consolidating these, and updating them to the state of the
art anytime recently, and if anyone has suggestions for new additions.

I'd also appreciate pointers to design patterns around a generalised
interface for these, that wouldn't take away from them.

Many Thanks for any comments.

-- 
Math/(~cherry)

Dtrace

[Giacomo Vattini]

Morning to all ,does someone know how to implement all the probes for dtrace as 
in FreeBSD, the oneliners toolkit are not working and everytime that i put as a 
probe TCP IP it gives me errors
Thank you for the suggestions

Re: CVS commit: src/sys/compat/netbsd32

[Martin Husemann]

On Tue, Apr 30, 2024 at 05:10:22PM +, Michael van Elst wrote:
> Module Name:  src
> Committed By: mlelstv
> Date: Tue Apr 30 17:10:22 UTC 2024
> 
> Modified Files:
>   src/sys/compat/netbsd32: netbsd32_sysent.c
> 
> Log Message:
> Enable compat sigreturn system call.

Please check the head of this file - it is not supposed to be manually edited.
Edit sys/compat/netbsd32/syscalls.master instead, commit that and then regen
all relevant files - but it seems that syscalls.master is correct and the
real issue is hidden somewhere else.

This problem is a lot more generic it seems, e.g. the same issue should exist
for the fh* syscalls at 298..300.

Martin

[PATCH] [RFC] i386/i386_mainbus: fix the build when MPBIOS and ACPI are disabled

[Paolo Pisati]

In sys/arch/i386/i386/i386_mainbus.c::i386_mainbus_rescan() mp_pci_scan() is
only executed if:

...
if (npcibus == 0 && mpacpi_active)
npcibus = mp_pci_scan(self, _pba, pcibusprint);
if (npcibus == 0 && mpbios_scanned != 0)
npcibus = mp_pci_scan(self, _pba, pcibusprint);
...

hence follow the same condition during the unwind.

Signed-off-by: Paolo Pisati 
---
 sys/arch/i386/i386/i386_mainbus.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/sys/arch/i386/i386/i386_mainbus.c 
b/sys/arch/i386/i386/i386_mainbus.c
index f6d2f24a593a..2834002e5989 100644
--- a/sys/arch/i386/i386/i386_mainbus.c
+++ b/sys/arch/i386/i386/i386_mainbus.c
@@ -178,7 +178,7 @@ i386_mainbus_childdetached(device_t self, device_t child)
if (sc->sc_pci == child)
sc->sc_pci = NULL;
 
-#if NPCI > 0
+#if NPCI > 0 && (defined(ACPI_SCANPCI) || defined(MPBIOS_SCANPCI))
mp_pci_childdetached(self, child);
 #endif
 }
-- 
2.34.1

[PATCH] x86/intr.c: fix build if NO_PCI_MSI_MSIX is defined

[Paolo Pisati]

...
/home/flag/bsd/netbsd/src/sys/arch/x86/x86/intr.c: In function 
'intr_create_intrid':
/home/flag/bsd/netbsd/src/sys/arch/x86/x86/intr.c:315:34: error: 
'MSI_INT_DEV_MASK' undeclared (first use in this function)
  315 |   pih = __SHIFTIN((uint64_t)dev, MSI_INT_DEV_MASK)
  |  ^~~~
...

Signed-off-by: Paolo Pisati 
---
 sys/arch/x86/x86/intr.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/sys/arch/x86/x86/intr.c b/sys/arch/x86/x86/intr.c
index a94ea7f17707..beb3dab5673b 100644
--- a/sys/arch/x86/x86/intr.c
+++ b/sys/arch/x86/x86/intr.c
@@ -138,6 +138,7 @@ __KERNEL_RCSID(0, "$NetBSD: intr.c,v 1.167 2024/03/05 
20:58:05 andvar Exp $");
 #include "opt_intrdebug.h"
 #include "opt_multiprocessor.h"
 #include "opt_acpi.h"
+#include "opt_pci.h"
 
 #include 
 #include 
-- 
2.34.1

Re: Creating a hackable kernel for AMD64

[Emile `iMil' Heitor]

On Tue, 23 Apr 2024, Taylor R Campbell wrote:


Cool, thanks!  The main thing you're trying to do -- pass a kernel on
the command line to qemu -- doesn't work on x86.


It does with these patches: 
https://github.com/NetBSDfr/NetBSD-src/tree/nbfr_master
They need to be reviewed before I merge them into current.

Cheers,


Emile `iMil' Heitor  | https://imil.net

Re: Creating a hackable kernel for AMD64

[Jared Barnak]

> Cool, thanks!  The main thing you're trying to do -- pass a kernel on
> the command line to qemu -- doesn't work on x86.  It does work on
> aarch64 (arm64), which is why I used that as the example here:

> https://www.netbsd.org/gallery/presentations/riastradh/eurobsdcon2023/getstarted.pdf#page=30

Whoops! Not sure how I missed that. Thanks!

> It also works on riscv and alpha, and possibly some others -- but not
> x86.  So if you want to run x86, it's a little different.  We should
> really reduce these differences, but for now this is what you'll have
> to do:

This makes much more sense. Genuinely appreciate it!

> netbsd-DEBUG_KERNEL.gz is just a kernel, not a whole disk image with
> bootloader, userland, and so on.  What you want is a live disk image.
>
> On aarch64, build.sh release leaves a live disk image in
> $RELEASEDIR/evbarm-aarch64/binary/gzimg/arm64.img.gz, but on x86 the
> live image works differently -- you need to do:

> ./build.sh ... live-image

> and then you'll find it at

> $RELEASEDIR/images/NetBSD-10.99.XXX-amd64-live.img.gz

> This you can just gunzip and pass as a raw disk image to qemu.

This makes it much easier! Is there any reason that:

```
qemu-img create -f qcow2 disk.img 10G
```

> For amd64 we don't build netbsd.img -- the kernel image is just called
> `netbsd'.  (On aarch64, there is a netbsd.img as well as netbsd, and
> some important difference between them that I don't remember the
> details of.)

Okie dokie! Might do my development in VMWare or Virtualbox. In that
case, I might need to:

```
build.sh iso-image-source
```

And just to clarify, `iso-image-source` is needed to retuin the debug
information when the kernel was compiled with `kernel.gdb=DEBUG_KERNEL`
(`DEBUG_KERNEL` being my configuration for a debuggable kernel). Or am
I misunderstanding `src/BUILDING`?

> (Might want to use /bin/sh here, unless you really do have a
> /urs/bin/sh?)

Thanks for catching this! Promptly fixed

> Right -- passing a kernel image on the command line isn't supported
> yet on amd64.  It does work on aarch64 (arm64), riscv, and alpha, but
> not amd64, which is why I added this note:
>
> https://www.netbsd.org/gallery/presentations/riastradh/eurobsdcon2023/getstarted.pdf#page=32
>
> When I'm doing kernel development under qemu, I usually use aarch64 or
> alpha because of this, not x86.
>
> When I do use x86 under qemu, and I need to update the kernel, I serve
> it from the host by serving the objdir via http, and download it into
> the guest with the ftp(1) command, like:
>
> host$ cd ~/netbsd/current/obj.amd64
> host$ /usr/libexec/httpd -b -f -s -I 54321 -X .
>
> guest# ftp -o /netbsd.new 
> http://169.254.123.45:54321/sys/arch/amd64/compile/DEBUG/netbsd
> guest# mv /netbsd /onetbsd
> guest# mv /netbsd.new /netbsd

Super helpful! Thanks so much! I genuinely appreciate the response!

On Tue, Apr 23, 2024 at 11:39 AM Taylor R Campbell
 wrote:
>
> > Date: Tue, 23 Apr 2024 10:52:41 -0400
> > From: Jared Barnak 
> >
> > I watched "How to get started hacking NetBSD" and it was great, I just
> > noticed that some things didn't work when cross compiling for amd64. and
> > frankly, I'm not sure what I did wrong.
>
> Cool, thanks!  The main thing you're trying to do -- pass a kernel on
> the command line to qemu -- doesn't work on x86.  It does work on
> aarch64 (arm64), which is why I used that as the example here:
>
> https://www.netbsd.org/gallery/presentations/riastradh/eurobsdcon2023/getstarted.pdf#page=30
>
> It also works on riscv and alpha, and possibly some others -- but not
> x86.  So if you want to run x86, it's a little different.  We should
> really reduce these differences, but for now this is what you'll have
> to do:
>
> > I unzipped the  `netbsd-DEBUG_KERNEL.gz` file and redirected the output to
> > `vm/disk.img` with:
> >
> > ```
> > gunzip -c  ~/Projects/netbsd/vm/disk.img
> > ```
>
> netbsd-DEBUG_KERNEL.gz is just a kernel, not a whole disk image with
> bootloader, userland, and so on.  What you want is a live disk image.
>
> On aarch64, build.sh release leaves a live disk image in
> $RELEASEDIR/evbarm-aarch64/binary/gzimg/arm64.img.gz, but on x86 the
> live image works differently -- you need to do:
>
> ./build.sh ... live-image
>
> and then you'll find it at
>
> $RELEASEDIR/images/NetBSD-10.99.XXX-amd64-live.img.gz
>
> This you can just gunzip and pass as a raw disk image to qemu.
>
> > And I cannot figure out why, but I don't have a `netbsd.img` under
> >
> > /home/jared/Projects/netbsd/obj/sys/arch/amd64/compile
>
> For amd64 we don't build netbsd.img -- the kernel image is just called
> `netbsd'.  (On aarch64, there is a netbsd.img as well as netbsd, and
> some important difference between them that I don't remember the
> details of.)
>
> > so the following does not work:
> >
> > ```
> > #!/urs/bin/sh
>
> (Might want to use /bin/sh here, unless you really do have a
> /urs/bin/sh?)
>
> > /usr/bin/qemu-system-x86_64 \
> > -kernel netbsd.img \ # This file does 

tty process group use-after-free fix RFC

[Jonathan A. Kollasch]

I've discovered that there's a use-after-free bug that is triggered by
multiply-opened tty(4) devices where one process uses FIOSETOWN and then
exits.  It occurs, for example, when first ntpd(8) opens a GPS_NMEA(0)
refclock (/dev/gps0 -> /dev/ttyU0), and subsequently gpsd(8) opens the
same tty.  ntpd(8) does a FIOSETOWN ioctl on this tty soon after startup
that sets tp->t_pgrp.  (gpsd does not seem to explicitly FIOSETOWN or
TIOCSCTTY.)  Subsequently when you stop the ntpd process (with
gpsd still running) tp->t_pgrp points to freed chunk of memory.  Soon,
ttysigintr() tries to send SIGIO to this process group, and things
explode (the symptom may vary, as the pgrp memory could have been
re-used, or kASan may have intercepted it, or you just segfault in the
kernel).

I see that the TIOCSCTTY case in ttioctl() calls proc_sesshold(), but
the FIOSETOWN or TIOCSPGRP cases, which are vaguely similar, do not.

The attached patch prevents the UAF bug I observed, but I'm not confident
it's correct; and I'm even less confident for the TIOCSPGRP case which I
was unable to test.  I am mostly concerned it may result in a memory leak,
or at worst a deadlock, but perhaps that's better than crashing the kernel.

Jonathan Kollasch
Index: src/sys/kern/tty.c
===
RCS file: /cvsroot/src/sys/kern/tty.c,v
retrieving revision 1.312
diff -d -u -a -p -r1.312 tty.c
--- src/sys/kern/tty.c	7 Dec 2023 09:00:32 -	1.312
+++ src/sys/kern/tty.c	24 Apr 2024 01:58:14 -
@@ -1441,6 +1441,7 @@ unlock_constty:	mutex_exit(_lock
 			return (EPERM);
 		}
 		mutex_spin_enter(_lock);
+		proc_sesshold(pgrp->pg_session);
 		tp->t_pgrp = pgrp;
 		mutex_spin_exit(_lock);
 		mutex_exit(_lock);
@@ -1464,6 +1465,7 @@ unlock_constty:	mutex_exit(_lock
 			return (EPERM);
 		}
 		mutex_spin_enter(_lock);
+		proc_sesshold(pgrp->pg_session);
 		tp->t_pgrp = pgrp;
 		mutex_spin_exit(_lock);
 		mutex_exit(_lock);

Re: Creating a hackable kernel for AMD64

[Taylor R Campbell]

> Date: Tue, 23 Apr 2024 10:52:41 -0400
> From: Jared Barnak 
> 
> I watched "How to get started hacking NetBSD" and it was great, I just
> noticed that some things didn't work when cross compiling for amd64. and
> frankly, I'm not sure what I did wrong.

Cool, thanks!  The main thing you're trying to do -- pass a kernel on
the command line to qemu -- doesn't work on x86.  It does work on
aarch64 (arm64), which is why I used that as the example here:

https://www.netbsd.org/gallery/presentations/riastradh/eurobsdcon2023/getstarted.pdf#page=30

It also works on riscv and alpha, and possibly some others -- but not
x86.  So if you want to run x86, it's a little different.  We should
really reduce these differences, but for now this is what you'll have
to do:

> I unzipped the  `netbsd-DEBUG_KERNEL.gz` file and redirected the output to
> `vm/disk.img` with:
> 
> ```
> gunzip -c  ~/Projects/netbsd/vm/disk.img
> ```

netbsd-DEBUG_KERNEL.gz is just a kernel, not a whole disk image with
bootloader, userland, and so on.  What you want is a live disk image.

On aarch64, build.sh release leaves a live disk image in
$RELEASEDIR/evbarm-aarch64/binary/gzimg/arm64.img.gz, but on x86 the
live image works differently -- you need to do:

./build.sh ... live-image

and then you'll find it at

$RELEASEDIR/images/NetBSD-10.99.XXX-amd64-live.img.gz

This you can just gunzip and pass as a raw disk image to qemu.

> And I cannot figure out why, but I don't have a `netbsd.img` under
> 
> /home/jared/Projects/netbsd/obj/sys/arch/amd64/compile

For amd64 we don't build netbsd.img -- the kernel image is just called
`netbsd'.  (On aarch64, there is a netbsd.img as well as netbsd, and
some important difference between them that I don't remember the
details of.)

> so the following does not work:
> 
> ```
> #!/urs/bin/sh

(Might want to use /bin/sh here, unless you really do have a
/urs/bin/sh?)

> /usr/bin/qemu-system-x86_64 \
> -kernel netbsd.img \ # This file does not exist

Right -- passing a kernel image on the command line isn't supported
yet on amd64.  It does work on aarch64 (arm64), riscv, and alpha, but
not amd64, which is why I added this note:

https://www.netbsd.org/gallery/presentations/riastradh/eurobsdcon2023/getstarted.pdf#page=32

When I'm doing kernel development under qemu, I usually use aarch64 or
alpha because of this, not x86.

When I do use x86 under qemu, and I need to update the kernel, I serve
it from the host by serving the objdir via http, and download it into
the guest with the ftp(1) command, like:

host$ cd ~/netbsd/current/obj.amd64
host$ /usr/libexec/httpd -b -f -s -I 54321 -X .

guest# ftp -o /netbsd.new 
http://169.254.123.45:54321/sys/arch/amd64/compile/DEBUG/netbsd
guest# mv /netbsd /onetbsd
guest# mv /netbsd.new /netbsd

> I'm also not sure what to substitute `-cpu` for when using `amd64`.

No need, default should work fine.

Creating a hackable kernel for AMD64

[Jared Barnak]

Hey,

I watched "How to get started hacking NetBSD" and it was great, I just
noticed that some things didn't work when cross compiling for amd64. and
frankly, I'm not sure what I did wrong.


I use the following script to build the kernel

```
#/usr/bin/sh

export DESTDIR=/home/jared/Projects/netbsd/obj/destdir
export RELEASEDIR=/home/jared/Projects/netbsd/obj/releasedir
export
KERNDIR=/home/jared/Projects/netbsd/obj/sys/arch/amd64/compile/DEBUG_KERNEL


cd src/
./build.sh -u -O ../obj -U -m amd64 -N1 -j24 -V MKCROSSGDB=yes tools
kernel.gdb=DEBUG_KERNEL

cd ../
```

And this creates the following directory

```
/home/jared/Projects/netbsd/obj/releasedir/amd64/binary/kernel/DEBUG_KERNEL
├── MD5
├── netbsd-DEBUG_KERNEL.gz
├── netbsd-GENERIC.gz
├── netbsd-GENERIC_KASLR.gz
├── netbsd-GENERIC.symbols.gz
├── netbsd-INSTALL.gz
├── netbsd-INSTALL.symbols.gz
├── netbsd-INSTALL_XEN3_DOMU.gz
├── netbsd-XEN3_DOM0.gz
├── netbsd-XEN3_DOMU.gz
└── SHA512
```

I unzipped the  `netbsd-DEBUG_KERNEL.gz` file and redirected the output to
`vm/disk.img` with:

```
gunzip -c  ~/Projects/netbsd/vm/disk.img
```

And I created a sparse disk image with (note: `-oseek` is not a valid
switch on my machine and `10g` didn't work, so I substituted it with `2^30`
which is `1073741824` ):

```
$ dd if=/dev/zero of=disk.img seek=1073741824 bs=1 count=1
1+0 records in
1+0 records out
1 byte copied, 0.000180831 s, 5.5 kB/s
```

And I cannot figure out why, but I don't have a `netbsd.img` under

```
/home/jared/Projects/netbsd/obj/sys/arch/amd64/compile
```

so the following does not work:

```
#!/urs/bin/sh

/usr/bin/qemu-system-x86_64 \
-kernel netbsd.img \ # This file does not exist
-machine 'ubuntu'
-append "root=dk1" \
-M virt \
-smp 2 \
-m 1g \
-drive if=none,file=disk.img,id=disk,format=raw \
-device virtio-blk-device,drive=disk \
-device virtio-rng-device \
-nic user,model=virtio-net-pci \
-nographic
```

And when I run that, I get:

```
qemu: could not open kernel file 'netbsd.img': No such file or directory
```

I'm also not sure what to substitute `-cpu` for when using `amd64`.

Some guidance would be great!

just one of five disks detected at arcmsr(4) since upgrade to NetBSD 10

[Bert Kiers]

Hi,

This morning I upgraded a box from 9.3_STABLE to NetBSD 10.0. With 
NetBSD 9.3 it probed five disks at the Areca ARC-1222 RAID controller, 
now just one. A powercycle and "scsictl scsibus0 scan all all" did not help.



NetBSD 9.3:

Apr 18 12:22:20 janeway /netbsd: [   1.0262686] arcmsr0 at pci1 dev 0 
function 0
Apr 18 12:22:20 janeway /netbsd: [   1.0262686] arcmsr0: interrupting at 
ioapic0 pin 16
Apr 18 12:22:20 janeway /netbsd: [   1.0262686] arcmsr0: Areca ARC-1222 
Host Adapter RAID controller
Apr 18 12:22:20 janeway /netbsd: [   1.0262686] arcmsr0: 8 ports, 256MB 
SDRAM, firmware 
Apr 18 12:22:20 janeway /netbsd: [   1.0262686] scsibus0 at arcmsr0: 16 
targets, 8 luns per target

...
Apr 18 12:22:20 janeway /netbsd: [   1.2718484] sd0 at scsibus0 target 0 
lun 0:  disk fixed
Apr 18 12:22:20 janeway /netbsd: [   1.2718484] sd0: 3726 GB, 64600 cyl, 
252 head, 480 sec, 512 bytes/sect x 7814037168 sectors
Apr 18 12:22:20 janeway /netbsd: [   1.2718484] sd1 at scsibus0 target 0 
lun 1:  disk fixed
Apr 18 12:22:20 janeway /netbsd: [   1.2718484] sd1: 3726 GB, 64600 cyl, 
252 head, 480 sec, 512 bytes/sect x 7814037168 sectors
Apr 18 12:22:20 janeway /netbsd: [   1.2718484] sd2 at scsibus0 target 0 
lun 2:  disk fixed
Apr 18 12:22:20 janeway /netbsd: [   1.2718484] sd2: 3726 GB, 64600 cyl, 
252 head, 480 sec, 512 bytes/sect x 7814037168 sectors
Apr 18 12:22:20 janeway /netbsd: [   1.2718484] sd3 at scsibus0 target 0 
lun 3:  disk fixed
Apr 18 12:22:20 janeway /netbsd: [   1.2718484] sd3: 3726 GB, 64600 cyl, 
252 head, 480 sec, 512 bytes/sect x 7814037168 sectors
Apr 18 12:22:20 janeway /netbsd: [   1.2718484] sd4 at scsibus0 target 0 
lun 4:  disk fixed
Apr 18 12:22:20 janeway /netbsd: [   1.2718484] sd4: 3726 GB, 64600 cyl, 
252 head, 480 sec, 512 bytes/sect x 7814037168 sectors
Apr 18 12:22:20 janeway /netbsd: [   1.2811828] autoconfiguration error: 
arcmsr0: unable to query firmware for sensor info



NetBSD 10.0:

Apr 18 14:08:20 janeway /netbsd: [   1.0299139] arcmsr0 at pci1 dev 0 
function 0
Apr 18 14:08:20 janeway /netbsd: [   1.0299139] arcmsr0: interrupting at 
ioapic0 pin 16
Apr 18 14:08:20 janeway /netbsd: [   1.0299139] arcmsr0: Areca ARC-1222 
Host Adapter RAID controller
Apr 18 14:08:20 janeway /netbsd: [   1.0299139] arcmsr0: 8 ports, 256MB 
SDRAM, firmware 
Apr 18 14:08:20 janeway /netbsd: [   1.0299139] scsibus0 at arcmsr0: 16 
targets, 8 luns per target

...
Apr 18 14:08:20 janeway /netbsd: [   1.0700815] sd0 at scsibus0 target 0 
lun 0:  disk fixed
Apr 18 14:08:20 janeway /netbsd: [   1.0700815] sd0: 3726 GB, 64600 cyl, 
252 head, 480 sec, 512 bytes/sect x 7814037168 sectors

...
Apr 18 14:08:20 janeway /netbsd: [   1.0765269] autoconfiguration error: 
arcmsr0: unable to query firmware for sensor info



root@janeway:/home/kiers# sysctl hw.disknames
hw.disknames = sd0 wd0 wd1


The card has ethernet with a webinterface and everything is fine there.
The five disks are a zpool, but that seems irrelevant.


Any ideas before I send-pr?

Thanks,
Bert

Re: [RFC] new APIs to use wskbd(4) input on non-wsdisplay tty devices

[Izumi Tsutsui]

tls@ wrote:

> On Fri, Apr 12, 2024 at 09:13:17AM -0400, Thor Lancelot Simon wrote:
> > On Sat, Apr 06, 2024 at 11:56:27PM +0900, Izumi Tsutsui wrote:
> > > 
> > > I'd like to add new APIs to use wskbd(4) input on non-wsdisplay
> > > tty devices, especially news68k that can use a putchar function
> > > provided by firmware PROM as a kernel console device.
> > 
> > Wouldn't a tty be a better abstraction for this? Lots of minicomputers
> > had incredibly dumb serial consoles.
> 
> I think the above is not clear.  To rephrase my question: what is the
> wscons layer actually adding here?  Would it not be simpler to interface
> at the tty layer instead?

- The tty layer requires both getchar() and putchar().

- For a glass console, completely different two drivers are necessary,
  i.e. keyboard driver for getchar() and display driver for putchar().

- On news68k, it's not easy to implement framebuffer driver due to
  lack of information, but we can use a putchar() function provided
  by a NEWS PROM firmware because it also supports VT emulation.

- On the other hand, we cannot use getchar() function provided by a
  NEWS PROM firmware for useland because it uses busy loop to wait
  keyboard input. But we already has a wskbd driver for NEWS machines.

- So I would like propose adding APIs "to use wskbd driver just for
  keyboard inputs" for non-wscons(4) tty drivers

See sys/arch/sun3/dev/kd.c for another example that use
"PROM function for putchar() to display and
 sys/dev/sun/kbd.c via zs(4) for getchar() from keyboard".

 https://github.com/NetBSD/src/blob/6053b8d/sys/arch/sun3/dev/kd.c
---
static void 
kdstart(struct tty *tp)
{
struct clist *cl;
int s1, s2;

s1 = splsoftclock();
s2 = spltty();
if (tp->t_state & (TS_BUSY|TS_TTSTOP|TS_TIMEOUT))
goto out;

cl = >t_outq;
if (ttypull(tp)) {
if (kd_is_console) {
tp->t_state |= TS_BUSY;
if ((s1 & PSL_IPL) == 0) {
/* called at level zero - update screen now. */
splx(s2);
kd_putfb(tp);

[...]
}

[...]

/*
 * Put text on the screen using the PROM monitor.
 * This can take a while, so to avoid missing
 * interrupts, this is called at splsoftclock.
 */
static void 
kd_putfb(struct tty *tp)
{
char buf[PUT_WSIZE];
struct clist *cl = >t_outq;
char *p, *end;
int len;

while ((len = q_to_b(cl, buf, PUT_WSIZE-1)) > 0) {
/* PROM will barf if high bits are set. */
p = buf;
end = buf + len;
while (p < end)
*p++ &= 0x7f;
(romVectorPtr->fbWriteStr)(buf, len);
}
}

[...]

/*
 * Our "interrupt" routine for input. This is called by
 * the keyboard driver (dev/sun/kbd.c) at spltty.
 */
void 
kd_cons_input(int c)
{
struct kd_softc *kd = _softc;
struct tty *tp;

/* XXX: Make sure the device is open. */
tp = kd->kd_tty;
if (tp == NULL)
return;
if ((tp->t_state & TS_ISOPEN) == 0)
return;

(*tp->t_linesw->l_rint)(c, tp);
}
---

sys/arch/news68k/news68k/romcons.c has the similar implementation:
 
https://github.com/tsutsui/netbsd-src/blob/c74b64a/sys/arch/news68k/news68k/romcons.c

---
static void
romcons_start(struct tty *tp)
{
int s, len;
uint8_t buf[BURSTLEN];

s = spltty();
if (tp->t_state & (TS_TIMEOUT | TS_BUSY | TS_TTSTOP)) {
splx(s);
return;
}
tp->t_state |= TS_BUSY;
splx(s);
len = q_to_b(>t_outq, buf, BURSTLEN);
s = splhigh();
rom_write(1, buf, len);
splx(s);
s = spltty();
tp->t_state &= ~TS_BUSY;
if (ttypull(tp)) {
tp->t_state |= TS_TIMEOUT;
callout_schedule(>t_rstrt_ch, 1);
}
splx(s);
}

[...]

#if NWSKBD > 0
static void
romcons_kbdinput(device_t self, int ks)
{
struct romcons_softc *sc = device_private(self);
struct tty *tp;

KASSERT(sc != NULL);
tp = sc->sc_tty;
if (tp && (tp->t_state & TS_ISOPEN))
(*tp->t_linesw->l_rint)(ks, tp);
}
#endif
---
Izumi Tsutsui

Re: [RFC] new APIs to use wskbd(4) input on non-wsdisplay tty devices

[Thor Lancelot Simon]

On Fri, Apr 12, 2024 at 09:13:17AM -0400, Thor Lancelot Simon wrote:
> On Sat, Apr 06, 2024 at 11:56:27PM +0900, Izumi Tsutsui wrote:
> > 
> > I'd like to add new APIs to use wskbd(4) input on non-wsdisplay
> > tty devices, especially news68k that can use a putchar function
> > provided by firmware PROM as a kernel console device.
> 
> Wouldn't a tty be a better abstraction for this? Lots of minicomputers
> had incredibly dumb serial consoles.

I think the above is not clear.  To rephrase my question: what is the
wscons layer actually adding here?  Would it not be simpler to interface
at the tty layer instead?

Thor

Re: [RFC] new APIs to use wskbd(4) input on non-wsdisplay tty devices

[Thor Lancelot Simon]

On Sat, Apr 06, 2024 at 11:56:27PM +0900, Izumi Tsutsui wrote:
> 
> I'd like to add new APIs to use wskbd(4) input on non-wsdisplay
> tty devices, especially news68k that can use a putchar function
> provided by firmware PROM as a kernel console device.

Wouldn't a tty be a better abstraction for this? Lots of minicomputers
had incredibly dumb serial consoles.

NetBSD-10.0 virtual memory improvements

[Sad Clouds]

Hello, I was interested to read the following release notes:

Various optimizations for the machine-independent virtual memory system:
- Switched to a faster radix tree algorithm for memory page lookups.
- Improved tracking of clean/dirty pages, speeding up fsync(2) on large
  files by orders of magnitude.
- Improved parallelization: rewritten page allocator with awareness of
  CPU topology, replaced global counters with per-CPU counters, and
  reduced lock contention.

I ran a quick performance test on the scalability of NetBSD-10.0 with
regard to handling page faults on x86-64 for up to 8 concurrent
threads. To be honest, I'm not noticing any improvements compared to
NetBSD-9.3. Compared to Linux, FreeBSD and Solaris, NetBSD is way at
the bottom in terms of paging performance.

Is there a specific sysctl that needs to be enabled or the "improved
parallelization" above is referring to some other aspect of page
handling?

I'm not subscribed to this list, please CC me in your replies.

Thanks.

Re: tickless kernel - high level roadmap ideas

[Mathew, Cherry G.*]

> On Wed, 10 Apr 2024 12:37:31 +, Ice Cream  
> said:

>> I've been working a little bit on how to tackle tickless in the
>> kernel. I'll be hoping to send some preliminary patches later this month
>> or so - so this is mostly an email to co-ordinate with anyone else
>> working or aiming to work on similar things.

> Hi, I'm aiming to work on this project as well.

>> One of the first things needed will be to discover and manage underlying
>> hardware timers which can interrupt at arbitrary (and/or repetitive)
>> timeouts. Therefore, one of the first things I'll be working on is how
>> to abstract this management. Since we need to support a variety of
>> platform/CPU/CPU-package configurations, this will be the part with a
>> large number of cross platform dependencies.

> Since you'll be working on this, I'll start with researching about the next 
> step,
> that is, the callout variants using bintime and the usage of the new timer 
> API.

Brilliant! Let's take the details offline, so that we don't spam the
list. I'll email you separately.

-- 
Math/(~cherry)

Re: [RFC] new APIs to use wskbd(4) input on non-wsdisplayttydevices

[Izumi Tsutsui]

uwe@ wrote:

> > On the other hand, news68k (and sun) machines have putchar()
> > that also handles virtual terminal ops like backspace, CR/LF,
> > and even scrolling at the bottom of screen. In this case
> > no VT emulation layer is necessary in the kernel side,
> > so kernel's putc(9) just calls firmware's putchar(),
> > and for userland processes we can simply pass translated
> > wskbd inputs to line discipline of the tty device.
> > 
> > That's the reason why I proposed to add register/deregister
> > APIs to pass wskbd data to romcons tty device.
> > 
> > What do you think about this case?
> 
> Add trivial wsemul_none (or wsemul_delegate, or whatever a suitable
> name might be) that does even less than wsemul_dumb and only ever uses
> putchar to pass chars to the firmware emulator?

I considered about it when I tried to implement news68k ROM console
driver, but I gave it up because:

- wscons(9) was the most complicated driver with undocumented APIs
  (struct wsemul_ops etc.)
- wsdisplay(9) would pull a lot of unnecessary sources like wsdisplay.c
  etc. (for screen ops) while TERM=wsvt25 could not be used
- there was no sample implementation using "putchar() with VT emulation"
- on the other hand there were simple implementation for PROM console
  functions (sys/arch/sun3/dev/kd.c and sys/dev/ofw/ofcons.c etc.)
  and actually it was quite simple to connect wskbd(4) to "romcons"
  driver

Thanks,
---
Izumi Tsutsui

Re: tickless kernel - high level roadmap ideas

[Ice Cream]

> I've been working a little bit on how to tackle tickless in the
> kernel. I'll be hoping to send some preliminary patches later this month
> or so - so this is mostly an email to co-ordinate with anyone else
> working or aiming to work on similar things.

Hi, I'm aiming to work on this project as well.

> One of the first things needed will be to discover and manage underlying
> hardware timers which can interrupt at arbitrary (and/or repetitive)
> timeouts. Therefore, one of the first things I'll be working on is how
> to abstract this management. Since we need to support a variety of
> platform/CPU/CPU-package configurations, this will be the part with a
> large number of cross platform dependencies.

Since you'll be working on this, I'll start with researching about the next 
step,
that is, the callout variants using bintime and the usage of the new timer API.

-- IC

tickless kernel - high level roadmap ideas

[Mathew, Cherry G.*]

Hello NetBSD,

I've been working a little bit on how to tackle tickless in the
kernel. I'll be hoping to send some preliminary patches later this month
or so - so this is mostly an email to co-ordinate with anyone else
working or aiming to work on similar things.

The only discernable change from an API consumption perspective for the
rest of the kernel would be the appearance of bintime(9) variants of
callout(9). Functionally, when tickless is enabled finally, any
APIs which depend on sleep/wakeup semantics will have finer than 1/hz
granularity. I can't think of any other discernable consumer facing
changes at this point - feel free to mention them.

To implement tickless however, things under the hood will need to change
quite a bit.

One of the first things needed will be to discover and manage underlying
hardware timers which can interrupt at arbitrary (and/or repetitive)
timeouts. Therefore, one of the first things I'll be working on is how
to abstract this management. Since we need to support a variety of
platform/CPU/CPU-package configurations, this will be the part with a
large number of cross platform dependencies.

Once this API is in place, the callout variants using bintime, described
above, will need to be enabled to use this timer API in order to provide
sub 1/hz granularity.

Finally, consumers such as sleep/wakeup providers (cv_timedwaitbt(9),
kpause(9) etc) and a whole bunch of other functions which depend upwards
on these and others will need to be enabled to use the new sub 1/hz
callout APIs.

>From my initial experiments, and discussions with core@ , it looks like
all this can be done mostly without changing the current hardclock(9)
semantics - those semantics can simply be converted to use the new timer
management API described above.

Finally, the system time model can be updated to use high-res timers - I
won't go into detail here, because it's a bit premature to speculate
what that might look like - there seem to be a bunch of dependencies,
including into CSF(9), softint(9), heartbeat(9), and anything that polls
for timestamp updates (eg: timecounter(9) ).

Looking forward to your comments and thoughts.

Best,
-- 
Math/(~cherry)

Re: [RFC] new APIs to use wskbd(4) input on non-wsdisplayttydevices

[Valery Ushakov]

On Mon, Apr 08, 2024 at 23:27:11 +0900, Izumi Tsutsui wrote:

> macallan@ wrote:
[...]
> > Oh, so it's an entire terminal emulation, not just something that lets
> > you draw characters?
> 
> Ah, maybe I see misunderstandings among us.
> 
> In sgimips crmfb and newport cases, a putchar() function provided
> by the firmware just draws a character (glyph) at the cursor
> or specified position. All virtual terminal emulation ops are
> done in wsdisplay(9) and vcons(9) layer and MD drivers just draw
> characters (or whitespace) per vcons pseudo text VRAM attributes.
> 
> On the other hand, news68k (and sun) machines have putchar()
> that also handles virtual terminal ops like backspace, CR/LF,
> and even scrolling at the bottom of screen. In this case
> no VT emulation layer is necessary in the kernel side,
> so kernel's putc(9) just calls firmware's putchar(),
> and for userland processes we can simply pass translated
> wskbd inputs to line discipline of the tty device.
> 
> That's the reason why I proposed to add register/deregister
> APIs to pass wskbd data to romcons tty device.
> 
> What do you think about this case?

Add trivial wsemul_none (or wsemul_delegate, or whatever a suitable
name might be) that does even less than wsemul_dumb and only ever uses
putchar to pass chars to the firmware emulator?

-uwe

Re: [RFC] new APIs to use wskbd(4) input on non-wsdisplayttydevices

[Izumi Tsutsui]

macallan@ wrote:

> On Sun, 7 Apr 2024 14:49:42 +0900
> Izumi Tsutsui  wrote:
> 
> > macallan@ wrote:
> > 
> > > > I'd like to add new APIs to use wskbd(4) input on non-wsdisplay
> > > > tty devices, especially news68k that can use a putchar function
> > > > provided by firmware PROM as a kernel console device.
> > > > 
> > > > # Details
> > > > 
> > > > The news68k machines have various framebuffer options, but
> > > > most of them are not "dumb" framebuffer that can be handled
> > > > by raspos(9) but some of them have own processors (68000 etc.)
> > > > to handle various acceleration etc. so it's difficult to implement
> > > > framebuffer drivers.  
> > > 
> > > You can use wsdisplay without rasops or a mappable framebuffer.
> > > Examples would be things like newport or crmfb on sgimips.  
> > 
> > It looks both newport and crmfb has copy and erase ops.
> > As I wrote another post, news68k PROM only provides putchar() function
> > (At least there is no info other framebuffer ops).
> 
> There is already code in vcons which uses putchar() to implement
> copy*() ( to avoid slow fb reads ), using it for erase*() is trivial.
> 
> > This means all pseudo terminal ops (backspace, clear screen etc.)
> > are handled by the PROM.  I still wonder if wscons(9) provides
> > proper interface in such case.
> 
> With vcons you can implement pretty much everything using just putchar
> - since there's a character and attribute buffer you just do your
> operations on the buffer and update the screen character by character.
> For copy operations on dumb framebuffers this is usually faster since
> it avoids framebuffer reads which can be painfully slow. Genfb uses a
> lot of tricks like that.
> Not the intended use when I wrote it, but it's a welcome side effect.
> In fact I wrote colour and virtual console support for STI like that,
> didn't commit it because it interferes with the early console support
> which I have no way to test. STI has no inverse operation ( or any ROP
> support at all )  so I implemented cursor() using putchar.
> 
> > > > To support "text only" framebuffer console, we can use putchar
> > > > functions provided by the firmware PROM.  
> > > 
> > > Does it let you draw characters wherever you want with colours of
> > > inverse attrubute?
> > > If so, that's enough for wsdisplay - sti on hppa does just that.   
> > 
> > Probably no color or inverse ops, unless PROM putchar() supports
> > proper escape sequence of them.
> 
> Oh, so it's an entire terminal emulation, not just something that lets
> you draw characters?

Ah, maybe I see misunderstandings among us.

In sgimips crmfb and newport cases, a putchar() function provided
by the firmware just draws a character (glyph) at the cursor
or specified position. All virtual terminal emulation ops are
done in wsdisplay(9) and vcons(9) layer and MD drivers just draw
characters (or whitespace) per vcons pseudo text VRAM attributes.

On the other hand, news68k (and sun) machines have putchar()
that also handles virtual terminal ops like backspace, CR/LF,
and even scrolling at the bottom of screen. In this case
no VT emulation layer is necessary in the kernel side,
so kernel's putc(9) just calls firmware's putchar(),
and for userland processes we can simply pass translated
wskbd inputs to line discipline of the tty device.

That's the reason why I proposed to add register/deregister
APIs to pass wskbd data to romcons tty device.

What do you think about this case?

Thanks,
---
Izumi Tsutsui

Re: fullfs

[Igor Kolesnik]

> On 6 Apr 2024, at 11:26, Ice Cream  wrote:
> 
> Hi, I already worked on this. You can check the patch here
> https://mail-index.netbsd.org/tech-kern/2024/03/30/msg029539.html
> 
>> I've copied nullfs to fullfs.  I did "make" and nothing happed,
>> I tried 'build.sh' and nothing, like 'fullfs' does not exist.
> 
> Did you add fullfs to the Makefile(s)?
> 

I didn't push it so hard.  From your patch I can see that there are many
places I had to look for.

igor

Re: Kernel work that needs help?

[Hubert Feyrer]

Maybe habe a Look at our Problem Report (PR) database.

 - Hubert


> Am 07.04.2024 um 22:00 schrieb Ice Cream :
> 
> I would like to contribute to development and am looking
> for ongoing kernel dev work that needs help.
> 
> I worked on one of the projects (fullfs) from the project list
> and am looking for more. Most projects on the list are years
> old or not with clear definition or not of my general interest.
> 
> I would like to join projects that are being actively worked
> on and have well defined milestones for someone relatively
> inexperienced. So, if you're working on something awesome
> then let me know :-)
> 
> -- IC

Kernel work that needs help?

[Ice Cream]

I would like to contribute to development and am looking
for ongoing kernel dev work that needs help.

I worked on one of the projects (fullfs) from the project list
and am looking for more. Most projects on the list are years
old or not with clear definition or not of my general interest.

I would like to join projects that are being actively worked
on and have well defined milestones for someone relatively
inexperienced. So, if you're working on something awesome
then let me know :-)

-- IC

Re: [RFC] new APIs to use wskbd(4) input on non-wsdisplay ttydevices

[Michael]

On Sun, 7 Apr 2024 14:49:42 +0900
Izumi Tsutsui  wrote:

> macallan@ wrote:
> 
> > > I'd like to add new APIs to use wskbd(4) input on non-wsdisplay
> > > tty devices, especially news68k that can use a putchar function
> > > provided by firmware PROM as a kernel console device.
> > > 
> > > # Details
> > > 
> > > The news68k machines have various framebuffer options, but
> > > most of them are not "dumb" framebuffer that can be handled
> > > by raspos(9) but some of them have own processors (68000 etc.)
> > > to handle various acceleration etc. so it's difficult to implement
> > > framebuffer drivers.  
> > 
> > You can use wsdisplay without rasops or a mappable framebuffer.
> > Examples would be things like newport or crmfb on sgimips.  
> 
> It looks both newport and crmfb has copy and erase ops.
> As I wrote another post, news68k PROM only provides putchar() function
> (At least there is no info other framebuffer ops).

There is already code in vcons which uses putchar() to implement
copy*() ( to avoid slow fb reads ), using it for erase*() is trivial.

> This means all pseudo terminal ops (backspace, clear screen etc.)
> are handled by the PROM.  I still wonder if wscons(9) provides
> proper interface in such case.

With vcons you can implement pretty much everything using just putchar
- since there's a character and attribute buffer you just do your
operations on the buffer and update the screen character by character.
For copy operations on dumb framebuffers this is usually faster since
it avoids framebuffer reads which can be painfully slow. Genfb uses a
lot of tricks like that.
Not the intended use when I wrote it, but it's a welcome side effect.
In fact I wrote colour and virtual console support for STI like that,
didn't commit it because it interferes with the early console support
which I have no way to test. STI has no inverse operation ( or any ROP
support at all )  so I implemented cursor() using putchar.

> > > To support "text only" framebuffer console, we can use putchar
> > > functions provided by the firmware PROM.  
> > 
> > Does it let you draw characters wherever you want with colours of
> > inverse attrubute?
> > If so, that's enough for wsdisplay - sti on hppa does just that.   
> 
> Probably no color or inverse ops, unless PROM putchar() supports
> proper escape sequence of them.

Oh, so it's an entire terminal emulation, not just something that lets
you draw characters?

have fun
Michael

Re: [RFC] new APIs to use wskbd(4) input on non-wsdisplay ttydevices

[Izumi Tsutsui]

macallan@ wrote:

> > I'd like to add new APIs to use wskbd(4) input on non-wsdisplay
> > tty devices, especially news68k that can use a putchar function
> > provided by firmware PROM as a kernel console device.
> > 
> > # Details
> > 
> > The news68k machines have various framebuffer options, but
> > most of them are not "dumb" framebuffer that can be handled
> > by raspos(9) but some of them have own processors (68000 etc.)
> > to handle various acceleration etc. so it's difficult to implement
> > framebuffer drivers.
> 
> You can use wsdisplay without rasops or a mappable framebuffer.
> Examples would be things like newport or crmfb on sgimips.

It looks both newport and crmfb has copy and erase ops.
As I wrote another post, news68k PROM only provides putchar() function
(At least there is no info other framebuffer ops).

This means all pseudo terminal ops (backspace, clear screen etc.)
are handled by the PROM.  I still wonder if wscons(9) provides
proper interface in such case.

kd_cons_input() in sys/arch/sun3/dev/kd.c has the similar storategy
 https://github.com/NetBSD/src/blob/2722c57/sys/arch/sun3/dev/kd.c#L401-L419
 https://github.com/NetBSD/src/blob/2722c57/sys/dev/sun/kbd.c#L733-L746
but sun3 port doesn't use wscons. (yet?)

> > To support "text only" framebuffer console, we can use putchar
> > functions provided by the firmware PROM.
> 
> Does it let you draw characters wherever you want with colours of
> inverse attrubute?
> If so, that's enough for wsdisplay - sti on hppa does just that. 

Probably no color or inverse ops, unless PROM putchar() supports
proper escape sequence of them.

Thanks,
---
Izumi Tsutsui

Re: [RFC] new APIs to use wskbd(4) input on non-wsdisplay tty devices

[Izumi Tsutsui]

uwe@ wrote:

> On Sat, Apr 06, 2024 at 23:56:27 +0900, Izumi Tsutsui wrote:
> 
> > To support "text only" framebuffer console, we can use putchar
> > functions provided by the firmware PROM.
> 
> Is that a console-typewriter--like device without addressable cursor
> terminal emulation?  Can you use wsemul_dumb to avoid rasops ?  It
> still uses copy/erase, but with trivial argument values that can be
> reversed back to puchar calls for tab/lf (from a very quick look).

The PROM provides (at least I can use) only a "putchar()" function.
No other framebuffer ops like bitblt.
For now I cannot find any proper implementation example in such case..

> > The attached patch provides new two APIs
> > - wskbd_consdev_kbdinput_register()
> > - wskbd_consdev_kbdinput_deregister()
> > to allow a kernel to use wskbd(9) for non-wsdisplay tty device.
> 
> AFAIU, there's nothing console-specific in this (except that it's
> first use is going to be for a console), so may be it would be better
> to drop the "consdev" from the name?

Hmm.

Actually it is not a "console" device (by sys/dev/cons.c) of the kernel,
but I just intended it as a "screen console (not a serial one)".

What name should we use in such case?
dispdev? scrndev? putchar?
"wskbd_ttyinput_register()" and "ttydev" (for members) or something?

> > Index: sys/dev/wscons/wskbd.c
> > ===
> > RCS file: /cvsroot/src/sys/dev/wscons/wskbd.c,v
> > retrieving revision 1.143
> > diff -u -p -d -r1.143 wskbd.c
> > --- sys/dev/wscons/wskbd.c  5 Feb 2019 10:04:49 -   1.143
> > +++ sys/dev/wscons/wskbd.c  6 Apr 2024 06:59:50 -
> [...]
> > @@ -706,6 +709,24 @@ wskbd_input(device_t dev, u_int type, in
> > }
> >  #endif
> >  
> > +#if NWSDISPLAY == 0
> > +   if (sc->sc_translating) {
> 
> The #endif above is for NWSDISPLAY > 0, so may be get rid of the
> ifdefs and use plain ifs?

It looks wskbd(9) assumes always attached to wsdisplay(9) (ttyE?)
if wsdisplay(4) is configured (in the above #if NWSDISPLAY >0 block)
even if no wsdisplay(4) devices are actually attached.

In my news68k case, I thought we should not override the current
assumptions even in the wskbd_consdev_kbdinput_register() case.

I have a feeling we should have separate 'if (sc->sc_translating)'
blocks for both case. Maybe we can simply use #else like:
#if NWSDISPLAY > 0
 [send kbd inputs to wsdisplay_kbdinput() and return unconditionally]
#else
 [send kbd inputs to sc_consdev_kbdinput() and return if it's registered]
#endif

Thanks,
---
Izumi Tsutsui

Re: [RFC] new APIs to use wskbd(4) input on non-wsdisplay tty devices

[Michael]

Hello,

On Sat, 6 Apr 2024 23:56:27 +0900
Izumi Tsutsui  wrote:

> I'd like to add new APIs to use wskbd(4) input on non-wsdisplay
> tty devices, especially news68k that can use a putchar function
> provided by firmware PROM as a kernel console device.
> 
> # Details
> 
> The news68k machines have various framebuffer options, but
> most of them are not "dumb" framebuffer that can be handled
> by raspos(9) but some of them have own processors (68000 etc.)
> to handle various acceleration etc. so it's difficult to implement
> framebuffer drivers.

You can use wsdisplay without rasops or a mappable framebuffer.
Examples would be things like newport or crmfb on sgimips.

> To support "text only" framebuffer console, we can use putchar
> functions provided by the firmware PROM.

Does it let you draw characters wherever you want with colours of
inverse attrubute?
If so, that's enough for wsdisplay - sti on hppa does just that. 

have fun
Michael

Re: [RFC] new APIs to use wskbd(4) input on non-wsdisplay tty devices

[Valery Ushakov]

On Sat, Apr 06, 2024 at 23:56:27 +0900, Izumi Tsutsui wrote:

> To support "text only" framebuffer console, we can use putchar
> functions provided by the firmware PROM.

Is that a console-typewriter--like device without addressable cursor
terminal emulation?  Can you use wsemul_dumb to avoid rasops ?  It
still uses copy/erase, but with trivial argument values that can be
reversed back to puchar calls for tab/lf (from a very quick look).


> The attached patch provides new two APIs
> - wskbd_consdev_kbdinput_register()
> - wskbd_consdev_kbdinput_deregister()
> to allow a kernel to use wskbd(9) for non-wsdisplay tty device.

AFAIU, there's nothing console-specific in this (except that it's
first use is going to be for a console), so may be it would be better
to drop the "consdev" from the name?


> Index: sys/dev/wscons/wskbd.c
> ===
> RCS file: /cvsroot/src/sys/dev/wscons/wskbd.c,v
> retrieving revision 1.143
> diff -u -p -d -r1.143 wskbd.c
> --- sys/dev/wscons/wskbd.c5 Feb 2019 10:04:49 -   1.143
> +++ sys/dev/wscons/wskbd.c6 Apr 2024 06:59:50 -
[...]
> @@ -706,6 +709,24 @@ wskbd_input(device_t dev, u_int type, in
>   }
>  #endif
>  
> +#if NWSDISPLAY == 0
> + if (sc->sc_translating) {

The #endif above is for NWSDISPLAY > 0, so may be get rid of the
ifdefs and use plain ifs?

Thanks.

-uwe

[RFC] new APIs to use wskbd(4) input on non-wsdisplay tty devices

[Izumi Tsutsui]

# tl;dr

I'd like to add new APIs to use wskbd(4) input on non-wsdisplay
tty devices, especially news68k that can use a putchar function
provided by firmware PROM as a kernel console device.

# Details

The news68k machines have various framebuffer options, but
most of them are not "dumb" framebuffer that can be handled
by raspos(9) but some of them have own processors (68000 etc.)
to handle various acceleration etc. so it's difficult to implement
framebuffer drivers.

To support "text only" framebuffer console, we can use putchar
functions provided by the firmware PROM.

However, to use getchar() functions we cannot use getchar
provided by PROM because it always stalls until any key is
pressed.

NetBSD/sun3 has the similar framework in sys/arch/sun3/dev/kd.c,
i.e. it uses a PROM function for putchar but use own keyboard
driver (sys/dev/sun/kbd.c) to handle kernel console.

I'd like to implement the similar framework on NetBSD/news68k.
news68k port already has wskbd driver (delived from newsmips port),
but currently a design of wskbd(9) assumes always it used with
wsdisplay(9) tty (i.e. ttyE?) device and there is no API to use
wskbd(9) input for other drivers than wsdisplay(9).

The attached patch provides new two APIs
- wskbd_consdev_kbdinput_register()
- wskbd_consdev_kbdinput_deregister()
to allow a kernel to use wskbd(9) for non-wsdisplay tty device.

Once "sc_consdev_kbdinput()" callback funtion is registered
via the wskbd_consdev_kbdinput_register() by the MD attachment,
MI wskbd(9) just puts the input data from wskbd(9) to the callback
fundtions, if NWSDISPLAY is not configured.

I think the change is minimum and it doesn't break any existing
ports that use wsdisplay(9) ttys, but I'd like to call for
comments about these implementation design.

Supporting non-serial console on news68k port has been in
my TODO list for ~20 years so I'd like to commit this
if there is no particular objection.

There is a video of NetBSD/news68k 9.3 with the similar patch:
 https://www.youtube.com/watch?v=kPOiHG20Ye8


Index: sys/dev/wscons/wskbd.c
===
RCS file: /cvsroot/src/sys/dev/wscons/wskbd.c,v
retrieving revision 1.143
diff -u -p -d -r1.143 wskbd.c
--- sys/dev/wscons/wskbd.c  5 Feb 2019 10:04:49 -   1.143
+++ sys/dev/wscons/wskbd.c  6 Apr 2024 06:59:50 -
@@ -214,6 +214,9 @@ struct wskbd_softc {
/* optional table to translate scancodes in event mode */
int sc_evtrans_len;
keysym_t*sc_evtrans;
+
+   wskbd_consdev_kbdinput *sc_consdev_kbdinput;
+   device_t sc_consdev;
 };
 
 #define MOD_SHIFT_L(1 << 0)
@@ -420,6 +423,8 @@ wskbd_attach(device_t parent, device_t s
sc->sc_hotkeycookie = NULL;
sc->sc_evtrans_len = 0;
sc->sc_evtrans = NULL;
+   sc->sc_consdev_kbdinput = NULL;
+   sc->sc_consdev = NULL;
 
 #if NWSMUX > 0 || NWSDISPLAY > 0
sc->sc_base.me_ops = _srcops;
@@ -664,9 +669,7 @@ void
 wskbd_input(device_t dev, u_int type, int value)
 {
struct wskbd_softc *sc = device_private(dev);
-#if NWSDISPLAY > 0
int num, i;
-#endif
 
if (sc->sc_repeating) {
sc->sc_repeating = 0;
@@ -706,6 +709,24 @@ wskbd_input(device_t dev, u_int type, in
}
 #endif
 
+#if NWSDISPLAY == 0
+   if (sc->sc_translating) {
+   keysym_t ks;
+
+   if (sc->sc_consdev_kbdinput) {
+   num = wskbd_translate(sc->id, type, value);
+   for (i = 0; i < num; i++) {
+   ks = sc->id->t_symbols[i];
+   if (KS_GROUP(ks) == KS_GROUP_Plain &&
+   KS_VALUE(ks) <= 0x7f)
+   (*sc->sc_consdev_kbdinput)(
+   sc->sc_consdev, KS_VALUE(ks));
+   }
+   return;
+   }
+   }
+#endif
+
wskbd_deliver_event(sc, type, value);
 
 #if defined(WSKBD_EVENT_AUTOREPEAT)
@@ -1683,6 +1704,36 @@ wskbd_hotkey_deregister(device_t self)
sc->sc_hotkeycookie = NULL;
 }
 
+device_t
+wskbd_consdev_kbdinput_register(device_t self, wskbd_consdev_kbdinput *kifn,
+device_t cons_dv)
+{
+   struct wskbd_softc *sc = device_private(self);
+
+   if (sc == NULL)
+   return NULL;
+
+   if (kifn == NULL)
+   return NULL;
+
+   sc->sc_consdev_kbdinput = kifn;
+   sc->sc_consdev = cons_dv;
+   aprint_verbose_dev(self, "connecting to %s\n", device_xname(cons_dv));
+
+   return sc->sc_base.me_dv;
+}
+
+void
+wskbd_consdev_kbdinput_deregister(device_t self)
+{
+   struct wskbd_softc *sc = device_private(self);
+
+   KASSERT(sc != NULL);
+
+   sc->sc_consdev_kbdinput = NULL;
+   sc->sc_consdev = NULL;
+}
+
 static int
 wskbd_translate(struct wskbd_internal *id, u_int type, int value)
 {
Index: 

Re: fullfs

[Ice Cream]

Hi, I already worked on this. You can check the patch here
https://mail-index.netbsd.org/tech-kern/2024/03/30/msg029539.html

> I've copied nullfs to fullfs.  I did "make" and nothing happed,
> I tried 'build.sh' and nothing, like 'fullfs' does not exist.

Did you add fullfs to the Makefile(s)?

-- IC

fullfs

[Igor Kolesnik]

Hi;
I'd like to take this project.

I've copied nullfs to fullfs.  I did "make" and nothing happed,
I tried 'build.sh' and nothing, like 'fullfs' does not exist.

Re: Using mmap(2) in sort(1) instead of temp files

[Ice Cream]

 > PS: It was mentioned in the TODO file
 > > speed up sort(1) by using mmap(2) rather than temp files

I can't find this reference :-(

It's in src/doc/TODO (line 41)
https://github.com/NetBSD/src/blob/6b2da37d700ac979e2a4dc4413082e5a879be9b6/doc/TODO#L41



(I reposted this on tech-userlevel, you might also be interested
in the discussion there.)

-- IC

Re: Using mmap(2) in sort(1) instead of temp files

[David Holland]

On Fri, Apr 05, 2024 at 06:48:05AM +, David Holland wrote:
 >- read or write errors on memory mapped files result in SIGSEGV,
 >  which is annoying to deal with and does actually turn up in the
 >  field sometimes (*);

Oops, looks like I forgot to populate the asterisk.

(*) https://gnats.netbsd.org/11904

-- 
David A. Holland
dholl...@netbsd.org

Re: Using mmap(2) in sort(1) instead of temp files

[David Holland]

On Wed, Apr 03, 2024 at 05:40:47PM +, Ice Cream wrote:
 > I'm trying to speed up sort(1) by using mmap(2) instead of temp
 > files.
 > 
 > ftmp() (see code below) is called in the sort functions to create and
 > return a temp file. mkstemp() is used to create the temp file, then
 > the file pointer (returned by fdopen) is returned to the sort functions
 > for use. I'm trying to understand where and how mmap should come
 > into the picture here, and how to implement this feature.

I expect the intent was to mmap the temporary files to avoid the
overhead incurred by stdio. (As opposed to just allocating memory,
which as Mouse points out carries problems.)

I'm not sure this is actually a good idea. Using raw file handles
instead of stdio FILEs might provide some speedup (depending on the
write patterns and how big the blocks written are) but it's never been
entirely clear that mmap is actually substantively faster than using
raw file handles. Meanwhile, there are several disadvantages:
   - as Mouse pointed out, you need to know the size in advance;
   - read or write errors on memory mapped files result in SIGSEGV,
 which is annoying to deal with and does actually turn up in the
 field sometimes (*);
   - even if you apply MADV_SEQUENTIAL with madvise(2) the mmap
 interface can't really do as good a job of prefetching;
   - on 32-bit platforms the size is limited.
 
FWIW.

 > PS: It was mentioned in the TODO file
 > > speed up sort(1) by using mmap(2) rather than temp files
 
I can't find this reference :-(

-- 
David A. Holland
dholl...@netbsd.org

Re: Using mmap(2) in sort(1) instead of temp files

[Mouse]

Why is this on tech-kern?  It seems to me it belongs on tech-userlevel.

> I'm trying to speed up sort(1) by using mmap(2) instead of temp
> files.

If you're going to sort in RAM, why bother with temporaries at all?
Just slurp it all in and sort it in core.

But.

Part of the point of using temp files, it seems to me, is to be able to
sort datasets larger than will fit memory.

Unless NetBSD is prepared to completely desupport small machines like
the MicroVAX-II or shark, I think this might be a misguided thing to
do.  Given the way swap works, I suspect it will work better to use of
temp files instead of mmap()ed memory to sort datasets larger than will
fit in RAM, even if VM is available.  Furthermore, VM can be limited;
sorting input bigger than 3G on i386 shouldn't break (and, from a
usability standpoint, shouldn't even require any special options).
Even on 64-bit, VM can be comparatively small; on a 9.1 amd64 machine
at work, proc.$$.rlimit.datasize.hard is only 8 gigs.

At the very least, I would strongly recommend adding an option to
disable this, to continue to use real files for temporaries.

> ftmp() (see code below) is called in the sort functions to create and
> return a temp file.  mkstemp() is used to create the temp file, then
> the file pointer (returned by fdopen) is returned to the sort
> functions for use.  I'm trying to understand where and how mmap
> should come into the picture here, and how to implement this feature.

I think the biggest issue you'll have (aside the ones raised above) is
that an mmap()ed memory block has a fixed size, set at map time.  Files
are sized much more dynamically.  I suspect you'll end up
(re)implementing a ramfs (a simplified one, because the application
needs are relatively simple, but still.)

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Using mmap(2) in sort(1) instead of temp files

[Ice Cream]

I'm trying to speed up sort(1) by using mmap(2) instead of temp
files.

ftmp() (see code below) is called in the sort functions to create and
return a temp file. mkstemp() is used to create the temp file, then
the file pointer (returned by fdopen) is returned to the sort functions
for use. I'm trying to understand where and how mmap should come
into the picture here, and how to implement this feature.

According to me, replacing fdopen with mmap, then returning the
address of the mapped region to the sort functions should do it.
This will also require the sort functions themselves to be modified
to work with the mapped region instead of a file. In addition to this,
what else should be taken care of?

PS: It was mentioned in the TODO file
> speed up sort(1) by using mmap(2) rather than temp files

-- IC

Code snippet:

FILE *
ftmp(void)
{
sigset_t set, oset;
FILE *fp;
int fd;
char path[MAXPATHLEN];

(void)snprintf(path, sizeof(path), "%s%s%s", tmpdir,
   (tmpdir[strlen(tmpdir)-1] != '/') ? "/" : "", _NAME_TMP);

sigfillset();
(void)sigprocmask(SIG_BLOCK, , );
if ((fd = mkstemp(path)) < 0)
err(2, "ftmp: mkstemp(\"%s\")", path);
if (!(fp = fdopen(fd, "w+")))
err(2, "ftmp: fdopen(\"%s\")", path);
if (!DEBUG('t'))
(void)unlink(path);

(void)sigprocmask(SIG_SETMASK, , NULL);
return (fp);
}

Gsoc, Emulating missing linux syscalls

[张正]

Hi! Sorry for such a late contact, I’ve already written a proper 
proposal for apply compat_linux project by following the instruction and 
guidelines from the link given by the NetBSD Gsoc idea list page.I try to 
answer as much questions as I can in the guidelines page and by follow the 
questions,i also try and compile the NetBSD kernel and read through the code 
base that relevant with compat_linux.And I think it’s very weird to apply Gsoc 
project and submit the proposal without having any contact with the 
developer/maintainer of the project.So anyways, sorry for dragging this till 
now.
As a junior developer I’ve been working on several open source project 
on GitHub with the git/github pr workflow before,but as I read through the 
NetBSD contribute guidelines page, it seems the traditional mail list and 
problem database /problem report is the way to collaborate in NetBSD,so is it a 
“must do” here to use the cvs/svn version control software and the mail list 
for contribute here?
In the compat_linux project page, saying that 
"you should find at least one Linux binary that does not yet run on 
NetBSD using compat_linux to use as a test case (your   mentor may have 
suggestions)”
Is there any suggestion on which linux binary isn’t yet supported by 
compat_linux as a test case?

Inquiry about the "Auto Create Swap on Memory Pressure" Project in GSoC

[hanabi]

Dear Mentors,

I am interested in the "Auto Create Swap on Memory Pressure" project
(https://wiki.netbsd.org/projects/project/swap-auto/) in GSoC. I would
greatly appreciate it if you could inform me whether there is an
existing suitable candidate for this project at the moment.

If no one has been chosen yet, could you provide me with the
requirements necessary for participating in this project? I am eager
to learn more about this exciting opportunity.

Best Regards,
Tengfei Yu

Re: MCLADDREFERENCE() incrementing the wrong ext_refcnt?

[Edgar Fuß]

> So I think
> atomic_inc_uint(&(o)->m_ext.ext_refcnt);\
> should really be
> atomic_inc_uint(&(o)->m_ext_ref->m_ext.ext_refcnt); \
> which, of course, is the same thing if MEXT_ISEMBEDDED(o) is true.

> Am I getting something wrong?
Self-answer: Yes.
m_ext is m_ext_ref->m_ext_storage, so the additional indirection is 
already performed.

[GSoC] Questions about emulating missing linux system calls project

[Jamgadepatil Shivraj Shashikant]

Hello,
For the past few days, I have been trying to assess support for Linux
system calls on a NetBSD system. I am using Linux Test Project
for identifying system calls that are yet to be emulated on NetBSD.
I have setup QEMU with GENERIC kernel for testing purpose
(Thanks iMil for helping me out!) with COMPAT_LINUX option
enabled.

I have following particular questions:


  1.
I wish to emulate support for pidfd_* family of system calls,
open_tree, openat2, clone3. I believe they still lack
emulation support. Where should I look to identify the
possible mappings for these system calls in NetBSD
kernel?

  2.
I found around 40 system calls that currently lack
emulation support. Most of them are the system calls

that aren't popular or are more relevant to Linux kernel. Only
some of the newer system calls seem more likely to be able
to be mapped directly to NetBSD system calls. Is it expected
to actually implement some of them in NetBSD kernel and
then add them to compat layer as part of this project?

Also, what's the state of splice and sendfile system calls? I
can see that they have been implemented as part of
GSoC 2022 but the test cases relating to them still can't
find these system calls.


  1.
With respect to system calls relating to scheduler, I get error
relating to not finding some particular files in proc directory.
I have set up proc as mentioned in documentation but I still
get these errors when I run schedular related system calls.
Does proc need to be configured in different way or do schedular
related system calls actually lack emulation support?


I would also like to thank Theodore for helping me understand the work he did 
last year.

Thanking you,
Shivraj

[PATCH] Implement Fullfs file system

[Ice Cream]

(Resubmitting since it went unnoticed last time)
This patch implements the basic Fullfs file system where writes
fail with ENOSPC, as described here:
https://wiki.netbsd.org/projects/project/fullfs/

-- ICdiff --git a/distrib/sets/lists/base/mi b/distrib/sets/lists/base/mi
index fe14a819cd9f..e70342028a72 100644
--- a/distrib/sets/lists/base/mi
+++ b/distrib/sets/lists/base/mi
@@ -414,6 +414,7 @@
 ./sbin/mount_fdescbase-miscfs-root
 ./sbin/mount_ffsbase-sysutil-root
 ./sbin/mount_filecorebase-filecorefs-root
+./sbin/mount_fullbase-miscfs-root
 ./sbin/mount_hfsbase-hfs-root
 ./sbin/mount_hfspbase-obsolete		obsolete
 ./sbin/mount_kernfsbase-sysutil-root
@@ -1114,6 +1115,7 @@
 ./usr/include/miscfsbase-c-usr
 ./usr/include/miscfs/fdesc			base-c-usr
 ./usr/include/miscfs/fifofs			base-c-usr
+./usr/include/miscfs/fullfs			base-c-usr
 ./usr/include/miscfs/genfs			base-c-usr
 ./usr/include/miscfs/kernfs			base-c-usr
 ./usr/include/miscfs/nullfs			base-c-usr
diff --git a/distrib/sets/lists/comp/mi b/distrib/sets/lists/comp/mi
index b8952f060b89..530e97226710 100644
--- a/distrib/sets/lists/comp/mi
+++ b/distrib/sets/lists/comp/mi
@@ -2670,6 +2670,7 @@
 ./usr/include/milter/mfdef.h			comp-obsolete		obsolete
 ./usr/include/miscfs/fdesc/fdesc.h		comp-c-include
 ./usr/include/miscfs/fifofs/fifo.h		comp-c-include
+./usr/include/miscfs/fullfs/full.h		comp-c-include
 ./usr/include/miscfs/genfs/genfs.h		comp-c-include
 ./usr/include/miscfs/genfs/genfs_node.h		comp-c-include
 ./usr/include/miscfs/genfs/layer.h		comp-c-include
diff --git a/distrib/sets/lists/man/mi b/distrib/sets/lists/man/mi
index ec52c3c82aed..b08803cc8911 100644
--- a/distrib/sets/lists/man/mi
+++ b/distrib/sets/lists/man/mi
@@ -6210,6 +6210,7 @@
 ./usr/share/man/html8/mount_fdesc.html		man-miscfs-htmlman	html
 ./usr/share/man/html8/mount_ffs.html		man-sysutil-htmlman	html
 ./usr/share/man/html8/mount_filecore.html	man-filecorefs-htmlman	html
+./usr/share/man/html8/mount_full.html		man-miscfs-htmlman	html
 ./usr/share/man/html8/mount_hfs.html		man-hfs-htmlman		html
 ./usr/share/man/html8/mount_kernfs.html		man-sysutil-htmlman	html
 ./usr/share/man/html8/mount_lfs.html		man-sysutil-htmlman	html
@@ -9599,6 +9600,7 @@
 ./usr/share/man/man8/mount_fdesc.8		man-miscfs-man		.man
 ./usr/share/man/man8/mount_ffs.8		man-sysutil-man		.man
 ./usr/share/man/man8/mount_filecore.8		man-filecorefs-man	.man
+./usr/share/man/man8/mount_full.8		man-miscfs-man		.man
 ./usr/share/man/man8/mount_hfs.8		man-hfs-man		.man
 ./usr/share/man/man8/mount_hfsp.8		man-obsolete		obsolete
 ./usr/share/man/man8/mount_kernfs.8		man-sysutil-man		.man
diff --git a/sbin/Makefile b/sbin/Makefile
index 8cc5125b95c9..2f3036fcfd8a 100644
--- a/sbin/Makefile
+++ b/sbin/Makefile
@@ -33,6 +33,7 @@ SUBDIR+= mount_ext2fs
 SUBDIR+= mount_fdesc
 SUBDIR+= mount_filecore
 SUBDIR+= mount_ffs
+SUBDIR+= mount_full
 SUBDIR+= mount_hfs
 SUBDIR+= mount_kernfs
 SUBDIR+= mount_lfs
diff --git a/sbin/mount_full/Makefile b/sbin/mount_full/Makefile
new file mode 100644
index ..7c3790d28c0f
--- /dev/null
+++ b/sbin/mount_full/Makefile
@@ -0,0 +1,17 @@
+#	$NetBSD: Makefile,v 1.13 2020/07/26 08:20:22 mlelstv Exp $
+#	@(#)Makefile	8.3 (Berkeley) 3/27/94
+
+.include 
+
+PROG=	mount_full
+SRCS=	mount_full.c pathadj.c
+MAN=	mount_full.8
+
+MOUNT=  ${NETBSDSRCDIR}/sbin/mount
+CPPFLAGS+= -I${NETBSDSRCDIR}/sys -I${MOUNT}
+.PATH:  ${MOUNT}
+
+DPADD+=${LIBUTIL}
+LDADD+=-lutil
+
+.include 
diff --git a/sbin/mount_full/mount_full.8 b/sbin/mount_full/mount_full.8
new file mode 100644
index ..d59d41deda9b
--- /dev/null
+++ b/sbin/mount_full/mount_full.8
@@ -0,0 +1,12 @@
+.Dd March 28, 2024
+.Dt MOUNT_FULL 8
+.Os
+.Sh NAME
+.Nm mount_full
+.Nd a filesystem where writes fail with disk
+full
+.Sh SYNOPSIS
+.Nm
+.Op Fl o Ar options
+.Ar target
+.Ar mount-point
diff --git a/sbin/mount_full/mount_full.c b/sbin/mount_full/mount_full.c
new file mode 100644
index ..507a1c2a2f94
--- /dev/null
+++ b/sbin/mount_full/mount_full.c
@@ -0,0 +1,127 @@
+/*	$NetBSD: mount_null.c,v 1.21 2020/07/26 08:20:22 mlelstv Exp $	*/
+
+/*
+ * Copyright (c) 1992, 1993, 1994
+ *	The Regents of the University of California.  All rights reserved.
+ *
+ * This code is derived from software donated to Berkeley by
+ * Jan-Simon Pendry.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *notice, this list of conditions and the following disclaimer in the
+ *documentation and/or other materials provided with the distribution.
+ * 3. Neither the name of the University nor the names of its contributors
+ *may be used to endorse or promote products derived from this software
+ * 

Re: [GSOC][NEWBIE]

[Jamgadepatil Shivraj Shashikant]

Hello iMil,
Thanks a lot for the help. I was able to set up the qemu VM following the 
commands you specified.
Once again thanks a lot for your help.

Thanking you,
Shivraj


From: Emile 'iMil' Heitor 
Sent: Thursday, March 28, 2024 10:23 PM
To: Jamgadepatil Shivraj Shashikant ; tech-kern@netbsd.org 

Subject: Re: [GSOC][NEWBIE]

External Email


On 3/28/24 17:44, Jamgadepatil Shivraj Shashikant wrote:

> I even tried gziped kernel image stored in releasedir directory, so I
> guess it's not only PVH boot that was causing the problem. I suspected I
> might have built the kernel incorrectly, so I also tried booting from
> the daily kernel build provided on the github but the kvm error persists.
> qemu command I am using for booting the system is: qemu-system-x86_64 -m
> 2048 -kernel netbsd-GENERIC.gz -drive
> if=virtio,file=disk.qcow2,format=qcow2 -enable-kvm

Don't gzip the kernel, here's an example session:

$ git branch
* perf
$ ./build.sh -U -u -j4 -T obj/tooldir -m amd64 tools
$ ./build.sh -U -u -j4 -T obj/tooldir -m amd64 kernel=MICROVM
$ KERNEL=sys/arch/amd64/compile/obj/MICROVM/netbsd
$ IMG=NetBSD-10.99.10-amd64-live.img # fetch and gunzip this from
https://nycdn.netbsd.org/pub/NetBSD-daily/HEAD/202403200640Z/images/NetBSD-10.99.
10-amd64-live.img.gz
$ qemu-system-x86_64 \
 -M microvm,x-option-roms=off,rtc=on,acpi=off,pic=off \
 -enable-kvm -m 512 -cpu host,+invtsc \
 -append "root=ld0a console=com rw -v" -display none \
 -device virtio-blk-device,drive=hd0 \
 -drive file=${IMG},format=raw,id=hd0 \
 -netdev user,id=net0,hostfwd=tcp::10022-:22 \
 -device virtio-net-device,netdev=net0 \
 -kernel ${KERNEL} -serial mon:stdio

FYI I'll be mostly AFK till next Tuesday.

Cheers,

--

Emile `iMil' Heitor  | https://imil.net

GSoC application

[Luka Filipovic]

About the project: * What is the goal of the project?- The goal is to create an algorithm that automatically creates swap on memory pressure.  * What will be the deliverables of the project?- I will keep track of the project in the form of documentation so it can further be refined in the official manual.   The project will deliver the algorithm for creating, resizing, and deleting swap files. * Give an overview of how you intend to reach the project's goal in the form of milestones and a schedule.- For my schedule I will be unable to get a lot of work done until the beginning of July because of my exams.  As for the milestones, the first goals are:  1. Determining the threshold when the swapping starts   2. Creating the swap files and deciding which disk to place the file on  3. Implementing the algorithm for expanding or shrinking the swap file  4. Deleting the swap file when it is no longer needed  5. Optimizing the algorithm (if it isn't fast or efficient enough)  6. Fixing any miscellaneous bugs * Is similar software already available elsewhere?- It is present in the Linux kernel. * Is the project a port of software, or a rewrite?- It will have to be a rewrite. About your project and NetBSD: * Have you installed NetBSD and made first experiences with hands-on configuration?- I have downloaded it and set it up on a VM. * Have you found the relevant places that your project is based on in the source code, and read through it?- I've familiarized myself with the sys/kern/ section, to be more specific the section for memory allocation and the uvm_swap.c file to see how swapping is implemented in the kernel. * How will your project integrate into NetBSD? - I will primarily integrate the project through the kernel subsystem. * What interfaces in NetBSD will your project use?- I can't say for certain at this time. This will have to be further discussed with my mentor. * To what degree are you familiar with those interfaces?- I've only read into the source code to familiarize myself with it. About you: * Can you list some prior projects that you have worked on so far?- As university project I've made a kernel from scratch for the RISC-V processor architecture and also modified the xv6 kernel to support swapping and detection and prevention of thrashing.  These were solo projects for my classes that were written in C, C++ and Assembly, so I have a decent amount of experience in them. * Do you have any prior experience with programming NetBSD?- This is my first time working on NetBSD but I've familiarized myself enough to feel comfortable enough to write in it. * Have you previously discussed your project within NetBSD, either on a mailing list or with some specific developers?- I've never contacted NetBSD before. * How do we contact you?- The easiest way is with this email filipovic.luka@gmail.com * Is there anything else you'd like us to know?- I'd once again like to point out because of my university and its exams I won't be able to get a lot of work done until the beginning of July. Thank you for reading my application and I'm looking forward to working with you.   Sent from Mail for Windows 

Luka_Filipovic_cv (1).pdf
Description: Adobe PDF document

Re: [GSOC][NEWBIE]

[Jamgadepatil Shivraj Shashikant]

Hello iMil,
Thanks for your suggestions. I built the kernel from the sources available on 
the github link you provided. I am still running into the similar issues

Re: [GSOC][NEWBIE]

[Emile 'iMil' Heitor]

On 3/28/24 17:44, Jamgadepatil Shivraj Shashikant wrote:

I even tried gziped kernel image stored in releasedir directory, so I 
guess it's not only PVH boot that was causing the problem. I suspected I 
might have built the kernel incorrectly, so I also tried booting from 
the daily kernel build provided on the github but the kvm error persists.
qemu command I am using for booting the system is: qemu-system-x86_64 -m 
2048 -kernel netbsd-GENERIC.gz -drive 
if=virtio,file=disk.qcow2,format=qcow2 -enable-kvm


Don't gzip the kernel, here's an example session:

$ git branch
* perf
$ ./build.sh -U -u -j4 -T obj/tooldir -m amd64 tools
$ ./build.sh -U -u -j4 -T obj/tooldir -m amd64 kernel=MICROVM
$ KERNEL=sys/arch/amd64/compile/obj/MICROVM/netbsd
$ IMG=NetBSD-10.99.10-amd64-live.img # fetch and gunzip this from 
https://nycdn.netbsd.org/pub/NetBSD-daily/HEAD/202403200640Z/images/NetBSD-10.99.

10-amd64-live.img.gz
$ qemu-system-x86_64 \
-M microvm,x-option-roms=off,rtc=on,acpi=off,pic=off \
-enable-kvm -m 512 -cpu host,+invtsc \
-append "root=ld0a console=com rw -v" -display none \
-device virtio-blk-device,drive=hd0 \
-drive file=${IMG},format=raw,id=hd0 \
-netdev user,id=net0,hostfwd=tcp::10022-:22 \
-device virtio-net-device,netdev=net0 \
-kernel ${KERNEL} -serial mon:stdio

FYI I'll be mostly AFK till next Tuesday.

Cheers,

--

Emile `iMil' Heitor  | https://imil.net

Re: [GSOC][NEWBIE]

[Emile 'iMil' Heitor]

Hi Shivraj,

On 3/27/24 18:27, Jamgadepatil Shivraj Shashikant wrote:
I am using Linux as my host OS and have cross-compiled the NetBSD kernel 
but loading the custom kernel results in a KVM emulation error. I am 


If you seek to run a NetBSD/amd64 kernel with the -kernel qemu/kvm
option, I suggest you give a try to this development branch:
https://github.com/NetBSDfr/NetBSD-src/tree/perf
Official NetBSD source tree doesn't have PVH boot merged yet, this
branch has it, along with performance improvements.

Cheers,

--

Emile `iMil' Heitor  | https://imil.net

Re: MCLADDREFERENCE() incrementing the wrong ext_refcnt?

[Ryota Ozaki]

Hi,

On Sat, Mar 23, 2024 at 12:44 AM Edgar Fuß  wrote:
>
> Hello.
>
> I'm under the impression that MCLADDREFERENCE() may increment the wrong
> ext_refcnt.
>
> In case it's permitted (I cant't find anything to the contrary) to
> call MCLADDREFERENCE(m1, m2) and then MCLADDREFERENCE(m2, m3), then the
> second call will increment m2's ext_refcnt where it should be incrementing
> m1's one (e.g. the one all of m1, m2 and m3's m_ext_ref are pointing to), no?
>
> So I think
> atomic_inc_uint(&(o)->m_ext.ext_refcnt);\
> should really be
> atomic_inc_uint(&(o)->m_ext_ref->m_ext.ext_refcnt); \
> which, of course, is the same thing if MEXT_ISEMBEDDED(o) is true.
>
> Am I getting something wrong?

I think you're right.

IIUC use-after-free can occur in some cases. In the case of your example,
if the mbufs are freed in the order of m1, m3 and m2, a freed buffer of m1
can be accessed via m2 after m3 is freed.

I'll commit your fix.

Thanks,
  ozaki-r

Re: Fullfs file system

[Ice Cream]

I figured it out. The basic Fullfs now works
(writes fail with ENOSPC). Could someone
review my code?

-- ICdiff --git a/distrib/sets/lists/base/mi b/distrib/sets/lists/base/mi
index fe14a819cd9f..e70342028a72 100644
--- a/distrib/sets/lists/base/mi
+++ b/distrib/sets/lists/base/mi
@@ -414,6 +414,7 @@
 ./sbin/mount_fdescbase-miscfs-root
 ./sbin/mount_ffsbase-sysutil-root
 ./sbin/mount_filecorebase-filecorefs-root
+./sbin/mount_fullbase-miscfs-root
 ./sbin/mount_hfsbase-hfs-root
 ./sbin/mount_hfspbase-obsolete		obsolete
 ./sbin/mount_kernfsbase-sysutil-root
@@ -1114,6 +1115,7 @@
 ./usr/include/miscfsbase-c-usr
 ./usr/include/miscfs/fdesc			base-c-usr
 ./usr/include/miscfs/fifofs			base-c-usr
+./usr/include/miscfs/fullfs			base-c-usr
 ./usr/include/miscfs/genfs			base-c-usr
 ./usr/include/miscfs/kernfs			base-c-usr
 ./usr/include/miscfs/nullfs			base-c-usr
diff --git a/distrib/sets/lists/comp/mi b/distrib/sets/lists/comp/mi
index b8952f060b89..530e97226710 100644
--- a/distrib/sets/lists/comp/mi
+++ b/distrib/sets/lists/comp/mi
@@ -2670,6 +2670,7 @@
 ./usr/include/milter/mfdef.h			comp-obsolete		obsolete
 ./usr/include/miscfs/fdesc/fdesc.h		comp-c-include
 ./usr/include/miscfs/fifofs/fifo.h		comp-c-include
+./usr/include/miscfs/fullfs/full.h		comp-c-include
 ./usr/include/miscfs/genfs/genfs.h		comp-c-include
 ./usr/include/miscfs/genfs/genfs_node.h		comp-c-include
 ./usr/include/miscfs/genfs/layer.h		comp-c-include
diff --git a/distrib/sets/lists/man/mi b/distrib/sets/lists/man/mi
index ec52c3c82aed..b08803cc8911 100644
--- a/distrib/sets/lists/man/mi
+++ b/distrib/sets/lists/man/mi
@@ -6210,6 +6210,7 @@
 ./usr/share/man/html8/mount_fdesc.html		man-miscfs-htmlman	html
 ./usr/share/man/html8/mount_ffs.html		man-sysutil-htmlman	html
 ./usr/share/man/html8/mount_filecore.html	man-filecorefs-htmlman	html
+./usr/share/man/html8/mount_full.html		man-miscfs-htmlman	html
 ./usr/share/man/html8/mount_hfs.html		man-hfs-htmlman		html
 ./usr/share/man/html8/mount_kernfs.html		man-sysutil-htmlman	html
 ./usr/share/man/html8/mount_lfs.html		man-sysutil-htmlman	html
@@ -9599,6 +9600,7 @@
 ./usr/share/man/man8/mount_fdesc.8		man-miscfs-man		.man
 ./usr/share/man/man8/mount_ffs.8		man-sysutil-man		.man
 ./usr/share/man/man8/mount_filecore.8		man-filecorefs-man	.man
+./usr/share/man/man8/mount_full.8		man-miscfs-man		.man
 ./usr/share/man/man8/mount_hfs.8		man-hfs-man		.man
 ./usr/share/man/man8/mount_hfsp.8		man-obsolete		obsolete
 ./usr/share/man/man8/mount_kernfs.8		man-sysutil-man		.man
diff --git a/sbin/Makefile b/sbin/Makefile
index 8cc5125b95c9..2f3036fcfd8a 100644
--- a/sbin/Makefile
+++ b/sbin/Makefile
@@ -33,6 +33,7 @@ SUBDIR+= mount_ext2fs
 SUBDIR+= mount_fdesc
 SUBDIR+= mount_filecore
 SUBDIR+= mount_ffs
+SUBDIR+= mount_full
 SUBDIR+= mount_hfs
 SUBDIR+= mount_kernfs
 SUBDIR+= mount_lfs
diff --git a/sbin/mount_full/Makefile b/sbin/mount_full/Makefile
new file mode 100644
index ..7c3790d28c0f
--- /dev/null
+++ b/sbin/mount_full/Makefile
@@ -0,0 +1,17 @@
+#	$NetBSD: Makefile,v 1.13 2020/07/26 08:20:22 mlelstv Exp $
+#	@(#)Makefile	8.3 (Berkeley) 3/27/94
+
+.include 
+
+PROG=	mount_full
+SRCS=	mount_full.c pathadj.c
+MAN=	mount_full.8
+
+MOUNT=  ${NETBSDSRCDIR}/sbin/mount
+CPPFLAGS+= -I${NETBSDSRCDIR}/sys -I${MOUNT}
+.PATH:  ${MOUNT}
+
+DPADD+=${LIBUTIL}
+LDADD+=-lutil
+
+.include 
diff --git a/sbin/mount_full/mount_full.8 b/sbin/mount_full/mount_full.8
new file mode 100644
index ..d59d41deda9b
--- /dev/null
+++ b/sbin/mount_full/mount_full.8
@@ -0,0 +1,12 @@
+.Dd March 28, 2024
+.Dt MOUNT_FULL 8
+.Os
+.Sh NAME
+.Nm mount_full
+.Nd a filesystem where writes fail with disk
+full
+.Sh SYNOPSIS
+.Nm
+.Op Fl o Ar options
+.Ar target
+.Ar mount-point
diff --git a/sbin/mount_full/mount_full.c b/sbin/mount_full/mount_full.c
new file mode 100644
index ..507a1c2a2f94
--- /dev/null
+++ b/sbin/mount_full/mount_full.c
@@ -0,0 +1,127 @@
+/*	$NetBSD: mount_null.c,v 1.21 2020/07/26 08:20:22 mlelstv Exp $	*/
+
+/*
+ * Copyright (c) 1992, 1993, 1994
+ *	The Regents of the University of California.  All rights reserved.
+ *
+ * This code is derived from software donated to Berkeley by
+ * Jan-Simon Pendry.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *notice, this list of conditions and the following disclaimer in the
+ *documentation and/or other materials provided with the distribution.
+ * 3. Neither the name of the University nor the names of its contributors
+ *may be used to endorse or promote products derived from this software
+ *without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE REGENTS 

[GSOC][NEWBIE]

[Jamgadepatil Shivraj Shashikant]

Hello, I am Shivraj. I found the project about "adding support for emulation of 
missing linux syscalls" to NetBSD kernel intriguing.
I have previously worked on projects involving the Linux kernel. I have worked 
on adding system calls to Linux kernel for isolating cores, implementing a 
different scheduler in the kernel for normal processes and to provide support 
for anonymous memory checkpointing for a user process by writing a custom page 
fault handler.
I have been exploring NetBSD source tree for a few days. I am trying to set up 
my development environment with qemu to check out some ideas and understand 
things more clearly. I tried to boot from the kernel compiled from sources, but 
I am still unable to do so. I would appreciate all the help people here can 
provide to guide me to set up the development environment.
I am using Linux as my host OS and have cross-compiled the NetBSD kernel but 
loading the custom kernel results in a KVM emulation error. I am unable to find 
the path where compressed kernel image is generated. Images generated in 
releasedir results in errors or a frozen qemu screen.

Thanking you,
Shivraj Jamgade
MTech CSA
Indian Institute of Science, Bangalore

Re: Forcing a USB device to "ugen"

[Taylor R Campbell]

> Date: Tue, 26 Mar 2024 17:41:52 -0400
> From: Thor Lancelot Simon 
> 
> On Tue, Mar 26, 2024 at 12:25:07AM +, Taylor R Campbell wrote:
> > 
> > We should really expose a /dev/ugen* instance for _every_ USB device;
> > those that have kernel drivers attached have only limited access via
> > /dev/ugen* (no reads, writes, transfer ioctls, ), until you do
> > ioctl(USB_KICK_OUT_KERNEL_DRIVER) or whatever, at which point the
> > kernel driver will detach and the user program can take over instead
> > and use the full ugen(4) API.
> 
> I don't think this can be safely allowed at security level > 0, unless,
> perhaps, it's restricted from working on devices that would match disk
> drivers.

ioctl(USB_KICK_OUT_KERNEL_DRIVER) would attempt to detach the driver
without DETACH_FORCE, so disk devices with file systems mounted would
simply decline and the ioctl would fail.

Re: Forcing a USB device to "ugen"

[Jason Thorpe]

> On Mar 26, 2024, at 2:41 PM, Thor Lancelot Simon  wrote:
> 
> I don't think this can be safely allowed at security level > 0, unless,
> perhaps, it's restricted from working on devices that would match disk
> drivers.

The driver being asked to detach could certainly refuse to do so based on why 
it’s being asked.

-- thorpej

Re: Forcing a USB device to "ugen"

[Thor Lancelot Simon]

On Tue, Mar 26, 2024 at 12:25:07AM +, Taylor R Campbell wrote:
> 
> We should really expose a /dev/ugen* instance for _every_ USB device;
> those that have kernel drivers attached have only limited access via
> /dev/ugen* (no reads, writes, transfer ioctls, ), until you do
> ioctl(USB_KICK_OUT_KERNEL_DRIVER) or whatever, at which point the
> kernel driver will detach and the user program can take over instead
> and use the full ugen(4) API.

I don't think this can be safely allowed at security level > 0, unless,
perhaps, it's restricted from working on devices that would match disk
drivers.