Re: Proposal to obsolete SYS_pipe

[Valery Ushakov]

On Tue, Dec 26, 2017 at 01:29:42 +, Christos Zoulas wrote:

> In article ,
> Kamil Rytarowski   wrote:
> >-=-=-=-=-=-
> >-=-=-=-=-=-
> >
> >On 25.12.2017 17:43, Christos Zoulas wrote:
> >> On Dec 25,  4:42pm, n...@gmx.com (Kamil Rytarowski) wrote:
> >> -- Subject: Re: Proposal to obsolete SYS_pipe
> >> 
> >> | I've extracted two changes from the original mail:
> >> | 
> >> | https://mail-index.netbsd.org/tech-kern/2017/12/25/msg022836.html
> >> 
> >> Yes, the first patch is exactly what I had in mind; remove the
> >> assembly stubs from libc and make pipe() a wrapper for pipe2().
> >> The second patch sounds good too, but it is not in the email...
> >> 
> >> christos
> >> 
> >
> >I've included the missing patch in the subsequent mail:
> >
> >https://mail-index.netbsd.org/tech-kern/2017/12/25/msg022840.html
> >
> >Patch (pasted here for the reference):
> >
> >http://netbsd.org/~kamil/patch-00041-refactor-pipe1.txt
> 
> I am good with both since they eliminate the MD code and simplify
> the MI code. The only drawback is that sys_pipe (the system call)
> is not handled directly anymore by libc, but that's not an issue
> except for the slight performance loss (which does not really matter
> the moment you start doing I/O).

Why can't we just leave pipe() alone?  There are other syscalls that
return two values, e.g. fork.  The MD asm stubs are trivial and they
are already written.  They've been there for ages.  Why the sudden
desire to "create movment"?

The pipe1() change is a good thing, OTOH.

-uwe

Re: xcall while cold == 1

[Masanobu SAITOH]

On 2017/12/25 20:26, Martin Husemann wrote:

On Mon, Dec 25, 2017 at 03:42:06PM +0900, Masanobu SAITOH wrote:

  Is this intended behavior? Is it possible to use xcall while
cold==1?


Cold is (as you noted) not the right condition (but pretty close).

Xcalls don't really make any sense before cpus have been spun up.
In your case it might be good to do the loop checking for SPCF_RUNNING
and if <= 1 is found, use the code path for single cpu systems (the
current else statatement).


In init_main.c::configure2():


cold = 0;   /* clocks are running, we're warm now! */
s = splsched();
curcpu()->ci_schedstate.spc_flags |= SPCF_RUNNING;
splx(s);

/* Boot the secondary processors. */
for (CPU_INFO_FOREACH(cii, ci)) {
uvm_cpu_attach(ci);
}
mp_online = true;


so checking mp_online is the best, right?


Martin




--
---
SAITOH Masanobu (msai...@execsw.org
 msai...@netbsd.org)

Re: Proposal to obsolete SYS_pipe

[Christos Zoulas]

In article ,
Kamil Rytarowski   wrote:
>-=-=-=-=-=-
>-=-=-=-=-=-
>
>On 25.12.2017 17:43, Christos Zoulas wrote:
>> On Dec 25,  4:42pm, n...@gmx.com (Kamil Rytarowski) wrote:
>> -- Subject: Re: Proposal to obsolete SYS_pipe
>> 
>> | I've extracted two changes from the original mail:
>> | 
>> | https://mail-index.netbsd.org/tech-kern/2017/12/25/msg022836.html
>> 
>> Yes, the first patch is exactly what I had in mind; remove the
>> assembly stubs from libc and make pipe() a wrapper for pipe2().
>> The second patch sounds good too, but it is not in the email...
>> 
>> christos
>> 
>
>I've included the missing patch in the subsequent mail:
>
>https://mail-index.netbsd.org/tech-kern/2017/12/25/msg022840.html
>
>Patch (pasted here for the reference):
>
>http://netbsd.org/~kamil/patch-00041-refactor-pipe1.txt

I am good with both since they eliminate the MD code and simplify
the MI code. The only drawback is that sys_pipe (the system call)
is not handled directly anymore by libc, but that's not an issue
except for the slight performance loss (which does not really matter
the moment you start doing I/O).

christos

Re: Proposal to obsolete SYS_pipe

[Christos Zoulas]

In article ,
Kamil Rytarowski   wrote:
>-=-=-=-=-=-
>-=-=-=-=-=-
>
>On 25.12.2017 16:37, Kamil Rytarowski wrote:
>> On 24.12.2017 22:25, Kamil Rytarowski wrote:
>>> I propose to deprecate SYS_pipe.
>>>
>>> It is a special syscall that returns two integers from one function
>>> call. Fanciness is not compatible with regular C syntax and it demands
>>> per-cpu assembly wrappers and rump-kernel workarounds. It's not easily
>>> usable with syscall(2).
>>>
>>> OpenBSD and FreeBSD already deprecated this traditional pipe(2) syscall
>>> replacing it with a more portable equivalent in C.
>>>
>>> Changes:
>>>  - I've marked pipe(2) as compat_80.
>>>  - I've removed generation of garbage in retval[2] from pipe2(2).
>>>  - I've reimplemented in C the pipe(2) syscall with pipe2(2).
>>>  - I've adjusted the surrounding code for the changes.
>>>
>>> http://netbsd.org/~kamil/patch-00039-obsolete-SYS_pipe.txt
>>>
>> 
>> I've extracted two patches from the above proposal.
>> 
>> In these patches SYS_pipe is not marked COMPAT_80 and not removed from
>> rump. I've left it as it is.
>> 
>> 1. Implement pipe() with pipe2(2) in libc:
>> 
>> New source code is now Machine Independent.
>> 
>> http://netbsd.org/~kamil/patch-00040-implement-pipe-with-pipe2-in-libc.txt
>> 
>> The generated code in libc for x86_64 is also simpler and shorter:
>> 
>> 0008b2a2 <_pipe>:
>>8b2a2:   31 f6   xor%esi,%esi
>>8b2a4:   e9 b7 f5 fa ff  jmpq   3a860 
>> 
>> 2. Refactor pipe1() kernel-internal function to operate over int[2]
>> rather than register_t[2]. Stop returning garbage through retval[2] from
>> pipe2(2).
>> 
>
>http://netbsd.org/~kamil/patch-00041-refactor-pipe1.txt

That looks good too (the code is simpler).

christos

Re: Proposal to obsolete SYS_pipe

[Kamil Rytarowski]

On 25.12.2017 17:18, Valery Ushakov wrote:
> On Mon, Dec 25, 2017 at 16:37:43 +0100, Kamil Rytarowski wrote:
> 
>> On 24.12.2017 22:25, Kamil Rytarowski wrote:
>>
>>> http://netbsd.org/~kamil/patch-00039-obsolete-SYS_pipe.txt
>>
>> I've extracted two patches from the above proposal.
>>
>> In these patches SYS_pipe is not marked COMPAT_80 and not removed from
>> rump. I've left it as it is.
>>
>> 1. Implement pipe() with pipe2(2) in libc:
>>
>> New source code is now Machine Independent.
>>
>> http://netbsd.org/~kamil/patch-00040-implement-pipe-with-pipe2-in-libc.txt
>>
>> The generated code in libc for x86_64 is also simpler and shorter:
>>
>> 0008b2a2 <_pipe>:
>>8b2a2:   31 f6   xor%esi,%esi
>>8b2a4:   e9 b7 f5 fa ff  jmpq   3a860 
> 
> But you incur the price of pipe2's copyout().  I'm curious, does
> anyone know how things like SMAP contribute to that price?
> 
> 

I've benchmarked pipe() vs pipe2() calling the entry calls from
unmodified libc.

NetBSD/amd64 with SMAP and SMEP, version 8.99.7 (kernel: Nov 24th)

http://netbsd.org/~kamil/pipe-bench.c

pipe2() is slower than pipe() and takes 8% more time in my microbenchmarks.

Therefore right, there is performance hit.

How about committing the pipe.c version to libc and keep it as a
fallback for archs without pipe.S?

>> 2. Refactor pipe1() kernel-internal function to operate over int[2]
>> rather than register_t[2].  Stop returning garbage through retval[2]
>> from pipe2(2).
> 
> Please, can you be more specific with your characterizations.
> "Returning garbage" is vague, and without further details (that you do
> know yourself but don't disclose) makes every reader expend time and
> mental effort to figure out what are you really talking about.
> 
> For the reference, sys_pipe2() overwrites retval[1] with the second
> descriptor b/c it passes retval[] to pipe1(), like sys_pipe() does.
> But what is the intended effect for pipe() causes retval[1] register
> to be clobbered for pipe2().
> 
> 
> -uwe
> 

pipe2() returns two integers (values), the 2nd one is a copy of the 2nd
file descriptor that lands in fildes[2]. This is a side effect of
reusing the code for sys_pipe() and not cleaning it up.

The first returned value is (on success) 0.

I've introduced a small refactoring that does not operate over retval[],
but on an array int[2]. A user sets retval[] for pipe() when desired and
needed.



signature.asc
Description: OpenPGP digital signature

Re: Proposal to obsolete SYS_pipe

[Kamil Rytarowski]

On 25.12.2017 17:43, Christos Zoulas wrote:
> On Dec 25,  4:42pm, n...@gmx.com (Kamil Rytarowski) wrote:
> -- Subject: Re: Proposal to obsolete SYS_pipe
> 
> | I've extracted two changes from the original mail:
> | 
> | https://mail-index.netbsd.org/tech-kern/2017/12/25/msg022836.html
> 
> Yes, the first patch is exactly what I had in mind; remove the
> assembly stubs from libc and make pipe() a wrapper for pipe2().
> The second patch sounds good too, but it is not in the email...
> 
> christos
> 

I've included the missing patch in the subsequent mail:

https://mail-index.netbsd.org/tech-kern/2017/12/25/msg022840.html

Patch (pasted here for the reference):

http://netbsd.org/~kamil/patch-00041-refactor-pipe1.txt



signature.asc
Description: OpenPGP digital signature

Re: Proposal to obsolete SYS_pipe

[Kamil Rytarowski]

On 25.12.2017 16:37, Kamil Rytarowski wrote:
> On 24.12.2017 22:25, Kamil Rytarowski wrote:
>> I propose to deprecate SYS_pipe.
>>
>> It is a special syscall that returns two integers from one function
>> call. Fanciness is not compatible with regular C syntax and it demands
>> per-cpu assembly wrappers and rump-kernel workarounds. It's not easily
>> usable with syscall(2).
>>
>> OpenBSD and FreeBSD already deprecated this traditional pipe(2) syscall
>> replacing it with a more portable equivalent in C.
>>
>> Changes:
>>  - I've marked pipe(2) as compat_80.
>>  - I've removed generation of garbage in retval[2] from pipe2(2).
>>  - I've reimplemented in C the pipe(2) syscall with pipe2(2).
>>  - I've adjusted the surrounding code for the changes.
>>
>> http://netbsd.org/~kamil/patch-00039-obsolete-SYS_pipe.txt
>>
> 
> I've extracted two patches from the above proposal.
> 
> In these patches SYS_pipe is not marked COMPAT_80 and not removed from
> rump. I've left it as it is.
> 
> 1. Implement pipe() with pipe2(2) in libc:
> 
> New source code is now Machine Independent.
> 
> http://netbsd.org/~kamil/patch-00040-implement-pipe-with-pipe2-in-libc.txt
> 
> The generated code in libc for x86_64 is also simpler and shorter:
> 
> 0008b2a2 <_pipe>:
>8b2a2:   31 f6   xor%esi,%esi
>8b2a4:   e9 b7 f5 fa ff  jmpq   3a860 
> 
> 2. Refactor pipe1() kernel-internal function to operate over int[2]
> rather than register_t[2]. Stop returning garbage through retval[2] from
> pipe2(2).
> 

http://netbsd.org/~kamil/patch-00041-refactor-pipe1.txt

> This simplifies the existing code, that looks like developed when
> sizeof(register_t) was equal to sizeof(int).
> 
> 
> 
> Before the changes:
> 
> $ ktruss -i ./a.out
> [...]
>  15131  1 a.outpipe2(0x7f7fff2e62b8, 0)= 0, 4
> [...]
> 
> After the changes:
> 
> $ ktruss -i ./a.out
> [...[
>782  1 a.outpipe2(0x7f7fff97e850, 0)= 0
> [...]
> 




signature.asc
Description: OpenPGP digital signature

Re: Proposal to obsolete SYS_pipe

[Christos Zoulas]

On Dec 25,  4:42pm, n...@gmx.com (Kamil Rytarowski) wrote:
-- Subject: Re: Proposal to obsolete SYS_pipe

| I've extracted two changes from the original mail:
| 
| https://mail-index.netbsd.org/tech-kern/2017/12/25/msg022836.html

Yes, the first patch is exactly what I had in mind; remove the
assembly stubs from libc and make pipe() a wrapper for pipe2().
The second patch sounds good too, but it is not in the email...

christos

Re: Proposal to obsolete SYS_pipe

[Valery Ushakov]

On Mon, Dec 25, 2017 at 16:37:43 +0100, Kamil Rytarowski wrote:

> On 24.12.2017 22:25, Kamil Rytarowski wrote:
>
> > http://netbsd.org/~kamil/patch-00039-obsolete-SYS_pipe.txt
> 
> I've extracted two patches from the above proposal.
> 
> In these patches SYS_pipe is not marked COMPAT_80 and not removed from
> rump. I've left it as it is.
> 
> 1. Implement pipe() with pipe2(2) in libc:
> 
> New source code is now Machine Independent.
> 
> http://netbsd.org/~kamil/patch-00040-implement-pipe-with-pipe2-in-libc.txt
> 
> The generated code in libc for x86_64 is also simpler and shorter:
> 
> 0008b2a2 <_pipe>:
>8b2a2:   31 f6   xor%esi,%esi
>8b2a4:   e9 b7 f5 fa ff  jmpq   3a860 

But you incur the price of pipe2's copyout().  I'm curious, does
anyone know how things like SMAP contribute to that price?


> 2. Refactor pipe1() kernel-internal function to operate over int[2]
> rather than register_t[2].  Stop returning garbage through retval[2]
> from pipe2(2).

Please, can you be more specific with your characterizations.
"Returning garbage" is vague, and without further details (that you do
know yourself but don't disclose) makes every reader expend time and
mental effort to figure out what are you really talking about.

For the reference, sys_pipe2() overwrites retval[1] with the second
descriptor b/c it passes retval[] to pipe1(), like sys_pipe() does.
But what is the intended effect for pipe() causes retval[1] register
to be clobbered for pipe2().


-uwe

Re: Proposal to obsolete SYS_pipe

[Kamil Rytarowski]

On 24.12.2017 22:25, Kamil Rytarowski wrote:
> I propose to deprecate SYS_pipe.
> 
> It is a special syscall that returns two integers from one function
> call. Fanciness is not compatible with regular C syntax and it demands
> per-cpu assembly wrappers and rump-kernel workarounds. It's not easily
> usable with syscall(2).
> 
> OpenBSD and FreeBSD already deprecated this traditional pipe(2) syscall
> replacing it with a more portable equivalent in C.
> 
> Changes:
>  - I've marked pipe(2) as compat_80.
>  - I've removed generation of garbage in retval[2] from pipe2(2).
>  - I've reimplemented in C the pipe(2) syscall with pipe2(2).
>  - I've adjusted the surrounding code for the changes.
> 
> http://netbsd.org/~kamil/patch-00039-obsolete-SYS_pipe.txt
> 

I've extracted two patches from the above proposal.

In these patches SYS_pipe is not marked COMPAT_80 and not removed from
rump. I've left it as it is.

1. Implement pipe() with pipe2(2) in libc:

New source code is now Machine Independent.

http://netbsd.org/~kamil/patch-00040-implement-pipe-with-pipe2-in-libc.txt

The generated code in libc for x86_64 is also simpler and shorter:

0008b2a2 <_pipe>:
   8b2a2:   31 f6   xor%esi,%esi
   8b2a4:   e9 b7 f5 fa ff  jmpq   3a860 

2. Refactor pipe1() kernel-internal function to operate over int[2]
rather than register_t[2]. Stop returning garbage through retval[2] from
pipe2(2).

This simplifies the existing code, that looks like developed when
sizeof(register_t) was equal to sizeof(int).



Before the changes:

$ ktruss -i ./a.out
[...]
 15131  1 a.outpipe2(0x7f7fff2e62b8, 0)= 0, 4
[...]

After the changes:

$ ktruss -i ./a.out
[...[
   782  1 a.outpipe2(0x7f7fff97e850, 0)= 0
[...]



signature.asc
Description: OpenPGP digital signature

Re: Proposal to obsolete SYS_pipe

[Christos Zoulas]

In article ,
Kamil Rytarowski   wrote:
>-=-=-=-=-=-
>-=-=-=-=-=-
>
>On 25.12.2017 03:42, John Nemeth wrote:
>> On Dec 24,  9:37pm, Mouse wrote:
>> }
>> } > http://netbsd.org/~kamil/patch-00039-obsolete-SYS_pipe.txt
>> } 
>> } I see no pipe2(2), nor change from pipe(2) to pipe(3) (with an xref to
>> } pipe2(2)), both of which, it seems to me, should be part of this.
>> 
>>  From: http://netbsd.gw.com/cgi-bin/man-cgi?pipe2+2+NetBSD-current
>> 
>> HISTORY
>>  A pipe() function call appeared in Version 6 AT UNIX.  The pipe2()
>>  function is inspired from Linux and appeared in NetBSD 6.0.
>> 
>> My NetBSD 7.x systems have the manpage as well.  One might wish to
>> look for manpages on a system newer then 1.4T.  :->
>> 
>>  The big thing is that I don't see what the difference between
>> pipe(2) and pipe2(2) are, other then that pipe2(2) takes an extra
>> flags argument, i.e. I don't see how it solves the problem stated
>> in the original message.
>> 
>
>The original problem:
>
>__syscall(SYS_pipe2, ... works, while syscall(SYS_pipe, ... is not
>easily usable.
>
>It's stated in the BUGS section of syscall(2):
>
>BUGS
> There is no way to simulate system calls that have multiple return
>values
> such as pipe(2).
>
>
>This propagates to rump special case handling and requests to use
>assembly snippets for every supported CPU. It might save little space in
>the kernel, but also save time of a developer porting the OS to a new
>CPU. Nobody will need to spend precious time on pipe.S anymore.
>
>I cannot remove entirely SYS_pipe, as it is used in software that bypass
>libc (like golang). Once compat_80 will be disabled at least in the
>default kernels, it will be gone from regular users. pipe2(2) is there
>since 5.99.x, and switching golang and other users to SYS_pipe2 won't
>affect any users of supported kernel versions.
>
>I've just entirely removed sys_pipe from rump and adjusted the needed
>code. There is certainly more room now to further simplify the rump
>retval[] code,but I have omitted it.
>
>I've skipped the change renaming or splitting the man pages.

It does not really matter; there are other syscalls that do this hack
like getpid() (ppid == retval[1]), get{u,g}id return the
(e{u,g}id == retval[1])... We'll need the compat code anyway in the
kernel (and perhaps in libc) since pipe is a bit popular -- which actually
makes it annoying because you'll only discover that you don't have the
compat code around unexpectedly. I don't see much value added with the
change, you can always use syscall pipe2 and the userland pipe() can use
pipe2(). I.e. I'd keep the kernel as is (we can always clean it up later)
and I would make the libc stuff use pipe2() so that the sanitizer parts
don't need the hack to work.

christos

Re: RFC: ipsec(4) pseudo interface

[Christos Zoulas]

On Dec 25,  4:54pm, k-nakah...@iij.ad.jp (Kengo NAKAHARA) wrote:
-- Subject: Re: RFC: ipsec(4) pseudo interface

| Here is the updated patch series and unified patch.
| - https://www.netbsd.org/~knakahara/if_ipsec/if_ipsec4.tgz
| - https://www.netbsd.org/~knakahara/if_ipsec/if_ipsec4-unified.patch

Thanks, LGTM!

christos

Re: Proposal to obsolete SYS_pipe

[Warner Losh]

On Dec 24, 2017 11:10 PM, "Robert Elz"  wrote:

Date:Sun, 24 Dec 2017 18:42:19 -0800
From:John Nemeth 
Message-ID:  <201712250242.vbp2gjjm017...@server.cornerstoneservice.ca>

  | HISTORY
  |  A pipe() function call appeared in Version 6 AT UNIX.

That I think would be a man page bug - pipe() was certainly in 5th
edition, but that is as far back as I go, so I am not sure when it
did appear - the syscall number suggests it was not in the very early
versions though (not 1st or 2nd edition probably.)


It is in the 3rd edition man pages, but is documented with only one return
code. The 4th edition manual looks very similar, but does have both values
documented. The source is fragmentary so it's hard to track down. 2nd
edition has no manuals, but no pipe in libc.

I just went through FreeBSD's system call man pages and corrected a number
of details like this...

Warner

Re: Proposal to obsolete SYS_pipe

[Kamil Rytarowski]

On 25.12.2017 03:42, John Nemeth wrote:
> On Dec 24,  9:37pm, Mouse wrote:
> }
> } > http://netbsd.org/~kamil/patch-00039-obsolete-SYS_pipe.txt
> } 
> } I see no pipe2(2), nor change from pipe(2) to pipe(3) (with an xref to
> } pipe2(2)), both of which, it seems to me, should be part of this.
> 
>  From: http://netbsd.gw.com/cgi-bin/man-cgi?pipe2+2+NetBSD-current
> 
> HISTORY
>  A pipe() function call appeared in Version 6 AT UNIX.  The pipe2()
>  function is inspired from Linux and appeared in NetBSD 6.0.
> 
> My NetBSD 7.x systems have the manpage as well.  One might wish to
> look for manpages on a system newer then 1.4T.  :->
> 
>  The big thing is that I don't see what the difference between
> pipe(2) and pipe2(2) are, other then that pipe2(2) takes an extra
> flags argument, i.e. I don't see how it solves the problem stated
> in the original message.
> 

The original problem:

__syscall(SYS_pipe2, ... works, while syscall(SYS_pipe, ... is not
easily usable.

It's stated in the BUGS section of syscall(2):

BUGS
 There is no way to simulate system calls that have multiple return
values
 such as pipe(2).


This propagates to rump special case handling and requests to use
assembly snippets for every supported CPU. It might save little space in
the kernel, but also save time of a developer porting the OS to a new
CPU. Nobody will need to spend precious time on pipe.S anymore.

I cannot remove entirely SYS_pipe, as it is used in software that bypass
libc (like golang). Once compat_80 will be disabled at least in the
default kernels, it will be gone from regular users. pipe2(2) is there
since 5.99.x, and switching golang and other users to SYS_pipe2 won't
affect any users of supported kernel versions.

I've just entirely removed sys_pipe from rump and adjusted the needed
code. There is certainly more room now to further simplify the rump
retval[] code,but I have omitted it.

I've skipped the change renaming or splitting the man pages.

> }-- End of excerpt from Mouse
> 




signature.asc
Description: OpenPGP digital signature