Gsoc, Emulating missing linux syscalls
Hi! Sorry for such a late contact, I’ve already written a proper proposal for apply compat_linux project by following the instruction and guidelines from the link given by the NetBSD Gsoc idea list page.I try to answer as much questions as I can in the guidelines page and by follow the questions,i also try and compile the NetBSD kernel and read through the code base that relevant with compat_linux.And I think it’s very weird to apply Gsoc project and submit the proposal without having any contact with the developer/maintainer of the project.So anyways, sorry for dragging this till now. As a junior developer I’ve been working on several open source project on GitHub with the git/github pr workflow before,but as I read through the NetBSD contribute guidelines page, it seems the traditional mail list and problem database /problem report is the way to collaborate in NetBSD,so is it a “must do” here to use the cvs/svn version control software and the mail list for contribute here? In the compat_linux project page, saying that "you should find at least one Linux binary that does not yet run on NetBSD using compat_linux to use as a test case (your mentor may have suggestions)” Is there any suggestion on which linux binary isn’t yet supported by compat_linux as a test case?
[GSoC 2024] - Expression of Interest - Emulating missing Linux syscalls
Hello, I am Trung Nguyen, and I'm looking for a chance to participate in GSoC 2024 with the *Emulating missing linux syscalls* project. I am new to NetBSD, so I would appreciate any feedback for my answers below. *About your project* - Goal of the project: Improve NetBSD's `compat_linux` to make at least one Linux binary that currently does not run properly under NetBSD work. - Deliverables of the project: Mostly code changes to `sys/compat/linux` and occasional patches to other parts of the kernel depending on the nature of the syscalls implemented/fixed. - Overall plan: + Get a simple Linux environment running with a shell, such as an Alpine chroot. + Improve related tracing tools like `strace`. + Run more complex binaries, especially application runtimes that heavily interact with the system such as `dotnet`. - Similar software I am aware of: + WSL1 on Windows NT 10.0 + FreeBSD's "Linuxulator" + blink (https://github.com/jart/blink). Runs entirely on a POSIX userland, making it portable but requires architecture emulation along with Linux syscall emulation, therefore less efficient than kernel modules. - The project is not a port of any software but improving existing NetBSD code. *About your project and NetBSD* - Experience with NetBSD: I am new to NetBSD. I have installed the OS and recompiled the kernel to enable `compat_linux`. I have tried an Alpine Linux root (my favorite option for testing any Linux compatibility layers), but the `busybox` binary responsible for the default shell does not work. - I have read through the source in `sys/compat/linux`. - The project should result in improvement to a kernel subsystem. - Interfaces: From what I understand: + A `probe` function checks for Linux binaries by looking at Linux-specific traits distinguishing it from other ELFs. + A struct called `emul` contains function pointers related to process lifetime and syscall dispatching for the compatibility layer. - Currently I am not too familiar with these interfaces and many other parts of NetBSD. However, as I tackle problems I can quickly read the relevant parts of the source. - Knowledge about the Linux ABI is required. - I am very familiar with important parts of the Linux syscall ABI, having worked on other projects emulating it in the past. *About you* - I am Trung Nguyen, also known as trungnt2910. I am an enthusiast in OS development and OS compatibility layers. My GitHub account is at https://github.com/trungnt2910. + Many projects I have worked on involve layers written in C or C++ translating to and from the Linux ABI, including: * Contributor to Darling: macOS binaries on Linux. * Collaborator of blink: Linux binaries on a variety of POSIX platforms. * Maintainer of hyclone: Haiku OS binaries on Linux. + I also have experience with kernel-land C/C++ development: * Haiku OS: Fix kernel bugs that prevented the dotnet port from running properly (Successful GSoC 2023 project: https://www.haiku-os.org/blog/trungnt2910/2023-08-20_gsoc_2023_dotnet_port_final_report#haikuhaiku ) * lxmonika: https://github.com/trungnt2910/lxmonika - A Windows driver framework enabling developers to write custom compatibility layers on the OS apart from WSL. - I am new to programming with NetBSD, though I am used to working in a generic Unix-like environment. - This mail is my first discussion of the project. - My preferred contact is at m...@trungnt2910.com. I look forward to your suggestions, and hope that this could eventually turn into a successful proposal for GSoC 2024. Kind regards, Trung Nguyen
Re: Emulating missing linux syscalls project
On Tue, Mar 28, 2023 at 03:32:35PM -0700, Zophiel wrote: > - What binary's in the current NetBSD stack does not run compact_test as a > test case increasing order of missing test cases ? The only regular compat_* tests running are compat_netbsd32, that is e.g. testing a NetBSD/i386 userland on a NetBSD/amd64 machine. Martin
Emulating missing linux syscalls project
Good afternoon, I'm interested in the Gsoc project, "Emulating missing linux syscalls", I have some programming experience and project experience. In terms of experience-level i'm a junior developer when it comes to os-development and net-bsd so I imagine I have quite a lot to learn. My questions regarding the project: - What binary's in the current NetBSD stack does not run compact_test as a test case increasing order of missing test cases ? - What would be the best testing frameworks for identifying and scoping in on syscalls ? - What do you recommend for reading up on developing for NetBSD ? (Currently i'm working through some of the chapters of OSTEP get enough os-internals background) Thanks, Enoch Ayeh
GSOC2023 || Emulating missing linux syscalls
Hello All I just came across the NETBSD project in GSOC and saw that this idea is very interesting. Regarding the chat group, would you please help to be involved? -- Regards, Mohamed Osama
Re: [GSoC] Emulating missing Linux syscalls project questions
On Sun, 19 Mar 2023 at 04:22, Theodore Preduta wrote: > > > The Linux Test Project (http://linux-test-project.github.io) would help > > not only with finding missing syscalls, but also with finding bugs / > > missing functionality in the existing Linux emul code. > > Yes this is a great idea! Although my interpretation of the project > idea is that the expectations are that the binary is functional by the > end of the summer. I obviously will not be able to implement all > missing syscalls by the end of the summer, so I would have to draw an > arbitrary line as to what I would/would not try to implement. > > Which brings me to my next comment. > > > It would be nice to have this running on NetBSD. > > In what way exactly does the LTP not function on NetBSD? I tried it > today and (after a few hours of troubleshooting) seemingly got it to work. > > Some assorted notes about what I did/what it took to get it to work: > > - I only looked at system call tests (so for all I know the other types > of tests could be what you're referring to). > > - The actual testcases themselves can be trivially (just add -static) > statically compiled on any Linux distro and can be run individually just > fine, but the rest of the testing infrastructure cannot (because glibc). > (Most of my time spent on this was dealing with glibc versions) > > - Otherwise you can compile everything normally on OpenSUSE 15.4, and > with suse15_base installed the binaries will *almost* just work. > > - The ltp-pan binary does depend on /dev/kmsg (which doesn't currently > exist in the emul code), but only writes to it, so touch > /emul/linux/dev/kmsg is sufficient to trick it into working. > > - As expected, lots of tests fail, but also lots of tests pass! I > haven't looked to hard into the failing tests (yet), but I didn't find > anything too surprising in the list of failing tests. > > Overall, I did enjoy going down this rabbit hole! It definitely taught > me a few new things about how the emul subsystem behaves. I think making progress on running LTP on NetBSD (*), and fixing up a useful subset of missing or incomplete syscall implementations would make an _excellent_ GSoC project :) *: Even if its "Build all the test infrastructure on Linux, then run it on NetBSD under Linux emulation", or "Build and run it on Linux, with the individual tests run as a ssh to a NetBSD box" - as the more interesting part is what the tests show David
Re: [GSoC] Emulating missing Linux syscalls project questions
> The Linux Test Project (http://linux-test-project.github.io) would help > not only with finding missing syscalls, but also with finding bugs / > missing functionality in the existing Linux emul code. Yes this is a great idea! Although my interpretation of the project idea is that the expectations are that the binary is functional by the end of the summer. I obviously will not be able to implement all missing syscalls by the end of the summer, so I would have to draw an arbitrary line as to what I would/would not try to implement. Which brings me to my next comment. > It would be nice to have this running on NetBSD. In what way exactly does the LTP not function on NetBSD? I tried it today and (after a few hours of troubleshooting) seemingly got it to work. Some assorted notes about what I did/what it took to get it to work: - I only looked at system call tests (so for all I know the other types of tests could be what you're referring to). - The actual testcases themselves can be trivially (just add -static) statically compiled on any Linux distro and can be run individually just fine, but the rest of the testing infrastructure cannot (because glibc). (Most of my time spent on this was dealing with glibc versions) - Otherwise you can compile everything normally on OpenSUSE 15.4, and with suse15_base installed the binaries will *almost* just work. - The ltp-pan binary does depend on /dev/kmsg (which doesn't currently exist in the emul code), but only writes to it, so touch /emul/linux/dev/kmsg is sufficient to trick it into working. - As expected, lots of tests fail, but also lots of tests pass! I haven't looked to hard into the failing tests (yet), but I didn't find anything too surprising in the list of failing tests. Overall, I did enjoy going down this rabbit hole! It definitely taught me a few new things about how the emul subsystem behaves. Theo(dore)
Re: [GSoC] Emulating missing Linux syscalls project questions
Hi Theodore -- The Linux Test Project (http://linux-test-project.github.io) would help not only with finding missing syscalls, but also with finding bugs / missing functionality in the existing Linux emul code. It would be nice to have this running on NetBSD. Take care, Jared On Mon, 13 Mar 2023, Theodore Preduta wrote: As the subject suggests, I think the emulating missing Linux syscalls project might be fun. I am just wondering (1) Is there any documentation on the internals of this subsystem? (manpage and wiki seem to just be how to use it) (2) Is there a better binary-finding strategy than trying Linux binaries on NetBSD, and if they fail (have a script) compare the output of strace from a Linux run of the program with the table in sys/compat/linux/arch/*/linux_syscalls.c? (3) Do y'all have any suggestions for binaries? :P (4) Most of the (failing) binaries I've found so far seem to have the epoll set of system calls in common, is there's some technical reason why that's not been implemented yet? (or it it just a matter of no one has done it yet) Thanks in advance, Theo(dore)
Re: [GSoC] Emulating missing Linux syscalls project questions
On Mon, Mar 13, 2023 at 12:10:11AM -0400, Theodore Preduta wrote: > (1) Is there any documentation on the internals of this subsystem? (manpage > and wiki seem to just be how to use it) I wrote a series of articles 22 years ago on the topic. All links are now dead, but we still have them thanks to Internat Archive: https://web.archive.org/web/20050212165754/http://www.onlamp.com/pub/a/onlamp/2001/05/10/linux_bsd.html https://web.archive.org/web/20151001183225/http://archive.oreilly.com/pub/a/onlamp/2001/05/17/linux_bsd.html https://web.archive.org/web/20151001195242/http://archive.oreilly.com/pub/a/onlamp/2001/06/07/linux_bsd.html https://web.archive.org/web/20150910035226/http://archive.oreilly.com/pub/a/onlamp/2001/06/21/linux_bsd.html https://web.archive.org/web/20150910011921/http://archive.oreilly.com/pub/a/onlamp/2001/08/09/linux_bsd.html > (2) Is there a better binary-finding strategy than trying Linux binaries on > NetBSD In my experience, this is the best way of doing it. -- Emmanuel Dreyfus m...@netbsd.org
Re: [GSoC] Emulating missing Linux syscalls project questions
> (2) Is there a better binary-finding strategy than trying Linux > binaries on NetBSD, and if they fail (have a script) compare the > output of strace from a Linux run of the program with the table in > sys/compat/linux/arch/*/linux_syscalls.c? Better? Maybe, maybe not. But what I did in a similar case (an emulator that emulated just userland, handling directly anything that traps to the kernel on real hardware) was to `implement' any unimplemented syscalls with code that just prints a message and terminates. Here, that would map into unimplemented syscalls printing/logging something and killing the process. Obviously, that code would not survive into the end result, but something like it might be a useful intermediate step. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
[GSoC] Emulating missing Linux syscalls project questions
As the subject suggests, I think the emulating missing Linux syscalls project might be fun. I am just wondering (1) Is there any documentation on the internals of this subsystem? (manpage and wiki seem to just be how to use it) (2) Is there a better binary-finding strategy than trying Linux binaries on NetBSD, and if they fail (have a script) compare the output of strace from a Linux run of the program with the table in sys/compat/linux/arch/*/linux_syscalls.c? (3) Do y'all have any suggestions for binaries? :P (4) Most of the (failing) binaries I've found so far seem to have the epoll set of system calls in common, is there's some technical reason why that's not been implemented yet? (or it it just a matter of no one has done it yet) Thanks in advance, Theo(dore)
Re: Emulating missing linux syscalls
On Tue, Apr 19, 2022 at 4:38 AM Joerg Sonnenberger wrote:
>
> Am Tue, Apr 19, 2022 at 02:39:44AM +0530 schrieb Piyush Sachdeva:
> > On Sat, Apr 16, 2022 at 2:06 AM Joerg Sonnenberger wrote:
> > >
> > > Am Wed, Apr 13, 2022 at 09:51:31PM - schrieb Christos Zoulas:
> > > > In article , Joerg Sonnenberger
> > > > wrote:
> > > > >Am Tue, Apr 12, 2022 at 04:56:05PM - schrieb Christos Zoulas:
> > > >
> > > > >splice(2) as a concept is much older than the current Linux
> > > > >implementation.
> > > > >There is no reason why zero-copying for sockets should require a
> > > > >different system call for zero-copying from/to pipes. There are valid
> > > > >reasons for other combinations, too. Consider /bin/cp for example.
> > > >
> > > > You don't need two system calls because the kernel knows the type of
> > > > the file descriptors and can dispatch to different implementations.
> > > > One of the questions is do you provide the means to pass an additional
> > > > header/trailer to the output data like FreeBSD does for its sendfile(2)
> > > > implementation?
> > > >
> > > > int
> > > > splice(int infd, off_t *inoff, int outfd, off_t *outoff, size_t len,
> > > > const struct {
> > > > struct iov *head;
> > > > size_t headcnt;
> > > > struct iov *tail;
> > > > size_t tailcnt;
> > > > } *ht, int flags);
> > >
> > > There are essentially two use cases here:
> > > (1) I want a simple interface to transfer data from one fd to another
> > > without extra copies.
> > >
> > > (2) I wanto avoid copies AND I want to avoid system calls.
> > >
> > > For the former:
> > > int splice(int dstfd, int srcfd, off_t *len);
> > >
> > > is more than good enough. "Transfer up to [*len] octets from srcfd to
> > > dstfd, updating [len] with the actually transferred amount and returning
> > > the first error if any.
> > >
> > > For the second category, an interface more like the posix_spawn
> > > interface (but without all the extra allocations) would be useful.
> > >
> >
> > Therefore, having the above const struct *ht to support
> > mmap() will be a good option I guess.
>
> It covers a very limited subset of the desired options. Basically, what
> you want in this case is something like:
>
> int splicev(int dstfd, struct spliceop ops[], size_t *lenops, off_t
> *outoff);
>
> where spliceops is used to specify the supported operations:
> - read from a fd with possible seek
> - read from memory
> - seek output
> and maybe other operations I can't think of right now. lenops provides
> the number of operations in input and the remaining operations on
> return, outoff is the remaining output in the current block. Some
> variant of this might be possible.
Thank you Joerg and Christos for helping me with this.
I have successfully submitted a proposal for this project through the GSoC
portal. Hope to make the cut this time :)
--
Regards,
Piyush
Re: Emulating missing linux syscalls
Am Tue, Apr 19, 2022 at 02:39:44AM +0530 schrieb Piyush Sachdeva:
> On Sat, Apr 16, 2022 at 2:06 AM Joerg Sonnenberger wrote:
> >
> > Am Wed, Apr 13, 2022 at 09:51:31PM - schrieb Christos Zoulas:
> > > In article , Joerg Sonnenberger
> > > wrote:
> > > >Am Tue, Apr 12, 2022 at 04:56:05PM - schrieb Christos Zoulas:
> > >
> > > >splice(2) as a concept is much older than the current Linux
> > > >implementation.
> > > >There is no reason why zero-copying for sockets should require a
> > > >different system call for zero-copying from/to pipes. There are valid
> > > >reasons for other combinations, too. Consider /bin/cp for example.
> > >
> > > You don't need two system calls because the kernel knows the type of
> > > the file descriptors and can dispatch to different implementations.
> > > One of the questions is do you provide the means to pass an additional
> > > header/trailer to the output data like FreeBSD does for its sendfile(2)
> > > implementation?
> > >
> > > int
> > > splice(int infd, off_t *inoff, int outfd, off_t *outoff, size_t len,
> > > const struct {
> > > struct iov *head;
> > > size_t headcnt;
> > > struct iov *tail;
> > > size_t tailcnt;
> > > } *ht, int flags);
> >
> > There are essentially two use cases here:
> > (1) I want a simple interface to transfer data from one fd to another
> > without extra copies.
> >
> > (2) I wanto avoid copies AND I want to avoid system calls.
> >
> > For the former:
> > int splice(int dstfd, int srcfd, off_t *len);
> >
> > is more than good enough. "Transfer up to [*len] octets from srcfd to
> > dstfd, updating [len] with the actually transferred amount and returning
> > the first error if any.
> >
> > For the second category, an interface more like the posix_spawn
> > interface (but without all the extra allocations) would be useful.
> >
>
> Therefore, having the above const struct *ht to support
> mmap() will be a good option I guess.
It covers a very limited subset of the desired options. Basically, what
you want in this case is something like:
int splicev(int dstfd, struct spliceop ops[], size_t *lenops, off_t
*outoff);
where spliceops is used to specify the supported operations:
- read from a fd with possible seek
- read from memory
- seek output
and maybe other operations I can't think of right now. lenops provides
the number of operations in input and the remaining operations on
return, outoff is the remaining output in the current block. Some
variant of this might be possible.
Joerg
Re: Emulating missing linux syscalls
On Sat, Apr 16, 2022 at 2:06 AM Joerg Sonnenberger wrote:
>
> Am Wed, Apr 13, 2022 at 09:51:31PM - schrieb Christos Zoulas:
> > In article , Joerg Sonnenberger
> > wrote:
> > >Am Tue, Apr 12, 2022 at 04:56:05PM - schrieb Christos Zoulas:
> >
> > >splice(2) as a concept is much older than the current Linux implementation.
> > >There is no reason why zero-copying for sockets should require a
> > >different system call for zero-copying from/to pipes. There are valid
> > >reasons for other combinations, too. Consider /bin/cp for example.
> >
> > You don't need two system calls because the kernel knows the type of
> > the file descriptors and can dispatch to different implementations.
> > One of the questions is do you provide the means to pass an additional
> > header/trailer to the output data like FreeBSD does for its sendfile(2)
> > implementation?
> >
> > int
> > splice(int infd, off_t *inoff, int outfd, off_t *outoff, size_t len,
> > const struct {
> > struct iov *head;
> > size_t headcnt;
> > struct iov *tail;
> > size_t tailcnt;
> > } *ht, int flags);
>
> There are essentially two use cases here:
> (1) I want a simple interface to transfer data from one fd to another
> without extra copies.
>
> (2) I wanto avoid copies AND I want to avoid system calls.
>
> For the former:
> int splice(int dstfd, int srcfd, off_t *len);
>
> is more than good enough. "Transfer up to [*len] octets from srcfd to
> dstfd, updating [len] with the actually transferred amount and returning
> the first error if any.
>
> For the second category, an interface more like the posix_spawn
> interface (but without all the extra allocations) would be useful.
>
Therefore, having the above const struct *ht to support
mmap() will be a good option I guess.
> > >I was saying that the Linux system call can be implemented without a
> > >kernel backend, because I don't consider zero copy a necessary part of
> > >the interface contract. It's a perfectly valid, if a bit slower
> > >implementation to do allocate a kernel buffer and do IO via that.
> >
> > Of course, but how do you make an existing binary use it? LD_PRELOAD
> > a binary to override the symbol in the linux glibc? By that logic you
> > don't need an in kernel linux emulation, you can do it all in userland :-)
>
> You still provide the system call as front end, but internally implement
> it on top of regular read/write to a temporary buffer.
>
Got it, thank you Joerg!
--
Regards,
Piyush
Re: Emulating missing linux syscalls
Am Wed, Apr 13, 2022 at 09:51:31PM - schrieb Christos Zoulas:
> In article , Joerg Sonnenberger
> wrote:
> >Am Tue, Apr 12, 2022 at 04:56:05PM - schrieb Christos Zoulas:
>
> >splice(2) as a concept is much older than the current Linux implementation.
> >There is no reason why zero-copying for sockets should require a
> >different system call for zero-copying from/to pipes. There are valid
> >reasons for other combinations, too. Consider /bin/cp for example.
>
> You don't need two system calls because the kernel knows the type of
> the file descriptors and can dispatch to different implementations.
> One of the questions is do you provide the means to pass an additional
> header/trailer to the output data like FreeBSD does for its sendfile(2)
> implementation?
>
> int
> splice(int infd, off_t *inoff, int outfd, off_t *outoff, size_t len,
> const struct {
> struct iov *head;
> size_t headcnt;
> struct iov *tail;
> size_t tailcnt;
> } *ht, int flags);
There are essentially two use cases here:
(1) I want a simple interface to transfer data from one fd to another
without extra copies.
(2) I wanto avoid copies AND I want to avoid system calls.
For the former:
int splice(int dstfd, int srcfd, off_t *len);
is more than good enough. "Transfer up to [*len] octets from srcfd to
dstfd, updating [len] with the actually transferred amount and returning
the first error if any.
For the second category, an interface more like the posix_spawn
interface (but without all the extra allocations) would be useful.
> >I was saying that the Linux system call can be implemented without a
> >kernel backend, because I don't consider zero copy a necessary part of
> >the interface contract. It's a perfectly valid, if a bit slower
> >implementation to do allocate a kernel buffer and do IO via that.
>
> Of course, but how do you make an existing binary use it? LD_PRELOAD
> a binary to override the symbol in the linux glibc? By that logic you
> don't need an in kernel linux emulation, you can do it all in userland :-)
You still provide the system call as front end, but internally implement
it on top of regular read/write to a temporary buffer.
Joerg
Re: Emulating missing linux syscalls
On Thu, Apr 14, 2022 at 3:22 AM Christos Zoulas wrote:
>
> In article , Joerg Sonnenberger
> wrote:
> >Am Tue, Apr 12, 2022 at 04:56:05PM - schrieb Christos Zoulas:
>
> >splice(2) as a concept is much older than the current Linux implementation.
> >There is no reason why zero-copying for sockets should require a
> >different system call for zero-copying from/to pipes. There are valid
> >reasons for other combinations, too. Consider /bin/cp for example.
>
I was under the assumption that zero-copying would be a preference.
I did go through /bin/cp and the important copy_file() function. There
mmap() is being used and then data is written to the destination using
write(2) in chunks. Thanks for this Joerg!
> You don't need two system calls because the kernel knows the type of
> the file descriptors and can dispatch to different implementations.
Yes. Therefore, I am assuming that only one general splice(2) function
will be implemented and in case it's supplied a socketfd, it will behave like
sendfile(2). (as it is also clear from the function def you have
provided under.)
Also, add the sendfile(2) functionality and have it invoke splice(2).
> One of the questions is do you provide the means to pass an additional
> header/trailer to the output data like FreeBSD does for its sendfile(2)
> implementation?
>
> int
> splice(int infd, off_t *inoff, int outfd, off_t *outoff, size_t len,
> const struct {
> struct iov *head;
> size_t headcnt;
> struct iov *tail;
> size_t tailcnt;
> } *ht, int flags);
>
I will be more than happy to provide the functionality
(taking reference from writev(2) for the struct iovec
and the FreeBSD implementation of sendfile(2)).
> >I was saying that the Linux system call can be implemented without a
> >kernel backend, because I don't consider zero copy a necessary part of
> >the interface contract. It's a perfectly valid, if a bit slower
> >implementation to do allocate a kernel buffer and do IO via that.
>
Right, Joerg!
As I was initially also hoping to broaden the project by actually
adding the syscall
to the NetBSD kernel as well (adds a feature) and then have the
compat_linux layer
profit from that call. Unless that is something you are trying to
avoid/steer away from.
Now the final question for me is:
The splice() prototype that you just mentioned above, Christos. Is
that for a NetBSD syscall
(as I would hope given the struct iovec parameter) and then have both
splice(2) and
sendfile(2) implemented in compat_linux layer profiting from this syscall?
Or is it just splice(2) and sendfile(2) (which will call splice(2))
both in the linux layer
only?
> Of course, but how do you make an existing binary use it? LD_PRELOAD
> a binary to override the symbol in the linux glibc? By that logic you
> don't need an in kernel linux emulation, you can do it all in userland :-)
>
Christos, if you can shine some more light on this.
I guess this will make a great proposal and I will send you something by Monday,
I hope, for a first pass.
Hope to hear from you soon.
--
Regards,
Piyush
Re: Emulating missing linux syscalls
In article , Joerg Sonnenberger wrote:
>Am Tue, Apr 12, 2022 at 04:56:05PM - schrieb Christos Zoulas:
>splice(2) as a concept is much older than the current Linux implementation.
>There is no reason why zero-copying for sockets should require a
>different system call for zero-copying from/to pipes. There are valid
>reasons for other combinations, too. Consider /bin/cp for example.
You don't need two system calls because the kernel knows the type of
the file descriptors and can dispatch to different implementations.
One of the questions is do you provide the means to pass an additional
header/trailer to the output data like FreeBSD does for its sendfile(2)
implementation?
int
splice(int infd, off_t *inoff, int outfd, off_t *outoff, size_t len,
const struct {
struct iov *head;
size_t headcnt;
struct iov *tail;
size_t tailcnt;
} *ht, int flags);
>I was saying that the Linux system call can be implemented without a
>kernel backend, because I don't consider zero copy a necessary part of
>the interface contract. It's a perfectly valid, if a bit slower
>implementation to do allocate a kernel buffer and do IO via that.
Of course, but how do you make an existing binary use it? LD_PRELOAD
a binary to override the symbol in the linux glibc? By that logic you
don't need an in kernel linux emulation, you can do it all in userland :-)
christos
Re: Emulating missing linux syscalls
Am Tue, Apr 12, 2022 at 04:56:05PM - schrieb Christos Zoulas: > In article , Joerg Sonnenberger > wrote: > >Am Tue, Apr 12, 2022 at 12:29:21PM - schrieb Christos Zoulas: > >> In article > >, > >> Piyush Sachdeva wrote: > >> >-=-=-=-=-=- > >> > > >> >Dear Stephen Borrill, > >> >My name is Piyush, and I was looking into the > >> >'Emulating missing Linux syscalls' project hoping to contribute > >> >to this year's GSoC. > >> > > >> >I wanted to be sure of a few basic things before I go ahead: > >> >- linux binaries are found in- src/sys/compat/linux > >> >- particular implementation in - src/sys/compat/linux/common > >> >- a few architecture-specific implementations in- > >> > src/sys/compat/linux/arch/. > >> >- The src/sys/compat/linux/arch//linux_syscalls.c file > >> > lists of system calls, and states if a particular syscall is present or > >> >not. > >> > > >> >I was planning to work on the 'sendfile()' syscall, which I believe > >> >is unimplemented for amd64 and a few other architectures as well. > >> > > >> >Considering the above points, I was hoping you could point me in > >> >the right direction for this project. Hope to hear from you soon. > >> > >> I would look into porting the FreeBSD implementation of sendfile to NetBSD. > > > >sendfile(2) for Linux compat can be emulated in the kernel without > >backing. That said, a real splice(2) or even splicev(2) would be really > >nice to have. But that's a different project and arguable, a potentially > >more generally useful one, too. > > > Yes, splice is more general (as opposed to send a file to a socket), but I > think splice has limitations too (one of the fds needs to be a pipe). > Is that true only for linux? splice(2) as a concept is much older than the current Linux implementation. There is no reason why zero-copying for sockets should require a different system call for zero-copying from/to pipes. There are valid reasons for other combinations, too. Consider /bin/cp for example. I was saying that the Linux system call can be implemented without a kernel backend, because I don't consider zero copy a necessary part of the interface contract. It's a perfectly valid, if a bit slower implementation to do allocate a kernel buffer and do IO via that. Joerg
Re: Emulating missing linux syscalls
Thank you Christos and Joerg! On Tue, Apr 12, 2022 at 10:26 PM Christos Zoulas wrote: > > In article , Joerg Sonnenberger > wrote: > > >>> I would look into porting the FreeBSD implementation of sendfile to > >>> NetBSD. > I had a look at the FreeBSD implementation. What I found was- - linux_sendfile() function in freebsd-src/sys/compat/linux/linux_socket.c ends up calling linux_sendfile_common() (in the same file) which in turn calls fo_sendfile(). I am guessing, this is supported by sendfile(2) syscall which is present in the FreeBSD kernel. - Therefore in case the implementation just needs to be ported, then it would make for a very simple project. > > >> sendfile(2) for Linux compat can be emulated in the kernel without > >> backing. Joerg, would you please explain to me how this will be possible? As I understand, anything in the compat layer needs backing functions, and I didn't find anything pertaining to sendfile(2) as a syscall in the NetBSD kernel. Or maybe you are talking about using in-kernel support functions, which would have been used to support sendfile(2) had it been present? In this case, I guess, these functions support zero-copy and it will be great if you could point me to them. > > That said, a real splice(2) or even splicev(2) would be really > >> nice to have. But that's a different project and arguable, a potentially > >> more generally useful one, too. > > > > > Yes, splice is more general (as opposed to send a file to a socket), but I > > think splice has limitations too (one of the fds needs to be a pipe). > > Is that true only for linux? > splice(2) for sure is an amazing project, and as Christos said, splice(2) requires one of the fds to be a pipe. Given that splice is only implemented in linux (from what I found), we can have a slightly different implementation in the NetBSD kernel according to requirements (if allowed). I haven't found sendfile(2) or splice(2) syscall in the NetBSD kernel. I did find a reference to sendfile(), but that was for the tftp daemon. It will make an interesting project, to add support for these calls in the NetBSD kernel first. Later these very syscalls can back functionality for the linux compat layer as well. What I wish to know is, what other zero-copy functionality is already present in the NetBSD kernel, which can support these two system calls. I hope this makes some sense and please do correct me where I have made a wrong assumption. Hope to hear from you soon -- Regards, Piyush
Emulating missing linux syscalls
Hello, I have submitted my draft proposal on gsoc platform. Could you review it and let me know what you think? Thank you in advance Ahmed Ehab
Re: Emulating missing linux syscalls
In article , Joerg Sonnenberger wrote: >Am Tue, Apr 12, 2022 at 12:29:21PM - schrieb Christos Zoulas: >> In article >, >> Piyush Sachdeva wrote: >> >-=-=-=-=-=- >> > >> >Dear Stephen Borrill, >> >My name is Piyush, and I was looking into the >> >'Emulating missing Linux syscalls' project hoping to contribute >> >to this year's GSoC. >> > >> >I wanted to be sure of a few basic things before I go ahead: >> >- linux binaries are found in- src/sys/compat/linux >> >- particular implementation in - src/sys/compat/linux/common >> >- a few architecture-specific implementations in- >> > src/sys/compat/linux/arch/. >> >- The src/sys/compat/linux/arch//linux_syscalls.c file >> > lists of system calls, and states if a particular syscall is present or >> >not. >> > >> >I was planning to work on the 'sendfile()' syscall, which I believe >> >is unimplemented for amd64 and a few other architectures as well. >> > >> >Considering the above points, I was hoping you could point me in >> >the right direction for this project. Hope to hear from you soon. >> >> I would look into porting the FreeBSD implementation of sendfile to NetBSD. > >sendfile(2) for Linux compat can be emulated in the kernel without >backing. That said, a real splice(2) or even splicev(2) would be really >nice to have. But that's a different project and arguable, a potentially >more generally useful one, too. Yes, splice is more general (as opposed to send a file to a socket), but I think splice has limitations too (one of the fds needs to be a pipe). Is that true only for linux? christos
Re: Emulating missing linux syscalls
Am Tue, Apr 12, 2022 at 12:29:21PM - schrieb Christos Zoulas: > In article > , > Piyush Sachdeva wrote: > >-=-=-=-=-=- > > > >Dear Stephen Borrill, > >My name is Piyush, and I was looking into the > >'Emulating missing Linux syscalls' project hoping to contribute > >to this year's GSoC. > > > >I wanted to be sure of a few basic things before I go ahead: > >- linux binaries are found in- src/sys/compat/linux > >- particular implementation in - src/sys/compat/linux/common > >- a few architecture-specific implementations in- > > src/sys/compat/linux/arch/. > >- The src/sys/compat/linux/arch//linux_syscalls.c file > > lists of system calls, and states if a particular syscall is present or > >not. > > > >I was planning to work on the 'sendfile()' syscall, which I believe > >is unimplemented for amd64 and a few other architectures as well. > > > >Considering the above points, I was hoping you could point me in > >the right direction for this project. Hope to hear from you soon. > > I would look into porting the FreeBSD implementation of sendfile to NetBSD. sendfile(2) for Linux compat can be emulated in the kernel without backing. That said, a real splice(2) or even splicev(2) would be really nice to have. But that's a different project and arguable, a potentially more generally useful one, too. Joerg
Re: Emulating missing linux syscalls
In article , Piyush Sachdeva wrote: >-=-=-=-=-=- > >Dear Stephen Borrill, >My name is Piyush, and I was looking into the >'Emulating missing Linux syscalls' project hoping to contribute >to this year's GSoC. > >I wanted to be sure of a few basic things before I go ahead: >- linux binaries are found in- src/sys/compat/linux >- particular implementation in - src/sys/compat/linux/common >- a few architecture-specific implementations in- > src/sys/compat/linux/arch/. >- The src/sys/compat/linux/arch//linux_syscalls.c file > lists of system calls, and states if a particular syscall is present or >not. > >I was planning to work on the 'sendfile()' syscall, which I believe >is unimplemented for amd64 and a few other architectures as well. > >Considering the above points, I was hoping you could point me in >the right direction for this project. Hope to hear from you soon. I would look into porting the FreeBSD implementation of sendfile to NetBSD. christos
Emulating missing linux syscalls
Dear Stephen Borrill, My name is Piyush, and I was looking into the 'Emulating missing Linux syscalls' project hoping to contribute to this year's GSoC. I wanted to be sure of a few basic things before I go ahead: - linux binaries are found in- src/sys/compat/linux - particular implementation in - src/sys/compat/linux/common - a few architecture-specific implementations in- src/sys/compat/linux/arch/. - The src/sys/compat/linux/arch//linux_syscalls.c file lists of system calls, and states if a particular syscall is present or not. I was planning to work on the 'sendfile()' syscall, which I believe is unimplemented for amd64 and a few other architectures as well. Considering the above points, I was hoping you could point me in the right direction for this project. Hope to hear from you soon. -- Regards, Piyush
Re: Tentative Proposal for Gsoc : Emulating missing linux syscalls (350h)
On Sat, Mar 12, 2022 at 08:54:39AM +0100, Martin Husemann wrote: > After reading your proposal it seems to me you may have misunderstood > something about the project itself (and that is likely because it's > description is very vague). Aye, I've updated the wiki page a little bit to clarify somethings.
Re: Tentative Proposal for Gsoc : Emulating missing linux syscalls (350h)
On Sat, Mar 12, 2022 at 12:01:25AM +0200, Ahmed bahloul wrote: > Hello, > I have made my Tentative Proposal for : Emulating missing linux syscalls > project. Hello Ahmed, great that you are interested in this project and enhancing NetBSD! After reading your proposal it seems to me you may have misunderstood something about the project itself (and that is likely because it's description is very vague). The point of emulated Linux syscalls is: - they implement the exact same ABI that the original Linux system call has - they typically reuse as much of the existing kernel internal infrastructure available, so typically end up being a very tiny wrapper that just translates the ABI to structures the/a similar NetBSD native syscall implementation understands. So points 2 and 3 of your proposal make no sense. For point 1 the typical apporach would be very adhoc and empirical: try to run some Linux applications on you NetBSD machine and see which ones work and which fail. You mentors might suggest special interest apps here. For the ones that fail to run: find out which syscalls fail or are missing. The other (more systematical but tedious) aproach would be to compare the syscall tables, find out which of the missing ones are actually used in current Linux libc, and check if any "serious" applications or libs do use them. The implementation often is trivial and sometimes very much not - it totally depends on the syscall (and if some important syscall would be identified where implementation is far from trivial or need general kernel changes we likely would push them outside the scope of this project and make them a dedicated project for later). Testing is also simple if you do it adhoc during this project: just create a Linux binary to exercise the system call and run it under emulation. The more general "testing all our Linux syscall emulations" project is a separate project suggestion and a bit more involved, as it expects fully automatic test runs as the net result (like we do for native syscalls and netbsd 32 bit emulated syscalls on 64bit machines). Martin
