Gsoc, Emulating missing linux syscalls
Hi! Sorry for such a late contact, I’ve already written a proper proposal for apply compat_linux project by following the instruction and guidelines from the link given by the NetBSD Gsoc idea list page.I try to answer as much questions as I can in the guidelines page and by follow the questions,i also try and compile the NetBSD kernel and read through the code base that relevant with compat_linux.And I think it’s very weird to apply Gsoc project and submit the proposal without having any contact with the developer/maintainer of the project.So anyways, sorry for dragging this till now. As a junior developer I’ve been working on several open source project on GitHub with the git/github pr workflow before,but as I read through the NetBSD contribute guidelines page, it seems the traditional mail list and problem database /problem report is the way to collaborate in NetBSD,so is it a “must do” here to use the cvs/svn version control software and the mail list for contribute here? In the compat_linux project page, saying that "you should find at least one Linux binary that does not yet run on NetBSD using compat_linux to use as a test case (your mentor may have suggestions)” Is there any suggestion on which linux binary isn’t yet supported by compat_linux as a test case?
Re: [GSoC] Emulating missing Linux syscalls project questions
On Sun, 19 Mar 2023 at 04:22, Theodore Preduta wrote: > > > The Linux Test Project (http://linux-test-project.github.io) would help > > not only with finding missing syscalls, but also with finding bugs / > > missing functionality in the existing Linux emul code. > > Yes this is a great idea! Although my interpretation of the project > idea is that the expectations are that the binary is functional by the > end of the summer. I obviously will not be able to implement all > missing syscalls by the end of the summer, so I would have to draw an > arbitrary line as to what I would/would not try to implement. > > Which brings me to my next comment. > > > It would be nice to have this running on NetBSD. > > In what way exactly does the LTP not function on NetBSD? I tried it > today and (after a few hours of troubleshooting) seemingly got it to work. > > Some assorted notes about what I did/what it took to get it to work: > > - I only looked at system call tests (so for all I know the other types > of tests could be what you're referring to). > > - The actual testcases themselves can be trivially (just add -static) > statically compiled on any Linux distro and can be run individually just > fine, but the rest of the testing infrastructure cannot (because glibc). > (Most of my time spent on this was dealing with glibc versions) > > - Otherwise you can compile everything normally on OpenSUSE 15.4, and > with suse15_base installed the binaries will *almost* just work. > > - The ltp-pan binary does depend on /dev/kmsg (which doesn't currently > exist in the emul code), but only writes to it, so touch > /emul/linux/dev/kmsg is sufficient to trick it into working. > > - As expected, lots of tests fail, but also lots of tests pass! I > haven't looked to hard into the failing tests (yet), but I didn't find > anything too surprising in the list of failing tests. > > Overall, I did enjoy going down this rabbit hole! It definitely taught > me a few new things about how the emul subsystem behaves. I think making progress on running LTP on NetBSD (*), and fixing up a useful subset of missing or incomplete syscall implementations would make an _excellent_ GSoC project :) *: Even if its "Build all the test infrastructure on Linux, then run it on NetBSD under Linux emulation", or "Build and run it on Linux, with the individual tests run as a ssh to a NetBSD box" - as the more interesting part is what the tests show David
Re: [GSoC] Emulating missing Linux syscalls project questions
> The Linux Test Project (http://linux-test-project.github.io) would help > not only with finding missing syscalls, but also with finding bugs / > missing functionality in the existing Linux emul code. Yes this is a great idea! Although my interpretation of the project idea is that the expectations are that the binary is functional by the end of the summer. I obviously will not be able to implement all missing syscalls by the end of the summer, so I would have to draw an arbitrary line as to what I would/would not try to implement. Which brings me to my next comment. > It would be nice to have this running on NetBSD. In what way exactly does the LTP not function on NetBSD? I tried it today and (after a few hours of troubleshooting) seemingly got it to work. Some assorted notes about what I did/what it took to get it to work: - I only looked at system call tests (so for all I know the other types of tests could be what you're referring to). - The actual testcases themselves can be trivially (just add -static) statically compiled on any Linux distro and can be run individually just fine, but the rest of the testing infrastructure cannot (because glibc). (Most of my time spent on this was dealing with glibc versions) - Otherwise you can compile everything normally on OpenSUSE 15.4, and with suse15_base installed the binaries will *almost* just work. - The ltp-pan binary does depend on /dev/kmsg (which doesn't currently exist in the emul code), but only writes to it, so touch /emul/linux/dev/kmsg is sufficient to trick it into working. - As expected, lots of tests fail, but also lots of tests pass! I haven't looked to hard into the failing tests (yet), but I didn't find anything too surprising in the list of failing tests. Overall, I did enjoy going down this rabbit hole! It definitely taught me a few new things about how the emul subsystem behaves. Theo(dore)
Re: [GSoC] Emulating missing Linux syscalls project questions
Hi Theodore -- The Linux Test Project (http://linux-test-project.github.io) would help not only with finding missing syscalls, but also with finding bugs / missing functionality in the existing Linux emul code. It would be nice to have this running on NetBSD. Take care, Jared On Mon, 13 Mar 2023, Theodore Preduta wrote: As the subject suggests, I think the emulating missing Linux syscalls project might be fun. I am just wondering (1) Is there any documentation on the internals of this subsystem? (manpage and wiki seem to just be how to use it) (2) Is there a better binary-finding strategy than trying Linux binaries on NetBSD, and if they fail (have a script) compare the output of strace from a Linux run of the program with the table in sys/compat/linux/arch/*/linux_syscalls.c? (3) Do y'all have any suggestions for binaries? :P (4) Most of the (failing) binaries I've found so far seem to have the epoll set of system calls in common, is there's some technical reason why that's not been implemented yet? (or it it just a matter of no one has done it yet) Thanks in advance, Theo(dore)
Re: [GSoC] Emulating missing Linux syscalls project questions
On Mon, Mar 13, 2023 at 12:10:11AM -0400, Theodore Preduta wrote: > (1) Is there any documentation on the internals of this subsystem? (manpage > and wiki seem to just be how to use it) I wrote a series of articles 22 years ago on the topic. All links are now dead, but we still have them thanks to Internat Archive: https://web.archive.org/web/20050212165754/http://www.onlamp.com/pub/a/onlamp/2001/05/10/linux_bsd.html https://web.archive.org/web/20151001183225/http://archive.oreilly.com/pub/a/onlamp/2001/05/17/linux_bsd.html https://web.archive.org/web/20151001195242/http://archive.oreilly.com/pub/a/onlamp/2001/06/07/linux_bsd.html https://web.archive.org/web/20150910035226/http://archive.oreilly.com/pub/a/onlamp/2001/06/21/linux_bsd.html https://web.archive.org/web/20150910011921/http://archive.oreilly.com/pub/a/onlamp/2001/08/09/linux_bsd.html > (2) Is there a better binary-finding strategy than trying Linux binaries on > NetBSD In my experience, this is the best way of doing it. -- Emmanuel Dreyfus m...@netbsd.org
Re: [GSoC] Emulating missing Linux syscalls project questions
> (2) Is there a better binary-finding strategy than trying Linux > binaries on NetBSD, and if they fail (have a script) compare the > output of strace from a Linux run of the program with the table in > sys/compat/linux/arch/*/linux_syscalls.c? Better? Maybe, maybe not. But what I did in a similar case (an emulator that emulated just userland, handling directly anything that traps to the kernel on real hardware) was to `implement' any unimplemented syscalls with code that just prints a message and terminates. Here, that would map into unimplemented syscalls printing/logging something and killing the process. Obviously, that code would not survive into the end result, but something like it might be a useful intermediate step. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
[GSoC] Emulating missing Linux syscalls project questions
As the subject suggests, I think the emulating missing Linux syscalls project might be fun. I am just wondering (1) Is there any documentation on the internals of this subsystem? (manpage and wiki seem to just be how to use it) (2) Is there a better binary-finding strategy than trying Linux binaries on NetBSD, and if they fail (have a script) compare the output of strace from a Linux run of the program with the table in sys/compat/linux/arch/*/linux_syscalls.c? (3) Do y'all have any suggestions for binaries? :P (4) Most of the (failing) binaries I've found so far seem to have the epoll set of system calls in common, is there's some technical reason why that's not been implemented yet? (or it it just a matter of no one has done it yet) Thanks in advance, Theo(dore)
Re: Tentative Proposal for Gsoc : Emulating missing linux syscalls (350h)
On Sat, Mar 12, 2022 at 08:54:39AM +0100, Martin Husemann wrote: > After reading your proposal it seems to me you may have misunderstood > something about the project itself (and that is likely because it's > description is very vague). Aye, I've updated the wiki page a little bit to clarify somethings.
Re: Tentative Proposal for Gsoc : Emulating missing linux syscalls (350h)
On Sat, Mar 12, 2022 at 12:01:25AM +0200, Ahmed bahloul wrote: > Hello, > I have made my Tentative Proposal for : Emulating missing linux syscalls > project. Hello Ahmed, great that you are interested in this project and enhancing NetBSD! After reading your proposal it seems to me you may have misunderstood something about the project itself (and that is likely because it's description is very vague). The point of emulated Linux syscalls is: - they implement the exact same ABI that the original Linux system call has - they typically reuse as much of the existing kernel internal infrastructure available, so typically end up being a very tiny wrapper that just translates the ABI to structures the/a similar NetBSD native syscall implementation understands. So points 2 and 3 of your proposal make no sense. For point 1 the typical apporach would be very adhoc and empirical: try to run some Linux applications on you NetBSD machine and see which ones work and which fail. You mentors might suggest special interest apps here. For the ones that fail to run: find out which syscalls fail or are missing. The other (more systematical but tedious) aproach would be to compare the syscall tables, find out which of the missing ones are actually used in current Linux libc, and check if any "serious" applications or libs do use them. The implementation often is trivial and sometimes very much not - it totally depends on the syscall (and if some important syscall would be identified where implementation is far from trivial or need general kernel changes we likely would push them outside the scope of this project and make them a dedicated project for later). Testing is also simple if you do it adhoc during this project: just create a Linux binary to exercise the system call and run it under emulation. The more general "testing all our Linux syscall emulations" project is a separate project suggestion and a bit more involved, as it expects fully automatic test runs as the net result (like we do for native syscalls and netbsd 32 bit emulated syscalls on 64bit machines). Martin