Gsoc, Emulating missing linux syscalls

2024-03-31 Thread 张正
Hi! Sorry for such a late contact, I’ve already written a proper 
proposal for apply compat_linux project by following the instruction and 
guidelines from the link given by the NetBSD Gsoc idea list page.I try to 
answer as much questions as I can in the guidelines page and by follow the 
questions,i also try and compile the NetBSD kernel and read through the code 
base that relevant with compat_linux.And I think it’s very weird to apply Gsoc 
project and submit the proposal without having any contact with the 
developer/maintainer of the project.So anyways, sorry for dragging this till 
now.
As a junior developer I’ve been working on several open source project 
on GitHub with the git/github pr workflow before,but as I read through the 
NetBSD contribute guidelines page, it seems the traditional mail list and 
problem database /problem report is the way to collaborate in NetBSD,so is it a 
“must do” here to use the cvs/svn version control software and the mail list 
for contribute here?
In the compat_linux project page, saying that 
"you should find at least one Linux binary that does not yet run on 
NetBSD using compat_linux to use as a test case (your   mentor may have 
suggestions)”
Is there any suggestion on which linux binary isn’t yet supported by 
compat_linux as a test case?


Re: [GSoC] Emulating missing Linux syscalls project questions

2023-03-22 Thread David Brownlee
On Sun, 19 Mar 2023 at 04:22, Theodore Preduta  wrote:
>
> > The Linux Test Project (http://linux-test-project.github.io) would help
> > not only with finding missing syscalls, but also with finding bugs /
> > missing functionality in the existing Linux emul code.
>
> Yes this is a great idea!  Although my interpretation of the project
> idea is that the expectations are that the binary is functional by the
> end of the summer.  I obviously will not be able to implement all
> missing syscalls by the end of the summer, so I would have to draw an
> arbitrary line as to what I would/would not try to implement.
>
> Which brings me to my next comment.
>
> > It would be nice to have this running on NetBSD.
>
> In what way exactly does the LTP not function on NetBSD?  I tried it
> today and (after a few hours of troubleshooting) seemingly got it to work.
>
> Some assorted notes about what I did/what it took to get it to work:
>
> - I only looked at system call tests (so for all I know the other types
> of tests could be what you're referring to).
>
> - The actual testcases themselves can be trivially (just add -static)
> statically compiled on any Linux distro and can be run individually just
> fine, but the rest of the testing infrastructure cannot (because glibc).
> (Most of my time spent on this was dealing with glibc versions)
>
> - Otherwise you can compile everything normally on OpenSUSE 15.4, and
> with suse15_base installed the binaries will *almost* just work.
>
> - The ltp-pan binary does depend on /dev/kmsg (which doesn't currently
> exist in the emul code), but only writes to it, so touch
> /emul/linux/dev/kmsg is sufficient to trick it into working.
>
> - As expected, lots of tests fail, but also lots of tests pass!  I
> haven't looked to hard into the failing tests (yet), but I didn't find
> anything too surprising in the list of failing tests.
>
> Overall, I did enjoy going down this rabbit hole!  It definitely taught
> me a few new things about how the emul subsystem behaves.

I think making progress on running LTP on NetBSD (*), and fixing up a
useful subset of missing or incomplete syscall implementations would
make an _excellent_ GSoC project :)

*: Even if its "Build all the test infrastructure on Linux, then run
it on NetBSD under Linux emulation", or "Build and run it on Linux,
with the individual tests run as a ssh to a NetBSD box" - as the more
interesting part is what the tests show

David


Re: [GSoC] Emulating missing Linux syscalls project questions

2023-03-18 Thread Theodore Preduta
> The Linux Test Project (http://linux-test-project.github.io) would help
> not only with finding missing syscalls, but also with finding bugs /
> missing functionality in the existing Linux emul code.

Yes this is a great idea!  Although my interpretation of the project
idea is that the expectations are that the binary is functional by the
end of the summer.  I obviously will not be able to implement all
missing syscalls by the end of the summer, so I would have to draw an
arbitrary line as to what I would/would not try to implement.

Which brings me to my next comment.

> It would be nice to have this running on NetBSD.

In what way exactly does the LTP not function on NetBSD?  I tried it
today and (after a few hours of troubleshooting) seemingly got it to work.

Some assorted notes about what I did/what it took to get it to work:

- I only looked at system call tests (so for all I know the other types
of tests could be what you're referring to).

- The actual testcases themselves can be trivially (just add -static)
statically compiled on any Linux distro and can be run individually just
fine, but the rest of the testing infrastructure cannot (because glibc).
(Most of my time spent on this was dealing with glibc versions)

- Otherwise you can compile everything normally on OpenSUSE 15.4, and
with suse15_base installed the binaries will *almost* just work.

- The ltp-pan binary does depend on /dev/kmsg (which doesn't currently
exist in the emul code), but only writes to it, so touch
/emul/linux/dev/kmsg is sufficient to trick it into working.

- As expected, lots of tests fail, but also lots of tests pass!  I
haven't looked to hard into the failing tests (yet), but I didn't find
anything too surprising in the list of failing tests.

Overall, I did enjoy going down this rabbit hole!  It definitely taught
me a few new things about how the emul subsystem behaves.

Theo(dore)



Re: [GSoC] Emulating missing Linux syscalls project questions

2023-03-18 Thread Jared McNeill

Hi Theodore --

The Linux Test Project (http://linux-test-project.github.io) would help 
not only with finding missing syscalls, but also with finding bugs / 
missing functionality in the existing Linux emul code. It would be nice to 
have this running on NetBSD.


Take care,
Jared


On Mon, 13 Mar 2023, Theodore Preduta wrote:

As the subject suggests, I think the emulating missing Linux syscalls project 
might be fun. I am just wondering


(1) Is there any documentation on the internals of this subsystem? (manpage 
and wiki seem to just be how to use it)


(2) Is there a better binary-finding strategy than trying Linux binaries on 
NetBSD, and if they fail (have a script) compare the output of strace from a 
Linux run of the program with the table in 
sys/compat/linux/arch/*/linux_syscalls.c?


(3) Do y'all have any suggestions for binaries? :P

(4) Most of the (failing) binaries I've found so far seem to have the epoll 
set of system calls in common, is there's some technical reason why that's 
not been implemented yet? (or it it just a matter of no one has done it yet)


Thanks in advance,

Theo(dore)




Re: [GSoC] Emulating missing Linux syscalls project questions

2023-03-12 Thread Emmanuel Dreyfus
On Mon, Mar 13, 2023 at 12:10:11AM -0400, Theodore Preduta wrote:
> (1) Is there any documentation on the internals of this subsystem? (manpage
> and wiki seem to just be how to use it)

I wrote a series of articles 22 years ago on the topic. All links are now dead, 
but we still have them thanks to Internat Archive:
https://web.archive.org/web/20050212165754/http://www.onlamp.com/pub/a/onlamp/2001/05/10/linux_bsd.html
https://web.archive.org/web/20151001183225/http://archive.oreilly.com/pub/a/onlamp/2001/05/17/linux_bsd.html
https://web.archive.org/web/20151001195242/http://archive.oreilly.com/pub/a/onlamp/2001/06/07/linux_bsd.html
https://web.archive.org/web/20150910035226/http://archive.oreilly.com/pub/a/onlamp/2001/06/21/linux_bsd.html
https://web.archive.org/web/20150910011921/http://archive.oreilly.com/pub/a/onlamp/2001/08/09/linux_bsd.html

> (2) Is there a better binary-finding strategy than trying Linux binaries on
> NetBSD

In my experience, this is the best way of doing it.

-- 
Emmanuel Dreyfus
m...@netbsd.org


Re: [GSoC] Emulating missing Linux syscalls project questions

2023-03-12 Thread Mouse
> (2) Is there a better binary-finding strategy than trying Linux
> binaries on NetBSD, and if they fail (have a script) compare the
> output of strace from a Linux run of the program with the table in
> sys/compat/linux/arch/*/linux_syscalls.c?

Better?  Maybe, maybe not.  But what I did in a similar case (an
emulator that emulated just userland, handling directly anything that
traps to the kernel on real hardware) was to `implement' any
unimplemented syscalls with code that just prints a message and
terminates.  Here, that would map into unimplemented syscalls
printing/logging something and killing the process.

Obviously, that code would not survive into the end result, but
something like it might be a useful intermediate step.

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B


[GSoC] Emulating missing Linux syscalls project questions

2023-03-12 Thread Theodore Preduta
As the subject suggests, I think the emulating missing Linux syscalls 
project might be fun. I am just wondering


(1) Is there any documentation on the internals of this subsystem? 
(manpage and wiki seem to just be how to use it)


(2) Is there a better binary-finding strategy than trying Linux binaries 
on NetBSD, and if they fail (have a script) compare the output of strace 
from a Linux run of the program with the table in 
sys/compat/linux/arch/*/linux_syscalls.c?


(3) Do y'all have any suggestions for binaries? :P

(4) Most of the (failing) binaries I've found so far seem to have the 
epoll set of system calls in common, is there's some technical reason 
why that's not been implemented yet? (or it it just a matter of no one 
has done it yet)


Thanks in advance,

Theo(dore)


Re: Tentative Proposal for Gsoc : Emulating missing linux syscalls (350h)

2022-03-12 Thread nia
On Sat, Mar 12, 2022 at 08:54:39AM +0100, Martin Husemann wrote:
> After reading your proposal it seems to me you may have misunderstood
> something about the project itself (and that is likely because it's
> description is very vague).

Aye, I've updated the wiki page a little bit to clarify somethings.


Re: Tentative Proposal for Gsoc : Emulating missing linux syscalls (350h)

2022-03-11 Thread Martin Husemann
On Sat, Mar 12, 2022 at 12:01:25AM +0200, Ahmed bahloul wrote:
> Hello,
> I have made my Tentative Proposal for : Emulating missing linux syscalls
> project.

Hello Ahmed,

great that you are interested in this project and enhancing NetBSD!

After reading your proposal it seems to me you may have misunderstood
something about the project itself (and that is likely because it's
description is very vague).

The point of emulated Linux syscalls is:

 - they implement the exact same ABI that the original Linux system call has
 - they typically reuse as much of the existing kernel internal infrastructure
   available, so typically end up being a very tiny wrapper that just 
   translates the ABI to structures the/a similar NetBSD native syscall
   implementation understands.

So points 2 and 3 of your proposal make no sense.

For point 1 the typical apporach would be very adhoc and empirical: try 
to run some Linux applications on you NetBSD machine and see which ones
work and which fail. You mentors might suggest special interest apps
here.

For the ones that fail to run: find out which syscalls fail or are missing.

The other (more systematical but tedious) aproach would be to compare the
syscall tables, find out which of the missing ones are actually used in
current Linux libc, and check if any "serious" applications or libs do use
them.

The implementation often is trivial and sometimes very much not - it totally
depends on the syscall (and if some important syscall would be identified
where implementation is far from trivial or need general kernel changes
we likely would push them outside the scope of this project and make them
a dedicated project for later).

Testing is also simple if you do it adhoc during this project: just create
a Linux binary to exercise the system call and run it under emulation.

The more general "testing all our Linux syscall emulations" project is a
separate project suggestion and a bit more involved, as it expects fully
automatic test runs as the net result (like we do for native syscalls
and netbsd 32 bit emulated syscalls on 64bit machines).

Martin