Re: Supporting musl libc
On Tuesday, 17 May 2016 at 08:51:01 UTC, Jacob Carlborg wrote: As an alternative to glibc there's a C standard library called musl [1]. This is the C standard library used by ELLCC [2], a cross-compiler based on Clang. This cross-compiler makes it very easy to target other platforms and can be used as the C compiler when building with LDC. The issue is that musl doesn't support the functions defined by execinfo.h: backtrace, backtrace_symbols_fd and backtrace_symbols, since these are glibc extensions. As far as I can see, these functions are used in two places in druntime: src/rt/backtrace/dwarf.d [3] and src/core/runtime.d [4]. The imports of execinfo is guarded by version(CRuntime_Glibc). I see that CRuntime_Glibc is a predefined version identifier defined by the compiler on Linux. I'm not sure how to best handle different C standard libraries when it comes to choosing which one to use. Is it best to choose that when building the compiler or when building druntime? Or can it be a runtime option? [1] https://www.musl-libc.org [2] http://ellcc.org [3] https://github.com/dlang/druntime/blob/master/src/rt/backtrace/dwarf.d#L41 [4] https://github.com/dlang/druntime/blob/master/src/core/runtime.d#L433-L434 I tried to build dmd on musl, and it seems to work relatively well. I need to build dmd 2.067 for bootstrapping. It doesn't build out-of-box, but there's patches floating around. There're some missing symbols in druntime: a couple math related, backtrace() and backtrace_symbols(). The former ones can be workaround, but and proper solution is needed. The latter ones can be solved by linking in libbacktrace().
Re: Supporting musl libc
On Tuesday, 13 June 2017 at 10:10:38 UTC, Joseph Rushton Wakeling wrote: On Tuesday, 13 June 2017 at 09:42:43 UTC, Joakim wrote: Also, it's possible others are still adding Glibc stuff as just linux, as that's happened a couple times since then, but I don't follow it and tell them to change it, as it usually doesn't affect me on Android. If these are affecting you, submit a pull for druntime and ping me, I'll review it. That sounds like something that automated testing ought to be able to catch ... ? Depends how deep you want it go, what did you have in mind?
Re: Supporting musl libc
On Tuesday, 13 June 2017 at 09:42:43 UTC, Joakim wrote: Also, it's possible others are still adding Glibc stuff as just linux, as that's happened a couple times since then, but I don't follow it and tell them to change it, as it usually doesn't affect me on Android. If these are affecting you, submit a pull for druntime and ping me, I'll review it. That sounds like something that automated testing ought to be able to catch ... ?
Re: Supporting musl libc
On Monday, 10 October 2016 at 15:19:00 UTC, Daniel Kozak wrote: Dne 10.10.2016 v 15:27 openwrt via Digitalmars-d napsal(a): On Sunday, 9 October 2016 at 15:48:49 UTC, Daniel Kozak wrote: On Sunday, 9 October 2016 at 13:38:33 UTC, Jacob Carlborg wrote: On 2016-10-08 20:47, Daniel Kozak wrote: What is the current status? Without support of musl-libc, I can not ad support for a Alpine linux distribution. It is a shame because they already have go and rust support. I've not worked at this at all. For my use case it was easier to just build in Docker instead. I solved this by using libexecinfo library from freebsd openwrt also use musl, and I can not run my d code on it(unless I rebuild everythings with glic). Yes, even libexecinfo does not fixed all issues, we really need to add other C runtimes. In D runtime and phobos there is a lot of places which wrongly use version(linux) instead of CRuntime_Glibc. Sorry, didn't see your comment till this thread was just bumped. I was the one who separated out linux and Glibc in druntime, I noted then that some blocks were still mixed: https://github.com/dlang/druntime/pull/1010#issuecomment-69815792 Also, it's possible others are still adding Glibc stuff as just linux, as that's happened a couple times since then, but I don't follow it and tell them to change it, as it usually doesn't affect me on Android. If these are affecting you, submit a pull for druntime and ping me, I'll review it.
Re: Supporting musl libc
On Tuesday, 17 May 2016 at 08:51:01 UTC, Jacob Carlborg wrote: As an alternative to glibc there's a C standard library called musl [1]. [...] The issue is that musl doesn't support the functions defined by execinfo.h: backtrace, backtrace_symbols_fd and backtrace_symbols, since these are glibc extensions. As far as I can see, these functions are used in two places in druntime: src/rt/backtrace/dwarf.d [3] and src/core/runtime.d [4]. The imports of execinfo is guarded by version(CRuntime_Glibc). I see that CRuntime_Glibc is a predefined version identifier defined by the compiler on Linux. [...] I just ran into these problems while trying to get D running on a hardened Gentoo system using musl libc. Using a patched gdc-4.9.4 I've been able to compile and run a simple D "Hello world". That made me very happy! Here are the binaries: http://www.chadjoan.com/d/gdc-4.9.4-hardened-musl.tar.gz I have forked the dlang overlay in the hope that it can provide the necessary information if someone tries to compile the thing for themselves: https://github.com/chadjoan/dlang This is pretty recent work, so I haven't had time to use it for anything besides "Hello world". I'm also not sure how to run regression tests for this thing, if there even are any. So this is all very untested. I'm sharing a minimally useful milestone, and hopefully it is more than that. Background: Earlier, I modified the "dmd-2.067.1-r2" ebuild from the dlang overlay and actually managed to get it to compile and emit executables. The BSD execinfo did not even manage to run its own test program on my system without segfaulting. So I used libbacktrace for backtraces and had to do a bunch of other patching to get dmd/druntime/phobos to build on this system; it involved a lot of trial-and-error. Although it emitted executables, these executables would segfault or overstep security bounds enforced by the kernel (to the point where removing all PAX restrictions would not help it). I remember something about text relocations, but at this point it's been a while, so my memory of details is lacking. Diving into codegen is far too time-consuming in my situation, so that was a dead-end. I started over with gdc, because I knew that gcc could already generate working executables on the system. Most of the pain in this process involved two things: (1) coercing portage into building gdc in its own directory so that it didn't take over my system compiler and (2) fixing gcc/gdc build scripts. It was important to use portage and not just manually configure gdc, because I wanted to ensure that the resulting gdc build would be configured and patched in a way closely matching what my system compiler does (and thus having higher success chances). Trying to configure and patch it by hand, thus repeating a lot of hard work already done by Gentoo maintainers, is... not practical. As for the build scripts: I have no idea what conventions gcc maintainers use for these things, and thus how I might do basic things like use an environment variable from outside of the build process to affect what flags are passed to a compiler. There be hacks in my patching! As an example, it will unconditionally compile the D frontend with -fPIC; if you end up building this on another system and you didn't want that, then it's just too bad ;) So it took more trial-and-error, and maybe a little compromise, but eventually it got there. I hope this helps someone.
Re: Supporting musl libc
Dne 10.10.2016 v 15:27 openwrt via Digitalmars-d napsal(a): On Sunday, 9 October 2016 at 15:48:49 UTC, Daniel Kozak wrote: On Sunday, 9 October 2016 at 13:38:33 UTC, Jacob Carlborg wrote: On 2016-10-08 20:47, Daniel Kozak wrote: What is the current status? Without support of musl-libc, I can not ad support for a Alpine linux distribution. It is a shame because they already have go and rust support. I've not worked at this at all. For my use case it was easier to just build in Docker instead. I solved this by using libexecinfo library from freebsd openwrt also use musl, and I can not run my d code on it(unless I rebuild everythings with glic). Yes, even libexecinfo does not fixed all issues, we really need to add other C runtimes. In D runtime and phobos there is a lot of places which wrongly use version(linux) instead of CRuntime_Glibc.
Re: Supporting musl libc
On Sunday, 9 October 2016 at 15:48:49 UTC, Daniel Kozak wrote: On Sunday, 9 October 2016 at 13:38:33 UTC, Jacob Carlborg wrote: On 2016-10-08 20:47, Daniel Kozak wrote: What is the current status? Without support of musl-libc, I can not ad support for a Alpine linux distribution. It is a shame because they already have go and rust support. I've not worked at this at all. For my use case it was easier to just build in Docker instead. I solved this by using libexecinfo library from freebsd openwrt also use musl, and I can not run my d code on it(unless I rebuild everythings with glic).
Re: Supporting musl libc
On Sunday, 9 October 2016 at 13:38:33 UTC, Jacob Carlborg wrote: On 2016-10-08 20:47, Daniel Kozak wrote: What is the current status? Without support of musl-libc, I can not ad support for a Alpine linux distribution. It is a shame because they already have go and rust support. I've not worked at this at all. For my use case it was easier to just build in Docker instead. I solved this by using libexecinfo library from freebsd
Re: Supporting musl libc
On 2016-10-08 20:47, Daniel Kozak wrote: What is the current status? Without support of musl-libc, I can not ad support for a Alpine linux distribution. It is a shame because they already have go and rust support. I've not worked at this at all. For my use case it was easier to just build in Docker instead. -- /Jacob Carlborg
Re: Supporting musl libc
On Tuesday, 17 May 2016 at 08:51:01 UTC, Jacob Carlborg wrote: As an alternative to glibc there's a C standard library called musl [1]. This is the C standard library used by ELLCC [2], a cross-compiler based on Clang. This cross-compiler makes it very easy to target other platforms and can be used as the C compiler when building with LDC. The issue is that musl doesn't support the functions defined by execinfo.h: backtrace, backtrace_symbols_fd and backtrace_symbols, since these are glibc extensions. As far as I can see, these functions are used in two places in druntime: src/rt/backtrace/dwarf.d [3] and src/core/runtime.d [4]. The imports of execinfo is guarded by version(CRuntime_Glibc). I see that CRuntime_Glibc is a predefined version identifier defined by the compiler on Linux. I'm not sure how to best handle different C standard libraries when it comes to choosing which one to use. Is it best to choose that when building the compiler or when building druntime? Or can it be a runtime option? [1] https://www.musl-libc.org [2] http://ellcc.org [3] https://github.com/dlang/druntime/blob/master/src/rt/backtrace/dwarf.d#L41 [4] https://github.com/dlang/druntime/blob/master/src/core/runtime.d#L433-L434 What is the current status? Without support of musl-libc, I can not ad support for a Alpine linux distribution. It is a shame because they already have go and rust support.
Re: Supporting musl libc
On Tuesday, 17 May 2016 at 11:30:33 UTC, Jacob Carlborg wrote: On 2016-05-17 11:27, Joakim wrote: [...] It depends. One approach would be to add an option to the druntime makefile, which would add -version CRuntime_Musl when compiling. In that case version(CRuntime_Musl) needs to come before version(CRuntime_Glibc) when declaring the imports. It doesn't sound like a good idea, to depend on the order of the version statements, but it's still a possibility. I wouldn't take the existence of CRuntime_Glibc as etched in stone, but whatever happens, they will not both be defined at the same time, so I don't think you should worry about the order. You can modify dmd to replace Glibc with Musl for now, and worry about how it'll be defined officially later.
Re: Supporting musl libc
On 2016-05-17 15:42, Wyatt wrote: It's worth noting that musl does support a lot of GNU-isms, but the debug stuff is probably not on the table. Here's some background on that from Rich Felker himself: http://www.openwall.com/lists/musl/2015/04/09/3 Yeah, I saw that post. -- /Jacob Carlborg
Re: Supporting musl libc
On Tuesday, 17 May 2016 at 08:51:01 UTC, Jacob Carlborg wrote: The issue is that musl doesn't support the functions defined by execinfo.h: backtrace, backtrace_symbols_fd and backtrace_symbols, since these are glibc extensions. It's worth noting that musl does support a lot of GNU-isms, but the debug stuff is probably not on the table. Here's some background on that from Rich Felker himself: http://www.openwall.com/lists/musl/2015/04/09/3 -Wyatt
Re: Supporting musl libc
On 2016-05-17 11:27, Joakim wrote: It is a runtime option on Windows, when choosing between the Digital Mars C runtime and the MSVC runtime, which is why those are also predefined version identifiers. The one for Glibc was added as an afterthought in that Windows PR, and Bionic was also reserved last fall, though my PR to actually define it has been in limbo for a year. That's mostly my fault: I never got back to it when I focused on ldc and Android/ARM. There was a lot of debate among the core team when even the Windows CRuntime support was finalized, with Walter wanting to minimize such predefined versions and some of the core team arguing against it even after it was merged. Even the iOS predefined version took 5 months to get in, because there was debate whether Darwin should be allowed as a common version for all Apple platforms. The reason I'm mentioning all this is that adding predefined versions is a matter of much debate, but that's how you'll likely implement it as a runtime option. There is no good answer on this right now: I suggest you just implement it and worry about run-time versus compile-time when you're done. Since you'll be doing the same work either way, ie adding or separating out a bunch of version blocks for musl in druntime, do it any way you like for now and you can worry about merging that last bit about compile-time vs runtime at the end. It depends. One approach would be to add an option to the druntime makefile, which would add -version CRuntime_Musl when compiling. In that case version(CRuntime_Musl) needs to come before version(CRuntime_Glibc) when declaring the imports. It doesn't sound like a good idea, to depend on the order of the version statements, but it's still a possibility. -- /Jacob Carlborg
Re: Supporting musl libc
On 2016-05-17 11:32, Joakim wrote: Jacob, when you said "runtime option," you meant when running the compiler, right? Exactly. That's how I took it: the C library is chosen either when building the compiler or running the compiler. On Windows, it's when running the compiler. We introduced CRuntime_Glibc specifically to separate libc from kernel differences, e.g. to support android. Have a look at how -mscoff32 and Android are implememted in dmd. Android is not in dmd yet, as I mentioned, you can look at my small PR though: https://github.com/dlang/dmd/pull/3643 Looks like a "runtime" option. It's not hard to implement, just a question of which approach we would like to use. -- /Jacob Carlborg
Re: Supporting musl libc
On 2016-05-17 11:26, Martin Nowak wrote: Shouldn't be runtime detected, but a linktime/compile option. We introduced CRuntime_Glibc specifically to separate libc from kernel differences, e.g. to support android. Have a look at how -mscoff32 and Android are implememted in dmd. At best defining CRuntime_Glibc should move into dmd.conf, where you could also select a different phobos library to link. In this case when I said "runtime", I meant runtime for the compiler. -mscoff32 is a different object format that just happens to use a different C runtime as well. But I guess we could add a new flag, -cruntime=musl or something. -- /Jacob Carlborg
Re: Supporting musl libc
On Tuesday, 17 May 2016 at 08:51:01 UTC, Jacob Carlborg wrote: As an alternative to glibc there's a C standard library called musl [1]. This is the C standard library used by ELLCC [2], a cross-compiler based on Clang. This cross-compiler makes it very easy to target other platforms and can be used as the C compiler when building with LDC. The issue is that musl doesn't support the functions defined by execinfo.h: backtrace, backtrace_symbols_fd and backtrace_symbols, since these are glibc extensions. As far as I can see, these functions are used in two places in druntime: src/rt/backtrace/dwarf.d [3] and src/core/runtime.d [4]. The imports of execinfo is guarded by version(CRuntime_Glibc). I see that CRuntime_Glibc is a predefined version identifier defined by the compiler on Linux. I'm not sure how to best handle different C standard libraries when it comes to choosing which one to use. Is it best to choose that when building the compiler or when building druntime? Or can it be a runtime option? [1] https://www.musl-libc.org [2] http://ellcc.org [3] https://github.com/dlang/druntime/blob/master/src/rt/backtrace/dwarf.d#L41 [4] https://github.com/dlang/druntime/blob/master/src/core/runtime.d#L433-L434 It is a runtime option on Windows, when choosing between the Digital Mars C runtime and the MSVC runtime, which is why those are also predefined version identifiers. The one for Glibc was added as an afterthought in that Windows PR, and Bionic was also reserved last fall, though my PR to actually define it has been in limbo for a year. That's mostly my fault: I never got back to it when I focused on ldc and Android/ARM. There was a lot of debate among the core team when even the Windows CRuntime support was finalized, with Walter wanting to minimize such predefined versions and some of the core team arguing against it even after it was merged. Even the iOS predefined version took 5 months to get in, because there was debate whether Darwin should be allowed as a common version for all Apple platforms. The reason I'm mentioning all this is that adding predefined versions is a matter of much debate, but that's how you'll likely implement it as a runtime option. There is no good answer on this right now: I suggest you just implement it and worry about run-time versus compile-time when you're done. Since you'll be doing the same work either way, ie adding or separating out a bunch of version blocks for musl in druntime, do it any way you like for now and you can worry about merging that last bit about compile-time vs runtime at the end.
Re: Supporting musl libc
On Tuesday, 17 May 2016 at 08:51:01 UTC, Jacob Carlborg wrote: I'm not sure how to best handle different C standard libraries when it comes to choosing which one to use. Is it best to choose that when building the compiler or when building druntime? Or can it be a runtime option? Shouldn't be runtime detected, but a linktime/compile option. We introduced CRuntime_Glibc specifically to separate libc from kernel differences, e.g. to support android. Have a look at how -mscoff32 and Android are implememted in dmd. At best defining CRuntime_Glibc should move into dmd.conf, where you could also select a different phobos library to link.
Supporting musl libc
As an alternative to glibc there's a C standard library called musl [1]. This is the C standard library used by ELLCC [2], a cross-compiler based on Clang. This cross-compiler makes it very easy to target other platforms and can be used as the C compiler when building with LDC. The issue is that musl doesn't support the functions defined by execinfo.h: backtrace, backtrace_symbols_fd and backtrace_symbols, since these are glibc extensions. As far as I can see, these functions are used in two places in druntime: src/rt/backtrace/dwarf.d [3] and src/core/runtime.d [4]. The imports of execinfo is guarded by version(CRuntime_Glibc). I see that CRuntime_Glibc is a predefined version identifier defined by the compiler on Linux. I'm not sure how to best handle different C standard libraries when it comes to choosing which one to use. Is it best to choose that when building the compiler or when building druntime? Or can it be a runtime option? [1] https://www.musl-libc.org [2] http://ellcc.org [3] https://github.com/dlang/druntime/blob/master/src/rt/backtrace/dwarf.d#L41 [4] https://github.com/dlang/druntime/blob/master/src/core/runtime.d#L433-L434 -- /Jacob Carlborg
