libc dependency
Hi all! Is it correct that D produces executables that depend on libc? While this approach avoids a certain amount of maintenance cost doesn't it limit D's possibility to outdo C on the lowest level? Honey
Re: libc dependency
On Monday, 19 June 2017 at 20:20:23 UTC, Honey wrote: Is it correct that D produces executables that depend on libc? Usually yes, since that makes sense for most programs on operating systems, but you can pull it out if you want to for a specialized task. (Really, same boat C is in.)
Re: libc dependency
On Monday, 19 June 2017 at 20:26:43 UTC, Adam D. Ruppe wrote: On Monday, 19 June 2017 at 20:20:23 UTC, Honey wrote: Is it correct that D produces executables that depend on libc? Usually yes, since that makes sense for most programs on operating systems, but you can pull it out if you want to for a specialized task. (Really, same boat C is in.) Thanks, Adam! To elaborate on my second point: if one compares, say, FreeBSD's libc [1] with glibc [2] it is pretty obvious that not all implementations are optimized to the same degree. Does it make sense - at least in cerain cases - to provide and use a D implementation that is fast on all platforms, instead? [1] https://svnweb.freebsd.org/base/head/lib/libc/ [2] https://sourceware.org/git/?p=glibc.git;a=tree;hb=HEAD
Re: libc dependency
On 6/19/17 5:01 PM, Honey wrote: On Monday, 19 June 2017 at 20:26:43 UTC, Adam D. Ruppe wrote: On Monday, 19 June 2017 at 20:20:23 UTC, Honey wrote: Is it correct that D produces executables that depend on libc? Usually yes, since that makes sense for most programs on operating systems, but you can pull it out if you want to for a specialized task. (Really, same boat C is in.) Thanks, Adam! To elaborate on my second point: if one compares, say, FreeBSD's libc [1] with glibc [2] it is pretty obvious that not all implementations are optimized to the same degree. Does it make sense - at least in cerain cases - to provide and use a D implementation that is fast on all platforms, instead? It does, but it depends on what you want to replace. What specifically are you looking for? IIRC, Tango did not depend on libc at all. It only used system calls. So it certainly is possible. -Steve
Re: libc dependency
On Monday, 19 June 2017 at 21:35:56 UTC, Steven Schveighoffer wrote: IIRC, Tango did not depend on libc at all. It only used system calls. So it certainly is possible. How did they invoke those system calls? They are usually access via libc on POSIX systems, so you don't have to implement accessing e.g. vdso on Linux yourself.
Re: libc dependency
On Monday, 19 June 2017 at 21:35:56 UTC, Steven Schveighoffer wrote: It does, but it depends on what you want to replace. What specifically are you looking for? I might need a fast variant of memcmp. Writing it in D seems to be the natural choice. I see no reason why it should be slower than other well optimized C implementations. It will probably be faster than implementations in not so much tuned standard C libraries. Actually replacing memcmp does not seem worth the effort. However, I was wondering whether D could and should be used to replace more C 'legacy' - for fun and profit. ;-) IIRC, Tango did not depend on libc at all. It only used system calls. So it certainly is possible. I heard that Go's standard library is based on a similar approach. Not depending on libc at all might serve as a marketing instrument as well.
Re: libc dependency
On 6/19/17 5:45 PM, Moritz Maxeiner wrote: On Monday, 19 June 2017 at 21:35:56 UTC, Steven Schveighoffer wrote: IIRC, Tango did not depend on libc at all. It only used system calls. So it certainly is possible. How did they invoke those system calls? They are usually access via libc on POSIX systems, so you don't have to implement accessing e.g. vdso on Linux yourself. I may have misspoke. I mean they didn't depend on the library itself. I think they do depend on the C wrappers. So for instance, they didn't use FILE *, but instead used read/write/recv/send. I don't know what's involved in creating those wrappers, but I can't imagine it's difficult to do with D (it's probably actually easier than in C/assembly). -Steve
Re: libc dependency
On Mon, Jun 19, 2017 at 07:29:46PM -0400, Steven Schveighoffer via Digitalmars-d-learn wrote: > On 6/19/17 5:45 PM, Moritz Maxeiner wrote: > > On Monday, 19 June 2017 at 21:35:56 UTC, Steven Schveighoffer wrote: > > > IIRC, Tango did not depend on libc at all. It only used system > > > calls. So it certainly is possible. > > > > How did they invoke those system calls? They are usually access via > > libc on POSIX systems, so you don't have to implement accessing e.g. > > vdso on Linux yourself. > > I may have misspoke. I mean they didn't depend on the library itself. > I think they do depend on the C wrappers. > > So for instance, they didn't use FILE *, but instead used > read/write/recv/send. > > I don't know what's involved in creating those wrappers, but I can't > imagine it's difficult to do with D (it's probably actually easier > than in C/assembly). [...] The downside of that is that then we can no longer leverage the work done by libc authors to interface with that particular OS / platform. Meaning we have to maintain this low-level code ourselves, which includes potentially keeping up with OS-level changes that would otherwise be transparent had we used the libc wrappers. Sadly, we're not yet in the day when libd authors would do this for us with every new OS that comes out. :-D T -- Talk is cheap. Whining is actually free. -- Lars Wirzenius
Re: libc dependency
On Monday, 19 June 2017 at 23:29:46 UTC, Steven Schveighoffer wrote: I don't know what's involved in creating those wrappers, but I can't imagine it's difficult to do with D (it's probably actually easier than in C/assembly). Not difficult, but a tedious amount of work for no inherent benefit: You would have to recreate the correct system call calling semantics for every operating system / hardware combination that D is supposed to run on. I suggest reading [1] and [2] to get a feeling of the workload *just* for Linux. My unsolicited opinion: Unless someone pays you (for whatever reason) to explicitly not use a libc, just use a well tested libc for system calls and be done with it. [1] http://man7.org/linux/man-pages/man2/syscalls.2.html [2] http://man7.org/linux/man-pages/man7/vdso.7.html
Re: libc dependency
On Monday, 19 June 2017 at 21:01:35 UTC, Honey wrote: On Monday, 19 June 2017 at 20:26:43 UTC, Adam D. Ruppe wrote: On Monday, 19 June 2017 at 20:20:23 UTC, Honey wrote: Is it correct that D produces executables that depend on libc? Usually yes, since that makes sense for most programs on operating systems, but you can pull it out if you want to for a specialized task. (Really, same boat C is in.) Thanks, Adam! To elaborate on my second point: if one compares, say, FreeBSD's libc [1] with glibc [2] it is pretty obvious that not all implementations are optimized to the same degree. Does it make sense - at least in cerain cases - to provide and use a D implementation that is fast on all platforms, instead? [1] https://svnweb.freebsd.org/base/head/lib/libc/ [2] https://sourceware.org/git/?p=glibc.git;a=tree;hb=HEAD The issue with that is that you assume: 1) that a complete libc replacement can be created from scratch 2) that a correct libc replacement can be created from scratch 3) that a better libc replacement can be created from scratch 4) that this replacement can be maintained for years to come on all plateforms All those points are certainly theoretically possible, but getting all 4 at once isn't a small task. Point 2 and 3 in particular: whatever your libc is it's been tested, reflected upon and optimized for years by good programmers. There is absolutely no reason to throw away this work. Even if it was practical to do that and that the developper was confident enough that he can do better than libc to start working on it, point 4 would be out of reach given the current state of affairs. So, while replacing specific C functions by D implementations where the gain is clear seems like a good idea to me, dropping the dependency on libc seems more like a risky gamble than anything.
Re: libc dependency
On Tuesday, 20 June 2017 at 05:19:21 UTC, Cym13 wrote: The issue with that is that you assume: 1) that a complete libc replacement can be created from scratch 2) that a correct libc replacement can be created from scratch 3) that a better libc replacement can be created from scratch 4) that this replacement can be maintained for years to come on all plateforms For things like core.stdc it would be insane not to depend on libc. In fact, it would be strange if we did not call into libc, here. But does this really mean that depending on libc must be the default for everyone? I doubt that we would have to replace every function provided by libc. We only have to provide alternatives for what we use. Other than that I share your scepsis. I am not saying that we should actually drop the dependency. I was just curious to learn something.
Re: libc dependency
On Mon, 2017-06-19 at 22:41 +, Honey via Digitalmars-d-learn wrote: > […] > I heard that Go's standard library is based on a similar approach. > > Not depending on libc at all might serve as a marketing > instrument as well. But there is lots of paid resource in the core Go community which makes not using "middleware" feasible by providing your own. Also of course the Go/C interface is not as clean as is the case in D, so the need for Go-specific middleware is much, much higher. As D can call C linkage libraries, and libc is a library for interfacing to OSes, use it, get the abstraction, pay the (small) price, get someone else to do the maintenance. -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder signature.asc Description: This is a digitally signed message part
Re: libc dependency
On Monday, 19 June 2017 at 21:45:56 UTC, Moritz Maxeiner wrote: On Monday, 19 June 2017 at 21:35:56 UTC, Steven Schveighoffer wrote: IIRC, Tango did not depend on libc at all. It only used system calls. So it certainly is possible. How did they invoke those system calls? They are usually access via libc on POSIX systems, so you don't have to implement accessing e.g. vdso on Linux yourself. Tango just didn't use C abstractions (`FILE*`, say), but rather it would call functions such `read(2)`, `write(2)`, `seek(2)`, etc. and implement buffering/seeking/etc manually. So, the library just declared `extern (C) read (...);`/`extern (C) write`.. and expected for the right library (which doesn't have to be libc, just the one that exposes these syscall wrappers) linked.
Re: libc dependency
On Tuesday, 20 June 2017 at 11:00:00 UTC, Nemanja Boric wrote: On Monday, 19 June 2017 at 21:45:56 UTC, Moritz Maxeiner wrote: On Monday, 19 June 2017 at 21:35:56 UTC, Steven Schveighoffer wrote: IIRC, Tango did not depend on libc at all. It only used system calls. So it certainly is possible. How did they invoke those system calls? They are usually access via libc on POSIX systems, so you don't have to implement accessing e.g. vdso on Linux yourself. Tango just didn't use C abstractions (`FILE*`, say), but rather it would call functions such `read(2)`, `write(2)`, `seek(2)`, etc. and implement buffering/seeking/etc manually. Only the second one of those are the system calls I was talking about :p So, the library just declared `extern (C) read (...);`/`extern (C) write`.. and expected for the right library (which doesn't have to be libc, just the one that exposes these syscall wrappers) linked. If you have a library that exposes to you via C API those functions that you need of libc, then that library *is* your (albeit partial) libc (since a libc is defined as any library that exposes such functions). In any case, that's the same way that druntime depends on libc AFAIK.
Re: libc dependency
On Tuesday, 20 June 2017 at 11:08:15 UTC, Moritz Maxeiner wrote: On Tuesday, 20 June 2017 at 11:00:00 UTC, Nemanja Boric wrote: On Monday, 19 June 2017 at 21:45:56 UTC, Moritz Maxeiner wrote: On Monday, 19 June 2017 at 21:35:56 UTC, Steven Schveighoffer wrote: IIRC, Tango did not depend on libc at all. It only used system calls. So it certainly is possible. How did they invoke those system calls? They are usually access via libc on POSIX systems, so you don't have to implement accessing e.g. vdso on Linux yourself. Tango just didn't use C abstractions (`FILE*`, say), but rather it would call functions such `read(2)`, `write(2)`, `seek(2)`, etc. and implement buffering/seeking/etc manually. Only the second one of those are the system calls I was talking about :p Yeah, I saw the Steve's answer late, but there's no editing posts here :(. :-) So, the library just declared `extern (C) read (...);`/`extern (C) write`.. and expected for the right library (which doesn't have to be libc, just the one that exposes these syscall wrappers) linked. If you have a library that exposes to you via C API those functions that you need of libc, then that library *is* your (albeit partial) libc (since a libc is defined as any library that exposes such functions). In any case, that's the same way that druntime depends on libc AFAIK. Yeah, that stands, but I thought the point of this topic was that (IIRC, not a phobos experienced user) Phobos (_not_ druntime) uses C _standard library_ abstractions for many things, and that's why the OP said we can't beat C in speed. I don't think it's reasonable thing to think that writing syscall wrappers would help D gain to much speed :-).
Re: libc dependency
On Tuesday, 20 June 2017 at 11:18:17 UTC, Nemanja Boric wrote: Phobos (_not_ druntime) uses C _standard library_ abstractions for many things I just took a quick look and it seems that I'm remembering this wrong.
Re: libc dependency
On 2017-06-20 01:29, Steven Schveighoffer wrote: I may have misspoke. I mean they didn't depend on the library itself. I think they do depend on the C wrappers. So for instance, they didn't use FILE *, but instead used read/write/recv/send. They did use the Posix and Windows API functions. I can see some imports for libc and Posix functions spread out here and there [1], but I'm not too familiar with are system calls and which are not. It's certainly no way near how much is used in Phobos today. [1] http://dsource.org/projects/tango/browser/trunk/tango/sys/Process.d#L17 -- /Jacob Carlborg
Re: libc dependency
On 2017-06-20 09:48, Russel Winder via Digitalmars-d-learn wrote: But there is lots of paid resource in the core Go community which makes not using "middleware" feasible by providing your own. Also of course the Go/C interface is not as clean as is the case in D, so the need for Go-specific middleware is much, much higher. As D can call C linkage libraries, and libc is a library for interfacing to OSes, use it, get the abstraction, pay the (small) price, get someone else to do the maintenance. Yes. But it would be nice to not be dependent on glibc. If we could use musl it would be a lot easier to create our own tool chain (and get full support for static liking). Avoiding the need to download the C tool chain to be able to use D. I've heard that many times, especially on macOS: "oh, you need to download Xcode to use D?". -- /Jacob Carlborg
Re: libc dependency
On Tuesday, 20 June 2017 at 11:26:44 UTC, Jacob Carlborg wrote: Yes. But it would be nice to not be dependent on glibc. If we could use musl it would be a lot easier to create our own tool chain (and get full support for static liking). Yes, please. Avoiding the need to download the C tool chain to be able to use D. I've heard that many times, especially on macOS: "oh, you need to download Xcode to use D?". Last time I checked you only needed the Xcode command line tools (which are small), not the whole thing. In any case, what would you propose to be bundled as the linker (if we can't depend on the one provided by Xcode)?
Re: libc dependency
On 2017-06-20 13:51, Moritz Maxeiner wrote: Last time I checked you only needed the Xcode command line tools (which are small), not the whole thing. Yes. But I think there are a few things missing, depending on what you need. There's some LLDB library that is missing from the command line tools that is used for editor/IDE integration. In any case, what would you propose to be bundled as the linker (if we can't depend on the one provided by Xcode)? LLD, the LLVM linker [1]. As far as I understand it, it also support cross-platform linking. Using LDC, musl and LLD you have a fully working cross-compiler tool chain. You might need some extra libraries as well, depending on what you need. [1] https://lld.llvm.org -- /Jacob Carlborg
Re: libc dependency
On 6/20/17 7:23 AM, Jacob Carlborg wrote: On 2017-06-20 01:29, Steven Schveighoffer wrote: I may have misspoke. I mean they didn't depend on the library itself. I think they do depend on the C wrappers. So for instance, they didn't use FILE *, but instead used read/write/recv/send. They did use the Posix and Windows API functions. I can see some imports for libc and Posix functions spread out here and there [1], but I'm not too familiar with are system calls and which are not. It's certainly no way near how much is used in Phobos today. I remember they specifically did not want to depend on anything in the C library. In windows, they definitely did not depend on C at all, and did not link with msvcrt or any other C runtime. The calls were all to the ntdll.dll. Really this is the way it *should* be in Posix as well, but they just stuff everything into libc. As it stands, Phobos uses C runtime features extensively, and it would be hard to get rid of the calls. -Steve
Re: libc dependency
On 6/19/17 7:47 PM, H. S. Teoh via Digitalmars-d-learn wrote: On Mon, Jun 19, 2017 at 07:29:46PM -0400, Steven Schveighoffer via Digitalmars-d-learn wrote: On 6/19/17 5:45 PM, Moritz Maxeiner wrote: On Monday, 19 June 2017 at 21:35:56 UTC, Steven Schveighoffer wrote: IIRC, Tango did not depend on libc at all. It only used system calls. So it certainly is possible. How did they invoke those system calls? They are usually access via libc on POSIX systems, so you don't have to implement accessing e.g. vdso on Linux yourself. I may have misspoke. I mean they didn't depend on the library itself. I think they do depend on the C wrappers. So for instance, they didn't use FILE *, but instead used read/write/recv/send. I don't know what's involved in creating those wrappers, but I can't imagine it's difficult to do with D (it's probably actually easier than in C/assembly). [...] The downside of that is that then we can no longer leverage the work done by libc authors to interface with that particular OS / platform. Meaning we have to maintain this low-level code ourselves, which includes potentially keeping up with OS-level changes that would otherwise be transparent had we used the libc wrappers. Sadly, we're not yet in the day when libd authors would do this for us with every new OS that comes out. :-D Of course! It would not be a small task. But D has some rather nice generational features. One of my favorite Dconf talks of all time is still Mike Franklin's talk from 2014 (http://dconf.org/2014/talks/franklin.html), which has a part in it where he wrote D code to parse the pdf spec of a microcontroller and AUTOMATICALLY GENERATE the correct register mappings. Think about that -- 0 effort to maintain hundreds or thousands of registers, with no mistakes! I'm frankly disappointed that D did not service Mike enough to continue using D, as he seemed a very bright and motivated user. I think we could do something similar in D for the OS calls. What is the point, you might ask? The only true point is to have control over every part of the userspace call chains. You don't need a FILE * for stdout? Fine, just use this small wrapper. Who knows what magic can be performed if the compiler has full access to the system calls for inlining and optimization. I'm not saying I want to do this or that it's really important. But there are possible reasons to do it, even if it's just for the coolness factor :) -Steve
Re: libc dependency
On Tuesday, 20 June 2017 at 12:09:06 UTC, Jacob Carlborg wrote: LLD, the LLVM linker [1]. As far as I understand it, it also support cross-platform linking. Using LDC, musl and LLD you have a fully working cross-compiler tool chain. You might need some extra libraries as well, depending on what you need. Hm, so then we could provide a self-contained installer for all major platforms that bundles all you need to get started with D. Neat.
Re: libc dependency
On Tuesday, 20 June 2017 at 18:51:17 UTC, Moritz Maxeiner wrote: On Tuesday, 20 June 2017 at 12:09:06 UTC, Jacob Carlborg wrote: LLD, the LLVM linker [1]. As far as I understand it, it also support cross-platform linking. Using LDC, musl and LLD you have a fully working cross-compiler tool chain. You might need some extra libraries as well, depending on what you need. Hm, so then we could provide a self-contained installer for all major platforms that bundles all you need to get started with D. Neat. Yes, that's part of the idea behind the ongoing work to integrate LLD into LDC: https://github.com/ldc-developers/ldc/pull/2142 For Windows, we use the MS C runtime, though, and the legal situation around redistribution seems a bit unclear. — David
Re: libc dependency
On 2017-06-20 14:10, Steven Schveighoffer wrote: I remember they specifically did not want to depend on anything in the C library. I can only tell you what I read from the source code :) . I didn't know enough about these things back in the D1 and Tango days. -- /Jacob Carlborg
Re: libc dependency
On 2017-06-20 21:59, David Nadlinger wrote: For Windows, we use the MS C runtime, though, and the legal situation around redistribution seems a bit unclear. Musl (or similar) should be available as an alternative. That will make it easier to cross-compile as well. But I guess MS C runtime is required for linking with other existing C libraries. -- /Jacob Carlborg
Re: libc dependency
On Wednesday, 21 June 2017 at 06:58:43 UTC, Jacob Carlborg wrote: Musl (or similar) should be available as an alternative. That will make it easier to cross-compile as well. This is not relevant for cross-compilation, as long as you have the libraries available. You can actually link a D Windows x64/MSVCRT executable from Linux today if you copy over the necessary libraries. The question is just how we can make this as easy as possible for users. — David
Re: libc dependency
On 2017-06-21 17:51, David Nadlinger wrote: This is not relevant for cross-compilation, as long as you have the libraries available. You can actually link a D Windows x64/MSVCRT executable from Linux today if you copy over the necessary libraries. The question is just how we can make this as easy as possible for users. Perhaps not when targeting Windows but when targeting Linux. Can you target a specific version of libc or do you need to use static linking? -- /Jacob Carlborg
