Re: [rust-dev] Rethinking Linking in Rust
On 11/18/2013 08:01 PM, Zack Corr wrote: On Tue, Nov 19, 2013 at 11:13 AM, Brian Anderson mailto:bander...@mozilla.com>> wrote: Of course this conflicts with the `link` attribute of crates, which I think is poorly named anyway. Perhaps #[link] in its current usage should be renamed to #[crate]? I think that would make more sense. This would be my preference. ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Rethinking Linking in Rust
On 11/18/2013 06:22 PM, Alex Crichton wrote: * #[link(...)] becomes the new method of specifying linkage semantics on extern blocks, and it may be used similarly to link_args today I'd kind of like for this to be available at the crate level too since most libraries don't use OS X two-level namespaces and it's more convient to me to just put all the linkage at the top of the crate. Of course this conflicts with the `link` attribute of crates, which I think is poorly named anyway. What purpose did you have in mind for the #[link] attribute at the top of the crate? Is the crate saying how it should be linked to other crates? It would be the same purpose as putting them on extern blocks - to tell the linker what other libraries to link to. It would function exactly the same. The only reason link attributes ever *need* to be specifically on an extern block is to support OS X two-level namespaces (which I don't know anything about and haven't actually seen). I don't really understand what 'once' implies in `link(once)` and how it relates to statics. If a static library *must* be linked, then dynamic libraries may not be linked? Why is that? If 'once' implies 'static', can we just say 'link(static)'? I assume some argument propagation is going to come into play here ... Will also need to accomodate some other common features like, e.g. `link(framework = "foo")` or something for OS X frameworks. What this ended up turning out as is #[link(name = “foo”, kind = “static”)]. I think that a “framework” kind would fit quite well for this use case. .rlib files also need the crate metadata. I agree What happens when two upstream crates link to the same native static library? In the final link step they are both going to be linked in, and I presume there's some kind of conflict? Right now they’re both linked in. I didn’t envision this as a use case for rustc to complain about, but it would be simple enough to iterate over all upstream crates and see if the same static library were linked twice. My implementation requires metadata about the linkage regardless. How does one opt into linking to dynamic libraries? Without some further mechanism everybody will be linking to the static libstd. I’ve reserved another -Z flag for ‘-Z prefer-dynamic' I took this to mean that we would just be packaging up the static libraries to save them for the final link step (since the rlib is just a .o file, not pre-linked to it's static lib dependencies). The effect of this though may be that all downstream crates implicitly have access to all the static library's symbols. This actually what currently happens. The resulting rlib file already contains all of the static native libraries bundled inside of it. This is achieved via ld’s -r option. It is true that all of the symbols leak through, however, and this is unfortunate. I’d like to explore methods of preventing this, but I only know of one way currently. We’d create a list of all symbols in the rust library (an actual file on the filesystem), and then pass that to the linker saying “here are all the exported symbols, discard everything else.” ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Rethinking Linking in Rust
On Tue, Nov 19, 2013 at 11:13 AM, Brian Anderson wrote: > Of course this conflicts with the `link` attribute of crates, which I > think is poorly named anyway. > Perhaps #[link] in its current usage should be renamed to #[crate]? I think that would make more sense. ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Rethinking Linking in Rust
>> * #[link(...)] becomes the new method of specifying linkage semantics on >> extern >> blocks, and it may be used similarly to link_args today > > I'd kind of like for this to be available at the crate level too since most > libraries don't use OS X two-level namespaces and it's more convient to me to > just put all the linkage at the top of the crate. Of course this conflicts > with the `link` attribute of crates, which I think is poorly named anyway. What purpose did you have in mind for the #[link] attribute at the top of the crate? Is the crate saying how it should be linked to other crates? > I don't really understand what 'once' implies in `link(once)` and how it > relates to statics. If a static library *must* be linked, then dynamic > libraries may not be linked? Why is that? If 'once' implies 'static', can we > just say 'link(static)'? I assume some argument propagation is going to come > into play here ... > > Will also need to accomodate some other common features like, e.g. > `link(framework = "foo")` or something for OS X frameworks. What this ended up turning out as is #[link(name = “foo”, kind = “static”)]. I think that a “framework” kind would fit quite well for this use case. > .rlib files also need the crate metadata. I agree > What happens when two upstream crates link to the same native static library? > In the final link step they are both going to be linked in, and I presume > there's some kind of conflict? Right now they’re both linked in. I didn’t envision this as a use case for rustc to complain about, but it would be simple enough to iterate over all upstream crates and see if the same static library were linked twice. My implementation requires metadata about the linkage regardless. > How does one opt into linking to dynamic libraries? Without some further > mechanism everybody will be linking to the static libstd. I’ve reserved another -Z flag for ‘-Z prefer-dynamic' > I took this to mean that we would just be packaging up the static libraries > to save them for the final link step (since the rlib is just a .o file, not > pre-linked to it's static lib dependencies). The effect of this though may be > that all downstream crates implicitly have access to all the static library's > symbols. This actually what currently happens. The resulting rlib file already contains all of the static native libraries bundled inside of it. This is achieved via ld’s -r option. It is true that all of the symbols leak through, however, and this is unfortunate. I’d like to explore methods of preventing this, but I only know of one way currently. We’d create a list of all symbols in the rust library (an actual file on the filesystem), and then pass that to the linker saying “here are all the exported symbols, discard everything else.” ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Rethinking Linking in Rust
On 11/15/2013 03:01 AM, Niko Matsakis wrote: A few quick questions and comments: On Fri, Nov 15, 2013 at 12:09:28AM -0800, Alex Crichton wrote: I've been thinking about static linking recently, along with a little bit of linking in general, and I wanted to see what others thought. # The Goal Primarily, I believe that if desired, rustc should be able to generate an executable or dynamic library with no dependence on any rust libraries. This includes things like librustrt and libextra. Rust shouldn't be striving to lift dependence on system libraries, that'll come at later times if need be. It seems like you *are* striving to lift dependencies on non-rust libraries, though. For example, you mention having libuv be statically imported into a .rlib file etc. Or did I misunderstand? I took this to mean that we would just be packaging up the static libraries to save them for the final link step (since the rlib is just a .o file, not pre-linked to it's static lib dependencies). The effect of this though may be that all downstream crates implicitly have access to all the static library's symbols. ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Rethinking Linking in Rust
On 11/15/2013 03:01 AM, Niko Matsakis wrote: A few quick questions and comments: On Fri, Nov 15, 2013 at 12:09:28AM -0800, Alex Crichton wrote: As a side node, after writing all this up, I remembered LTO as an option for generating libraries. I don't think I know enough about LTO to be able to say whether it would fit in this system or not, but my basic understanding is that an LTO library is just "IR in a box". We could add a --lto output option which has pretty much the same semantics as the --rlib option, but with a different format. Again though, I haven't thought how native libraries would fit into that scenario, but I believe that we could fairly easily accommodate LTO in a system like this. Unless I'm missing something, it seems like what we would want to do is to have the .rlib file contain LLVM IR, at least for the Rust code that was compiled / statically linked against. If we can, I think we should just make LTO happen by default whenver you statically link, rather than having it be a separate option, but it is fine if that doesn't work yet in the first version (e.g., because .rlib is just a .o file). Still I agree that your scheme accommodates it just fine. I'm not sure I agree that LTO should be the default - for large applications it's going to be brutal on LLVM and I imagine we'll quickly hit scenarios where we run out of RAM. For something like a production Servo I imagine we would go to whatever lengths necessary to LTO everything, but it would be cheaper and faster to do normal static linking. Some empirical evidence about how far we can push LLVM's LTO would be helpful though. ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Rethinking Linking in Rust
On 11/15/2013 12:09 AM, Alex Crichton wrote: I've been thinking about static linking recently, along with a little bit of linking in general, and I wanted to see what others thought. # The Goal Primarily, I believe that if desired, rustc should be able to generate an executable or dynamic library with no dependence on any rust libraries. This includes things like librustrt and libextra. Rust shouldn't be striving to lift dependence on system libraries, that'll come at later times if need be. Additionally, rustc should be able to generate libfoo.a where libfoo.a has no dependence on any rust libraries. This library can then be statically linked to another application. # Intermediate static libraries I personally know of no way to create a static library from a dynamic one, so to achieve this we would need to distribute libstd and libextra in some form that is not a shared library. This problem not only applies to libstd, but also to any rust library which wants to be statically linked. The first natural conclusion for for an intermediate format would be a .a file itself. Why not distribute libstd.a along with libstd.so. After all, a .a is only an archive which in our case would contain one .o file. In thinking about this though, I don't think that this is the best format. The main idea of providing intermediate .a files is to allow linkage to them via the normal system linker. To be usable, this would mean that all .a files rust generates would have to have their own statically linked version of libstd or otherwise everyone will have to find where libstd is guess the name and hash attached to it. This is infeasible for arbitrary libraries which could have arbitrarily many dependencies. # Native Libraries One part of linking which rust cannot forget is native libraries. Right now, native libraries are always linked against when compiling a local crate, but no native library dependencies are propagated among crates. Due to the nature of a static library and what I assume is the file format itself, a static rust library cannot link to its dependent dynamic libraries. We can, however, resolve all native static dependencies at compile time. # A Scheme for Linking With the above knowledge, I would propose the following linkage model for rust. There are four types of files that the rust compiler will generate: 1. An executable 2. A dynamic library (.so, .dylib, .dll) 3. A "rust" static library (.rlib) 4. A regular static library (.a, .lib) The "rust language" would ship with dynamic library files as well as .rlib files. There would be no .a files in the distribution. A rust static library would be a format defined by rust that is not available for or intended for external use. It is meant to be beneficial to the rust compiler and that's it. It just so happens that their first incarnation would be created similarly to `cp foo.o foo.rlib`. In addition to these changes, the linkage attributes would change to be as follows: * #[link_args] becomes gated behind a feature flag. I believe that this is still a very useful ability to pass arbitrary flags to the linker, but this is *not* a sanctioned way of doing so at all because of how platform specific it is * #[link(...)] becomes the new method of specifying linkage semantics on extern blocks, and it may be used similarly to link_args today I'd kind of like for this to be available at the crate level too since most libraries don't use OS X two-level namespaces and it's more convient to me to just put all the linkage at the top of the crate. Of course this conflicts with the `link` attribute of crates, which I think is poorly named anyway. * #[link(name = "foo")] specifies that this crate links to native library `foo` * #[link(once)] implies that the native library is a static library, hence it *must* be linked against in the current compilation, regardless of the output format Omission of `link(once)` assumes that the library is available at all destinations, and it may not be linked against in the current compilation unit. I don't really understand what 'once' implies in `link(once)` and how it relates to statics. If a static library *must* be linked, then dynamic libraries may not be linked? Why is that? If 'once' implies 'static', can we just say 'link(static)'? I assume some argument propagation is going to come into play here ... Will also need to accomodate some other common features like, e.g. `link(framework = "foo")` or something for OS X frameworks. ## The Linkage Step To see how this affects how artifacts are created, I'd like to go into detail about how each of the four output artifacts all interact with one another by describing the linkage phase of each output. For each of these, remember that the compiler's output is one .o file for each crate. Also remember that all rust libraries will always link to all upstream rust libraries. ### Linking Executables and Dynamic Librar
Re: [rust-dev] Rethinking Linking in Rust
What if the final executable also wants to link against a slightly newer
version of libfoo.a? I'm not even sure what ld would do then. Complains
about duplicate symbols? Picks one at random?
I think I'd rather have Rust object file along with a list of libraries
that will be needed for final linking. This could be a companion text file
or maybe a command line option to rustc, which dumps this info from
metadata.
And if I really do want a monolithic library file, I can always create one
with ar, can't I?
Vadim
On Fri, Nov 15, 2013 at 12:12 PM, Alex Crichton wrote:
> You would be required to specify that the native library would be static
> via
>
> #[link(name = "foo", static)]
> extern { ... }
>
> And then rustc would bundle libfoo.a with libmycomp.a. libmycomp.a
> would also include any upstream rust dependencies (libstd.a,
> libextra.a, etc.)
>
> On Fri, Nov 15, 2013 at 12:00 PM, Vadim wrote:
> > So in the case of --staticlib, if my Rust library, libmycomp.a, depended
> on
> > non-Rust local native library, libfoo.a, would Rust then bundle all
> modules
> > from libfoo into libmycomp? Or would it only do so for Rust libraries,
> > e.g. libstd.a?
>
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Rethinking Linking in Rust
You would be required to specify that the native library would be static via
#[link(name = "foo", static)]
extern { ... }
And then rustc would bundle libfoo.a with libmycomp.a. libmycomp.a
would also include any upstream rust dependencies (libstd.a,
libextra.a, etc.)
On Fri, Nov 15, 2013 at 12:00 PM, Vadim wrote:
> So in the case of --staticlib, if my Rust library, libmycomp.a, depended on
> non-Rust local native library, libfoo.a, would Rust then bundle all modules
> from libfoo into libmycomp? Or would it only do so for Rust libraries,
> e.g. libstd.a?
>
>
> On Fri, Nov 15, 2013 at 12:09 AM, Alex Crichton wrote:
>>
>> I've been thinking about static linking recently, along with a little bit
>> of
>> linking in general, and I wanted to see what others thought.
>>
>> # The Goal
>>
>> Primarily, I believe that if desired, rustc should be able to generate an
>> executable or dynamic library with no dependence on any rust libraries.
>> This
>> includes things like librustrt and libextra. Rust shouldn't be striving to
>> lift
>> dependence on system libraries, that'll come at later times if need be.
>>
>> Additionally, rustc should be able to generate libfoo.a where libfoo.a has
>> no
>> dependence on any rust libraries. This library can then be statically
>> linked to
>> another application.
>>
>> # Intermediate static libraries
>>
>> I personally know of no way to create a static library from a dynamic one,
>> so to
>> achieve this we would need to distribute libstd and libextra in some form
>> that
>> is not a shared library. This problem not only applies to libstd, but also
>> to
>> any rust library which wants to be statically linked.
>>
>> The first natural conclusion for for an intermediate format would be a .a
>> file
>> itself. Why not distribute libstd.a along with libstd.so. After all, a .a
>> is
>> only an archive which in our case would contain one .o file. In thinking
>> about
>> this though, I don't think that this is the best format. The main idea of
>> providing intermediate .a files is to allow linkage to them via the normal
>> system linker. To be usable, this would mean that all .a files rust
>> generates
>> would have to have their own statically linked version of libstd or
>> otherwise
>> everyone will have to find where libstd is guess the name and hash
>> attached to
>> it. This is infeasible for arbitrary libraries which could have
>> arbitrarily many
>> dependencies.
>>
>> # Native Libraries
>>
>> One part of linking which rust cannot forget is native libraries. Right
>> now,
>> native libraries are always linked against when compiling a local crate,
>> but no
>> native library dependencies are propagated among crates.
>>
>> Due to the nature of a static library and what I assume is the file format
>> itself, a static rust library cannot link to its dependent dynamic
>> libraries. We
>> can, however, resolve all native static dependencies at compile time.
>>
>> # A Scheme for Linking
>>
>> With the above knowledge, I would propose the following linkage model for
>> rust.
>>
>> There are four types of files that the rust compiler will generate:
>>
>> 1. An executable
>> 2. A dynamic library (.so, .dylib, .dll)
>> 3. A "rust" static library (.rlib)
>> 4. A regular static library (.a, .lib)
>>
>> The "rust language" would ship with dynamic library files as well as .rlib
>> files. There would be no .a files in the distribution.
>>
>> A rust static library would be a format defined by rust that is not
>> available
>> for or intended for external use. It is meant to be beneficial to the rust
>> compiler and that's it. It just so happens that their first incarnation
>> would be
>> created similarly to `cp foo.o foo.rlib`.
>>
>> In addition to these changes, the linkage attributes would change to be as
>> follows:
>>
>> * #[link_args] becomes gated behind a feature flag. I believe that this is
>> still
>> a very useful ability to pass arbitrary flags to the linker, but this is
>> *not*
>> a sanctioned way of doing so at all because of how platform specific it
>> is
>>
>> * #[link(...)] becomes the new method of specifying linkage semantics on
>> extern
>> blocks, and it may be used similarly to link_args today
>>
>> * #[link(name = "foo")] specifies that this crate links to native
>> library
>> `foo`
>> * #[link(once)] implies that the native library is a static library,
>> hence it
>> *must* be linked against in the current compilation, regardless of the
>> output format
>>
>> Omission of `link(once)` assumes that the library is available at all
>> destinations, and it may not be linked against in the current
>> compilation
>> unit.
>>
>> ## The Linkage Step
>>
>> To see how this affects how artifacts are created, I'd like to go into
>> detail
>> about how each of the four output artifacts all interact with one another
>> by
>> describing the linkage phase of each output. For each of these, remember
>> that
>> the compiler's output is one .o file for each c
Re: [rust-dev] Rethinking Linking in Rust
So in the case of --staticlib, if my Rust library, libmycomp.a, depended on non-Rust local native library, libfoo.a, would Rust then bundle all modules from libfoo into libmycomp? Or would it only do so for Rust libraries, e.g. libstd.a? On Fri, Nov 15, 2013 at 12:09 AM, Alex Crichton wrote: > I've been thinking about static linking recently, along with a little bit > of > linking in general, and I wanted to see what others thought. > > # The Goal > > Primarily, I believe that if desired, rustc should be able to generate an > executable or dynamic library with no dependence on any rust libraries. > This > includes things like librustrt and libextra. Rust shouldn't be striving to > lift > dependence on system libraries, that'll come at later times if need be. > > Additionally, rustc should be able to generate libfoo.a where libfoo.a has > no > dependence on any rust libraries. This library can then be statically > linked to > another application. > > # Intermediate static libraries > > I personally know of no way to create a static library from a dynamic one, > so to > achieve this we would need to distribute libstd and libextra in some form > that > is not a shared library. This problem not only applies to libstd, but also > to > any rust library which wants to be statically linked. > > The first natural conclusion for for an intermediate format would be a .a > file > itself. Why not distribute libstd.a along with libstd.so. After all, a .a > is > only an archive which in our case would contain one .o file. In thinking > about > this though, I don't think that this is the best format. The main idea of > providing intermediate .a files is to allow linkage to them via the normal > system linker. To be usable, this would mean that all .a files rust > generates > would have to have their own statically linked version of libstd or > otherwise > everyone will have to find where libstd is guess the name and hash > attached to > it. This is infeasible for arbitrary libraries which could have > arbitrarily many > dependencies. > > # Native Libraries > > One part of linking which rust cannot forget is native libraries. Right > now, > native libraries are always linked against when compiling a local crate, > but no > native library dependencies are propagated among crates. > > Due to the nature of a static library and what I assume is the file format > itself, a static rust library cannot link to its dependent dynamic > libraries. We > can, however, resolve all native static dependencies at compile time. > > # A Scheme for Linking > > With the above knowledge, I would propose the following linkage model for > rust. > > There are four types of files that the rust compiler will generate: > > 1. An executable > 2. A dynamic library (.so, .dylib, .dll) > 3. A "rust" static library (.rlib) > 4. A regular static library (.a, .lib) > > The "rust language" would ship with dynamic library files as well as .rlib > files. There would be no .a files in the distribution. > > A rust static library would be a format defined by rust that is not > available > for or intended for external use. It is meant to be beneficial to the rust > compiler and that's it. It just so happens that their first incarnation > would be > created similarly to `cp foo.o foo.rlib`. > > In addition to these changes, the linkage attributes would change to be as > follows: > > * #[link_args] becomes gated behind a feature flag. I believe that this is > still > a very useful ability to pass arbitrary flags to the linker, but this is > *not* > a sanctioned way of doing so at all because of how platform specific it > is > > * #[link(...)] becomes the new method of specifying linkage semantics on > extern > blocks, and it may be used similarly to link_args today > > * #[link(name = "foo")] specifies that this crate links to native library > `foo` > * #[link(once)] implies that the native library is a static library, > hence it > *must* be linked against in the current compilation, regardless of the > output format > > Omission of `link(once)` assumes that the library is available at all > destinations, and it may not be linked against in the current compilation > unit. > > ## The Linkage Step > > To see how this affects how artifacts are created, I'd like to go into > detail > about how each of the four output artifacts all interact with one another > by > describing the linkage phase of each output. For each of these, remember > that > the compiler's output is one .o file for each crate. Also remember that > all rust > libraries will always link to all upstream rust libraries. > > ### Linking Executables and Dynamic Libraries > > These two cases are very similar because they are creating the actual > "result > artifact" in terms of a file which will have no more linkage performed on > it. > The following components must be linked in to produce the artifact: > > * The local .o file > * All local native dependencies > * All upstre
Re: [rust-dev] Rethinking Linking in Rust
>> To this end, I mainly >> point out that rust should roll in local native static libraries, and >> just live with global native dynamic libraries. > > How does rustc know the difference? Because the "local native" libraries > are tagged as #[link(once)]? (nit: maybe link(static) would be clearer?) You're correct, this is the reason that I added the #[link(once)]. I don't want rustc to start guessing about LD_LIBRARY_PATH and weird business like that, so I'd rather that the author just be explicitly about the native library. I also agree that #[link(static)] is clearer. > Is this just a matter of changing what is hashed when we construct the > symbol name (or dropping the symbol hashes entirely)? That doesn't > seem very far. Are there are a lot of other things standing in the > way that immediately come to mind? Can we try and document what those > issues are? This is correct. This is tangentially related to https://github.com/mozilla/rust/issues/10207 along with https://github.com/mozilla/rust/issues/10208. I have many thoughts on this, although they don't quite relate to static linking, so I'll try to write them up later. The main idea is that the SVH from 10207 is in all symbol names, and the SVH is a hash of "all reachable things" which includes many things you may not initially consider (many of which you pointed out). ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Rethinking Linking in Rust
On Fri, Nov 15, 2013 at 09:28:49AM -0800, Alex Crichton wrote: > To this end, I mainly > point out that rust should roll in local native static libraries, and > just live with global native dynamic libraries. How does rustc know the difference? Because the "local native" libraries are tagged as #[link(once)]? (nit: maybe link(static) would be clearer?) > I should rephrase. I'm not an ABI expert, and my concerns actually > aren't really that related to ABI (although I know that many others do > have concerns about this). My primary concern is the current fragility > of a symbol in rust. If I add a doc-comment in libstd, I will likely > change many symbols in the output dynamic library. This is another > problem entirely, but it has repercussions for this right now. In > favoring dynamic libraries, rust is stating that it is available for > an in-place upgrade. The rust compiler is currently sufficiently far > from this goal that I do not believe that we should be favoring > dynamic linking. Is this just a matter of changing what is hashed when we construct the symbol name (or dropping the symbol hashes entirely)? That doesn't seem very far. Are there are a lot of other things standing in the way that immediately come to mind? Can we try and document what those issues are? Things that I can think of off the top of my head: - Static constants: what things are compiled in to foreign crates? - Enum discriminants: do we want to provide a (slow) way for crates to match against enum variants without inlining the discriminat value associated with each variant? - Inline and generic functions: currently, we export the bodies of fns that are marked #[inline] or which are public and generic. Clearly if one is concerned about binary compatibility, these functions are the most subtle ones to reason about, since a downstream consumer will continue to use the old version *even if* they link against your new library. It might make sense to have a rule that, in a dynamic linking scenario, no function body is exported that is not explicitly annotated for export. This implies that we have a way to "pre-instantiate" generic functions (like C++ does) and probably that we have an annotation that means "export this IR so it can be instantiated" but which does not provide an LLVM inline hint, since that might not be appropriate. > Does that make sense? This is a fairly major decision, and I want to > make sure that everyone's on board with it. Regardless of whether or not we are able to present a clear ABI compatibility story (and I agree we do not right now), it's always going to be a fairly complicated set of requirements that library authors will have to reason about. We can make rules that guarantee binary compatibility, but that does not imply semantic compatibility. The sense of a boolean parameter might change, for example, or the set of flags might change. Therefore, I can see an argument that says one ought to "opt in" to dynamic linking. OTOH, it's more the producer that has to be careful about using semantic versioning and so on, not so much the consumer, so I'm not sure whether that argument really makes sense. Niko ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Rethinking Linking in Rust
>> Hm, I suppose I should re-phrase. Rust's linkage model should not >> attempt to lift dependence on global native libraries. These global >> libraries (like libm and librt on linux) should be assumed to be >> everywhere. Our result artifacts must always be linked against them >> (if their functionality is used). On the other hand, any build >> artifacts that are a result of a local build process should not be >> considered downstream dependencies as well. To this end, I mainly >> point out that rust should roll in local native static libraries, and >> just live with global native dynamic libraries. > > This can't be assumed if we want to support freestanding use. Removing > the need for rust-core means support for environments without features > like floating point and amenities provided by a kernel to user-land > processes. Perhaps I'm misunderstanding what you mean here. In saying this is what I think that rust should do, I should also mention that I know of no other method of doing this. If rustc creates a static library, then it may have no dynamic dependencies encoded anywhere. Global dependencies are not a problem which can be solved by static linking or not, they are a problem of the standard library itself. Here's a little example. On linux, the libm library is a global system dynamic library (if I'm wrong, just assume it is). Rust's stdlib has a requirement on this library, acos. This is defined on floating point values (the acos) function, and this will eventually call the corresponding libm function. In this scenario, it is impossible to distribute a libstd which does *not* have a dependence on libm if you use the acos function (or at least it's impossible within the limits of my knowledge). Now that being said, all is not lost. First off, if we have a static libstd.rlib, then I believe that this would "just work". In my scheme, let's say you want to create a static library with no dependence on any global dynamic libraries. This means that you will statically link to libstd.rlib, creating libfoo.a. In doing so, the compiler will warn you about the dynamic library dependencies of libstd, in this case that includes libm. The compiler will *not* bring in any of them as dependencies (because it can't). When you link libfoo.a into your application, you will get an undefined reference error if you used the acos function, or you will get no error at all if you did not use the acos function. If you receive an error, you learn that the dynamic dependencies which were not linked are probably needed for those symbols. All in all, a major goal of this redesign is to support freestanding usage of rust. Rust's linkage model should not prevent you from using rust in virtually any location. This redesign is not a "silver bullet" in making freestanding rust work, there is more changes which need to happen elsewhere. Rust currently has a number of dynamic library dependencies on various platforms, and we need to figure out how to drop them in some situations. For example, perhaps the introduction of the libm dependency should only be done if you compile the num::f64 module, and perhaps this module shouldn't be compiled in the --cfg freestanding version of libstd. Does that make sense? I want to make sure that *linkage* does not block freestanding rust. If all of this were implemented tonight, I don't believe that rust would be "completely ready" for freestanding use, but it would be a whole lot closer. ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Rethinking Linking in Rust
On Fri, Nov 15, 2013 at 12:28 PM, Alex Crichton wrote: >>> Primarily, I believe that if desired, rustc should be able to generate an >>> executable or dynamic library with no dependence on any rust libraries. This >>> includes things like librustrt and libextra. Rust shouldn't be striving to >>> lift >>> dependence on system libraries, that'll come at later times if need be. >> >> It seems like you *are* striving to lift dependencies on non-rust >> libraries, though. For example, you mention having libuv be statically >> imported into a .rlib file etc. Or did I misunderstand? > > Hm, I suppose I should re-phrase. Rust's linkage model should not > attempt to lift dependence on global native libraries. These global > libraries (like libm and librt on linux) should be assumed to be > everywhere. Our result artifacts must always be linked against them > (if their functionality is used). On the other hand, any build > artifacts that are a result of a local build process should not be > considered downstream dependencies as well. To this end, I mainly > point out that rust should roll in local native static libraries, and > just live with global native dynamic libraries. This can't be assumed if we want to support freestanding use. Removing the need for rust-core means support for environments without features like floating point and amenities provided by a kernel to user-land processes. Perhaps I'm misunderstanding what you mean here. ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Rethinking Linking in Rust
>> Primarily, I believe that if desired, rustc should be able to generate an >> executable or dynamic library with no dependence on any rust libraries. This >> includes things like librustrt and libextra. Rust shouldn't be striving to >> lift >> dependence on system libraries, that'll come at later times if need be. > > It seems like you *are* striving to lift dependencies on non-rust > libraries, though. For example, you mention having libuv be statically > imported into a .rlib file etc. Or did I misunderstand? Hm, I suppose I should re-phrase. Rust's linkage model should not attempt to lift dependence on global native libraries. These global libraries (like libm and librt on linux) should be assumed to be everywhere. Our result artifacts must always be linked against them (if their functionality is used). On the other hand, any build artifacts that are a result of a local build process should not be considered downstream dependencies as well. To this end, I mainly point out that rust should roll in local native static libraries, and just live with global native dynamic libraries. > Does this imply that all Rust programs will be executed with whatever > specific version of libsundown and libuv was produced as part of the > Rust compiler build process? (I'm not implying this is a bad thing, > necessarily, just trying to understand) Correct. We as distributors of the rust compile could choose to make libsundown a dynamic library which is then distributed and upgradeable. We may also choose to link it statically to show that it cannot be in-place upgraded. >> I would propose that the compiler automatically favors static >> linkage over dynamic linkage due to various rust ABI issues. > > Could you elaborate? What issues are you thinking of? I should rephrase. I'm not an ABI expert, and my concerns actually aren't really that related to ABI (although I know that many others do have concerns about this). My primary concern is the current fragility of a symbol in rust. If I add a doc-comment in libstd, I will likely change many symbols in the output dynamic library. This is another problem entirely, but it has repercussions for this right now. In favoring dynamic libraries, rust is stating that it is available for an in-place upgrade. The rust compiler is currently sufficiently far from this goal that I do not believe that we should be favoring dynamic linking. By favoring static linking instead, we are lifting ourselves from this burden. When we upgrade the rust compiler a year from now (2.0, woo!), all previous rust executables will continue to run just fine (assuming they weren't linked dynamically). Additionally, any application which has a linked version of rust will continue to work. Does that make sense? This is a fairly major decision, and I want to make sure that everyone's on board with it. > Unless I'm missing something, it seems like what we would want to do > is to have the .rlib file contain LLVM IR, at least for the Rust code > that was compiled / statically linked against. If we can, I think we > should just make LTO happen by default whenver you statically link, > rather than having it be a separate option Interesting idea! Sounds like this definitely along the right lines, and so long as we don't advertise our .rlib format it sounds like we can silently do this at any time in the future. ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Rethinking Linking in Rust
A few quick questions and comments: On Fri, Nov 15, 2013 at 12:09:28AM -0800, Alex Crichton wrote: > I've been thinking about static linking recently, along with a little bit of > linking in general, and I wanted to see what others thought. > > # The Goal > > Primarily, I believe that if desired, rustc should be able to generate an > executable or dynamic library with no dependence on any rust libraries. This > includes things like librustrt and libextra. Rust shouldn't be striving to > lift > dependence on system libraries, that'll come at later times if need be. It seems like you *are* striving to lift dependencies on non-rust libraries, though. For example, you mention having libuv be statically imported into a .rlib file etc. Or did I misunderstand? > As mentioned above, these files are similar to the compiler's .o output. The > only other component which can be considered for inclusion in this .o file is > all native static library dependencies. These are encoded as #[link(once)] in > linkage attributes. The reason for doing this is that it's likely to be common > to have a local static library which is not available in distribution, but is > always available for the build process. Examples for the compiler include > libsundown, libuv, libuv_support, and maybe librustrt. Does this imply that all Rust programs will be executed with whatever specific version of libsundown and libuv was produced as part of the Rust compiler build process? (I'm not implying this is a bad thing, necessarily, just trying to understand) > I would propose that the compiler automatically favors static > linkage over dynamic linkage due to various rust ABI issues. Could you elaborate? What issues are you thinking of? > This default could be switched in the future, but it simply means > that if the compiler finds a .so and a .rlib, it will suck in the > .rlib before sucking in the .so. What does "suck in" here mean? I don't think this is as simple as what file the compiler opens, but rather it means that by default you'd get static linking and hence wouldn't require the .so to be distributed but also couldn't support independent library version upgrades, right? > What are others' thoughts on this? Is this too complex of a system? Is there a > glaring omission of use cases? I think it generally makes sense and doesn't seem overly complicated, though I'm not sure if there are missing use cases or not. > As a side node, after writing all this up, I remembered LTO as an > option for generating libraries. I don't think I know enough about LTO > to be able to say whether it would fit in this system or not, but my > basic understanding is that an LTO library is just "IR in a box". We > could add a --lto output option which has pretty much the same > semantics as the --rlib option, but with a different format. Again > though, I haven't thought how native libraries would fit into that > scenario, but I believe that we could fairly easily accommodate LTO in > a system like this. Unless I'm missing something, it seems like what we would want to do is to have the .rlib file contain LLVM IR, at least for the Rust code that was compiled / statically linked against. If we can, I think we should just make LTO happen by default whenver you statically link, rather than having it be a separate option, but it is fine if that doesn't work yet in the first version (e.g., because .rlib is just a .o file). Still I agree that your scheme accommodates it just fine. Niko ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
[rust-dev] Rethinking Linking in Rust
I've been thinking about static linking recently, along with a little bit of linking in general, and I wanted to see what others thought. # The Goal Primarily, I believe that if desired, rustc should be able to generate an executable or dynamic library with no dependence on any rust libraries. This includes things like librustrt and libextra. Rust shouldn't be striving to lift dependence on system libraries, that'll come at later times if need be. Additionally, rustc should be able to generate libfoo.a where libfoo.a has no dependence on any rust libraries. This library can then be statically linked to another application. # Intermediate static libraries I personally know of no way to create a static library from a dynamic one, so to achieve this we would need to distribute libstd and libextra in some form that is not a shared library. This problem not only applies to libstd, but also to any rust library which wants to be statically linked. The first natural conclusion for for an intermediate format would be a .a file itself. Why not distribute libstd.a along with libstd.so. After all, a .a is only an archive which in our case would contain one .o file. In thinking about this though, I don't think that this is the best format. The main idea of providing intermediate .a files is to allow linkage to them via the normal system linker. To be usable, this would mean that all .a files rust generates would have to have their own statically linked version of libstd or otherwise everyone will have to find where libstd is guess the name and hash attached to it. This is infeasible for arbitrary libraries which could have arbitrarily many dependencies. # Native Libraries One part of linking which rust cannot forget is native libraries. Right now, native libraries are always linked against when compiling a local crate, but no native library dependencies are propagated among crates. Due to the nature of a static library and what I assume is the file format itself, a static rust library cannot link to its dependent dynamic libraries. We can, however, resolve all native static dependencies at compile time. # A Scheme for Linking With the above knowledge, I would propose the following linkage model for rust. There are four types of files that the rust compiler will generate: 1. An executable 2. A dynamic library (.so, .dylib, .dll) 3. A "rust" static library (.rlib) 4. A regular static library (.a, .lib) The "rust language" would ship with dynamic library files as well as .rlib files. There would be no .a files in the distribution. A rust static library would be a format defined by rust that is not available for or intended for external use. It is meant to be beneficial to the rust compiler and that's it. It just so happens that their first incarnation would be created similarly to `cp foo.o foo.rlib`. In addition to these changes, the linkage attributes would change to be as follows: * #[link_args] becomes gated behind a feature flag. I believe that this is still a very useful ability to pass arbitrary flags to the linker, but this is *not* a sanctioned way of doing so at all because of how platform specific it is * #[link(...)] becomes the new method of specifying linkage semantics on extern blocks, and it may be used similarly to link_args today * #[link(name = "foo")] specifies that this crate links to native library `foo` * #[link(once)] implies that the native library is a static library, hence it *must* be linked against in the current compilation, regardless of the output format Omission of `link(once)` assumes that the library is available at all destinations, and it may not be linked against in the current compilation unit. ## The Linkage Step To see how this affects how artifacts are created, I'd like to go into detail about how each of the four output artifacts all interact with one another by describing the linkage phase of each output. For each of these, remember that the compiler's output is one .o file for each crate. Also remember that all rust libraries will always link to all upstream rust libraries. ### Linking Executables and Dynamic Libraries These two cases are very similar because they are creating the actual "result artifact" in terms of a file which will have no more linkage performed on it. The following components must be linked in to produce the artifact: * The local .o file * All local native dependencies * All upstream rust libraries (dynamic and static) * All non-once (dynamic) native libraries of upstream static crates. More on this later The result artifact needs to be a fully resolved destination artifact. The point of this is to have a dynamic dependency on all upstream dynamic libraries, and all upstream static libraries will have been sucked in to create the target. ### Creating rust static libraries (.rlib files) As mentioned above, these files are similar to the compiler's .o output. The only other component which can be consi
