Re: [rust-dev] Separated/Incremential compilation
Maybe this should be done upstream in LLVM, actually. Seems like work that would be applicable to e.g. clang with LTO as well. Tim Chevalier wrote: >On Fri, Nov 29, 2013 at 9:09 AM, Patrick Walton >wrote: >> I shouldn't say that Rust has no problems with build times--it could >always >> be faster, and in particular the memory representations are >inefficient, >> particularly around ASTs--but when you actually run with `-Z >time-passes`, >> you'll see that the vast majority of the time for any >reasonably-sized crate >> is spent in LLVM. There isn't much we can do to make that faster by >an order >> of magnitude, other than to try to push on the parallel per-function >> optimization and codegen work that is happening in some upstream >branches. >> Mergefunc, disabling exceptions, and the no-zeroing-out stuff that >Niko is >> doing would be nice, but they won't improve build times by an order >of >> magnitude. > >What about caching LLVM bitcode for individual Rust functions / items >(using workcache, for example) and only recompiling those items whose >dependencies have changed? Obviously this would be a lot of design and >implementation work, and one would want to do the math to make sure >it's likely to improve build performance, but offhand I can't see why >it's not feasible. > >The scenario I'm thinking of is "add a debug! statement to one >function, and only recompile the code for that function since its >interface hasn't changed". In that case, only regenerating code for >the changed function and not the entire crate should make a big >difference. > >Cheers, >Tim > >> >> Patrick >> >> [1]: >> >https://groups.google.com/forum/#!topic/mozilla.dev.platform/WjcCfckml4A >> >> >> ___ >> Rust-dev mailing list >> Rust-dev@mozilla.org >> https://mail.mozilla.org/listinfo/rust-dev > > > >-- >Tim Chevalier * http://catamorphism.org/ * Often in error, never in >doubt >"If you are silent about your pain, they'll kill you and say you >enjoyed it." >-- Zora Neale Hurston -- Sent from my Android phone with K-9 Mail. Please excuse my brevity.___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Separated/Incremential compilation
On 11/29/2013 03:01 AM, Léo Testard wrote: Hello, I think everyone here will agree to say that compilation times in Rust are problematic. Recently, there was an argument on IRC about reducing compilation times by reducing the use of GC and failures. Although I agree it's good to reduce Rustc's overhead, I think there are more important problems. The total duration of a build matters only because you have to recompile the whole crate on each modification. In C++, the duration of the complete build of a project matters less because when you compile incrementally, you only have to rebuild a couple of files - those you modified. I know the "1 crate = 1 compilation unit" is the model chosen by Rust, but this is a major issue for production. Nobody will ever use Rust in production if they have to recompile thousands of lines of code on each modification. On some of my personal projects, I "solved" this problem by splitting the codebase into several crates, that I compile statically, and then link together using extern mod. This is not really a solution, because this implies that there is no cyclic dependency between each of the small crates, or I end up with issues trying to compile it, because using extern mod requires that the library corresponding to that mod exists before compiling the crate that depends on it. But strictly speaking, a compiled crate is nothing more than a module hierarchy, and so is a single Rust source file, so we should be able to compile a single file to some sort of .o and then link all together to form a crate. References to modules outside of this file just require the first passes of the build, not the code generation, so it should be ok regarding to cyclic dependencies, and if not, we could still introduce some kind of auto-generated interface file, like Caml does. I know it's quite a big work, and that the current system is quite good, but having this is very important if we want Rust to be used in production. I agree that incremental recompilation would be a good thing to pursue, and I'd be happy to see someone work on it, but as you say it is a very difficult problem, and it's frankly a low priority. Some other compilition performance wins to pursue are removing metadata compression (causes dynamic linker crashes on linux), improving llvm perf, and improving linker perf. ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Separated/Incremential compilation
On Fri, Nov 29, 2013 at 9:09 AM, Patrick Walton wrote: > I shouldn't say that Rust has no problems with build times--it could always > be faster, and in particular the memory representations are inefficient, > particularly around ASTs--but when you actually run with `-Z time-passes`, > you'll see that the vast majority of the time for any reasonably-sized crate > is spent in LLVM. There isn't much we can do to make that faster by an order > of magnitude, other than to try to push on the parallel per-function > optimization and codegen work that is happening in some upstream branches. > Mergefunc, disabling exceptions, and the no-zeroing-out stuff that Niko is > doing would be nice, but they won't improve build times by an order of > magnitude. What about caching LLVM bitcode for individual Rust functions / items (using workcache, for example) and only recompiling those items whose dependencies have changed? Obviously this would be a lot of design and implementation work, and one would want to do the math to make sure it's likely to improve build performance, but offhand I can't see why it's not feasible. The scenario I'm thinking of is "add a debug! statement to one function, and only recompile the code for that function since its interface hasn't changed". In that case, only regenerating code for the changed function and not the entire crate should make a big difference. Cheers, Tim > > Patrick > > [1]: > https://groups.google.com/forum/#!topic/mozilla.dev.platform/WjcCfckml4A > > > ___ > Rust-dev mailing list > Rust-dev@mozilla.org > https://mail.mozilla.org/listinfo/rust-dev -- Tim Chevalier * http://catamorphism.org/ * Often in error, never in doubt "If you are silent about your pain, they'll kill you and say you enjoyed it." -- Zora Neale Hurston ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Separated/Incremential compilation
I experimented with LZ4. https://github.com/mozilla/rust/pull/6954. It isn't worth it, IMO. On Fri, Nov 29, 2013 at 1:51 PM, Carter Charbonneau wrote: > Why not lz4? It's faster than snappy. > > On Nov 29, 2013 11:29 AM, "Patrick Walton" wrote: >> >> On 11/29/13 10:26 AM, comex wrote: >>> >>> On Fri, Nov 29, 2013 at 12:07 PM, Daniel Micay >>> wrote: A minimal program definitely doesn't compile more slowly than `clang`: >>> >>> >>> Well, I said it was mostly unrelated. :) >>> >>> Importing the std crate is responsible for the overhead, but if the C >>> program can bring in basic library functions in the time it takes Rust >>> to bring in nothing, it's still faster. >>> >>> ...but according to Instruments, almost 70% of the compilation time >>> for an empty non-#[no_std] crate is being spent inside >>> flate::inflate_bytes (93ms). If that's accurate, it doesn't sound too >>> hard to fix, if it matters to anyone. Compilation time for more >>> substantial crates is more interesting, of course, but I do like the >>> instantaneous feeling of compiling small C utilities. >> >> >> Yeah, we need to rework the representation of metadata. I'd also like to >> try switching to Snappy at some point (or just not compressing). >> >> Patrick >> ___ >> Rust-dev mailing list >> Rust-dev@mozilla.org >> https://mail.mozilla.org/listinfo/rust-dev > > > ___ > Rust-dev mailing list > Rust-dev@mozilla.org > https://mail.mozilla.org/listinfo/rust-dev > ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Separated/Incremential compilation
Why not lz4? It's faster than snappy. On Nov 29, 2013 11:29 AM, "Patrick Walton" wrote: > On 11/29/13 10:26 AM, comex wrote: > >> On Fri, Nov 29, 2013 at 12:07 PM, Daniel Micay >> wrote: >> >>> A minimal program definitely doesn't compile more slowly than `clang`: >>> >> >> Well, I said it was mostly unrelated. :) >> >> Importing the std crate is responsible for the overhead, but if the C >> program can bring in basic library functions in the time it takes Rust >> to bring in nothing, it's still faster. >> >> ...but according to Instruments, almost 70% of the compilation time >> for an empty non-#[no_std] crate is being spent inside >> flate::inflate_bytes (93ms). If that's accurate, it doesn't sound too >> hard to fix, if it matters to anyone. Compilation time for more >> substantial crates is more interesting, of course, but I do like the >> instantaneous feeling of compiling small C utilities. >> > > Yeah, we need to rework the representation of metadata. I'd also like to > try switching to Snappy at some point (or just not compressing). > > Patrick > ___ > Rust-dev mailing list > Rust-dev@mozilla.org > https://mail.mozilla.org/listinfo/rust-dev > ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Separated/Incremential compilation
On 11/29/13 10:29 AM, György Andrasek wrote: On 11/29/2013 06:09 PM, Patrick Walton wrote: compilation times. This thread [1] reports build time increases of 6x-15x! No, he reported build time *improvements* of 6x-15x. Quoting later: Yeah, that's what I meant to say. Too early in the morning. :) Patrick ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Separated/Incremential compilation
On 11/29/13 10:26 AM, comex wrote: On Fri, Nov 29, 2013 at 12:07 PM, Daniel Micay wrote: A minimal program definitely doesn't compile more slowly than `clang`: Well, I said it was mostly unrelated. :) Importing the std crate is responsible for the overhead, but if the C program can bring in basic library functions in the time it takes Rust to bring in nothing, it's still faster. ...but according to Instruments, almost 70% of the compilation time for an empty non-#[no_std] crate is being spent inside flate::inflate_bytes (93ms). If that's accurate, it doesn't sound too hard to fix, if it matters to anyone. Compilation time for more substantial crates is more interesting, of course, but I do like the instantaneous feeling of compiling small C utilities. Yeah, we need to rework the representation of metadata. I'd also like to try switching to Snappy at some point (or just not compressing). Patrick ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Separated/Incremential compilation
On 11/29/2013 06:09 PM, Patrick Walton wrote: compilation times. This thread [1] reports build time increases of 6x-15x! No, he reported build time *improvements* of 6x-15x. Quoting later: > You're right, I did in fact use ccache because I was using the wrong mozconfig! (Rookie mistake.) With that fixed, we go down from 1m51s on my machine to 31s, which is only about 4x faster! ...which sounds about right for unified C++ builds. The downside being, of course, the memory usage and the longer edit/compile/run cycle. ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Separated/Incremential compilation
On Fri, Nov 29, 2013 at 12:07 PM, Daniel Micay wrote: > A minimal program definitely doesn't compile more slowly than `clang`: Well, I said it was mostly unrelated. :) Importing the std crate is responsible for the overhead, but if the C program can bring in basic library functions in the time it takes Rust to bring in nothing, it's still faster. ...but according to Instruments, almost 70% of the compilation time for an empty non-#[no_std] crate is being spent inside flate::inflate_bytes (93ms). If that's accurate, it doesn't sound too hard to fix, if it matters to anyone. Compilation time for more substantial crates is more interesting, of course, but I do like the instantaneous feeling of compiling small C utilities. ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Separated/Incremential compilation
On 11/29/13 3:01 AM, Léo Testard wrote: Although I agree it's good to reduce Rustc's overhead, I think there are more important problems. The total duration of a build matters only because you have to recompile the whole crate on each modification. In C++, the duration of the complete build of a project matters less because when you compile incrementally, you only have to rebuild a couple of files - those you modified. I know the "1 crate = 1 compilation unit" is the model chosen by Rust, but this is a major issue for production. Nobody will ever use Rust in production if they have to recompile thousands of lines of code on each modification. In practice this isn't as much of a benefit for C++ as claimed. In C++, when you change a header file you often have to rebuild huge numbers of files, and most nontrivial changes change header files. In fact, Firefox is moving away from the C++ model to a more Rust-like model of unified builds on a per-directory basis precisely because it improves compilation times. This thread [1] reports build time increases of 6x-15x! Furthermore, when you do link-time optimization, which is generally a requirement for software that needs to be fast these days, you can't really do separate compilation at all. I shouldn't say that Rust has no problems with build times--it could always be faster, and in particular the memory representations are inefficient, particularly around ASTs--but when you actually run with `-Z time-passes`, you'll see that the vast majority of the time for any reasonably-sized crate is spent in LLVM. There isn't much we can do to make that faster by an order of magnitude, other than to try to push on the parallel per-function optimization and codegen work that is happening in some upstream branches. Mergefunc, disabling exceptions, and the no-zeroing-out stuff that Niko is doing would be nice, but they won't improve build times by an order of magnitude. Patrick [1]: https://groups.google.com/forum/#!topic/mozilla.dev.platform/WjcCfckml4A ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Separated/Incremential compilation
On Fri, Nov 29, 2013 at 12:00 PM, comex wrote:
> On Fri, Nov 29, 2013 at 11:24 AM, Patrick Walton wrote:
>> I disagree. Rust doesn't compile that much slower than other languages at
>> this point.
>
> If by other languages you mean C++, C++ compile times are a huge
> turnoff for me, especially when I know that the work that
> theoretically needs to be done to compile a small incremental change
> is almost zero. Rust, which has actual modules, should be much faster
> to compile, not slower. Just my two cents.
>
> Of course, I don't know whether you're actually talking about C++;
> without experience compiling large Rust codebases, the only number
> I've gotten to know is rustc being about 4 times slower to compile an
> empty crate than clang for an empty c file (even if the latter has
> some standard #includes), but that's mostly unrelated.
A minimal program definitely doesn't compile more slowly than `clang`:
```
int main(void) {
return 0;
}
```
clang foo.c -emit-llvm 0.03s user 0.03s system 91% cpu 0.062 total
```
#[no_std];
#[start]
fn main(_: int, _: **u8) -> int {
0
}
```
rustc foo.rs --emit-llvm 0.02s user 0.01s system 91% cpu 0.025 total
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Separated/Incremential compilation
On Fri, Nov 29, 2013 at 11:24 AM, Patrick Walton wrote: > I disagree. Rust doesn't compile that much slower than other languages at > this point. If by other languages you mean C++, C++ compile times are a huge turnoff for me, especially when I know that the work that theoretically needs to be done to compile a small incremental change is almost zero. Rust, which has actual modules, should be much faster to compile, not slower. Just my two cents. Of course, I don't know whether you're actually talking about C++; without experience compiling large Rust codebases, the only number I've gotten to know is rustc being about 4 times slower to compile an empty crate than clang for an empty c file (even if the latter has some standard #includes), but that's mostly unrelated. ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Separated/Incremential compilation
On 11/29/13 3:01 AM, Léo Testard wrote: Hello, I think everyone here will agree to say that compilation times in Rust are problematic. I disagree. Rust doesn't compile that much slower than other languages at this point. > For example, I think that if we can reduce significantly the Rust compiler's compilation time, it could allow more developers to contribute to the Rust language (as they won't have to wait 30min for each small modifications in the compiler). This is because of bootstrapping, not the language. Patrick ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Separated/Incremential compilation
On Fri, Nov 29, 2013 at 6:22 AM, Guillaume HERVIER wrote: > +1 for this issue. I think that compilation time is really important if we > want Rust to be used as production language. > > For example, I think that if we can reduce significantly the Rust compiler's > compilation time, it could allow more developers to contribute to the Rust > language (as they won't have to wait 30min for each small modifications in > the compiler). > Personally, it's the only thing which blocks me when I want to contribute to > Rust, because I like to often compile code when I do small modifications to > test each of these small modifications, partly because I don't know the > language very well. The compiler has to bootstrap itself so incremental compilation won't decrease the build times. Splitting up crates into many crates does allow for incremental compilation, but it will also increase the *total* build time, since work is being redone for any generic or cross-crate inlined functions. The final binary will also be larger and slower than using a single crate, so supporting link-time optimization for release builds is important. ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Separated/Incremential compilation
+1 for this issue. I think that compilation time is really important if we want Rust to be used as production language. For example, I think that if we can reduce significantly the Rust compiler's compilation time, it could allow more developers to contribute to the Rust language (as they won't have to wait 30min for each small modifications in the compiler). Personally, it's the only thing which blocks me when I want to contribute to Rust, because I like to often compile code when I do small modifications to test each of these small modifications, partly because I don't know the language very well. On 11/29/2013 12:01 PM, Léo Testard wrote: > > Hello, > > I think everyone here will agree to say that compilation times in Rust > are problematic. Recently, there was an argument on IRC about reducing > compilation times by reducing the use of GC and failures. Although I > agree it's good to reduce Rustc's overhead, I think there are more > important problems. The total duration of a build matters only because > you have to recompile the whole crate on each modification. In C++, > the duration of the complete build of a project matters less because > when you compile incrementally, you only have to rebuild a couple of > files - those you modified. I know the "1 crate = 1 compilation unit" > is the model chosen by Rust, but this is a major issue for production. > Nobody will ever use Rust in production if they have to recompile > thousands of lines of code on each modification. > > On some of my personal projects, I "solved" this problem by splitting > the codebase into several crates, that I compile statically, and then > link together using extern mod. This is not really a solution, because > this implies that there is no cyclic dependency between each of the > small crates, or I end up with issues trying to compile it, because > using extern mod requires that the library corresponding to that mod > exists before compiling the crate that depends on it. > > But strictly speaking, a compiled crate is nothing more than a module > hierarchy, and so is a single Rust source file, so we should be able > to compile a single file to some sort of .o and then link all together > to form a crate. References to modules outside of this file just > require the first passes of the build, not the code generation, so it > should be ok regarding to cyclic dependencies, and if not, we could > still introduce some kind of auto-generated interface file, like Caml > does. I know it's quite a big work, and that the current system is > quite good, but having this is very important if we want Rust to be > used in production. > > Sorry if this topic has already been posted several times, but I feel > this is important, and the related issues seem to date (#2369). I > think it's a real mistake to report this to post-1.0. > > Leo > > > > ___ > Rust-dev mailing list > Rust-dev@mozilla.org > https://mail.mozilla.org/listinfo/rust-dev signature.asc Description: OpenPGP digital signature ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
