Re: [Mesa-dev] Rust drivers in Mesa
On Fri, 2020-10-02 at 18:21 -0500, Jason Ekstrand wrote: > > The good thing about C++ is that you can pick a subset of features > > that > > you like, and stick to them. Then you'll never have to deal with > > other > > the stuff if you don't want to. There exists a subset of the > > language > > which helps get rid of most memory management issues. > > The difficulty in Mesa has always been deciding what that subset is. > This thread is proof of that. So far we've had people extole > everything from basic type safety and memory management to template > meta-programming. Features which one developer enjoys another > loathes. This is why we tend to stick to C for the core: It's > limited > but those limitations we can sort-of all agree on it. Yes, the bad thing about C++ is that it has a lot of baggage. Sometimes I find myself enjoying writing C more, because then I get to spend more time on the actual problems I'm solving. In C++ there are a lot of ways to do everything so I have to think more about which fits the task at hand best. My only problem with C is that it lacks something like the STL which means that every project has to reinvent the wheel to some degree; and that it lacks things like generics, so there isn't a good way to avoid repetitive code. > IMO: Like many modern languages, Rust has a defined default coding > style and It's pretty reasonable. We should just go with it. Fair point. > > That said, I think it's perfectly okay to give it a try, write a > > backend compiler in Rust and see what happens. However, I don't > > think > > it is feasible or desireable to rewrite complicated parts from > > C/C++ to > > Rust. I think Daniel Steinberg's thoughts apply: > > https://daniel.haxx.se/blog/2017/03/27/curl-is-c/ > > Same. Different developers/teams make different choices in their > drivers and back-ends all the time. Some use C++ while others stickk > to C. They all have different coding styles and choose different C++ > feature sets (when they use C++). IMO, the choice to use a language > like Rust is no different. As long as it doesn't cause a giant > project-wide problem, I'm fine with it. Yeah, nothing wrong with that. I'll be curious to see how it turns out. - Tim ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] Rust drivers in Mesa
On Fri, Oct 2, 2020 at 5:51 PM wrote: > > On Fri, 2020-10-02 at 12:53 -0500, Jason Ekstrand wrote: > > On Fri, Oct 2, 2020 at 11:34 AM Eric Anholt wrote: > > > On Thu, Oct 1, 2020 at 6:36 PM Alyssa Rosenzweig > > > wrote: > > > > Hi all, > > > > > > > > Recently I've been thinking about the potential for the Rust > > > > programming > > > > language in Mesa. Rust bills itself a safe system programming > > > > language > > > > with comparable performance to C [0], which is a naturally fit > > > > for > > > > graphics driver development. > > > > > > > > [...] > > > > That said, I really do like Rust as a language. I'm not sure I'd go > > quite as far as Anholt but I feel like it has everything I like about > > C++ without most of what I dislike. I'm not sure how we would do > > things like ANV's GenXML multi-compile magic but I'm sure we could > > figure out a way. I'd love to see someone build something in Rust > > and > > figure out some best practices. > > > > --Jason > > The good thing about C++ is that you can pick a subset of features that > you like, and stick to them. Then you'll never have to deal with other > the stuff if you don't want to. There exists a subset of the language > which helps get rid of most memory management issues. The difficulty in Mesa has always been deciding what that subset is. This thread is proof of that. So far we've had people extole everything from basic type safety and memory management to template meta-programming. Features which one developer enjoys another loathes. This is why we tend to stick to C for the core: It's limited but those limitations we can sort-of all agree on it. > I would also argue that there exist a bunch of tools that can help make > C/C++ code stay safe, for example our collegaue Tony has recently > addressed some findings by an undefined behaviour sanitizer. There are > also free and open source static analyzers. > > I like the idea of Rust, especially their intention to focus on safety, > but honestly, I don't like how they realized it. > > The Rust syntax is slightly annoying. They departed from C/C++ enough > to make Rust look different, but then they got lazy and for some reason > they chose to keep the most annoying parts from C/C++ like curly braces > and semicolons, so even if we switch to Rust we can still enjoy > debating important topics like where to put curly braces, how many > spaces or tabs are best, and so on. IMO: Like many modern languages, Rust has a defined default coding style and It's pretty reasonable. We should just go with it. > My main concern is that memory is just one of many resources that we > need to use safely, but it's the only one that Rust concerns itself > with. Memory safety can give you a false sense of security. It's > possible to write insecure apps that are perfectly memory safe. > > That said, I think it's perfectly okay to give it a try, write a > backend compiler in Rust and see what happens. However, I don't think > it is feasible or desireable to rewrite complicated parts from C/C++ to > Rust. I think Daniel Steinberg's thoughts apply: > https://daniel.haxx.se/blog/2017/03/27/curl-is-c/ Same. Different developers/teams make different choices in their drivers and back-ends all the time. Some use C++ while others stickk to C. They all have different coding styles and choose different C++ feature sets (when they use C++). IMO, the choice to use a language like Rust is no different. As long as it doesn't cause a giant project-wide problem, I'm fine with it. Even bits in core like the SPIR-V parser, if all the developers who are working in that area want to write it in Rust, I'd say it's ok. I doubt it will actually happen because, has has been pointed out, those bits of the code base are all large and aging and rewriting the SPIR-V parser, as much as it might be fun from a "make it more secure" perspective, would likely introduce far more bugs in the short term than it would solve long-term. --Jason ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] Rust drivers in Mesa
On Fri, 2020-10-02 at 12:53 -0500, Jason Ekstrand wrote: > On Fri, Oct 2, 2020 at 11:34 AM Eric Anholt wrote: > > On Thu, Oct 1, 2020 at 6:36 PM Alyssa Rosenzweig > > wrote: > > > Hi all, > > > > > > Recently I've been thinking about the potential for the Rust > > > programming > > > language in Mesa. Rust bills itself a safe system programming > > > language > > > with comparable performance to C [0], which is a naturally fit > > > for > > > graphics driver development. > > > > > [...] > > That said, I really do like Rust as a language. I'm not sure I'd go > quite as far as Anholt but I feel like it has everything I like about > C++ without most of what I dislike. I'm not sure how we would do > things like ANV's GenXML multi-compile magic but I'm sure we could > figure out a way. I'd love to see someone build something in Rust > and > figure out some best practices. > > --Jason The good thing about C++ is that you can pick a subset of features that you like, and stick to them. Then you'll never have to deal with other the stuff if you don't want to. There exists a subset of the language which helps get rid of most memory management issues. I would also argue that there exist a bunch of tools that can help make C/C++ code stay safe, for example our collegaue Tony has recently addressed some findings by an undefined behaviour sanitizer. There are also free and open source static analyzers. I like the idea of Rust, especially their intention to focus on safety, but honestly, I don't like how they realized it. The Rust syntax is slightly annoying. They departed from C/C++ enough to make Rust look different, but then they got lazy and for some reason they chose to keep the most annoying parts from C/C++ like curly braces and semicolons, so even if we switch to Rust we can still enjoy debating important topics like where to put curly braces, how many spaces or tabs are best, and so on. My main concern is that memory is just one of many resources that we need to use safely, but it's the only one that Rust concerns itself with. Memory safety can give you a false sense of security. It's possible to write insecure apps that are perfectly memory safe. That said, I think it's perfectly okay to give it a try, write a backend compiler in Rust and see what happens. However, I don't think it is feasible or desireable to rewrite complicated parts from C/C++ to Rust. I think Daniel Steinberg's thoughts apply: https://daniel.haxx.se/blog/2017/03/27/curl-is-c/ Best regards, Tim ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] Rust drivers in Mesa
On Fri, Oct 2, 2020, 10:53 Jason Ekstrand wrote:
>
> 2. Rust's enums look awesome but are only mostly awesome:
> a. Pattern matching on them can lead to some pretty deep
> indentation which is a bit annoying.
> b. There's no good way to have multiple cases handled by the same
> code like you can with a C switch; you have to either repeat it or
> break it out into a generic helper.
>
You can use | in match which could help with 2.b:
https://doc.rust-lang.org/reference/expressions/match-expr.html
example:
match make_my_enum() {
B { strv: s, .. } | C(MyStruct(s)) => println!("B or C: the string is
{}", s),
A(_) => {}
}
struct MyStruct(String);
enum MyEnum {
A(String),
B {
intv: i32,
strv: String,
},
C(MyStruct),
}
Jacob Lifshay
___
mesa-dev mailing list
mesa-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] Rust drivers in Mesa
On Fri, Oct 2, 2020 at 11:34 AM Eric Anholt wrote:
>
> On Thu, Oct 1, 2020 at 6:36 PM Alyssa Rosenzweig
> wrote:
> >
> > Hi all,
> >
> > Recently I've been thinking about the potential for the Rust programming
> > language in Mesa. Rust bills itself a safe system programming language
> > with comparable performance to C [0], which is a naturally fit for
> > graphics driver development.
> >
> > Mesa today is written primarily in C, a notoriously low-level language,
> > with some components in C++. To handle the impedance mismatch, we've
> > built up a number of abstractions in-tree, including multiple ad hoc
> > code generators (GenXML, NIR algebraic passes, Bifrost disassembler). A
> > higher level language can help avoid the web of metaprogramming and
> > effect code that is simpler and easier to reason about. Similarly, a
> > better type system can aid static analysis.
> >
> > Beyond abstraction, Rust's differentiating feature is the borrow checker
> > to guarantee memory safety. Historically, safety has not been a primary
> > concern of graphics drivers, since drivers are implemented as regular
> > userspace code running in the process of the application calling them.
> > Unfortunately, now that OpenGL is being exposed to untrusted code via
> > WebGL, the driver does become an attack vector.
> >
> > For the time being, Mesa attempts to minimize memory bugs with defensive
> > programming, safe in-tree abstractions (including ralloc), and static
> > analysis via Coverity. Nevertheless, these are all heuristic solutions.
> > Static analysis is imperfect and in our case, proprietary software.
> > Ideally, the bugs we've been fixing via Coverity could be caught at
> > compile-time with a free and open source toolchain.
> >
> > As Rust would allow exactly this, I see the primary benefit of Rust in
> > verifying correctness and robustness, rather than security concerns per
> > se. Indeed, safety guarantees do translate well beyond WebGL.
> >
> > Practically, how would Rust fit in with our existing C codebase?
> > Obviously I'm not suggesting a rewrite of Mesa's more than 15 million
> > lines of C. Instead, I see value in introducing Rust in targeted parts
> > of the tree. In particular, I envision backend compilers written in part
> > in Rust. While creating an idiomatic Rust wrapper for NIR or Gallium
> > would be prohibitively costly for now, a backend compiler could be
> > written in Rust with IR builders exported for use of the NIR -> backend
> > IR translator written in C.
> >
> > This would have minimal impact on the tree. Users that are not building
> > such a driver would be unaffected. For those who _are_ building Rust
> > code, the Rust compiler would be added as a build-time dependency and
> > the (statically linked) Rust standard library would be added as a
> > runtime dependency. There is concern about the Rust compiler requiring
> > LLVM as a dependency, but again this is build-time, and no worse than
> > Mesa already requiring LLVM as a runtime dependency for llvmpipe and
> > clover. As for the standard library, it is possible to eliminate the
> > dependency as embedded Rust does, perhaps calling out to the C standard
> > library via the FFI, but this is likely quixotic. I do regret the binary
> > size increase, however.
> >
> > Implications for the build system vary. Rust prefers to be built by its
> > own package manager, Cargo, which is tricky to integrate with other
> > build systems. Actually, Meson has native support for Rust, invoking the
> > compiler directly and skipping Cargo, as if it were C code. This support
> > is not widely adopted as it prevents linking with external libraries
> > ("crates", in Rust parlance), with discussions between Rust and Meson
> > developers ending in a stand-still [1]. For Mesa, this might be just
> > fine. Our out-of-tree run-time dependencies are minimal for the C code,
> > and Rust's standard library largely avoids the need for us to maintain a
> > Rust version of util/ in-tree. If this proves impractical in the
> > long-term, it is possible to integrate Cargo with Meson on our end [2].
> >
> > One outstanding concern is build-time, which has been a notorious
> > growing pain for Rust due to both language design and LLVM itself [3],
> > although there is active work to improve both fronts [4][5]. I build
> > Mesa on my Arm laptop, so I suppose I'd be hurt more than many of us.
> > There's also awkward bootstrapping questions, but there is work here too
> > [6].
> >
> > If this is of interest, please discuss. It's clear to me Rust is not
> > going away any time soon, and I see value in Mesa embracing the new
> > technology. I'd like to hear other Mesa developers' thoughts.
>
> For me, every day I write C code, I wish I was writing rust. I've
> written hobby rust (https://crates.io/crates/gpu-trace-perf) and also
> dabbled in a huge project (https://servo.org/), and I've gone through
> a bit of the struggles with the borrow checker and come out the
Re: [Mesa-dev] Rust drivers in Mesa
On 02.10.20 08:14, Dave Airlie wrote: My feeling is the pieces that would benefit the most are the things touch the real world, GLSL compiler, SPIR-V handling, maybe some of the GL API space, but I also feel these are the messiest things to move to rust. I'm not sure you'd get much benefit from vulkan API space or touching kernel interfaces as much. Granted a backend compiler might be the easiest place to experiment. I think the problem is you really want to justify that whatever you introduce using rust is sufficiently better along any axis than c/c++ to justify the enhanced build time requirements and maintenance, otherwise someone will just rewrite it in C/C++ to avoid having to pull a rust compiler into their distro requires. Note that Firefox already requires rust to build for three years [1], so any distro shipping a supported version of Firefox has the rust compiler available. The LTS branch of Firefox is currently at version 78 and thus requires at least rust 1.41. [1]https://wiki.mozilla.org/Rust_Update_Policy_for_Firefox ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] Rust drivers in Mesa
On Fri, Oct 2, 2020 at 8:05 AM Dylan Baker wrote: > > I have serious concerns about cargo and crate usage. Cargo is basically npm > for rust, and shares all of the bad design decisions of npm, including > linking multiple versions of the same library together and ballooning > dependency lists that are fetched intrigued from the internet. This is both a > security problem and directly in conflict with meson's design off one and > only one version of a project. And while rust prevents certain kinds of bugs, > it doesn't prevent design bugs or malicious code. Add a meson developer the > rust community has been incredibly hard to work with and basically hostile to > every request we've made "cargo is hour you build rust", is essentially the > answer we've gotten from them at every turn. And if you're not going to use > cargo, is rust really a win? The standard library is rather minimal "because > just pull in 1000 crates". The distro people can correct me if I'm wrong, but > when librsvg went to rust it was a nightmare, several distros went a long > time without u pdates because of cargo. (I am not currently advocating for cargo usage, but...) The standard library is large compared to what we currently allow for Mesa dependencies (libexpat, libz, libdrm basically). First thought I had was that my hash table/set is garbage compared to theirs, and we could probably just wrap rust's with my C API and get a win for compiler perf. Having to write u_dynarray-using code makes one wish we had Vec<>. util/os_socket.c? We write all sorts of stuff to work around how we have no standard library worth speaking of currently, and time we spend optimizing hash_table.c and fnv1 (or oh wait xxhash32 or is that one actually best...) is time that we're not getting to spend on making graphics code better. If you're worried about cargo deps, introduce a cargo lockfile, even though we're a library. Lockfiles are for rust code used as a rust library, I see no reason not to lock when our library has a C interface. Then you don't have the multiple deps or random versions thing, it's up to us what we want to use, even if you're downloading git of that hash off the internet. Yes, there's the "if you blindly uprev, you can pull in malicious deps" thing. I feel pretty good about that given the rate of dependency compromise attacks I've seen detected in npm/cargo/etc., compared to the rate of CVEs we could generate if we chose to aim a fuzzer basically anywhere in our codebase. ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] Rust drivers in Mesa
On Thu, Oct 1, 2020 at 6:36 PM Alyssa Rosenzweig
wrote:
>
> Hi all,
>
> Recently I've been thinking about the potential for the Rust programming
> language in Mesa. Rust bills itself a safe system programming language
> with comparable performance to C [0], which is a naturally fit for
> graphics driver development.
>
> Mesa today is written primarily in C, a notoriously low-level language,
> with some components in C++. To handle the impedance mismatch, we've
> built up a number of abstractions in-tree, including multiple ad hoc
> code generators (GenXML, NIR algebraic passes, Bifrost disassembler). A
> higher level language can help avoid the web of metaprogramming and
> effect code that is simpler and easier to reason about. Similarly, a
> better type system can aid static analysis.
>
> Beyond abstraction, Rust's differentiating feature is the borrow checker
> to guarantee memory safety. Historically, safety has not been a primary
> concern of graphics drivers, since drivers are implemented as regular
> userspace code running in the process of the application calling them.
> Unfortunately, now that OpenGL is being exposed to untrusted code via
> WebGL, the driver does become an attack vector.
>
> For the time being, Mesa attempts to minimize memory bugs with defensive
> programming, safe in-tree abstractions (including ralloc), and static
> analysis via Coverity. Nevertheless, these are all heuristic solutions.
> Static analysis is imperfect and in our case, proprietary software.
> Ideally, the bugs we've been fixing via Coverity could be caught at
> compile-time with a free and open source toolchain.
>
> As Rust would allow exactly this, I see the primary benefit of Rust in
> verifying correctness and robustness, rather than security concerns per
> se. Indeed, safety guarantees do translate well beyond WebGL.
>
> Practically, how would Rust fit in with our existing C codebase?
> Obviously I'm not suggesting a rewrite of Mesa's more than 15 million
> lines of C. Instead, I see value in introducing Rust in targeted parts
> of the tree. In particular, I envision backend compilers written in part
> in Rust. While creating an idiomatic Rust wrapper for NIR or Gallium
> would be prohibitively costly for now, a backend compiler could be
> written in Rust with IR builders exported for use of the NIR -> backend
> IR translator written in C.
>
> This would have minimal impact on the tree. Users that are not building
> such a driver would be unaffected. For those who _are_ building Rust
> code, the Rust compiler would be added as a build-time dependency and
> the (statically linked) Rust standard library would be added as a
> runtime dependency. There is concern about the Rust compiler requiring
> LLVM as a dependency, but again this is build-time, and no worse than
> Mesa already requiring LLVM as a runtime dependency for llvmpipe and
> clover. As for the standard library, it is possible to eliminate the
> dependency as embedded Rust does, perhaps calling out to the C standard
> library via the FFI, but this is likely quixotic. I do regret the binary
> size increase, however.
>
> Implications for the build system vary. Rust prefers to be built by its
> own package manager, Cargo, which is tricky to integrate with other
> build systems. Actually, Meson has native support for Rust, invoking the
> compiler directly and skipping Cargo, as if it were C code. This support
> is not widely adopted as it prevents linking with external libraries
> ("crates", in Rust parlance), with discussions between Rust and Meson
> developers ending in a stand-still [1]. For Mesa, this might be just
> fine. Our out-of-tree run-time dependencies are minimal for the C code,
> and Rust's standard library largely avoids the need for us to maintain a
> Rust version of util/ in-tree. If this proves impractical in the
> long-term, it is possible to integrate Cargo with Meson on our end [2].
>
> One outstanding concern is build-time, which has been a notorious
> growing pain for Rust due to both language design and LLVM itself [3],
> although there is active work to improve both fronts [4][5]. I build
> Mesa on my Arm laptop, so I suppose I'd be hurt more than many of us.
> There's also awkward bootstrapping questions, but there is work here too
> [6].
>
> If this is of interest, please discuss. It's clear to me Rust is not
> going away any time soon, and I see value in Mesa embracing the new
> technology. I'd like to hear other Mesa developers' thoughts.
For me, every day I write C code, I wish I was writing rust. I've
written hobby rust (https://crates.io/crates/gpu-trace-perf) and also
dabbled in a huge project (https://servo.org/), and I've gone through
a bit of the struggles with the borrow checker and come out the other
side being really convinced that the language is worth it. Getting to
write rust for $dayjob is probably the only thing that could drag me
away from the Mesa project, which I love.
I think we'll miss out on a ton
Re: [Mesa-dev] Rust drivers in Mesa
On Fri, Oct 2, 2020 at 8:35 PM Dylan Baker wrote: > > Add a meson developer the rust community has been incredibly hard to work with > and basically hostile to every request we've made "cargo is hour you build > rust", > is essentially the answer we've gotten from them at every turn. > > On the meson front cargo is incredibly hard to integrate with meson, it's > essentially like calling cmake in autotools. > Also speaking as a meson developer, I'd like to share another point of view. As of today, Meson's integration with Cargo is not great. The only option you have is what Fractal does, as pointed out by Alyssa. However, I believe that the status quo is (at least in part) a consequence of us not merging the "unstable-cargo" Meson module[1] I wrote back in 2017. At the time we did not have a well-defined way to integrate Meson with other build systems, but now we do. I think we can give that another shot. The integration in that PR is not perfect, but it's the best start we can get. Mostly because we need to do quite some work on Cargo to make information available, standardization of some features used in build.rs, and so on. Some of this has already begun[2]. But setting all this aside, as Alyssa has already mentioned, you do not need Cargo to use Rust, and Meson already supports that use-case. You can compile and link Rust code like you would C or C++ code using Meson today. I think this can be a good fit for Mesa. Overall, my point of view is that as non-contributors, non-participants, and non-users, we have no voice in the future of Cargo or Rust. I can completely understand why it has been hard to work with them; it would be the same with any other FOSS project. The fix is straightforward: start using it, start participating, and *then* we will be in a position to complain and work together in making change happen. The fundamental property of software is that no one fixes anything till someone really wants it fixed. Cheers, Nirbheek 1. https://github.com/mesonbuild/meson/pull/2617 2. https://docs.rs/system-deps/1.3.1/system_deps/ ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] Rust drivers in Mesa
I have serious concerns about cargo and crate usage. Cargo is basically npm for
rust, and shares all of the bad design decisions of npm, including linking
multiple versions of the same library together and ballooning dependency lists
that are fetched intrigued from the internet. This is both a security problem
and directly in conflict with meson's design off one and only one version of a
project. And while rust prevents certain kinds of bugs, it doesn't prevent
design bugs or malicious code. Add a meson developer the rust community has
been incredibly hard to work with and basically hostile to every request we've
made "cargo is hour you build rust", is essentially the answer we've gotten
from them at every turn. And if you're not going to use cargo, is rust really a
win? The standard library is rather minimal "because just pull in 1000 crates".
The distro people can correct me if I'm wrong, but when librsvg went to rust it
was a nightmare, several distros went a long time without updates because of
cargo.
On the meson front cargo is incredibly hard to integrate with meson, it's
essentially like calling cmake in autotools.
Dylan
On Thu, Oct 1, 2020, at 18:35, Alyssa Rosenzweig wrote:
> Hi all,
>
> Recently I've been thinking about the potential for the Rust programming
> language in Mesa. Rust bills itself a safe system programming language
> with comparable performance to C [0], which is a naturally fit for
> graphics driver development.
>
> Mesa today is written primarily in C, a notoriously low-level language,
> with some components in C++. To handle the impedance mismatch, we've
> built up a number of abstractions in-tree, including multiple ad hoc
> code generators (GenXML, NIR algebraic passes, Bifrost disassembler). A
> higher level language can help avoid the web of metaprogramming and
> effect code that is simpler and easier to reason about. Similarly, a
> better type system can aid static analysis.
>
> Beyond abstraction, Rust's differentiating feature is the borrow checker
> to guarantee memory safety. Historically, safety has not been a primary
> concern of graphics drivers, since drivers are implemented as regular
> userspace code running in the process of the application calling them.
> Unfortunately, now that OpenGL is being exposed to untrusted code via
> WebGL, the driver does become an attack vector.
>
> For the time being, Mesa attempts to minimize memory bugs with defensive
> programming, safe in-tree abstractions (including ralloc), and static
> analysis via Coverity. Nevertheless, these are all heuristic solutions.
> Static analysis is imperfect and in our case, proprietary software.
> Ideally, the bugs we've been fixing via Coverity could be caught at
> compile-time with a free and open source toolchain.
>
> As Rust would allow exactly this, I see the primary benefit of Rust in
> verifying correctness and robustness, rather than security concerns per
> se. Indeed, safety guarantees do translate well beyond WebGL.
>
> Practically, how would Rust fit in with our existing C codebase?
> Obviously I'm not suggesting a rewrite of Mesa's more than 15 million
> lines of C. Instead, I see value in introducing Rust in targeted parts
> of the tree. In particular, I envision backend compilers written in part
> in Rust. While creating an idiomatic Rust wrapper for NIR or Gallium
> would be prohibitively costly for now, a backend compiler could be
> written in Rust with IR builders exported for use of the NIR -> backend
> IR translator written in C.
>
> This would have minimal impact on the tree. Users that are not building
> such a driver would be unaffected. For those who _are_ building Rust
> code, the Rust compiler would be added as a build-time dependency and
> the (statically linked) Rust standard library would be added as a
> runtime dependency. There is concern about the Rust compiler requiring
> LLVM as a dependency, but again this is build-time, and no worse than
> Mesa already requiring LLVM as a runtime dependency for llvmpipe and
> clover. As for the standard library, it is possible to eliminate the
> dependency as embedded Rust does, perhaps calling out to the C standard
> library via the FFI, but this is likely quixotic. I do regret the binary
> size increase, however.
>
> Implications for the build system vary. Rust prefers to be built by its
> own package manager, Cargo, which is tricky to integrate with other
> build systems. Actually, Meson has native support for Rust, invoking the
> compiler directly and skipping Cargo, as if it were C code. This support
> is not widely adopted as it prevents linking with external libraries
> ("crates", in Rust parlance), with discussions between Rust and Meson
> developers ending in a stand-still [1]. For Mesa, this might be just
> fine. Our out-of-tree run-time dependencies are minimal for the C code,
> and Rust's standard library largely avoids the need for us to maintain a
> Rust version of util/ in-tree. If this
Re: [Mesa-dev] Rust drivers in Mesa
On Fri, Oct 02, 2020 at 04:14:17PM +1000, Dave Airlie wrote:
> On Fri, 2 Oct 2020 at 15:01, Jason Ekstrand wrote:
> >
> > On Thu, Oct 1, 2020 at 10:56 PM Rob Clark wrote:
> > >
> > > On Thu, Oct 1, 2020 at 6:36 PM Alyssa Rosenzweig
> > > wrote:
> > > >
> > > > Implications for the build system vary. Rust prefers to be built by its
> > > > own package manager, Cargo, which is tricky to integrate with other
> > > > build systems. Actually, Meson has native support for Rust, invoking the
> > > > compiler directly and skipping Cargo, as if it were C code. This support
> > > > is not widely adopted as it prevents linking with external libraries
> > > > ("crates", in Rust parlance), with discussions between Rust and Meson
> > > > developers ending in a stand-still [1]. For Mesa, this might be just
> > > > fine. Our out-of-tree run-time dependencies are minimal for the C code,
> > > > and Rust's standard library largely avoids the need for us to maintain a
> > > > Rust version of util/ in-tree. If this proves impractical in the
> > > > long-term, it is possible to integrate Cargo with Meson on our end [2].
> > > >
> > >
> > > drive-by comment: for something like a gl driver that a lot of other
> > > things depend on, making it harder for us to depend on other external
> > > things is actually a good thing
> >
> > Generally, I'm a fan in concept. Generally, I feel like rust has most
> > of what I like from C++ without most of what I don't like. I
> > particularly like it's error handling mechanism, for instance. That
> > said, when it comes to things like the borrow checker, my little bit
> > of rust experience says that it doesn't actually remove bugs so much
> > as move them around.
> >
> > What's been stopping me is practicalities. Not only build systems but
> > the way in which everything in mesa is interconnected. Your
> > suggestion for building the entire back-end compiler with C-wrapped
> > helpers for NIR->compiler translation may be workable. We might also
> > be able to write NIR wrappers sufficient for reading NIR. It's not
> > actually that much API surface if all you want to do is read the data
> > structure.
> >
> > I've also thought of breaking off a component like ISL and converting
> > it. However, all the nicely contained pieces are also the ones that
> > wouldn't benefit as much. :-/
> >
>
> My feeling is the pieces that would benefit the most are the things
> touch the real world, GLSL compiler, SPIR-V handling, maybe some of
> the GL API space, but I also feel these are the messiest things to
> move to rust. I'm not sure you'd get much benefit from vulkan API
> space or touching kernel interfaces as much. Granted a backend
> compiler might be the easiest place to experiment.
>
> I think the problem is you really want to justify that whatever you
> introduce using rust is sufficiently better along any axis than c/c++
> to justify the enhanced build time requirements and maintenance,
> otherwise someone will just rewrite it in C/C++ to avoid having to
> pull a rust compiler into their distro requires.
>
> Dave.
Hi everyone,
I agree with the sentiments already expressed in this thread. Here are
some additional thoughts.
Since the original email mentioned the borrow checker in particular, my
view is that the borrow checker is one tool in Rust's safety toolkit
which is certainly useful, but I don't know if it's enough to be the
driving force for Rust adoption, at least given other adoption
constraints and trade-offs.
I have found that other tools like RAII/drop, the closely related smart
pointer types, and safe containers (vectors, strings etc.) even without
the borrow checker niceties, to be relatively more useful in preventing
memory errors. However, these are features that modern C++ also offers,
along with a seamless integration story with existing C and C++ code. I
find that Rust has an edge in thread-safety, but I am not sure if this
is a strong selling point in the context of Mesa, where the current
design seems to be well served (for now) by the traditional thread
safety patterns.
I like Rust more than C++ as a language, as I find it to be cleaner (in
part by virtue of being new) and more convenient. I especially enjoy the
better integrated functional aspects and the general high-level feel of
the language. For both C++ and Rust this comes with a caveat, though: a
reduction in the transparency of the cost of the code. I think Rust may
be a just bit more opaque in this department, since with C++ we can more
readily mentally map high-level concepts to well understood C
constructs. Still, and, more often than not, both C++ and Rust deliver
their high-level features very efficiently.
There is also the question of potentially constraining the pool of
contributors by not using one of the well established languages. Perhaps
that's not a big deal going forward given Rust's current momentum. One
could even argue that using Rust could even attract contributors,
although given
Re: [Mesa-dev] Rust drivers in Mesa
fransisco, alyssa et al: you may be interested to know that the Libre-SOC Kazan Vulkan driver, funded by NLnet, is written in rust. https://salsa.debian.org/Kazan-team/kazan the insights and analysis that you are going through is - was - pretty much exactly why we chose it (or, more accurately: jacob lifshay convinced me it was a damn good idea). it's a big project, so we would welcome opportunities for collaboration and cross-participation. l. --- crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68 ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] Rust drivers in Mesa
Hi, On Fri, 2 Oct 2020 at 08:31, Kristian Kristensen wrote: > On Fri, Oct 2, 2020 at 8:14 AM Dave Airlie wrote: >> My feeling is the pieces that would benefit the most are the things >> touch the real world, GLSL compiler, SPIR-V handling, maybe some of >> the GL API space, but I also feel these are the messiest things to >> move to rust. I'm not sure you'd get much benefit from vulkan API >> space or touching kernel interfaces as much. Granted a backend >> compiler might be the easiest place to experiment. >> >> I think the problem is you really want to justify that whatever you >> introduce using rust is sufficiently better along any axis than c/c++ >> to justify the enhanced build time requirements and maintenance, >> otherwise someone will just rewrite it in C/C++ to avoid having to >> pull a rust compiler into their distro requires. > > > Just as a data point, in ChromeOS we already build core parts of the OS with > rust. Our hypervisor (crosvm) is written in rust and using rust is generally > encouraged. That's just one distro, of course, but my sense is that rust > isn't going away - parts of GNOME are moving to rust as well - and it's a > good fit for a project like mesa. If we could agree that a rust dependency in > a gallium backend isn't any worse than a C++ backend (not talking about > subjective qualities of the languages, just build time deps), then it's not > so much a matter of "justifying" the use of rust, but a developer preference. > When we write a new compiler backend in modern C++, I don't think that > there's unanimous agreement that that's better than C along all axises, so > it's a bit of a high bar to raise for rust. > > I would also worry less about which components supposedly would theoretically > benefit the most from being written in rust. The GLSL front might be better > in rust, but rewriting it is a big investment and will introduce more bugs > and regressions before it gets to that hypothetical ideal state, for a > codebase that's otherwise in stasis. I think it makes more sense to apply the > benefits of rust to live code - code that we work on and write every day, > since that way we get to actually benefit from the features of the language. > I'm in favor of figuring out how to integrate rust code in mesa and maybe > rewrite a small helper library or binary to rust or incrementally start > converting a compiler backend or driver. Yep. Before we can land a single bit of code, we need to do a bunch of annoying things like build-system integration, FFI bridging, agreeing on conventions and style, etc etc. Trying to do that whilst also replacing the GLSL compiler or vtn is way too much; it's 100% doomed to failure, even if they're the highest-value targets on some axes. If you start with a leaf node somewhere greenfield (like a new Vulkan driver, or a new experimental backend), it makes it much easier to approach, because you've cut down on the number of moving parts (e.g. maintaining APIs in two directions), you don't have to replace something which already exists and people don't want to regress, and it also allows people who don't really care about it to ignore it until you've got all the impedance mismatches and teething pain figured out. After that you'll have a reasonable template for how you bridge the two language worlds and how you handle particular patterns, which others can replicate and improve on, than having to bridge the entire codebase in one go just to get anything working. But even then, there's a lot more to Rust than 'woo now everything is secure'. Complex state trackers like the GL API implementation could get a bunch of help from the type system; things like backends can get a lot smaller as collect/iterate/match/transform is much better supported by the language than it is in C. So the benefit in that case, apart from developer ergonomy of not having to write the same trivial patterns over and over again, is that the important parts of code becomes a lot easier to reason about. Cheers, Daniel ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] Rust drivers in Mesa
On Fri, 2 Oct 2020 at 15:01, Jason Ekstrand wrote:
>
> On Thu, Oct 1, 2020 at 10:56 PM Rob Clark wrote:
> >
> > On Thu, Oct 1, 2020 at 6:36 PM Alyssa Rosenzweig
> > wrote:
> > >
> > > Implications for the build system vary. Rust prefers to be built by its
> > > own package manager, Cargo, which is tricky to integrate with other
> > > build systems. Actually, Meson has native support for Rust, invoking the
> > > compiler directly and skipping Cargo, as if it were C code. This support
> > > is not widely adopted as it prevents linking with external libraries
> > > ("crates", in Rust parlance), with discussions between Rust and Meson
> > > developers ending in a stand-still [1]. For Mesa, this might be just
> > > fine. Our out-of-tree run-time dependencies are minimal for the C code,
> > > and Rust's standard library largely avoids the need for us to maintain a
> > > Rust version of util/ in-tree. If this proves impractical in the
> > > long-term, it is possible to integrate Cargo with Meson on our end [2].
> > >
> >
> > drive-by comment: for something like a gl driver that a lot of other
> > things depend on, making it harder for us to depend on other external
> > things is actually a good thing
>
> Generally, I'm a fan in concept. Generally, I feel like rust has most
> of what I like from C++ without most of what I don't like. I
> particularly like it's error handling mechanism, for instance. That
> said, when it comes to things like the borrow checker, my little bit
> of rust experience says that it doesn't actually remove bugs so much
> as move them around.
>
> What's been stopping me is practicalities. Not only build systems but
> the way in which everything in mesa is interconnected. Your
> suggestion for building the entire back-end compiler with C-wrapped
> helpers for NIR->compiler translation may be workable. We might also
> be able to write NIR wrappers sufficient for reading NIR. It's not
> actually that much API surface if all you want to do is read the data
> structure.
>
> I've also thought of breaking off a component like ISL and converting
> it. However, all the nicely contained pieces are also the ones that
> wouldn't benefit as much. :-/
>
My feeling is the pieces that would benefit the most are the things
touch the real world, GLSL compiler, SPIR-V handling, maybe some of
the GL API space, but I also feel these are the messiest things to
move to rust. I'm not sure you'd get much benefit from vulkan API
space or touching kernel interfaces as much. Granted a backend
compiler might be the easiest place to experiment.
I think the problem is you really want to justify that whatever you
introduce using rust is sufficiently better along any axis than c/c++
to justify the enhanced build time requirements and maintenance,
otherwise someone will just rewrite it in C/C++ to avoid having to
pull a rust compiler into their distro requires.
Dave.
___
mesa-dev mailing list
mesa-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/mesa-dev
