Re: LLVM Image Activator
On Sun, Jan 13, 2013 at 12:24:35PM -0800, John-Mark Gurney wrote: Nathan Whitehorn wrote this message on Sun, Jan 13, 2013 at 10:14 -0800: On 01/13/13 09:13, Konstantin Belousov wrote: On Sun, Jan 13, 2013 at 08:21:37AM -0800, Nathan Whitehorn wrote: On 01/13/13 05:20, Konstantin Belousov wrote: On Sun, Jan 13, 2013 at 12:41:09PM +0100, Ed Schouten wrote: Hi Kostik, 2013/1/7 Konstantin Belousov kostik...@gmail.com: I still do remember the buzz about the binary format 0xCAFEBABE, which AFAIR gained image activator support on several OSes, to be garbage collected. Maybe it would then be a good idea then to add some kind of general purpose remapping imgact? Example: /etc/imgacttab: cafebabe /usr/local/bin/java cffaedfe /usr/local/bin/osx_emulator 4243c0de /usr/bin/lli That way we still give people the freedom to play around with mapping their own executable formats, but don't need to maintain a bunch of imgacts. A generic module that could be somewhat customized at runtime to map offset+signature into the shebang path could be a possibility indeed. I strongly prefer to have it as module and not enabled by default. Asking Nathan for writing the thing is too much, IMHO, esp. in the response to the 50-lines hack. I think this is a good idea, since it both prevents a profusion of similar activators and works nicely in jails and similar environments. I probably won't write it quickly, but it should not take more than about 50 lines, so I can't imagine it will be that bad. There are some complications with this kind of design from the things in the XXX comment in imgact_llvm.c about handling argv[0] that I need to think some more about. Great. I do not believe in the 50 lines, but I am happy that you want to work this out. Why are you opposed to having it there by default? I think it's actually quite important that it be there by default. Having it not standard would be fine, but it should at least be in GENERIC. There are minimal security risks since it just munges begin_argv and doesn't even load the executable and it's little enough code that there should not be any kernel bloat to speak of. If things like this aren't enabled by default, no one can depend on them being there, no one will use it, and the point is entirely lost. All image activators demonstrated a constant stream of security holes. Even our ELF activator, and I was guilty there too. I definitely do not fight over the inclusion of the proposed activator into GENERIC, but do insist on the config option + module. OK, that sounds like a plan then. I'll try to code up something configurable in the next couple weeks, unless someone else beats me to it. I'll point out that file already has the magic (pun intended) that we are looking for, though I do realize that the code might be a bit much to import.. As someone who recently stuffed libmagic into a very constrained sandbox environment, I can safely assert that you don't want to go there. The code isn't written in a way that would make this easy and I definitely wouldn't want it in the kernel. -- Brooks pgp07tnkSUC1K.pgp Description: PGP signature
Re: LLVM Image Activator
Hi Kostik, 2013/1/7 Konstantin Belousov kostik...@gmail.com: I still do remember the buzz about the binary format 0xCAFEBABE, which AFAIR gained image activator support on several OSes, to be garbage collected. Maybe it would then be a good idea then to add some kind of general purpose remapping imgact? Example: /etc/imgacttab: cafebabe /usr/local/bin/java cffaedfe /usr/local/bin/osx_emulator 4243c0de /usr/bin/lli That way we still give people the freedom to play around with mapping their own executable formats, but don't need to maintain a bunch of imgacts. -- Ed Schouten e...@80386.nl ___ freebsd-toolchain@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain To unsubscribe, send any mail to freebsd-toolchain-unsubscr...@freebsd.org
Re: LLVM Image Activator
On Sun, Jan 13, 2013 at 12:41:09PM +0100, Ed Schouten wrote: Hi Kostik, 2013/1/7 Konstantin Belousov kostik...@gmail.com: I still do remember the buzz about the binary format 0xCAFEBABE, which AFAIR gained image activator support on several OSes, to be garbage collected. Maybe it would then be a good idea then to add some kind of general purpose remapping imgact? Example: /etc/imgacttab: cafebabe /usr/local/bin/java cffaedfe /usr/local/bin/osx_emulator 4243c0de /usr/bin/lli That way we still give people the freedom to play around with mapping their own executable formats, but don't need to maintain a bunch of imgacts. A generic module that could be somewhat customized at runtime to map offset+signature into the shebang path could be a possibility indeed. I strongly prefer to have it as module and not enabled by default. Asking Nathan for writing the thing is too much, IMHO, esp. in the response to the 50-lines hack. pgpczGgp_E_rw.pgp Description: PGP signature
Re: LLVM Image Activator
On 01/13/13 05:20, Konstantin Belousov wrote: On Sun, Jan 13, 2013 at 12:41:09PM +0100, Ed Schouten wrote: Hi Kostik, 2013/1/7 Konstantin Belousov kostik...@gmail.com: I still do remember the buzz about the binary format 0xCAFEBABE, which AFAIR gained image activator support on several OSes, to be garbage collected. Maybe it would then be a good idea then to add some kind of general purpose remapping imgact? Example: /etc/imgacttab: cafebabe /usr/local/bin/java cffaedfe /usr/local/bin/osx_emulator 4243c0de /usr/bin/lli That way we still give people the freedom to play around with mapping their own executable formats, but don't need to maintain a bunch of imgacts. A generic module that could be somewhat customized at runtime to map offset+signature into the shebang path could be a possibility indeed. I strongly prefer to have it as module and not enabled by default. Asking Nathan for writing the thing is too much, IMHO, esp. in the response to the 50-lines hack. I think this is a good idea, since it both prevents a profusion of similar activators and works nicely in jails and similar environments. I probably won't write it quickly, but it should not take more than about 50 lines, so I can't imagine it will be that bad. There are some complications with this kind of design from the things in the XXX comment in imgact_llvm.c about handling argv[0] that I need to think some more about. Why are you opposed to having it there by default? I think it's actually quite important that it be there by default. Having it not standard would be fine, but it should at least be in GENERIC. There are minimal security risks since it just munges begin_argv and doesn't even load the executable and it's little enough code that there should not be any kernel bloat to speak of. If things like this aren't enabled by default, no one can depend on them being there, no one will use it, and the point is entirely lost. -Nathan ___ freebsd-toolchain@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain To unsubscribe, send any mail to freebsd-toolchain-unsubscr...@freebsd.org
Re: LLVM Image Activator
Nathan Whitehorn wrote this message on Sun, Jan 13, 2013 at 10:14 -0800: On 01/13/13 09:13, Konstantin Belousov wrote: On Sun, Jan 13, 2013 at 08:21:37AM -0800, Nathan Whitehorn wrote: On 01/13/13 05:20, Konstantin Belousov wrote: On Sun, Jan 13, 2013 at 12:41:09PM +0100, Ed Schouten wrote: Hi Kostik, 2013/1/7 Konstantin Belousov kostik...@gmail.com: I still do remember the buzz about the binary format 0xCAFEBABE, which AFAIR gained image activator support on several OSes, to be garbage collected. Maybe it would then be a good idea then to add some kind of general purpose remapping imgact? Example: /etc/imgacttab: cafebabe /usr/local/bin/java cffaedfe /usr/local/bin/osx_emulator 4243c0de /usr/bin/lli That way we still give people the freedom to play around with mapping their own executable formats, but don't need to maintain a bunch of imgacts. A generic module that could be somewhat customized at runtime to map offset+signature into the shebang path could be a possibility indeed. I strongly prefer to have it as module and not enabled by default. Asking Nathan for writing the thing is too much, IMHO, esp. in the response to the 50-lines hack. I think this is a good idea, since it both prevents a profusion of similar activators and works nicely in jails and similar environments. I probably won't write it quickly, but it should not take more than about 50 lines, so I can't imagine it will be that bad. There are some complications with this kind of design from the things in the XXX comment in imgact_llvm.c about handling argv[0] that I need to think some more about. Great. I do not believe in the 50 lines, but I am happy that you want to work this out. Why are you opposed to having it there by default? I think it's actually quite important that it be there by default. Having it not standard would be fine, but it should at least be in GENERIC. There are minimal security risks since it just munges begin_argv and doesn't even load the executable and it's little enough code that there should not be any kernel bloat to speak of. If things like this aren't enabled by default, no one can depend on them being there, no one will use it, and the point is entirely lost. All image activators demonstrated a constant stream of security holes. Even our ELF activator, and I was guilty there too. I definitely do not fight over the inclusion of the proposed activator into GENERIC, but do insist on the config option + module. OK, that sounds like a plan then. I'll try to code up something configurable in the next couple weeks, unless someone else beats me to it. I'll point out that file already has the magic (pun intended) that we are looking for, though I do realize that the code might be a bit much to import.. -- John-Mark Gurney Voice: +1 415 225 5579 All that I will do, has been done, All that I have, has not. ___ freebsd-toolchain@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain To unsubscribe, send any mail to freebsd-toolchain-unsubscr...@freebsd.org
Re: LLVM Image Activator
On Sunday, January 06, 2013 01:02:21 PM Nathan Whitehorn wrote: Having LLVM/clang in the base system lets us do some interesting things that we couldn't do with GCC. One is that LLVM ships with a JIT for LLVM IR as well as components of a toolchain for it (this is what Google's pNACL uses) and that you can end up producing binary files that are in IR instead of native code. The IR isn't really cross-platform, but does let you do CPU-specific optimizations when executed by the JIT, etc. The attached patch causes the LLVM JIT (lli) to be built by default (adding ~20 seconds to buildworld on my five-year-old laptop) and adds a kernel image activator that invokes it when passed LLVM bitcode files. It's not completely finished (see the XXX comment in the middle), but it does work, as follows: $ clang -emit-llvm -c -o hw.ll hw.c $ file hw.ll hw.ll: LLVM bitcode $ lli hw.ll Hello world! $ chmod a+x hw.ll $ ./hw.ll Hello world! $ Is there any interest in having features like this? It seems like this could provides some interesting possibilities for us and nice integration from having imported clang into base. -Nathan This sounds neat indeed. Does the IR format provide any sort of notation for encoding the path to the interpreter (similar to ELF)? If not, you might want to at least make the path to 'lli' be configurable via a tunable and/or sysctl (e.g. if using a newer version of clang in /usr/local). -- John Baldwin ___ freebsd-toolchain@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain To unsubscribe, send any mail to freebsd-toolchain-unsubscr...@freebsd.org
Re: LLVM Image Activator
Hi, Just my 2c On Sunday, 6 January 2013 20:02:21 Nathan Whitehorn wrote: Having LLVM/clang in the base system lets us do some interesting things that we couldn't do with GCC. One is that LLVM ships with a JIT for LLVM IR as well as components of a toolchain for it (this is what Google's pNACL uses) and that you can end up producing binary files that are in IR instead of native code. The IR isn't really cross-platform, but does let you do CPU-specific optimizations when executed by the JIT, etc. The attached patch causes the LLVM JIT (lli) to be built by default (adding ~20 seconds to buildworld on my five-year-old laptop) and adds a kernel image activator that invokes it when passed LLVM bitcode files. It's not completely finished (see the XXX comment in the middle), but it does work, as follows: $ clang -emit-llvm -c -o hw.ll hw.c $ file hw.ll hw.ll: LLVM bitcode $ lli hw.ll Hello world! $ chmod a+x hw.ll $ ./hw.ll Hello world! $ Is there any interest in having features like this? It seems like this could provides some interesting possibilities for us and nice integration from having imported clang into base. Would it be possible to have this as a module (and thus in ports)? Or, perhaps, change the sources such these things could be loaded as a module... Also, with modification to LLVM, wouldn't the existing '#!' mechanism work (aka '#!/usr/bin/lli') at the beginning of the file? On an aside. Could you imagine shipping a 'x86' llvm-byte code and get the jit (with caching), thus your computer will get full support (speed) from the binaries and the binaries will work on all 'x86' related architectures. (My thinking is for things like i386 vs pentium4). Regards signature.asc Description: This is a digitally signed message part.
Re: LLVM Image Activator
On 01/07/13 08:25, David Naylor wrote: Hi, Just my 2c On Sunday, 6 January 2013 20:02:21 Nathan Whitehorn wrote: Having LLVM/clang in the base system lets us do some interesting things that we couldn't do with GCC. One is that LLVM ships with a JIT for LLVM IR as well as components of a toolchain for it (this is what Google's pNACL uses) and that you can end up producing binary files that are in IR instead of native code. The IR isn't really cross-platform, but does let you do CPU-specific optimizations when executed by the JIT, etc. The attached patch causes the LLVM JIT (lli) to be built by default (adding ~20 seconds to buildworld on my five-year-old laptop) and adds a kernel image activator that invokes it when passed LLVM bitcode files. It's not completely finished (see the XXX comment in the middle), but it does work, as follows: $ clang -emit-llvm -c -o hw.ll hw.c $ file hw.ll hw.ll: LLVM bitcode $ lli hw.ll Hello world! $ chmod a+x hw.ll $ ./hw.ll Hello world! $ Is there any interest in having features like this? It seems like this could provides some interesting possibilities for us and nice integration from having imported clang into base. Would it be possible to have this as a module (and thus in ports)? Or, perhaps, change the sources such these things could be loaded as a module... We do support that, but I'm not really sure what would be gained. We have LLVM in base; it seems a shame not to use it. Also, with modification to LLVM, wouldn't the existing '#!' mechanism work (aka '#!/usr/bin/lli') at the beginning of the file? In principle, yes, but we'd have to coordinate that with a lot of players since a number of tools operate on LLVM bitcode. The other obstacle is that LLVM bitcode is a giant binary blob, not text, so the standard #! seems a little out of place. On an aside. Could you imagine shipping a 'x86' llvm-byte code and get the jit (with caching), thus your computer will get full support (speed) from the binaries and the binaries will work on all 'x86' related architectures. (My thinking is for things like i386 vs pentium4). Yes, this is exactly the kind of thing I had in mind. -Nathan ___ freebsd-toolchain@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain To unsubscribe, send any mail to freebsd-toolchain-unsubscr...@freebsd.org
Re: LLVM Image Activator
On Mon, Jan 07, 2013 at 09:36:38AM -0500, John Baldwin wrote: On Sunday, January 06, 2013 01:02:21 PM Nathan Whitehorn wrote: Having LLVM/clang in the base system lets us do some interesting things that we couldn't do with GCC. One is that LLVM ships with a JIT for LLVM IR as well as components of a toolchain for it (this is what Google's pNACL uses) and that you can end up producing binary files that are in IR instead of native code. The IR isn't really cross-platform, but does let you do CPU-specific optimizations when executed by the JIT, etc. The attached patch causes the LLVM JIT (lli) to be built by default (adding ~20 seconds to buildworld on my five-year-old laptop) and adds a kernel image activator that invokes it when passed LLVM bitcode files. It's not completely finished (see the XXX comment in the middle), but it does work, as follows: $ clang -emit-llvm -c -o hw.ll hw.c $ file hw.ll hw.ll: LLVM bitcode $ lli hw.ll Hello world! $ chmod a+x hw.ll $ ./hw.ll Hello world! $ Is there any interest in having features like this? It seems like this could provides some interesting possibilities for us and nice integration from having imported clang into base. -Nathan This sounds neat indeed. Does the IR format provide any sort of notation for encoding the path to the interpreter (similar to ELF)? If not, you might want to at least make the path to 'lli' be configurable via a tunable and/or sysctl (e.g. if using a newer version of clang in /usr/local). I do not like one-purpose write-once never-used image activators. I still do remember the buzz about the binary format 0xCAFEBABE, which AFAIR gained image activator support on several OSes, to be garbage collected. For one thing, such hacks are very system-specific and thus are never used, causing quick bit rot. Second, more important, is that parsing of yet another format opens an attack vector on kernel, and due to the nature of the activators, on the user space. Look at the PE image activator history, including at least one SA, to see what I mean. And PE is real image format instead of hack. Even for ELF, there were recent panic-level bugs. Note that several language runtimes are happy to use the shebang for bytecode. For examples, look at the ocaml and lisps. If adding despite these arguments, please do not make the thing 'standard'. Provide the option to compile it in, and may be make it a module. pgpdPIA0aUmpv.pgp Description: PGP signature
