Re: prelink: is it worth it?
[benefit of prelink:] - almost all relocations a program has to perform are avoided. These can be very expensive when many dependencies and/or large symbol tables are involved. The latter is somewhat mitigated by the new symbol table hashing we implemented some time back but still. About 10% to 50% of the time on i686, this benefit of prelink is trashed by the randomization of the placement of [vdso], also known as linux-gate.so. If the page that the kernel chooses for [vdso] overlaps any pre-linked needed shared library, then ld-linux cannot avoid processing the relocations for that library. Often the cost snowballs as libraries that do not get their pre-linked pages are moved so that they interfere with subsequent libraries. [On x86_64 the vdso is at a special fixed address that cannot conflict.] Try this example from https://bugzilla.redhat.com/show_bug.cgi?id=162797 - for i in 0 1 2 3 4 5 6 7 8 9; do for j in 0 1 2 3 4 5 6 7 8 9; do for k in 0 1 2 3 4 5 6 7 8 9; do ldd /bin/cat done done done | grep libc | sort | uniq -c - For current Fedora 11 on i686, I see a conflict about 10% of the time, involving only ld-linux, libc, and [vdso]. This means that glibc must be dynamically relocated about 10% of the time anyway, even though glibc has been pre-linked, and even though /bin/cat is near minimal in its use of shared libraries. When a GNOME app uses 50 or more pre-linked shared libs, as claimed in another thread on this subject, then runtime conflict and expense are even more likely. If time performance matters a lot, then the kernel must co-operate when placing the vdso. A patch to FC5 was submitted and adopted some years ago to offer the choice of: no vdso, random vdso, vdso just below STACKTOP, vdso just below PT_INTERP (namely, ld-linux.so.2), vdso just below main. Maintenance suffered because exec_shield was not in the kernel mainline. None of the choices is available today. Even the remaining comment in Fedora's kernel/sysctl.c [just after int exec_shield = (10);] is incorrect. -- -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: prelink: is it worth it?
On Jul 9, 2009, at 9:45 AM, devzero2000 wrote: 2 - not checked if this problem is actual or not: prelink erases file-based capabilities https://bugzilla.redhat.com/show_bug.cgi?id=456105 Which remains 'NEW' a year after it was opened. It was recently reconfirmed by Tomas Mraz in F11. We have a number of apps that use file system capabilities, so we don't install prelink. This is frustrating since it impacts interactive app performance. joe -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: prelink: is it worth it?
Jakub Jelinek ja...@redhat.com wrote: Also, while prelink process has been fairly expensive some years ago, it is much faster these days; if you haven't installed any rpms in the last day, most of the days the cron job will just quit, if you have installed some, for libraries/binaries that don't need reprelinking it will just do a quick stat and nothing else, and even a full prelink takes just a minute or two. So what got faster, prelink or hardware? My experience, some years ago, was that a full prelink took 35-40 minutes. Since I didn't see any visible difference in performance I added turn off prelink to my list of things to do after installing Fedore. Ron -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: prelink: is it worth it?
On Fri, Jul 10, 2009 at 09:08:49AM +0100, Ron Yorston wrote: Jakub Jelinek ja...@redhat.com wrote: Also, while prelink process has been fairly expensive some years ago, it is much faster these days; if you haven't installed any rpms in the last day, most of the days the cron job will just quit, if you have installed some, for libraries/binaries that don't need reprelinking it will just do a quick stat and nothing else, and even a full prelink takes just a minute or two. So what got faster, prelink or hardware? My experience, some years ago, was that a full prelink took 35-40 minutes. Since I didn't see any visible difference in performance I added turn off prelink to my list of things to do after installing Fedore. Both. prelink's C++ optimizations used to be quite time consuming, that has been fixed, also there were changes to avoid reading all libraries and binaries, even when those haven't changed since last prelinking and don't need prelinking even now. Jakub -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: prelink: is it worth it?
On Fri, Jul 10, 2009 at 11:29:43PM +0200, yersinia wrote: Ok. But prelink it or not a requisite for ASLR or not ? In other word, besides performance is disabling prelink a security matter or not ? It is not bad to have some answer on this. ASLR happens with prelink or without. Particularly, PIEs (should be used for most of suid/network facing or otherwise security exposed programs) are always randomized, both the binary itself and all shared libraries it uses. Other than that, on prelinked system libraries are assigned random addresses whenever reprelinked, while when not prelinked, libraries are given random addresses on every exec. Non-PIE binaries have always fixed address. Jakub -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: prelink: is it worth it?
On Sat, Jul 11, 2009 at 12:26 AM, Jakub Jelinek ja...@redhat.com wrote: On Fri, Jul 10, 2009 at 11:29:43PM +0200, yersinia wrote: Ok. But prelink it or not a requisite for ASLR or not ? In other word, besides performance is disabling prelink a security matter or not ? It is not bad to have some answer on this. ASLR happens with prelink or without. Particularly, PIEs (should be used for most of suid/network facing or otherwise security exposed programs) are always randomized, both the binary itself and all shared libraries it uses. Other than that, on prelinked system libraries are assigned random addresses whenever reprelinked, while when not prelinked, libraries are given random addresses on every exec. Non-PIE binaries have always fixed address. Jakub Thank a lot for your answer: this was a delicate and very interesting issue, for me almost. Best regards -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: prelink: is it worth it?
I am curious as to this answer as well because prelink has been something that actually hurt my netbook in performance so I nuked it. -Adam -- http://maxamillion.googlepages.com - () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: prelink: is it worth it?
On Thu, Jul 9, 2009 at 4:36 PM, Adam Miller maxamill...@gmail.com wrote: I am curious as to this answer as well because prelink has been something that actually hurt my netbook in performance so I nuked it. There are also other two big problem, imho, now, with prelink support: 1 - it render impossibile to do a centralizzated integrity checker (with as example rfc.sf.net ): very very bad 2 - not checked if this problem is actual or not: prelink erases file-based capabilities https://bugzilla.redhat.com/show_bug.cgi?id=456105 If actual it render new rpm possibility in this area pointless -Adam -- http://maxamillion.googlepages.com - () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: prelink: is it worth it?
On Thu, 9 Jul 2009 10:32:01 -0400, Matthew Miller mat...@mattdm.org wrote: Apparently there was some fun with prelink breaking everything in rawhide recently: https://bugzilla.redhat.com/show_bug.cgi?id=509655. I didn't notice, because like Pete Zaitcev says in the comments, removing prelink is one of the first things I do. I meant that it was the first thing I did when the root cause became known. This is not the same thing as removing mlocate upon install that everyone do. That said, you raise an interesting question. -- Pete -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: prelink: is it worth it?
On Thu, Jul 09, 2009 at 08:58:08AM -0600, Pete Zaitcev wrote: Apparently there was some fun with prelink breaking everything in rawhide recently: https://bugzilla.redhat.com/show_bug.cgi?id=509655. I didn't notice, because like Pete Zaitcev says in the comments, removing prelink is one of the first things I do. I meant that it was the first thing I did when the root cause became known. This is not the same thing as removing mlocate upon install that everyone do. Sorry -- didn't mean to put words into your mouth. -- Matthew Miller mat...@mattdm.org Senior Systems Architect Cyberinfrastructure Labs Computing Information Technology Harvard School of Engineering Applied Sciences -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: prelink: is it worth it?
On Thu, Jul 09, 2009 at 10:32:01AM -0400, Matthew Miller wrote: Apparently there was some fun with prelink breaking everything in rawhide recently: https://bugzilla.redhat.com/show_bug.cgi?id=509655. I didn't notice, because like Pete Zaitcev says in the comments, removing prelink is one of the first things I do. I see it as adding unnecessary complexity and fragility, and it makes forensic verification difficult. Binaries can't be verified without being modified, which is far from ideal. And the error about dependencies having changed since prelinking is disturbingly frequent. On the other hand, smart people have worked on it. It's very likely that those smart people know things I don't. I can't find any good numbers anywhere demonstrating the concrete benefits provided by prelink. Is there data out there? Pretty charts and graphs would be nice. The only things I've been able to find are old and not very impressive: http://crast.us/james/articles/prelink.php http://smackerelofopinion.blogspot.com/2009/06/does-prelinking-speed-up-boot-times.html http://reallylongword.org/prelink-2/ Say on x86-64 F11 (with /etc/prelink.conf.d/nss-prelink.conf removed): LD_DEBUG=statistics /usr/lib64/openoffice.org3/program/swriter.bin 11799: 11799: runtime linker statistics: 11799: total startup time in dynamic loader: 18632372 clock cycles 11799: time needed for relocation: 2975643 clock cycles (15.9%) 11799: number of relocations: 0 11799: number of relocations from cache: 1527 11799: number of relative relocations: 0 11799:time needed to load objects: 13190177 clock cycles (70.7%) LD_USE_LOAD_BIAS=0 LD_DEBUG=statistics /usr/lib64/openoffice.org3/program/swriter.bin 11817: 11817: runtime linker statistics: 11817: total startup time in dynamic loader: 145813250 clock cycles 11817: time needed for relocation: 129375884 clock cycles (88.7%) 11817: number of relocations: 42207 11817: number of relocations from cache: 126013 11817: number of relative relocations: 149959 11817:time needed to load objects: 15248893 clock cycles (10.4%) The former is how long it takes before swriter.bin starts up when prelinked, the latter tells it not to use prelink information and do normal relocation. Try this on any other GUI app that links against dozens of shared libraries. Even on binaries that link against only very few libraries, but are executed often it makes a difference, say consider configure scripts or libtool that exec hundreds to thousands small programs. And time isn't the only saving, another is that fewer pages are dirtied. The rawhide prelink issue is just that we've added a new ELF extension to the toolchain (binutils, glibc) and prelink had to be changed to handle it. Jakub -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: prelink: is it worth it?
On Thu, Jul 9, 2009 at 4:55 PM, Ulrich Drepper drep...@redhat.com wrote: Adam Miller wrote: I am curious as to this answer as well because prelink has been something that actually hurt my netbook in performance so I nuked it. Performance only ever can be hurt because prelink runs periodically to prelink newly installed code or re-randomize to increase security. prelink has two benefits: - almost all relocations a program has to perform are avoided. These can be very expensive when many dependencies and/or large symbol tables are involved. The latter is somewhat mitigated by the new symbol table hashing we implemented some time back but still. - as a side effect of the first point some pages in the loaded binary are not copied-on-write. This can obviously have good effects on systems with little memory (netbooks). Just run your own tests on apps with many relocations and dependencies. FF, OO.org, most GUI apps come into mind. Use LD_DEBUG=statistics some/program to compare numbers. Run it with and without prelink (but always with hot disk cache to be fair). The number of cycles for total startup is representative of the win. Note, also small but frequently used apps benefit. I run gcc etc a lot and like every single saved cycle. But something one have to pay a security prize on not disabling it : it render impossible to have a centralizzated security integrity management (e.g. rfc.sf.net for example) or one have to skip from check the prelink binary. Very bad i think. -- ➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖ -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: prelink: is it worth it?
On Thu, Jul 09, 2009 at 07:55:38AM -0700, Ulrich Drepper wrote: hot disk cache to be fair). The number of cycles for total startup is representative of the win. I'm not sure that's the case. If I can get a 50% speed up to a program's startup times, that sounds great, but if I then leave that program running for days on end, I haven't actually won very much at all -- but I still pay the price continuously. (That price being: fragility, verifiability, and of course the prelinking activity itself.) Note, also small but frequently used apps benefit. I run gcc etc a lot and like every single saved cycle. Right, this actually seems more significant to me, even though it may be less important in terms of marketing to new Linux desktop users. But saving every last possible cycle is only one concern of many, particularly when deciding on a default configuration. -- Matthew Miller mat...@mattdm.org Senior Systems Architect Cyberinfrastructure Labs Computing Information Technology Harvard School of Engineering Applied Sciences -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: prelink: is it worth it?
On Thu, 9 Jul 2009, Matthew Miller wrote: On Thu, Jul 09, 2009 at 07:55:38AM -0700, Ulrich Drepper wrote: hot disk cache to be fair). The number of cycles for total startup is representative of the win. I'm not sure that's the case. If I can get a 50% speed up to a program's startup times, that sounds great, but if I then leave that program running for days on end, I haven't actually won very much at all -- but I still pay the price continuously. (That price being: fragility, verifiability, and of course the prelinking activity itself.) Note, also small but frequently used apps benefit. I run gcc etc a lot and like every single saved cycle. Right, this actually seems more significant to me, even though it may be less important in terms of marketing to new Linux desktop users. But saving every last possible cycle is only one concern of many, particularly when deciding on a default configuration. Is there a middle road which involves having prelink target specific pkgs/binaries and avoid others? -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: prelink: is it worth it?
On Thu, Jul 09, 2009 at 05:07:05PM +0200, yersinia wrote: But something one have to pay a security prize on not disabling it : it render impossible to have a centralizzated security integrity management (e.g. rfc.sf.net for example) or one have to skip from check the prelink binary. Very bad i think. That's what prelink -y is for, it verifies the binary would prelink from unprelinked state to bitwise same file and gives you the bits before prelinking, which you can use for verification. rpm -V uses this, why can't other security integrity apps do the same? Jakub -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: prelink: is it worth it?
On Thu, Jul 9, 2009 at 5:12 PM, Jakub Jelinek ja...@redhat.com wrote: On Thu, Jul 09, 2009 at 05:07:05PM +0200, yersinia wrote: But something one have to pay a security prize on not disabling it : it render impossible to have a centralizzated security integrity management (e.g. rfc.sf.net for example) or one have to skip from check the prelink binary. Very bad i think. That's what prelink -y is for, it verifies the binary would prelink from unprelinked state to bitwise same file and gives you the bits before prelinking, which you can use for verification. rpm -V uses this, why can't other security integrity apps do the same? Yes I know that rpm do this. But other centralizzated integrity checker, perhaps for portability between posix platform, at max permit to skip the check - OSSSEC for example iirc do this - on prelinked binary. regards Jakub -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: prelink: is it worth it?
On Thu, Jul 09, 2009 at 11:08:20AM -0400, Matthew Miller wrote: On Thu, Jul 09, 2009 at 07:55:38AM -0700, Ulrich Drepper wrote: hot disk cache to be fair). The number of cycles for total startup is representative of the win. I'm not sure that's the case. If I can get a 50% speed up to a program's startup times, that sounds great, but if I then leave that program running for days on end, I haven't actually won very much at all -- but I still pay the price continuously. (That price being: fragility, verifiability, and of course the prelinking activity itself.) Most of GNOME apps link against 50+ libraries, however small they are, many of them are executed frequently. Also, while prelink process has been fairly expensive some years ago, it is much faster these days; if you haven't installed any rpms in the last day, most of the days the cron job will just quit, if you have installed some, for libraries/binaries that don't need reprelinking it will just do a quick stat and nothing else, and even a full prelink takes just a minute or two. Jakub -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: prelink: is it worth it?
On Thu July 9 2009, yersinia wrote: But something one have to pay a security prize on not disabling it : it render impossible to have a centralizzated security integrity management (e.g. rfc.sf.net for example) or one have to skip from check the prelink binary. Very bad i think. You pay a security prize if you disable prelink, because it also performs address space randomization: http://lwn.net/Articles/190139/ Btw. you can also patch the remote integrity checker to use prelink to either get a checksum of the perlinked binary or undo the prelinking before checking it. Regards Till signature.asc Description: This is a digitally signed message part. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: prelink: is it worth it?
On Thu, Jul 09, 2009 at 04:45:55PM +0200, devzero2000 wrote: 2 - not checked if this problem is actual or not: prelink erases file-based capabilities And not just that - it nukes other stuff too. Old-style OCaml bytecode programs[1] and there was some MinGW binary problem too, can't find the BZ for that right now. I think the mistake was to change binaries directly. It should just store additional hints about binaries in a /var/ directory somewhere. Rich. [1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=460730 -- Richard Jones, Emerging Technologies, Red Hat http://et.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 75 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: prelink: is it worth it?
Till Maas wrote: I fail to understand what is nonsense, since you agree that prelink performs randomization. If the binary is not prelinked, the kernel randomizes the address spaces at runtime, which is more secure than the prelink-time randomization. Kevin Kofler -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: prelink: is it worth it?
On Thursday 09 July 2009 10:45:55 am devzero2000 wrote: There are also other two big problem, imho, now, with prelink support: 1 - it render impossibile to do a centralizzated integrity checker (with as example rfc.sf.net ): very very bad The aide program in rawhide is prelink friendly. So there are integrity checkers that can be used. As for security, prelink stirring around address space randomization is good for security. For example, this hack needed prelink not to have run to get the exploit reliable: http://invisiblethingslab.com/pub/xenfb-adventures-10.pdf There are more examples like this. -Steve -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: prelink: is it worth it?
On Thu, Jul 9, 2009 at 8:19 PM, Steve Grubb sgr...@redhat.com wrote: On Thursday 09 July 2009 10:45:55 am devzero2000 wrote: There are also other two big problem, imho, now, with prelink support: 1 - it render impossibile to do a centralizzated integrity checker (with as example rfc.sf.net ): very very bad The aide program in rawhide is prelink friendly. So there are integrity checkers that can be used. As for security, prelink stirring around address space randomization is good for security. For example, this hack needed prelink not to have run to get the exploit reliable: http://invisiblethingslab.com/pub/xenfb-adventures-10.pdf There are more examples like this. i know this ref. Tell me something other. I follow only 15 mailing list on these subjects. Anyway if prelink is a good thing for ASLR IT MUST BE DOCUMENTATED BETTER. I am sure anyone agreed on this. regards -Steve -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: prelink: is it worth it?
On Thu, 2009-07-09 at 17:06 +0200, Jakub Jelinek wrote: LD_DEBUG=statistics /usr/lib64/openoffice.org3/program/swriter.bin 11799: 11799: runtime linker statistics: 11799: total startup time in dynamic loader: 18632372 clock cycles 11799: time needed for relocation: 2975643 clock cycles (15.9%) 11799:number of relocations: 0 11799: number of relocations from cache: 1527 11799: number of relative relocations: 0 11799: time needed to load objects: 13190177 clock cycles (70.7%) LD_USE_LOAD_BIAS=0 LD_DEBUG=statistics /usr/lib64/openoffice.org3/program/swriter.bin 11817: 11817: runtime linker statistics: 11817: total startup time in dynamic loader: 145813250 clock cycles 11817: time needed for relocation: 129375884 clock cycles (88.7%) 11817:number of relocations: 42207 11817: number of relocations from cache: 126013 11817: number of relative relocations: 149959 11817: time needed to load objects: 15248893 clock cycles (10.4%) The former is how long it takes before swriter.bin starts up when prelinked, the latter tells it not to use prelink information and do normal relocation. Indeed, and Fedora OOo has a prelink-friendly set of launchers that explicitly link against the dsos that OOo will definitely dlopen on the various appropriate launch paths, and it's reassuring to see smaller numbers spat out at the end, but do we have actual numbers that measure the real-world difference it makes, or even a reliable mechanism that can be used to reproducibly and consistently measure the real-world difference that *any* given change makes. e.g. I've been totally unable to consistently measure a difference between compiling OOo with -fstrict-aliasing and the current -fno-strict-aliasing. And any other number of other proposed optimizations are massively difficult to measure reliably. C. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
