Re: Testing for memory safety issues with Address Sanitizer
Hi. On Mi, 2016-09-21 at 11:27 -0500, Michael Catanzaro wrote: > > Then, setting LD_PRELOAD=/usr/lib64/libasan.so worked for me. > > For some reason, Fedora doesn't seem to have a libasan-devel package, hm. On my system, I have: $ dpkg -S /usr/lib/gcc/x86_64-linux-gnu/5/libasan.so libgcc-5-dev:amd64: /usr/lib/gcc/x86_64-linux-gnu/5/libasan.so $ dpkg -S /usr/lib/x86_64-linux-gnu/libasan.so.2 libasan2:amd64: /usr/lib/x86_64-linux-gnu/libasan.so.2 > so there's no plain libasan.so. Really strange. I tried changing it to > libasan.so.3 in my jhbuildrc but actually couldn't figure out how to do > that; jhbuild is playing with LD_PRELOAD as well. :( I know little about jhbuild. I think you could try to jhbuild shell into your environment, look at $LD_PRELOAD, prefix it with the path to libasan, and then do "make" or whatever command it intended to execute. If that works, it needs to be made nicer, though. Another option altogether may be to statically link ASan with -static-asan. I haven't tried that yet, however. I also don't know how well it actually works in the particular case of ld_opened libraries that you are describing. Cheers, Tobi signature.asc Description: This is a digitally signed message part ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Testing for memory safety issues with Address Sanitizer
On Wed, 2016-09-21 at 11:27 -0500, Michael Catanzaro wrote: > For some reason, Fedora doesn't seem to have a libasan-devel package, > so there's no plain libasan.so. Really strange. I tried changing it > to libasan.so.3 in my jhbuildrc but actually couldn't figure out how > to do that; jhbuild is playing with LD_PRELOAD as well. :( Hi, these options used to work in some not so distant past for me (yeah, when playing with evolution-data-server and evolution): export CFLAGS="$CFLAGS -fsanitize=address,bounds,alignment,object-size -fno-omit-frame-pointer" export LDFLAGS="-fsanitize=address,bounds,alignment,object-size -lasan -lubsan -lpthread -ldl" export ASAN_OPTIONS=abort_on_error=1:detect_stack_use_after_return=0:detect_leaks=0:handle_segv=0:check_printf=0:detect_deadlocks=1:replace_str=1:replace_intrin=1:alloc_dealloc_mismatch=1:new_delete_type_mismatch=1:detect_container_overflow=1 I recall I was forced to use the LD_PRELOAD too. It was for cases when mixing with and without asan built code, if I recall correctly. You should teach jhbuild to add the asan/ubsan first, I'm afraid (or to concatenate the LD_PRELOAD with some shell value, similar to CFLAGS above. Bye, Milan ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Testing for memory safety issues with Address Sanitizer
On Wed, 2016-09-21 at 16:14 +0200, Tobias Mueller wrote: > Try LDFLAGS+=-lasan ==21260==ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD. > Another source of inconvenience is when compiling any library that > GCC needs itself, like gnome-calculator does. > Then, setting LD_PRELOAD=/usr/lib64/libasan.so worked for me. For some reason, Fedora doesn't seem to have a libasan-devel package, so there's no plain libasan.so. Really strange. I tried changing it to libasan.so.3 in my jhbuildrc but actually couldn't figure out how to do that; jhbuild is playing with LD_PRELOAD as well. :( Michael ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Testing for memory safety issues with Address Sanitizer
Hi. On Wed, Sep 21, 2016 at 08:17:03AM -0500, Michael Catanzaro wrote: > undefined symbol: __asan_option_detect_stack_use_after_return > > Is anything special needed for dlopened modules? Try LDFLAGS+=-lasan Another source of inconvenience is when compiling any library that GCC needs itself, like gnome-calculator does. Then, setting LD_PRELOAD=/usr/lib64/libasan.so worked for me. Cheers, Tobi ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Testing for memory safety issues with Address Sanitizer
On Mon, 2016-09-19 at 12:03 +0200, Hanno Böck wrote:
> ./configure CFLAGS="-fsanitize=address -g -fno-common
> -U_FORTIFY_SOURCE" CXXFLAGS="-fsanitize=address -g -fno-common
> -U_FORTIFY_SOURCE" LDFLAGS="-fsanitize=address -g -fno-common
> -U_FORTIFY_SOURCE"
Do you really need to pass e.g. -g to LDFLAGS?
I tried this in my jhbuildrc:
module_extra_env = {
'epiphany': {'CFLAGS': '-fno-omit-frame-pointer -fno-optimize-sibling-calls
-fno-common -fsanitize=address -U_FORTIFY_SOURCE',
'LDFLAGS': '-fno-common -fsanitize=address -U_FORTIFY_SOURCE
-Wl,-z,lazy',
'ASAN_OPTIONS': 'log_path=/home/mcatanzaro/asan-error'
}
}
But it doesn't work for Epiphany, the web extension is broken:
Error loading module
'/home/mcatanzaro/src/jhbuild/install/lib/epiphany/web-
extensions/libephywebextension.so':
/home/mcatanzaro/src/jhbuild/install/lib/epiphany/libephymisc.so:
undefined symbol: __asan_option_detect_stack_use_after_return
Is anything special needed for dlopened modules?
I'm pretty sure I had it working in the past as I did use asan once
before to fix some miscellaneous leaks, but I don't remember how I got
it to work. We should come up with some jhbuildrc foo that will work
without breaking modules.
Michael
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Testing for memory safety issues with Address Sanitizer
On Tue, 20 Sep 2016 23:02:54 -0700 Philip Withnall wrote: > It’s something which would be good to have running as part of CI, > along with the other -fsanitize options (address, thread, leak, > undefined) and things like -D_FORTIFY_SOURCE=1. I don't want to discourage you here, but from all the sanitizers asan is the one you most likely want to start with. It is really easy to use and gives you the most significant bugs for the least amount of worries. leak sanitizer is no sanitizer of its own, but part of the newer versions of asan (available since gcc 5 I think). But not everyone wants to fix every single leak and they are often harmless, so it has to be weighted whether to focus on that (it can be disabled with the env var ASAN_OPTIONS="detect_leaks=0") ubsan (undefined behavior) is also easy to use and can be combined with asan. However it also finds a lot of minor issues (mostly invalid shift operations) that are more cosmetic. msan and tsan (memory, threads) are significantly more bothersome to use. I haven't done a lot with tsan. with msan the challenge is that you need to compile *everything* with it, including all libraries. If you use c++ even parts of the libc. It also has a couple of pitfalls and you easily end up getting bogus false positives if you do anything wrong. It would of course be nice if one used all sanitizers and fixed all bugs, even the minor ones. But for practicality I'd say start by using asan. It will give you more results for less work. (also regarding fortify source: that's already the default on many linux systems, e.g. ubuntu, therefore I assume testing this is already practically happening. And it doesn't play well with the sanitizers, it's best to disable it when using asan with -U_FORTIFY_SOURCE) > Enabling asan in the jhbuild configuration by default would be one > option, with the idea that people would then develop using it by > default. However, asan interferes with Valgrind, and I think that > Valgrind is probably the more useful tool to have working in a > development environment like jhbuild. Be aware that asan basically replaces a lot of valgrind's functionality and can find bug classes that a runtime tool like valgrind can't find by design (e.g. stack buffer overreads). The only thing I'm aware what valgrind finds and asan doesn't are uninitialized memory reads (there's msan for that, but see above, harder to use). > I should reiterate that I think doing testing like this manually is a > trap. For this to be useful, it needs to be automated — that will > rely on each module having good test coverage, but that’s a good goal > to aim for anyway. I'm not familiar with gnome's testing strategies, but I'd assume it's a very different thing running a test suite versus running the actual application. In order to automate that one would need some kind of simulated gui where simulated user interactions happen. I still think it'd be valuable simply being able to run the whole gnome stack on asan and see what bugs crop up. In my experience gui applications are especially prone to use after free bugs (which may be very significant in increasing stability, as these are the bugs that you may hate as a developer - they may ocassionally cause crashes or malfunctions, but not in any reproducible way). -- Hanno Böck https://hboeck.de/ mail/jabber: ha...@hboeck.de GPG: FE73757FA60E4E21B937579FA5880072BBB51E42 pgpsQux4ZaOoc.pgp Description: OpenPGP digital signature ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Testing for memory safety issues with Address Sanitizer
Hi, On Mon, 2016-09-19 at 12:03 +0200, Hanno Böck wrote: > I recently reported a couple of bugs to GNOME-components that were > easily discovered with a feature called Address Sanitizer (asan): > https://bugzilla.gnome.org/show_bug.cgi?id=762417 (glib) > https://bugzilla.gnome.org/show_bug.cgi?id=762493 (glib) > https://bugzilla.gnome.org/show_bug.cgi?id=768441 (gnome-session) > > The first two are out of bounds reads in glib that get triggered by > running the test suite with "make check". The third is a buffer > overflow in gnome-session that gets triggered on every startup. Nice work! > Things that could be done, by increasing complexity: > > * Run the existing test suites with address sanitizer > This could be mostly automated, it's usually as simple as doing > something like: > ./configure CFLAGS="-fsanitize=address -g -fno-common > -U_FORTIFY_SOURCE" CXXFLAGS="-fsanitize=address -g -fno-common > -U_FORTIFY_SOURCE" LDFLAGS="-fsanitize=address -g -fno-common > -U_FORTIFY_SOURCE" > make > ASAN_OPTIONS="log_path=/var/log/asan/asan-error" make check It’s something which would be good to have running as part of CI, along with the other -fsanitize options (address, thread, leak, undefined) and things like -D_FORTIFY_SOURCE=1. Unfortunately, these flags are mostly mutually exclusive, which means that we’d need 5 new CI pipelines. I guess the right approach is to set up a few more parallel instances of gnome-continuous, each running with different compiler options. I’ll have a play with that tomorrow. I guess eventually it could be integrated into gnome-continuous as a new set of tree suffixes. > * Run GNOME itself and its applications with asan enabled. > Test suites are fine, but they don't replace real testing, i.e. > actually running the software. > Ideally one would built the full software stack with address sanitizer > enabled to find bugs both in the applications themselve and in > libraries. Automated tests complement manual testing, and require a lot less effort over time. (While it would be lovely in theory, in practice, very few maintainers are going to recompile each of their projects with 5 different sets of compiler options and manually test every feature in the UI before a release — that’s a huge amount of work.) Enabling asan in the jhbuild configuration by default would be one option, with the idea that people would then develop using it by default. However, asan interferes with Valgrind, and I think that Valgrind is probably the more useful tool to have working in a development environment like jhbuild. Additionally, this means consciously prioritising asan over any other -fsanitize or hardening flags (which is fine, but it is a tradeoff). > * Use Fuzzing. > This is especially interesting for everything that is parsing complex > data formats. > There has been a lot of improvement in fuzzing techniques lately and > advanced fuzzing tools like american fuzzy lop or libfuzzer can greatly > help in improving and securing code. > > I have some introductions to these things on the fuzzing project > webpage: > https://fuzzing-project.org/tutorials.html Not really related to asan, but good advice all the same. afl is pretty amazing. > I discussed this already with Tobias Mueller recently and he already > started finding a couple of issues with asan: > > https://bugzilla.gnome.org/show_bug.cgi?id=771607 > https://bugzilla.gnome.org/show_bug.cgi?id=771608 > https://bugzilla.gnome.org/show_bug.cgi?id=771609 > https://bugzilla.gnome.org/show_bug.cgi?id=771610 > https://bugzilla.gnome.org/show_bug.cgi?id=771612 > https://bugzilla.gnome.org/show_bug.cgi?id=771298 > https://bugzilla.gnome.org/show_bug.cgi?id=771289 I should reiterate that I think doing testing like this manually is a trap. For this to be useful, it needs to be automated — that will rely on each module having good test coverage, but that’s a good goal to aim for anyway. Philip signature.asc Description: This is a digitally signed message part ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Testing for memory safety issues with Address Sanitizer
Hi, I recently reported a couple of bugs to GNOME-components that were easily discovered with a feature called Address Sanitizer (asan): https://bugzilla.gnome.org/show_bug.cgi?id=762417 (glib) https://bugzilla.gnome.org/show_bug.cgi?id=762493 (glib) https://bugzilla.gnome.org/show_bug.cgi?id=768441 (gnome-session) The first two are out of bounds reads in glib that get triggered by running the test suite with "make check". The third is a buffer overflow in gnome-session that gets triggered on every startup. Given that these bugs were there I can only assume that nobody has used address sanitizer in the past to test gnome components - which is unfortunate, because it's a really powerful tool to discover hard to find memory safety issues. It's part of the common compilers (gcc,clang) and can be enabled simply by passing -fsanitize=address in CFLAGS, CXXFLAGS and LDFLAGS. Most commonly it finds out of bounds memory access (global, stack and heap, both read and write) and use after free bugs. Things that could be done, by increasing complexity: * Run the existing test suites with address sanitizer This could be mostly automated, it's usually as simple as doing something like: ./configure CFLAGS="-fsanitize=address -g -fno-common -U_FORTIFY_SOURCE" CXXFLAGS="-fsanitize=address -g -fno-common -U_FORTIFY_SOURCE" LDFLAGS="-fsanitize=address -g -fno-common -U_FORTIFY_SOURCE" make ASAN_OPTIONS="log_path=/var/log/asan/asan-error" make check The additional flags (-fno-common -U_FORTIFY_SOURCE) help avoiding false negatives from asan. The -g adds debugging info. Using the asan log_path makes sure one finds the error messages asan produces later. * Run GNOME itself and its applications with asan enabled. Test suites are fine, but they don't replace real testing, i.e. actually running the software. Ideally one would built the full software stack with address sanitizer enabled to find bugs both in the applications themselve and in libraries. I created a Gentoo-based system where almost everything is compiled with asan, this could serve as a starting point: https://wiki.gentoo.org/wiki/AddressSanitizer However of course this could also be done on any other system. * Use Fuzzing. This is especially interesting for everything that is parsing complex data formats. There has been a lot of improvement in fuzzing techniques lately and advanced fuzzing tools like american fuzzy lop or libfuzzer can greatly help in improving and securing code. I have some introductions to these things on the fuzzing project webpage: https://fuzzing-project.org/tutorials.html I discussed this already with Tobias Mueller recently and he already started finding a couple of issues with asan: https://bugzilla.gnome.org/show_bug.cgi?id=771607 https://bugzilla.gnome.org/show_bug.cgi?id=771608 https://bugzilla.gnome.org/show_bug.cgi?id=771609 https://bugzilla.gnome.org/show_bug.cgi?id=771610 https://bugzilla.gnome.org/show_bug.cgi?id=771612 https://bugzilla.gnome.org/show_bug.cgi?id=771298 https://bugzilla.gnome.org/show_bug.cgi?id=771289 -- Hanno Böck https://hboeck.de/ mail/jabber: ha...@hboeck.de GPG: FE73757FA60E4E21B937579FA5880072BBB51E42 pgpzE4V8krr6m.pgp Description: OpenPGP digital signature ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
