Re: [gsoc] syscall/libc fuzzer proposal
Mateusz, Now that NetBSD has dtrace (FBT) for the kernel, have you thought about how you might use write mode in dtrace to simulate failure? Is there value in introducing specific dtrace probes (once we have SDT probes) to support fuzzing? Are further changes required, such as allowing longer sleeps, so that it is easier to use fuzz and delays to cause unexpected behaviour to multi-threaded applications? Is there scope within the existing dtrace tool for creating a dfuzz tool? ... I could go on... Darren
Re: [gsoc] syscall/libc fuzzer proposal
Hello,
On Sat, Mar 20, 2010 at 05:32:28PM +0200, Mateusz Kocielski wrote:
(Eg.
int foo() { char d[10]; int i; i = 5; return d[i] }
will be translate into:
int foo() { char d[10]; int i; int _x; i = 5; _x = i; if ( _x 0 ||
_x 9 ) ERROR();
return d[_x]; }
)
I've seen some tools that are able to generate bound-checking code for
C. There is/was patches for gcc to enable bound-checking and there is
some kind program transformator. I do not have the links or names at
hand, but I would google the web for such tools, as there exists open
source tools that already do such kinds of transformation. Maybe they
are outdated or may not compile easily, but they are existent. I think
ccured is one of the tools capable of doing such transformations.
Btw. someone using ccured on NetBSD? I was very unlucky in compiling it
and did not spot it in pkgsrc.
Regards
Matthias Kretschmer
Re: [gsoc] syscall/libc fuzzer proposal
On Sat, Mar 20, 2010 at 08:53:12PM +0200, Mateusz Kocielski wrote:
2010/3/20 Thor Lancelot Simon t...@panix.com:
What is the benefit of this when compared to existing static-analysis
tools such as Coverity Scan, splint, or the Clang static analyzer? Will
this cover any cases they don't? If so, which ones?
Undecidability is the limit for static-analysis. Consider following program:
*b*
$ cat splint.c
#include stdio.h
#include stdlib.h
#include string.h
int main(int argc, char **argv)
{
int i;
char blah[10];
memset(blah, 0, sizeof(blah));
if ( argc 1 )
i = atoi(argv[1]);
else
i = 0;
printf(%d - %c\n, i, blah[i]);
return 0;
}
$ splint splint.c
Splint 3.1.2 --- 07 Sep 2009
Finished checking --- no warnings
$ ./splint 99
99 - 1
*b*
Static analysis used in splint is not strong enough to uncover bug.
For sure there exists static analysis which is able to find this bug,
but it might be a good subject for PhD thesis. :)
For what it's worth, here is what devel/frama-c has to say about this
sample, with frama-c -val splint.c:
splint.c:14:[kernel] warning: accessing out of bounds index. assert ((0 ≤ i) ∧
(i 10));
splint.c:14:[kernel] warning: accessing uninitialized left-value blah[i]:
assert(TODO)
splint.c:14:[kernel] warning: completely unspecified value in {{
blah - [0..72],0%8 ;}} (size:8). This path is assumed to
be dead.
However, the results are much more difficult to analyse with examples
from our source tree. However, with ASCL annotations, it could be very
interesting to use it to a larger scale.
regards,
antoine
pgp0u5cncF115.pgp
Description: PGP signature
Re: [gsoc] syscall/libc fuzzer proposal
On Sat, Mar 20, 2010 at 3:32 PM, Mateusz Kocielski m.kociel...@gmail.com wrote: Hi, My proposal is to write syscall/libc fuzzer, i've written down my thoughts, please let me know what you think about it. I would appreciate your feedback. I'm open for any ideas or comments. 1. What is fuzzing? Fuzz testing is a software testing technique that provides random/invalid data to the program and then checks if the program failed or something unexpected happened. Hello Mateusz, Your proposal seems very interesting. But given that this is about testing and NetBSD, have you considered how does your proposal relate to our testing infrastructure, ATF? Would be interesting to think about it and improve the description of your project! If you have questions about ATF, feel free to ask me off-list. Thanks! -- Julio Merino
Re: [gsoc] syscall/libc fuzzer proposal
On Sat, 20 Mar 2010, Mateusz Kocielski wrote: ...your ideas? Reminds me of 1991's crashme: http://crashme.codeplex.com/ The idea sounds more like a research project to me... - Hubert
Re: [gsoc] syscall/libc fuzzer proposal
2010/3/20 Thor Lancelot Simon t...@panix.com:
What is the benefit of this when compared to existing static-analysis
tools such as Coverity Scan, splint, or the Clang static analyzer? Will
this cover any cases they don't? If so, which ones?
Undecidability is the limit for static-analysis. Consider following program:
*b*
$ cat splint.c
#include stdio.h
#include stdlib.h
#include string.h
int main(int argc, char **argv)
{
int i;
char blah[10];
memset(blah, 0, sizeof(blah));
if ( argc 1 )
i = atoi(argv[1]);
else
i = 0;
printf(%d - %c\n, i, blah[i]);
return 0;
}
$ splint splint.c
Splint 3.1.2 --- 07 Sep 2009
Finished checking --- no warnings
$ ./splint 99
99 - 1
*b*
Static analysis used in splint is not strong enough to uncover bug.
For sure there exists static analysis which is able to find this bug,
but it might be a good subject for PhD thesis. :)
We will put splint.c through our translator and receive something like
this (assert is just an example, additional lines has got //T
comments):
*b*
$ cat translated.c
#include stdio.h
#include stdlib.h
#include string.h
#include assert.h
int main(int argc, char **argv)
{
int j; // T
int i;
char blah[10];
memset(blah, 0, sizeof(blah));
if ( argc 1 )
i = atoi(argv[1]);
else
i = 0;
j = i; // T
assert ( j = 0 j 10 ); // T
printf(%d - %c, i, blah[j]);
return 0;
}
$ ./translated 99
translated: translated.c:20: main: Assertion `j = 0 j 10' failed.
Przerwane (Aborted)
*b*
My example is a bit tendentious and trivial, but it shows that
transforming programs by adding there
assertions/bound-checkers/whatever can support fuzz testing to uncover
some bugs.
--
Mateusz Kocielski
Re: [gsoc] syscall/libc fuzzer proposal
On Sat, Mar 20, 2010 at 01:54:49PM -0400, Elad Efrat wrote: Thor Lancelot Simon wrote: If not, I don't think this adds any benefit to your proposal and is likely to simply be a distraction; I'd urge you in that case to drop it. Strongly seconded. There are so many great ways to improve NetBSD and wasting time and money on fuzzing is about as suboptimal as it gets. Um. First of all, that's not what Thor said; second of all, you really should not be telling potential gsoc students that their project ideas are flatly worthless, even if your judgment were correct; and third, I'm rather surprised that anyone who claims to work on security would call testing and analysis tools worthless. Let's try not to scare everyone off, ok? -- David A. Holland dholl...@netbsd.org
Re: [gsoc] syscall/libc fuzzer proposal
On Sat, Mar 20, 2010 at 12:40:12PM -0400, Thor Lancelot Simon wrote: As a part of my work I would like to write a translator for C language and a small library. Their goal would be to detect integer overflows, stack overflows, problems with static array indexing, etc (when such occur during the program execution). It will enable me to uncover more bugs in the software. What is the benefit of this when compared to existing static-analysis tools such as Coverity Scan, splint, or the Clang static analyzer? Will this cover any cases they don't? If so, which ones? AIUI from chat, the idea is to increase the probability that if the testing causes something bogus to happen, the bogus behavior will result in an easily identifiable abort. This seems like a valid line of reasoning; all the same, implementing such a tool is a fairly big (and annoying) pile of grunt work. Plus various variations on it have been done before. (Some of which might be worth looking into, actually.) -- David A. Holland dholl...@netbsd.org
Re: [gsoc] syscall/libc fuzzer proposal
On Sat, Mar 20, 2010 at 3:24 PM, David Holland dholland-t...@netbsd.org wrote: On Sat, Mar 20, 2010 at 01:54:49PM -0400, Elad Efrat wrote: Thor Lancelot Simon wrote: If not, I don't think this adds any benefit to your proposal and is likely to simply be a distraction; I'd urge you in that case to drop it. Strongly seconded. There are so many great ways to improve NetBSD and wasting time and money on fuzzing is about as suboptimal as it gets. Um. First of all, that's not what Thor said; Sorry? Are you saying that me agreeing with Thor that unless this proposal shows some clear advantage over what we already have -- specifically Coverity Scan -- it should probably be dropped is not what Thor said? second of all, you really should not be telling potential gsoc students that their project ideas are flatly worthless, even if your judgment were correct; I said exactly what I think and I'll repeat it again: there are many ways to improve NetBSD. Wasting both time and money, even if it's someone else's, on things that aren't likely to benefit NetBSD in the long term is a waste. There's a list of projects NetBSD's interested in, and when someone proposes a project not on the list it should be reviewed. What I said is my opinion. I don't decide which projects are selected, nor do I participate or plan to participate; it's an honest, objective opinion. and third, I'm rather surprised that anyone who claims to work on security would call testing and analysis tools worthless. I don't *claim* anything, David; I *work*, at least as opposed to, say, assigning bugs to me, claiming for years I'll do something about them (together with many other grand ideas) and instead fix, I dunno, whitespace and grammar issues. Take your preaching elsewhere; I couldn't care less. As for the issue at hand, well, I suggest you look at what the proposal is, what we already have for years, and draw your own conclusions. -e.
Re: [gsoc] syscall/libc fuzzer proposal
On Sat, Mar 20, 2010 at 03:40:33PM -0400, Elad Efrat wrote: If not, I don't think this adds any benefit to your proposal and is likely to simply be a distraction; I'd urge you in that case to drop it. Strongly seconded. There are so many great ways to improve NetBSD and wasting time and money on fuzzing is about as suboptimal as it gets. Um. First of all, that's not what Thor said; Sorry? Are you saying that me agreeing with Thor that unless this proposal shows some clear advantage over what we already have -- specifically Coverity Scan -- it should probably be dropped is not what Thor said? He was talking about the bounds-checking translation tool part. You were attacking the entire thing. second of all, you really should not be telling potential gsoc students that their project ideas are flatly worthless, even if your judgment were correct; I said exactly what I think Which was tactless and rude. If someone comes along with an idea that's basically a waste of time, they should be gently steered towards something else. Students don't always have good ideas; that's why they need mentoring and advising, but you don't mentor and advise very effectively by being hostile and dismissive. Also, outside of the specific gsoc context, we have a long-standing custom in this project to not tell other people what to spend their time on or what is and isn't valuable. and third, I'm rather surprised that anyone who claims to work on security would call testing and analysis tools worthless. I don't *claim* anything, David; I *work*, at least as opposed to, say, assigning bugs to me, claiming for years I'll do something about them (together with many other grand ideas) and instead fix, I dunno, whitespace and grammar issues. Take your preaching elsewhere; I couldn't care less. Is that what you think I do? (And if so, do you really want to get into ad hominems? You're on fairly shaky ground.) As for the issue at hand, well, I suggest you look at what the proposal is, what we already have for years, and draw your own conclusions. Yes, I have; it needs to be fleshed out into a real project proposal (as is to be expected at this stage, after all) but I don't see anything inherently wrong with it so far. -- David A. Holland dholl...@netbsd.org
Re: [gsoc] syscall/libc fuzzer proposal
On Mar 20, 2010, at 3:35 PM, David Holland wrote: On Sat, Mar 20, 2010 at 12:40:12PM -0400, Thor Lancelot Simon wrote: As a part of my work I would like to write a translator for C language and a small library. Their goal would be to detect integer overflows, stack overflows, problems with static array indexing, etc (when such occur during the program execution). It will enable me to uncover more bugs in the software. What is the benefit of this when compared to existing static-analysis tools such as Coverity Scan, splint, or the Clang static analyzer? Will this cover any cases they don't? If so, which ones? AIUI from chat, the idea is to increase the probability that if the testing causes something bogus to happen, the bogus behavior will result in an easily identifiable abort. This seems like a valid line of reasoning; all the same, implementing such a tool is a fairly big (and annoying) pile of grunt work. Plus various variations on it have been done before. (Some of which might be worth looking into, actually.) Yes. Beyond that, though, I think that fuzzing is a very useful, valid security *testing* tool. Thor asked how this compared to Coverity and the like. Unless I badly misunderstand what Coverity is about, it's a static analysis tool. Static analysis is great -- when it does the job. It can find bugs that would require really unlikely test cases and timing. But there are things that static analysis can't do, even in principle. Fuzzing is one tool, among many, for dynamic testing. Now -- as has been noted, just saying fuzzing isn't enough. There are tools around that do it; there's also the NetBSD testing framework. Integrating all that, plus new code and perhaps new analysis, would probably be a better way to proceed. Note carefully that I said probably -- I haven't fleshed out my ideas any more than the original poster. But I do encourage the original poster to proceed and develop the proposal some more. --Steve Bellovin, http://www.cs.columbia.edu/~smb
Re: [gsoc] syscall/libc fuzzer proposal
On Sat, Mar 20, 2010 at 05:32:28PM +0200, Mateusz Kocielski wrote: 4. What are my main goals? * write syscall/libc fuzzer * develop additional tools to fuzzer environment * cover as much project code as possible during testing Mateusz, I'm going to seize on one of your goals, sorry. :-) I'm awfully interested in measuring instruction statement coverage in NetBSD, and in using actual/simulated conditions to exercise error paths There may be a GSoC project in that, too. Before I forget: fuzzing the network stacks (netinet, netinet6, net80211) could be interesting. Dave -- David Young OJC Technologies dyo...@ojctech.com Urbana, IL * (217) 278-3933
Re: [gsoc] syscall/libc fuzzer proposal
On Sat, Mar 20, 2010 at 08:53:12PM +0200, Mateusz Kocielski wrote: 2010/3/20 Thor Lancelot Simon t...@panix.com: What is the benefit of this when compared to existing static-analysis tools such as Coverity Scan, splint, or the Clang static analyzer? ?Will this cover any cases they don't? ?If so, which ones? Undecidability is the limit for static-analysis. Consider following program: I did. Doesn't GCC's existing SSP, FORTIFY_SOURCE, and/or mudflap mode already catch such cases? (I am pretty sure Coverity would also catch them at compile time). Thor
Re: [gsoc] syscall/libc fuzzer proposal
On Sat, Mar 20, 2010 at 05:32:28PM +0200, Mateusz Kocielski wrote:
As a part of my work I would like to write a translator for C language and a
small library. Their goal would be to detect integer overflows, stack
overflows,
problems with static array indexing, etc (when such occur during the program
execution). It will enable me to uncover more bugs in the software.
(Eg.
int foo() { char d[10]; int i; i = 5; return d[i] }
will be translate into:
int foo() { char d[10]; int i; int _x; i = 5; _x = i; if ( _x 0 ||
_x 9 ) ERROR();
return d[_x]; }
)
I think gcc has support for something like this. Take a look at the -fmudflap
option.
eric
Re: [gsoc] syscall/libc fuzzer proposal
On Sat, Mar 20, 2010 at 06:38:57PM -0500, David Young wrote: On Sat, Mar 20, 2010 at 05:32:28PM +0200, Mateusz Kocielski wrote: 4. What are my main goals? * write syscall/libc fuzzer * develop additional tools to fuzzer environment * cover as much project code as possible during testing Mateusz, I'm going to seize on one of your goals, sorry. :-) I'm awfully interested in measuring instruction statement coverage in NetBSD, and in using actual/simulated conditions to exercise error paths There may be a GSoC project in that, too. I once spent some time at work playing around with gcc's -ftest-coverage option, and using lcov to display the results. Other than having to write a short hack to fix paths in the gcda file gcc creates it worked ok, but collecting the coverage wasn't really the hard part. It's the work of actually exercsing the code paths, and making it easy enough to do so with the coverage option enabled that people actually use it, or setting up something automatic to do the tests, where things get a bit difficult. IMO, that's certainly a worthwhile project, but keep in mind that it's not just a write some code project, but also a get people to buy into it one, which can be considerably more involved. eric
