Re: Dealing with static code analysis in Fedora
A long time ago I looked at automating some of the invariants in libvirt. I didn't get very far, but you can see what I did here: http://people.redhat.com/~rjones/cil-analysis-of-libvirt/ The thing that is interesting about libvirt is there are two classes of static error. The first is stuff like freeing pointers twice along paths, already very well covered by static analysis tools. But the second are very libvirt-specific invariants. For example, the virError* functions must only be called exactly once along error paths, and never along normal return paths. Or another example: libvirt has its own memory allocation functions, and those must be used, in a certain way, and not say C functions like 'malloc'. Other large projects probably have similar requirements. The kernel certainly does, and has its own checker for them. Your Python ref-counting checker is kind of an example of this, albeit one that applies to a larger selection of projects, not just a single one. For these project-specific tests, I think the only way we're ever going to be able to automate it is to allow custom tests to be written. For the analysis above, I was attempting (and mostly failing) to use OCaml + CIL. CLANG + custom C++ (gord help us) might be another way to do this. -- Agreed about the rest of it, and static analysis is definitely something that should be welcomed to Fedora. By the way I think the wiki page should at least mention Coverity because it's such a big player in this area, even though it'll be rejected because it doesn't fit the Free Software requirement that you listed. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Static Analysis SIG ? (was Re: Dealing with static code analysis in Fedora)
On Thu, 2012-12-13 at 21:45 +0200, Alek Paunov wrote: On 11.12.2012 23:52, David Malcolm wrote: We'd be able to run all of the code in Fedora through static analysis tools, and slurp the results into the database Dave, I really do not know what to say first :-). The subject is so important and there are so many aspects and application fields - IMHO, the topic is the most important one in the devel list lately (and is in _direct_ relation with the all other _hot_ topics - ABI stability, upgradeability, collections, reliable/automated migrations, packagers productivity, rawhide, etc.) I very much agree, of course :) We rely far too much on manual tasks when we build Fedora, and there are plenty that we could be automating. Within the domain of making Fedora more stable, static code analysis should be the partner of the AutoQA effort: the former can detect bugs at compile-time, the latter at run-time (hopefully before it reaches any systems that people care about). The two approaches are of course complementary: programmers tend to overemphasize the value of static code analysis - but both are valuable. I hope this thread will be long and fruitful discussion with the final effect to change Fedora to something that all motivated devs in the list expect it to become. Just few preliminary questions about your insights in the future: 1) What about dumping the GCC structs to the DB during the OS/Repos processing from the same beginning (means something more powerful than dxr.mozilla.org, and possibility to engage various static analysis people to the project, like Masaryk University team as Michal reported, without the locking to concrete compiler technology/encoding) Yes - we could use that to build a great source cross-referencer for all of Fedora. I can think of plenty of uses for this (e.g. upstream added a new parameter to this library function; how much is going to break?). FWIW, I'm trying to focus my efforts on bug detection, though, and I'm not sure how far one can get with a database of the IR of every function. I've been looking at using LTO to do interprocedural analysis across source files via my gcc plugin, and I have that working (not yet released), so I can run analyses at whole libraries at link-time, from within GCC. One drawback of that is that (IIRC) GCC's LTO representation within the .o files is specific to the precise GCC version, and IIRC there aren't any guarantees about forward or backward compatibility, compared to say, LLVM bitcode. It might be possible to patch GCC to use GIMPLE textual dumps (compressed?) as an intermediate format, storing that within the .o files in a similar way to the LTO implementation. But I'm speculating here. 2) Clang world enrolled the (suspicious) term Compilation database as the safe sequence and arguments of the compiler invocations for a package build. What is your opinion for abstracting build systems to the DB in the same way in Fedora (based on the GCC plugin)? I hadn't heard of that; presumably you're referring to: http://clang.llvm.org/docs/JSONCompilationDatabase.html right? That sounds reminiscent of the fake-make program that Steve Grubb wrote and mentioned elsewhere in this thread: http://lists.fedoraproject.org/pipermail/devel/2012-December/175259.html http://people.redhat.com/sgrubb/swa/cwe/index.html As for both (2) and the SA part of (1), they seem to me to be coming at this from a slightly different direction to the one I'm interested in: they're approaching the problem of I have a static analysis tool that finds bugs in, say C++ code, how do I get it to run on all of Fedora without having to dealing with the bazillion different build invocations, Makefiles, autoconf, cmake, custom scripts etc across all of the packages. As described in my other mail, I think it's possible to reliably run a static analysis tool whilst rebuilding an srpm by hacking up mock to inject the analysis payload, and then harvesting report files from the chroot, as described in: http://lists.fedoraproject.org/pipermail/devel/2012-December/175258.html [search for nasty nasty hack... but it works within that post :)]. There's still the messy business of actually doing the rebuilds, of course (I believe we can set up some guest VMs within Fedora's infrastructure for big workloads - massively handwaving here, of course). The problem I'm most interested in is I've run my tool on lots of packages and it generated lots of results. What do we do with all of these results? - I want to come up with a better answer than file lots of bugzillas, since that approach would suck: what happens next time you want to run the tool? (newer version of tool, or newer version of srpm). Note that we're already running an analysis tool on all of the C/C++ tool in Fedora: the compiler. How good is everyone at reading all of the compiler warnings from their builds? (or does everyone use -Werror?) The system I'm envisaging
Re: Dealing with static code analysis in Fedora
On 11.12.2012 23:52, David Malcolm wrote: We'd be able to run all of the code in Fedora through static analysis tools, and slurp the results into the database Dave, I really do not know what to say first :-). The subject is so important and there are so many aspects and application fields - IMHO, the topic is the most important one in the devel list lately (and is in _direct_ relation with the all other _hot_ topics - ABI stability, upgradeability, collections, reliable/automated migrations, packagers productivity, rawhide, etc.) I hope this thread will be long and fruitful discussion with the final effect to change Fedora to something that all motivated devs in the list expect it to become. Just few preliminary questions about your insights in the future: 1) What about dumping the GCC structs to the DB during the OS/Repos processing from the same beginning (means something more powerful than dxr.mozilla.org, and possibility to engage various static analysis people to the project, like Masaryk University team as Michal reported, without the locking to concrete compiler technology/encoding) 2) Clang world enrolled the (suspicious) term Compilation database as the safe sequence and arguments of the compiler invocations for a package build. What is your opinion for abstracting build systems to the DB in the same way in Fedora (based on the GCC plugin)? 3) As I said already, IMHO, this thread is the most practically important topic in Fedora. What about SIG/Team? I think base of 8-10 high experienced part-time contributors will be enough for your spec and 1)-like enhancements. Kind Regards, Alek P.S. Fedora infrastructure resources are mandatory for the final Fedora repos cooking, but I think that the community is able to provide less secure, but much more in volume resources for the analysis workers (Fedora can just supply small enslaving script for the dedicated VM) -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Dealing with static code analysis in Fedora
Hi Dave! We have started a similar project in ABRT some year ago. The original purpose was to automatically determine some crash characteristics (security impact, unchecked user input etc.). The overall idea is rebuilding the given package with some compiler plugin and go through the AST based on coredump's stack. We have built an infrastructure working on top of ABRT Server[1]. This is because of its backend synchronizing with several Fedora services (koji, pkgdb, bugzilla) storing all the data in one database and thus ability to make fast cross-service queries. The storage (that's how we call it) is more complex - it uses DB for relational data and saves large files (like RPMs) on the FS. The storage contains (almost) all packages built in koji. We were trying to use mock yum for a long time, but we were not able to make it work correctly with such a complex repo. That's why we have created our own dependency solver based on libsolv (using the DB as metadata source) and use rpm directly to install the resulting set. This resulted into a fast chroot installer able to interrupt build process in many phases. Some guys working on static analysis at Masaryk University, Brno, CZ showed interest in the project. They are working on various clang plugins for static analysis, but have no real input data. We agreed to implement a service rebuilding Fedora packages with LLVM/Clang and providing the output (LLVM bitcode files). They will provide the analysis tools. This showed up as a non-trivial problem, because a simple CC=clang is not enough :). The thing is however implemented and I've personally rebuilt Fedora a few times, leaving me with some 60% of packages successfully processed and a few ideas for improvement. The service has not yet been deployed in Fedora infrastructure, because we 1) don't consider it ready 2) lack hardware capacity. This is more or less the current state. We are open to any discussion, new ideas, use cases or patches :). Sources: http://git.fedorahosted.org/cgit/faf.git/ RFE: https://fedorahosted.org/abrt/newticket?component=faf Michal ABRT [1] http://abrt.fedoraproject.org On 2012-12-11 22:52, David Malcolm wrote: A while back I ran my static checker on all of the Python extension modules in Fedora 17: http://fedoraproject.org/wiki/Features/StaticAnalysisOfPythonRefcounts I wrote various scripts to build the packages in a mock environment that injects my checker into gcc, then wrote various scripts to triage the results. I then filed bugs by hand for the most important results, writing some more scripts along the way to make the process easier. This led to some valuable bug fixes, but the mechanism for running the analysis was very ad hoc and doesn't scale. In particular, we don't yet have an automated way of rerunning the tests, whilst using the old results as a baseline. For example it would be most useful if only new problems could be reported, and if the system (whatever it is) remembered when a report has been marked as a true bug or as a false positive. Similarly, there's no automated way of saying this particular test is bogus; ignore it for now. I'm wondering if there's a Free Software system for doing this kind of thing, and if not, I'm thinking of building it. What I have in mind is a web app backed by a database (perhaps checker.fedoraproject.org ?) We'd be able to run all of the code in Fedora through static analysis tools, and slurp the results into the database: primarily my cpychecker work, but we could also run the clang analyzer etc. I've also been working on another as-yet-unreleased static analysis tool for which I'd want a db for the results. What I have working is a way to inject an analysis payload into gcc within a mock build, which dumps JSON report files into the chroot without disturbing the real build. The idea is then to gather up the JSON files and insert the report data into the db, tagging it with version information. There are two dimensions to the version information: (A) the version of the software under analysis (name-version-release.arch) (B) the version of the tool doing the analysis We could use (B) within the system to handle the release cycle of a static analysis tool. Initially, any such analysis tools would be regarded as experimental, and package maintainers could happily ignore the results of such a tool. The maintainer of an analysis tool could work on bug fixes and heuristics to get the signal:noise ratio of the tool up to an acceptable level, and then the status of the analysis tool could be upgraded to an alpha level or beyond. Functional Requirements: * a collection of reports (not bugs): * interprocedural control flow, potentially across multiple source files (potentially with annotations, such as value of variables, call stack?) * syntax highlighting * capturing of all relevant source (potentially with headers as
Re: Dealing with static code analysis in Fedora
On Wed, 2012-12-12 at 01:00 -0200, Paulo César Pereira de Andrade wrote: (Thanks; various replies inline below) 2012/12/11 David Malcolm dmalc...@redhat.com: A while back I ran my static checker on all of the Python extension modules in Fedora 17: http://fedoraproject.org/wiki/Features/StaticAnalysisOfPythonRefcounts I wrote various scripts to build the packages in a mock environment that injects my checker into gcc, then wrote various scripts to triage the results. I then filed bugs by hand for the most important results, writing some more scripts along the way to make the process easier. This led to some valuable bug fixes, but the mechanism for running the analysis was very ad hoc and doesn't scale. I think it could be useful at least as a generic tool where one would just do something like: make CC=gcc-with-python-plugin like some time ago one could run make CC=cgcc to see what sparse would tell. Or maybe think of it as a tool like rpmlint. That's what I first tried, but there are plenty of packages that don't use make. I then tried setting the rpm build flags to add my plugin to gcc, but I ran into enough packages that didn't respect them that it's simplest to hack up /usr/bin/gcc within the built chroot to automatically add the checker. (This means that e.g. all the little programs that configure runs get the checker run on them too, but that isn't a major issue). In particular, we don't yet have an automated way of rerunning the tests, whilst using the old results as a baseline. For example it would be most useful if only new problems could be reported, and if the system (whatever it is) remembered when a report has been marked as a true bug or as a false positive. Similarly, there's no automated way of saying this particular test is bogus; ignore it for now. Something like valgrind's .supp files? Yes, though I was thinking of a web ui backed by a web UI to make it trivial to flag something. I don't know if e.g. a collection of files backed by git is going to be as flexible... Not sure. I'm wondering if there's a Free Software system for doing this kind of thing, and if not, I'm thinking of building it. What I have in mind is a web app backed by a database (perhaps checker.fedoraproject.org ?) Remembers me of http://upstream-tracker.org/ Thanks for the link, I hadn't seen it. Looks nice, and has some good UI ideas, but it's not quite what I had in mind - it seems to have a fixed collection of tests, each of which is very different from the other, whereas I want a dynamic collection of tests, all within the pattern of paths through source code, with tracking of the quality of those tests. Any static analysis tool will have false positives, and we need a mechanism in place to cope with that, or we'll just drown in the noise. We'd be able to run all of the code in Fedora through static analysis tools, and slurp the results into the database: primarily my cpychecker work, but we could also run the clang analyzer etc. I've also been working on another as-yet-unreleased static analysis tool for which I'd want a db for the results. What I have working is a way to inject an analysis payload into gcc within a mock build, which dumps JSON report files into the chroot without disturbing the real build. The idea is then to gather up the JSON files and insert the report data into the db, tagging it with version information. There are two dimensions to the version information: (A) the version of the software under analysis (name-version-release.arch) (B) the version of the tool doing the analysis We could use (B) within the system to handle the release cycle of a static analysis tool. Initially, any such analysis tools would be regarded as experimental, and package maintainers could happily ignore the results of such a tool. The maintainer of an analysis tool could work on bug fixes and heuristics to get the signal:noise ratio of the tool up to an acceptable level, and then the status of the analysis tool could be upgraded to an alpha level or beyond. Functional Requirements: * a collection of reports (not bugs): * interprocedural control flow, potentially across multiple source files (potentially with annotations, such as value of variables, call stack?) * syntax highlighting * capturing of all relevant source (potentially with headers as well?) * visualization of control flow so that you can see the path through the code that leads to the error * support for my cpychecker analysis * support for an as-yet-unreleased interprocedural static analysis tool I've been working on * support for reports from the clang static analyzer * ability to mark a report as: * a true bug (and a way to act on it, e.g. escalate to bugzilla or to the relevant upstream
Re: Dealing with static code analysis in Fedora
On Wed, 2012-12-12 at 16:46 +0100, Michal Toman wrote: Hi Dave! We have started a similar project in ABRT some year ago. The original purpose was to automatically determine some crash characteristics (security impact, unchecked user input etc.). The overall idea is rebuilding the given package with some compiler plugin and go through the AST based on coredump's stack. Sounds interesting. Is the idea to capture some higher-level information about the code? If I'm understanding things right, your code is looking at a coredump, which is a snapshot in time of the state of a process (the moment of death, in fact). For example, each thread will have the program counter at a specific location. What I'm trying to track is a predicted path through the code (potentially with state annotations): first the program counter is at foo, then it's at bar, and then at baz a crash happens. We have built an infrastructure working on top of ABRT Server[1]. This is because of its backend synchronizing with several Fedora services (koji, pkgdb, bugzilla) storing all the data in one database and thus ability to make fast cross-service queries. The storage (that's how we call it) is more complex - it uses DB for relational data and saves large files (like RPMs) on the FS. I had a go at reading through the faf sources. Am I right in thinking that the storage you refer to is the collection of Django apps within the pyfaf subdirectory: http://git.fedorahosted.org/cgit/faf.git/tree/pyfaf The storage contains (almost) all packages built in koji. We were trying to use mock yum for a long time, but we were not able to make it work correctly with such a complex repo. That's why we have created our own dependency solver based on libsolv (using the DB as metadata source) and use rpm directly to install the resulting set. This resulted into a fast chroot installer able to interrupt build process in many phases. Some guys working on static analysis at Masaryk University, Brno, CZ showed interest in the project. They are working on various clang plugins for static analysis, but have no real input data. We agreed to implement a service rebuilding Fedora packages with LLVM/Clang and providing the output (LLVM bitcode files). They will provide the analysis tools. This showed up as a non-trivial problem, because a simple CC=clang is not enough :). The thing is however implemented and I've personally rebuilt Fedora a few times, leaving me with some 60% of packages successfully processed and a few ideas for improvement. The service has not yet been deployed in Fedora infrastructure, because we 1) don't consider it ready 2) lack hardware capacity. As noted in another post, when I tried running all of Fedora through my gcc-with-cpychecker, I first tried setting CC, but plenty of things don't use make. I then tried setting the rpm build flags, but plenty of things ignore them. I finally hit on the idea of moving /usr/bin/gcc in the chroot to /usr/bin/the-real-gcc, and replacing /usr/bin/gcc with a script to invoke the-real-gcc with the relevant flags to inject my gcc plugin, with similar hacks for g++ and c++. Heh, it's just a temporary chroot :) See http://git.fedorahosted.org/cgit/gcc-python-plugin.git/tree/misc/fedora/mass-rebuild.py for the gory details. It's a nasty nasty hack... but it works. Presumably you could do something similar for your LLVM work. This is more or less the current state. We are open to any discussion, new ideas, use cases or patches :). If I understand your work, you're attempting to gather LLVM bitcode for all of the sources in Fedora, so that analysis tools can be readily run on them. You're also dealing with the issue of tracking what is Fedora within a database, which is no mean feat (this was where I stumbled in my old rpmgrok tool). I'm approaching this from a different angle: my analysis tools are run directly from within GCC during a build, and the results are stored somewhere. The issue I want to solve is that of managing false positives. Any static code analysis tool is by necessity just an approximation of the actual behavior of the code, and so will have false alarms. We need a way of keeping those false alarms to an acceptable level whilst gleaning useful data from the analysis. So my proposal is less about: here's a big collection of data you can run tests on, and more about: here's a big body of test *results* of unknown quality: what can we do with it so that tests can be improved to the point where package maintainers might want to care about the results, and at that point, management of said results. Or, from an other perspective, the use cases I'm interested are: * Anna the Analysis Tool Author: Anna has written a static analysis tool that detects a particular antipattern in, say, C++ code. It works for her example code, but she wants to try running it on all of Fedora. She expects her tool to crash and to
Re: Dealing with static code analysis in Fedora
On Wednesday, December 12, 2012 01:00:36 AM Paulo César Pereira de Andrade wrote: A while back I ran my static checker on all of the Python extension modules in Fedora 17: http://fedoraproject.org/wiki/Features/StaticAnalysisOfPythonRefcounts I wrote various scripts to build the packages in a mock environment that injects my checker into gcc, then wrote various scripts to triage the results. I then filed bugs by hand for the most important results, writing some more scripts along the way to make the process easier. This led to some valuable bug fixes, but the mechanism for running the analysis was very ad hoc and doesn't scale. I think it could be useful at least as a generic tool where one would just do something like: make CC=gcc-with-python-plugin like some time ago one could run make CC=cgcc to see what sparse would tell. Or maybe think of it as a tool like rpmlint. I wrote a program, fake-make which collects everything so that programs like cppcheck can be run with correct defines, paths, and files. Instructions are here: http://people.redhat.com/sgrubb/swa/cwe/index.html That said, what's really needed is every analyzer to output messages with something in common so that results can be compared. That something in common is CWE (Common Weakness Enumeration). I was working on a mapping for cppcheck to CWE so that it could be correlated with other tools. the advantage of CWE is that its also married to CAPEC (Common Attack Pattern enumeration and classification). This mapping shows some possible ways that the bug being found could be exploited depending on other mitigating factors. So, what would be nice is to figure out how to get all the static analyzers and compilers outputting CWE information. Then define a common format so that correlation tools can be built. If several tools report the same issue at the same line, then its probably not a false positive and someone should look at it. But at the same time, not all bugs are created equal. A buffer overflow is a worse problem than unchecked return code (unless its setuid(2)). There is a scoring framework, CWSS (Common Weakness Scoring System) that can be used to rank bugs so they can be prioritized. It also takes into account the effect of the bug withon the program its found in. For example, buffer overflow in network app or daemon is more critical that same issue in a program run by authenticated users such as ps. Don't get me wrong, there are corners cases...but some heuristic has to be used and decisions have to be made. So.. this would be my advice...try to follow these standards. Its all part of a larger project to track weaknesses, combine with configuration information, and network IDS systems for real time situational awareness. -Steve -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Dealing with static code analysis in Fedora
On Wed, 2012-12-12 at 15:03 -0500, Steve Grubb wrote: On Wednesday, December 12, 2012 01:00:36 AM Paulo César Pereira de Andrade wrote: A while back I ran my static checker on all of the Python extension modules in Fedora 17: http://fedoraproject.org/wiki/Features/StaticAnalysisOfPythonRefcounts I wrote various scripts to build the packages in a mock environment that injects my checker into gcc, then wrote various scripts to triage the results. I then filed bugs by hand for the most important results, writing some more scripts along the way to make the process easier. This led to some valuable bug fixes, but the mechanism for running the analysis was very ad hoc and doesn't scale. I think it could be useful at least as a generic tool where one would just do something like: make CC=gcc-with-python-plugin like some time ago one could run make CC=cgcc to see what sparse would tell. Or maybe think of it as a tool like rpmlint. I wrote a program, fake-make which collects everything so that programs like cppcheck can be run with correct defines, paths, and files. Instructions are here: http://people.redhat.com/sgrubb/swa/cwe/index.html That said, what's really needed is every analyzer to output messages with something in common so that results can be compared. That something in common is CWE (Common Weakness Enumeration). I was working on a mapping for cppcheck to CWE so that it could be correlated with other tools. the advantage of CWE is that its also married to CAPEC (Common Attack Pattern enumeration and classification). This mapping shows some possible ways that the bug being found could be exploited depending on other mitigating factors. So, what would be nice is to figure out how to get all the static analyzers and compilers outputting CWE information. Then define a common format so that correlation tools can be built. If several tools report the same issue at the same line, then its probably not a false positive and someone should look at it. But at the same time, not all bugs are created equal. A buffer overflow is a worse problem than unchecked return code (unless its setuid(2)). There is a scoring framework, CWSS (Common Weakness Scoring System) that can be used to rank bugs so they can be prioritized. It also takes into account the effect of the bug withon the program its found in. For example, buffer overflow in network app or daemon is more critical that same issue in a program run by authenticated users such as ps. Don't get me wrong, there are corners cases...but some heuristic has to be used and decisions have to be made. So.. this would be my advice...try to follow these standards. Its all part of a larger project to track weaknesses, combine with configuration information, and network IDS systems for real time situational awareness. Thanks for the feedback. CWE appears to be a good fit for what I have in mind, so I intend to allow analysis tools to (optionally) supply a CWE id in the reports we capture, so we ought to be able to e.g. query on CWE id in the DB (and I'm adding CWE codes right now to my own checker tool). CWSS doesn't seem to be such a natural fit though: how can we map from source code through to the scenarios described in the CWSS document? setuid binaries is one thing that occurs to me, but I'm not sure how to bridge from the world of, say, GCC's internal representation at build time through to say this code is being run by a regular user let alone the entity has control over [...] a router ; perhaps with a list of ELF files that get run in/linked to setuid binaries? (but that requires bridging the world of compile time vs installation path). Ideas welcome. Dave -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Dealing with static code analysis in Fedora
2012/12/11 David Malcolm dmalc...@redhat.com: A while back I ran my static checker on all of the Python extension modules in Fedora 17: http://fedoraproject.org/wiki/Features/StaticAnalysisOfPythonRefcounts I wrote various scripts to build the packages in a mock environment that injects my checker into gcc, then wrote various scripts to triage the results. I then filed bugs by hand for the most important results, writing some more scripts along the way to make the process easier. This led to some valuable bug fixes, but the mechanism for running the analysis was very ad hoc and doesn't scale. I think it could be useful at least as a generic tool where one would just do something like: make CC=gcc-with-python-plugin like some time ago one could run make CC=cgcc to see what sparse would tell. Or maybe think of it as a tool like rpmlint. In particular, we don't yet have an automated way of rerunning the tests, whilst using the old results as a baseline. For example it would be most useful if only new problems could be reported, and if the system (whatever it is) remembered when a report has been marked as a true bug or as a false positive. Similarly, there's no automated way of saying this particular test is bogus; ignore it for now. Something like valgrind's .supp files? I'm wondering if there's a Free Software system for doing this kind of thing, and if not, I'm thinking of building it. What I have in mind is a web app backed by a database (perhaps checker.fedoraproject.org ?) Remembers me of http://upstream-tracker.org/ We'd be able to run all of the code in Fedora through static analysis tools, and slurp the results into the database: primarily my cpychecker work, but we could also run the clang analyzer etc. I've also been working on another as-yet-unreleased static analysis tool for which I'd want a db for the results. What I have working is a way to inject an analysis payload into gcc within a mock build, which dumps JSON report files into the chroot without disturbing the real build. The idea is then to gather up the JSON files and insert the report data into the db, tagging it with version information. There are two dimensions to the version information: (A) the version of the software under analysis (name-version-release.arch) (B) the version of the tool doing the analysis We could use (B) within the system to handle the release cycle of a static analysis tool. Initially, any such analysis tools would be regarded as experimental, and package maintainers could happily ignore the results of such a tool. The maintainer of an analysis tool could work on bug fixes and heuristics to get the signal:noise ratio of the tool up to an acceptable level, and then the status of the analysis tool could be upgraded to an alpha level or beyond. Functional Requirements: * a collection of reports (not bugs): * interprocedural control flow, potentially across multiple source files (potentially with annotations, such as value of variables, call stack?) * syntax highlighting * capturing of all relevant source (potentially with headers as well?) * visualization of control flow so that you can see the path through the code that leads to the error * support for my cpychecker analysis * support for an as-yet-unreleased interprocedural static analysis tool I've been working on * support for reports from the clang static analyzer * ability to mark a report as: * a true bug (and a way to act on it, e.g. escalate to bugzilla or to the relevant upstream tracker) * a false positive (and a way for the analysis maintainer to act on it) * other bug associations with a report? (e.g. if the wording from the tool's message could be improved) * ability to have a conversation about a report within the UI as a series of comments (similar to bugzilla). * automated report matching between successive runs, so that the markings can be inherited * scriptable triage, so that we can write scripts that mark all reports matching a certain pattern e.g. as being bogus, as being security sensitive, etc * potentially: debug data (from the analysis tool) associated with a report, so that the maintainers of the tool can analyze a false positive * ability to store crash results where some code broke a static analysis tool, so that the tool can be fixed * association between reports and builds * association between builds and source packages * association between packages and people, so that you can see what reports are associated with you (perhaps via the pkgdb?) * prioritization of reports to be generated by the tool * association between reports and tools (and tool versions) * quality marking of tool versions, so that we can ignore alpha
