Re: Correctly configuring Apache Commons components for oss-fuzz

[Oliver Chang]

Hi Mark,

Thanks for the early feedback.

Re a), unfortunately I'm not aware of an easy way to do this with our
current bug tracking system (Monorail). If it's an important feature to
have, one way to achieve this may be to set up a separate "
security-oss-fuzz-not...@commons.apache.org" group or something similar to
be CCed on all issues, which just forwards any notifications to the main "
secur...@commons.apache.org" list. The main list can then filter out emails
based on the recipient to avoid duplication. Would that work?

Re b), thank you for the feedback. We will be working on making our bug
reports contain more actionable context in the notifications themselves.

Best,
Oliver

On Sun, 20 Nov 2022 at 21:24, Mark Thomas  wrote:

> Hi Oliver,
>
> The following are a couple of (hopefully) low hanging fruit that will
> smooth a couple of rough edges. These aren't the biggest issues - just
> something to get started with.
>
> a) It would be very helpful if there was an option to enable sending of
> notifications for your own comments.
>
> b) It would be helpful if more (actually all) of the issue detail was
> included in the notification emails.
>
> Mark
>
>
> On 18/11/2022 00:02, Oliver Chang wrote:
> > Thanks Mark.
> >
> > Please let us know how we can help make this fuzzing experience better
> > for you. We're also happy to jump on a call to walk through your
> > concerns and reach a good outcome.
> >
> > Best regards,
> > --
> > Oliver
> >
> >
> > On Thu, 17 Nov 2022 at 06:56, Mark Thomas  > > wrote:
> >
> > I haven't forgotten about this. I am currently working through the
> open
> > issues. I want to complete first that so feedback isn't skewed by a
> > single project.
> >
> > Mark
> >
> >
> > On 11/11/2022 14:45, Roman Wagner wrote:
> >  > Hi Mark,
> >  >
> >  > I think the best way forward is to collaborate and have a short
> > feedback
> >  > loop.
> >  >
> >  > Did you mean build failures by “Invalid due to broken test”? If
> > yes, I am
> >  > not sure what we can do about the broken tests since those are
> > already
> >  > executed and tested by check build scripts locally and in a CI/CD
> > pipeline.
> >  > Build and Coverage failures are sometimes supposed to happen if
> > there are
> >  > changes in the target repository itself or if there are
> > infrastructure
> >  > issues in OSS-Fuzz. We will investigate those issues in more
> > detail. Maybe
> >  > some filter in the apache mailing list is helpful for you in the
> > short
> >  > term, Fuzzing and Coverage build issues have a "build failure"
> > string in
> >  > the subject. That would enable you to focus on the reports only.
> >  >
> >  > In order to make sure that we get high-quality tests and results,
> >  > maintainer feedback from apache will be very valuable and
> > welcome. You have
> >  > the best domain knowledge about your code base and can give
> > valuable hints
> >  > on which APIs to tackle best. There was already some valuable
> > feedback for
> >  > Apache Tomcat in
> > https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=53153
> > .
> >  > Let us extend  this collaboration. We can discuss and agree on
> > the attack
> >  > vectors in apache-commons components.
> >  >
> >  > Best regards
> >  > Roman
> >  >
> >  > On Thu, Nov 10, 2022 at 10:29 AM Mark Thomas  > > wrote:
> >  >
> >  >> Oliver,
> >  >>
> >  >> My requirements regarding configuration are:
> >  >>
> >  >> - secur...@commons.apache.org
> >  MUST be notified of all
> security
> >  >> vulnerability reports for all Apache Commons components
> >  >>
> >  >> - a mechanism MUST be provided for the
> > secur...@commons.apache.org 
> >  >> Google user to view all historical reports that were not
> > previously
> >  >> notified to that address
> >  >>
> >  >> - if any further Apache Commons components get added to oss-fuzz
> >  >> then secur...@commons.apache.org
> >  MUST receive notification of
> any
> >  >> issues as they are found
> >  >>
> >  >> - more generally, if *any* Apache Software Foundation project is
> > added
> >  >> to oss-fuzz then the notifications for that project MUST
> > include the
> >  >> relevant security team for that project
> >  >>
> >  >> If you can achieve the above with the current structure then
> great.
> >  >>
> >  >>
> >  >> Separately, there is a concern regarding the false positive
> > rate. With
> >  >> the oss-fuzz integration provided by Code Intelligence we have

Re: Correctly configuring Apache Commons components for oss-fuzz

[Mark Thomas]

Hi Oliver,

The following are a couple of (hopefully) low hanging fruit that will 
smooth a couple of rough edges. These aren't the biggest issues - just 
something to get started with.


a) It would be very helpful if there was an option to enable sending of
   notifications for your own comments.

b) It would be helpful if more (actually all) of the issue detail was
   included in the notification emails.

Mark


On 18/11/2022 00:02, Oliver Chang wrote:

Thanks Mark.

Please let us know how we can help make this fuzzing experience better 
for you. We're also happy to jump on a call to walk through your 
concerns and reach a good outcome.


Best regards,
--
Oliver


On Thu, 17 Nov 2022 at 06:56, Mark Thomas > wrote:


I haven't forgotten about this. I am currently working through the open
issues. I want to complete first that so feedback isn't skewed by a
single project.

Mark


On 11/11/2022 14:45, Roman Wagner wrote:
 > Hi Mark,
 >
 > I think the best way forward is to collaborate and have a short
feedback
 > loop.
 >
 > Did you mean build failures by “Invalid due to broken test”? If
yes, I am
 > not sure what we can do about the broken tests since those are
already
 > executed and tested by check build scripts locally and in a CI/CD
pipeline.
 > Build and Coverage failures are sometimes supposed to happen if
there are
 > changes in the target repository itself or if there are
infrastructure
 > issues in OSS-Fuzz. We will investigate those issues in more
detail. Maybe
 > some filter in the apache mailing list is helpful for you in the
short
 > term, Fuzzing and Coverage build issues have a "build failure"
string in
 > the subject. That would enable you to focus on the reports only.
 >
 > In order to make sure that we get high-quality tests and results,
 > maintainer feedback from apache will be very valuable and
welcome. You have
 > the best domain knowledge about your code base and can give
valuable hints
 > on which APIs to tackle best. There was already some valuable
feedback for
 > Apache Tomcat in
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=53153
.
 > Let us extend  this collaboration. We can discuss and agree on
the attack
 > vectors in apache-commons components.
 >
 > Best regards
 > Roman
 >
 > On Thu, Nov 10, 2022 at 10:29 AM Mark Thomas mailto:ma...@apache.org>> wrote:
 >
 >> Oliver,
 >>
 >> My requirements regarding configuration are:
 >>
 >> - secur...@commons.apache.org
 MUST be notified of all security
 >>     vulnerability reports for all Apache Commons components
 >>
 >> - a mechanism MUST be provided for the
secur...@commons.apache.org 
 >>     Google user to view all historical reports that were not
previously
 >>     notified to that address
 >>
 >> - if any further Apache Commons components get added to oss-fuzz
 >>     then secur...@commons.apache.org
 MUST receive notification of any
 >>     issues as they are found
 >>
 >> - more generally, if *any* Apache Software Foundation project is
added
 >>     to oss-fuzz then the notifications for that project MUST
include the
 >>     relevant security team for that project
 >>
 >> If you can achieve the above with the current structure then great.
 >>
 >>
 >> Separately, there is a concern regarding the false positive
rate. With
 >> the oss-fuzz integration provided by Code Intelligence we have
seen the
 >> following with Apache Tomcat in a little under 3 months.
 >>
 >> Total "vulnerability" reports: 39
 >>
 >> Invalid due to broken test: 31%
 >> False positive:             52%
 >> Bugs:                       18%
 >> Valid security issues:       0%
 >>
 >> To add some commentary:
 >> - the bugs were minor / extreme edge cases users were unlikely
to hit
 >> - false positives were all due to the tests being based on invalid
 >>     assumptions regarding whether input was expected to be
trusted or not
 >>
 >>
 >> If those statistics were repeated across multiple Apache Commons
 >> components, the volume of invalid reports would be more than the
 >> volunteers of the Apache Commons security team could handle.
 >>
 >> I have no objection to being overwhelmed with valid security
 >> vulnerability reports. If that ever happened, we would find a way to
 >> deal with it.
 >>
 >> I do have very strong objections to being overwhelmed with invalid
 >> security vulnerability reports. If we see the same false
positive rate