> Wow, great job! Until recently I had to upload SBOMs to Dependency Track manually. I suppose you have some sort of script for that?
I looked briefly at it before, but yes I hope that finally this and next week I will be able to refresh and re-upload all the few hundreds (or even thousands) of SBOMS for Airflow + our ~90 providers we have :) - see below - via API (which I already got credentials from Arnout that need refreshing). We are a bit extreme case, but I think it can show what can be done when you have a lot of interconnected components released by a PMC each with own set of dependencies and with the need of those components to be able to be installed together (including optionality). > The first challenge that comes to mind is what version of our transitive dependencies should libraries specify in their SBOMs. I stress out the case of libraries, because for applications it is easier: they list the libraries they embed in their distribution. > There is also another solution in Maven: I could add `bar` as a direct dependency of my project and bump its version to `1.0.1`. This way users can also benefit from the version bump, but this adds additional maintenance work: Dependabot will track new versions of `bar` and the list of project dependencies becomes long and messy. > What do you think we should do in this case? > This is something I've been thinking about for a while: a (Maven) project should not have optional dependencies. It just makes a mess in the end and usually surprises users with class not found errors at runtime. I can share a complete experience on how we are doing it all in Airflow. We have had a solution for it for a long time - and perfected over time (I introduced it some 4 years ago, before we started releasing Airflow 2). We are using the mechanism of constraints of pip for that. Together with every release of Airflow (Airflow is rather special because it is both application and library) we produce a complete set of dependencies (including transitive) frozen at release time [1]. This is all fully automated and built-in our CI - the constraints are upgraded and automatically committed after every successful CI "canary" build and those are the frozen dependencies most of our PRs are tested against (so that new versions of released dependencies are not impacting regular PRs. Whenever we release Airflow we freeze the constraints with a tag in GitHub. For example [2] are the frozen constraints for Airflow 2.8.1 Those are the "golden set" of dependencies which were available at the moment of release of Airflow X.Y.Z and provider(s) N.M.O. We now have a mechanism to generate SBOMS for past versions of Airflow using those (and this is how we are going to continue also in the future). So basically our SBOM is a snapshot of all the "highest non-conflicting" versions of dependencies at the moment of release of a given component version. I think that's the most reasonable approach for us, maybe it can be applied for others. And what's interesting - those constraints are a complete set of dependencies for Airflow + ALL OPTIONAL sets of dependencies. In our case - every provider we release is such an optional set of dependencies. Note - those "golden" constraints are also the mechanism we recommend our users to install Airflow in reproducible way [3] (because we want to keep our upper-bound limits mostly opened, allowing the users to upgrade to new versions of dependencies released after we release Airflow, but since they can break Airflow we also wanted to have a way to be able to install Airflow in a reproducible way by our users even if someone releases future version of dependency that breaks Airflow). So choosing those "frozen" constraints as source of SBOMS is very good for us - because those are also user-facing and the actual way we recommend users to install airflow is with those versions. Also reference images we publish (and freeze at release time [4]) use the very same set of constraints. I know it's a Python world so it might be a bit different, but I think a similar approach can be - I think used for all libraries as well in different languages. Over the last few years when we released (long before SBOMS and CRA) I've managed to work out a very nice and pretty interconnected approach - where everything revolves around knowing what complete set of dependencies is "golden" at the moment of release. And this is great to see how it nicely fits now with SBOMS (and we have years of history that we can reliably tap in to produce SBOMS for all historical versions of Airflow 2 and all providers, which is pretty cool.. > It would be nice to integrate Dependency Track into our release process. If the process of uploading SBOMs to Dependency Track is simplified, I could: Oh we absolutely plan to do it as well and publish SBOM by release managers automatically at the moment of release :D. And we are not far from that. [1] https://github.com/apache/airflow/commits/constraints-main/ [2] https://github.com/apache/airflow/commits/constraints-2.8.1/ [3] https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html [4] https://airflow.apache.org/docs/docker-stack/index.html#fixing-images-at-release-time On Mon, Feb 19, 2024 at 6:24 PM Gary Gregory <garydgreg...@gmail.com> wrote: > This is something I've been thinking about for a while: a (Maven) project > should not have optional dependencies. It just makes a mess in the end and > usually surprises users with class not found errors at runtime. > > This makes more sense if you split up a project into a multi-module (Maven) > project to only bring in required dependencies in the module that actually > needs it. > > Then I can add a check (somehow) that warns me (or fails) if my POMs use > optional dependencies. Another informative check would be if transitive > dependencies have optional dependencies. > > Gary > > On Mon, Feb 19, 2024, 12:12 PM Piotr P. Karwasz <piotr.karw...@gmail.com> > wrote: > > > Hi Arnout, > > > > Wow, great job! Until recently I had to upload SBOMs to Dependency > > Track manually. I suppose you have some sort of script for that? > > > > On Mon, 19 Feb 2024 at 15:20, Arnout Engelen <enge...@apache.org> wrote: > > > More and more Apache projects are producing SBOMs as part of their > > release > > > process. Challenges producing and consuming SBOMs are definitely > on-topic > > > for this list, and ideally we can consolidate that knowledge on the > > wiki[0] > > > > The first challenge that comes to mind is what version of our > > transitive dependencies should libraries specify in their SBOMs. I > > stress out the case of libraries, because for applications it is > > easier: they list the libraries they embed in their distribution. > > > > I am a big fan of Maven's dependency management, so whenever our > > direct dependency `foo` depends on a vulnerable `bar-1.0.0`, I bump > > the version of `bar` in the dependency management section of my > > project to `1.0.1` and the problem is solved! My test suite will run > > using `bar-1.0.1`, my SBOM will contain `bar-1.0.1` and Dependency > > Track will not complain. > > > > However this is just a trick: users of my library will still have > > `bar-1.0.0` on their classpath, unless they also bump the version of > > `bar` or the `foo` project releases a version that depends on > > `bar-1.0.1`. > > > > There is also another solution in Maven: I could add `bar` as a direct > > dependency of my project and bump its version to `1.0.1`. This way > > users can also benefit from the version bump, but this adds additional > > maintenance work: Dependabot will track new versions of `bar` and the > > list of project dependencies becomes long and messy. > > > > What do you think we should do in this case? > > > > > If you know of any other projects to include, would like help setting > up > > > SBOM publishing for your project, contribute 'nightly' SBOM snapshots, > or > > > discuss other things SBOM, I'm all ears! > > > > It would be nice to integrate Dependency Track into our release > > process. If the process of uploading SBOMs to Dependency Track is > > simplified, I could: > > > > * compile a snapshot, > > * check if there are dependency alerts (Dependabot gives us alerts > > for direct dependencies, but not the transitive ones), > > * bump the vulnerable transitive dependency versions (if possible), > > * prepare a release candidate. > > > > Piotr > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: > security-discuss-unsubscr...@community.apache.org > > For additional commands, e-mail: > > security-discuss-h...@community.apache.org > > > > >
