> So the question is how do we proceed? Do I contact INFRA to enable it for the main repo?
No objections from me, is this something we should vote on though? Perhaps we already have lazy consensus :) > more concretely how do we deal with these PRs in a practical sense? Do we rename them and create an associated JIRA for tracking? Yes I wonder about this too. I only have more questions though: Would we set reviewers [1] to make sure these PRs get attention? Who should they go to? It would be nice if we could have reviewers per dependency, similar to [2], but it looks like it has to be the same person or group for all PRs. What about our existing dependency tracking which sends the dependency update message to dev@ and files JIRAs? Probably we could just leave that on in parallel for now, and consider turning it down later if dependabot is working well. Brian [1] https://docs.github.com/en/code-security/supply-chain-security/keeping-your-dependencies-updated-automatically/configuration-options-for-dependency-updates#reviewers [2] https://github.com/apache/beam/tree/243128a8fc52798e1b58b0cf1a271d95ee7aa241/ownership On Wed, May 12, 2021 at 5:43 AM Ismaël Mejía <ieme...@gmail.com> wrote: > My excuses Brian I had not seen your question: > > > - Will dependabot work ok with the version ranges that we specify? For > example some Python dependencies have upper bounds for the next major > version, some for the next minor version. Is dependabot smart enough to try > bumping the appropriate version number? > > Yes, it does and we can also explicitly set it to ignore certain versions > or a all for each dependency if we don't want to have any PR upgrade for it. > > As a follow up on this I received an email from my Beam fork this morning > reporting a CVE issue on one of the website dependencies, it is a moderate > issue since this is a dep for the website generation, so it won't affect > Beam users) but it is a clear example of the utility of dependabot. > > So the question is how do we proceed? Do I contact INFRA to enable it for > the main repo? and more concretely how do we deal with these PRs in a > practical sense? Do we rename them and create an associated JIRA for > tracking? > > Other opinions? > > Ismaël > > > > On Fri, Apr 16, 2021 at 5:36 PM Brian Hulette <bhule...@google.com> wrote: > >> Yeah I can see the advantage in tooling like this for easy upgrades. I >> suspect many of the outdated Python dependencies fall under this category, >> but the toil of creating a PR and verifying it passes tests is enough of a >> barrier that we just haven't done it. Having a bot create the PR and >> trigger CI to verify it would be helpful IMO. >> >> Some questions/concerns I have: >> - I think many python upgrades will still require manual work: >> - We also have pinned versions for some Python dependencies in >> base_image_requirements.txt [1] >> - We test with multiple major versions of pyarrow. We'd want to add a >> new test environment [2] when bumping to the next major version >> - Will dependabot work ok with the version ranges that we specify? For >> example some Python dependencies have upper bounds for the next major >> version, some for the next minor version. Is dependabot smart enough to try >> bumping the appropriate version number? >> >> Brian >> >> [1] >> https://github.com/apache/beam/blob/master/sdks/python/container/base_image_requirements.txt >> >> [2] >> https://github.com/apache/beam/blob/985e2f095d150261e998f58cf048e48a909d5b2b/sdks/python/tox.ini#L231 >> >> On Fri, Apr 16, 2021 at 7:16 AM Ismaël Mejía <ieme...@gmail.com> wrote: >> >>> Oh forgot to mention one alternative that we do in the Avro project, >>> it is that we don't create issues for the dependabot PRs and then we >>> search all the commits authored by dependabot and include them in the >>> release notes to track dependency upgrades. >>> >>> On Fri, Apr 16, 2021 at 4:02 PM Ismaël Mejía <ieme...@gmail.com> wrote: >>> > >>> > > Quite often, dependency upgrade to latest versions leads to either >>> compilation errors or failed tests and it should be resolved manually or >>> declined. Having this, maybe I miss something, but I don’t see what kind of >>> advantages automatic upgrade will bring to us except that we don’t need to >>> create a PR manually (which is a not big deal). >>> > >>> > The advantage is exactly that, that we don't have to create and track >>> > dependency updates manually, it will be done by the bot and we will >>> > only have to do the review and guarantee that no issues are >>> > introduced. I forgot to mention but we can create exception rules so >>> > no further upgrades will be proposed for some dependencies e.g. >>> > Hadoop, Netty (Java 11 flavor) etc. I forgot to mention another big >>> > advantage that is the detailed security report that will help us >>> > prioritize dependency upgrades. >>> > >>> > > Regarding another issue - it’s already a problem, imho. Since we >>> have a one Jira per package upgrade now and usually it “accumulates” all >>> package upgrades and it’s not closed once upgrade is done, we don’t have a >>> reliable way to notify in release notes about all dependency upgrades for >>> current release. One of the way is to mention the package upgrade in >>> CHANGES.md which seems not very relible because it's quite easy to forget >>> to do. I’d prefer to have a dedicated Jira issue for every upgrade and it >>> will be included into releases notes almost automatically. >>> > >>> > Yes it seems the best for release note tracking to create the issue >>> > and rename the PR title for this, but that would be part of the >>> > review/merge process, so up to the Beam committers to do it >>> > systematically and given how well we respect the commit naming / >>> > squashing rules I am not sure if we will win much by having another >>> > useless rule. >>> > >>> > On Fri, Apr 16, 2021 at 3:24 PM Alexey Romanenko >>> > <aromanenko....@gmail.com> wrote: >>> > > >>> > > Quite often, dependency upgrade to latest versions leads to either >>> compilation errors or failed tests and it should be resolved manually or >>> declined. Having this, maybe I miss something, but I don’t see what kind of >>> advantages automatic upgrade will bring to us except that we don’t need to >>> create a PR manually (which is a not big deal). >>> > > >>> > > Regarding another issue - it’s already a problem, imho. Since we >>> have a one Jira per package upgrade now and usually it “accumulates” all >>> package upgrades and it’s not closed once upgrade is done, we don’t have a >>> reliable way to notify in release notes about all dependency upgrades for >>> current release. One of the way is to mention the package upgrade in >>> CHANGES.md which seems not very relible because it's quite easy to forget >>> to do. I’d prefer to have a dedicated Jira issue for every upgrade and it >>> will be included into releases notes almost automatically. >>> > > >>> > > > On 16 Apr 2021, at 14:15, Ismaël Mejía <ieme...@gmail.com> wrote: >>> > > > >>> > > > Hello, >>> > > > >>> > > > Github has a bot that creates automatically Dependency Update PRs >>> and >>> > > > report security issues called dependabot. >>> > > > >>> > > > I was wondering if we should enable it for Beam. I tested it in my >>> > > > personal Beam fork and it seems to be working well, it created >>> > > > dependency updates for both Python and JS (website) dependencies. >>> > > > The bot seems to be having problems to understand our gradle >>> > > > dependency definitions for Java but that's something we can >>> address in >>> > > > the future to benefit of the updates. Also it did not propose >>> go-lang >>> > > > updates (probably for the same reason). >>> > > > >>> > > > If the community agrees I will create a ticket for INFRA to enable >>> it. >>> > > > We might be getting extra PRs (at the beginning) and we have to be >>> > > > cautious about updates that might have unintended consequences for >>> > > > example we should not merge non stable dependency updates (those >>> > > > ending on -rc1 or -beta on Java) that >>> > > > might be proposed or dependencies that committers are aware we >>> should >>> > > > not update for example projects where their main stable version is >>> not >>> > > > the most recent one like Hadoop or dependencies that do not support >>> > > > our ongoing language target version (e.g. Java 11 only deps). >>> > > > >>> > > > Another issue is that these dependency updates might not get a JIRA >>> > > > associated with them so we need to decide if (1) we create one and >>> > > > rename/associate the PR with it, or (2) we just decide not to have >>> > > > JIRAs for dependency updates. >>> > > > >>> > > > WDYT? other pros/cons that I can be missing? >>> > > > >>> > > > Ismaël >>> > > >>> >>
