Re: Scala Steward for dependency updates?
Sounds like there are no objections, and some additional good reasons I didn't mention to use this tool. I'll make the pull request to add apache/daffodil as a repo for Scala Steward to track. I'd recommend creating a ticket for any updates you think would be useful, such as the versioning scheme. We don't want to lose track of these kinds of ideas. On 3/4/21 2:26 PM, Interrante, John A (GE Research, US) wrote: > P.S. I've been meaning to pass along another useful Scala ecosystem article > I read recently [1]. Library writers are being asked to define a `ThisBuild > / versionScheme` setting to declare a library's semantic versioning scheme > (like Daffodil's in our case) so that sbt can issue more reliable (no false > positives) warnings about conflicts between incompatible versions of > transitive library dependencies. I think we should define this setting to > tell downstream applications which semantic versioning scheme we use for > Daffodil's version numbers. > > [1] > https://scala-lang.org/blog/2021/02/16/preventing-version-conflicts-with-versionscheme.html > > -Original Message- > From: Steve Lawrence > Sent: Thursday, March 4, 2021 1:10 PM > To: dev@daffodil.apache.org > Subject: EXT: Scala Steward for dependency updates? > > I just stumbled across Scala Steward [1]. From their website, "Scala Steward > is a bot that helps you keep library dependencies and sbt plugins up-to-date." > > This bot periodical checks to see if there are any newer versions of > dependencies, and if detected will create a pull request to update that > dependency in the project/Dependencies.scala file. > > I've enabled it on my fork as a test, and it just created a bunch of pull > requests, so you can see what it looks like at my fork: > > https://github.com/stevedlawrence/daffodil/pulls > > The benefit here is we can rely on this bot to keep our deps updated so we > don't fall behind, and can rely on our GitHub actions to test if anything > breaks for a particular dependency. So much of the process becomes automated. > > Some parts are still manual, like checking that the license for the > dependency hasn't changed, and updating the bin.NOTICE file which mentions > library versions, so there's still some work. But it it at least automates > part of the process. > > It also has a config file if needed to do things like pin certain > dependencies to a version if needed, configure pull request messages, etc. My > fork above just uses the default configuration. > > If we do want to enable this, all we need to do is create a pull request to > add "apache/daffodil" to to scala steward repo's file--pretty simple change. > > > Thoughts? > > [1] https://github.com/scala-steward-org/scala-steward >
RE: Scala Steward for dependency updates?
P.S. I've been meaning to pass along another useful Scala ecosystem article I read recently [1]. Library writers are being asked to define a `ThisBuild / versionScheme` setting to declare a library's semantic versioning scheme (like Daffodil's in our case) so that sbt can issue more reliable (no false positives) warnings about conflicts between incompatible versions of transitive library dependencies. I think we should define this setting to tell downstream applications which semantic versioning scheme we use for Daffodil's version numbers. [1] https://scala-lang.org/blog/2021/02/16/preventing-version-conflicts-with-versionscheme.html -Original Message- From: Steve Lawrence Sent: Thursday, March 4, 2021 1:10 PM To: dev@daffodil.apache.org Subject: EXT: Scala Steward for dependency updates? I just stumbled across Scala Steward [1]. From their website, "Scala Steward is a bot that helps you keep library dependencies and sbt plugins up-to-date." This bot periodical checks to see if there are any newer versions of dependencies, and if detected will create a pull request to update that dependency in the project/Dependencies.scala file. I've enabled it on my fork as a test, and it just created a bunch of pull requests, so you can see what it looks like at my fork: https://github.com/stevedlawrence/daffodil/pulls The benefit here is we can rely on this bot to keep our deps updated so we don't fall behind, and can rely on our GitHub actions to test if anything breaks for a particular dependency. So much of the process becomes automated. Some parts are still manual, like checking that the license for the dependency hasn't changed, and updating the bin.NOTICE file which mentions library versions, so there's still some work. But it it at least automates part of the process. It also has a config file if needed to do things like pin certain dependencies to a version if needed, configure pull request messages, etc. My fork above just uses the default configuration. If we do want to enable this, all we need to do is create a pull request to add "apache/daffodil" to to scala steward repo's file--pretty simple change. Thoughts? [1] https://github.com/scala-steward-org/scala-steward
RE: Scala Steward for dependency updates?
I like the idea of using the Scala Stewart bot to keep Daffodil's library dependencies and sbt plugins up-to-date very much. In fact, I noticed two more benefits you didn't mention due to a significant part of the Scala ecosystem already using the Scala Stewart bot: [1] https://github.com/scala-steward-org/scala-steward/blob/master/docs/scalafix-migrations.md Some Scala libraries have written Scalafix rewrite rules for certain version updates which require code changes in addition to bumps in version numbers. If the bot knows about these rules (there's a config file contributors already have added to), the bot can automatically rewrite code that would not compile with the new version of the dependency. [2] https://github.com/scala-steward-org/scala-steward/blob/master/docs/artifact-migrations.md Some dependencies change their group ids, artifact ids, or both, across some version updates. If the bot knows about these artifact migration rules (again, there's a config file contributors already have added to), the bot can migrate dependencies to newer versions with different group/artifact ids automatically. We can add Daffodil migration rules to this config file ourselves if we change our Daffodil modules' names in the future. As Mike said, we need to take precautions against software supply chain hacks. The bot signs commits with a PGP key which has a known fingerprint and can be found at a known URL. Even if we can automate that signature check, we still should require two committers to review and approve each pull request (while manually updating files like bin.NOTICE by pushing another commit to the pull request, making sure to squash commits) before we merge the bot's pull requests. I agree that we're safer reviewing each dependency update one at a time (making sure to wait for CI checks to pass) than batching them together in warehouse-size chunks too. FYI, the bot makes version bumps conservatively. If there are newer patch, minor, and major versions available, the bot will propose an update to the latest patch version at the same major and minor version. If the dependency is on the latest patch version next time, then the bot will propose an update to the latest minor version at the same major version. Only when the dependency is on the latest minor version at the same major version, does the bot finally propose an update to the latest major version. FYI, once we merge a dependency update, the bot may send us another pull request making another update for the same dependency but it's a good thing to be conservative like that. John -Original Message- From: Steve Lawrence Sent: Thursday, March 4, 2021 1:10 PM To: dev@daffodil.apache.org Subject: EXT: Scala Steward for dependency updates? I just stumbled across Scala Steward [1]. From their website, "Scala Steward is a bot that helps you keep library dependencies and sbt plugins up-to-date." This bot periodical checks to see if there are any newer versions of dependencies, and if detected will create a pull request to update that dependency in the project/Dependencies.scala file. I've enabled it on my fork as a test, and it just created a bunch of pull requests, so you can see what it looks like at my fork: https://github.com/stevedlawrence/daffodil/pulls The benefit here is we can rely on this bot to keep our deps updated so we don't fall behind, and can rely on our GitHub actions to test if anything breaks for a particular dependency. So much of the process becomes automated. Some parts are still manual, like checking that the license for the dependency hasn't changed, and updating the bin.NOTICE file which mentions library versions, so there's still some work. But it it at least automates part of the process. It also has a config file if needed to do things like pin certain dependencies to a version if needed, configure pull request messages, etc. My fork above just uses the default configuration. If we do want to enable this, all we need to do is create a pull request to add "apache/daffodil" to to scala steward repo's file--pretty simple change. Thoughts? [1] https://github.com/scala-steward-org/scala-steward
Re: Scala Steward for dependency updates?
There's been this increase of software supply chain hacks these days. Since this bot is on our ongoing development branch, and so long as we watch these changes and verify the dependencies it chooses before merging these PRs, then this makes this version updating an incremental effort done as things come along. We would inspect and merge the pull requests individually. That's far better than the hated job to do this in bulk and verify just before a release. I like the notion that these updates occur as early as possible, so developers get experience with the new versions over time. I very much like that they're one library at a time per pull request. Bite sized unit of work. From: Steve Lawrence Sent: Thursday, March 4, 2021 1:10 PM To: dev@daffodil.apache.org Subject: Scala Steward for dependency updates? I just stumbled across Scala Steward [1]. From their website, "Scala Steward is a bot that helps you keep library dependencies and sbt plugins up-to-date." This bot periodical checks to see if there are any newer versions of dependencies, and if detected will create a pull request to update that dependency in the project/Dependencies.scala file. I've enabled it on my fork as a test, and it just created a bunch of pull requests, so you can see what it looks like at my fork: https://github.com/stevedlawrence/daffodil/pulls The benefit here is we can rely on this bot to keep our deps updated so we don't fall behind, and can rely on our GitHub actions to test if anything breaks for a particular dependency. So much of the process becomes automated. Some parts are still manual, like checking that the license for the dependency hasn't changed, and updating the bin.NOTICE file which mentions library versions, so there's still some work. But it it at least automates part of the process. It also has a config file if needed to do things like pin certain dependencies to a version if needed, configure pull request messages, etc. My fork above just uses the default configuration. If we do want to enable this, all we need to do is create a pull request to add "apache/daffodil" to to scala steward repo's file--pretty simple change. Thoughts? [1] https://github.com/scala-steward-org/scala-steward
Scala Steward for dependency updates?
I just stumbled across Scala Steward [1]. From their website, "Scala Steward is a bot that helps you keep library dependencies and sbt plugins up-to-date." This bot periodical checks to see if there are any newer versions of dependencies, and if detected will create a pull request to update that dependency in the project/Dependencies.scala file. I've enabled it on my fork as a test, and it just created a bunch of pull requests, so you can see what it looks like at my fork: https://github.com/stevedlawrence/daffodil/pulls The benefit here is we can rely on this bot to keep our deps updated so we don't fall behind, and can rely on our GitHub actions to test if anything breaks for a particular dependency. So much of the process becomes automated. Some parts are still manual, like checking that the license for the dependency hasn't changed, and updating the bin.NOTICE file which mentions library versions, so there's still some work. But it it at least automates part of the process. It also has a config file if needed to do things like pin certain dependencies to a version if needed, configure pull request messages, etc. My fork above just uses the default configuration. If we do want to enable this, all we need to do is create a pull request to add "apache/daffodil" to to scala steward repo's file--pretty simple change. Thoughts? [1] https://github.com/scala-steward-org/scala-steward
