+1 for going by the commits since this is what matters at the end of the day. Also, many issues may not get tagged correctly for a given release due to either the contributor not tagging the issue or due to commits for the issue spanning multiple Beam releases.
For example, For all commits in a given release RC: * If we find a Github issue for the commit: add a notice to the Github issue * Else: add the notice to a generic issue for the release including tags for the commit ID, PR author, and the committer who merged the PR. Thanks, Cham On Mon, Oct 23, 2023 at 11:49 AM Danny McCormick via dev < dev@beam.apache.org> wrote: > I'd probably vote to include both the issue filer and the contributor. It > is pretty equally straightforward - one way to do this would be using all > issues related to that release's milestone and extracting the issue author > and the issue closer. > > This does leave out the (unfortunately sizable) set of contributions that > don't have an associated issue; if we're worried about that, we could > always fall back to anyone with a commit in the last release who doesn't > have an associated issue (aka what I thought we were initially proposing > and what I think Airflow does today). > > I'm pretty much +1 on any sort of automation here, and it certainly can > come in stages :) > > On Mon, Oct 23, 2023 at 1:50 PM Johanna Öjeling via dev < > dev@beam.apache.org> wrote: > >> Yes that's a good point to include also those who created the issue. >> >> On Mon, Oct 23, 2023, 19:18 Robert Bradshaw via dev <dev@beam.apache.org> >> wrote: >> >>> On Mon, Oct 23, 2023 at 7:26 AM Danny McCormick via dev < >>> dev@beam.apache.org> wrote: >>> >>>> So to summarize, I think there's broad consensus (or at least lazy >>>> consensus) around the following: >>>> >>>> - (1) Updating our release email/guidelines to be more specific about >>>> what we mean by release validation/how to be helpful during this process. >>>> This includes both encouraging validation within each user's own code base >>>> and encouraging people to document/share their process of validation and >>>> link it in the release spreadsheet. >>>> - (2) Doing something like what Airflow does (#29424 >>>> <https://github.com/apache/airflow/issues/29424>) and creating an >>>> issue asking people who have contributed to the current release to help >>>> validate their changes. >>>> >>>> I'm also +1 on doing both of these. The first bit (updating our >>>> guidelines) is relatively easy - it should just require updating >>>> https://github.com/apache/beam/blob/master/contributor-docs/release-guide.md#vote-and-validate-the-release-candidate >>>> . >>>> >>>> I took a look at the second piece (copying what Airflow does) to see if >>>> we could just copy their automation, but it looks like it's tied to >>>> airflow breeze >>>> <https://github.com/apache/airflow/blob/main/dev/breeze/src/airflow_breeze/provider_issue_TEMPLATE.md.jinja2> >>>> (their repo-specific automation tooling), so we'd probably need to build >>>> the automation ourselves. It shouldn't be terrible, basically we'd want a >>>> GitHub Action that compares the current release tag with the last release >>>> tag, grabs all the commits in between, parses them to get the author, and >>>> creates an issue with that data, but it does represent more effort than >>>> just updating a markdown file. There might even be an existing Action that >>>> can help with this, I haven't looked too hard. >>>> >>> >>> I was thinking along the lines of a script that would scrape the issues >>> resolved in a given release and add a comment to them noting that the >>> change is in release N and encouraging (with clear instructions) how this >>> can be validated. Creating a "validate this release" issue with all >>> "contributing" participants could be an interesting way to do this as well. >>> (I think it'd be valuable to get those who filed the issue, not just those >>> who fixed it, to validate.) >>> >>> >>>> As our next release manager, I'm happy to review PRs for either of >>>> these if anyone wants to volunteer to help out. If not, I'm happy to update >>>> the guidelines, but I probably won't have time to add the commit inspection >>>> tooling (I'm planning on throwing any extra time towards continuing to >>>> automate release candidate creation which is currently a more impactful >>>> problem IMO). I would very much like it if both of these things happened >>>> though :) >>>> >>>> Thanks, >>>> Danny >>>> >>>> On Mon, Oct 23, 2023 at 10:05 AM XQ Hu <x...@google.com> wrote: >>>> >>>>> +1. This is a great idea to try. @Danny McCormick >>>>> <dannymccorm...@google.com> FYI as our next release manager. >>>>> >>>>> On Wed, Oct 18, 2023 at 2:30 PM Johanna Öjeling via dev < >>>>> dev@beam.apache.org> wrote: >>>>> >>>>>> When I have contributed to Apache Airflow, they have tagged all >>>>>> contributors concerned in a GitHub issue when the RC is available and >>>>>> asked >>>>>> us to validate it. Example: #29424 >>>>>> <https://github.com/apache/airflow/issues/29424>. >>>>>> >>>>>> I found that to be an effective way to notify contributors of the RC >>>>>> and nudge them to help out. In the issue description there is a reference >>>>>> to the guidelines on how to test the RC and a note that people are >>>>>> encouraged to vote on the mailing list (which could admittedly be more >>>>>> highlighted because I did not pay attention to it until now and was >>>>>> unaware >>>>>> that contributors had a vote). >>>>>> >>>>>> It might be an idea to consider something similar here to increase >>>>>> the participation? >>>>>> >>>>>> On Tue, Oct 17, 2023 at 7:01 PM Jack McCluskey via dev < >>>>>> dev@beam.apache.org> wrote: >>>>>> >>>>>>> I'm +1 on helping explain what we mean by "validate the RC" since >>>>>>> we're really just asking users to see if their existing use cases work >>>>>>> along with our typical slate of tests. I don't know if offloading that >>>>>>> work >>>>>>> to our active validators is the right approach though, >>>>>>> documentation/screen >>>>>>> share of their specific workflow is definitely less useful than having a >>>>>>> more general outline of how to install the RC and things to look out for >>>>>>> when testing. >>>>>>> >>>>>>> On Tue, Oct 17, 2023 at 12:55 PM Austin Bennett <aus...@apache.org> >>>>>>> wrote: >>>>>>> >>>>>>>> Great effort. I'm also interested in streamlining releases -- so >>>>>>>> if there are alot of manual tests that could be automated, would be >>>>>>>> great >>>>>>>> to discover and then look to address. >>>>>>>> >>>>>>>> On Tue, Oct 17, 2023 at 8:47 AM Robert Bradshaw via dev < >>>>>>>> dev@beam.apache.org> wrote: >>>>>>>> >>>>>>>>> +1 >>>>>>>>> >>>>>>>>> I would also strongly suggest that people try out the release >>>>>>>>> against their own codebases. This has the benefit of ensuring the >>>>>>>>> release >>>>>>>>> won't break your own code when they go out, and stress-tests the new >>>>>>>>> code >>>>>>>>> against real-world pipelines. (Ideally our own tests are all passing, >>>>>>>>> and >>>>>>>>> this validation is automated as much as possible (though ensuring it >>>>>>>>> matches our documentation and works in a clean environment still has >>>>>>>>> value), but there's a lot of code and uses out there that we don't >>>>>>>>> have >>>>>>>>> access to during normal Beam development.) >>>>>>>>> >>>>>>>>> On Tue, Oct 17, 2023 at 8:21 AM Svetak Sundhar via dev < >>>>>>>>> dev@beam.apache.org> wrote: >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> I’ve participated in RC testing for a few releases and have >>>>>>>>>> observed a bit of a knowledge gap in how releases can be tested. >>>>>>>>>> Given that >>>>>>>>>> Beam encourages contributors to vote on RC’s regardless of tenure, >>>>>>>>>> and that >>>>>>>>>> voting on an RC is a relatively low-effort, high leverage way to >>>>>>>>>> influence >>>>>>>>>> the release of the library, I propose the following: >>>>>>>>>> >>>>>>>>>> During the vote for the next release, voters can document the >>>>>>>>>> process they followed on a separate document, and add the link on >>>>>>>>>> column G >>>>>>>>>> here >>>>>>>>>> <https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=437054928>. >>>>>>>>>> One step further, could be a screencast of running the test, and >>>>>>>>>> attaching >>>>>>>>>> a link of that. >>>>>>>>>> >>>>>>>>>> We can keep repeating this through releases until we have >>>>>>>>>> documentation for many of the different tests. We can then add these >>>>>>>>>> docs >>>>>>>>>> into the repo. >>>>>>>>>> >>>>>>>>>> I’m proposing this because I’ve gathered the following feedback >>>>>>>>>> from colleagues that are tangentially involved with Beam: They are >>>>>>>>>> interested in participating in release validation, but don’t know >>>>>>>>>> how to >>>>>>>>>> get started. Happy to hear other suggestions too, if there are any to >>>>>>>>>> address the above. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Svetak Sundhar >>>>>>>>>> >>>>>>>>>> Data Engineer >>>>>>>>>> s <nellywil...@google.com>vetaksund...@google.com >>>>>>>>>> >>>>>>>>>>
