I review a decent number of PRs for Apache Beam, and I've built some of my own tooling to help keep track of open PRs. I wrote a script that pulls metadata about all relevant PRs and uses some heuristics to categorize them into: - incoming review - outgoing review - "CC'd" - where I've been mentioned but am not the reviewer or author
In the first two cases I try to highlight the ones that need my attention, simply by detecting if I'm the person who took the most recent action or not. This works reasonably well but gets tripped up on several edge cases: 1) The author might push multiple commits before they're actually ready for more feedback. 2) A PR might need feedback from multiple reviewers (e.g. people with domain knowledge of certain areas). I've been planning to make my script stateful so that I can mark a PR as "not my turn" (i.e. unhighlight this until there is more activity), and maybe "never my turn" (i.e. I've finished reviewing this, it's waiting on someone else), to handle these cases. The idea of a "Addressing Feedback" -> "Waiting on Review" label that is automatically transitioned when there is activity would run into these same edge cases. If a reviewer had the ability to bump the label back to "Addressing Feedback", that would at least address #1. I think Wes's proposal (a read-only web UI) would likely also run into these edge cases since it stores no state of its own to deconflict in those situations. Brian On Tue, Jun 29, 2021 at 6:26 AM Wes McKinney <wesmck...@gmail.com> wrote: > On Tue, Jun 29, 2021 at 3:10 PM Andrew Lamb <al...@influxdata.com> wrote: > > > > The thing that would make me more efficient reviewing PRs is figuring out > > which one of the open reviews are ready for additional feedback. > > Yes, I think this would be the single most significant quality-of-life > improvement for reviewers. > > > I think the idea of a webapp or something that shows active reviews would > > be helpful (though I get most of that from appropriate email filters). > > > > What about a system involving labels (for which there is already a basic > > GUI in github)? Something low tech like > > > > (Waiting for Review) > > (Addressing Feedback) > > (Approved, waiting for Merge) > > > > With maybe some automation prompting people to add the "Waiting on > Review" > > label when they want feedback > > I think it would have to be a bot that automatically sets the labels. > If it requires contributors to take some action outside of pushing new > work (new commits or a rebased version of the patch) to the PR and > leaving responses to comments on the PR, the system is likely to fail > some non-trivial percentage of the time. > Given the quality of off-the-shelf web app components nowadays (e.g. > https://material-ui.com), throwing together a read-only PR dashboard > that shows what has changed since you last interacted with them (along > with some other helpful things, like whether the build is passing) is > "probably" not a super heavy lift. I haven't done any frontend > development in years so while the backend part (writing Python code to > wrangle data from GitHub's REST API and put it in a SQLite database) > wouldn't take very long I would need some help on the front end > portion and setting it up for deployment on DigitalOcean or somewhere. > > > Andrew > > > > On Tue, Jun 29, 2021 at 4:28 AM Wes McKinney <wesmck...@gmail.com> > wrote: > > > > > hi folks, > > > > > > I've noted that the volume of PRs for Arrow has been steadily > > > increasing (and will likely continue to increase), and while I've > > > personally had less time for development / maintenance / code reviews > > > over the last year, I would like to have a discussion about what we > > > could do to improve our tooling for maintainers to optimize the > > > efficiency of time spent tending to the PR queue. In my own > > > experience, I have felt that I have wasted a lot of time digging > > > around the queue looking for PRs that are awaiting feedback or need to > > > be merged. > > > > > > I note first of all that around 70 out of 173 open PRs have been > > > updated in the last 7 days, so while there is some PR staleness, to > > > have nearly half of the PRs active is pretty good. That said, ~70 > > > active PRs is a lot of PRs to tend to. > > > > > > I scraped the project's code review comment history, and here are the > > > individuals who have left the most comments on PRs since genesis > > > > > > pitrou 6802 > > > wesm 5023 > > > emkornfield 3032 > > > bkietz 2834 > > > kou 1489 > > > nealrichardson 1439 > > > fsaintjacques 1356 > > > kszucs 1250 > > > alamb 1133 > > > jorisvandenbossche 1094 > > > liyafan82 831 > > > lidavidm 816 > > > westonpace 794 > > > xhochy 770 > > > nevi-me 643 > > > BryanCutler 639 > > > jorgecarleitao 635 > > > cpcloud 551 > > > sunchao 536 > > > ianmcook 499 > > > > > > Since we're probably stuck using GitHub to receive code contributions > > > (as opposed to systems — Gerrit is one I'm familiar with — that > > > provide more structure for reviewers to track the patches they "own" > > > as well as the outgoing/incoming state of reviews), I am wondering > > > what kinds of tools we could create to make it easier for maintainers > > > to keep track of PRs they are shepherding through the contribution > > > process. Ideally this wouldn't involve maintainers having to engage in > > > some explicit action like assigning themselves as a PR reviewer. > > > > > > Here's one idea: a web application that displays "your reviews", a > > > table of PRs that you have interacted with in any way (commented, left > > > code review, assigned as reviewer, someone mentioned you, etc.) sorted > > > either by last commit or last comment to assess "freshness". So if you > > > comment on a PR or leave a code review, it will automatically show up > > > in "your reviews". It could also indicate whether there has been > > > activity on the PR since the last time you interacted with it. > > > > > > Having now used the GitHub API to pull comments from PRs for the above > > > analysis, there is certainly enough information available to help > > > create this kind of tool. I'd be willing to contribute to building the > > > backend of such a web application. > > > > > > This is just one idea, but I am curious to hear from others who are > > > spending a lot of time doing code review / PR merging to see what > > > might help them use their time more effectively. > > > > > > Thanks, > > > Wes > > > >
