Hi Jarek, Thank you very much for all the efforts in building the solutions. I recently also read through the following discussions [1,2,3], and think about whether there is a good approach on tackling the challenge.
I believe integrating with LLM is a good approach, especially can leverage its reasoning capabilities to provide a better triage. Existing products such as Copilot Code Review can also provide insightful triage as previously proposed by Kaxil. I also find another direction that also looks promising to me is to use a *small language model (SLM)*, a model with 2-4 B parameters, which can be run on standard Github runners, using CPU-only, to triage issues and PRs. I've built a github action *SLM Triage* ( https://github.com/marketplace/actions/slm-triage). What advantages does SLM offer? * It can be run on a standard GitHub runner, on CPU, and finish execution in around 3 - 5 minutes * There is no API cost, billing set up with LLM service * It runs on GitHub events, when an issue or PR is opened, and capable to triage issues as long as there are GitHub runners available * It can be simply integrated into GitHub Actions without infrastructure, or local setup. What are the current limitations? * It doesn't have enough domain knowledge about a specific codebase, so it can only triage based on high-level context, and relevancy between context information and code changes * It has limited reasoning capability * It has limited context window (128k context window size, some might have ~256k) Why I think it can be a potential direction * I feel some issues or PRs can be triage based on some basic heuristics and rules * Even though context window is limited, if the process is triggered when issue opened, the context window is good enough to capture issue description, pr description, and even code change * It is easier to set up for a broader open-source community, and probably more cost efficient, it can scale based on workflow adoption * It can take action through API such as comment on an issue, add label, close an issue or PR, etc. based on the triage result. I also attempted to triage multiple issues and PRs on airflow repository, and check the actual issues/PRs (I created a script to dry-run and inspect the triage result and reasoning). The result looks promising, but sometimes I found it is "a bit strict" and needs some improvements in terms of prompting. I wonder if this is a valid idea, but it would be great if the idea can potentially help. Thanks, Kevin Yang [1] https://github.com/orgs/community/discussions/185387 [2] https://github.com/ossf/wg-vulnerability-disclosures/issues/178 [3] https://www.reddit.com/r/opensource/comments/1q3f89b/open_source_is_being_ddosed_by_ai_slop_and_github/#:~:text=FunBrilliant5713-,Open%20source%20is%20being%20DDoSed%20by%20AI%20slop%20and%20GitHub,which%20submissions%20came%20from%20Copilot . On Tue, Mar 10, 2026 at 9:13 PM Jarek Potiuk <[email protected]> wrote: > Just to update everyone: I've auto-triaged a bunch of PRs—the tool works > very well IMHO, but we will know after the authors see them and review > > Some stats (I will gather more in the next days as I am adding timing and > further improvements): > > * I triaged about 100 PRs in under an hour of elapsed time (I > also corrected, improved and noted some fixes, so it will be faster) > * I converted 30 of those into Drafts and closed a few > * I have not marked any as ready to review yet, but I will do that tomorrow > * The LLM (Claude) assessment is quite fast - faster than I thought. > Parallelizing it also helps. LLM assessment takes between 20 s and 2 > minutes (elapsed), but usually, only a few pull requests (15% or less) are > LLM assessed in a batch, so this is not a bottleneck. I will also modify > the tool to start reviewing deterministic things before LLMs complete - > which should speed up the whole process even more > * The LLM assessments are pretty good - but a few were significantly wrong > and I would not post them. It's good we have Human-In-The-Loop and in the > driver's seat. > > Overall - I think the tool is doing very well what I wanted. But let's see > the improvements over the next few days, observe how authors react, and > determine if it can actually help maintainers > > I added a few PRs as improvements; looking forward to reviews, : > > * https://github.com/apache/airflow/pull/63318 > * https://github.com/apache/airflow/pull/63317 > * https://github.com/apache/airflow/pull/63315 > * https://github.com/apache/airflow/pull/63319 > * https://github.com/apache/airflow/pull/63320 > > J. > > > > On Tue, Mar 10, 2026 at 10:18 PM Jarek Potiuk <[email protected]> wrote: > > > Lazy consensus reached. I will try it out tonight. I added more signals > > (unresolved review comments) and filtering options ( > > https://github.com/apache/airflow/pull/63300) that will be useful during > > this phase. > > > > On Fri, Mar 6, 2026 at 9:08 PM Jarek Potiuk <[email protected]> wrote: > > > >> Hello here, > >> > >> I am asking a lazy consensus on the approach proposed in > >> https://lists.apache.org/thread/ly6lrm2gc4p7p54vomr8621nmb1pvlsk > >> regarding our approach to triaging PRs. > >> > >> The lazy consensus will last till Tuesday 10 pm CEST ( > >> > https://www.timeanddate.com/countdown/generic?iso=20260310T22&p0=262&font=cursive > >> ) > >> > >> Summary of the proposal > >> > >> This is the proposed update to the PR contributing guidelines: > >> > >> > Start with **Draft**: Until you are sure that your PR passes all the > >> quality checks and tests, keep it in **Draft** status. This will signal > to > >> maintainers that the PR is not yet ready for review and it will prevent > >> maintainers from accidentally merging it before it's ready. Once you are > >> sure that your PR is ready for review, you can mark it as "Ready for > >> review" in the GitHub UI. Our regular check will convert all PRs from > >> non-collaborators that do not pass our quality gates to Draft status, > so if > >> you see that your PR is in Draft status and you haven't set it to Draft. > >> Check the comments to see what needs to be fixed. > >> > >> That's a "broad" description of the process; details will be worked out > >> while testing the solution. > >> > >> The PR: https://github.com/apache/airflow/pull/62682 > >> > >> My testing approach is to start with individual areas, update and > perfect > >> the tool, gradually increase the reach of it and engage others - then we > >> might think about more regular process involving more maintainers. > >> > >> J. > >> > > >
