Thanks for the feedback! More than happy if I could implement these options and integrations. I will look into the current implementation and draft PRs by the upcoming week.
Best, Kevin Yang On Wed, Mar 11, 2026 at 4:05 AM Jarek Potiuk <[email protected]> wrote: > You can absolutely add the option to use any agent or model to the tool I > created. Currently it can use copilot, Claude, codex - but you can add PR > to use any model - it is build for that purpose. > > This is integrated with breeze uatctually even automatically stores which > model you use and continue using it. The interface to LLm ia super Simple. > It does not even use Pydantic AI - it just generates prompt and parses the > output. so by all means - adding a way to use any other LLM. > > 90% of the work done by the tool is deterministic; it only asks the LLM > when it is in doubt. > > So - by all means, PRs to use any other LLMs - whether local or remote - > are most welcome. Also we can add opencode and ollama integration > > [image: image.png] > > J. > > On Wed, Mar 11, 2026, 03:32 Kevin Yang <[email protected]> wrote: > >> Hi Jarek, >> >> Thank you very much for all the efforts in building the solutions. I >> recently also read through the following discussions [1,2,3], and think >> about whether there is a good approach on tackling the challenge. >> >> I believe integrating with LLM is a good approach, especially can leverage >> its reasoning capabilities to provide a better triage. Existing products >> such as Copilot Code Review can also provide insightful triage as >> previously proposed by Kaxil. >> >> I also find another direction that also looks promising to me is to >> use a *small >> language model (SLM)*, a model with 2-4 B parameters, which can be run on >> standard Github runners, using CPU-only, to triage issues and PRs. I've >> built a github action *SLM Triage* ( >> https://github.com/marketplace/actions/slm-triage). >> >> What advantages does SLM offer? >> * It can be run on a standard GitHub runner, on CPU, and finish execution >> in around 3 - 5 minutes >> * There is no API cost, billing set up with LLM service >> * It runs on GitHub events, when an issue or PR is opened, and capable to >> triage issues as long as there are GitHub runners available >> * It can be simply integrated into GitHub Actions without infrastructure, >> or local setup. >> >> What are the current limitations? >> * It doesn't have enough domain knowledge about a specific codebase, so it >> can only triage based on high-level context, and relevancy between context >> information and code changes >> * It has limited reasoning capability >> * It has limited context window (128k context window size, some might have >> ~256k) >> >> Why I think it can be a potential direction >> * I feel some issues or PRs can be triage based on some basic heuristics >> and rules >> * Even though context window is limited, if the process is triggered when >> issue opened, the context window is good enough to capture issue >> description, pr description, and even code change >> * It is easier to set up for a broader open-source community, and probably >> more cost efficient, it can scale based on workflow adoption >> * It can take action through API such as comment on an issue, add label, >> close an issue or PR, etc. based on the triage result. >> >> I also attempted to triage multiple issues and PRs on airflow repository, >> and check the actual issues/PRs (I created a script to dry-run and inspect >> the triage result and reasoning). The result looks promising, but >> sometimes >> I found it is "a bit strict" and needs some improvements in terms of >> prompting. >> >> I wonder if this is a valid idea, but it would be great if the idea can >> potentially help. >> >> Thanks, >> Kevin Yang >> >> [1] https://github.com/orgs/community/discussions/185387 >> [2] https://github.com/ossf/wg-vulnerability-disclosures/issues/178 >> [3] >> >> https://www.reddit.com/r/opensource/comments/1q3f89b/open_source_is_being_ddosed_by_ai_slop_and_github/#:~:text=FunBrilliant5713-,Open%20source%20is%20being%20DDoSed%20by%20AI%20slop%20and%20GitHub,which%20submissions%20came%20from%20Copilot >> . >> >> On Tue, Mar 10, 2026 at 9:13 PM Jarek Potiuk <[email protected]> wrote: >> >> > Just to update everyone: I've auto-triaged a bunch of PRs—the tool works >> > very well IMHO, but we will know after the authors see them and review >> > >> > Some stats (I will gather more in the next days as I am adding timing >> and >> > further improvements): >> > >> > * I triaged about 100 PRs in under an hour of elapsed time (I >> > also corrected, improved and noted some fixes, so it will be faster) >> > * I converted 30 of those into Drafts and closed a few >> > * I have not marked any as ready to review yet, but I will do that >> tomorrow >> > * The LLM (Claude) assessment is quite fast - faster than I thought. >> > Parallelizing it also helps. LLM assessment takes between 20 s and 2 >> > minutes (elapsed), but usually, only a few pull requests (15% or less) >> are >> > LLM assessed in a batch, so this is not a bottleneck. I will also >> modify >> > the tool to start reviewing deterministic things before LLMs complete - >> > which should speed up the whole process even more >> > * The LLM assessments are pretty good - but a few were significantly >> wrong >> > and I would not post them. It's good we have Human-In-The-Loop and in >> the >> > driver's seat. >> > >> > Overall - I think the tool is doing very well what I wanted. But let's >> see >> > the improvements over the next few days, observe how authors react, and >> > determine if it can actually help maintainers >> > >> > I added a few PRs as improvements; looking forward to reviews, : >> > >> > * https://github.com/apache/airflow/pull/63318 >> > * https://github.com/apache/airflow/pull/63317 >> > * https://github.com/apache/airflow/pull/63315 >> > * https://github.com/apache/airflow/pull/63319 >> > * https://github.com/apache/airflow/pull/63320 >> > >> > J. >> > >> > >> > >> > On Tue, Mar 10, 2026 at 10:18 PM Jarek Potiuk <[email protected]> wrote: >> > >> > > Lazy consensus reached. I will try it out tonight. I added more >> signals >> > > (unresolved review comments) and filtering options ( >> > > https://github.com/apache/airflow/pull/63300) that will be useful >> during >> > > this phase. >> > > >> > > On Fri, Mar 6, 2026 at 9:08 PM Jarek Potiuk <[email protected]> wrote: >> > > >> > >> Hello here, >> > >> >> > >> I am asking a lazy consensus on the approach proposed in >> > >> https://lists.apache.org/thread/ly6lrm2gc4p7p54vomr8621nmb1pvlsk >> > >> regarding our approach to triaging PRs. >> > >> >> > >> The lazy consensus will last till Tuesday 10 pm CEST ( >> > >> >> > >> https://www.timeanddate.com/countdown/generic?iso=20260310T22&p0=262&font=cursive >> > >> ) >> > >> >> > >> Summary of the proposal >> > >> >> > >> This is the proposed update to the PR contributing guidelines: >> > >> >> > >> > Start with **Draft**: Until you are sure that your PR passes all >> the >> > >> quality checks and tests, keep it in **Draft** status. This will >> signal >> > to >> > >> maintainers that the PR is not yet ready for review and it will >> prevent >> > >> maintainers from accidentally merging it before it's ready. Once you >> are >> > >> sure that your PR is ready for review, you can mark it as "Ready for >> > >> review" in the GitHub UI. Our regular check will convert all PRs from >> > >> non-collaborators that do not pass our quality gates to Draft status, >> > so if >> > >> you see that your PR is in Draft status and you haven't set it to >> Draft. >> > >> Check the comments to see what needs to be fixed. >> > >> >> > >> That's a "broad" description of the process; details will be worked >> out >> > >> while testing the solution. >> > >> >> > >> The PR: https://github.com/apache/airflow/pull/62682 >> > >> >> > >> My testing approach is to start with individual areas, update and >> > perfect >> > >> the tool, gradually increase the reach of it and engage others - >> then we >> > >> might think about more regular process involving more maintainers. >> > >> >> > >> J. >> > >> >> > > >> > >> >
