lupyuen opened a new issue, #18906: URL: https://github.com/apache/nuttx/issues/18906
Over the past 234 hours: NuttX CI Checks have become much slower, and it's beyond our control. According to our [NuttX Build Monitor](https://github.com/apache/nuttx/issues/18659) at https://lupyuen.github.io/nuttx-github-jobs/build-monitor <img width="2253" height="1849" alt="Image" src="https://github.com/user-attachments/assets/1e100cd6-a764-4299-9c7a-1ab0d5f75bfd"; /> The Top Bar shows that our CI Checks are taking up to __9 hours__ to complete. Normally they complete in __3.5 hours__: https://lupyuen.github.io/nuttx-github-jobs/build-monitor <img width="2090" height="463" alt="Image" src="https://github.com/user-attachments/assets/ec8aa38d-e26b-45f2-aef6-32b92cdef2d2"; /> _Why are our CI Checks so slow? Has something changed in our CI Workflow?_ Let's click into PR 18903. The __Elapsed Duration__ is indeed 8 hour plus (instead of 3.5 hours): https://github.com/apache/nuttx/actions/runs/26104003327 <img width="1857" height="1750" alt="Image" src="https://github.com/user-attachments/assets/f316e6b6-4c3f-41f2-aeb9-4df9bc824d6b"; /> That's the __Elapsed Time__, but it's not the actual __Execution Time__ for building NuttX. To show the Actual GitHub Runner Minutes consumed: Click "Run Details > Usage". This CI Build consumed __30 hours__ of GitHub Runners (1 day 6 hours): https://github.com/apache/nuttx/actions/runs/26104003327/usage <img width="1857" height="1375" alt="Image" src="https://github.com/user-attachments/assets/89276ded-46b5-422c-9f66-dd2a5396d187"; /> And __30 hours__ of GitHub Runners is quite normal for a Complete CI Build. _Why is Elapsed Time so high, when GitHub Runner Time is normal?_ It means that NuttX CI is __waiting for an available GitHub Runner__. NuttX CI wasted time on the waiting, not the execution. Remember that All Apache Projects are sharing the __Same Pool of GitHub Runners?__ It's highly likely that Other Apache Projects are consuming Too Many GitHub Runners, and depriving us of GitHub Runners. _Recently we seem to be running out of GitHub Runners. Why?_ Possibly because Other Apache Projects are overwhelmed by __AI-Generated PRs__? Watch out illogical AI-Generated PRs like this, we shouldn't waste any GitHub Runners on them (should we block them?) - https://github.com/apache/nuttx-apps/pull/3486 _Are we really sure it's not something we changed in NuttX CI?_ Check the NuttX Mirror Repo. Complete CI Builds are still completing in 3 hours, which means that NuttX CI is still working OK. NuttX Mirror Repo uses its own GitHub Runners, not from the Apache Pool of GitHub Runners: https://github.com/NuttX/nuttx/actions <img width="1857" height="1375" alt="Image" src="https://github.com/user-attachments/assets/759c8379-62c5-43cd-a2a2-fe14ba38ab59"; /> _Why is our GitHub Full-Time Runner Metric so high?_ Our own GitHub Full-Time Runner Metric shows that we consumed 30 Full-Time GitHub Runners over the past day, way beyond our quota of 25 Full-Time GitHub Runners: https://lupyuen.github.io/nuttx-metrics/github-fulltime-runners.png <img width="1178" height="891" alt="Image" src="https://github.com/user-attachments/assets/914f3d37-6632-42a8-97a1-6e42e0faecf9"; /> But this number is incorrect. The official report from ASF shows that we consumed only 16 Full-Time GitHub Runners: https://infra-reports.apache.org/#ghactions&hours=24&limit=15&group=name <img width="1528" height="1237" alt="Image" src="https://github.com/user-attachments/assets/febf5760-86ba-4739-ad8e-297fe555b607"; /> _What's wrong with our GitHub Full-Time Runner Metric?_ That's because we don't have access to the Special GitHub APIs needed to extract the Actual Execution Time from each GitHub Job. We only have the Elapsed Time, which is incorrectly inflated, that's why our GitHub Full-Time Runner Metric incorrectly estimates the Execution Time based on the Elapsed Time. In future we might use the [Downloaded GitHub Job Status](https://github.com/lupyuen/nuttx-github-jobs/tree/main/success) to compute more accurately the Actual Execution Time and GitHub Runner Minutes. Be careful when using [PR Review with GitHub Copilot](https://github.com/apache/nuttx/pull/18886#pullrequestreview-4303632900), they will consume our precious GitHub Runners (but ASF doesn't have a report that tells us how many GitHub Runners are used by Copilot sigh) - ["GitHub Copilot code review will start consuming GitHub Actions minutes on June 1, 2026"](https://github.blog/changelog/2026-04-27-github-copilot-code-review-will-start-consuming-github-actions-minutes-on-june-1-2026/) FYI @simbit18 :-) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
