Dev-iL commented on PR #76279:
URL: https://github.com/apache/airflow/pull/76279#issuecomment-6625505436

   Thank you for taking my concerns seriously. Let me narrow what I'm worried 
about.
   
   > Each runner runs one job at a time. GitHub's docs say that [each standard 
GitHub-hosted runner is a new virtual 
machine](https://docs.github.com/en/actions/concepts/runners/github-hosted-runners#overview-of-github-hosted-runners),
 and that [each runner can run a single job at a 
time](https://docs.github.com/en/actions/get-started/understand-github-actions#runners).
 RunsOn's docs say that RunsOn starts [a fresh, ephemeral EC2 runner for every 
job](https://runs-on.com/docs/). In the PROD build job, no other step runs at 
the same time as the export step. Also, with -T0, zstd tries to detect the 
number of physical CPU cores and uses that many threads. So zstd should not 
start more threads than the machine has cores.
   
   "One VM per job" isn't "one set of physical cores per job". vCPUs are 
time-sliced onto host hardware shared with other tenants, and `-T0` sizes its 
worker pool from the topology the guest is *told* about, not from what the host 
can actually deliver at that moment. If the host is oversubscribed, the guest 
sees N runnable cores and spawns N workers, but the hypervisor can only 
time-slice them. Neither the guest scheduler nor the hypervisor can create the 
capacity that the thread count assumed, and context switching only divides the 
available time (while adding cache/TLB churn). The guest sees this as steal 
time and can't mitigate it. Note also that vCPUs are often hyperthreads, so "N 
cores" may be closer to N/2 real ones.
   
   > The same command already runs at scale. The CI image build has used zstd 
-6 -T0 since https://github.com/apache/airflow/pull/73997 was merged on 
2026-10-01.
   
   Thank you for pointing that out! Let's compare the runtime **and failure 
rates** of the "Build CI image" job in the period 
[before](https://github.com/apache/airflow/actions/metrics/performance?dateRangeType=DATE_RANGE_TYPE_CUSTOM&sort=jobExecutions&tab=jobs&filters=%22Build+CI+image%22&range=1788220800000-1790812800000)
 and 
[after](https://github.com/apache/airflow/actions/metrics/performance?dateRangeType=DATE_RANGE_TYPE_CUSTOM&sort=jobExecutions&tab=jobs&filters=%22Build+CI+image%22&range=1790899200000-1793750400000)
 deploying that change:
   - Before: 
   <img width="924" height="366" alt="image" 
src="https://github.com/user-attachments/assets/f2f76d71-1763-4aa7-8b3b-d1c594ebe065";
 />
   
   - After:
   <img width="924" height="366" alt="image" 
src="https://github.com/user-attachments/assets/c3353264-85b3-4210-95f0-37739e2d9c3f";
 />
   
   So what we gained in shorter compression time we lost in queue time and 
multiple times over in failed CI minutes. I don't claim that this change is 
uniquely responsible for the trend, but the correlation cannot be ignored. To 
make sure we should check the failure logs for the frequency of `"The hosted 
runner lost communication with the server"` messages.
   
   > zstd needs less CPU time in total than the old deflate compression. 
   
   No argument there. But it also requires more memory. The failures I'm 
worried about aren't slow compression, but the runner agent being starved. On 
GitHub-hosted and self-hosted runners alike, a step that saturates every core 
can cause "The hosted runner lost communication with the server", which kills 
the job and its logs. This is documented as a cause of that error, and projects 
have hit it with unbounded parallel builds (e.g. pytorch/pytorch#92056 capped 
Bazel's CPU/memory for exactly this reason). `-T0` leaves nothing for the 
runner listener or dockerd, whether or not the host is oversubscribed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to