Dev-iL commented on PR #76279: URL: https://github.com/apache/airflow/pull/76279#issuecomment-6625505436
Thank you for taking my concerns seriously. Let me narrow what I'm worried about. > Each runner runs one job at a time. GitHub's docs say that [each standard GitHub-hosted runner is a new virtual machine](https://docs.github.com/en/actions/concepts/runners/github-hosted-runners#overview-of-github-hosted-runners), and that [each runner can run a single job at a time](https://docs.github.com/en/actions/get-started/understand-github-actions#runners). RunsOn's docs say that RunsOn starts [a fresh, ephemeral EC2 runner for every job](https://runs-on.com/docs/). In the PROD build job, no other step runs at the same time as the export step. Also, with -T0, zstd tries to detect the number of physical CPU cores and uses that many threads. So zstd should not start more threads than the machine has cores. "One VM per job" isn't "one set of physical cores per job". vCPUs are time-sliced onto host hardware shared with other tenants, and `-T0` sizes its worker pool from the topology the guest is *told* about, not from what the host can actually deliver at that moment. If the host is oversubscribed, the guest sees N runnable cores and spawns N workers, but the hypervisor can only time-slice them. Neither the guest scheduler nor the hypervisor can create the capacity that the thread count assumed, and context switching only divides the available time (while adding cache/TLB churn). The guest sees this as steal time and can't mitigate it. Note also that vCPUs are often hyperthreads, so "N cores" may be closer to N/2 real ones. > The same command already runs at scale. The CI image build has used zstd -6 -T0 since https://github.com/apache/airflow/pull/73997 was merged on 2026-10-01. Thank you for pointing that out! Let's compare the runtime **and failure rates** of the "Build CI image" job in the period [before](https://github.com/apache/airflow/actions/metrics/performance?dateRangeType=DATE_RANGE_TYPE_CUSTOM&sort=jobExecutions&tab=jobs&filters=%22Build+CI+image%22&range=1788220800000-1790812800000) and [after](https://github.com/apache/airflow/actions/metrics/performance?dateRangeType=DATE_RANGE_TYPE_CUSTOM&sort=jobExecutions&tab=jobs&filters=%22Build+CI+image%22&range=1790899200000-1793750400000) deploying that change: - Before: <img width="924" height="366" alt="image" src="https://github.com/user-attachments/assets/f2f76d71-1763-4aa7-8b3b-d1c594ebe065" /> - After: <img width="924" height="366" alt="image" src="https://github.com/user-attachments/assets/c3353264-85b3-4210-95f0-37739e2d9c3f" /> So what we gained in shorter compression time we lost in queue time and multiple times over in failed CI minutes. I don't claim that this change is uniquely responsible for the trend, but the correlation cannot be ignored. To make sure we should check the failure logs for the frequency of `"The hosted runner lost communication with the server"` messages. > zstd needs less CPU time in total than the old deflate compression. No argument there. But it also requires more memory. The failures I'm worried about aren't slow compression, but the runner agent being starved. On GitHub-hosted and self-hosted runners alike, a step that saturates every core can cause "The hosted runner lost communication with the server", which kills the job and its logs. This is documented as a cause of that error, and projects have hit it with unbounded parallel builds (e.g. pytorch/pytorch#92056 capped Bazel's CPU/memory for exactly this reason). `-T0` leaves nothing for the runner listener or dockerd, whether or not the host is oversubscribed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
