damccorm opened a new pull request, #41147: URL: https://github.com/apache/beam/pull/41147
## Summary Fixes recurring `1/0 NVIDIA T4 GPUs` quota exhaustion failures in `:sdks:python:test-suites:dataflow:py3XX:vllmTests` during `PostCommit Python` ([run 42721261611](https://github.com/apache/beam/actions/runs/42721261611/job/129515512951) and [run 42694734966](https://github.com/apache/beam/actions/runs/42694734966)). ### Root Cause - When `beam_Inference_Python_Benchmarks_Dataflow` runs GPU benchmarks in `us-central1` (requesting up to 50 `nvidia-tesla-t4` workers alongside 6 static T4 VMs in `us-central1`), all 48 `NVIDIA_T4_GPUS` in `us-central1` are consumed (`1/0 NVIDIA T4 GPUs` required/available). - Meanwhile, `apache-beam-testing` has an independent regional quota of `32` `NVIDIA_T4_GPUS` in `us-west1` (`us-west1-a`, `us-west1-b`) with `0` usage. ### Fix - Update `vllmBaseArgMap` and `tensorRTtests` in `sdks/python/test-suites/dataflow/common.gradle` to run in `us-west1` using the US multi-region `gs://temp-storage-for-end-to-end-tests/{staging-it,temp-it}` locations. ## Verification - Triggered `PostCommit Python` workflow run on branch: https://github.com/apache/beam/actions/runs/42750972886 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
