shunping opened a new pull request, #40674:
URL: https://github.com/apache/beam/pull/40674
## Problem
The Python GCS client library (`storage.Client`) only reads the universe
domain from `client_options.universe_domain`. It ignores
`GOOGLE_CLOUD_UNIVERSE_DOMAIN` and the universe domain on the credentials.
Beam creates the client without `client_options`, so it always assumes
`googleapis.com`. With Trusted Partner Cloud (TPC) credentials, client creation
fails with:
> ValueError: The configured universe domain (googleapis.com) does not match
> the universe domain found in the credentials (...)
This breaks:
- `gcsio.create_storage_client()`, which every GCS read and write uses.
- The soft-delete check in `GoogleCloudOptions`. This fails during pipeline
validation, so the job can't even be submitted.
## Changes
- **`gcsio.create_storage_client`**: passes
`client_options={"universe_domain": ...}`, taking the value from
`GOOGLE_CLOUD_UNIVERSE_DOMAIN` or, if that's unset, the credentials. Nothing
changes for the default universe.
- **`GoogleCloudOptions._warn_if_soft_delete_policy_enabled`**:
- Builds `GcsIO` from the user's pipeline options instead of empty
defaults, so `--project`, `--impersonate_service_account` and the universe
domain are respected.
- Catches any exception and logs a warning. This check is only meant to
warn, so it should never block submission.
Part of #40569
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
