shunping opened a new pull request, #40686: URL: https://github.com/apache/beam/pull/40686
## Summary The Go SDK's GCS and Dataflow clients always talk to the default `*.googleapis.com` endpoints. In a non-default universe such as Trusted Partner Cloud (TPC), they fail: - **GCS:** `storage.NewClient` reads the universe domain only from `option.WithUniverseDomain` or `GOOGLE_CLOUD_UNIVERSE_DOMAIN`. It ignores the credentials' universe domain. The auth library checks on every request that the client's universe matches the credentials' universe, so TPC credentials fail with a universe-mismatch error. - **Dataflow:** `dataflowlib.NewClient` used the deprecated `df.New(httpClient)`. With no `--dataflow_endpoint` and no env var, it always resolved to `https://dataflow.googleapis.com/`. This PR follows the Python SDK fix for GCS (#40674). It also adds a `--gcs_endpoint` override, in line with Java's `--gcsEndpoint` and the existing `--dataflow_endpoint`. ## Changes **`gcpopts`** - New `--gcs_endpoint` flag and `GetGcsEndpoint()`. Workers don't parse flags, so the getter falls back to the `gcs_endpoint` pipeline option. That is how the value reaches workers; the flag is intentionally not in the Dataflow `flagFilter`. - New `GetUniverseDomain(ctx)`, shared by both clients. It reads `GOOGLE_CLOUD_UNIVERSE_DOMAIN` first, then the default credentials' universe domain. - The result is cached, because GCS clients are created often and the credentials lookup may call the GCE metadata server. - Failed lookups are not cached. **`gcsx`** - `NewClient` passes `option.WithEndpoint` when `--gcs_endpoint` is set. - It passes `option.WithUniverseDomain` when the universe domain is not `googleapis.com`. - `NewUnauthenticatedClient` also uses `--gcs_endpoint`. **`dataflowlib`** - `NewClient` now uses `df.NewService(ctx, ...)` instead of the deprecated `df.New(google.DefaultClient(...))` plus a `BasePath` overwrite. - `--dataflow_endpoint` is passed via `option.WithEndpoint`. The universe domain is passed via `option.WithUniverseDomain`. **`dataflow` runner** - The soft-delete bucket check now uses `gcsx.NewClient` instead of calling `storage.NewClient(ctx)` directly. It gets the same endpoint and universe handling. ## Behavior changes - Dataflow client credentials now come from the newer Google auth library (through `df.NewService`) instead of `oauth2/google`. A mismatch between the client's and the credentials' universe now fails with a clear error instead of a confusing server-side failure. - A `--dataflow_endpoint` value containing `://` is used as-is, as before. A bare `host[:port]` is now combined with the default endpoint. Previously it was set as `BasePath` directly and produced an invalid URL. - Nothing changes in the default universe (`googleapis.com`) when no flags or env vars are set. ## Usage in TPC Either rely on the universe domain from the credentials or the env var: ``` export GOOGLE_CLOUD_UNIVERSE_DOMAIN=<universe-domain> ``` or set the endpoints explicitly: ``` --dataflow_endpoint=https://dataflow.<universe-domain>/ --gcs_endpoint=https://storage.<universe-domain>/storage/v1/ ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
