shahar1 commented on PR #39203:
URL: https://github.com/apache/beam/pull/39203#issuecomment-4862989957
**Live verification against real GCP projects (all three SDKs)**
Each SDK ran the same three scenarios on a local runner, reading
`bigquery-public-data.usa_names.usa_1910_2013` via a query, with the
pipeline/parent project set to a project I own ("project A") and the quota
project varied:
| Scenario | Expected | Python (`ReadFromBigQuery(quota_project_id=...)`,
DIRECT_READ) | Java (`--bigQueryQuotaProjectId`, DIRECT_READ) | Go
(`bigqueryio.WithQuotaProject`) |
|---|---|---|---|---|
| No quota project (control) | Read succeeds | ✅ | ✅ | ✅ |
| Quota project the caller has no `serviceusage.services.use` on | 403
`USER_PROJECT_DENIED` | ✅ | ✅ | ✅ |
| Quota project I own, with the BigQuery API **disabled** |
`SERVICE_DISABLED` naming the quota project | ✅ | ✅ | ✅ |
The third scenario is a positive attribution proof with zero cost: the API
rejected the request *as a consumer of the quota project*, demonstrating the
request was attributed to it rather than to project A.
Two findings from the live tests:
1. **Bug caught and fixed** (commit 0f949cce66e): the transform-level
`quota_project_id` in Python never reached the apitools jobs client — both
sources built the client from pipeline options only, so only the
`--quota_project_id` pipeline option worked. The live negative test initially
*succeeded* when it should have failed, exposing this.
2. **BigQuery Storage Read API ignores `x-goog-user-project`**: verified
with a gRPC interceptor that the header is sent on the wire, yet
`CreateReadSession` succeeds even when the quota project has the Storage API
disabled. Per the [Storage Read API
docs](https://cloud.google.com/bigquery/docs/reference/storage), read-session
quota/billing attach to the `parent` project of `CreateReadSession` (which Beam
already sets from the billing project). So for `DIRECT_READ`, attribution of
the *session/streams* is governed by the parent project; the quota project
applies to the jobs/tables/datasets API calls (query jobs, exports, metadata),
which is where the enforcement above was observed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
