shunping opened a new pull request, #40244:
URL: https://github.com/apache/beam/pull/40244
**This is the final stage of the GCS client library migration.**
`GcsUtilV2` is the `java-storage` backed replacement for the `gcsio` based
`GcsUtilV1`, selected with the `use_gcsutil_v2` experiment. There were some
behavior gaps between the two. This PR closes those gaps, and fixes a bug that
made the experiment a no-op on Dataflow.
Parity gaps closed:
| Gap | Before | Now |
| :--- | :--- | :--- |
| Client configuration | Application default credentials; `--gcsEndpoint`
ignored | Pipeline credentials are passed through (`null` → `NoCredentials`),
`--gcsEndpoint` applied as the client host |
| Metrics | `open()` / `create()` returned raw channels, so every GCS metric
was dropped | Channels wrapped in `Counting{Seekable,Writable}ByteChannel`,
plus a `ServiceCallMetric` for `GcsGet` / `GcsInsert` |
| HTTP transport counters | Not reported | Request count, request shape and
status classes, gated on `--gcsPerformanceMetrics` |
| Facade routing | Only the `GcsPath`-typed methods reached V2, so match,
list, copy, rename, delete and bucket creation stayed on V1 | The
`String`-typed and legacy-model methods have V2 branches too, converting `Blob`
/ `BucketInfo` back to the JSON API model so callers need no change |
| Upload chunk size | java-storage default, so the same workload cost a
different number of requests | Mirrors gcsio's `AsyncWriteChannelOptions`
default |
| Leaked channel | `create()` opened a V2 channel, discarded it, then opened
a second on V1 | Returns the V2 channel |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
