shunping opened a new issue, #40712: URL: https://github.com/apache/beam/issues/40712
### What needs to happen? Umbrella issue for moving the Java SDK's GCS access (`GcsUtil`, used by `GcsFileSystem` and anything reading or writing `gs://` paths) off the gcsio / `google-api-services-storage` stack (GcsUtil V1) and onto the Cloud Storage Java client library `google-cloud-storage` (GcsUtil V2). ### Background `GcsUtil` is a facade over two implementations: - **`GcsUtilV1`**: the existing gcsio-based implementation. - **`GcsUtilV2`**: the new implementation built on `com.google.cloud.storage.Storage`. V2 was introduced behind the `use_gcsutil_v2` experiment. It then gained feature and performance parity with V1, with parity integration tests (`GcsUtilIT`) running both implementations side by side. ### Rollout plan 1. Implement V2 behind an opt-in experiment (`use_gcsutil_v2`). **Done.** 2. Reach feature and performance parity, with test coverage. **Done.** 3. Make V2 the default. Keep `disable_gcsutil_v2` as a fallback; `use_grpc_for_gcs` also stays on V1. **In progress.** 4. Let V2 run as the default for at least one release and fix whatever turns up. 5. Remove V1 and the gcsio dependency, and retire the `use_gcsutil_v2` / `disable_gcsutil_v2` experiments. ### Experiments | Experiment | Effect (once V2 is the default) | |---|---| | *(none)* | GcsUtil V2 | | `disable_gcsutil_v2` | Fall back to GcsUtil V1 | | `use_grpc_for_gcs` | GcsUtil V1 over gRPC (gRPC is not supported by V2 yet) | | `use_gcsutil_v2` | Ignored; kept so existing pipelines don't break | ### Completed work - [x] GCS client library migration in Java SDK, part 1 (#36876) - [x] GCS client library migration in Java SDK, part 2 (#37502) - [x] GCS client library migration in Java SDK, part 2b (#37592) - [x] Remove test-only functions in GcsUtil and move related unit tests to V1 (#37597) - [x] GCS client library migration in Java SDK, part 3 (#37900) - [x] Extend GCS performance metrics and unify the GCS metric namespace (#40142) - [x] Fill gaps between GcsUtilV2 and GcsUtilV1 (#40244) - [x] Route the remaining GCS calls through GcsUtilV2 when enabled (#40273) - [x] Re-arrange GcsUtil unit tests and integration tests (#40279) - [x] Add V2 unit tests and V1/V2 parity ITs for GcsUtil (#40319) - [x] Parallelize per-file rewrites in GcsUtilV2 copy and move (#40376) - [x] Close GcsUtilV2's small-file performance gaps with GcsUtilV1 (#40677) ### Remaining work - [ ] Use GcsUtil V2 by default, with `disable_gcsutil_v2` as the fallback - [ ] Decide whether V2 should support gRPC, or whether `use_grpc_for_gcs` goes away along with V1 - [ ] Let V2 run as the default for at least one release - [ ] Remove `GcsUtilV1` and the gcsio dependency - [ ] Retire the `use_gcsutil_v2` and `disable_gcsutil_v2` experiments ### Issue Priority Priority: 2 (default / most normal work should be filed as P2) ### Issue Components - [ ] Component: Python SDK - [x] Component: Java SDK - [ ] Component: Go SDK - [ ] Component: Typescript SDK - [ ] Component: IO connector - [ ] Component: Beam YAML - [ ] Component: Beam examples - [ ] Component: Beam playground - [ ] Component: Beam katas - [ ] Component: Website - [ ] Component: Infrastructure - [ ] Component: Spark Runner - [ ] Component: Flink Runner - [ ] Component: Prism Runner - [ ] Component: Twister2 Runner - [ ] Component: Hazelcast Jet Runner - [ ] Component: Google Cloud Dataflow Runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
