[
https://issues.apache.org/jira/browse/BEAM-8889?focusedWorklogId=375143&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-375143
]
ASF GitHub Bot logged work on BEAM-8889:
----------------------------------------
Author: ASF GitHub Bot
Created on: 21/Jan/20 18:35
Start Date: 21/Jan/20 18:35
Worklog Time Spent: 10m
Work Description: veblush commented on pull request #10617: [BEAM-8889]
adding gRPC connectivity to Beam/GCS connector
URL: https://github.com/apache/beam/pull/10617#discussion_r369172468
##########
File path:
sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java
##########
@@ -427,22 +440,14 @@ public WritableByteChannel create(GcsPath path, String
type) throws IOException
*/
public WritableByteChannel create(GcsPath path, String type, Integer
uploadBufferSizeBytes)
throws IOException {
- GoogleCloudStorageWriteChannel channel =
- new GoogleCloudStorageWriteChannel(
- executorService,
- storageClient,
- new ClientRequestHelper<>(),
- path.getBucket(),
- path.getObject(),
- type,
- /* kmsKeyName= */ null,
- AsyncWriteChannelOptions.newBuilder().build(),
- new ObjectWriteConditions(),
- Collections.emptyMap());
+ WritableByteChannel channel = getCloudStorage().create(new
StorageResourceId(path.getBucket()));
if (uploadBufferSizeBytes != null) {
- channel.setUploadBufferSize(uploadBufferSizeBytes);
+ if (channel instanceof GoogleCloudStorageWriteChannel) {
Review comment:
I'm a bit hesitant to have this but i'm okay if we have to. @chamikaramj Do
you think this is still necessary? If this is required, we might consider to
have new API in `getCloudStorage().create` to accept `uploadBufferSizeBytes` to
avoid code for each type of channels.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 375143)
Remaining Estimate: 167h 10m (was: 167h 20m)
Time Spent: 50m (was: 40m)
> Make GcsUtil use GoogleCloudStorage
> -----------------------------------
>
> Key: BEAM-8889
> URL: https://issues.apache.org/jira/browse/BEAM-8889
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
> Affects Versions: 2.16.0
> Reporter: Esun Kim
> Assignee: VASU NORI
> Priority: Major
> Labels: gcs
> Original Estimate: 168h
> Time Spent: 50m
> Remaining Estimate: 167h 10m
>
> [GcsUtil|https://github.com/apache/beam/blob/master/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java]
> is a primary class to access Google Cloud Storage on Apache Beam. Current
> implementation directly creates GoogleCloudStorageReadChannel and
> GoogleCloudStorageWriteChannel by itself to read and write GCS data rather
> than using
> [GoogleCloudStorage|https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorage.java]
> which is an abstract class providing basic IO capability which eventually
> creates channel objects. This request is about updating GcsUtil to use
> GoogleCloudStorage to create read and write channel, which is expected
> flexible because it can easily pick up the new change; e.g. new channel
> implementation using new protocol without code change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
