[jira] [Work logged] (BEAM-6155) Migrate the Go SDK to the modern GCS library
[ https://issues.apache.org/jira/browse/BEAM-6155?focusedWorklogId=181333&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-181333 ] ASF GitHub Bot logged work on BEAM-6155: Author: ASF GitHub Bot Created on: 05/Jan/19 01:45 Start Date: 05/Jan/19 01:45 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #7292: [BEAM-6155] Plumb the contexts though the gcsx library. URL: https://github.com/apache/beam/pull/7292 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 181333) Time Spent: 1h 10m (was: 1h) > Migrate the Go SDK to the modern GCS library > > > Key: BEAM-6155 > URL: https://issues.apache.org/jira/browse/BEAM-6155 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Andrew Brampton >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > The gcsx package is using the google.golang.org/api/storage/v1 GCS library. > That library has been deprecated for ~6 months, and the recommendation is to > use the newer > [cloud.google.com/go/storage|https://godoc.org/cloud.google.com/go/storage] > package. That package supports newer features, and has built in connection > pooling, timeout support, retry with exponential backoff, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6155) Migrate the Go SDK to the modern GCS library
[ https://issues.apache.org/jira/browse/BEAM-6155?focusedWorklogId=179521&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-179521 ] ASF GitHub Bot logged work on BEAM-6155: Author: ASF GitHub Bot Created on: 28/Dec/18 16:46 Start Date: 28/Dec/18 16:46 Worklog Time Spent: 10m Work Description: lostluck commented on issue #7292: [BEAM-6155] Plumb the contexts though the gcsx library. URL: https://github.com/apache/beam/pull/7292#issuecomment-450389637 LGTM @aaltay Please merge at your convenience. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 179521) Time Spent: 1h (was: 50m) > Migrate the Go SDK to the modern GCS library > > > Key: BEAM-6155 > URL: https://issues.apache.org/jira/browse/BEAM-6155 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Andrew Brampton >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > The gcsx package is using the google.golang.org/api/storage/v1 GCS library. > That library has been deprecated for ~6 months, and the recommendation is to > use the newer > [cloud.google.com/go/storage|https://godoc.org/cloud.google.com/go/storage] > package. That package supports newer features, and has built in connection > pooling, timeout support, retry with exponential backoff, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6155) Migrate the Go SDK to the modern GCS library
[ https://issues.apache.org/jira/browse/BEAM-6155?focusedWorklogId=179219&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-179219 ] ASF GitHub Bot logged work on BEAM-6155: Author: ASF GitHub Bot Created on: 27/Dec/18 20:31 Start Date: 27/Dec/18 20:31 Worklog Time Spent: 10m Work Description: apilloud commented on issue #7292: [BEAM-6155] Plumb the contexts though the gcsx library. URL: https://github.com/apache/beam/pull/7292#issuecomment-450225949 R: @lostluck This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 179219) Time Spent: 50m (was: 40m) > Migrate the Go SDK to the modern GCS library > > > Key: BEAM-6155 > URL: https://issues.apache.org/jira/browse/BEAM-6155 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Andrew Brampton >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > The gcsx package is using the google.golang.org/api/storage/v1 GCS library. > That library has been deprecated for ~6 months, and the recommendation is to > use the newer > [cloud.google.com/go/storage|https://godoc.org/cloud.google.com/go/storage] > package. That package supports newer features, and has built in connection > pooling, timeout support, retry with exponential backoff, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6155) Migrate the Go SDK to the modern GCS library
[ https://issues.apache.org/jira/browse/BEAM-6155?focusedWorklogId=175834&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-175834 ] ASF GitHub Bot logged work on BEAM-6155: Author: ASF GitHub Bot Created on: 16/Dec/18 16:54 Start Date: 16/Dec/18 16:54 Worklog Time Spent: 10m Work Description: bramp opened a new pull request #7292: [BEAM-6155] Plumb the contexts though the gcsx library. URL: https://github.com/apache/beam/pull/7292 Plumb the context through the gcsx library. This is a continuation from https://github.com/apache/beam/pull/7182 Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 175834) Time Spent: 40m (was: 0.5h) > Migrate the Go SDK to the modern GCS library > > > Key: BEAM-6155 > URL: https://issues.apache.org/jira/browse/BEAM-6155 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Andrew Brampton >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > The gcsx package is using the google.golang.org/api/storage/v1 GCS library. > That library has been deprecated for ~6 months, and the recommendation is to > use the newer > [cloud.google.com/go/storage|https://godoc.org/cloud.google.com/go/storage] > package. That package supports newer features, and has built in connection > pooling, timeout support, retry with exponential backoff, etc. -- This message was sent by Atlassian JIRA (
[jira] [Work logged] (BEAM-6155) Migrate the Go SDK to the modern GCS library
[
https://issues.apache.org/jira/browse/BEAM-6155?focusedWorklogId=172911&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172911
]
ASF GitHub Bot logged work on BEAM-6155:
Author: ASF GitHub Bot
Created on: 07/Dec/18 02:03
Start Date: 07/Dec/18 02:03
Worklog Time Spent: 10m
Work Description: aaltay closed pull request #7182: [BEAM-6155] Updates
the GCS library the Go SDK uses.
URL: https://github.com/apache/beam/pull/7182
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go
b/sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go
index dede7a51c1af..a59b81d93ffc 100644
--- a/sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go
+++ b/sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go
@@ -19,11 +19,11 @@ import (
"fmt"
"io"
+ "cloud.google.com/go/storage"
pb "github.com/apache/beam/sdks/go/pkg/beam/model/jobmanagement_v1"
"github.com/apache/beam/sdks/go/pkg/beam/util/gcsx"
"github.com/golang/protobuf/proto"
"golang.org/x/net/context"
- "google.golang.org/api/storage/v1"
)
// RetrievalServer is a artifact retrieval server backed by Google
@@ -41,7 +41,7 @@ func ReadProxyManifest(ctx context.Context, object string)
(*pb.ProxyManifest, e
return nil, fmt.Errorf("invalid manifest object %v: %v",
object, err)
}
- cl, err := gcsx.NewClient(ctx, storage.DevstorageReadOnlyScope)
+ cl, err := gcsx.NewClient(ctx, storage.ScopeReadOnly)
if err != nil {
return nil, fmt.Errorf("failed to create GCS client: %v", err)
}
@@ -88,22 +88,22 @@ func (s *RetrievalServer) GetArtifact(req
*pb.GetArtifactRequest, stream pb.Arti
bucket, object := parseObject(blob)
- client, err := gcsx.NewClient(stream.Context(),
storage.DevstorageReadOnlyScope)
+ client, err := gcsx.NewClient(stream.Context(), storage.ScopeReadOnly)
if err != nil {
return fmt.Errorf("Failed to create client for %v: %v", key,
err)
}
// Stream artifact in up to 1MB chunks.
-
- resp, err := client.Objects.Get(bucket, object).Download()
+ ctx := context.TODO()
+ r, err := client.Bucket(bucket).Object(object).NewReader(ctx)
if err != nil {
return fmt.Errorf("Failed to read object for %v: %v", key, err)
}
- defer resp.Body.Close()
+ defer r.Close()
data := make([]byte, 1<<20)
for {
- n, err := resp.Body.Read(data)
+ n, err := r.Read(data)
if n > 0 {
if err := stream.Send(&pb.ArtifactChunk{Data:
data[:n]}); err != nil {
return fmt.Errorf("chunk send failed: %v", err)
diff --git a/sdks/go/pkg/beam/artifact/gcsproxy/staging.go
b/sdks/go/pkg/beam/artifact/gcsproxy/staging.go
index 51ffec6fe129..109eb29cdb43 100644
--- a/sdks/go/pkg/beam/artifact/gcsproxy/staging.go
+++ b/sdks/go/pkg/beam/artifact/gcsproxy/staging.go
@@ -26,11 +26,11 @@ import (
"path"
"sync"
+ "cloud.google.com/go/storage"
pb "github.com/apache/beam/sdks/go/pkg/beam/model/jobmanagement_v1"
"github.com/apache/beam/sdks/go/pkg/beam/util/gcsx"
"github.com/golang/protobuf/proto"
"golang.org/x/net/context"
- "google.golang.org/api/storage/v1"
)
// StagingServer is a artifact staging server backed by Google Cloud Storage
@@ -81,7 +81,7 @@ func (s *StagingServer) CommitManifest(ctx context.Context,
req *pb.CommitManife
return nil, fmt.Errorf("failed to marshal proxy manifest: %v",
err)
}
- cl, err := gcsx.NewClient(ctx, storage.DevstorageReadWriteScope)
+ cl, err := gcsx.NewClient(ctx, storage.ScopeReadWrite)
if err != nil {
return nil, fmt.Errorf("failed to create GCS client: %v", err)
}
@@ -135,7 +135,7 @@ func (s *StagingServer) PutArtifact(ps
pb.ArtifactStagingService_PutArtifactServ
// Stream content to GCS. We don't have to worry about partial
// or abandoned writes, because object writes are atomic.
- cl, err := gcsx.NewClient(ps.Context(),
storage.DevstorageReadWriteScope)
+ cl, err := gcsx.NewClient(ps.Context(), storage.ScopeReadWrite)
if err != nil {
return fmt.Errorf("failed to create GCS client: %v", err)
}
diff --git a/sdks/go/pkg/beam/io/filesystem/gcs/gcs.go
b/sdks/go/pkg/beam/io/filesystem/gcs/gcs.go
index 9a7b265796d7..356c718a024d 100644
--- a/sdks/go/pkg/beam/io/filesystem/gcs/gcs.go
+++ b/sdks/go/pkg/beam/io/filesyste
[jira] [Work logged] (BEAM-6155) Migrate the Go SDK to the modern GCS library
[ https://issues.apache.org/jira/browse/BEAM-6155?focusedWorklogId=172905&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172905 ] ASF GitHub Bot logged work on BEAM-6155: Author: ASF GitHub Bot Created on: 07/Dec/18 00:59 Start Date: 07/Dec/18 00:59 Worklog Time Spent: 10m Work Description: aaltay commented on issue #7182: [BEAM-6155] Updates the GCS library the Go SDK uses. URL: https://github.com/apache/beam/pull/7182#issuecomment-445085546 Run Go PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 172905) Time Spent: 20m (was: 10m) > Migrate the Go SDK to the modern GCS library > > > Key: BEAM-6155 > URL: https://issues.apache.org/jira/browse/BEAM-6155 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Andrew Brampton >Assignee: Robert Burke >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > The gcsx package is using the google.golang.org/api/storage/v1 GCS library. > That library has been deprecated for ~6 months, and the recommendation is to > use the newer > [cloud.google.com/go/storage|https://godoc.org/cloud.google.com/go/storage] > package. That package supports newer features, and has built in connection > pooling, timeout support, retry with exponential backoff, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6155) Migrate the Go SDK to the modern GCS library
[ https://issues.apache.org/jira/browse/BEAM-6155?focusedWorklogId=171411&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171411 ] ASF GitHub Bot logged work on BEAM-6155: Author: ASF GitHub Bot Created on: 02/Dec/18 21:56 Start Date: 02/Dec/18 21:56 Worklog Time Spent: 10m Work Description: bramp opened a new pull request #7182: [BEAM-6155] Updates the GCS library the Go SDK uses. URL: https://github.com/apache/beam/pull/7182 Updates the GCS library the Go SDK uses. This keeps the API the same, but uses the modern GCS library which correctly supports contexts, exponential backoff, etc Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 171411) Time Spent: 10m Remaining Estimate: 0h > Migrate the Go SDK to the modern GCS library > > > Key: BEAM-6155 > URL: https://issues.apache.org/jira/browse/BEAM-6155 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Andrew Brampton >Assignee: Robert Burke >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > The gcsx package is using the google.golang.org/api/storage/v1 GCS library. > That library has been deprecated for ~6 months, and the recommendation is to > use the newer > [cloud.google.com/go/s
