[jira] [Work logged] (BEAM-3991) Update dependency 'google-api-services-storage' to latest version
[
https://issues.apache.org/jira/browse/BEAM-3991?focusedWorklogId=93336=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-93336
]
ASF GitHub Bot logged work on BEAM-3991:
Author: ASF GitHub Bot
Created on: 20/Apr/18 18:15
Start Date: 20/Apr/18 18:15
Worklog Time Spent: 10m
Work Description: kennknowles closed pull request #5046: [BEAM-3991]
Updating Google API client dependencies to 1.23 versions
URL: https://github.com/apache/beam/pull/5046
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/build_rules.gradle b/build_rules.gradle
index ada4c8b5976..a00643df153 100644
--- a/build_rules.gradle
+++ b/build_rules.gradle
@@ -162,7 +162,7 @@ offlineDependencies {
def google_cloud_bigdataoss_version = "1.4.5"
def bigtable_version = "1.0.0"
def bigtable_proto_version = "1.0.0-pre3"
-def google_clients_version = "1.22.0"
+def google_clients_version = "1.23.0"
def google_auth_version = "0.7.1"
def grpc_version = "1.2.0"
def protobuf_version = "3.2.0"
@@ -218,11 +218,11 @@ ext.library = [
google_api_client_java6:
"com.google.api-client:google-api-client-java6:$google_clients_version",
google_api_common: "com.google.api:api-common:1.0.0-rc2",
google_api_services_bigquery:
"com.google.apis:google-api-services-bigquery:v2-rev374-$google_clients_version",
-google_api_services_clouddebugger:
"com.google.apis:google-api-services-clouddebugger:v2-rev8-$google_clients_version",
-google_api_services_cloudresourcemanager:
"com.google.apis:google-api-services-cloudresourcemanager:v1-rev6-$google_clients_version",
+google_api_services_clouddebugger:
"com.google.apis:google-api-services-clouddebugger:v2-rev233-$google_clients_version",
+google_api_services_cloudresourcemanager:
"com.google.apis:google-api-services-cloudresourcemanager:v1-rev477-$google_clients_version",
google_api_services_dataflow:
"com.google.apis:google-api-services-dataflow:v1b3-rev221-$google_clients_version",
-google_api_services_pubsub:
"com.google.apis:google-api-services-pubsub:v1-rev10-$google_clients_version",
-google_api_services_storage:
"com.google.apis:google-api-services-storage:v1-rev71-$google_clients_version",
+google_api_services_pubsub:
"com.google.apis:google-api-services-pubsub:v1-rev382-$google_clients_version",
+google_api_services_storage:
"com.google.apis:google-api-services-storage:v1-rev124-$google_clients_version",
google_auth_library_credentials:
"com.google.auth:google-auth-library-credentials:$google_auth_version",
google_auth_library_oauth2_http:
"com.google.auth:google-auth-library-oauth2-http:$google_auth_version",
google_cloud_core: "com.google.cloud:google-cloud-core:1.0.2",
diff --git a/pom.xml b/pom.xml
index e14a190f2e7..3e42a31f53f 100644
--- a/pom.xml
+++ b/pom.xml
@@ -108,13 +108,13 @@
1.0.0-rc2
2.33
1.8.2
-v2-rev374-1.22.0
+v2-rev374-1.23.0
1.0.0
1.0.0-pre3
-v1-rev6-1.22.0
+
v1-rev477-1.23.0
0.1.18
-v2-rev8-1.22.0
-v1b3-rev221-1.22.0
+v2-rev233-1.23.0
+v1b3-rev221-1.23.0
0.5.160222
1.4.0
1.3.0
@@ -122,7 +122,7 @@
1.0-rc2
1.5.3
0.7.1
-1.22.0
+1.23.0
1.4.5
1.0.2
0.5.160304
@@ -145,13 +145,13 @@
4.1.8.Final
1.1.33.Fork26
3.2.0
-v1-rev10-1.22.0
+v1-rev382-1.23.0
1.7.25
0.20.0b-beta
2.3.0
4.3.5.RELEASE
3.1.4
-v1-rev71-1.22.0
+v1-rev124-1.23.0
4.4.1
4.3.5.RELEASE
1.1.4
diff --git
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index f7eca249952..dc8ddb77163 100644
---
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -29,7 +29,7 @@
import com.fasterxml.jackson.databind.Module;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.api.client.googleapis.json.GoogleJsonResponseException;
-import com.google.api.services.clouddebugger.v2.Clouddebugger;
+import com.google.api.services.clouddebugger.v2.CloudDebugger;
import com.google.api.services.clouddebugger.v2.model.Debuggee;
import com.google.api.services.clouddebugger.v2.model.RegisterDebuggeeRequest;
import com.google.api.services.clouddebugger.v2.model.RegisterDebuggeeResponse;
@@ -617,14 +617,14 @@ private void
maybeRegisterDebuggee(DataflowPipelineOptions options, String uniqu
throw new
[jira] [Work logged] (BEAM-3991) Update dependency 'google-api-services-storage' to latest version
[ https://issues.apache.org/jira/browse/BEAM-3991?focusedWorklogId=93335=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-93335 ] ASF GitHub Bot logged work on BEAM-3991: Author: ASF GitHub Bot Created on: 20/Apr/18 18:15 Start Date: 20/Apr/18 18:15 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5046: [BEAM-3991] Updating Google API client dependencies to 1.23 versions URL: https://github.com/apache/beam/pull/5046#issuecomment-383179651 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 93335) Time Spent: 1h 10m (was: 1h) > Update dependency 'google-api-services-storage' to latest version > - > > Key: BEAM-3991 > URL: https://issues.apache.org/jira/browse/BEAM-3991 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp, sdk-py-core >Reporter: Chamikara Jayalath >Assignee: Chamikara Jayalath >Priority: Blocker > Fix For: 2.5.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently we use version 'v1-rev71-1.22.0' which is deprecated and about two > years old. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3991) Update dependency 'google-api-services-storage' to latest version
[ https://issues.apache.org/jira/browse/BEAM-3991?focusedWorklogId=92939=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92939 ] ASF GitHub Bot logged work on BEAM-3991: Author: ASF GitHub Bot Created on: 19/Apr/18 23:47 Start Date: 19/Apr/18 23:47 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #5046: [BEAM-3991] Updating Google API client dependencies to 1.23 versions URL: https://github.com/apache/beam/pull/5046#issuecomment-382915557 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 92939) Time Spent: 1h (was: 50m) > Update dependency 'google-api-services-storage' to latest version > - > > Key: BEAM-3991 > URL: https://issues.apache.org/jira/browse/BEAM-3991 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp, sdk-py-core >Reporter: Chamikara Jayalath >Assignee: Chamikara Jayalath >Priority: Blocker > Fix For: 2.5.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Currently we use version 'v1-rev71-1.22.0' which is deprecated and about two > years old. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3991) Update dependency 'google-api-services-storage' to latest version
[ https://issues.apache.org/jira/browse/BEAM-3991?focusedWorklogId=90234=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90234 ] ASF GitHub Bot logged work on BEAM-3991: Author: ASF GitHub Bot Created on: 12/Apr/18 00:00 Start Date: 12/Apr/18 00:00 Worklog Time Spent: 10m Work Description: chamikaramj closed pull request #5105: [BEAM-3991] Updates gcsio to use a API specific batch endpoint. URL: https://github.com/apache/beam/pull/5105 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/sdks/python/apache_beam/io/gcp/gcsio.py b/sdks/python/apache_beam/io/gcp/gcsio.py index f687686fd64..e428e3b1401 100644 --- a/sdks/python/apache_beam/io/gcp/gcsio.py +++ b/sdks/python/apache_beam/io/gcp/gcsio.py @@ -91,6 +91,17 @@ # GcsIO.delete_batch(). MAX_BATCH_OPERATION_SIZE = 100 +# Batch endpoint URL for GCS. +# We have to specify an API specific endpoint here since Google APIs global +# batch endpoints will be deprecated on 03/25/2019. +# See https://developers.googleblog.com/2018/03/discontinuing-support-for-json-rpc-and.html. # pylint: disable=line-too-long +# Currently apitools library uses a global batch endpoint by default: +# https://github.com/google/apitools/blob/master/apitools/base/py/batch.py#L152 +# TODO: remove this constant and it's usage after apitools move to using an API +# specific batch endpoint or after Beam gcsio module start using a GCS client +# library that does not use global batch endpoints. +GCS_BATCH_ENDPOINT = 'https://www.googleapis.com/batch/storage/v1' + def proxy_info_from_environment_var(proxy_env_var): """Reads proxy info from the environment and converts to httplib2.ProxyInfo. @@ -274,6 +285,7 @@ def delete_batch(self, paths): if not paths: return [] batch_request = BatchApiRequest( +batch_url=GCS_BATCH_ENDPOINT, retryable_codes=retry.SERVER_ERROR_OR_TIMEOUT_CODES) for path in paths: bucket, object_path = parse_gcs_path(path) @@ -328,6 +340,7 @@ def copy_batch(self, src_dest_pairs): if not src_dest_pairs: return [] batch_request = BatchApiRequest( +batch_url=GCS_BATCH_ENDPOINT, retryable_codes=retry.SERVER_ERROR_OR_TIMEOUT_CODES) for src, dest in src_dest_pairs: src_bucket, src_path = parse_gcs_path(src) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90234) Time Spent: 50m (was: 40m) > Update dependency 'google-api-services-storage' to latest version > - > > Key: BEAM-3991 > URL: https://issues.apache.org/jira/browse/BEAM-3991 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp, sdk-py-core >Reporter: Chamikara Jayalath >Assignee: Chamikara Jayalath >Priority: Blocker > Fix For: 2.5.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Currently we use version 'v1-rev71-1.22.0' which is deprecated and about two > years old. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3991) Update dependency 'google-api-services-storage' to latest version
[ https://issues.apache.org/jira/browse/BEAM-3991?focusedWorklogId=90233=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90233 ] ASF GitHub Bot logged work on BEAM-3991: Author: ASF GitHub Bot Created on: 12/Apr/18 00:00 Start Date: 12/Apr/18 00:00 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #5105: [BEAM-3991] Updates gcsio to use a API specific batch endpoint. URL: https://github.com/apache/beam/pull/5105#issuecomment-380632160 Thanks. Merging. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90233) Time Spent: 40m (was: 0.5h) > Update dependency 'google-api-services-storage' to latest version > - > > Key: BEAM-3991 > URL: https://issues.apache.org/jira/browse/BEAM-3991 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp, sdk-py-core >Reporter: Chamikara Jayalath >Assignee: Chamikara Jayalath >Priority: Blocker > Fix For: 2.5.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Currently we use version 'v1-rev71-1.22.0' which is deprecated and about two > years old. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3991) Update dependency 'google-api-services-storage' to latest version
[ https://issues.apache.org/jira/browse/BEAM-3991?focusedWorklogId=90205=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90205 ] ASF GitHub Bot logged work on BEAM-3991: Author: ASF GitHub Bot Created on: 11/Apr/18 22:39 Start Date: 11/Apr/18 22:39 Worklog Time Spent: 10m Work Description: chamikaramj commented on a change in pull request #5105: [BEAM-3991] Updates gcsio to use a API specific batch endpoint. URL: https://github.com/apache/beam/pull/5105#discussion_r180918984 ## File path: sdks/python/apache_beam/io/gcp/gcsio.py ## @@ -91,6 +91,16 @@ # GcsIO.delete_batch(). MAX_BATCH_OPERATION_SIZE = 100 +# Batch endpoint URL for GCS. +# We have to specify an API specific endpoint here since Google APIs global +# batch endpoints will be deprecated on 03/25/2019. +# See https://developers.googleblog.com/2018/03/discontinuing-support-for-json-rpc-and.html. +# Currently apitools library uses a global batch endpoint by default: https://github.com/google/apitools/blob/master/apitools/base/py/batch.py#L152 +# TODO: remove this constant and it's usage after apitools move to using an API Review comment: Created https://issues.apache.org/jira/browse/BEAM-4055 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90205) Time Spent: 0.5h (was: 20m) > Update dependency 'google-api-services-storage' to latest version > - > > Key: BEAM-3991 > URL: https://issues.apache.org/jira/browse/BEAM-3991 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp, sdk-py-core >Reporter: Chamikara Jayalath >Assignee: Chamikara Jayalath >Priority: Blocker > Fix For: 2.5.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently we use version 'v1-rev71-1.22.0' which is deprecated and about two > years old. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3991) Update dependency 'google-api-services-storage' to latest version
[ https://issues.apache.org/jira/browse/BEAM-3991?focusedWorklogId=90151=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90151 ] ASF GitHub Bot logged work on BEAM-3991: Author: ASF GitHub Bot Created on: 11/Apr/18 19:49 Start Date: 11/Apr/18 19:49 Worklog Time Spent: 10m Work Description: aaltay commented on a change in pull request #5105: [BEAM-3991] Updates gcsio to use a API specific batch endpoint. URL: https://github.com/apache/beam/pull/5105#discussion_r180876774 ## File path: sdks/python/apache_beam/io/gcp/gcsio.py ## @@ -91,6 +91,16 @@ # GcsIO.delete_batch(). MAX_BATCH_OPERATION_SIZE = 100 +# Batch endpoint URL for GCS. +# We have to specify an API specific endpoint here since Google APIs global +# batch endpoints will be deprecated on 03/25/2019. +# See https://developers.googleblog.com/2018/03/discontinuing-support-for-json-rpc-and.html. +# Currently apitools library uses a global batch endpoint by default: https://github.com/google/apitools/blob/master/apitools/base/py/batch.py#L152 +# TODO: remove this constant and it's usage after apitools move to using an API Review comment: Let's create a JIRA to move off of apitools. That library itself is in maintenance mode. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90151) Time Spent: 20m (was: 10m) > Update dependency 'google-api-services-storage' to latest version > - > > Key: BEAM-3991 > URL: https://issues.apache.org/jira/browse/BEAM-3991 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp, sdk-py-core >Reporter: Chamikara Jayalath >Assignee: Chamikara Jayalath >Priority: Blocker > Fix For: 2.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Currently we use version 'v1-rev71-1.22.0' which is deprecated and about two > years old. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3991) Update dependency 'google-api-services-storage' to latest version
[ https://issues.apache.org/jira/browse/BEAM-3991?focusedWorklogId=90133=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90133 ] ASF GitHub Bot logged work on BEAM-3991: Author: ASF GitHub Bot Created on: 11/Apr/18 19:07 Start Date: 11/Apr/18 19:07 Worklog Time Spent: 10m Work Description: chamikaramj opened a new pull request #5105: [BEAM-3991] Updates gcsio to use a API specific batch endpoint. URL: https://github.com/apache/beam/pull/5105 We have to specify an API specific endpoint here since Google APIs global batch endpoints will be deprecated on 03/25/2019. See https://developers.googleblog.com/2018/03/discontinuing-support-for-json-rpc-and.html. Currently apitools library uses a global batch endpoint by default: https://github.com/google/apitools/blob/master/apitools/base/py/batch.py#L152 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90133) Time Spent: 10m Remaining Estimate: 0h > Update dependency 'google-api-services-storage' to latest version > - > > Key: BEAM-3991 > URL: https://issues.apache.org/jira/browse/BEAM-3991 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp, sdk-py-core >Reporter: Chamikara Jayalath >Assignee: Chamikara Jayalath >Priority: Blocker > Fix For: 2.5.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently we use version 'v1-rev71-1.22.0' which is deprecated and about two > years old. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
