This is an automated email from the ASF dual-hosted git repository.
martijnvisser pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new 492a8862482 [FLINK-33704][Filesytems] Update GCS filesystems to latest
available versions. This closes #23837
492a8862482 is described below
commit 492a886248208904276fcd2bda138a79c86bc71c
Author: Martijn Visser <2989614+martijnvis...@users.noreply.github.com>
AuthorDate: Thu Nov 30 15:55:05 2023 +0100
[FLINK-33704][Filesytems] Update GCS filesystems to latest available
versions. This closes #23837
* [FLINK-33704][Filesystem] Update GCS filesystem dependencies
- Update GS SDK from 2.15.0 to 2.29.1
- Update GS Hadoop Connector from 2.2.15 to 2.2.18
- Update GS GRPC to highest shared version
* [FLINK-33704] Make NOTICE inclusion of com.google.re2j for GCS consistent
RE2J is listed for the other filesystems as Go license, we should remain
consistent throughout the project
---
docs/content.zh/docs/deployment/filesystems/gcs.md | 4 +-
docs/content/docs/deployment/filesystems/gcs.md | 4 +-
flink-filesystems/flink-gs-fs-hadoop/pom.xml | 10 +-
.../src/main/resources/META-INF/NOTICE | 105 +++++++++++----------
4 files changed, 64 insertions(+), 59 deletions(-)
diff --git a/docs/content.zh/docs/deployment/filesystems/gcs.md
b/docs/content.zh/docs/deployment/filesystems/gcs.md
index 363cd2fec90..7edf78b2e61 100644
--- a/docs/content.zh/docs/deployment/filesystems/gcs.md
+++ b/docs/content.zh/docs/deployment/filesystems/gcs.md
@@ -55,7 +55,7 @@ Note that these examples are *not* exhaustive and you can use
GCS in other place
Flink provides the `flink-gs-fs-hadoop` file system to write to GCS.
This implementation is self-contained with no dependency footprint, so there
is no need to add Hadoop to the classpath to use it.
-`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the
*gs://* scheme. It uses Google's
[gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/hadoop3-2.2.15)
Hadoop library to access GCS. It also uses Google's
[google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.15.0)
library to provide `RecoverableWriter` support.
+`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the
*gs://* scheme. It uses Google's
[gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/hadoop3-2.2.18)
Hadoop library to access GCS. It also uses Google's
[google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.29.1)
library to provide `RecoverableWriter` support.
This file system can be used with the [FileSystem connector]({{< ref
"docs/connectors/datastream/filesystem.md" >}}).
@@ -68,7 +68,7 @@ cp ./opt/flink-gs-fs-hadoop-{{< version >}}.jar
./plugins/gs-fs-hadoop/
### Configuration
-The underlying Hadoop file system can be configured using the [Hadoop
configuration
keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.15/gcs/CONFIGURATION.md)
for `gcs-connector` by adding the configurations to your `flink-conf.yaml`.
+The underlying Hadoop file system can be configured using the [Hadoop
configuration
keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.18/gcs/CONFIGURATION.md)
for `gcs-connector` by adding the configurations to your `flink-conf.yaml`.
For example, `gcs-connector` has a `fs.gs.http.connect-timeout` configuration
key. If you want to change it, you need to set `gs.http.connect-timeout: xyz`
in `flink-conf.yaml`. Flink will internally translate this back to
`fs.gs.http.connect-timeout`.
diff --git a/docs/content/docs/deployment/filesystems/gcs.md
b/docs/content/docs/deployment/filesystems/gcs.md
index 97e008eca33..3bd130e2046 100644
--- a/docs/content/docs/deployment/filesystems/gcs.md
+++ b/docs/content/docs/deployment/filesystems/gcs.md
@@ -55,7 +55,7 @@ Note that these examples are *not* exhaustive and you can use
GCS in other place
Flink provides the `flink-gs-fs-hadoop` file system to write to GCS.
This implementation is self-contained with no dependency footprint, so there
is no need to add Hadoop to the classpath to use it.
-`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the
*gs://* scheme. It uses Google's
[gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/hadoop3-2.2.15)
Hadoop library to access GCS. It also uses Google's
[google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.15.0)
library to provide `RecoverableWriter` support.
+`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the
*gs://* scheme. It uses Google's
[gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/hadoop3-2.2.18)
Hadoop library to access GCS. It also uses Google's
[google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.29.1)
library to provide `RecoverableWriter` support.
This file system can be used with the [FileSystem connector]({{< ref
"docs/connectors/datastream/filesystem.md" >}}).
@@ -68,7 +68,7 @@ cp ./opt/flink-gs-fs-hadoop-{{< version >}}.jar
./plugins/gs-fs-hadoop/
### Configuration
-The underlying Hadoop file system can be configured using the [Hadoop
configuration
keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.15/gcs/CONFIGURATION.md)
for `gcs-connector` by adding the configurations to your `flink-conf.yaml`.
+The underlying Hadoop file system can be configured using the [Hadoop
configuration
keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.18/gcs/CONFIGURATION.md)
for `gcs-connector` by adding the configurations to your `flink-conf.yaml`.
For example, `gcs-connector` has a `fs.gs.http.connect-timeout` configuration
key. If you want to change it, you need to set `gs.http.connect-timeout: xyz`
in `flink-conf.yaml`. Flink will internally translate this back to
`fs.gs.http.connect-timeout`.
diff --git a/flink-filesystems/flink-gs-fs-hadoop/pom.xml
b/flink-filesystems/flink-gs-fs-hadoop/pom.xml
index 4f98ffe3077..15841c1d1f5 100644
--- a/flink-filesystems/flink-gs-fs-hadoop/pom.xml
+++ b/flink-filesystems/flink-gs-fs-hadoop/pom.xml
@@ -34,10 +34,10 @@ under the License.
<properties>
<!-- If updating these dependency versions, please also update
the corresponding links -->
<!-- in the GCS file system documentation. -->
- <fs.gs.sdk.version>2.15.0</fs.gs.sdk.version>
-
<fs.gs.connector.version>hadoop3-2.2.15</fs.gs.connector.version>
+ <fs.gs.sdk.version>2.29.1</fs.gs.sdk.version>
+
<fs.gs.connector.version>hadoop3-2.2.18</fs.gs.connector.version>
<!-- Set this to the highest version of grpc artifacts from
gcs-connector and google-cloud-storage -->
- <fs.gs.grpc.version>1.50.3</fs.gs.grpc.version>
+ <fs.gs.grpc.version>1.59.1</fs.gs.grpc.version>
</properties>
<dependencies>
@@ -98,6 +98,10 @@ under the License.
<groupId>com.google.guava</groupId>
<artifactId>failureaccess</artifactId>
</exclusion>
+ <exclusion>
+ <groupId>com.google.errorprone</groupId>
+
<artifactId>error_prone_annotations</artifactId>
+ </exclusion>
<!-- exclude dependency because of its GPLv2
license, see https://github.com/apache/flink/pull/15599#issuecomment-850241316
-->
<exclusion>
<groupId>javax.annotation</groupId>
diff --git
a/flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/NOTICE
b/flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/NOTICE
index 59b3e8cc22c..7643f00c6c4 100644
--- a/flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/NOTICE
+++ b/flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/NOTICE
@@ -9,53 +9,55 @@ This project bundles the following dependencies under the
Apache Software Licens
- com.fasterxml.jackson.core:jackson-core:2.15.3
- com.google.android:annotations:4.1.1.4
- com.google.api-client:google-api-client-jackson2:2.0.1
-- com.google.api-client:google-api-client:2.0.1
-- com.google.api.grpc:gapic-google-cloud-storage-v2:2.15.0-alpha
-- com.google.api.grpc:grpc-google-cloud-storage-v2:2.15.0-alpha
-- com.google.api.grpc:grpc-google-iam-v1:1.6.7
+- com.google.api-client:google-api-client:2.2.0
+- com.google.api.grpc:gapic-google-cloud-storage-v2:2.29.1-alpha
+- com.google.api.grpc:grpc-google-cloud-storage-v2:2.29.1-alpha
- com.google.api.grpc:proto-google-cloud-monitoring-v3:1.64.0
-- com.google.api.grpc:proto-google-cloud-storage-v2:2.15.0-alpha
-- com.google.api.grpc:proto-google-common-protos:2.10.0
-- com.google.api.grpc:proto-google-iam-v1:1.6.7
+- com.google.api.grpc:proto-google-cloud-storage-v2:2.29.1-alpha
+- com.google.api.grpc:proto-google-common-protos:2.28.0
+- com.google.api.grpc:proto-google-iam-v1:1.23.0
- com.google.apis:google-api-services-iamcredentials:v1-rev20211203-2.0.0
-- com.google.apis:google-api-services-storage:v1-rev20220705-2.0.0
-- com.google.auto.value:auto-value-annotations:1.10
-- com.google.cloud.bigdataoss:gcs-connector:hadoop3-2.2.15
-- com.google.cloud.bigdataoss:gcsio:2.2.15
-- com.google.cloud.bigdataoss:util-hadoop:hadoop3-2.2.15
-- com.google.cloud.bigdataoss:util:2.2.15
-- com.google.cloud:google-cloud-core-grpc:2.8.27
-- com.google.cloud:google-cloud-core-http:2.8.27
-- com.google.cloud:google-cloud-core:2.8.27
+- com.google.apis:google-api-services-storage:v1-rev20231028-2.0.0
+- com.google.auto.value:auto-value-annotations:1.10.4
+- com.google.cloud.bigdataoss:gcs-connector:hadoop3-2.2.18
+- com.google.cloud.bigdataoss:gcsio:2.2.18
+- com.google.cloud.bigdataoss:util-hadoop:hadoop3-2.2.18
+- com.google.cloud.bigdataoss:util:2.2.18
+- com.google.cloud:google-cloud-core-grpc:2.27.0
+- com.google.cloud:google-cloud-core-http:2.27.0
+- com.google.cloud:google-cloud-core:2.27.0
- com.google.cloud:google-cloud-monitoring:1.82.0
-- com.google.cloud:google-cloud-storage:2.15.0
-- com.google.code.gson:gson:2.10
+- com.google.cloud:google-cloud-storage:2.29.1
+- com.google.code.gson:gson:2.10.1
- com.google.flogger:flogger-system-backend:0.7.1
- com.google.flogger:flogger:0.7.1
- com.google.flogger:google-extensions:0.7.1
- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
-- com.google.http-client:google-http-client-apache-v2:1.42.3
-- com.google.http-client:google-http-client-appengine:1.42.3
-- com.google.http-client:google-http-client-gson:1.42.3
-- com.google.http-client:google-http-client-jackson2:1.42.3
-- com.google.http-client:google-http-client:1.42.3
+- com.google.http-client:google-http-client-apache-v2:1.43.3
+- com.google.http-client:google-http-client-appengine:1.43.3
+- com.google.http-client:google-http-client-gson:1.43.3
+- com.google.http-client:google-http-client-jackson2:1.43.3
+- com.google.http-client:google-http-client:1.43.3
- com.google.oauth-client:google-oauth-client:1.34.1
- com.lmax:disruptor:3.4.2
- commons-codec:commons-codec:1.15
-- io.grpc:grpc-alts:1.50.3
-- io.grpc:grpc-api:1.50.3
-- io.grpc:grpc-auth:1.50.3
-- io.grpc:grpc-census:1.50.3
-- io.grpc:grpc-context:1.50.3
-- io.grpc:grpc-core:1.50.3
-- io.grpc:grpc-googleapis:1.50.3
-- io.grpc:grpc-grpclb:1.50.3
-- io.grpc:grpc-netty-shaded:1.50.3
-- io.grpc:grpc-protobuf-lite:1.50.3
-- io.grpc:grpc-protobuf:1.50.3
-- io.grpc:grpc-services:1.50.3
-- io.grpc:grpc-stub:1.50.3
-- io.grpc:grpc-xds:1.50.3
+- io.grpc:grpc-alts:1.59.1
+- io.grpc:grpc-api:1.59.1
+- io.grpc:grpc-auth:1.59.1
+- io.grpc:grpc-census:1.59.1
+- io.grpc:grpc-context:1.59.1
+- io.grpc:grpc-core:1.59.1
+- io.grpc:grpc-googleapis:1.59.1
+- io.grpc:grpc-grpclb:1.59.1
+- io.grpc:grpc-inprocess:1.59.1
+- io.grpc:grpc-netty-shaded:1.59.1
+- io.grpc:grpc-protobuf-lite:1.59.1
+- io.grpc:grpc-protobuf:1.59.1
+- io.grpc:grpc-rls:1.59.1
+- io.grpc:grpc-services:1.59.1
+- io.grpc:grpc-stub:1.59.1
+- io.grpc:grpc-util:1.59.1
+- io.grpc:grpc-xds:1.59.1
- io.opencensus:opencensus-api:0.31.1
- io.opencensus:opencensus-contrib-exemplar-util:0.31.0
- io.opencensus:opencensus-contrib-grpc-metrics:0.31.0
@@ -66,30 +68,29 @@ This project bundles the following dependencies under the
Apache Software Licens
- io.opencensus:opencensus-impl:0.31.0
- io.opencensus:opencensus-impl-core:0.31.0
- io.opencensus:opencensus-proto:0.2.0
-- io.perfmark:perfmark-api:0.25.0
+- io.perfmark:perfmark-api:0.26.0
- org.apache.httpcomponents:httpclient:4.5.13
- org.apache.httpcomponents:httpcore:4.4.14
- org.conscrypt:conscrypt-openjdk-uber:2.5.2
-This project bundles the following dependencies under BSD-2 License
(https://opensource.org/licenses/BSD-2-Clause).
+This project bundles the following dependencies under BSD-3 License
(https://opensource.org/licenses/BSD-3-Clause).
See bundled license files for details.
-- com.google.api:api-common:2.2.2
-- com.google.api:gax-grpc:2.19.5
-- com.google.api:gax-httpjson:0.104.5
-- com.google.api:gax:2.19.5
+- com.google.api:api-common:2.20.0
+- com.google.api:gax-grpc:2.37.0
+- com.google.api:gax-httpjson:2.37.0
+- com.google.api:gax:2.37.0
+- com.google.auth:google-auth-library-credentials:1.20.0
+- com.google.auth:google-auth-library-oauth2-http:1.20.0
+- com.google.protobuf:protobuf-java-util:3.24.4
+- com.google.protobuf:protobuf-java:3.24.4
+- org.threeten:threetenbp:1.6.8
-This project bundles the following dependencies under BSD-3 License
(https://opensource.org/licenses/BSD-3-Clause).
+This project bundles the following dependencies under the Go License
(https://golang.org/LICENSE).
See bundled license files for details.
-
-- com.google.auth:google-auth-library-credentials:1.12.1
-- com.google.auth:google-auth-library-oauth2-http:1.12.1
-- com.google.protobuf:protobuf-java-util:3.21.9
-- com.google.protobuf:protobuf-java:3.21.9
-- com.google.re2j:re2j:1.6
-- org.threeten:threetenbp:1.6.4
+- com.google.re2j:re2j:1.7
This project bundles the following dependencies under the MIT License.
See bundled license files for details.
-- org.codehaus.mojo:animal-sniffer-annotations:1.22
+- org.codehaus.mojo:animal-sniffer-annotations:1.23
