This is an automated email from the ASF dual-hosted git repository. martijnvisser pushed a commit to branch release-1.18 in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/release-1.18 by this push: new 8951f907b46 [FLINK-33704][Filesytems] Update GCS filesystems to latest available versions. This closes #23935 8951f907b46 is described below commit 8951f907b4633d6826852d62920ef638905c7c78 Author: Martijn Visser <2989614+martijnvis...@users.noreply.github.com> AuthorDate: Mon Dec 18 09:05:00 2023 +0100 [FLINK-33704][Filesytems] Update GCS filesystems to latest available versions. This closes #23935 * [FLINK-33704][Filesystem] Update GCS filesystem dependencies - Update GS SDK from 2.15.0 to 2.29.1 - Update GS Hadoop Connector from 2.2.15 to 2.2.18 - Update GS GRPC to highest shared version * [FLINK-33704] Make NOTICE inclusion of com.google.re2j for GCS consistent RE2J is listed for the other filesystems as Go license, we should remain consistent throughout the project (cherry picked from commit 492a886248208904276fcd2bda138a79c86bc71c) --- docs/content.zh/docs/deployment/filesystems/gcs.md | 4 +- docs/content/docs/deployment/filesystems/gcs.md | 4 +- flink-filesystems/flink-gs-fs-hadoop/pom.xml | 10 +- .../src/main/resources/META-INF/NOTICE | 105 +++++++++++---------- 4 files changed, 64 insertions(+), 59 deletions(-) diff --git a/docs/content.zh/docs/deployment/filesystems/gcs.md b/docs/content.zh/docs/deployment/filesystems/gcs.md index 363cd2fec90..7edf78b2e61 100644 --- a/docs/content.zh/docs/deployment/filesystems/gcs.md +++ b/docs/content.zh/docs/deployment/filesystems/gcs.md @@ -55,7 +55,7 @@ Note that these examples are *not* exhaustive and you can use GCS in other place Flink provides the `flink-gs-fs-hadoop` file system to write to GCS. This implementation is self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use it. -`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the *gs://* scheme. It uses Google's [gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/hadoop3-2.2.15) Hadoop library to access GCS. It also uses Google's [google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.15.0) library to provide `RecoverableWriter` support. +`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the *gs://* scheme. It uses Google's [gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/hadoop3-2.2.18) Hadoop library to access GCS. It also uses Google's [google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.29.1) library to provide `RecoverableWriter` support. This file system can be used with the [FileSystem connector]({{< ref "docs/connectors/datastream/filesystem.md" >}}). @@ -68,7 +68,7 @@ cp ./opt/flink-gs-fs-hadoop-{{< version >}}.jar ./plugins/gs-fs-hadoop/ ### Configuration -The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.15/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your `flink-conf.yaml`. +The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.18/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your `flink-conf.yaml`. For example, `gcs-connector` has a `fs.gs.http.connect-timeout` configuration key. If you want to change it, you need to set `gs.http.connect-timeout: xyz` in `flink-conf.yaml`. Flink will internally translate this back to `fs.gs.http.connect-timeout`. diff --git a/docs/content/docs/deployment/filesystems/gcs.md b/docs/content/docs/deployment/filesystems/gcs.md index 97e008eca33..3bd130e2046 100644 --- a/docs/content/docs/deployment/filesystems/gcs.md +++ b/docs/content/docs/deployment/filesystems/gcs.md @@ -55,7 +55,7 @@ Note that these examples are *not* exhaustive and you can use GCS in other place Flink provides the `flink-gs-fs-hadoop` file system to write to GCS. This implementation is self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use it. -`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the *gs://* scheme. It uses Google's [gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/hadoop3-2.2.15) Hadoop library to access GCS. It also uses Google's [google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.15.0) library to provide `RecoverableWriter` support. +`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the *gs://* scheme. It uses Google's [gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/hadoop3-2.2.18) Hadoop library to access GCS. It also uses Google's [google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.29.1) library to provide `RecoverableWriter` support. This file system can be used with the [FileSystem connector]({{< ref "docs/connectors/datastream/filesystem.md" >}}). @@ -68,7 +68,7 @@ cp ./opt/flink-gs-fs-hadoop-{{< version >}}.jar ./plugins/gs-fs-hadoop/ ### Configuration -The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.15/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your `flink-conf.yaml`. +The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.18/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your `flink-conf.yaml`. For example, `gcs-connector` has a `fs.gs.http.connect-timeout` configuration key. If you want to change it, you need to set `gs.http.connect-timeout: xyz` in `flink-conf.yaml`. Flink will internally translate this back to `fs.gs.http.connect-timeout`. diff --git a/flink-filesystems/flink-gs-fs-hadoop/pom.xml b/flink-filesystems/flink-gs-fs-hadoop/pom.xml index 623f7068d56..294c3c4636e 100644 --- a/flink-filesystems/flink-gs-fs-hadoop/pom.xml +++ b/flink-filesystems/flink-gs-fs-hadoop/pom.xml @@ -34,10 +34,10 @@ under the License. <properties> <!-- If updating these dependency versions, please also update the corresponding links --> <!-- in the GCS file system documentation. --> - <fs.gs.sdk.version>2.15.0</fs.gs.sdk.version> - <fs.gs.connector.version>hadoop3-2.2.15</fs.gs.connector.version> + <fs.gs.sdk.version>2.29.1</fs.gs.sdk.version> + <fs.gs.connector.version>hadoop3-2.2.18</fs.gs.connector.version> <!-- Set this to the highest version of grpc artifacts from gcs-connector and google-cloud-storage --> - <fs.gs.grpc.version>1.50.3</fs.gs.grpc.version> + <fs.gs.grpc.version>1.59.1</fs.gs.grpc.version> </properties> <dependencies> @@ -90,6 +90,10 @@ under the License. <groupId>com.google.guava</groupId> <artifactId>failureaccess</artifactId> </exclusion> + <exclusion> + <groupId>com.google.errorprone</groupId> + <artifactId>error_prone_annotations</artifactId> + </exclusion> <!-- exclude dependency because of its GPLv2 license, see https://github.com/apache/flink/pull/15599#issuecomment-850241316 --> <exclusion> <groupId>javax.annotation</groupId> diff --git a/flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/NOTICE b/flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/NOTICE index 7a34d1b4ef6..d27a1484d83 100644 --- a/flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/NOTICE +++ b/flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/NOTICE @@ -9,53 +9,55 @@ This project bundles the following dependencies under the Apache Software Licens - com.fasterxml.jackson.core:jackson-core:2.14.3 - com.google.android:annotations:4.1.1.4 - com.google.api-client:google-api-client-jackson2:2.0.1 -- com.google.api-client:google-api-client:2.0.1 -- com.google.api.grpc:gapic-google-cloud-storage-v2:2.15.0-alpha -- com.google.api.grpc:grpc-google-cloud-storage-v2:2.15.0-alpha -- com.google.api.grpc:grpc-google-iam-v1:1.6.7 +- com.google.api-client:google-api-client:2.2.0 +- com.google.api.grpc:gapic-google-cloud-storage-v2:2.29.1-alpha +- com.google.api.grpc:grpc-google-cloud-storage-v2:2.29.1-alpha - com.google.api.grpc:proto-google-cloud-monitoring-v3:1.64.0 -- com.google.api.grpc:proto-google-cloud-storage-v2:2.15.0-alpha -- com.google.api.grpc:proto-google-common-protos:2.10.0 -- com.google.api.grpc:proto-google-iam-v1:1.6.7 +- com.google.api.grpc:proto-google-cloud-storage-v2:2.29.1-alpha +- com.google.api.grpc:proto-google-common-protos:2.28.0 +- com.google.api.grpc:proto-google-iam-v1:1.23.0 - com.google.apis:google-api-services-iamcredentials:v1-rev20211203-2.0.0 -- com.google.apis:google-api-services-storage:v1-rev20220705-2.0.0 -- com.google.auto.value:auto-value-annotations:1.10 -- com.google.cloud.bigdataoss:gcs-connector:hadoop3-2.2.15 -- com.google.cloud.bigdataoss:gcsio:2.2.15 -- com.google.cloud.bigdataoss:util-hadoop:hadoop3-2.2.15 -- com.google.cloud.bigdataoss:util:2.2.15 -- com.google.cloud:google-cloud-core-grpc:2.8.27 -- com.google.cloud:google-cloud-core-http:2.8.27 -- com.google.cloud:google-cloud-core:2.8.27 +- com.google.apis:google-api-services-storage:v1-rev20231028-2.0.0 +- com.google.auto.value:auto-value-annotations:1.10.4 +- com.google.cloud.bigdataoss:gcs-connector:hadoop3-2.2.18 +- com.google.cloud.bigdataoss:gcsio:2.2.18 +- com.google.cloud.bigdataoss:util-hadoop:hadoop3-2.2.18 +- com.google.cloud.bigdataoss:util:2.2.18 +- com.google.cloud:google-cloud-core-grpc:2.27.0 +- com.google.cloud:google-cloud-core-http:2.27.0 +- com.google.cloud:google-cloud-core:2.27.0 - com.google.cloud:google-cloud-monitoring:1.82.0 -- com.google.cloud:google-cloud-storage:2.15.0 -- com.google.code.gson:gson:2.10 +- com.google.cloud:google-cloud-storage:2.29.1 +- com.google.code.gson:gson:2.10.1 - com.google.flogger:flogger-system-backend:0.7.1 - com.google.flogger:flogger:0.7.1 - com.google.flogger:google-extensions:0.7.1 - com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava -- com.google.http-client:google-http-client-apache-v2:1.42.3 -- com.google.http-client:google-http-client-appengine:1.42.3 -- com.google.http-client:google-http-client-gson:1.42.3 -- com.google.http-client:google-http-client-jackson2:1.42.3 -- com.google.http-client:google-http-client:1.42.3 +- com.google.http-client:google-http-client-apache-v2:1.43.3 +- com.google.http-client:google-http-client-appengine:1.43.3 +- com.google.http-client:google-http-client-gson:1.43.3 +- com.google.http-client:google-http-client-jackson2:1.43.3 +- com.google.http-client:google-http-client:1.43.3 - com.google.oauth-client:google-oauth-client:1.34.1 - com.lmax:disruptor:3.4.2 - commons-codec:commons-codec:1.15 -- io.grpc:grpc-alts:1.50.3 -- io.grpc:grpc-api:1.50.3 -- io.grpc:grpc-auth:1.50.3 -- io.grpc:grpc-census:1.50.3 -- io.grpc:grpc-context:1.50.3 -- io.grpc:grpc-core:1.50.3 -- io.grpc:grpc-googleapis:1.50.3 -- io.grpc:grpc-grpclb:1.50.3 -- io.grpc:grpc-netty-shaded:1.50.3 -- io.grpc:grpc-protobuf-lite:1.50.3 -- io.grpc:grpc-protobuf:1.50.3 -- io.grpc:grpc-services:1.50.3 -- io.grpc:grpc-stub:1.50.3 -- io.grpc:grpc-xds:1.50.3 +- io.grpc:grpc-alts:1.59.1 +- io.grpc:grpc-api:1.59.1 +- io.grpc:grpc-auth:1.59.1 +- io.grpc:grpc-census:1.59.1 +- io.grpc:grpc-context:1.59.1 +- io.grpc:grpc-core:1.59.1 +- io.grpc:grpc-googleapis:1.59.1 +- io.grpc:grpc-grpclb:1.59.1 +- io.grpc:grpc-inprocess:1.59.1 +- io.grpc:grpc-netty-shaded:1.59.1 +- io.grpc:grpc-protobuf-lite:1.59.1 +- io.grpc:grpc-protobuf:1.59.1 +- io.grpc:grpc-rls:1.59.1 +- io.grpc:grpc-services:1.59.1 +- io.grpc:grpc-stub:1.59.1 +- io.grpc:grpc-util:1.59.1 +- io.grpc:grpc-xds:1.59.1 - io.opencensus:opencensus-api:0.31.1 - io.opencensus:opencensus-contrib-exemplar-util:0.31.0 - io.opencensus:opencensus-contrib-grpc-metrics:0.31.0 @@ -66,30 +68,29 @@ This project bundles the following dependencies under the Apache Software Licens - io.opencensus:opencensus-impl:0.31.0 - io.opencensus:opencensus-impl-core:0.31.0 - io.opencensus:opencensus-proto:0.2.0 -- io.perfmark:perfmark-api:0.25.0 +- io.perfmark:perfmark-api:0.26.0 - org.apache.httpcomponents:httpclient:4.5.13 - org.apache.httpcomponents:httpcore:4.4.14 - org.conscrypt:conscrypt-openjdk-uber:2.5.2 -This project bundles the following dependencies under BSD-2 License (https://opensource.org/licenses/BSD-2-Clause). +This project bundles the following dependencies under BSD-3 License (https://opensource.org/licenses/BSD-3-Clause). See bundled license files for details. -- com.google.api:api-common:2.2.2 -- com.google.api:gax-grpc:2.19.5 -- com.google.api:gax-httpjson:0.104.5 -- com.google.api:gax:2.19.5 +- com.google.api:api-common:2.20.0 +- com.google.api:gax-grpc:2.37.0 +- com.google.api:gax-httpjson:2.37.0 +- com.google.api:gax:2.37.0 +- com.google.auth:google-auth-library-credentials:1.20.0 +- com.google.auth:google-auth-library-oauth2-http:1.20.0 +- com.google.protobuf:protobuf-java-util:3.24.4 +- com.google.protobuf:protobuf-java:3.24.4 +- org.threeten:threetenbp:1.6.8 -This project bundles the following dependencies under BSD-3 License (https://opensource.org/licenses/BSD-3-Clause). +This project bundles the following dependencies under the Go License (https://golang.org/LICENSE). See bundled license files for details. - -- com.google.auth:google-auth-library-credentials:1.12.1 -- com.google.auth:google-auth-library-oauth2-http:1.12.1 -- com.google.protobuf:protobuf-java-util:3.21.9 -- com.google.protobuf:protobuf-java:3.21.9 -- com.google.re2j:re2j:1.6 -- org.threeten:threetenbp:1.6.4 +- com.google.re2j:re2j:1.7 This project bundles the following dependencies under the MIT License. See bundled license files for details. -- org.codehaus.mojo:animal-sniffer-annotations:1.22 +- org.codehaus.mojo:animal-sniffer-annotations:1.23