[GitHub] [flink] MartijnVisser commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2

2022-04-26 Thread GitBox 


MartijnVisser commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1110280345

   S3 E2E tests also passed (ignoring the post step of caching the Maven repo), 
per 
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=35145&view=results
   
   Merging this now


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] MartijnVisser commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2

2022-04-26 Thread GitBox 


MartijnVisser commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1110130110

   @AHeise @tweise What's your opinion on backporting this to Flink 1.14 and 
Flink 1.15? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] MartijnVisser commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2

2022-04-26 Thread GitBox 


MartijnVisser commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1110129504

   @tweise @AHeise I've completed the verification of the dependency tree. I've 
also included a notice for the so called `Apache Hadoop Relocated (Shaded) 
Third-party Libs` with regards to the Shaded Protobuf and/or Shaded Guava, 
similar like we do for `flink-python` (with Beam) and for the HBase connectors. 
   
   I've also rebased this PR to be sure, I'll let CI verify this once more and 
if everything turns green, I'll merge it. 
   
   Thanks again @chinmayms @tweise @AHeise 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] MartijnVisser commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2

2022-04-25 Thread GitBox 


MartijnVisser commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1108878229

   @tweise Are you referring to the ones like 
`org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7` and 
`org.apache.hadoop.thirdparty:hadoop-shaded-guava` ? 
   
   @AHeise I ran and compared the output of `mvn dependency:tree`. The removed 
entries are not listed in that output anymore, so I've cleaned them up. I'm 
suspecting that the downgrade notice is specific for the Azure FS 
implementation: that's currently using 
`org.apache.hadoop:hadoop-azure:jar:3.2.2:compile` in `master`, which depends 
on `com.google.guava:guava:jar:27.0-jre:compile`. See:
   
   ```
   [INFO] +- org.apache.hadoop:hadoop-azure:jar:3.2.2:compile
   [INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.5.13:compile
   [INFO] |  |  +- org.apache.httpcomponents:httpcore:jar:4.4.14:compile
   [INFO] |  |  \- commons-codec:commons-codec:jar:1.15:compile
   [INFO] |  +- com.microsoft.azure:azure-storage:jar:7.0.0:compile
   [INFO] |  |  +- com.fasterxml.jackson.core:jackson-core:jar:2.13.2:compile
   [INFO] |  |  \- com.microsoft.azure:azure-keyvault-core:jar:1.0.0:compile
   [INFO] |  +- com.google.guava:guava:jar:27.0-jre:compile
   [INFO] |  |  +- com.google.guava:failureaccess:jar:1.0:compile
   [INFO] |  |  +- 
com.google.guava:listenablefuture:jar:.0-empty-to-avoid-conflict-with-guava:compile
   [INFO] |  |  +- org.checkerframework:checker-qual:jar:2.5.2:compile
   [INFO] |  |  +- 
com.google.errorprone:error_prone_annotations:jar:2.2.0:compile
   [INFO] |  |  +- com.google.j2objc:j2objc-annotations:jar:1.1:compile
   [INFO] |  |  \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:compile
   [INFO] |  +- org.eclipse.jetty:jetty-util-ajax:jar:9.3.24.v20180605:compile
   [INFO] |  |  \- org.eclipse.jetty:jetty-util:jar:9.3.24.v20180605:compile
   [INFO] |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
   [INFO] |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
   [INFO] |  \- org.wildfly.openssl:wildfly-openssl:jar:1.0.7.Final:compile
   ```
   
   The newer version has a smaller dependency footprint, probably due to them 
switching to the shaded version of Guava. 
   ```
   [INFO] +- org.apache.hadoop:hadoop-azure:jar:3.3.2:compile
   [INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.5.13:compile
   [INFO] |  |  +- org.apache.httpcomponents:httpcore:jar:4.4.14:compile
   [INFO] |  |  \- commons-codec:commons-codec:jar:1.15:compile
   [INFO] |  +- com.microsoft.azure:azure-storage:jar:7.0.1:compile
   [INFO] |  |  +- com.fasterxml.jackson.core:jackson-core:jar:2.13.2:compile
   [INFO] |  |  \- com.microsoft.azure:azure-keyvault-core:jar:1.0.0:compile
   [INFO] |  +- 
org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.1.1:compile
   [INFO] |  +- org.eclipse.jetty:jetty-util-ajax:jar:9.3.24.v20180605:compile
   [INFO] |  |  \- org.eclipse.jetty:jetty-util:jar:9.3.24.v20180605:compile
   [INFO] |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
   [INFO] |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
   [INFO] |  \- org.wildfly.openssl:wildfly-openssl:jar:1.0.7.Final:compile
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] MartijnVisser commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2

2022-04-22 Thread GitBox 


MartijnVisser commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1106821488

   @tweise The S3 tests have passed, see 
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=35010&view=results
 - Can you review the PR? CC @chinmayms 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] MartijnVisser commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2

2022-04-21 Thread GitBox 


MartijnVisser commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1105705076

   @chinmayms Ah I missed your comment. I'll first try with the removal of the 
exclusion, else I'll explicitly add it. Most likely it will be during the 
weekend


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] MartijnVisser commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2

2022-04-21 Thread GitBox 


MartijnVisser commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1105613053

   I'm suspecting this is caused by the exclusion of commons-compress, see 
https://github.com/apache/flink/blob/master/flink-filesystems/flink-fs-hadoop-shaded/pom.xml#L157-L160


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] MartijnVisser commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2

2022-04-21 Thread GitBox 


MartijnVisser commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1105255497

   One of the S3 tests is failing unfortunately:
   
   ```
   2022-04-21T14:01:10.5059692Z Apr 21 14:01:09 
org.apache.flink.client.program.ProgramInvocationException: The main method 
caused an error: org.apache.flink.client.program.ProgramInvocationException: 
Job failed (JobID: c97647441357fd84a29e08d77d040975)
   2022-04-21T14:01:10.5061026Z Apr 21 14:01:09 at 
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
 ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
   2022-04-21T14:01:10.5062344Z Apr 21 14:01:09 at 
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
 ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
   2022-04-21T14:01:10.5063591Z Apr 21 14:01:09 at 
org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) 
~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
   2022-04-21T14:01:10.5064689Z Apr 21 14:01:09 at 
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:836) 
~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
   2022-04-21T14:01:10.5065843Z Apr 21 14:01:09 at 
org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:247) 
~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
   2022-04-21T14:01:10.5067009Z Apr 21 14:01:09 at 
org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1078) 
~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
   2022-04-21T14:01:10.5068184Z Apr 21 14:01:09 at 
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1156) 
~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
   2022-04-21T14:01:10.5069338Z Apr 21 14:01:09 at 
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
 [flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
   2022-04-21T14:01:10.5070557Z Apr 21 14:01:09 at 
org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1156) 
[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
   2022-04-21T14:01:10.5071520Z Apr 21 14:01:09 Caused by: 
java.util.concurrent.ExecutionException: 
org.apache.flink.client.program.ProgramInvocationException: Job failed (JobID: 
c97647441357fd84a29e08d77d040975)
   2022-04-21T14:01:10.5072419Z Apr 21 14:01:09 at 
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) 
~[?:1.8.0_322]
   2022-04-21T14:01:10.5073108Z Apr 21 14:01:09 at 
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) 
~[?:1.8.0_322]
   2022-04-21T14:01:10.5074296Z Apr 21 14:01:09 at 
org.apache.flink.client.program.ContextEnvironment.getJobExecutionResult(ContextEnvironment.java:117)
 ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
   2022-04-21T14:01:10.5075572Z Apr 21 14:01:09 at 
org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:74)
 ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
   2022-04-21T14:01:10.5076359Z Apr 21 14:01:09 at 
org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:93) 
~[?:?]
   2022-04-21T14:01:10.5077158Z Apr 21 14:01:09 at 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_322]
   2022-04-21T14:01:10.5077791Z Apr 21 14:01:09 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[?:1.8.0_322]
   2022-04-21T14:01:10.5078622Z Apr 21 14:01:09 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_322]
   2022-04-21T14:01:10.5079285Z Apr 21 14:01:09 at 
java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_322]
   2022-04-21T14:01:10.5080384Z Apr 21 14:01:09 at 
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
 ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
   2022-04-21T14:01:10.5081009Z Apr 21 14:01:09 ... 8 more
   2022-04-21T14:01:10.5081639Z Apr 21 14:01:09 Caused by: 
org.apache.flink.client.program.ProgramInvocationException: Job failed (JobID: 
c97647441357fd84a29e08d77d040975)
   2022-04-21T14:01:10.5082795Z Apr 21 14:01:09 at 
org.apache.flink.client.deployment.ClusterClientJobClientAdapter.lambda$null$6(ClusterClientJobClientAdapter.java:130)
 ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
   2022-04-21T14:01:10.5083778Z Apr 21 14:01:09 at 
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616) 
~[?:1.8.0_322]
   2022-04-21T14:01:10.5084587Z Apr 21 14:01:09 at 
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
 ~[?:1.8.0_322]
   2022-04-21T14:01:10.5085313Z Apr 21 14:01:09 at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) 
~[?:1.8.0_322]
   2022-04-21T14:01:10.5086106Z Apr 21 14:01:09 at 
java.util.concurrent.CompletableFuture.com

[GitHub] [flink] MartijnVisser commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2

2022-04-21 Thread GitBox 


MartijnVisser commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1105111870

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] MartijnVisser commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2

2022-04-21 Thread GitBox 


MartijnVisser commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1104925163

   @chinmayms I see that @zentol has closed your PR, so I've cherrypicked and 
co-authored your changes and put them into this PR. I was only starting to see 
what changes where necessary for the dependency upgrade, so I'm really happy 
that you already helped with a fix :)
   
   I'll need to run the S3 tests (they are not running as part of this CI) so 
I'll do that. If @tweise can review it once more, we should be able to merge 
this in the next couple of days. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org