Re: [I] [SUPPORT] EMR on EKS version 6.15.0, Spark 3.4.1 and Hudi 0.14.0 getting java.io.IOException: Failed to delete: /usr/lib/hudi/. [hudi]
CTTY commented on issue #10376: URL: https://github.com/apache/hudi/issues/10376#issuecomment-1974124259 I think we need to understand why Spark is trying to delete Hudi jars here, can you share the previous log so we can see what Spark is doing before this exception? On the other hand, I wouldn't use `/usr/lib/hudi/` to put your own script as the entire directory is owned by root user there and you may run into some weird permission issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT] EMR on EKS version 6.15.0, Spark 3.4.1 and Hudi 0.14.0 getting java.io.IOException: Failed to delete: /usr/lib/hudi/. [hudi]
Lakshmi-Holla12 commented on issue #10376: URL: https://github.com/apache/hudi/issues/10376#issuecomment-1876894215 Hi @ad1happy2go @CTTY, I have been testing EMR on EKS with different EMR versions: **emr-6.15.0, emr-6.14.0, emr-6.13.0, and emr-6.11.0.** I found that emr-6.11.0 works fine with Spark 3.3.2 and hudi 0.13.0. However, emr-6.14.0 and emr-6.13.0, which use Spark 3.4.1 and hudi 0.13.1, started to fail with the same error message: "java.io.IOException: Failed to delete: /usr/lib/hudi/./hudi-spark3-bundle_2.12-0.13.1-amzn-2.jar". Similarly, emr-6.15.0, which uses Spark 3.4.1 and hudi 0.14.0, also encountered the same issue with the error message: "java.io.IOException: Failed to delete: /usr/lib/hudi/./hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar". I am not sure what changes in Spark 3.4.1 could be causing the deletion of these files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT] EMR on EKS version 6.15.0, Spark 3.4.1 and Hudi 0.14.0 getting java.io.IOException: Failed to delete: /usr/lib/hudi/. [hudi]
ad1happy2go commented on issue #10376: URL: https://github.com/apache/hudi/issues/10376#issuecomment-1875578004 @CTTY Any insights on this. thanks in advance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT] EMR on EKS version 6.15.0, Spark 3.4.1 and Hudi 0.14.0 getting java.io.IOException: Failed to delete: /usr/lib/hudi/. [hudi]
Lakshmi-Holla12 commented on issue #10376: URL: https://github.com/apache/hudi/issues/10376#issuecomment-1875098138 Hi @ad1happy2go, I'm still experiencing the issue with EMR 6.15. By default, Spark jar is located at /usr/lib/hudi, which I am currently using. During the spark-submit, I provide the following parameters `--jars /usr/lib/hudi/hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar` and `--py-files /usr/lib/hudi/`. However, I'm not sure why Spark is trying to delete the files. I've tried a few steps to resolve the issue but to no luck: 1. I attempted putting the Hudi Spark jar in a different location and ran into the same error. 2. I removed the `--jars` parameter during Spark submit, this time it looks for `--py-files` and throws the same error with not able to delete /usr/lib/hudi/./. 3. I tried setting this configuration `spark.hadoop.fs.s3a.deletion.enable`, `false`. This was meant to disable the deletion of temporary files by the S3A filesystem. However, I'm still facing the same issue. 4. I also tried setting `spark.hadoop.io.native.lib.available`, `false`. This was to disable the use of native Unix OS commands for deletion. However, I'm still facing the same issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT] EMR on EKS version 6.15.0, Spark 3.4.1 and Hudi 0.14.0 getting java.io.IOException: Failed to delete: /usr/lib/hudi/. [hudi]
ad1happy2go commented on issue #10376: URL: https://github.com/apache/hudi/issues/10376#issuecomment-1874094831 @Lakshmi-Holla12 Were you able to resolve this issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT] EMR on EKS version 6.15.0, Spark 3.4.1 and Hudi 0.14.0 getting java.io.IOException: Failed to delete: /usr/lib/hudi/. [hudi]
ad1happy2go commented on issue #10376: URL: https://github.com/apache/hudi/issues/10376#issuecomment-1867797289 @Lakshmi-Holla12 Interesting. Not sure why spark is trying to delete the jar from that location. Can you try putting hudi spark jar in some different location and point that to it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [SUPPORT] EMR on EKS version 6.15.0, Spark 3.4.1 and Hudi 0.14.0 getting java.io.IOException: Failed to delete: /usr/lib/hudi/. [hudi]
Lakshmi-Holla12 opened a new issue, #10376: URL: https://github.com/apache/hudi/issues/10376 **Describe the problem you faced** Getting Failed to delete the hudi jar from /usr/lib/hudi/./ location after upgrading the EMR to 6.15.0 which has in built hudi 0.14.0 and Spark 3.4.1 version. The application fails at the start of the spark application with the below error message, ``` Files local:///usr/lib/hudi/hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar from /usr/lib/hudi/hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar to /usr/lib/hudi/./hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar 23/12/19 10:11:38 WARN JavaUtils: Attempt to delete using native Unix OS command failed for path = /usr/lib/hudi/./hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar. Falling back to Java IO way java.io.IOException: Failed to delete: /usr/lib/hudi/./hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar at org.apache.spark.network.util.JavaUtils.deleteRecursivelyUsingUnixNative(JavaUtils.java:177) ~[spark-network-common_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2] at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:113) ~[spark-network-common_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2] at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:94) ~[spark-network-common_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2] at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1232) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2] at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$14(SparkSubmit.scala:436) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2] at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286) ~[scala-library-2.12.15.jar:?] at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) ~[scala-library-2.12.15.jar:?] at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) ~[scala-library-2.12.15.jar:?] at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) ~[scala-library-2.12.15.jar:?] at scala.collection.TraversableLike.map(TraversableLike.scala:286) ~[scala-library-2.12.15.jar:?] at scala.collection.TraversableLike.map$(TraversableLike.scala:279) ~[scala-library-2.12.15.jar:?] at scala.collection.AbstractTraversable.map(Traversable.scala:108) ~[scala-library-2.12.15.jar:?] at org.apache.spark.deploy.SparkSubmit.downloadResourcesToCurrentDirectory$1(SparkSubmit.scala:427) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2] at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$17(SparkSubmit.scala:452) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2] at scala.Option.map(Option.scala:230) ~[scala-library-2.12.15.jar:?] at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:452) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2] at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1001) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2] at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2] at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2] at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2] at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1158) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2] at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1167) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2] at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2] Exception in thread "main" java.io.IOException: Failed to delete: /usr/lib/hudi/./hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar at org.apache.spark.network.util.JavaUtils.deleteRecursivelyUsingJavaIO(JavaUtils.java:150) at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:121) at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:94) at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1232) at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$14(SparkSubmit.scala:436) at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at scala.collection.TraversableLike.map(TraversableLike.scala:286) at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
