Lakshmi-Holla12 commented on issue #10376: URL: https://github.com/apache/hudi/issues/10376#issuecomment-1875098138
Hi @ad1happy2go, I'm still experiencing the issue with EMR 6.15. By default, Spark jar is located at /usr/lib/hudi, which I am currently using. During the spark-submit, I provide the following parameters `--jars /usr/lib/hudi/hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar` and `--py-files /usr/lib/hudi/<my src code path>`. However, I'm not sure why Spark is trying to delete the files. I've tried a few steps to resolve the issue but to no luck: 1. I attempted putting the Hudi Spark jar in a different location and ran into the same error. 2. I removed the `--jars` parameter during Spark submit, this time it looks for `--py-files` and throws the same error with not able to delete /usr/lib/hudi/./<my src code path>. 3. I tried setting this configuration `spark.hadoop.fs.s3a.deletion.enable`, `false`. This was meant to disable the deletion of temporary files by the S3A filesystem. However, I'm still facing the same issue. 4. I also tried setting `spark.hadoop.io.native.lib.available`, `false`. This was to disable the use of native Unix OS commands for deletion. However, I'm still facing the same issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org