[GitHub] [spark] mridulm commented on pull request #38377: [SPARK-40901][CORE] Unable to store Spark Driver logs with Absolute Hadoop based URI FS Path

mridulm commented on PR #38377: URL: https://github.com/apache/spark/pull/38377#issuecomment-1311068364 Merged to master. Thanks for fixing this @swamirishi ! Thanks for the review @dongjoon-hyun :-) -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] mridulm commented on pull request #38377: [SPARK-40901][CORE] Unable to store Spark Driver logs with Absolute Hadoop based URI FS Path

mridulm commented on PR #38377: URL: https://github.com/apache/spark/pull/38377#issuecomment-1310907480 I wanted to give @dongjoon-hyun a chance to take a look if he has bandwidth, given he had reviewed it. Will merge it later tonight if there are no other comments -- This is an

[GitHub] [spark] mridulm commented on pull request #38377: [SPARK-40901][CORE] Unable to store Spark Driver logs with Absolute Hadoop based URI FS Path

mridulm commented on PR #38377: URL: https://github.com/apache/spark/pull/38377#issuecomment-1303007426 Two points: * spark.driver.log.dfsDir is typically expected to be a path to hdfs - so resolving it relative to current working directory does not make sense * If `rootDir` is

[GitHub] [spark] mridulm commented on pull request #38377: [SPARK-40901][CORE] Unable to store Spark Driver logs with Absolute Hadoop based URI FS Path

mridulm commented on PR #38377: URL: https://github.com/apache/spark/pull/38377#issuecomment-1301454290 I am not sure I follow - can you give an example ? Based on what the PR is doing, I would expect both to be equivalent. -- This is an automated message from the Apache Git Service.

[GitHub] [spark] mridulm commented on pull request #38377: [SPARK-40901][CORE] Unable to store Spark Driver logs with Absolute Hadoop based URI FS Path

mridulm commented on PR #38377: URL: https://github.com/apache/spark/pull/38377#issuecomment-1299613189 Makes sense ... why not simply `val dfsLogFile = new Path(rootDir, appId + DRIVER_LOG_FILE_SUFFIX)` instead btw ? I am trying to see if I am missing anything here ... -- This is an

[GitHub] [spark] mridulm commented on pull request #38377: [SPARK-40901][CORE] Unable to store Spark Driver logs with Absolute Hadoop based URI FS Path

mridulm commented on PR #38377: URL: https://github.com/apache/spark/pull/38377#issuecomment-1297907890 QQ: Any details for the query [above](https://github.com/apache/spark/pull/38377#pullrequestreview-1153485176) ? -- This is an automated message from the Apache Git Service. To