cxzl25 opened a new pull request, #51083:
URL: https://github.com/apache/spark/pull/51083
### What changes were proposed in this pull request?
### Why are the changes needed?
Now the `artifacts` directory will try to be created at workdir, which may
not have permission to create write.
`artifacts` is a temporary directory, and although we have an exit cleanup
mechanism, it's better to put it in tmpdir, driver because `artifacts` may not
be cleaned up after an OOM exit.
```java
org.apache.spark.sql.execution.QueryExecutionException: java.io.IOException:
Failed to create a temp directory (under artifacts) after 10 attempts!
at
org.apache.spark.network.util.JavaUtils.createDirectory(JavaUtils.java:411)
at
org.apache.spark.util.SparkFileUtils.createDirectory(SparkFileUtils.scala:95)
at
org.apache.spark.util.SparkFileUtils.createDirectory$(SparkFileUtils.scala:94)
at org.apache.spark.util.Utils$.createDirectory(Utils.scala:99)
at org.apache.spark.util.Utils$.createTempDir(Utils.scala:249)
at
org.apache.spark.sql.artifact.ArtifactManager$.artifactRootDirectory$lzycompute(ArtifactManager.scala:468)
at
org.apache.spark.sql.artifact.ArtifactManager$.artifactRootDirectory(ArtifactManager.scala:467)
at
org.apache.spark.sql.artifact.ArtifactManager.artifactRootPath(ArtifactManager.scala:60)
at
org.apache.spark.sql.artifact.ArtifactManager.<init>(ArtifactManager.scala:70)
at
org.apache.spark.sql.internal.BaseSessionStateBuilder.$anonfun$artifactManager$2(BaseSessionStateBuilder.scala:395)
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
local test
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
