This is an automated email from the ASF dual-hosted git repository.
yangjie01 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 09e726d3658f [SPARK-51605][CONNECT] Try to create the parent directory
before touching the logFile
09e726d3658f is described below
commit 09e726d3658f4ee6f2e96375db9b5551165d5a36
Author: yangjie01 <[email protected]>
AuthorDate: Thu Mar 27 13:06:09 2025 +0800
[SPARK-51605][CONNECT] Try to create the parent directory before touching
the logFile
### What changes were proposed in this pull request?
This PR try to create the parent directory before touching the log file in
`connect.SparkSession.withLocalConnectServer` to avoid issues when the parent
directory does not exist.
### Why are the changes needed?
When the `logs` directory does not exist under the `SPARK_HOME` path,
executing `bin/spark-shell --remote local` will result in the following error:
```
bin/spark-shell --remote local
WARNING: Using incubator modules: jdk.incubator.vector
Exception in thread "main" java.nio.file.NoSuchFileException:
/Users/yangjie01/Tools/spark-4.1.0-SNAPSHOT-bin-3.4.1/logs
at
java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at
java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at
java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:148)
at java.base/java.nio.file.Files.readAttributes(Files.java:1851)
at
java.base/sun.nio.fs.PollingWatchService.doPrivilegedRegister(PollingWatchService.java:173)
at
java.base/sun.nio.fs.PollingWatchService$2.run(PollingWatchService.java:154)
at
java.base/sun.nio.fs.PollingWatchService$2.run(PollingWatchService.java:151)
at
java.base/java.security.AccessController.doPrivileged(AccessController.java:569)
at
java.base/sun.nio.fs.PollingWatchService.register(PollingWatchService.java:150)
at java.base/sun.nio.fs.UnixPath.register(UnixPath.java:885)
at java.base/java.nio.file.Path.register(Path.java:894)
at
org.apache.spark.sql.connect.SparkSession$.waitUntilFileExists(SparkSession.scala:717)
at
org.apache.spark.sql.connect.SparkSession$.$anonfun$withLocalConnectServer$13(SparkSession.scala:798)
at
org.apache.spark.sql.connect.SparkSession$.$anonfun$withLocalConnectServer$13$adapted(SparkSession.scala:791)
at scala.Option.foreach(Option.scala:437)
at
org.apache.spark.sql.connect.SparkSession$.withLocalConnectServer(SparkSession.scala:791)
at
org.apache.spark.sql.application.ConnectRepl$.doMain(ConnectRepl.scala:67)
at
org.apache.spark.sql.application.ConnectRepl$.main(ConnectRepl.scala:57)
at org.apache.spark.sql.application.ConnectRepl.main(ConnectRepl.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1027)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:204)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:227)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:96)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1132)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1141)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
25/03/26 15:39:40 INFO ShutdownHookManager: Shutdown hook called
25/03/26 15:39:40 INFO ShutdownHookManager: Deleting directory
/private/var/folders/j2/cfn7w6795538n_416_27rkqm0000gn/T/spark-fe4c9d71-b7d7-437e-b486-514cc538cccc
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GitHub Acitons
- Manual Check:
1. Package a Spark Client using the command `dev/make-distribution.sh --tgz
-Phive`.
2. Execute `bin/spark-shell --remote local`. Although the logs directory
does not exist, the aforementioned error is no longer reported.
3. After exiting the Connect REPL, execute `bin/spark-shell --remote local`
again. At this point, the logs directory already exists, and the shell will
start successfully. (Due to the unresolved issue SPARK-51606, it is necessary
to manually kill the Connect Server process after exiting the Connect REPL.)
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #50421 from LuciferYang/SPARK-51605.
Authored-by: yangjie01 <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
---
.../src/main/scala/org/apache/spark/sql/connect/SparkSession.scala | 1 +
1 file changed, 1 insertion(+)
diff --git
a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/SparkSession.scala
b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/SparkSession.scala
index 4d93f797e1ec..52c8da9ecd1f 100644
---
a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/SparkSession.scala
+++
b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/SparkSession.scala
@@ -789,6 +789,7 @@ object SparkSession extends SparkSessionCompanion with
Logging {
Option(System.getenv("SPARK_LOG_DIR"))
.orElse(Option(System.getenv("SPARK_HOME")).map(p => Paths.get(p,
"logs").toString))
.foreach { p =>
+ Files.createDirectories(Paths.get(p))
val logFile = Paths
.get(
p,
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
