sharma-0311 opened a new pull request, #56029: URL: https://github.com/apache/spark/pull/56029
### What changes were proposed in this pull request? Auto-create `spark.eventLog.dir` and `spark.history.fs.logDirectory` at startup if they do not exist, instead of failing with a FileNotFoundException. **FsHistoryProvider** (SPARK-56966): In `startPolling()`, the FileNotFoundException catch block now calls `FileSystem.mkdirs` before giving up. If creation succeeds the directory is treated as valid; if it fails, the original warning-and-skip behavior is preserved. **EventLogFileWriters** (SPARK-56967): `requireLogBaseDirAsDirectory()` now checks `fileSystem.exists` first and calls `FileSystem.mkdirs` if the path is absent. Works for local FS, HDFS, and S3-compatible paths. ### Why are the changes needed? Users frequently hit startup failures because the log directory has not been pre-created, even when the path and permissions are correct. Auto-creating it reduces operational friction with no regression for directories that already exist. ### Does this PR introduce any user-facing change? Yes — previously both code paths failed on a missing directory. Now they create it automatically. Behavior is unchanged when the directory exists. ### How was this patch tested? Updated FsHistoryProviderSuite and EventLogFileWritersSuite with tests covering the auto-creation path. Resolves https://issues.apache.org/jira/browse/SPARK-56966 Resolves https://issues.apache.org/jira/browse/SPARK-56967 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
