sharma-0311 opened a new pull request, #56032: URL: https://github.com/apache/spark/pull/56032
### What changes were proposed in this pull request? Auto-create the event log directory (`spark.eventLog.dir`) and the history server log directory (`spark.history.fs.logDirectory`) if they do not exist at startup, instead of failing with a `FileNotFoundException`. **FsHistoryProvider** (`SPARK-56966`): In `startPolling()`, the `FileNotFoundException` catch block now calls `FileSystem.mkdirs` with `LOG_FOLDER_PERMISSIONS` before giving up. If creation succeeds the directory is treated as valid; if it fails, the original warning-and-skip behavior is preserved. **EventLogFileWriters** (`SPARK-56967`): `requireLogBaseDirAsDirectory()` now checks `fileSystem.exists` first and calls `FileSystem.mkdirs` if the path is absent. This works for local FS, HDFS, and S3-compatible paths. ### Why are the changes needed? Users frequently see startup failures because the log directory has not been pre-created, even when the path and permissions are otherwise correct. Auto-creating it reduces operational friction with no behavioral regression for directories that already exist. ### Does this PR introduce _any_ user-facing change? Yes — previously both code paths threw/warned on a missing directory. Now they silently create it. The behavior when a directory exists is unchanged. ### How was this patch tested? - [ ] Unit tests for `FsHistoryProviderSuite` covering the new mkdir path - [ ] Unit tests for `EventLogFileWritersSuite` covering auto-creation - [ ] Manual test with a non-existent local path and an S3 path Closes #<PR number> Resolves https://issues.apache.org/jira/browse/SPARK-56966 Resolves https://issues.apache.org/jira/browse/SPARK-56967 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
