dongjoon-hyun opened a new pull request, #59680: URL: https://github.com/apache/spark/pull/59680
### What changes were proposed in this pull request? This PR aims to fix `FileSystemRecoveryModeFactory` to handle `none` as a codec of `spark.deploy.recoveryCompressionCodec` instead of throwing an exception. Like the other short codec names, `none` is matched case-insensitively. ### Why are the changes needed? Since Apache Spark 4.0.0 ([SPARK-46216](https://github.com/apache/spark/pull/44129)), the documentation of `spark.deploy.recoveryCompressionCodec` lists `none` as a valid value. https://github.com/apache/spark/blob/cd06735c008285bddb9bd976b3e931f37eb4990f/docs/spark-standalone.md?plain=1#L939-L942 > A compression codec for persistence engines. none (default), lz4, lzf, snappy, and zstd. Currently, only FILESYSTEM mode supports this configuration. However, `FileSystemRecoveryModeFactory` passes any configured value to `CompressionCodec.createCodec`, which doesn't know `none`. As a result, `Master.onStart` fails with the following error when `none` is given explicitly, and the Master never becomes `ALIVE`. ``` ERROR Inbox: Ignoring error org.apache.spark.SparkIllegalArgumentException: [CODEC_NOT_AVAILABLE.WITH_CONF_SUGGESTION] The codec none is not available. Consider to set the config "spark.io.compression.codec" to "snappy". SQLSTATE: 56038 at org.apache.spark.errors.SparkCoreErrors$.codecNotAvailableError(SparkCoreErrors.scala:476) at org.apache.spark.io.CompressionCodec$.$anonfun$createCodec$2(CompressionCodec.scala:96) at scala.Option.getOrElse(Option.scala:201) at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:96) at org.apache.spark.deploy.master.FileSystemRecoveryModeFactory.$anonfun$createPersistenceEngine$2(RecoveryModeFactory.scala:61) at scala.Option.map(Option.scala:242) at org.apache.spark.deploy.master.FileSystemRecoveryModeFactory.createPersistenceEngine(RecoveryModeFactory.scala:61) at org.apache.spark.deploy.master.Master.onStart(Master.scala:189) ... ``` This is consistent with `spark.eventLog.compression.codec`, which handles `none` in the same way since [SPARK-45261](https://github.com/apache/spark/pull/43038). ### Does this PR introduce _any_ user-facing change? Yes. `spark.deploy.recoveryCompressionCodec=none` works as documented now, which means no compression like the default (unset). Previously, the Master failed to initialize the `FILESYSTEM` recovery mode with `CODEC_NOT_AVAILABLE` and never became `ALIVE`. This bug exists since [v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19). ### How was this patch tested? Pass the CIs with the newly added test cases. The new test cases fail without the main code change. ``` $ build/sbt 'core/testOnly org.apache.spark.deploy.master.RecoverySuite org.apache.spark.deploy.master.PersistenceEngineSuite' ... [info] - SPARK-46216: Recovery without compression (62 milliseconds) [info] - SPARK-46216: Recovery with compression (51 milliseconds) [info] - SPARK-60430: Recovery with explicit none codec (none) (55 milliseconds) [info] - SPARK-60430: Recovery with explicit none codec (NONE) (79 milliseconds) ... [info] Tests: succeeded 26, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
