anshulbaliga7 commented on PR #58167: URL: https://github.com/apache/spark/pull/58167#issuecomment-5782764732
Thanks @cloud-fan for the review, have addressed both as: - The payload now carries a CRC32 over the version and count bytes (13 bytes total), and the reader rejects a version mismatch, checksum mismatch, negative count, or trailing bytes before comparing. Any rejection takes the tolerated "detection inactive" path. The corruption test now includes your exact case: the count bytes rewritten to 5 with the recorded checksum left intact, which is tolerated, plus a control that restores the original payload and confirms a real missing trailing file still throws `CHECKPOINT_TRUNCATED_DIRECTORY`. - Added two failure-injection tests using `LocalFileSystem` subclasses, following the existing `MkdirsFailingFilesystem` pattern: one throws from `create` on the `_num_partitions` temp file, one returns `false` from `rename` onto the final path. Both assert the action completes, the RDD is checkpointed, the inactive-check WARN is logged, neither the final nor the temp file is left behind, and the directory reads back as a pre-SPARK-58883 checkpoint. PR description updated to name `CHECKPOINT_TRUNCATED_DIRECTORY`, the on-disk format, and the `spark.checkpoint.verifyPartitionCount.enabled` switch. Can you PTAL again? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
