anshulbaliga7 commented on code in PR #58167:
URL: https://github.com/apache/spark/pull/58167#discussion_r3862075367
##########
core/src/main/scala/org/apache/spark/rdd/ReliableCheckpointRDD.scala:
##########
@@ -83,6 +85,16 @@ private[spark] class ReliableCheckpointRDD[T: ClassTag](
throw SparkCoreErrors.invalidCheckpointDirectoryError(path,
expectedFileName)
}
}
+ // If a partition-count metadata file is present, verify no trailing files
are missing.
+ // Directories written by earlier Spark versions have no such file; a
missing file is
+ // silently tolerated for backward compatibility. See SPARK-58883.
+ ReliableCheckpointRDD.readPartitionCountFromCheckpointDir(context,
checkpointPath)
+ .foreach { expected =>
+ if (inputFiles.length != expected) {
+ throw
SparkCoreErrors.checkpointRDDHasDifferentNumberOfPartitionsFromOriginalRDDError(
+ id, expected, id, inputFiles.length)
Review Comment:
Same RDD id in both template slots, no directory in the message. Split into
a new condition, `CHECKPOINT_TRUNCATED_DIRECTORY`, scoped to this read-only
check that carries path, expected, found.
`CHECKPOINT_RDD_PARTITION_COUNT_MISMATCH` is untouched and still only reachable
from the write-path comparison.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
