[ https://issues.apache.org/jira/browse/SPARK-17417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dhruve Ashar updated SPARK-17417: --------------------------------- Summary: Fix sorting of part files while reconstructing RDD/partition from checkpointed files. (was: Fix # of partitions for RDD while checkpointing - Currently limited by 10000(%05d)) > Fix sorting of part files while reconstructing RDD/partition from > checkpointed files. > ------------------------------------------------------------------------------------- > > Key: SPARK-17417 > URL: https://issues.apache.org/jira/browse/SPARK-17417 > Project: Spark > Issue Type: Bug > Components: Spark Core > Reporter: Dhruve Ashar > > Spark currently assumes # of partitions to be less than 100000 and uses %05d > padding. > If we exceed this no., the sort logic in ReliableCheckpointRDD gets messed up > and fails. This is because of part-files are sorted and compared as strings. > This leads filename order to be part-10000, part-100000, ... instead of > part-10000, part-10001, ..., part-100000 and while reconstructing the > checkpointed RDD the job fails. > Possible solutions: > - Bump the padding to allow more partitions or > - Sort the part files extracting a sub-portion as string and then verify the > RDD -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org