[ https://issues.apache.org/jira/browse/SPARK-11459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14990676#comment-14990676 ]
Ryan Williams commented on SPARK-11459: --------------------------------------- I'm mostly interested in saving RDDs to disk with kryo-serde ([SPARK-11461|https://issues.apache.org/jira/browse/SPARK-11461]). The existing checkpoint APIs are functionally exactly what I want, but they mandate putting a UUID in the directory name and fixing the basename to the RDD ID, somewhat unnecessarily. Letting the user opt in to specifying the path is a simple way to get at the functionality that I want without having to do something possibly more invasive e.g. for SPARK-11461, and there's not really a danger of it conflicting with existing checkpoint usages. It could even be exposed via a different method on SparkContext/RDD, if overloading the semantics of {{checkpoint}} is the concern. Another, orthogonal option I've worked on a little is basically copy/pasting a bunch of the checkpointing logic into a Spark package that hangs methods off of SparkContext and RDD that do [checkpointing with configurable path naming], but that's an unideal use-case for Spark packages: it is a bunch of code that's already in Spark, that I'd have to keep up to date, etc. > Allow configuring checkpoint dir, filenames > ------------------------------------------- > > Key: SPARK-11459 > URL: https://issues.apache.org/jira/browse/SPARK-11459 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 1.5.1 > Reporter: Ryan Williams > Priority: Minor > > I frequently want to persist some RDDs to disk and choose the names of the > files that they are saved as. > Currently, the {{RDD.checkpoint}} flow [writes to a directory with a UUID in > its > name|https://github.com/apache/spark/blob/v1.5.1/core/src/main/scala/org/apache/spark/SparkContext.scala#L2050], > and the file is [always named after the RDD's > ID|https://github.com/apache/spark/blob/v1.5.1/core/src/main/scala/org/apache/spark/rdd/ReliableRDDCheckpointData.scala#L96]. > Is there any reason not to allow the user to e.g. pass a string to > {{RDD.checkpoint}} that will set the location that the RDD is checkpointed to? -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org