[ https://issues.apache.org/jira/browse/SPARK-19045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15791135#comment-15791135 ]
Sean Owen commented on SPARK-19045: ----------------------------------- I disagree; the check looks correct: {code} if (!isLocal && Utils.nonLocalPaths(directory).isEmpty) { logWarning("Spark is not running in local mode, therefore the checkpoint directory " + s"must not be on the local filesystem. Directory '$directory' " + "appears to be on the local filesystem.") } {code} Standalone mode != local. Spark also can't know whether it's an NFS mount. The warning is just a warning anyway. > irrelevant warning when creating a checkpoint dir > ------------------------------------------------- > > Key: SPARK-19045 > URL: https://issues.apache.org/jira/browse/SPARK-19045 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 2.1.0 > Reporter: Assaf Mendelson > Priority: Trivial > > When I do: > sc.setCheckpointDir("/mydir") I get a warning: > 17/01/01 06:26:42 WARN SparkContext: Spark is not running in local mode, > therefore the checkpoint directory must not be on the local filesystem. > Directory '/mydir' appears to be on the local filesystem. > This occurs even though I use a single computer with local filesystem (using > spark standalone). Reading the code it seems the same error would occur if I > would use a cluster and an NFS share. > Checkpoint should work on a local directory as long as there is a single node > (even when using a resource manager rather than a single node) and should > work in a cluster with locally mounted NFS share. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org