[
https://issues.apache.org/jira/browse/SPARK-6313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14362687#comment-14362687
]
Patrick Wendell commented on SPARK-6313:
----------------------------------------
[~joshrosen] changing default caching behavior seems like it could silently
regress performance for the vas majority of users who aren't on NFS. What about
a hotfix for 1.3.1 that just exposes the config for NFS users (this is very
small population), but doesn't change the default. That may be sufficient in
itself... or if we want a real fix that makes it work out-of-the-box on NDFS,
we can put it in 1.4.
> Fetch File Lock file creation doesnt work when Spark working dir is on a NFS
> mount
> ----------------------------------------------------------------------------------
>
> Key: SPARK-6313
> URL: https://issues.apache.org/jira/browse/SPARK-6313
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.2.0, 1.3.0, 1.2.1
> Reporter: Nathan McCarthy
> Priority: Critical
>
> When running in cluster mode and mounting the spark work dir on a NFS volume
> (or some volume which doesn't support file locking), the fetchFile (used for
> downloading JARs etc on the executors) method in Spark Utils class will fail.
> This file locking was introduced as an improvement with SPARK-2713.
> See
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/Utils.scala#L415
>
> Introduced in 1.2 in commit;
> https://github.com/apache/spark/commit/7aacb7bfad4ec73fd8f18555c72ef696
> As this locking is for optimisation for fetching files, could we take a
> different approach here to create a temp/advisory lock file?
> Typically you would just mount local disks (in say ext4 format) and provide
> this as a comma separated list however we are trying to run Spark on MapR.
> With MapR we can do a loop back mount to a volume on the local node and take
> advantage of MapRs disk pools. This also means we dont need specific mounts
> for Spark and improves the generic nature of the cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
