[
https://issues.apache.org/jira/browse/SPARK-6313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14359964#comment-14359964
]
Nathan McCarthy commented on SPARK-6313:
----------------------------------------
Suggestion along the lines of;
https://github.com/apache/lucene-solr/blob/5314a56924f46522993baf106e6deca0e48a967f/lucene/core/src/java/org/apache/lucene/store/SimpleFSLockFactory.java
or
https://github.com/graphhopper/graphhopper/blob/master/core/src/main/java/com/graphhopper/storage/SimpleFSLockFactory.java
> Fetch File Lock file creation doesnt work when Spark working dir is on a NFS
> mount
> ----------------------------------------------------------------------------------
>
> Key: SPARK-6313
> URL: https://issues.apache.org/jira/browse/SPARK-6313
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.3.0
> Reporter: Nathan McCarthy
>
> When running in cluster mode and mounting the spark work dir on a NFS volume
> (or some volume which doesn't support file locking), the fetchFile (used for
> downloading JARs etc on the executors) method in Spark Utils class will fail.
> This file locking was introduced as an improvement with SPARK-2713.
> See
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/Utils.scala#L415
>
> Introduced in 1.2 in commit;
> https://github.com/apache/spark/commit/7aacb7bfad4ec73fd8f18555c72ef696
> As this locking is for optimisation for fetching files, could we take a
> different approach here to create a temp/advisory lock file?
> Typically you would just mount local disks (in say ext4 format) and provide
> this as a comma separated list however we are trying to run Spark on MapR.
> With MapR we can do a loop back mount to a volume on the local node and take
> advantage of MapRs disk pools. This also means we dont need specific mounts
> for Spark and improves the generic nature of the cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
