[
https://issues.apache.org/jira/browse/SPARK-6619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-6619.
---------------------------------
Resolution: Incomplete
> Improve Jar caching on executors
> --------------------------------
>
> Key: SPARK-6619
> URL: https://issues.apache.org/jira/browse/SPARK-6619
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: Mingyu Kim
> Priority: Major
> Labels: bulk-closed
>
> Taking SPARK-2713 one step further so that
> - The cached jars can be used by multiple applications. In order to do that,
> I'm planning to use MD5 as the cache key as opposed to url hash and timestamp.
> - The cached jars are hard-linked to the work directory as opposed to being
> copied.
> Re: perf. Computing MD5 using "openssl" on my local Macbook Pro took 1.2s for
> 158 jars with the total size of 56MB, and this takes ~10s to ship to the
> executor at the start-up.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
