[
https://issues.apache.org/jira/browse/MESOS-336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13972248#comment-13972248
]
Benjamin Hindman commented on MESOS-336:
----------------------------------------
I think the miscommunication here might be because (Vinod and) I assume that
the URI is identifying a specific _resource_ that we're trying to download.
Think of it like doing a wget on a URI, if the URI is
'http://ip1:port1/path/to/resouce.txt' then we expect wget to put a file in our
directory called 'resource.txt'. So if later we did a wget on
'ftp://ip2:port2:/different/path/to/resource.txt' then wget would try to
overwrite the local 'resource.txt' file in the directory. Thus, the resource is
"named" 'resource.txt', so we can easily check "if the requested file exists"
and then compare checksums. And yes, the checksum comes from the framework
putting it there, as you suggested.
Clearly there are some URIs that won't have a clear "name" that identifies a
resource, for example someone might have the HTTP URL
'http://ip:port/path/to/download?file=resource.txt'. In this case wget would
put the result in a file called 'download?file=resource.txt', which isn't a
great "name". My expectation right now was to ignore this case. That is, we'll
always "name" a resource as wget would. In the end we'll be checking the
checksum, and that's most important, even if it does mean we'll be re-download
things more often then we like.
For the specific HTTP case we can add more smarts later, like using ETag as
[~tknaup] suggested, as well as looking at the 'Content-Disposition' of our
response (to get better names).
Note that the checksum still does not become irrelevant if we used the entire
URI to name the cache file because the framework might explicitly be trying to
say: I've updated the object at this URI so I want you to re-download the new
version. This is the expected behavior if you think about something like an
HTTP cache. HTTP can exploit the 'If-Modified-Since' headers and we could too,
but since some protocols don't have anything analogous we proposed using
checksums (i.e., does FTP have any notion of If-Modified-Since? Or HDFS?).
> Mesos slave should cache executors
> ----------------------------------
>
> Key: MESOS-336
> URL: https://issues.apache.org/jira/browse/MESOS-336
> Project: Mesos
> Issue Type: Improvement
> Components: slave
> Reporter: brian wickman
> Assignee: Bernd Mathiske
> Labels: newbie
>
> The slave should be smarter about how it handles pulling down executors. In
> our environment, executors rarely change but the slave will always pull it
> down from regardless HDFS. This puts undue stress on our HDFS clusters, and
> is not resilient to reduced HDFS availability.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
