[
https://issues.apache.org/jira/browse/MESOS-9348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659865#comment-16659865
]
James Peach commented on MESOS-9348:
------------------------------------
One approach here is to URL-encode the output filename for the HDFS command.
Experimentally, it looks like this is required, since the command errors out on
unsafe characters:
{noformat}
# hdfs dfs -copyToLocal
hdfs:///artifacts/8c/99/4b/8c994b489674589a58805e2e695e98674b9dd793411579f0fbaea3459f94f86e/connector/%5BRELEASE%5D/connector-%5BRELEASE%5D.jar
$(pwd)/%255B-jpeach-].jar
copyToLocal: unexpected URISyntaxException
{noformat}
> URL-encoded HDFS artifacts can't be fetched through the cache.
> --------------------------------------------------------------
>
> Key: MESOS-9348
> URL: https://issues.apache.org/jira/browse/MESOS-9348
> Project: Mesos
> Issue Type: Bug
> Components: fetcher
> Reporter: James Peach
> Priority: Major
>
> The {{hdfs dfs}} command always does a URI decode on the target output file.
> This means that the output file gets stored in the fetcher cache under the
> wrong filename and we can never retrieve it.
> Here's an example of how the command behaves:
> {noformat}
> [/tmp]# hdfs dfs -copyToLocal
> hdfs:///artifacts/8c/99/4b/8c994b489674589a58805e2e695e98674b9dd793411579f0fbaea3459f94f86e/connector/%5BRELEASE%5D/connector-%5BRELEASE%5D.jar
> $(pwd)/%5B-jpeach-%5D.jar
> [/tmp]# ls -l *jpeach*
> -rw-r--r-- 1 root root 7285799 Oct 22 23:29 [-jpeach-].jar
> {noformat}
> Here's how this plays out in the fetcher:
> {noformat}
> W1022 23:22:13.649587 3186459 fetcher.cpp:395] Copying instead of extracting
> resource from URI with 'extract' flag, because it does not seem to be an
> archive:
> hdfs:///artifacts/8c/99/4b/8c994b489674589a58805e2e695e98674b9dd793411579f0fbaea3459f94f86e/connector/%5BRELEASE%5D/connector-%5BRELEASE%5D.jar
> cp: cannot stat `/srv/mesos/fetch/jarvis/c67-connector-_ASE%5D.jar': No such
> file or directory
> E1022 23:22:13.652987 3186459 fetcher.cpp:613] EXIT with status 1: Failed to
> fetch
> 'hdfs:///artifacts/8c/99/4b/8c994b489674589a58805e2e695e98674b9dd793411579f0fbaea3459f94f86e/connector/%5BRELEASE%5D/connector-%5BRELEASE%5D.jar':
> cp failed with status: 256
> ...
> # ls -latr /srv/mesos/fetch
> ...
> -rw-r--r-- 1 jarvis jarvis 7285799 Oct 22 23:22 c67-connector-_ASE].jar
> {noformat}
> The fetcher has downloaded the artifact into the cache, but can't copy it
> into the sandbox because it was downloaded to the wrong filename.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
