[ https://issues.apache.org/jira/browse/MESOS-9348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659865#comment-16659865 ]
James Peach commented on MESOS-9348: ------------------------------------ One approach here is to URL-encode the output filename for the HDFS command. Experimentally, it looks like this is required, since the command errors out on unsafe characters: {noformat} # hdfs dfs -copyToLocal hdfs:///artifacts/8c/99/4b/8c994b489674589a58805e2e695e98674b9dd793411579f0fbaea3459f94f86e/connector/%5BRELEASE%5D/connector-%5BRELEASE%5D.jar $(pwd)/%255B-jpeach-].jar copyToLocal: unexpected URISyntaxException {noformat} > URL-encoded HDFS artifacts can't be fetched through the cache. > -------------------------------------------------------------- > > Key: MESOS-9348 > URL: https://issues.apache.org/jira/browse/MESOS-9348 > Project: Mesos > Issue Type: Bug > Components: fetcher > Reporter: James Peach > Priority: Major > > The {{hdfs dfs}} command always does a URI decode on the target output file. > This means that the output file gets stored in the fetcher cache under the > wrong filename and we can never retrieve it. > Here's an example of how the command behaves: > {noformat} > [/tmp]# hdfs dfs -copyToLocal > hdfs:///artifacts/8c/99/4b/8c994b489674589a58805e2e695e98674b9dd793411579f0fbaea3459f94f86e/connector/%5BRELEASE%5D/connector-%5BRELEASE%5D.jar > $(pwd)/%5B-jpeach-%5D.jar > [/tmp]# ls -l *jpeach* > -rw-r--r-- 1 root root 7285799 Oct 22 23:29 [-jpeach-].jar > {noformat} > Here's how this plays out in the fetcher: > {noformat} > W1022 23:22:13.649587 3186459 fetcher.cpp:395] Copying instead of extracting > resource from URI with 'extract' flag, because it does not seem to be an > archive: > hdfs:///artifacts/8c/99/4b/8c994b489674589a58805e2e695e98674b9dd793411579f0fbaea3459f94f86e/connector/%5BRELEASE%5D/connector-%5BRELEASE%5D.jar > cp: cannot stat `/srv/mesos/fetch/jarvis/c67-connector-_ASE%5D.jar': No such > file or directory > E1022 23:22:13.652987 3186459 fetcher.cpp:613] EXIT with status 1: Failed to > fetch > 'hdfs:///artifacts/8c/99/4b/8c994b489674589a58805e2e695e98674b9dd793411579f0fbaea3459f94f86e/connector/%5BRELEASE%5D/connector-%5BRELEASE%5D.jar': > cp failed with status: 256 > ... > # ls -latr /srv/mesos/fetch > ... > -rw-r--r-- 1 jarvis jarvis 7285799 Oct 22 23:22 c67-connector-_ASE].jar > {noformat} > The fetcher has downloaded the artifact into the cache, but can't copy it > into the sandbox because it was downloaded to the wrong filename. -- This message was sent by Atlassian JIRA (v7.6.3#76005)