> On Nov. 4, 2014, 12:12 a.m., Vinod Kone wrote: > > src/launcher/fetcher.cpp, line 230 > > <https://reviews.apache.org/r/27483/diff/3/?file=747858#file747858line230> > > > > Why the fall through to HDFS here?
My reasoning is that if hadoop client is configured with some ftp/ftps/http/https credentials that are not in the urls, the hadoop client would be able to fetch those resource that the libcurl client can't. I am open to changing this if this line of thinking seems confusing. > On Nov. 4, 2014, 12:12 a.m., Vinod Kone wrote: > > src/launcher/fetcher.cpp, lines 245-246 > > <https://reviews.apache.org/r/27483/diff/3/?file=747858#file747858line245> > > > > Seems weird to do this after hdfs fetch. Is HDFS local copy as fast as > > "cp" ? > > > > I would change the semantics as follows: > > > > ``` > > if (local) { > > return fetchWithLocalCopy(); > > } > > > > if (http* or ftp*) { > > return fetchWithNet(); > > } > > > > return fetchWithHadoopClient(); > > > > ``` I am pretty sure that native cp is faster than hadoop client local copy. I can make this change. - Ankur ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27483/#review59682 ----------------------------------------------------------- On Nov. 4, 2014, 12:02 a.m., Ankur Chauhan wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/27483/ > ----------------------------------------------------------- > > (Updated Nov. 4, 2014, 12:02 a.m.) > > > Review request for mesos and Timothy St. Clair. > > > Bugs: MESOS-1711 > https://issues.apache.org/jira/browse/MESOS-1711 > > > Repository: mesos-git > > > Description > ------- > > Previously, the fetcher used a hardcoded list of schemes to determine what > URIs could be fetched by hadoop (if available). This is now changed such that > we first check if hadoop can fetch them for us and then we fallback to the > os::net and then a local copy method (same as it used to be). This allows > users to fetch artifacts from arbitrary filesystems as long as hadoop is > correctly configured (in core-site.xml). > > > Diffs > ----- > > src/hdfs/hdfs.hpp bbfeddef106c598d8379ced085ef0605c4b2f380 > src/launcher/fetcher.cpp 9323c28237010fa065ef34d74435c151ded530a8 > > Diff: https://reviews.apache.org/r/27483/diff/ > > > Testing > ------- > > make check > sudo bin/mesos-tests.sh --verbose > support/mesos-style.py > > > Thanks, > > Ankur Chauhan > >
