Unfortunately, this will not get in 0.21.0 as we're tagging that today. Please tag the ticket(s) as Target Version = 0.22.0.
Ian On Mon, Nov 3, 2014 at 10:22 AM, Ankur Chauhan <an...@malloc64.com> wrote: > Hi Tim/others, > > Is this to be included in the 0.21.0 release? If so, I don't know how to > tag it etc. I would really (shamelessly) love it to be included as it would > really simplify my intended usecase of using snackfs (cassandra backed > filesystem). > > -- Ankur > > > On 3 Nov 2014, at 09:28, Ankur Chauhan <an...@malloc64.com> wrote: > > > > Yea, I saw those today morning. I'll hold off a little mesos-336 changes > a lot of stuff. > > > > Sent from my iPhone > > > > On Nov 3, 2014, at 9:18 AM, Adam Bordelon <a...@mesosphere.io <mailto: > a...@mesosphere.io>> wrote: > > > >> + Bernd, who has done some fetcher work, including additional testing, > for MESOS-1316, MESOS-1945, and MESOS-336 > >> > >> On Mon, Nov 3, 2014 at 9:04 AM, Dominic Hamon <dha...@twopensource.com > <mailto:dha...@twopensource.com>> wrote: > >> Hi Ankur > >> > >> I think this is a great approach. It makes the code much simpler, > extensible, and more testable. Anyone that's heard me rant knows I am a big > fan of unit tests over integration tests, so this shouldn't surprise anyone > :) > >> > >> If you haven't already, please read the documentation on contributing > to Mesos and the style guide to ensure all the naming is as expected, then > you can push the patch to reviewboard to get it reviewed and committed. > >> > >> On Mon, Nov 3, 2014 at 12:49 AM, Ankur Chauhan <an...@malloc64.com > <mailto:an...@malloc64.com>> wrote: > >> Hi, > >> > >> I did some learning today! This is pretty much a very rough draft of > the tests/refactor of mesos-fetcher that I have come up with. Again, If > there are some obvious mistakes, please let me know. (this is my first pass > after all). > >> https://github.com/ankurcha/mesos/compare/prefer_2 < > https://github.com/ankurcha/mesos/compare/prefer_2> > >> > >> My main intention is to break the logic of the fetcher info some very > discrete components that i can write tests against. I am still re-learning > cpp/mesos code styles etc so I may be a little slow to catch up but I would > really appreciate any comments and/or suggestions. > >> > >> -- Ankur > >> @ankurcha > >> > >>> On 2 Nov 2014, at 18:17, Ankur Chauhan <an...@malloc64.com <mailto: > an...@malloc64.com>> wrote: > >>> > >>> Hi, > >>> > >>> I noticed that the current set of tests in > `src/tests/fetcher_tests.cpp` is pretty coarse grained and are more on the > lines of a functional test. I was going to add some tests but it seems like > if I am to do that I would need to add a test dependency on hadoop. > >>> > >>> As an alternative, I propose adding a good set of unit tests around > the methods used by `src/launcher/fetcher.cpp` and `src/hdfs/hdfs.cpp`. > This should be able to catch a good portion of cases at the same time > keeping the dependencies and runtime of tests low. What do you guys thing > about this? > >>> > >>> PS: I am pretty green in terms of gtest and the overall c++ testing > methodology. Can someone give me pointers to good examples of tests in the > codebase. > >>> > >>> -- Ankur > >>> > >>>> On 1 Nov 2014, at 22:54, Adam Bordelon <a...@mesosphere.io <mailto: > a...@mesosphere.io>> wrote: > >>>> > >>>> Thank you Ankur. At first glance, it looks great. We'll do a more > thorough review of it very soon. > >>>> I know Tim St. Clair had some ideas for fixing MESOS-1711 < > https://issues.apache.org/jira/browse/MESOS-1711>; he may want to review > too. > >>>> > >>>> On Sat, Nov 1, 2014 at 8:49 PM, Ankur Chauhan <an...@malloc64.com > <mailto:an...@malloc64.com>> wrote: > >>>> Hi Tim, > >>>> > >>>> I just created a review https://reviews.apache.org/r/27483/ < > https://reviews.apache.org/r/27483/> It's my first stab at it and I will > try to add more tests as I figure out how to do the hadoop mocking and > stuff. Have a look and let me know what you think about it so far. > >>>> > >>>> -- Ankur > >>>> > >>>>> On 1 Nov 2014, at 20:05, Ankur Chauhan <an...@malloc64.com <mailto: > an...@malloc64.com>> wrote: > >>>>> > >>>>> Yea, i saw that the minute i pressed send. I'll start the review > board so that people can have a look at the change. > >>>>> > >>>>> -- Ankur > >>>>> > >>>>>> On 1 Nov 2014, at 20:01, Tim Chen <t...@mesosphere.io <mailto: > t...@mesosphere.io>> wrote: > >>>>>> > >>>>>> Hi Ankur, > >>>>>> > >>>>>> There is a fetcher_tests.cpp in src/tests. > >>>>>> > >>>>>> Tim > >>>>>> > >>>>>> On Sat, Nov 1, 2014 at 7:27 PM, Ankur Chauhan <an...@malloc64.com > <mailto:an...@malloc64.com>> wrote: > >>>>>> Hi Tim, > >>>>>> > >>>>>> I am trying to find/write some test cases. I couldn't find a > fetcher_tests.{cpp|hpp} so once I have something, I'll post on review > board. I am new to gmock/gtest so bear with me while i get up to speed. > >>>>>> > >>>>>> -- Ankur > >>>>>> > >>>>>>> On 1 Nov 2014, at 19:23, Timothy Chen <t...@mesosphere.io <mailto: > t...@mesosphere.io>> wrote: > >>>>>>> > >>>>>>> Hi Ankur, > >>>>>>> > >>>>>>> Can you post on reviewboard? We can discuss more about the code > there. > >>>>>>> > >>>>>>> Tim > >>>>>>> > >>>>>>> Sent from my iPhone > >>>>>>> > >>>>>>> On Nov 1, 2014, at 6:29 PM, Ankur Chauhan <an...@malloc64.com > <mailto:an...@malloc64.com>> wrote: > >>>>>>> > >>>>>>>> Hi Tim, > >>>>>>>> > >>>>>>>> I don't think there is an issue which is directly in line with > what i wanted but the closest one that I could find in JIRA is > https://issues.apache.org/jira/browse/MESOS-1711 < > https://issues.apache.org/jira/browse/MESOS-1711> > >>>>>>>> > >>>>>>>> I have a branch ( > https://github.com/ankurcha/mesos/compare/prefer_hadoop_fetcher < > https://github.com/ankurcha/mesos/compare/prefer_hadoop_fetcher> ) that > has a change that would enable users to specify whatever hdfs compatible > uris to the mesos-fetcher but maybe you can weight in on it. Do you think > this is the right track? if so, i would like to pick this issue and submit > a patch for review. > >>>>>>>> > >>>>>>>> -- Ankur > >>>>>>>> > >>>>>>>> > >>>>>>>>> On 1 Nov 2014, at 04:32, Tom Arnfeld <t...@duedil.com <mailto: > t...@duedil.com>> wrote: > >>>>>>>>> > >>>>>>>>> Completely +1 to this. There are now quite a lot of hadoop > compatible filesystem wrappers out in the wild and this would certainly be > very useful. > >>>>>>>>> > >>>>>>>>> I'm happy to contribute a patch. Here's a few related issues > that might be of interest; > >>>>>>>>> > >>>>>>>>> - https://issues.apache.org/jira/browse/MESOS-1887 < > https://issues.apache.org/jira/browse/MESOS-1887> > >>>>>>>>> - https://issues.apache.org/jira/browse/MESOS-1316 < > https://issues.apache.org/jira/browse/MESOS-1316> > >>>>>>>>> - https://issues.apache.org/jira/browse/MESOS-336 < > https://issues.apache.org/jira/browse/MESOS-336> > >>>>>>>>> - https://issues.apache.org/jira/browse/MESOS-1248 < > https://issues.apache.org/jira/browse/MESOS-1248> > >>>>>>>>> > >>>>>>>>> On 31 October 2014 22:39, Tim Chen <t...@mesosphere.io <mailto: > t...@mesosphere.io>> wrote: > >>>>>>>>> I believe there is already a JIRA ticket for this, if you search > for fetcher in Mesos JIRA I think you can find it. > >>>>>>>>> > >>>>>>>>> Tim > >>>>>>>>> > >>>>>>>>> On Fri, Oct 31, 2014 at 3:27 PM, Ankur Chauhan < > an...@malloc64.com <mailto:an...@malloc64.com>> wrote: > >>>>>>>>> Hi, > >>>>>>>>> > >>>>>>>>> I have been looking at some of the stuff around the fetcher and > saw something interesting. The code for fetcher::fetch method is dependent > on a hard coded list of url schemes. No doubt that this works but is very > restrictive. > >>>>>>>>> Hadoop/HDFS in general is pretty flexible when it comes to being > able to fetch stuff from urls and has the ability to fetch a large number > of types of urls and can be extended by adding configuration into the > conf/hdfs-site.xml and core-site.xml > >>>>>>>>> > >>>>>>>>> What I am proposing is that we refactor the fetcher.cpp to > prefer to use the hdfs (using hdfs/hdfs.hpp) to do all the fetching if > HADOOP_HOME is set and $HADOOP_HOME/bin/hadoop is available. This logic > already exists and we can just use it. The fallback logic for using > net::download or local file copy is may be left in place for installations > that do not have hadoop configured. This means that if hadoop is present we > can directly fetch urls such as tachyon://... snackfs:// ... cfs:// .... > ftp://... s3://... http:// ... file:// with no extra effort. This makes > up for a much better experience when it comes to debugging and > extensibility. > >>>>>>>>> > >>>>>>>>> What do others think about this? > >>>>>>>>> > >>>>>>>>> - Ankur > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>> > >>>> > >>> > >> > >> > >> > >> > >> -- > >> Dominic Hamon | @mrdo | Twitter > >> There are no bad ideas; only good ideas that go horribly wrong. > >> > >
