Why rely on url scheme for fetching?

Hi, I have been looking at some of the stuff around the fetcher and saw something interesting. The code for fetcher::fetch method is dependent on a hard coded list of url schemes. No doubt that this works but is very restrictive. Hadoop/HDFS in general is pretty flexible when it comes to being

Re: Why rely on url scheme for fetching?

I believe there is already a JIRA ticket for this, if you search for fetcher in Mesos JIRA I think you can find it. Tim On Fri, Oct 31, 2014 at 3:27 PM, Ankur Chauhan wrote: > Hi, > > I have been looking at some of the stuff around the fetcher and saw > something interesting. The code for fetch

Re: Why rely on url scheme for fetching?

Completely +1 to this. There are now quite a lot of hadoop compatible filesystem wrappers out in the wild and this would certainly be very useful. I'm happy to contribute a patch. Here's a few related issues that might be of interest; - https://issues.apache.org/jira/browse/MESOS-1887 - https://i

Re: Why rely on url scheme for fetching?

Hi Tim, I don't think there is an issue which is directly in line with what i wanted but the closest one that I could find in JIRA is https://issues.apache.org/jira/browse/MESOS-1711 I have a branch ( https://github.com/ankurcha/mesos/compare

Re: Why rely on url scheme for fetching?

Hi Ankur, Can you post on reviewboard? We can discuss more about the code there. Tim Sent from my iPhone > On Nov 1, 2014, at 6:29 PM, Ankur Chauhan wrote: > > Hi Tim, > > I don't think there is an issue which is directly in line with what i wanted > but the closest one that I could find in

Re: Why rely on url scheme for fetching?

Hi Tim, I am trying to find/write some test cases. I couldn't find a fetcher_tests.{cpp|hpp} so once I have something, I'll post on review board. I am new to gmock/gtest so bear with me while i get up to speed. -- Ankur > On 1 Nov 2014, at 19:23, Timothy Chen wrote: > > Hi Ankur, > > Can you

Re: Why rely on url scheme for fetching?

Hi Ankur, There is a fetcher_tests.cpp in src/tests. Tim On Sat, Nov 1, 2014 at 7:27 PM, Ankur Chauhan wrote: > Hi Tim, > > I am trying to find/write some test cases. I couldn't find a > fetcher_tests.{cpp|hpp} so once I have something, I'll post on review > board. I am new to gmock/gtest so b

Re: Why rely on url scheme for fetching?

Yea, i saw that the minute i pressed send. I'll start the review board so that people can have a look at the change. -- Ankur > On 1 Nov 2014, at 20:01, Tim Chen wrote: > > Hi Ankur, > > There is a fetcher_tests.cpp in src/tests. > > Tim > > On Sat, Nov 1, 2014 at 7:27 PM, Ankur Chauhan

Re: Why rely on url scheme for fetching?

Hi Tim, I just created a review https://reviews.apache.org/r/27483/ It's my first stab at it and I will try to add more tests as I figure out how to do the hadoop mocking and stuff. Have a look and let me know what you think about it so far. -- Ankur > On

Re: Why rely on url scheme for fetching?

Thank you Ankur. At first glance, it looks great. We'll do a more thorough review of it very soon. I know Tim St. Clair had some ideas for fixing MESOS-1711 ; he may want to review too. On Sat, Nov 1, 2014 at 8:49 PM, Ankur Chauhan wrote: > Hi Ti

Re: Why rely on url scheme for fetching?

Hi, I noticed that the current set of tests in `src/tests/fetcher_tests.cpp` is pretty coarse grained and are more on the lines of a functional test. I was going to add some tests but it seems like if I am to do that I would need to add a test dependency on hadoop. As an alternative, I propos

Re: Why rely on url scheme for fetching?

Hi, I did some learning today! This is pretty much a very rough draft of the tests/refactor of mesos-fetcher that I have come up with. Again, If there are some obvious mistakes, please let me know. (this is my first pass after all). https://github.com/ankurcha/mesos/compare/prefer_2

Re: Why rely on url scheme for fetching?

Hi Ankur I think this is a great approach. It makes the code much simpler, extensible, and more testable. Anyone that's heard me rant knows I am a big fan of unit tests over integration tests, so this shouldn't surprise anyone :) If you haven't already, please read the documentation on contributi

Re: Why rely on url scheme for fetching?

+ Bernd, who has done some fetcher work, including additional testing, for MESOS-1316, MESOS-1945, and MESOS-336 On Mon, Nov 3, 2014 at 9:04 AM, Dominic Hamon wrote: > Hi Ankur > > I think this is a great approach. It makes the code much simpler, > extensible, and more testable. Anyone that's he

Re: Why rely on url scheme for fetching?

Yea, I saw those today morning. I'll hold off a little mesos-336 changes a lot of stuff. Sent from my iPhone > On Nov 3, 2014, at 9:18 AM, Adam Bordelon wrote: > > + Bernd, who has done some fetcher work, including additional testing, for > MESOS-1316, MESOS-1945, and MESOS-336 > >> On Mon,

Re: Why rely on url scheme for fetching?

Hi Tim/others, Is this to be included in the 0.21.0 release? If so, I don't know how to tag it etc. I would really (shamelessly) love it to be included as it would really simplify my intended usecase of using snackfs (cassandra backed filesystem). -- Ankur > On 3 Nov 2014, at 09:28, Ankur Chau

Re: Why rely on url scheme for fetching?

Hi, Okay, thanks Ian. What's the expected ETA on getting 0.21.0 out? jira didn't have a release date set. -- Ankur > On 3 Nov 2014, at 10:37, Ian Downes wrote: > > Unfortunately, this will not get in 0.21.0 as we're tagging that today. > > Please tag the ticket(s) as Target Version = 0.22.0.

Re: Why rely on url scheme for fetching?

I think it's too late to be included, since it's going to take some rounds of review, and Ian is cutting the release today. We'll have to tag this for the next release. Tim On Mon, Nov 3, 2014 at 10:22 AM, Ankur Chauhan wrote: > Hi Tim/others, > > Is this to be included in the 0.21.0 release? I

Re: Why rely on url scheme for fetching?

That's cool. I think if it gets reviews and gets an okay I'll modify my deployment and build myself some deb packages with these changes till 0.22.0 ships. Sent from my iPhone > On Nov 3, 2014, at 12:49 PM, Timothy Chen wrote: > > I think it's too late to be included, since it's going to tak

Re: Why rely on url scheme for fetching?

Hi Ankur, Sounds good, you can also use https://github.com/mesosphere/mesos-deb-packaging for building out deb packages from source. Thanks, Tim On Mon, Nov 3, 2014 at 1:21 PM, Ankur Chauhan wrote: > That's cool. I think if it gets reviews and gets an okay I'll modify my > deployment and buil

Re: Why rely on url scheme for fetching?

Hi ben, Thanks for the follow up. I think thats a perfectly fine game plan. I think I can break down things into smaller, more isolated chunks. But from the looks of MESOS-1316 the invocation code and the testing code seems to change a lot (in a good way), so to avoid a bunch of wasted cycles,