Re: Mesos/Marathon/HAProxy Logging

This may help: http://serverfault.com/questions/331079/haproxy-and-forwarding-client-ip-address-to-servers We use similar options to ensure we have the remote ip. > On 25 Aug 2015, at 09:30, John Omernik wrote: > > I have been playing with an application that is a very simple app: A > webserv

Re: MesosCon Seattle attendee introduction thread

Hi all, I am Ankur Chauhan. I am a Sr. Software engineer with the Reporting and Analytics team at Brightcove Inc. I have been evaluating, tinkering, developing with mesos for about an year now. My latest adventure has been in the spark mesos integration and writing the new apache flink - mesos

Questions about framework development - (HA and reconciling state)

Hi all, I am working on creating an integration between Apache Flink (http://flink.apache.org) and mesos which would be similar to the way the current hadoop-mesos integration works using the java mesos client. My current idea is that the scheduler will also run a JobManager process (similar t

Re: Cluster autoscaling in Spark+Mesos ?

-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, @Sharma - Is mantis/fenzo available on github or something, I did find some maven artifacts but the repository netflix/fenzo is a 404. I am interested in learning about the bin packing logic of fenzo. - -- Ankur Chauhan On 04/06/2015 22:35

Re: Cluster autoscaling in Spark+Mesos ?

chanism to perhaps restart the > Spark consumers into refreshed Spark context's which are aware of > the new cluster topology? > > On Thu, Jun 4, 2015 at 4:23 PM, Ankur Chauhan <mailto:an...@malloc64.com>> wrote: > > AFAIK Mesos does not support host level auto-

Re: Cluster autoscaling in Spark+Mesos ?

-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 AFAIK Mesos does not support host level auto-scaling because that is not the scope of the mesos-master or mesos-slave. In EC2 (like in my case) we have autoscaling groups set with cloudwatch metrics hooked up to scaling policies. In our case, we have t

Re: [DISCUSS] Renaming Mesos Slave

-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 +1 master/slave James made some very good points and there is no technical reason for wasting time on this. On 04/06/2015 08:45, James Vanns wrote: > +1 master/slave, no change needed. > > I couldn't agree more. This is a barmy request; master/slave

Re: [DISCUSS] Renaming Mesos Slave

Adding my two cents: If I have seen the following terminology being in a some places: Mesos-master => director/orchestrator : the reasoning here is simple, there is always only one orchestrator and has the job of directing others to do as per the current requirements. The group of orchestrators i

Re: Mesos 0.22.0

-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 +1 On 20/01/2015 20:00, Manivannan wrote: > +1 > > On Wed, Jan 21, 2015 at 7:04 AM, Vinod Kone > wrote: > > +1 > > @vinodkone > > On Jan 20, 2015, at 12:03 PM, Chris Aniszczyk > wrote: > >> d

Re: [VOTE] Release Apache Mesos 0.21.1 (rc2)

a minor improvement I'm inclined to include it for >>> the next major release which shouldn't be too far away from this release. >>> >>> If anyone else thinks otherwise please let me know. >>> >>> Tim >>> >>>> On Fri, Dec

Re: [VOTE] Release Apache Mesos 0.21.1 (rc2)

Sorry for a late join in can we get https://issues.apache.org/jira/plugins/servlet/mobile#issue/MESOS-1711 in too or is it too late? -- ankur Sent from my iPhone > On Dec 19, 2014, at 12:23, Tim Chen wrote: > > Hi all, > > Please vote on releasing the following candidate as Apache Mesos 0.21

Re: Help needed

The other option is to use hdfs or s3 or ceph or gluster, basically what I mean to say is to introduce a layer of indirection and have a dfs service with a userland "file system driver". I have used s3fs in my "experiments" with success although i would like to hear from others who have some re

Re: Help needed

You could use HDFS of Mount NFs on all nodes and then mount that inside all the containers as a volume. Sent from my iPhone > On Nov 22, 2014, at 9:40 AM, Qiang wrote: > > I having been working with docker and mesos recently and one of the app I am > going to dockerize relies on file storage

Re: Relative dates are parsed incorrectly.

Please have a look at http://mesos.apache.org/documentation/latest/mesos-developers-guide/ That should help you get setup/started. Github pull requests are not the suggested medium for contributing changes, you will need to submit patches to review board. Ankur Sent from my iPhone > On Nov 1

Re: [ANN] Mesos resources searchable

WOW! That is amazing. All these things in a single place is awesome. -- Ankur > On 10 Nov 2014, at 15:41, Otis Gospodnetic wrote: > > Hi everyone, > > Quick announcement that we've added Apache Mesos and made all its resources > searchable: > > http://search-hadoop.com/mesos

Re: Why rely on url scheme for fetching?

>>> extensible, and more testable. Anyone that's heard me rant knows I am a big >>> fan of unit tests over integration tests, so this shouldn't surprise anyone >>> :) >>> >>> If you haven't already, please read the documentation on contri

Re: Why rely on url scheme for fetching?

ded, since it's going to take some > rounds of review, and Ian is cutting the release today. > > We'll have to tag this for the next release. > > Tim > >> On Mon, Nov 3, 2014 at 10:22 AM, Ankur Chauhan wrote: >> Hi Tim/others, >> >> Is this to

Re: Why rely on url scheme for fetching?

(s) as Target Version = 0.22.0. > > Ian > > On Mon, Nov 3, 2014 at 10:22 AM, Ankur Chauhan <mailto:an...@malloc64.com>> wrote: > >> Hi Tim/others, >> >> Is this to be included in the 0.21.0 release? If so, I don't know how to >> tag it etc.

Re: Why rely on url scheme for fetching?

n contributing to >> Mesos and the style guide to ensure all the naming is as expected, then you >> can push the patch to reviewboard to get it reviewed and committed. >> >> On Mon, Nov 3, 2014 at 12:49 AM, Ankur Chauhan > <mailto:an...@malloc64.com>> wrote: >

Re: Why rely on url scheme for fetching?

nit tests over integration tests, so this shouldn't surprise anyone >> :) >> >> If you haven't already, please read the documentation on contributing to >> Mesos and the style guide to ensure all the naming is as expected, then you >> can push the patch to

Re: Why rely on url scheme for fetching?

mments and/or suggestions. -- Ankur @ankurcha > On 2 Nov 2014, at 18:17, Ankur Chauhan wrote: > > Hi, > > I noticed that the current set of tests in `src/tests/fetcher_tests.cpp` is > pretty coarse grained and are more on the lines of a functional test. I was > going to

Re: AWS - Master and Slave can't communicate

Custom TCP Rule - TCP - 2888 - 0.0.0.0/0 > Custom TCP Rule - TCP - 3888 - 0.0.0.0/0 > > > I don't know if this is correct or I've missed to open some other port. > >> On Mon, Nov 3, 2014 at 2:55 PM, Ankur Chauhan wrote: >> Are you running VPC? If so, what are

Re: AWS - Master and Slave can't communicate

Are you running VPC? If so, what are your routing /security group rules ? -- Ankur > On 2 Nov 2014, at 19:27, Matt Ivankov wrote: > > Hi, > > I'm new to Mesos and Marathon and I'd like to install it on AWS. For start, I > wanted to create 1 master and 1 slave (Mesos version 0.20.1). > > I've

Re: Why rely on url scheme for fetching?

it very soon. > I know Tim St. Clair had some ideas for fixing MESOS-1711 > <https://issues.apache.org/jira/browse/MESOS-1711>; he may want to review too. > > On Sat, Nov 1, 2014 at 8:49 PM, Ankur Chauhan <mailto:an...@malloc64.com>> wrote: > Hi Tim, > > I jus

Re: Why rely on url scheme for fetching?

. -- Ankur > On 1 Nov 2014, at 20:05, Ankur Chauhan wrote: > > Yea, i saw that the minute i pressed send. I'll start the review board so > that people can have a look at the change. > > -- Ankur > >> On 1 Nov 2014, at 20:01, Tim Chen > <mailto:t...@mesosphe

Re: Why rely on url scheme for fetching?

Yea, i saw that the minute i pressed send. I'll start the review board so that people can have a look at the change. -- Ankur > On 1 Nov 2014, at 20:01, Tim Chen wrote: > > Hi Ankur, > > There is a fetcher_tests.cpp in src/tests. > > Tim > > On Sat, Nov 1

Re: Why rely on url scheme for fetching?

Hi Ankur, > > Can you post on reviewboard? We can discuss more about the code there. > > Tim > > Sent from my iPhone > > On Nov 1, 2014, at 6:29 PM, Ankur Chauhan <mailto:an...@malloc64.com>> wrote: > >> Hi Tim, >> >> I don't think ther

Re: Why rely on url scheme for fetching?

31 October 2014 22:39, Tim Chen <mailto:t...@mesosphere.io>> wrote: > I believe there is already a JIRA ticket for this, if you search for fetcher > in Mesos JIRA I think you can find it. > > Tim > > On Fri, Oct 31, 2014 at 3:27 PM, Ankur Chauhan <mailto:an...@mall

Re: build failure on mac os x

r us not > having done that, yet. > > Let us know if that solved your problem. > > Till > >> On Nov 1, 2014, at 10:03 AM, Ankur Chauhan > <mailto:an...@malloc64.com>> wrote: >> >> I am trying to build the latest (master) mesos source and keep getting

build failure on mac os x

I am trying to build the latest (master) mesos source and keep getting this error: In file included from ../../src/state/log.cpp:25:0: ../../3rdparty/libprocess/3rdparty/stout/include/stout/svn.hpp:21:23: fatal error: svn_delta.h: No such file or directory #include ^ com

Why rely on url scheme for fetching?

Hi, I have been looking at some of the stuff around the fetcher and saw something interesting. The code for fetcher::fetch method is dependent on a hard coded list of url schemes. No doubt that this works but is very restrictive. Hadoop/HDFS in general is pretty flexible when it comes to being

Re: CDH5.2.3 on mesos

the property > hadoop.skip.worker.version.check to True - In the later case I wish you good > luck... Neverhteless, I am not sure if this error can also appear in other > cases. > > Hope this helps. > > Stratos > > >> On Tue, Oct 28, 2014 at 12:30 PM, Ankur

Re: CDH5.2.3 on mesos

Anyone else have something to add on this? -- Ankur Chauhan > On 28 Oct 2014, at 02:10, Ankur Chauhan wrote: > > Hi tom, > > I was basically following the readme. This gist has the list of commands how > i am setting up things > https://gist.github.com/ankurcha/a9504b0e4

Re: CDH5.2.3 on mesos

hare your hadoop configuration and perhaps a > description of the problems you're having? > > Tom. > > > > On Tue, Oct 28, 2014 at 8:53 AM, Ankur Chauhan <mailto:an...@malloc64.com>> wrote: > > H, > > > I was trying to setup mesos/hadoop with t

CDH5.2.3 on mesos

H, I was trying to setup mesos/hadoop with the latest CDH version (MR1) and it seems like the instructions are sort of out of date and I also tried the suggestions in https://github.com/mesos/hadoop/issues/25 but after 4 hours of flailing around I am

Re: Docker: pull on app start?

I had a similar issue and what I ended up doing was to explicitly set the version tag instead of just saying :latest. That makes the whole system much more stable/predictable. For the cost of a single curl call when you push is much better for debugging issues and the state of the system. -- Ank

Re: Exposing host services in docker container

> Try the following instead: > HOST_IP=`hostname -i` > docker run -it --rm dockerfile/mongodb bash -c "mongo --host $HOST_IP:27017" > > Tim > > On Sun, Oct 26, 2014 at 11:28 PM, Ankur Chauhan <mailto:an...@malloc64.com>> wrote: > Hi Tim, > > Thanks

Re: Exposing host services in docker container

container should be able to access any > port in the host, if it's cross hosts then you have to setup your own bridge > and use the lxc-conf option, or use something like pipework. > > I'm adding the lxc-conf options into the next release of mesos. > > Tim > >

Exposing host services in docker container

Hi all, So I have been dabbling with docker containers in and mesos. I have the following scenario and i was wondering if someone had experience with something like this. I have a bunch of dockerized apps running and the slave hosts have some common services running, something like mongos and

Re: Do i really need HDFS?

<< "' to '" << path << "'"; >Try result = hdfs.copyToLocal(uri, path); > if (result.isError()) { > LOG(ERROR) << "HDFS copyToLocal failed: " << result.error(); > return Error(result.error()); >} > --

Re: Do i really need HDFS?

This is what I also intend to do. Is a s3 path considered non-hdfs? If so, how does it know the credentials to use to fetch the file. Sent from my iPhone > On Oct 21, 2014, at 5:16 AM, David Greenberg wrote: > > We use spark without HDFS--in our case, we just use ansible to copy the spark >

Re: Do i really need HDFS?

t;> these required 'feature sets' is of interest to many. >> >> >> James >> >> >> >> >>> On 10/20/14 07:08, David Greenberg wrote: >>> You certainly don't need hdfs if you've got other infrastructure th

Do i really need HDFS?

P_HOME and get away with it? It would really help how other have their mesos setup in production or what they would recommend regarding my setup? --Ankur Chauhan

Re: Killing Docker containers

is something we want to make configurable. > > ryan > > On 15 October 2014 08:32, Ankur Chauhan <mailto:an...@malloc64.com>> wrote: > ++ I was planning on submitting that patch. But if someone has this sorted > out already, I'll defer. > > Sent from my

Re: Killing Docker containers

++ I was planning on submitting that patch. But if someone has this sorted out already, I'll defer. Sent from my iPhone > On Oct 14, 2014, at 2:19 PM, Ryan Thomas wrote: > > The docker stop command will attempt to kill the container if it doesn't stop > in 10 seconds by default. I think we s

Re: Multiple Network interfaces

Is using bridged networking with docker an option. That would map the service port (fixed) to a dynamically assigned port. Granted it will not utilize different nics but it's something. Sent from my iPhone > On Oct 8, 2014, at 8:52 AM, Jay Buffington wrote: > > My reading of the code is that

Re: Potential Seattle Mesos User Group

++ u would like to help. Contact the me directly and we can work out things. Sent from my iPhone > On Sep 21, 2014, at 10:29 AM, Dave Lester wrote: > > Hi All, > > I've chatted with a few different people (including some folks in the Twitter > Seattle office) about potentially starting a Sea

Re: Potential Seattle Mesos User Group

++ I would like to help. Sent from my iPhone > On Sep 21, 2014, at 10:29 AM, Dave Lester wrote: > > Hi All, > > I've chatted with a few different people (including some folks in the Twitter > Seattle office) about potentially starting a Seattle Mesos User Group. Does > anyone else want to h

Re: Frontend loadbalancer configuration for long running tasks

x27;d be > interested to see if this solution works for you! > > Thanks, > Tom > > On Thu, Sep 11, 2014 at 3:37 AM, Ankur Chauhan wrote: > >> I looked around a bit but the docs on aurora are almost non-existant when >> you go into anything more that the basics. Sing

Re: Frontend loadbalancer configuration for long running tasks

ody, > > Ankur's post is general enough, allowing me to reiterate the question: > > Does anyone know about similar HAProxy solutions for Aurora? > > Thanks, > Stephan > > > On Di, 2014-09-09 at 01:52 -0700, Ankur Chauhan wrote: > > Hi all, > > > >

Re: Frontend loadbalancer configuration for long running tasks

. As always, any comments suggestions are welcome! -- Ankur Chauhan On Tue, Sep 9, 2014 at 11:54 AM, Mike Michel wrote: > Hi, > > > > i changed the haproxy script so it makes use of haproxys ACL feature. I > need to have subdomains for every new app a user creates but always th

Re: Frontend loadbalancer configuration for long running tasks

se can be found at > https://github.com/opencredo/mesos_service_discovery > > The main advantage of this project is that it doesn't depend on DNS, so it > avoids issues with expiring TTLs and application caches, which can be > problematic. > > Cheers, > Bart &g

Re: Frontend loadbalancer configuration for long running tasks

/Opensourcing-Bamboo > > Cheers, > > Ryan > On 9 Sep 2014 18:53, "Ankur Chauhan" wrote: > >> Hi all, >> >> (Please let me know if this is not the correct place for such a question). >> I have been looking at mesos + marathon + haproxy as a way of deployin

Frontend loadbalancer configuration for long running tasks

Hi all, (Please let me know if this is not the correct place for such a question). I have been looking at mesos + marathon + haproxy as a way of deploying long running web applications. Mesos coupled with marathon's /tasks api gives me all the information needed to get a haproxy configured and loa

Logging for containers

in it. Something like /mnt/logs//taskId_.log Option 2: send the logs to syslog/rsyslog and use a centralized logging server of sorts Are there any other options? -- Ankur Chauhan

Re: Introducing Portainer

Could you share a link to the repo? On Wed, Sep 3, 2014 at 3:20 PM, Tom Arnfeld wrote: > Hey everyone, > > Thought it would be worth sharing this on the mailing list. We've recently > open sourced a Mesos framework called Portainer, which is for *building* > docker containers on top of your clu

Re: MongoDB on mesos

much as hoping a whole replica set is not moved. > Aurora, for example, implements this with a 'dedicated' scheduling > constraint specified in the job description. > > -=Bill > > > On Tue, Sep 2, 2014 at 9:01 PM, Ankur Chauhan > wrote: > >> Hi Vinod, &g

Re: MongoDB on mesos

work in this > area. > > > On Tue, Sep 2, 2014 at 2:50 PM, Ankur Chauhan wrote: > >> Hi all, >> >> I apologise for the repost but wanted to get some responses. I am evaluating >> mesos and wanted to know if anyone has successfully used >> mesos (with or

MongoDB on mesos

Hi all, I apologise for the repost but wanted to get some responses. I am evaluating mesos and wanted to know if anyone has successfully used mesos (with or without docker) as a means of managing a mongodb deployment. Currently, I have 8 shards where each shard is a 3x replicated replicaset and co

MongoDB on mesos

Hi all, I am evaluating mesos and wanted to know if anyone has successfully used mesos (with or without docker) as a means of managing a mongodb deployment. Currently, I have 8 shards where each shard is a 3x replicated replicaset and contains > 500GB of data. I was wondering if someone has tried

Re: Mesos autoscaling script

Looks cool! I'll check it out. Sent from my iPhone > On Aug 27, 2014, at 2:00 PM, Michael Babineau > wrote: > > I wrote a script to automatically launch/terminate Mesos slaves on EC2 based > on available resources: > https://github.com/thefactory/autoscale-python/blob/master/examples/mesos_e

Re: Mesos(phere) chef cookbook

This is great! This was probably 80% of the reason I couldn't push for mesos in my cluster! > On Jan 16, 2014, at 10:17, Vinod Kone wrote: > > Looks good! > > > @vinodkone > > >> On Thu, Jan 16, 2014 at 6:53 AM, Connor Doyle wrote: >> This is great news! >> Thanks Ray, >> -- >> Connor Doy

Re: Porting an app

at our deployment strategies, so I may not have been super clear about some stuff or over simplified some critical points. Please let me know what you think about this, I would love to contribute to marathon/mesos although I am pretty green as far as scala is concerned. Java is my strong su

Re: Porting an app

Thanks everyone for all the help. Marathon does seem like a good framework but my use case requires the app to evaluate it's own health and scale up based on internal load stats (SLA requirements) and I don't know if marathon supports that. This is the main reason why i am looking at building out m

Porting an app

Hi all, I just got started on mesos and I am trying to port a netty based webapp to use a mesos as a framework. I wrote a Executor and a Scheduler. But I am not quite sure as to how to go about getting everything to work together. I can't really find an example that is an end-to-end use case. B