Update to Spar Mesos docs possibly? LIBPROCESS_IP needs to be set for client mode

In going through running various Spark jobs, both Spark 1.5.2 and the new Spark 1.6 SNAPSHOTs, on a Mesos cluster (currently 0.25), we noticed that is in order to run the Spark shells (both python and scala), we needed to set the LIBPROCESS_IP environment variable before running. Was curious if

Re: Update to Spar Mesos docs possibly? LIBPROCESS_IP needs to be set for client mode

The more I read/look into this..it's because the Spark Mesos Scheduler resolves to something can't be reached (e.g. localhost). So, maybe this just needs to be added to the docs, if your host cannot or does not resolve to the IP address you want. Maybe just a footnote or something? On Wed,

Re: Update to Spar Mesos docs possibly? LIBPROCESS_IP needs to be set for client mode

Found this thread that talked about it to help understand it better: https://mail-archives.apache.org/mod_mbox/mesos-user/201507.mbox/%3ccajq68qf9pejgnwomasm2dqchyaxpcaovnfkfgggxxpzj2jo...@mail.gmail.com%3E > > When you run Spark on Mesos it needs to run > > spark driver > mesos scheduler > >

Re: Update to Spar Mesos docs possibly? LIBPROCESS_IP needs to be set for client mode

Basically, my hostname doesn't resolve to an "accessible" IP address...which isn't a big deal, I normally set SPARK_LOCAL_IP when I am doing things on a YARN cluster. But, we've moved to a Mesos Cluster recently, and had to track down when it wasn't working...I assumed (badly obviously) that

Re: Update to Spar Mesos docs possibly? LIBPROCESS_IP needs to be set for client mode

LIBPROCESS_IP has zero hits in the Spark code base. This seems to be a Mesos-specific setting. Have you tried setting SPARK_LOCAL_IP? On Wed, Dec 16, 2015 at 5:07 PM, Aaron wrote: > Found this thread that talked about it to help understand it better: > > >

Re: Update to Spar Mesos docs possibly? LIBPROCESS_IP needs to be set for client mode

Sure, documenting this would be great, I just wanted to understand the context. There is a related ticket: SPARK-5488 . Would you mind opening a PR? On Wed, Dec 16, 2015 at 5:11 PM, Aaron wrote: > Basically, my hostname

Re: Update to Spar Mesos docs possibly? LIBPROCESS_IP needs to be set for client mode

Hi Aaron, I never had to use that variable. What is it for? On Wed, Dec 16, 2015 at 2:00 PM, Aaron wrote: > In going through running various Spark jobs, both Spark 1.5.2 and the > new Spark 1.6 SNAPSHOTs, on a Mesos cluster (currently 0.25), we > noticed that is in order

Re: Update to Spar Mesos docs possibly? LIBPROCESS_IP needs to be set for client mode

Yes if want to manually override what IP to use to be contacted by the master you can set LIPROCESS_IP and LIBPROCESS_PORT. It is a Mesos specific settings. We can definitely update the docs. Note that in the future as we move to use the new Mesos Http API these configurations won't be needed

Re: Update to Spar Mesos docs possibly? LIBPROCESS_IP needs to be set for client mode

Wrt to PR, sure, let me update the documentation, i'll send it out shortly. My Fork is on Github..is the PR from there ok? Cheers, Aaron On Wed, Dec 16, 2015 at 11:33 AM, Timothy Chen wrote: > Yes if want to manually override what IP to use to be contacted by the > master