Take a look at https://github.com/lresende/docker-systemml-notebook
I am not using yarn, but have spark, hdfs, zeppelin working all together. On Wednesday, April 6, 2016, John Omernik <j...@omernik.com> wrote: > So, I am running into this again, net=host isn't an option for me, because > then my hadoop client conflicts with the nodes hadoop client. So I am > trying to work out net=bridge, however, even specifying the driver port and > host in spark doesn't work because spark tries to listen on only the > hostname specified in spark.driver.host, not on all interfaces, thus it > doesn't work inside the container. Anyone else have luck with running > Zeppelin in a Docker container, and using Yarn? > > On Wed, Jun 17, 2015 at 12:20 PM, John Omernik <j...@omernik.com > <javascript:_e(%7B%7D,'cvml','j...@omernik.com');>> wrote: > >> My challenge is I am running this in mesos, therefore I have a limited >> amount of ports and I would need to know them ahead of time. Even with >> that ,the IP provided to the Application Master by the is internal Docker >> IP, so that proves challenging as well. Any other thoughts on this? >> >> On Wed, Jun 17, 2015 at 12:03 PM, Jong Wook Kim <jongw...@nyu.edu >> <javascript:_e(%7B%7D,'cvml','jongw...@nyu.edu');>> wrote: >> >>> I just used the --net=host option to make the container just use the >>> host’s network stack which is probably the easiest way. >>> >>> if you are not allowed to do that, change the container’s >>> /proc/sys/net/ipv4/ip_local_port_range >>> to have a narrower port range, like 59000-59999 and forward the entire >>> range. >>> >>> >>> On Jun 18, 2015, at 1:15 AM, John Omernik <j...@omernik.com >>> <javascript:_e(%7B%7D,'cvml','j...@omernik.com');>> wrote: >>> >>> So I think I ran across an issue with Zeppelin/Docker but I wanted to >>> describe here and see what people thought. >>> >>> Basically I have a YARN cluster (Yarn... well Myriad on Mesos, but for >>> pyspark, it works like Yarn). >>> >>> I have setup a docker container with all I need including Spark etc. >>> >>> When I try to run pyspark, my Application Master in Yarn tries to >>> connect back to the driver which is inconveniently located in a docker >>> container, and the only ports exposed are those for Zeppelin. (My yarn >>> spark applicaiton is accepted, but I see in the logs, and it never "runs" >>> >>> 15/06/17 11:04:29 ERROR yarn.ApplicationMaster: Failed to connect to driver >>> at 172.17.0.16:59601, retrying ... >>> 15/06/17 11:04:32 ERROR yarn.ApplicationMaster: Failed to connect to driver >>> at 172.17.0.16:59601, retrying . >>> >>> >>> Ok, so yarn-client mode won't work (easily), however, yarn-cluster mode >>> doesn't work either because you can't run a shell in yarn-cluster mode. >>> >>> So this is all just trying to run $SPARK_HOME/bin/pyspark --master >>> yarn-client from inside of my Zeppelin Docker container (not even running >>> Zeppelin yet) >>> >>> I am guessing that the issue is part of my issues I am having getting >>> the spark interpreter running in Zeppelin because Zeppelin is doing just >>> that, running pyspark in yarn-client mode. >>> >>> Ok, so how have people running Zeppelin in Docker dealt with this? What >>> other ideas should I look into here? >>> >>> >>> >>> >> > -- Sent from my Mobile device
