Yup, alright, same solution then :)
On Tue, Jun 17, 2014 at 7:39 PM, Mohit Jaggi <mohitja...@gmail.com> wrote: > I used --privileged to start the container and then unmounted /etc/hosts. > Then I created a new /etc/hosts file > > > On Tue, Jun 17, 2014 at 4:58 PM, Aaron Davidson <ilike...@gmail.com> > wrote: > >> I remember having to do a similar thing in the spark docker scripts for >> testing purposes. Were you able to modify the /etc/hosts directly? I >> remember issues with that as docker apparently mounts it as part of its >> read-only filesystem. >> >> >> On Tue, Jun 17, 2014 at 4:36 PM, Mohit Jaggi <mohitja...@gmail.com> >> wrote: >> >>> It was a DNS issue. AKKA apparently uses the hostname of the endpoints >>> and hence they need to be resolvable. In my case the hostname of the docker >>> container was a randomly generated string and was not resolvable. I added a >>> workaround (entry in etc/hosts file of spark master) for now. If anyone can >>> point to a more elegant solution, that would be awesome! >>> >>> >>> On Tue, Jun 17, 2014 at 3:48 PM, Mohit Jaggi <mohitja...@gmail.com> >>> wrote: >>> >>>> I am using cutting edge code from git but doing my own sbt assembly. >>>> >>>> >>>> On Mon, Jun 16, 2014 at 10:28 PM, Andre Schumacher < >>>> schum...@icsi.berkeley.edu> wrote: >>>> >>>>> >>>>> Hi, >>>>> >>>>> are you using the amplab/spark-1.0.0 images from the global registry? >>>>> >>>>> Andre >>>>> >>>>> On 06/17/2014 01:36 AM, Mohit Jaggi wrote: >>>>> > Hi Folks, >>>>> > >>>>> > I am having trouble getting spark driver running in docker. If I run >>>>> a >>>>> > pyspark example on my mac it works but the same example on a docker >>>>> image >>>>> > (Via boot2docker) fails with following logs. I am pointing the spark >>>>> driver >>>>> > (which is running the example) to a spark cluster (driver is not >>>>> part of >>>>> > the cluster). I guess this has something to do with docker's >>>>> networking >>>>> > stack (it may be getting NAT'd) but I am not sure why (if at all) the >>>>> > spark-worker or spark-master is trying to create a new TCP >>>>> connection to >>>>> > the driver, instead of responding on the connection initiated by the >>>>> driver. >>>>> > >>>>> > I would appreciate any help in figuring this out. >>>>> > >>>>> > Thanks, >>>>> > >>>>> > Mohit. >>>>> > >>>>> > --------logs-------- >>>>> > >>>>> > Spark Executor Command: "java" "-cp" >>>>> > >>>>> "::/home/ayasdi/spark/conf:/home/xxxx/spark/assembly/target/scala-2.10/spark-assembly-1.0.0-hadoop1.0.4.jar" >>>>> > "-Xms2g" "-Xmx2g" "-Xms512M" "-Xmx512M" >>>>> > "org.apache.spark.executor.CoarseGrainedExecutorBackend" >>>>> > "akka.tcp://spark@fc31887475e3:43921/user/CoarseGrainedScheduler" >>>>> "1" >>>>> > "cobalt" "24" "akka.tcp://sparkWorker@aaaa:33952/user/Worker" >>>>> > "app-20140616152201-0021" >>>>> > >>>>> > ======================================== >>>>> > >>>>> > >>>>> > log4j:WARN No appenders could be found for logger >>>>> > (org.apache.hadoop.conf.Configuration). >>>>> > >>>>> > log4j:WARN Please initialize the log4j system properly. >>>>> > >>>>> > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig >>>>> for >>>>> > more info. >>>>> > >>>>> > 14/06/16 15:22:05 INFO SparkHadoopUtil: Using Spark's default log4j >>>>> > profile: org/apache/spark/log4j-defaults.properties >>>>> > >>>>> > 14/06/16 15:22:05 INFO SecurityManager: Changing view acls to: >>>>> ayasdi,root >>>>> > >>>>> > 14/06/16 15:22:05 INFO SecurityManager: SecurityManager: >>>>> authentication >>>>> > disabled; ui acls disabled; users with view permissions: Set(xxx, >>>>> xxx) >>>>> > >>>>> > 14/06/16 15:22:05 INFO Slf4jLogger: Slf4jLogger started >>>>> > >>>>> > 14/06/16 15:22:05 INFO Remoting: Starting remoting >>>>> > >>>>> > 14/06/16 15:22:06 INFO Remoting: Remoting started; listening on >>>>> addresses >>>>> > :[akka.tcp://sparkExecutor@aaaa:33536] >>>>> > >>>>> > 14/06/16 15:22:06 INFO Remoting: Remoting now listens on addresses: >>>>> > [akka.tcp://sparkExecutor@aaaa:33536] >>>>> > >>>>> > 14/06/16 15:22:06 INFO CoarseGrainedExecutorBackend: Connecting to >>>>> driver: >>>>> > akka.tcp://spark@fc31887475e3:43921/user/CoarseGrainedScheduler >>>>> > >>>>> > 14/06/16 15:22:06 INFO WorkerWatcher: Connecting to worker >>>>> > akka.tcp://sparkWorker@aaaa:33952/user/Worker >>>>> > >>>>> > 14/06/16 15:22:06 WARN Remoting: Tried to associate with unreachable >>>>> remote >>>>> > address [akka.tcp://spark@fc31887475e3:43921]. Address is now gated >>>>> for >>>>> > 60000 ms, all messages to this address will be delivered to dead >>>>> letters. >>>>> > >>>>> > 14/06/16 15:22:06 ERROR CoarseGrainedExecutorBackend: Driver >>>>> Disassociated >>>>> > [akka.tcp://sparkExecutor@aaaa:33536] -> >>>>> [akka.tcp://spark@fc31887475e3:43921] >>>>> > disassociated! Shutting down. >>>>> > >>>>> >>>>> >>>> >>> >> >
