Hi, Before considering running on Mesos, did you try to submit the application on Spark deployed without Mesos on Docker containers ?
Currently investigating this idea to deploy quickly a complete set of clusters with Docker, I'm interested by your findings on sharing the settings of Kafka and Zookeeper across nodes. How many broker and zookeeper do you use ? Regards, On Mon, May 5, 2014 at 10:11 PM, Gerard Maas <gerard.m...@gmail.com> wrote: > Hi all, > > I'm currently working on creating a set of docker images to facilitate > local development with Spark/streaming on Mesos (+zk, hdfs, kafka) > > After solving the initial hurdles to get things working together in docker > containers, now everything seems to start-up correctly and the mesos UI > shows slaves as they are started. > > I'm trying to submit a job from IntelliJ and the jobs submissions seem to > get lost in Mesos translation. The logs are not helping me to figure out > what's wrong, so I'm posting them here in the hope that they can ring a > bell and somebdoy could provide me a hint on what's wrong/missing with my > setup. > > > ---- DRIVER (IntelliJ running a Job.scala main) ---- > 14/05/05 21:52:31 INFO MetadataCleaner: Ran metadata cleaner for > SHUFFLE_BLOCK_MANAGER > 14/05/05 21:52:31 INFO BlockManager: Dropping broadcast blocks older than > 1399319251962 > 14/05/05 21:52:31 INFO BlockManager: Dropping non broadcast blocks older > than 1399319251962 > 14/05/05 21:52:31 INFO MetadataCleaner: Ran metadata cleaner for > BROADCAST_VARS > 14/05/05 21:52:31 INFO MetadataCleaner: Ran metadata cleaner for > BLOCK_MANAGER > 14/05/05 21:52:32 INFO MetadataCleaner: Ran metadata cleaner for > HTTP_BROADCAST > 14/05/05 21:52:32 INFO MetadataCleaner: Ran metadata cleaner for > MAP_OUTPUT_TRACKER > 14/05/05 21:52:32 INFO MetadataCleaner: Ran metadata cleaner for > SPARK_CONTEXT > > > ---- MESOS MASTER ---- > I0505 19:52:39.718080 388 master.cpp:690] Registering framework > 201405051517-67113388-5050-383-6995 at scheduler(1)@127.0.1.1:58115 > I0505 19:52:39.718261 388 master.cpp:493] Framework > 201405051517-67113388-5050-383-6995 disconnected > I0505 19:52:39.718277 389 hierarchical_allocator_process.hpp:332] Added > framework 201405051517-67113388-5050-383-6995 > I0505 19:52:39.718312 388 master.cpp:520] Giving framework > 201405051517-67113388-5050-383-6995 0ns to failover > I0505 19:52:39.718431 389 hierarchical_allocator_process.hpp:408] > Deactivated framework 201405051517-67113388-5050-383-6995 > W0505 19:52:39.718459 388 master.cpp:1388] Master returning resources > offered to framework 201405051517-67113388-5050-383-6995 because the > framework has terminated or is inactive > I0505 19:52:39.718567 388 master.cpp:1376] Framework failover timeout, > removing framework 201405051517-67113388-5050-383-6995 > > > > ---- MESOS SLAVE ---- > I0505 19:49:27.662019 20 slave.cpp:1191] Asked to shut down framework > 201405051517-67113388-5050-383-6803 by master@172.17.0.4:5050 > W0505 19:49:27.662072 20 slave.cpp:1206] Cannot shut down unknown > framework 201405051517-67113388-5050-383-6803 > I0505 19:49:28.662153 18 slave.cpp:1191] Asked to shut down framework > 201405051517-67113388-5050-383-6804 by master@172.17.0.4:5050 > W0505 19:49:28.662212 18 slave.cpp:1206] Cannot shut down unknown > framework 201405051517-67113388-5050-383-6804 > I0505 19:49:29.662199 13 slave.cpp:1191] Asked to shut down framework > 201405051517-67113388-5050-383-6805 by master@172.17.0.4:5050 > W0505 19:49:29.662256 13 slave.cpp:1206] Cannot shut down unknown > framework 201405051517-67113388-5050-383-6805 > I0505 19:49:30.662443 16 slave.cpp:1191] Asked to shut down framework > 201405051517-67113388-5050-383-6806 by master@172.17.0.4:5050 > W0505 19:49:30.662489 16 slave.cpp:1206] Cannot shut down unknown > framework 201405051517-67113388-5050-383-6806 > > > Thanks in advance, > > Gerard. > -- Benjamin Bouillé +33 665 050 285
