Hi Gerard, thank you for your feedbacks.
On Mon, May 5, 2014 at 11:17 PM, Gerard Maas <gerard.m...@gmail.com> wrote: > Hi Benjamin, > > Yes, we initially used a modified version of the AmpLabs docker scripts > [1]. The amplab docker images are a good starting point. > One of the biggest hurdles has been HDFS, which requires reverse-DNS and I > didn't want to go the dnsmasq route to keep the containers relatively > simple to use without the need of external scripts. Ended up running a > 1-node setup nnode+dnode. I'm still looking for a better solution for HDFS > [2] > > Our usecase using docker is to easily create local dev environments both > for development and for automated functional testing (using cucumber). My > aim is to strongly reduce the time of the develop-deploy-test cycle. > That also means that we run the minimum number of instances required to > have a functionally working setup. E.g. 1 Zookeeper, 1 Kafka broker, ... > > For the actual cluster deployment we have Chef-based devops toolchain that > put things in place on public cloud providers. > Personally, I think Docker rocks and would like to replace those complex > cookbooks with Dockerfiles once the technology is mature enough. > > -greetz, Gerard. > > [1] https://github.com/amplab/docker-scripts > [2] > http://stackoverflow.com/questions/23410505/how-to-run-hdfs-cluster-without-dns > > > On Mon, May 5, 2014 at 11:00 PM, Benjamin <bboui...@gmail.com> wrote: > >> Hi, >> >> Before considering running on Mesos, did you try to submit the >> application on Spark deployed without Mesos on Docker containers ? >> >> Currently investigating this idea to deploy quickly a complete set of >> clusters with Docker, I'm interested by your findings on sharing the >> settings of Kafka and Zookeeper across nodes. How many broker and zookeeper >> do you use ? >> >> Regards, >> >> >> >> On Mon, May 5, 2014 at 10:11 PM, Gerard Maas <gerard.m...@gmail.com>wrote: >> >>> Hi all, >>> >>> I'm currently working on creating a set of docker images to facilitate >>> local development with Spark/streaming on Mesos (+zk, hdfs, kafka) >>> >>> After solving the initial hurdles to get things working together in >>> docker containers, now everything seems to start-up correctly and the mesos >>> UI shows slaves as they are started. >>> >>> I'm trying to submit a job from IntelliJ and the jobs submissions seem >>> to get lost in Mesos translation. The logs are not helping me to figure out >>> what's wrong, so I'm posting them here in the hope that they can ring a >>> bell and somebdoy could provide me a hint on what's wrong/missing with my >>> setup. >>> >>> >>> ---- DRIVER (IntelliJ running a Job.scala main) ---- >>> 14/05/05 21:52:31 INFO MetadataCleaner: Ran metadata cleaner for >>> SHUFFLE_BLOCK_MANAGER >>> 14/05/05 21:52:31 INFO BlockManager: Dropping broadcast blocks older >>> than 1399319251962 >>> 14/05/05 21:52:31 INFO BlockManager: Dropping non broadcast blocks older >>> than 1399319251962 >>> 14/05/05 21:52:31 INFO MetadataCleaner: Ran metadata cleaner for >>> BROADCAST_VARS >>> 14/05/05 21:52:31 INFO MetadataCleaner: Ran metadata cleaner for >>> BLOCK_MANAGER >>> 14/05/05 21:52:32 INFO MetadataCleaner: Ran metadata cleaner for >>> HTTP_BROADCAST >>> 14/05/05 21:52:32 INFO MetadataCleaner: Ran metadata cleaner for >>> MAP_OUTPUT_TRACKER >>> 14/05/05 21:52:32 INFO MetadataCleaner: Ran metadata cleaner for >>> SPARK_CONTEXT >>> >>> >>> ---- MESOS MASTER ---- >>> I0505 19:52:39.718080 388 master.cpp:690] Registering framework >>> 201405051517-67113388-5050-383-6995 at scheduler(1)@127.0.1.1:58115 >>> I0505 19:52:39.718261 388 master.cpp:493] Framework >>> 201405051517-67113388-5050-383-6995 disconnected >>> I0505 19:52:39.718277 389 hierarchical_allocator_process.hpp:332] >>> Added framework 201405051517-67113388-5050-383-6995 >>> I0505 19:52:39.718312 388 master.cpp:520] Giving framework >>> 201405051517-67113388-5050-383-6995 0ns to failover >>> I0505 19:52:39.718431 389 hierarchical_allocator_process.hpp:408] >>> Deactivated framework 201405051517-67113388-5050-383-6995 >>> W0505 19:52:39.718459 388 master.cpp:1388] Master returning resources >>> offered to framework 201405051517-67113388-5050-383-6995 because the >>> framework has terminated or is inactive >>> I0505 19:52:39.718567 388 master.cpp:1376] Framework failover timeout, >>> removing framework 201405051517-67113388-5050-383-6995 >>> >>> >>> >>> ---- MESOS SLAVE ---- >>> I0505 19:49:27.662019 20 slave.cpp:1191] Asked to shut down framework >>> 201405051517-67113388-5050-383-6803 by master@172.17.0.4:5050 >>> W0505 19:49:27.662072 20 slave.cpp:1206] Cannot shut down unknown >>> framework 201405051517-67113388-5050-383-6803 >>> I0505 19:49:28.662153 18 slave.cpp:1191] Asked to shut down framework >>> 201405051517-67113388-5050-383-6804 by master@172.17.0.4:5050 >>> W0505 19:49:28.662212 18 slave.cpp:1206] Cannot shut down unknown >>> framework 201405051517-67113388-5050-383-6804 >>> I0505 19:49:29.662199 13 slave.cpp:1191] Asked to shut down framework >>> 201405051517-67113388-5050-383-6805 by master@172.17.0.4:5050 >>> W0505 19:49:29.662256 13 slave.cpp:1206] Cannot shut down unknown >>> framework 201405051517-67113388-5050-383-6805 >>> I0505 19:49:30.662443 16 slave.cpp:1191] Asked to shut down framework >>> 201405051517-67113388-5050-383-6806 by master@172.17.0.4:5050 >>> W0505 19:49:30.662489 16 slave.cpp:1206] Cannot shut down unknown >>> framework 201405051517-67113388-5050-383-6806 >>> >>> >>> Thanks in advance, >>> >>> Gerard. >>> >> >> >> >> -- >> Benjamin Bouillé >> +33 665 050 285 >> > > -- Benjamin Bouillé +33 665 050 285
