Finally got the answer. Actually it works fine. The allocation behavior on mesos is a little different from yarn/standalone. Seems the executor in mesos is lazily allocated (only when job is executed) while executor in yarn/standalone is allocated when spark-shell is started.
On Tue, Sep 8, 2015 at 10:39 PM, canan chen <ccn...@gmail.com> wrote: > Yes, I follow the guide in this doc, and run it as mesos client mode > > On Tue, Sep 8, 2015 at 6:31 PM, Akhil Das <ak...@sigmoidanalytics.com> > wrote: > >> In which mode are you submitting your application? (coarse-grained or >> fine-grained(default)). Have you gone through this documentation already? >> http://spark.apache.org/docs/latest/running-on-mesos.html#using-a-mesos-master-url >> >> Thanks >> Best Regards >> >> On Tue, Sep 8, 2015 at 12:54 PM, canan chen <ccn...@gmail.com> wrote: >> >>> Hi all, >>> >>> I try to run spark on mesos, but it looks like I can not allocate >>> resources from mesos. I am not expert of mesos, but from the mesos log, it >>> seems spark always decline the offer from mesos. Not sure what's wrong, >>> maybe need some configuration change. Here's the mesos master log >>> >>> I0908 15:08:16.515960 301916160 master.cpp:1767] Received registration >>> request for framework 'Spark shell' at >>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133 >>> I0908 15:08:16.520545 301916160 master.cpp:1834] Registering framework >>> 20150908-143320-16777343-5050-41965-0000 (Spark shell) at >>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133 with >>> checkpointing disabled and capabilities [ ] >>> I0908 15:08:16.522307 300843008 hierarchical.hpp:386] Added framework >>> 20150908-143320-16777343-5050-41965-0000 >>> I0908 15:08:16.525845 301379584 master.cpp:4290] Sending 1 offers to >>> framework 20150908-143320-16777343-5050-41965-0000 (Spark shell) at >>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133 >>> I0908 15:08:16.637677 302452736 master.cpp:2884] Processing DECLINE call >>> for offers: [ 20150908-143320-16777343-5050-41965-O0 ] for framework >>> 20150908-143320-16777343-5050-41965-0000 (Spark shell) at >>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133 >>> I0908 15:08:16.639197 299233280 hierarchical.hpp:761] Recovered >>> cpus(*):8; mem(*):15360; disk(*):470842; ports(*):[31000-32000] (total: >>> cpus(*):8; mem(*):15360; disk(*):470842; ports(*):[31000-32000], allocated: >>> ) on slave 20150908-143320-16777343-5050-41965-S0 from framework >>> 20150908-143320-16777343-5050-41965-0000 >>> I0908 15:08:21.786932 300306432 master.cpp:4290] Sending 1 offers to >>> framework 20150908-143320-16777343-5050-41965-0000 (Spark shell) at >>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133 >>> I0908 15:08:21.789979 298696704 master.cpp:2884] Processing DECLINE call >>> for offers: [ 20150908-143320-16777343-5050-41965-O1 ] for framework >>> 20150908-143320-16777343-5050-41965-0000 (Spark shell) at >>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133 >>> >> >> >
