By the way, 64 MB of RAM per machine is really small, I'm surprised Spark can even start up on that! Perhaps you meant to set SPARK_DAEMON_MEMORY so that the actual worker process itself would be small, but SPARK_WORKER_MEMORY (which controls the amount of memory available for Spark executors) should be at least 512 MB, and ideally many times that.
On Sun, Apr 13, 2014 at 1:14 AM, Aaron Davidson <[email protected]> wrote: > This was actually a bug in the log message itself, where the Master would > print its own ip and port instead of the registered worker's. It has been > fixed in 0.9.1 and 1.0.0 (here's the patch: > https://github.com/apache/spark/commit/c0795cf481d47425ec92f4fd0780e2e0b3fdda85 > ). > > Sorry about the confusion! > > > On Sat, Apr 12, 2014 at 4:36 PM, Mark Baker <[email protected]> wrote: > >> On Sat, Apr 12, 2014 at 9:19 AM, ge ko <[email protected]> wrote: >> > Hi, >> > >> > I'm wondering why the master is registering itself at startup, exactly 3 >> > times (same number as the number of workers). Log excerpt: >> > "" >> > 2014-04-11 21:08:15,363 INFO akka.event.slf4j.Slf4jLogger: Slf4jLogger >> > started >> > 2014-04-11 21:08:15,478 INFO Remoting: Starting remoting >> > 2014-04-11 21:08:15,838 INFO Remoting: Remoting started; listening on >> > addresses :[akka.tcp://[email protected]:7077] >> > 2014-04-11 21:08:16,252 INFO org.apache.spark.deploy.master.Master: >> Starting >> > Spark master at spark://hadoop-pg-5.cluster:7077 >> > 2014-04-11 21:08:16,299 INFO org.eclipse.jetty.server.Server: >> > jetty-7.x.y-SNAPSHOT >> > 2014-04-11 21:08:16,341 INFO org.eclipse.jetty.server.AbstractConnector: >> > Started [email protected]:18080 >> > 2014-04-11 21:08:16,343 INFO >> org.apache.spark.deploy.master.ui.MasterWebUI: >> > Started Master web UI at http://hadoop-pg-5.cluster:18080 >> > 2014-04-11 21:08:16,374 INFO org.apache.spark.deploy.master.Master: I >> have >> > been elected leader! New state: ALIVE >> > 2014-04-11 21:08:21,492 INFO org.apache.spark.deploy.master.Master: >> > Registering worker hadoop-pg-5.cluster:7078 with 2 cores, 64.0 MB RAM >> > 2014-04-11 21:08:31,362 INFO org.apache.spark.deploy.master.Master: >> > Registering worker hadoop-pg-5.cluster:7078 with 2 cores, 64.0 MB RAM >> > 2014-04-11 21:08:34,819 INFO org.apache.spark.deploy.master.Master: >> > Registering worker hadoop-pg-5.cluster:7078 with 2 cores, 64.0 MB RAM >> > "" >> > >> > The workers should be hadoop-pg-7/-8/-9 >> > This seems strange to me, or do I just interpret the log entry wrong >> ?!?! >> > Perhaps this relates to my other post titled "TaskSchedulerImpl: >> Initial job >> > has not accepted any resources" and both issues are caused by the same >> > problem/misconfiguration. >> > >> > All the nodes can reach each other via ping and telnet on the >> corresponding >> > ports. >> > >> > Any hints what is going wrong there? >> >> I had this happen in a virtualbox configuration I was testing with, >> and never got it fixed, but suspected it was because I wasn't using >> FQDNs (as you're also not doing). I had found - can't find it again >> now, of course - a message in the archives from somebody also >> suffering from this issue while still using FQDNs, but claimed that >> they found an error in /etc/hosts. >> >> Good luck. >> > >
