Re: Difference between sparkDriver and "executor ID driver"

2015-09-16 Thread Hemant Bhanawat
1. When you call new SparkContext(), spark driver is started which internally create Akka ActorSystem which registers on this port. 2. Since you are running in local mode, starting of executor is short circuited and an Executor object is created in the same process (see LocalEndpoint). This

Difference between sparkDriver and "executor ID driver"

2015-09-15 Thread Muler
I'm running Spark in local mode and getting these two log messages who appear to be similar. I want to understand what each is doing: 1. [main] util.Utils (Logging.scala:logInfo(59)) - Successfully started service 'sparkDriver' on port 60782. 2. [main] executor.Executor