Yes, scheduling is centralized in the driver. For debugging, I think you'd want to set the executor JVM, not the worker JVM flags.
On Thu, Jun 30, 2016 at 11:36 AM, cbruegg <m...@cbruegg.com> wrote: > Hello everyone, > > I'm a student assistant in research at the University of Paderborn, working > on integrating Spark (v1.6.2) with a new network resource management > system. > I have already taken a deep dive into the source code of spark-core w.r.t. > its scheduling systems. > > We are running a cluster in standalone mode consisting of a master node and > three slave nodes. Am I right to assume that tasks are scheduled within the > TaskSchedulerImpl using the DAGScheduler in this mode? I need to find a > place where the execution plan (and each stage) for a job is computed and > can be analyzed, so I placed some breakpoints in these two classes. > > The remote debugging session within IntelliJ IDEA has been established by > running the following commands on the master node before: > > export SPARK_WORKER_OPTS="-Xdebug > -Xrunjdwp:server=y,transport=dt_socket,address=4000,suspend=n" > export SPARK_MASTER_OPTS="-Xdebug > -Xrunjdwp:server=y,transport=dt_socket,address=4000,suspend=n" > > Port 4000 has been forwarded to my local machine. Unfortunately, none of my > breakpoints through the class get hit when I invoke a task like > sc.parallelize(1 to 1000).count() in spark-shell on the master node (using > --master spark://...), though when I pause all threads I can see that the > process I am debugging runs some kind of event queue, which means that the > debugger is connected to /something/. > > Do I rely on false assumptions or should these breakpoints in fact get hit? > I am not too familiar with Spark, so please bear with me if I got something > wrong. Many thanks in advance for your help. > > Best regards, > Christian Brüggemann > > > > -- > View this message in context: > http://apache-spark-developers-list.1001551.n3.nabble.com/Debugging-Spark-itself-in-standalone-cluster-mode-tp18139.html > Sent from the Apache Spark Developers List mailing list archive at > Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
