Hello, list.
I'm trying to debug my spark application on Intellij IDE. Before I submit
my job, I ran the command line:
export
SPARK_SUBMIT_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=4000
after that:
bin/spark-submit app-jar-with-dependencies.jar <arguments>
The IDE connects with the running job but all code that is running on
worker machine is unreachable to debug. See below:
rdd.foreachPartition(partition -> { //breakpoint stop here
partition.forEachRemaining(message -> {
//breakpoint doenst stop here
})
});
Does anyone know if is is possible? How? Any ideas?
