[
https://issues.apache.org/jira/browse/SPARK-29202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sandeep Katta updated SPARK-29202:
----------------------------------
Description:
Run the below command
./bin/spark-sql --master yarn
--driver-java-options="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5555"
In Spark 2.3.3
/opt/softwares/Java/jdk1.8.0_211/bin/java -cp
/opt/BigdataTools/spark-2.3.3-bin-hadoop2.7/conf/:/opt/BigdataTools/spark-2.3.3-bin-hadoop2.7/jars/*:/opt/BigdataTools/hadoop-3.2.0/etc/hadoop/
-Xmx1g -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5555
org.apache.spark.deploy.SparkSubmit --master yarn --conf
spark.driver.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5555
--class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver spark-internal
In Spark 3.0
/opt/softwares/Java/jdk1.8.0_211/bin/java -cp
/opt/apache/git/sparkSourceCode/spark/conf/:/opt/apache/git/sparkSourceCode/spark/assembly/target/scala-2.12/jars/*:/opt/BigdataTools/hadoop-3.2.0/etc/hadoop/
org.apache.spark.deploy.SparkSubmit --master yarn --conf
spark.driver.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5556
--class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver spark-internal
We can see that java options are not passed to driver process in spark3
> --driver-java-options are not passed to driver process in yarn client mode
> --------------------------------------------------------------------------
>
> Key: SPARK-29202
> URL: https://issues.apache.org/jira/browse/SPARK-29202
> Project: Spark
> Issue Type: Bug
> Components: Deploy
> Affects Versions: 3.0.0
> Reporter: Sandeep Katta
> Priority: Major
>
> Run the below command
> ./bin/spark-sql --master yarn
> --driver-java-options="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5555"
>
> In Spark 2.3.3
> /opt/softwares/Java/jdk1.8.0_211/bin/java -cp
> /opt/BigdataTools/spark-2.3.3-bin-hadoop2.7/conf/:/opt/BigdataTools/spark-2.3.3-bin-hadoop2.7/jars/*:/opt/BigdataTools/hadoop-3.2.0/etc/hadoop/
> -Xmx1g -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5555
> org.apache.spark.deploy.SparkSubmit --master yarn --conf
> spark.driver.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5555
> --class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver
> spark-internal
>
> In Spark 3.0
> /opt/softwares/Java/jdk1.8.0_211/bin/java -cp
> /opt/apache/git/sparkSourceCode/spark/conf/:/opt/apache/git/sparkSourceCode/spark/assembly/target/scala-2.12/jars/*:/opt/BigdataTools/hadoop-3.2.0/etc/hadoop/
> org.apache.spark.deploy.SparkSubmit --master yarn --conf
> spark.driver.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5556
> --class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver
> spark-internal
> We can see that java options are not passed to driver process in spark3
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
