[
https://issues.apache.org/jira/browse/SPARK-29070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Evans updated SPARK-29070:
-------------------------------
Description:
{{org.apache.spark.launcher.SparkLauncher}} wraps a {{ProcessBuilder}}, and
builds up a full command line to {{spark-submit}} using a builder pattern.
When {{startApplication}} is finally called, a full command line is
materialized out of all the options, then invoked via the {{ProcessBuilder}}.
In scenarios where another application is submitting to Spark, it would be
extremely useful from a support and debugging standpoint to be able to see the
full {{spark-submit}} command that is actually used (so that the same
submission can be tested standalone, arguments tweaked, etc.). Currently, the
only way this gets captured is to {{stderr}} if the
{{SPARK_PRINT_LAUNCH_COMMAND}} environment variable is set. This is cumbersome
in the context of an application that is wrapping Spark and already using the
APIs.
I propose simply making {{SparkSubmit}} log the full command line it is about
to launch, so that clients can see it directly in their log files, rather than
having to capture and search through {{stderr}}.
was:
{{org.apache.spark.launcher.SparkLauncher}} wraps a {{ProcessBuilder}}, and
builds up a full command line to {{spark-submit}} using a builder pattern.
When {{startApplication}} is finally called, a
In scenarios where another application is submitting to Spark, it would be
extremely useful from a support and debugging standpoint to be able to see the
full {{spark-submit}} command that is actually used (so that the same
submission can be tested standalone, arguments tweaked, etc.). Currently, the
only way this gets captured is to {{stderr}} if the
{{SPARK_PRINT_LAUNCH_COMMAND}} environment variable is set. This is cumbersome
in the context of an application that is wrapping Spark and already using the
APIs.
I propose simply adding a getter method to {{SparkSubmit}} that allows clients
to retrieve what the full command line will be, so they can log this however
they wish (or do anything else with it).
> Make SparkLauncher log full spark-submit command line
> -----------------------------------------------------
>
> Key: SPARK-29070
> URL: https://issues.apache.org/jira/browse/SPARK-29070
> Project: Spark
> Issue Type: Improvement
> Components: Spark Submit
> Affects Versions: 2.4.5
> Reporter: Jeff Evans
> Priority: Minor
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> {{org.apache.spark.launcher.SparkLauncher}} wraps a {{ProcessBuilder}}, and
> builds up a full command line to {{spark-submit}} using a builder pattern.
> When {{startApplication}} is finally called, a full command line is
> materialized out of all the options, then invoked via the {{ProcessBuilder}}.
> In scenarios where another application is submitting to Spark, it would be
> extremely useful from a support and debugging standpoint to be able to see
> the full {{spark-submit}} command that is actually used (so that the same
> submission can be tested standalone, arguments tweaked, etc.). Currently,
> the only way this gets captured is to {{stderr}} if the
> {{SPARK_PRINT_LAUNCH_COMMAND}} environment variable is set. This is
> cumbersome in the context of an application that is wrapping Spark and
> already using the APIs.
> I propose simply making {{SparkSubmit}} log the full command line it is about
> to launch, so that clients can see it directly in their log files, rather
> than having to capture and search through {{stderr}}.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
