Thanks for the pointers. We figured out the stdout/stderr capture
piece. I was just looking to capture the full command in order to
help debug issues we run into with the submit depending on various
combinations of all parameters/classpath, and also to isolate job
specific issues from our
BTW the SparkLauncher API has hooks to capture the stderr of the
spark-submit process into the logging system of the parent process.
Check the API javadocs since it's been forever since I looked at that.
On Wed, Apr 24, 2019 at 1:58 PM Marcelo Vanzin wrote:
>
> Setting the
You could set the env var SPARK_PRINT_LAUNCH_COMMAND and spark-submit
will print it, but it will be printed by the subprocess and not yours
unless you redirect the stdout
Also the command is what spark-submit generates, so it is quite more
verbose and includes the classpath etc.
I think the
Setting the SPARK_PRINT_LAUNCH_COMMAND env variable to 1 in the
launcher env will make Spark code print the command to stderr. Not
optimal but I think it's the only current option.
On Wed, Apr 24, 2019 at 1:55 PM Jeff Evans
wrote:
>
> The org.apache.spark.launcher.SparkLauncher is used to
