BTW the SparkLauncher API has hooks to capture the stderr of the spark-submit process into the logging system of the parent process. Check the API javadocs since it's been forever since I looked at that.
On Wed, Apr 24, 2019 at 1:58 PM Marcelo Vanzin <van...@cloudera.com> wrote: > > Setting the SPARK_PRINT_LAUNCH_COMMAND env variable to 1 in the > launcher env will make Spark code print the command to stderr. Not > optimal but I think it's the only current option. > > On Wed, Apr 24, 2019 at 1:55 PM Jeff Evans > <jeffrey.wayne.ev...@gmail.com> wrote: > > > > The org.apache.spark.launcher.SparkLauncher is used to construct a > > spark-submit invocation programmatically, via a builder pattern. In > > our application, which uses a SparkLauncher internally, I would like > > to log the full spark-submit command that it will invoke to our log > > file, in order to aid in debugging/support. However, I can't figure > > out a way to do this. This snippet would work, except for the fact > > that the createBuilder method is private. > > > > sparkLauncher.createBuilder().command() > > > > Is there an alternate way of doing this? The Spark version is > > 2.11:2.4.0. Thanks. > > > > --------------------------------------------------------------------- > > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > > > > -- > Marcelo -- Marcelo --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
