Github user vanzin commented on the issue: https://github.com/apache/spark/pull/14185 I tried playing a little bit with what this API would look like, and I'm starting to question my previous idea that exposing the ProcessBuilder is the way to go here... The above issue with logging redirection is just one source of issues. There are other problems, such as the API becoming a little bit convoluted: ``` SparkLauncher launcher = ...; ProcessBuilder pb = launcher.createProcessBuilder(); launcher.startApplication(pb); ``` And all the different ways to start the Spark app (3 different methods in SparkLauncher + `ProcessBuilder.start()`). At this point I'm starting to think it might be better to mirror parts of the ProcessBuilder API that are interesting. e.g., have: ``` SparkLauncher directory(File directory) SparkLauncher redirectErrorStream(boolean redirectErrorStream) SparkLauncher redirectError(ProcessBuilder.Redirect destination) SparkLauncher redirectOutput(ProcessBuilder.Redirect destination) ``` Optionally these (since you can use `Redirect.to(File)`): ``` SparkLauncher redirectError(File destination) SparkLauncher redirectOutput(File destination) ``` And add this one which implements the current logger redirection: ``` SparkLauncher redirectToLog(String loggerName) ``` By default logging redirection would be done when using `startApplication`, using the current semantics, unless the user has overridden that by calling one of the new methods (which would also apply to `launch`). This adds more methods and is a bit more work, but it avoids certains oddities in the API, avoids overloading `startApplication`, and hides ProcessBuilder APIs we don't want to expose (like `command()`). What do you think?
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org