[ https://issues.apache.org/jira/browse/SPARK-37373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-37373: ------------------------------------ Assignee: Attila Zsolt Piros (was: Apache Spark) > Collect LocalSparkContext worker logs in case of test failure > ------------------------------------------------------------- > > Key: SPARK-37373 > URL: https://issues.apache.org/jira/browse/SPARK-37373 > Project: Spark > Issue Type: Improvement > Components: Tests > Affects Versions: 3.2.0 > Reporter: Attila Zsolt Piros > Assignee: Attila Zsolt Piros > Priority: Major > > About 50 test suites are using LocalSparkContext by specifying > "local-cluster" as the cluster URL. In this case executor logs will be under > the worker dir which is a temp directory and as such will be deleted at > shutdown (for details see > https://github.com/apache/spark/blob/0a4961df29aab6912492e87e4e719865fe20d981/core/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala#L70) > So when a test fails and the error was on the executor side the log will be > lost. > This is only for local cluster tests and not for standalone tests where logs > will be kept in the "<SPARK_HOME>/work". -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org