Jack Hu created SPARK-36252: ------------------------------- Summary: Add log files rolling policy for driver running in cluster mode with spark standalone cluster Key: SPARK-36252 URL: https://issues.apache.org/jira/browse/SPARK-36252 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 3.1.2 Reporter: Jack Hu
For a long running driver in cluster mode, there is no rolling policy, the log (stdout/stderr) may accupy lots of space, user needs a external tool to clean the old logs, it's not friendly. For executor, following 5 configurations is used to control the log file rolling policy: {code:java} spark.executor.logs.rolling.maxRetainedFiles spark.executor.logs.rolling.enableCompression spark.executor.logs.rolling.maxSize spark.executor.logs.rolling.strategy spark.executor.logs.rolling.time.interval {code} For driver running in cluster mode: 1. reuse the executor settings 2. similar to executor: add following configurations (only works for stderr/stdout for driver in cluster mode) {code:java} spark.driver.logs.rolling.maxRetainedFiles spark.driver.logs.rolling.enableCompression spark.driver.logs.rolling.maxSize spark.driver.logs.rolling.strategy spark.driver.logs.rolling.time.interval {code} #2 seems better, do you agree? -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org