Github user ankuriitg commented on the issue:
https://github.com/apache/spark/pull/22504
Thanks for your comment @tgravescs
Moving the file to yarn logs app dir is optional and users can choose to
disable that. In my case, If they only enable syncToHdfs, the file will be
moved to the configured hdfs dir.
Regarding cleaning, I have another PR ready, which adds to the existing
logic of Spark History Server cleaner and cleans these driver logs as well. It
will use the existing max age and polling interval configured for event logs.
(I can merge that code in this PR as well)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
