Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21895 @jerryshao that can be done but I see mainly 2 problems IIUC: - your suggestion about blacklisting has the same "caching" and "memory leakage" problems of the solution proposed here, ie. if permissions on the file are changed, we wouldn't be aware until STS is restarted and we need to store in memory the set of the files in blacklist (they may be much less than the total number of files, this is true, so probably this is not a big problem); - with your suggestion, we will also try to delete the log file IIUC, so we have to lower to debug also the log related to access denied when deleting the files (I don't think this is a big issue, but this is something which has to be taken into account, as user may miss other issues which are currently evident with the normal logging level, eg. if the file is readonly for the spark user). What do you think? Thanks.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org