[ https://issues.apache.org/jira/browse/SPARK-31346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jackey Lee updated SPARK-31346: ------------------------------- Description: In InsertIntoHiveTable and InsertIntoHiveDirCommand, we use deleteExternalTmpPath to clean temporary directories after Job committed and cancel deleteOnExit if succeeded. But sometimes (e.g., when speculative task is enabled), temporary directories may be left uncleaned. This is happened if there are still some tasks running after we called deleteExternalTmpPath. Thus it maybe necessary to keep deleteOnExit, even if temporary directory has already deleted, to make sure the temporary directories cleaned. (was: In InsertIntoHiveTable and InsertIntoHiveDirCommand, we use deleteExternalTmpPath to clean temporary directories after Job committed and cancel deleteOnExit if succeeded. But sometimes (e.g., when speculative task is enabled), temporary directories may be left uncleaned. This is happened if there are still some tasks running after we called deleteExternalTmpPath. Thus it maybe necessary to keep deleteOnExit, even if temporary directory has already deleted, to verify the temporary directories cleaned.) Summary: Add new configuration to make sure temporary directory cleaned (was: Add new configuration to verify temporary directory cleaned) > Add new configuration to make sure temporary directory cleaned > -------------------------------------------------------------- > > Key: SPARK-31346 > URL: https://issues.apache.org/jira/browse/SPARK-31346 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.1.0 > Reporter: Jackey Lee > Priority: Major > > In InsertIntoHiveTable and InsertIntoHiveDirCommand, we use > deleteExternalTmpPath to clean temporary directories after Job committed and > cancel deleteOnExit if succeeded. But sometimes (e.g., when speculative task > is enabled), temporary directories may be left uncleaned. This is happened if > there are still some tasks running after we called deleteExternalTmpPath. > Thus it maybe necessary to keep deleteOnExit, even if temporary directory has > already deleted, to make sure the temporary directories cleaned. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org