[jira] [Commented] (SPARK-42466) spark.kubernetes.file.upload.path not deleting files under HDFS after job completes
[ https://issues.apache.org/jira/browse/SPARK-42466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17692340#comment-17692340 ] Apache Spark commented on SPARK-42466: -- User 'shrprasa' has created a pull request for this issue: https://github.com/apache/spark/pull/40128 > spark.kubernetes.file.upload.path not deleting files under HDFS after job > completes > --- > > Key: SPARK-42466 > URL: https://issues.apache.org/jira/browse/SPARK-42466 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 3.2.0 >Reporter: Jagadeeswara Rao >Priority: Major > > In cluster mode after uploading files to HDFS location using > spark.kubernetes.file.upload.path property files are not getting cleared . > File is successfully uploaded to hdfs location in this format > spark-upload-[randomUUID] using {{KubernetesUtils}} is requested to > uploadFileUri . > [https://github.com/apache/spark/blob/76a134ade60a9f354aca01eaca0b2e2477c6bd43/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala#L310] > following is driver log , driver is completed successfully and shutdownhook > is not cleared the hdfs files. > {code:java} > 23/02/16 18:06:56 INFO KubernetesClusterSchedulerBackend: Shutting down all > executors > 23/02/16 18:06:56 INFO > KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each > executor to shut down > 23/02/16 18:06:56 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has > been closed. > 23/02/16 18:06:57 INFO MapOutputTrackerMasterEndpoint: > MapOutputTrackerMasterEndpoint stopped! > 23/02/16 18:06:57 INFO MemoryStore: MemoryStore cleared > 23/02/16 18:06:57 INFO BlockManager: BlockManager stopped > 23/02/16 18:06:57 INFO BlockManagerMaster: BlockManagerMaster stopped > 23/02/16 18:06:57 INFO > OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: > OutputCommitCoordinator stopped! > 23/02/16 18:06:57 INFO SparkContext: Successfully stopped SparkContext > 23/02/16 18:06:57 INFO ShutdownHookManager: Shutdown hook called > 23/02/16 18:06:57 INFO ShutdownHookManager: Deleting directory > /tmp/spark-efb8f725-4ead-4729-a8e0-f478280121b7 > 23/02/16 18:06:57 INFO ShutdownHookManager: Deleting directory > /spark-local2/spark-66dbf7e6-fe7e-4655-8724-69d76d93fc1f > 23/02/16 18:06:57 INFO ShutdownHookManager: Deleting directory > /spark-local1/spark-53aefaee-58a5-4fce-b5b0-5e29f42e337f{code} > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-42466) spark.kubernetes.file.upload.path not deleting files under HDFS after job completes
[ https://issues.apache.org/jira/browse/SPARK-42466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17692339#comment-17692339 ] Apache Spark commented on SPARK-42466: -- User 'shrprasa' has created a pull request for this issue: https://github.com/apache/spark/pull/40128 > spark.kubernetes.file.upload.path not deleting files under HDFS after job > completes > --- > > Key: SPARK-42466 > URL: https://issues.apache.org/jira/browse/SPARK-42466 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 3.2.0 >Reporter: Jagadeeswara Rao >Priority: Major > > In cluster mode after uploading files to HDFS location using > spark.kubernetes.file.upload.path property files are not getting cleared . > File is successfully uploaded to hdfs location in this format > spark-upload-[randomUUID] using {{KubernetesUtils}} is requested to > uploadFileUri . > [https://github.com/apache/spark/blob/76a134ade60a9f354aca01eaca0b2e2477c6bd43/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala#L310] > following is driver log , driver is completed successfully and shutdownhook > is not cleared the hdfs files. > {code:java} > 23/02/16 18:06:56 INFO KubernetesClusterSchedulerBackend: Shutting down all > executors > 23/02/16 18:06:56 INFO > KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each > executor to shut down > 23/02/16 18:06:56 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has > been closed. > 23/02/16 18:06:57 INFO MapOutputTrackerMasterEndpoint: > MapOutputTrackerMasterEndpoint stopped! > 23/02/16 18:06:57 INFO MemoryStore: MemoryStore cleared > 23/02/16 18:06:57 INFO BlockManager: BlockManager stopped > 23/02/16 18:06:57 INFO BlockManagerMaster: BlockManagerMaster stopped > 23/02/16 18:06:57 INFO > OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: > OutputCommitCoordinator stopped! > 23/02/16 18:06:57 INFO SparkContext: Successfully stopped SparkContext > 23/02/16 18:06:57 INFO ShutdownHookManager: Shutdown hook called > 23/02/16 18:06:57 INFO ShutdownHookManager: Deleting directory > /tmp/spark-efb8f725-4ead-4729-a8e0-f478280121b7 > 23/02/16 18:06:57 INFO ShutdownHookManager: Deleting directory > /spark-local2/spark-66dbf7e6-fe7e-4655-8724-69d76d93fc1f > 23/02/16 18:06:57 INFO ShutdownHookManager: Deleting directory > /spark-local1/spark-53aefaee-58a5-4fce-b5b0-5e29f42e337f{code} > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-42466) spark.kubernetes.file.upload.path not deleting files under HDFS after job completes
[ https://issues.apache.org/jira/browse/SPARK-42466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17690381#comment-17690381 ] Shrikant Prasad commented on SPARK-42466: - working on the fix. > spark.kubernetes.file.upload.path not deleting files under HDFS after job > completes > --- > > Key: SPARK-42466 > URL: https://issues.apache.org/jira/browse/SPARK-42466 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 3.2.0 >Reporter: Jagadeeswara Rao >Priority: Major > > In cluster mode after uploading files to HDFS location using > spark.kubernetes.file.upload.path property files are not getting cleared . > File is successfully uploaded to hdfs location in this format > spark-upload-[randomUUID] using {{KubernetesUtils}} is requested to > uploadFileUri . > [https://github.com/apache/spark/blob/76a134ade60a9f354aca01eaca0b2e2477c6bd43/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala#L310] > following is driver log , driver is completed successfully and shutdownhook > is not cleared the hdfs files. > {code:java} > 23/02/16 18:06:56 INFO KubernetesClusterSchedulerBackend: Shutting down all > executors > 23/02/16 18:06:56 INFO > KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each > executor to shut down > 23/02/16 18:06:56 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has > been closed. > 23/02/16 18:06:57 INFO MapOutputTrackerMasterEndpoint: > MapOutputTrackerMasterEndpoint stopped! > 23/02/16 18:06:57 INFO MemoryStore: MemoryStore cleared > 23/02/16 18:06:57 INFO BlockManager: BlockManager stopped > 23/02/16 18:06:57 INFO BlockManagerMaster: BlockManagerMaster stopped > 23/02/16 18:06:57 INFO > OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: > OutputCommitCoordinator stopped! > 23/02/16 18:06:57 INFO SparkContext: Successfully stopped SparkContext > 23/02/16 18:06:57 INFO ShutdownHookManager: Shutdown hook called > 23/02/16 18:06:57 INFO ShutdownHookManager: Deleting directory > /tmp/spark-efb8f725-4ead-4729-a8e0-f478280121b7 > 23/02/16 18:06:57 INFO ShutdownHookManager: Deleting directory > /spark-local2/spark-66dbf7e6-fe7e-4655-8724-69d76d93fc1f > 23/02/16 18:06:57 INFO ShutdownHookManager: Deleting directory > /spark-local1/spark-53aefaee-58a5-4fce-b5b0-5e29f42e337f{code} > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org