[jira] [Updated] (SPARK-39324) Log ExecutorDecommission as INFO level in TaskSchedulerImpl

2022-12-16 Thread Dongjoon Hyun (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-39324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dongjoon Hyun updated SPARK-39324:
--
Parent Issue: SPARK-41550  (was: SPARK-41515)

> Log ExecutorDecommission as INFO level in TaskSchedulerImpl
> ---
>
> Key: SPARK-39324
> URL: https://issues.apache.org/jira/browse/SPARK-39324
> Project: Spark
>  Issue Type: Sub-task
>  Components: Spark Core
>Affects Versions: 3.4.0
>Reporter: Dongjoon Hyun
>Assignee: Dongjoon Hyun
>Priority: Major
> Fix For: 3.4.0
>
>
> Like the other module, `TaskSchedulerImpl` should log the decommission as 
> `INFO` level.
> {code}
> 22/05/28 01:25:28 INFO KubernetesClusterSchedulerBackend: Decommission 
> executors: 8
> 22/05/28 01:25:28 INFO KubernetesClusterSchedulerBackend: Notify executor 8 
> to decommissioning.
> 22/05/28 01:25:28 INFO BlockManagerMasterEndpoint: Mark BlockManagers 
> (BlockManagerId(8, 100.103.40.13, 43353, None)) as being decommissioning.
> 22/05/28 01:25:29 ERROR TaskSchedulerImpl: Lost executor 8 on 100.103.40.13: 
> Executor decommission.
> 22/05/28 01:25:29 INFO ExecutorMonitor: Executor 8 is removed. Remove reason 
> statistics: ...
> 22/05/28 01:25:29 INFO DAGScheduler: Executor lost: 8 (epoch 7)
> 22/05/28 01:25:29 INFO BlockManagerMasterEndpoint: Trying to remove executor 
> 8 from BlockManagerMaster.
> 22/05/28 01:25:29 INFO BlockManagerMasterEndpoint: Removing block manager 
> BlockManagerId(8, 100.103.40.13, 43353, None)
> 22/05/28 01:25:29 INFO BlockManagerMaster: Removed 8 successfully in 
> removeExecutor
> 22/05/28 01:25:29 INFO DAGScheduler: Shuffle files lost for executor: 8 
> (epoch 7)
> 22/05/28 01:25:34 INFO BlockManagerMaster: Removal of executor 8 requested
> 22/05/28 01:25:34 INFO BlockManagerMasterEndpoint: Trying to remove executor 
> 8 from BlockManagerMaster.
> 22/05/28 01:25:34 INFO 
> KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asked to remove 
> non-existent executor 8
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-39324) Log ExecutorDecommission as INFO level in TaskSchedulerImpl

2022-12-13 Thread Dongjoon Hyun (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-39324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dongjoon Hyun updated SPARK-39324:
--
Parent: SPARK-41515
Issue Type: Sub-task  (was: Improvement)

> Log ExecutorDecommission as INFO level in TaskSchedulerImpl
> ---
>
> Key: SPARK-39324
> URL: https://issues.apache.org/jira/browse/SPARK-39324
> Project: Spark
>  Issue Type: Sub-task
>  Components: Spark Core
>Affects Versions: 3.4.0
>Reporter: Dongjoon Hyun
>Assignee: Dongjoon Hyun
>Priority: Major
> Fix For: 3.4.0
>
>
> Like the other module, `TaskSchedulerImpl` should log the decommission as 
> `INFO` level.
> {code}
> 22/05/28 01:25:28 INFO KubernetesClusterSchedulerBackend: Decommission 
> executors: 8
> 22/05/28 01:25:28 INFO KubernetesClusterSchedulerBackend: Notify executor 8 
> to decommissioning.
> 22/05/28 01:25:28 INFO BlockManagerMasterEndpoint: Mark BlockManagers 
> (BlockManagerId(8, 100.103.40.13, 43353, None)) as being decommissioning.
> 22/05/28 01:25:29 ERROR TaskSchedulerImpl: Lost executor 8 on 100.103.40.13: 
> Executor decommission.
> 22/05/28 01:25:29 INFO ExecutorMonitor: Executor 8 is removed. Remove reason 
> statistics: ...
> 22/05/28 01:25:29 INFO DAGScheduler: Executor lost: 8 (epoch 7)
> 22/05/28 01:25:29 INFO BlockManagerMasterEndpoint: Trying to remove executor 
> 8 from BlockManagerMaster.
> 22/05/28 01:25:29 INFO BlockManagerMasterEndpoint: Removing block manager 
> BlockManagerId(8, 100.103.40.13, 43353, None)
> 22/05/28 01:25:29 INFO BlockManagerMaster: Removed 8 successfully in 
> removeExecutor
> 22/05/28 01:25:29 INFO DAGScheduler: Shuffle files lost for executor: 8 
> (epoch 7)
> 22/05/28 01:25:34 INFO BlockManagerMaster: Removal of executor 8 requested
> 22/05/28 01:25:34 INFO BlockManagerMasterEndpoint: Trying to remove executor 
> 8 from BlockManagerMaster.
> 22/05/28 01:25:34 INFO 
> KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asked to remove 
> non-existent executor 8
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-39324) Log ExecutorDecommission as INFO level in TaskSchedulerImpl

2022-05-27 Thread Dongjoon Hyun (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-39324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dongjoon Hyun updated SPARK-39324:
--
Description: 
Like the other module, `TaskSchedulerImpl` should log the decommission as 
`INFO` level.
{code}
22/05/28 01:25:28 INFO KubernetesClusterSchedulerBackend: Decommission 
executors: 8
22/05/28 01:25:28 INFO KubernetesClusterSchedulerBackend: Notify executor 8 to 
decommissioning.
22/05/28 01:25:28 INFO BlockManagerMasterEndpoint: Mark BlockManagers 
(BlockManagerId(8, 100.103.40.13, 43353, None)) as being decommissioning.
22/05/28 01:25:29 ERROR TaskSchedulerImpl: Lost executor 8 on 100.103.40.13: 
Executor decommission.
22/05/28 01:25:29 INFO ExecutorMonitor: Executor 8 is removed. Remove reason 
statistics: ...
22/05/28 01:25:29 INFO DAGScheduler: Executor lost: 8 (epoch 7)
22/05/28 01:25:29 INFO BlockManagerMasterEndpoint: Trying to remove executor 8 
from BlockManagerMaster.
22/05/28 01:25:29 INFO BlockManagerMasterEndpoint: Removing block manager 
BlockManagerId(8, 100.103.40.13, 43353, None)
22/05/28 01:25:29 INFO BlockManagerMaster: Removed 8 successfully in 
removeExecutor
22/05/28 01:25:29 INFO DAGScheduler: Shuffle files lost for executor: 8 (epoch 
7)
22/05/28 01:25:34 INFO BlockManagerMaster: Removal of executor 8 requested
22/05/28 01:25:34 INFO BlockManagerMasterEndpoint: Trying to remove executor 8 
from BlockManagerMaster.
22/05/28 01:25:34 INFO 
KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asked to remove 
non-existent executor 8
{code}

  was:
{code}
22/05/28 01:25:28 INFO KubernetesClusterSchedulerBackend: Decommission 
executors: 8
22/05/28 01:25:28 INFO KubernetesClusterSchedulerBackend: Notify executor 8 to 
decommissioning.
22/05/28 01:25:28 INFO BlockManagerMasterEndpoint: Mark BlockManagers 
(BlockManagerId(8, 100.103.40.13, 43353, None)) as being decommissioning.
22/05/28 01:25:29 ERROR TaskSchedulerImpl: Lost executor 8 on 100.103.40.13: 
Executor decommission.
22/05/28 01:25:29 INFO ExecutorMonitor: Executor 8 is removed. Remove reason 
statistics: ...
22/05/28 01:25:29 INFO DAGScheduler: Executor lost: 8 (epoch 7)
22/05/28 01:25:29 INFO BlockManagerMasterEndpoint: Trying to remove executor 8 
from BlockManagerMaster.
22/05/28 01:25:29 INFO BlockManagerMasterEndpoint: Removing block manager 
BlockManagerId(8, 100.103.40.13, 43353, None)
22/05/28 01:25:29 INFO BlockManagerMaster: Removed 8 successfully in 
removeExecutor
22/05/28 01:25:29 INFO DAGScheduler: Shuffle files lost for executor: 8 (epoch 
7)
22/05/28 01:25:34 INFO BlockManagerMaster: Removal of executor 8 requested
22/05/28 01:25:34 INFO BlockManagerMasterEndpoint: Trying to remove executor 8 
from BlockManagerMaster.
22/05/28 01:25:34 INFO 
KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asked to remove 
non-existent executor 8
{code}


> Log ExecutorDecommission as INFO level in TaskSchedulerImpl
> ---
>
> Key: SPARK-39324
> URL: https://issues.apache.org/jira/browse/SPARK-39324
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core
>Affects Versions: 3.4.0
>Reporter: Dongjoon Hyun
>Priority: Major
>
> Like the other module, `TaskSchedulerImpl` should log the decommission as 
> `INFO` level.
> {code}
> 22/05/28 01:25:28 INFO KubernetesClusterSchedulerBackend: Decommission 
> executors: 8
> 22/05/28 01:25:28 INFO KubernetesClusterSchedulerBackend: Notify executor 8 
> to decommissioning.
> 22/05/28 01:25:28 INFO BlockManagerMasterEndpoint: Mark BlockManagers 
> (BlockManagerId(8, 100.103.40.13, 43353, None)) as being decommissioning.
> 22/05/28 01:25:29 ERROR TaskSchedulerImpl: Lost executor 8 on 100.103.40.13: 
> Executor decommission.
> 22/05/28 01:25:29 INFO ExecutorMonitor: Executor 8 is removed. Remove reason 
> statistics: ...
> 22/05/28 01:25:29 INFO DAGScheduler: Executor lost: 8 (epoch 7)
> 22/05/28 01:25:29 INFO BlockManagerMasterEndpoint: Trying to remove executor 
> 8 from BlockManagerMaster.
> 22/05/28 01:25:29 INFO BlockManagerMasterEndpoint: Removing block manager 
> BlockManagerId(8, 100.103.40.13, 43353, None)
> 22/05/28 01:25:29 INFO BlockManagerMaster: Removed 8 successfully in 
> removeExecutor
> 22/05/28 01:25:29 INFO DAGScheduler: Shuffle files lost for executor: 8 
> (epoch 7)
> 22/05/28 01:25:34 INFO BlockManagerMaster: Removal of executor 8 requested
> 22/05/28 01:25:34 INFO BlockManagerMasterEndpoint: Trying to remove executor 
> 8 from BlockManagerMaster.
> 22/05/28 01:25:34 INFO 
> KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asked to remove 
> non-existent executor 8
> {code}



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional