[jira] [Commented] (AIRFLOW-3320) Sagemaker operator never ends when having "Stopped" status
[
https://issues.apache.org/jira/browse/AIRFLOW-3320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681516#comment-16681516
]
John Cheng commented on AIRFLOW-3320:
-
Oh, I didn't notice. Should be fixed after studying the latest code.
> Sagemaker operator never ends when having "Stopped" status
> --
>
> Key: AIRFLOW-3320
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3320
> Project: Apache Airflow
> Issue Type: Bug
> Components: aws
>Reporter: John Cheng
>Assignee: Yang Yu
>Priority: Minor
>
> Sagemaker operator never ends when having "Stopped" status.
> {code:java}
> [2018-11-08 20:15:27,864] {logging_mixin.py:95} INFO - [2018-11-08
> 20:15:27,864] {sagemaker_hook.py:129} INFO - Job still running for 2690
> seconds... current status is InProgress
> [2018-11-08 20:15:32,917] {logging_mixin.py:95} INFO - [2018-11-08
> 20:15:32,917] {sagemaker_hook.py:129} INFO - Job still running for 2695
> seconds... current status is Stopping
> [2018-11-08 20:15:37,963] {logging_mixin.py:95} INFO - [2018-11-08
> 20:15:37,963] {sagemaker_hook.py:129} INFO - Job still running for 2700
> seconds... current status is Stopping
> [2018-11-08 20:15:43,012] {logging_mixin.py:95} INFO - [2018-11-08
> 20:15:43,012] {sagemaker_hook.py:129} INFO - Job still running for 2705
> seconds... current status is Stopping
> [2018-11-08 20:15:48,060] {logging_mixin.py:95} INFO - [2018-11-08
> 20:15:48,060] {sagemaker_hook.py:129} INFO - Job still running for 2710
> seconds... current status is Stopping
> [2018-11-08 20:15:53,108] {logging_mixin.py:95} INFO - [2018-11-08
> 20:15:53,108] {sagemaker_hook.py:129} INFO - Job still running for 2715
> seconds... current status is Stopping
> [2018-11-08 20:15:58,153] {logging_mixin.py:95} INFO - [2018-11-08
> 20:15:58,153] {sagemaker_hook.py:129} INFO - Job still running for 2720
> seconds... current status is Stopping
> [2018-11-08 20:16:03,206] {logging_mixin.py:95} INFO - [2018-11-08
> 20:16:03,206] {sagemaker_hook.py:129} INFO - Job still running for 2725
> seconds... current status is Stopping
> [2018-11-08 20:16:08,255] {logging_mixin.py:95} INFO - [2018-11-08
> 20:16:08,255] {sagemaker_hook.py:129} INFO - Job still running for 2730
> seconds... current status is Stopping
> [2018-11-08 20:16:13,298] {logging_mixin.py:95} INFO - [2018-11-08
> 20:16:13,298] {sagemaker_hook.py:129} INFO - Job still running for 2735
> seconds... current status is Stopping
> [2018-11-08 20:16:18,349] {logging_mixin.py:95} INFO - [2018-11-08
> 20:16:18,349] {sagemaker_hook.py:129} INFO - Job still running for 2740
> seconds... current status is Stopping
> [2018-11-08 20:16:23,398] {logging_mixin.py:95} INFO - [2018-11-08
> 20:16:23,398] {sagemaker_hook.py:129} INFO - Job still running for 2745
> seconds... current status is Stopping
> [2018-11-08 20:16:28,445] {logging_mixin.py:95} INFO - [2018-11-08
> 20:16:28,445] {sagemaker_hook.py:129} INFO - Job still running for 2750
> seconds... current status is Stopping
> [2018-11-08 20:16:33,497] {logging_mixin.py:95} INFO - [2018-11-08
> 20:16:33,497] {sagemaker_hook.py:129} INFO - Job still running for 2755
> seconds... current status is Stopping
> [2018-11-08 20:16:38,545] {logging_mixin.py:95} INFO - [2018-11-08
> 20:16:38,545] {sagemaker_hook.py:129} INFO - Job still running for 2760
> seconds... current status is Stopping
> [2018-11-08 20:16:43,593] {logging_mixin.py:95} INFO - [2018-11-08
> 20:16:43,593] {sagemaker_hook.py:129} INFO - Job still running for 2765
> seconds... current status is Stopping
> [2018-11-08 20:16:48,639] {logging_mixin.py:95} INFO - [2018-11-08
> 20:16:48,639] {sagemaker_hook.py:129} INFO - Job still running for 2770
> seconds... current status is Stopping
> [2018-11-08 20:16:53,724] {logging_mixin.py:95} INFO - [2018-11-08
> 20:16:53,724] {sagemaker_hook.py:129} INFO - Job still running for 2775
> seconds... current status is Stopping
> [2018-11-08 20:16:58,773] {logging_mixin.py:95} INFO - [2018-11-08
> 20:16:58,773] {sagemaker_hook.py:129} INFO - Job still running for 2780
> seconds... current status is Stopping
> [2018-11-08 20:17:03,835] {logging_mixin.py:95} INFO - [2018-11-08
> 20:17:03,835] {sagemaker_hook.py:129} INFO - Job still running for 2785
> seconds... current status is Stopping
> [2018-11-08 20:17:08,880] {logging_mixin.py:95} INFO - [2018-11-08
> 20:17:08,880] {sagemaker_hook.py:129} INFO - Job still running for 2790
> seconds... current status is Stopping
> [2018-11-08 20:17:13,931] {logging_mixin.py:95} INFO - [2018-11-08
> 20:17:13,931] {sagemaker_hook.py:129} INFO - Job still running for 2795
> seconds... current status is Stopping
> [2018-11-08 20:17:18,976] {logging_mixin.py:95} INFO - [2018-11-08
> 20:17:18,976] {sagemaker_hook.py:129} INFO - Job
[jira] [Commented] (AIRFLOW-3320) Sagemaker operator never ends when having "Stopped" status
[
https://issues.apache.org/jira/browse/AIRFLOW-3320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681206#comment-16681206
]
Ash Berlin-Taylor commented on AIRFLOW-3320:
Are you running on master? If so can you check again with the latest version?
We've just merged a big couple of PRs that totally reworks the SageMaker code
and hopefully this is fixed.
> Sagemaker operator never ends when having "Stopped" status
> --
>
> Key: AIRFLOW-3320
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3320
> Project: Apache Airflow
> Issue Type: Bug
> Components: aws
>Reporter: John Cheng
>Assignee: Yang Yu
>Priority: Minor
>
> Sagemaker operator never ends when having "Stopped" status.
> {code:java}
> [2018-11-08 20:15:27,864] {logging_mixin.py:95} INFO - [2018-11-08
> 20:15:27,864] {sagemaker_hook.py:129} INFO - Job still running for 2690
> seconds... current status is InProgress
> [2018-11-08 20:15:32,917] {logging_mixin.py:95} INFO - [2018-11-08
> 20:15:32,917] {sagemaker_hook.py:129} INFO - Job still running for 2695
> seconds... current status is Stopping
> [2018-11-08 20:15:37,963] {logging_mixin.py:95} INFO - [2018-11-08
> 20:15:37,963] {sagemaker_hook.py:129} INFO - Job still running for 2700
> seconds... current status is Stopping
> [2018-11-08 20:15:43,012] {logging_mixin.py:95} INFO - [2018-11-08
> 20:15:43,012] {sagemaker_hook.py:129} INFO - Job still running for 2705
> seconds... current status is Stopping
> [2018-11-08 20:15:48,060] {logging_mixin.py:95} INFO - [2018-11-08
> 20:15:48,060] {sagemaker_hook.py:129} INFO - Job still running for 2710
> seconds... current status is Stopping
> [2018-11-08 20:15:53,108] {logging_mixin.py:95} INFO - [2018-11-08
> 20:15:53,108] {sagemaker_hook.py:129} INFO - Job still running for 2715
> seconds... current status is Stopping
> [2018-11-08 20:15:58,153] {logging_mixin.py:95} INFO - [2018-11-08
> 20:15:58,153] {sagemaker_hook.py:129} INFO - Job still running for 2720
> seconds... current status is Stopping
> [2018-11-08 20:16:03,206] {logging_mixin.py:95} INFO - [2018-11-08
> 20:16:03,206] {sagemaker_hook.py:129} INFO - Job still running for 2725
> seconds... current status is Stopping
> [2018-11-08 20:16:08,255] {logging_mixin.py:95} INFO - [2018-11-08
> 20:16:08,255] {sagemaker_hook.py:129} INFO - Job still running for 2730
> seconds... current status is Stopping
> [2018-11-08 20:16:13,298] {logging_mixin.py:95} INFO - [2018-11-08
> 20:16:13,298] {sagemaker_hook.py:129} INFO - Job still running for 2735
> seconds... current status is Stopping
> [2018-11-08 20:16:18,349] {logging_mixin.py:95} INFO - [2018-11-08
> 20:16:18,349] {sagemaker_hook.py:129} INFO - Job still running for 2740
> seconds... current status is Stopping
> [2018-11-08 20:16:23,398] {logging_mixin.py:95} INFO - [2018-11-08
> 20:16:23,398] {sagemaker_hook.py:129} INFO - Job still running for 2745
> seconds... current status is Stopping
> [2018-11-08 20:16:28,445] {logging_mixin.py:95} INFO - [2018-11-08
> 20:16:28,445] {sagemaker_hook.py:129} INFO - Job still running for 2750
> seconds... current status is Stopping
> [2018-11-08 20:16:33,497] {logging_mixin.py:95} INFO - [2018-11-08
> 20:16:33,497] {sagemaker_hook.py:129} INFO - Job still running for 2755
> seconds... current status is Stopping
> [2018-11-08 20:16:38,545] {logging_mixin.py:95} INFO - [2018-11-08
> 20:16:38,545] {sagemaker_hook.py:129} INFO - Job still running for 2760
> seconds... current status is Stopping
> [2018-11-08 20:16:43,593] {logging_mixin.py:95} INFO - [2018-11-08
> 20:16:43,593] {sagemaker_hook.py:129} INFO - Job still running for 2765
> seconds... current status is Stopping
> [2018-11-08 20:16:48,639] {logging_mixin.py:95} INFO - [2018-11-08
> 20:16:48,639] {sagemaker_hook.py:129} INFO - Job still running for 2770
> seconds... current status is Stopping
> [2018-11-08 20:16:53,724] {logging_mixin.py:95} INFO - [2018-11-08
> 20:16:53,724] {sagemaker_hook.py:129} INFO - Job still running for 2775
> seconds... current status is Stopping
> [2018-11-08 20:16:58,773] {logging_mixin.py:95} INFO - [2018-11-08
> 20:16:58,773] {sagemaker_hook.py:129} INFO - Job still running for 2780
> seconds... current status is Stopping
> [2018-11-08 20:17:03,835] {logging_mixin.py:95} INFO - [2018-11-08
> 20:17:03,835] {sagemaker_hook.py:129} INFO - Job still running for 2785
> seconds... current status is Stopping
> [2018-11-08 20:17:08,880] {logging_mixin.py:95} INFO - [2018-11-08
> 20:17:08,880] {sagemaker_hook.py:129} INFO - Job still running for 2790
> seconds... current status is Stopping
> [2018-11-08 20:17:13,931] {logging_mixin.py:95} INFO - [2018-11-08
> 20:17:13,931] {sagemaker_hook.py:129} INFO - Job still running for 2795
> seconds... current status
