june109109 commented on issue #50587:
URL: https://github.com/apache/airflow/issues/50587#issuecomment-3031420054

   @ellisms 
   I'm sorry for late response.
   
   ```
   ~ $ aws emr describe-notebook-execution --notebook-execution-id 
ex-J0BYN0I0N06S4UB8DAAFKBJXKKWO5
   {
       "NotebookExecution": {
           "NotebookExecutionId": "ex-J0BYN0I0N06S4UB8DAAFKBJXKKWO5",
           "EditorId": "e-AH8OW5O68ZRXGXYFFLF14HIAG",
           "ExecutionEngine": {
               "Id": "j-19O4HGHRC48IE",
               "Type": "EMR",
               "MasterInstanceSecurityGroupId": "sg-0c2a01931b0a299fd"
           },
           "NotebookExecutionName": "user_operator_test-user_nb",
           "NotebookParams": "{\"query\": \"SELECT * FROM 
tos.tcic_nrtm_integ_cont_prod WHERE strd_dt = '250702'\", \"key2\": 123, 
\"default_datestrings\": {\"yyyymmdd\": \"20250703\", \"yyyy-mm-dd\": 
\"2025-07-03\"}}",
           "Status": "FINISHED",
           "StartTime": "2025-07-03T08:09:36.552000+00:00",
           "EndTime": "2025-07-03T08:10:52.757000+00:00",
           "Arn": 
"arn:aws:elasticmapreduce:ap-northeast-2:302263049595:notebook-execution/ex-J0BYN0I0N06S4UB8DAAFKBJXKKWO5",
           "OutputNotebookURI": 
"s3://cosmos-computing-test-emr-20250609234141565800000008/poc/e-AH8OW5O68ZRXGXYFFLF14HIAG/executions/ex-J0BYN0I0N06S4UB8DAAFKBJXKKWO5/query.ipynb",
           "LastStateChangeReason": "Execution is finished for cluster 
j-19O4HGHRC48IE.",
           "NotebookInstanceSecurityGroupId": "",
           "Tags": [],
           "OutputNotebookS3Location": {
               "Bucket": "cosmos-computing-test-emr-20250609234141565800000008",
               "Key": 
"poc/e-AH8OW5O68ZRXGXYFFLF14HIAG/executions/ex-J0BYN0I0N06S4UB8DAAFKBJXKKWO5/query.ipynb"
           },
           "OutputNotebookFormat": "null"
       }
   }
   
   ```
   
   It seems issue occurs because the EMR waiter's internal argument path for 
status is set to "notebookexecution.status",
   but the actual response from describe_notebook_execution uses 
"NotebookExecution.Status" (with upper case letter). The waiter may looks for 
response["notebookexecution"]["status"], but actual response is 
response["NotebookExecution"]["Status"], so waits while notebook is finished. 
   
   I tried custom waiter definition and found out it works well with my case. 
   
   ```
   NOTEBOOK_WAITER_MODEL = {
       "version": 2,
       "waiters": {
           "notebook_execution_terminal": {
               "operation": "DescribeNotebookExecution",
               "delay": 30,
               "maxAttempts": 120,
               "acceptors": [
                   {
                       "matcher": "path",
                       "argument": "NotebookExecution.Status",
                       "expected": "FINISHED",
                       "state": "success"
                   },
                   {
                       "matcher": "path",
                       "argument": "NotebookExecution.Status",
                       "expected": "FAILED",
                       "state": "failure"
                   },
   
               ]
           }
       }
   }
   
       emr_hook = EmrHook(aws_conn_id=self.aws_conn_id)
       emr_client = emr_hook.get_conn()
       waiter = BaseBotoWaiter(emr_client, 
NOTEBOOK_WAITER_MODEL).waiter("notebook_execution_terminal")
       waiter.wait(
           NotebookExecutionId=notebook_execution_id,
           WaiterConfig=prune_dict({
               "Delay": self.waiter_delay,
               "MaxAttempts": self.waiter_max_attempts,
           }),
       )
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to