bryanyang0528 commented on a change in pull request #4919: [AIRFLOW-4093] Throw exception if job failed or cancelled or retry too many times URL: https://github.com/apache/airflow/pull/4919#discussion_r267341823
########## File path: airflow/contrib/operators/aws_athena_operator.py ########## @@ -74,7 +76,16 @@ def execute(self, context): self.result_configuration['OutputLocation'] = self.output_location self.query_execution_id = self.hook.run_query(self.query, self.query_execution_context, self.result_configuration, self.client_request_token) - self.hook.poll_query_status(self.query_execution_id) + query_status = self.hook.poll_query_status(self.query_execution_id, self.max_tries) + + if not query_status or query_status in AWSAthenaHook.FAILURE_STATES: + raise Exception( + 'Athena job failed. Final state is {}, query_execution_id is {}.' + .format(query_status, self.query_execution_id)) + elif query_status in AWSAthenaHook.INTERMEDIATE_STATES: Review comment: @XD-DENG According to the source code of `query_status`: ``` if query_state is None: self.log.info('Trial {try_number}: Invalid query state. Retrying again'.format( try_number=try_number)) ``` This query might something wrong so that this query is not sent to the queue of Athena, so I think it should show `failed` if `query_status` is None. Does it make any sense or there is somthing I missing? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services