vincbeck commented on code in PR #51883:
URL: https://github.com/apache/airflow/pull/51883#discussion_r2154892781
##########
providers/amazon/src/airflow/providers/amazon/aws/hooks/emr.py:
##########
@@ -311,6 +311,24 @@ def cancel_running_jobs(
return count
+ def cancel_job(self, application_id: str, job_run_id: str) -> None:
+ """
+ Cancel a specific EMR Serverless job run.
+
+ :param application_id: The ID of the EMR Serverless application
+ :param job_run_id: The ID of the job run to cancel
+ :raises AirflowException: If cancellation fails
+ """
+ try:
+ self.log.info("Cancelling EMR Serverless job %s in application
%s", job_run_id, application_id)
+ response = self.conn.cancel_job_run(applicationId=application_id,
jobRunId=job_run_id)
+ http_code = response["ResponseMetadata"]["HTTPStatusCode"]
+ if http_code != 200:
+ raise AirflowException(f"Failed to cancel job run
{job_run_id}: HTTP {http_code}")
Review Comment:
As far as I know this can never happen. If the call is not successful,
you'll get an exception.
Other than that, this method is just a thin wrapper on top of
`self.conn.cancel_job_run`, I dont think it is necessary to create a method for
that. You can call boto3 API directly in the operator.
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/emr.py:
##########
@@ -1280,6 +1288,9 @@ def execute_complete(self, context: Context, event:
dict[str, Any] | None = None
if validated_event["status"] == "success":
self.log.info("Serverless job completed")
return validated_event["job_id"]
+ self.log.info("Cancelling EMR Serverless job %s", self.job_id)
+ self.hook.conn.cancel_job_run(applicationId=self.application_id,
jobRunId=self.job_id)
Review Comment:
You should never use instance variable in `execute_complete`. The methods
`execute` and `execute_method` are executed with 2 different instances, thus
you have no guarantee these variables are set and equal the same across
instances. You should only use data from the event. The job ID is available in
`validated_event["job_id"]` but you need to modify the trigger to also send the
application ID as part of the event
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
