o-nikolas commented on code in PR #71546:
URL: https://github.com/apache/airflow/pull/71546#discussion_r3786599924
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -447,8 +452,10 @@ def _find_previous_job_run(self, context: Context,
task_uuid: str) -> str | None
self.log.info("Previous Glue job_run_id: %s, state: %s",
previous_job_run_id, state)
if self.is_job_active(state):
return previous_job_run_id
- except Exception:
- self.log.warning("Failed to get previous Glue job run state",
exc_info=True)
+ except ClientError:
Review Comment:
These now catch much less than before. So things that used to just cause a
fresh Glue Job to be created now fail the entire task? Was that intentional?
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py:
##########
@@ -180,6 +180,8 @@ class GlueJobOperator(ResumableJobMixin,
AwsBaseOperator[GlueJobHook]):
operator_extra_links = (GlueJobRunDetailsLink(),)
TASK_UUID_ARG = "--airflow_task_uuid"
+ TASK_UUID_SCAN_PAGE_SIZE = 50
Review Comment:
Why not max this out to the 200 limit? You'll avoid more possible throttling
errors if you request as many as possible each time through. Also you'll have
a higher chance of getting a hit if the job is busy and you need to go far into
history to find the existing uuid match.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
