ferruzzi commented on code in PR #46950:
URL: https://github.com/apache/airflow/pull/46950#discussion_r1968437877
##########
providers/amazon/src/airflow/providers/amazon/aws/hooks/glue.py:
##########
@@ -245,6 +246,9 @@ def display_logs_from(log_group: str, continuation_token:
str | None) -> str | N
for response in paginator.paginate(
logGroupName=log_group,
logStreamNames=[run_id],
+ startTime=int(
+ (datetime.now() - timedelta(hours=24)).timestamp() *
1000
+ ), # 24 hours ago in milliseconds
Review Comment:
Also, entirely unrelated to this PR, but while looking to see if we have a
get_job_run variant in the hook I noticed this oddity:
In the Glue hook [we implement list_jobs](list) which just calls and returns
`get_jobs()`, but the API **_has_** a [list_jobs
call](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue/client/list_jobs.html).....
we should maybe look into untangling that. It looks like get_jobs is
returning a list of full details for each job, what I would expect to see from
a describe_jobs() - which doesn't exist - but the API's version of list_jobs is
just returning a list of names, so hat would be a breaking change...
IMHO (unless there is some context I'm missing, of course) having
`hook.conn.list_jobs()` return something entirely different than
`hook.list_jobs()` is a bad idea. Maybe we should deprecate our
hook.list_jobs() and either rename it hook.describe_jobs() or, preferably, drop
it entirely since it's just a thin-wrapper renaming an existing api call. It's
a bad habit on our side to rename an existing provider's API feature like that.
That's not part of this PR, but maybe something to do next if you want to
poke at the Glue stuff more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
