Re: [I] Glue Job Operator failing when deferrable=True [airflow]
wilsonhooi86 commented on issue #55007: URL: https://github.com/apache/airflow/issues/55007#issuecomment-3758462254 Hi @digvijay-y , apology for late reply. I was able to get to work for `apache-airflow-providers-amazon==9.12.0` This is because `apache-airflow-providers-amazon==9.12.0` requires `boto3 `and `botocore ` 1.37.2 version onwards. `aiobotocore ` package version is quite tricky as it works for selected `boto3 `and `botocore ` versions Sample constraints: ``` apache-airflow-providers-amazon==9.12.0 boto3==1.37.2 botocore==1.37.2 aiobotocore==2.22.0 ... ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Glue Job Operator failing when deferrable=True [airflow]
digvijay-y commented on issue #55007: URL: https://github.com/apache/airflow/issues/55007#issuecomment-3620139084 > [@wilsonhooi86](https://github.com/wilsonhooi86) can you try it with `botocore>=1.35.74,<1.35. 82` for `aiobotocore==2.16.0` The issue persist because of the version compatibility between these 2 and AWS client library. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Glue Job Operator failing when deferrable=True [airflow]
digvijay-y commented on issue #55007: URL: https://github.com/apache/airflow/issues/55007#issuecomment-3620086310 @wilsonhooi86 can you try it with ```botocore>=1.35.74,<1.35. 82``` for ```aiobotocore==2.16.0``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Glue Job Operator failing when deferrable=True [airflow]
digvijay-y commented on issue #55007: URL: https://github.com/apache/airflow/issues/55007#issuecomment-3620058774 @o-nikolas can you reopen this issue, I guess it was not completely solved. @wilsonhooi86 I will look more into this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Glue Job Operator failing when deferrable=True [airflow]
wilsonhooi86 commented on issue #55007:
URL: https://github.com/apache/airflow/issues/55007#issuecomment-3616231088
Good Day @digvijay-y , may I know which airflow version was this issue
resolved? Can we reopen this issue?
Recently we are using below airflow provider version but the error still
persist.
Apache Airflow Provider(s)
amazon
Versions of Apache Airflow Providers
apache-airflow-providers-amazon==9.17.0
Apache Airflow version
Airflow 2.10.3
Deployment
MWAA Airflow
https://github.com/user-attachments/assets/d1665f68-d8db-4b09-9910-6903622c18f6";
/>
```
ip-100-65-41-135.eu-central-1.compute.internal
*** Reading remote log from Cloudwatch log_group:
airflow-gtm-core-ostr-dev-euc1-mwaa-environment-public-sandbox-Task log_stream:
dag_id=glo-src_can-innomar-prd/run_id=manual__2025-12-05T10_19_50.473718+00_00/task_id=glue_priority_class_1.alt_fund_d/attempt=1.log
[Invalid date] {local_task_job_runner.py:123} ▶ Pre task execution logs
[Invalid date] {glue.py:220} INFO - Initializing AWS Glue Job:
gtm-core-dl-dev-euc1-glue-job-innomar. Wait for completion: True
[Invalid date] {glue.py:454} INFO - Checking if job already exists:
gtm-core-dl-dev-euc1-glue-job-innomar
[Invalid date] {base.py:84} INFO - Retrieving connection 'aws_default'
[Invalid date] {glue.py:241} INFO - You can monitor this Glue Job run at:
https://console.aws.amazon.com/gluestudio/home?region=eu-central-1#/job/gtm-core-dl-dev-euc1-glue-job-innomar/run/jr_af011a908310600895274468813f904a9bb973c7791f83e9cb15ffe515e0cba0
[Invalid date] {taskinstance.py:288} INFO - Pausing task as DEFERRED.
dag_id=glo-src_can-innomar-prd, task_id=glue_priority_class_1.alt_fund_d,
run_id=manual__2025-12-05T10:19:50.473718+00:00,
execution_date=20251205T101950, start_date=20251205T101955
[Invalid date] {taskinstance.py:340} ▶ Post task execution logs
[Invalid date] {base.py:84} INFO - Retrieving connection 'aws_default'
[Invalid date] {local_task_job_runner.py:123} ▶ Pre task execution logs
[Invalid date] {baseoperator.py:1798} ERROR - Trigger failed:
Traceback (most recent call last):
File
"/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/jobs/triggerer_job_runner.py",
line 558, in cleanup_finished_triggers
result = details["task"].result()
File
"/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/jobs/triggerer_job_runner.py",
line 630, in run_trigger
async for event in trigger.run():
File
"/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/amazon/aws/triggers/base.py",
line 145, in run
async with await hook.get_async_conn() as client:
File
"/usr/local/airflow/.local/lib/python3.11/site-packages/aiobotocore/session.py",
line 25, in __aenter__
self._client = await self._coro
File
"/usr/local/airflow/.local/lib/python3.11/site-packages/aiobotocore/session.py",
line 215, in _create_client
client = await client_creator.create_client(
^^^
File
"/usr/local/airflow/.local/lib/python3.11/site-packages/aiobotocore/client.py",
line 82, in create_client
client_args = self._get_client_args(
^^
File
"/usr/local/airflow/.local/lib/python3.11/site-packages/aiobotocore/client.py",
line 291, in _get_client_args
return args_creator.get_client_args(
^
File
"/usr/local/airflow/.local/lib/python3.11/site-packages/aiobotocore/args.py",
line 99, in get_client_args
ruleset_resolver = self._build_endpoint_resolver(
^^
File
"/usr/local/airflow/.local/lib/python3.11/site-packages/aiobotocore/args.py",
line 165, in _build_endpoint_resolver
resolver_builtins = self.compute_endpoint_resolver_builtin_defaults(
TypeError: ClientArgsCreator.compute_endpoint_resolver_builtin_defaults()
missing 2 required positional arguments: 'credentials' and
'account_id_endpoint_mode'
[Invalid date] {taskinstance.py:3311} ERROR - Task failed with exception
Traceback (most recent call last):
File
"/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py",
line 767, in _execute_task
result = _execute_callable(context=context, **execute_callable_kwargs)
^
File
"/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py",
line 733, in _execute_callable
return ExecutionCallableRunner(
File
"/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/utils/operator_helpers.p
Re: [I] Glue Job Operator failing when deferrable=True [airflow]
o-nikolas closed issue #55007: Glue Job Operator failing when deferrable=True URL: https://github.com/apache/airflow/issues/55007 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Glue Job Operator failing when deferrable=True [airflow]
o-nikolas commented on issue #55007: URL: https://github.com/apache/airflow/issues/55007#issuecomment-3559584576 > [@o-nikolas](https://github.com/o-nikolas) I looked through it! The issue was to due version mismatch, it was update by some one. I think this issue should be closed. Sounds good, I'll close it! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Glue Job Operator failing when deferrable=True [airflow]
digvijay-y commented on issue #55007: URL: https://github.com/apache/airflow/issues/55007#issuecomment-3559290882 @o-nikolas I looked through it! The issue was to due version mismatch, it was update by some one. I think this issue should be closed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Glue Job Operator failing when deferrable=True [airflow]
o-nikolas commented on issue #55007: URL: https://github.com/apache/airflow/issues/55007#issuecomment-3519787187 Sounds good, thanks for verifying! @digvijay-y -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Glue Job Operator failing when deferrable=True [airflow]
digvijay-y commented on issue #55007: URL: https://github.com/apache/airflow/issues/55007#issuecomment-3519754742 @o-nikolas I think this was taken care of with another issue. Let me check it again. If not, I will send a PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Glue Job Operator failing when deferrable=True [airflow]
o-nikolas commented on issue #55007: URL: https://github.com/apache/airflow/issues/55007#issuecomment-3519733459 @digvijay-y are you still working on this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Glue Job Operator failing when deferrable=True [airflow]
vincbeck commented on issue #55007: URL: https://github.com/apache/airflow/issues/55007#issuecomment-3245512462 All yours! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Glue Job Operator failing when deferrable=True [airflow]
digvijay-y commented on issue #55007: URL: https://github.com/apache/airflow/issues/55007#issuecomment-3239167563 I'm up for it! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Glue Job Operator failing when deferrable=True [airflow]
eladkal commented on issue #55007: URL: https://github.com/apache/airflow/issues/55007#issuecomment-3234385422 cc @o-nikolas @vincbeck -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
