nrobinson-intelycare commented on issue #45622: URL: https://github.com/apache/airflow/issues/45622#issuecomment-2588124655
> I am trying to understand the whole picture. When using `BatchOperator`, it can successfully create the job but when the trigger gets initialized (because the operator is run in deferrable mode) it fails instantly? Does it fail right away when initializing the trigger or later while pooling the batch job status? > > Since the deployment is "Virtualenv installation", I assume the triggerer and the scheduler are the same machine? If so, I cannot explain why the job gets created successfully created but the trigger fails to fetch AWS creds when getting created. If the job gets created successfully it means AWS creds are there and valid. > > If the scheduler and triggerer are different machines, I could find explanations. Hey @vincbeck , thanks for looking into this. The scheduler and triggerer are on the same machine, and are both using the IAM role of the EC2 instance. I think the trigger is failing right away. Here is the full task log from another BatchOperator failing in the same way, coincidentally on the same day. [dag_id=foxy_pendo_run_id=scheduled__2025-01-10T20_30_00+00_00_task_id=foxy_pendo_batch_job_attempt=1.log](https://github.com/user-attachments/files/18401616/dag_id.foxy_pendo_run_id.scheduled__2025-01-10T20_30_00%2B00_00_task_id.foxy_pendo_batch_job_attempt.1.log) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
