yermalov-here opened a new pull request, #33952:
URL: https://github.com/apache/airflow/pull/33952
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
airflow 2.7.0
amazon-provider 8.6.0
**Problem**
AWS EmrStepSensor in deferred mode falls back to `aws_default` connection
after deferral and ignores the connection id specified.
**Root-cause**
Internally EmrStepSensorTrigger initializes EmrHook without specifying a
value for the `aws_conn_id` and instead sets the `emr_conn_id` which is not
used by the trigger. Thus after serialization in the Trigerrer the default
value for the `aws_conn_id` is used (which is `aws_default`).
See the example log in the bottom (Airflow there doesn't have the
`aws_default` connection configured). Note that before deferral it uses the
`aws` connection, but afterwards switches to `aws_default.`
**Scope**
This PR
1. fixes the EmrHook initialization in the EmrStepSensorTrigger
2. extends the emr triggers serialization test to check the hooks'
`aws_conn_id` value
3. and makes aws_conn_id assignment explicit for hook initialization in the
EmrContainerTrigger to avoid the same problem in case of implementation changes.
```
[2023-01-01, 13:52:27 UTC] {task_command.py:415} INFO - Running
<TaskInstance: dagname.wait_emr_step scheduled__2023-01-01 00:00:00+00:00
[running]> on host INSTANCE_01
[2023-01-01, 13:52:27 UTC] {taskinstance.py:1660} INFO - Exporting env vars:
AIRFLOW_CTX_DAG_EMAIL='[email protected]' AIRFLOW_CTX_DAG_OWNER='airflow'
AIRFLOW_CTX_DAG_ID='dagname' AIRFLOW_CTX_TASK_ID='wait_emr_step'
AIRFLOW_CTX_EXECUTION_DATE='2023-01-01T00:00:00' AIRFLOW_CTX_TRY_NUMBER='2'
AIRFLOW_CTX_DAG_RUN_ID='scheduled__2023-01-01 00:00:00+00:00'
[2023-01-01, 13:52:27 UTC] {base.py:73} INFO - Using connection ID 'aws' for
task execution.
[2023-01-01, 13:52:27 UTC] {emr.py:575} INFO - Poking step s-STEPID on
cluster j-CLUSTERID
[2023-01-01, 13:52:28 UTC] {emr.py:79} INFO - Job flow currently PENDING
[2023-01-01, 13:52:28 UTC] {taskinstance.py:1526} INFO - Pausing task as
DEFERRED. dag_id=dagname, task_id=wait_emr_step,
execution_date=20230101T000000, start_date=20230101T000000
[2023-01-01, 13:52:28 UTC] {local_task_job_runner.py:225} INFO - Task exited
with return code 100 (task deferral)
[2023-01-01, 13:52:28 UTC] {base_aws.py:566} WARNING - Unable to find AWS
Connection ID 'aws_default', switching to empty.
[2023-01-01, 13:52:28 UTC] {base_aws.py:160} INFO - No connection ID
provided. Fallback on boto3 credential strategy (region_name=None). See:
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
[2023-01-01, 13:52:40 UTC] {taskinstance.py:1159} INFO - Dependencies all
met for dep_context=non-requeueable deps ti=<TaskInstance:
dagname.wait_emr_step scheduled__2023-01-01 00:00:00+00:00 [queued]>
[2023-01-01, 13:52:40 UTC] {taskinstance.py:1159} INFO - Dependencies all
met for dep_context=requeueable deps ti=<TaskInstance: dagname.wait_emr_step
scheduled__2023-01-01 00:00:00+00:00 [queued]>
[2023-01-01, 13:52:40 UTC] {taskinstance.py:1359} INFO - Resuming after
deferral
[2023-01-01, 13:52:40 UTC] {taskinstance.py:1382} INFO - Executing
<Task(EmrStepSensor): wait_emr_step> on 2023-01-01 00:00:00+00:00
[2023-01-01, 13:52:40 UTC] {standard_task_runner.py:57} INFO - Started
process 259 to run task
[2023-01-01, 13:52:40 UTC] {standard_task_runner.py:84} INFO - Running:
['airflow', 'tasks', 'run', 'dagname', 'wait_emr_step', 'scheduled__2023-01-01
00:00:00+00:00', '--job-id', '2661', '--raw', '--subdir', 'DAGS_FOLDER/dag.py',
'--cfg-path', '/tmp/tmpbugsw1g7']
[2023-01-01, 13:52:40 UTC] {standard_task_runner.py:85} INFO - Job 2661:
Subtask wait_emr_step
[2023-01-01, 13:52:40 UTC] {task_command.py:415} INFO - Running
<TaskInstance: dagname.wait_emr_step scheduled__2023-01-01 00:00:00+00:00
[running]> on host INSTANCE_02
[2023-01-01, 13:52:40 UTC] {taskinstance.py:1720} ERROR - Trigger failed:
Traceback (most recent call last):
File "airflow/jobs/triggerer_job_runner.py", line 527, in
cleanup_finished_triggers
result = details["task"].result()
File "airflow/jobs/triggerer_job_runner.py", line 599, in run_trigger
async for event in trigger.run():
File "airflow/providers/amazon/aws/triggers/base.py", line 118, in run
async with hook.async_conn as client:
File "aiobotocore/session.py", line 27, in __aenter__
self._client = await self._coro
File "aiobotocore/session.py", line 211, in _create_client
client = await client_creator.create_client(
File "aiobotocore/client.py", line 76, in create_client
client_args = self._get_client_args(
File "aiobotocore/client.py", line 267, in _get_client_args
return args_creator.get_client_args(
File "aiobotocore/args.py", line 31, in get_client_args
final_args = self.compute_client_args(
File "botocore/args.py", line 219, in compute_client_args
endpoint_config = self._compute_endpoint_config(
File "botocore/args.py", line 368, in _compute_endpoint_config
return self._resolve_endpoint(**resolve_endpoint_kwargs)
File "botocore/args.py", line 473, in _resolve_endpoint
return endpoint_bridge.resolve(
File "botocore/client.py", line 595, in resolve
resolved = self.endpoint_resolver.construct_endpoint(
File "botocore/regions.py", line 229, in construct_endpoint
result = self._endpoint_for_partition(
File "botocore/regions.py", line 277, in _endpoint_for_partition
raise NoRegionError()
botocore.exceptions.NoRegionError: You must specify a region.
[2023-01-01, 13:52:40 UTC] {taskinstance.py:1943} ERROR - Task failed with
exception
airflow.exceptions.TaskDeferralError: Trigger failure
```
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]