yermalov-here opened a new pull request, #33952:
URL: https://github.com/apache/airflow/pull/33952

   <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
   
      http://www.apache.org/licenses/LICENSE-2.0
   
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   airflow 2.7.0
   amazon-provider 8.6.0
   
   **Problem**
   AWS EmrStepSensor in deferred mode falls back to `aws_default` connection 
after deferral and ignores the connection id specified.
   
   **Root-cause**
   Internally EmrStepSensorTrigger initializes EmrHook without specifying a 
value for the `aws_conn_id` and instead sets the `emr_conn_id` which is not 
used by the trigger. Thus after serialization in the Trigerrer the default 
value for the `aws_conn_id` is used (which is `aws_default`).
   See the example log in the bottom (Airflow there doesn't have the 
`aws_default` connection configured). Note that before deferral it uses the 
`aws` connection, but afterwards switches to `aws_default.`
   
   **Scope**
   This PR 
   1. fixes the EmrHook initialization in the EmrStepSensorTrigger
   2. extends the emr triggers serialization test to check the hooks' 
`aws_conn_id` value
   3. and makes aws_conn_id assignment explicit for hook initialization in the 
EmrContainerTrigger to avoid the same problem in case of implementation changes.
   
   ```
   [2023-01-01, 13:52:27 UTC] {task_command.py:415} INFO - Running 
<TaskInstance: dagname.wait_emr_step scheduled__2023-01-01 00:00:00+00:00 
[running]> on host INSTANCE_01
   [2023-01-01, 13:52:27 UTC] {taskinstance.py:1660} INFO - Exporting env vars: 
AIRFLOW_CTX_DAG_EMAIL='[email protected]' AIRFLOW_CTX_DAG_OWNER='airflow' 
AIRFLOW_CTX_DAG_ID='dagname' AIRFLOW_CTX_TASK_ID='wait_emr_step' 
AIRFLOW_CTX_EXECUTION_DATE='2023-01-01T00:00:00' AIRFLOW_CTX_TRY_NUMBER='2' 
AIRFLOW_CTX_DAG_RUN_ID='scheduled__2023-01-01 00:00:00+00:00'
   [2023-01-01, 13:52:27 UTC] {base.py:73} INFO - Using connection ID 'aws' for 
task execution.
   [2023-01-01, 13:52:27 UTC] {emr.py:575} INFO - Poking step s-STEPID on 
cluster j-CLUSTERID
   [2023-01-01, 13:52:28 UTC] {emr.py:79} INFO - Job flow currently PENDING
   [2023-01-01, 13:52:28 UTC] {taskinstance.py:1526} INFO - Pausing task as 
DEFERRED. dag_id=dagname, task_id=wait_emr_step, 
execution_date=20230101T000000, start_date=20230101T000000
   [2023-01-01, 13:52:28 UTC] {local_task_job_runner.py:225} INFO - Task exited 
with return code 100 (task deferral)
   [2023-01-01, 13:52:28 UTC] {base_aws.py:566} WARNING - Unable to find AWS 
Connection ID 'aws_default', switching to empty.
   [2023-01-01, 13:52:28 UTC] {base_aws.py:160} INFO - No connection ID 
provided. Fallback on boto3 credential strategy (region_name=None). See: 
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
   [2023-01-01, 13:52:40 UTC] {taskinstance.py:1159} INFO - Dependencies all 
met for dep_context=non-requeueable deps ti=<TaskInstance: 
dagname.wait_emr_step scheduled__2023-01-01 00:00:00+00:00 [queued]>
   [2023-01-01, 13:52:40 UTC] {taskinstance.py:1159} INFO - Dependencies all 
met for dep_context=requeueable deps ti=<TaskInstance: dagname.wait_emr_step 
scheduled__2023-01-01 00:00:00+00:00 [queued]>
   [2023-01-01, 13:52:40 UTC] {taskinstance.py:1359} INFO - Resuming after 
deferral
   [2023-01-01, 13:52:40 UTC] {taskinstance.py:1382} INFO - Executing 
<Task(EmrStepSensor): wait_emr_step> on 2023-01-01 00:00:00+00:00
   [2023-01-01, 13:52:40 UTC] {standard_task_runner.py:57} INFO - Started 
process 259 to run task
   [2023-01-01, 13:52:40 UTC] {standard_task_runner.py:84} INFO - Running: 
['airflow', 'tasks', 'run', 'dagname', 'wait_emr_step', 'scheduled__2023-01-01 
00:00:00+00:00', '--job-id', '2661', '--raw', '--subdir', 'DAGS_FOLDER/dag.py', 
'--cfg-path', '/tmp/tmpbugsw1g7']
   [2023-01-01, 13:52:40 UTC] {standard_task_runner.py:85} INFO - Job 2661: 
Subtask wait_emr_step
   [2023-01-01, 13:52:40 UTC] {task_command.py:415} INFO - Running 
<TaskInstance: dagname.wait_emr_step scheduled__2023-01-01 00:00:00+00:00 
[running]> on host INSTANCE_02
   [2023-01-01, 13:52:40 UTC] {taskinstance.py:1720} ERROR - Trigger failed:
   Traceback (most recent call last):
     File "airflow/jobs/triggerer_job_runner.py", line 527, in 
cleanup_finished_triggers
       result = details["task"].result()
     File "airflow/jobs/triggerer_job_runner.py", line 599, in run_trigger
       async for event in trigger.run():
     File "airflow/providers/amazon/aws/triggers/base.py", line 118, in run
       async with hook.async_conn as client:
     File "aiobotocore/session.py", line 27, in __aenter__
       self._client = await self._coro
     File "aiobotocore/session.py", line 211, in _create_client
       client = await client_creator.create_client(
     File "aiobotocore/client.py", line 76, in create_client
       client_args = self._get_client_args(
     File "aiobotocore/client.py", line 267, in _get_client_args
       return args_creator.get_client_args(
     File "aiobotocore/args.py", line 31, in get_client_args
       final_args = self.compute_client_args(
     File "botocore/args.py", line 219, in compute_client_args
       endpoint_config = self._compute_endpoint_config(
     File "botocore/args.py", line 368, in _compute_endpoint_config
       return self._resolve_endpoint(**resolve_endpoint_kwargs)
     File "botocore/args.py", line 473, in _resolve_endpoint
       return endpoint_bridge.resolve(
     File "botocore/client.py", line 595, in resolve
       resolved = self.endpoint_resolver.construct_endpoint(
     File "botocore/regions.py", line 229, in construct_endpoint
       result = self._endpoint_for_partition(
     File "botocore/regions.py", line 277, in _endpoint_for_partition
       raise NoRegionError()
   botocore.exceptions.NoRegionError: You must specify a region.
   [2023-01-01, 13:52:40 UTC] {taskinstance.py:1943} ERROR - Task failed with 
exception
   airflow.exceptions.TaskDeferralError: Trigger failure
   ```
   
   
   <!-- Please keep an empty line above the dashes. -->
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to