baolsen opened a new issue #12639:
URL: https://github.com/apache/airflow/issues/12639
**Apache Airflow version**: 1.10.8 (Patched with latest AWS Hook)
**Environment**:
- **Cloud provider or hardware configuration**: 4 VCPU 8GB RAM VM
- **OS** (e.g. from /etc/os-release): RHEL 7.7
- **Kernel** (e.g. `uname -a`): Linux 3.10.0-957.el7.x86_64 #1 SMP Thu Oct 4
20:48:51 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
- **Install tools**:
- **Others**:
The AWS Hook functionality for AssumeRoleWithSAML is not available in this
version, we manually added it via patching the hook file.
**What happened**:
In our environment the IDP HTTP request to obtain SAML assertion fails
temporarily with HTTP 400 with no way to retry. This causes the task to fail.
**What you expected to happen**:
We would like a way to automatically retry the IDP request if we get HTTP
400. Ideally the Retry would be configurable for other sites/scenarios.
**How to reproduce it**:
Very difficult to reproduce easily outside of our environment.
You'd need to use AWS AssumeRoleWithSAML for authentication, requiring an
IDP, and have to simulate failures on that IDP.
**Anything else we need to know**:
After many successful connections we get the following intermittent issue,
which causes task to fail.
Retrying the task manually works because the issue is temporary.
Site-specific details have been redacted from the log.
<details><summary>example.log</summary>
[2020-10-30 14:54:25,739] {logging_mixin.py:112} INFO - [2020-10-30
14:54:25,739] {aws_hook.py:63} INFO - Airflow Connection:
aws_conn_id=redacted_conn_id
[2020-10-30 14:54:25,754] {logging_mixin.py:112} INFO - [2020-10-30
14:54:25,754] {aws_hook.py:99} INFO - No credentials retrieved from Connection
redacted_conn_id
[2020-10-30 14:54:25,754] {logging_mixin.py:112} INFO - [2020-10-30
14:54:25,754] {aws_hook.py:105} INFO - Retrieving config_kwargs from
Connection.extra_config['config_kwargs']: {'proxies': {'http':
'http://redacted:8080', 'https': 'http://redacted:8080'}, 'retries': {'mode':
'standard'}}
[2020-10-30 14:54:25,754] {logging_mixin.py:112} INFO - [2020-10-30
14:54:25,754] {aws_hook.py:111} INFO - Retrieving region_name from
Connection.extra_config['region_name']
[2020-10-30 14:54:25,754] {logging_mixin.py:112} INFO - [2020-10-30
14:54:25,754] {aws_hook.py:114} INFO - region_name=redacted
[2020-10-30 14:54:25,754] {logging_mixin.py:112} INFO - [2020-10-30
14:54:25,754] {aws_hook.py:132} INFO - role_arn is
arn:aws:iam::redacted:role/r-redacted
[2020-10-30 14:54:25,754] {logging_mixin.py:112} INFO - [2020-10-30
14:54:25,754] {aws_hook.py:146} INFO - Creating sts_session with
aws_access_key_id=None
[2020-10-30 14:54:27,931] {logging_mixin.py:112} INFO - [2020-10-30
14:54:27,931] {aws_hook.py:164} INFO - assume_role_method=assume_role_with_saml
[2020-10-30 14:54:27,931] {logging_mixin.py:112} INFO - [2020-10-30
14:54:27,931] {aws_hook.py:244} INFO - idp_url= https://redacted/REDACTED
[2020-10-30 14:54:31,902] {logging_mixin.py:112} INFO - [2020-10-30
14:54:31,902] {gssapi_.py:233} ERROR - handle_other(): Mutual authentication
unavailable on 400 response
[2020-10-30 14:54:31,904] {taskinstance.py:1128} ERROR - 400 Client Error:
Bad Request for url: https://redacted/REDACTED
Traceback (most recent call last):
File
"/home/redacted/venv/lib64/python3.6/site-packages/airflow/models/taskinstance.py",
line 966, in _run_raw_task
result = task_copy.execute(context=context)
File
"/home/redacted/venv/lib64/python3.6/site-packages/airflow/sensors/base_sensor_operator.py",
line 107, in execute
while not self.poke(context):
File
"/home/redacted/venv/lib64/python3.6/site-packages/airflow/contrib/sensors/emr_base_sensor.py",
line 41, in poke
response = self.get_emr_response()
File
"/home/redacted/venv/lib64/python3.6/site-packages/airflow/contrib/sensors/emr_step_sensor.py",
line 51, in get_emr_response
emr = EmrHook(aws_conn_id=self.aws_conn_id).get_conn()
File
"/home/redacted/venv/lib64/python3.6/site-packages/airflow/contrib/hooks/emr_hook.py",
line 38, in get_conn
self.conn = self.get_client_type('emr', self.region_name)
File
"/home/redacted/venv/lib64/python3.6/site-packages/airflow/contrib/hooks/aws_hook.py",
line 308, in get_client_type
session, endpoint_url = self._get_credentials(region_name)
File
"/home/redacted/venv/lib64/python3.6/site-packages/airflow/contrib/hooks/aws_hook.py",
line 178, in _get_credentials
assume_role_kwargs
File
"/home/redacted/venv/lib64/python3.6/site-packages/airflow/contrib/hooks/aws_hook.py",
line 270, in _assume_role_with_saml
idp_reponse.raise_for_status()
File
"/home/redacted/venv/lib64/python3.6/site-packages/requests/models.py", line
940, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url:
https://redacted/REDACTED
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
