Daniel Callander created AIRFLOW-6791:
-----------------------------------------
Summary: AWSBatchOperator crash when "parameter" arguments not
passed
Key: AIRFLOW-6791
URL: https://issues.apache.org/jira/browse/AIRFLOW-6791
Project: Apache Airflow
Issue Type: Bug
Components: aws, operators
Affects Versions: 1.10.9
Environment: Python 3.7 on Linux
Reporter: Daniel Callander
Attachments: requirements.txt
Hey folks,
After upgrading to 1.10.9 we've discovered that using the AWSBatchOperator
without passing the parameter argument (e.g., defaults to None) that it causes
boto to explode in a gnarly error.
{noformat}
task2 = AWSBatchOperator(
job_name="dc-test-alchemy-batch",
job_definition="slingshot-dc-test-alchemy-batch-jobdef-1",
job_queue="AwsBatch-batch-queue",
aws_conn_id="aws_skyscanner",
dag=(dag),
overrides={},
task_id="dc-test-alchemy-batch"
){noformat}
{noformat}
[2020-02-13 10:48:20.298] ERROR:airflow.task.handle_failure:1128 Parameter
validation failed:
Invalid type for parameter parameters, value: None, type: <class 'NoneType'>,
valid types: <class 'dict'>
Traceback (most recent call last):
File
"/usr/local/lib/python3.7/site-packages/airflow/contrib/operators/awsbatch_operator.py",
line 137, in execute
containerOverrides=self.overrides,
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 276,
in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 559,
in _make_api_call
api_params, operation_model, context=request_context)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 607,
in _convert_to_request_dict
api_params, operation_model)
File "/usr/local/lib/python3.7/site-packages/botocore/validate.py", line 297,
in serialize_to_request
raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter parameters, value: None, type: <class 'NoneType'>,
valid types: <class 'dict'>{noformat}
It seems boto is upset anything was passed at all rather than an empty dict.
Specifically passing _parameters={}_ solves the issue.
It seems there's precedent for this as array_properties already gets changed
from None to {} if not passed.
[https://github.com/apache/airflow/blob/1.10.9/airflow/contrib/operators/awsbatch_operator.py#L109]
This is with *boto3==1.11.16* and *botocore==1.14.16* on *airflow==1.10.9*.
I've attached our full frozen requirements. The Airflow version we use is
forked but we've only added some extra metric collection — I absolutely promise
the changes are tiny and unrelated to this.
Mostly it seems like this is a bug in boto3 itself but I figured it was worth
reporting incase anyone else runs into it (haven't been able to find any other
issues).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
