[
https://issues.apache.org/jira/browse/AIRFLOW-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik resolved AIRFLOW-2939.
---------------------------------
Resolution: Fixed
Fix Version/s: 1.10.2
Resolved by https://github.com/apache/incubator-airflow/pull/4371
> `set` fails in case of `exisiting_files is None` and in case of `json.dumps`
> ----------------------------------------------------------------------------
>
> Key: AIRFLOW-2939
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2939
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
> Affects Versions: 2.0.0
> Reporter: Kiyoshi Nomo
> Assignee: Yohei Onishi
> Priority: Major
> Fix For: 1.10.2
>
>
> h1. Problems
> h2. TypeError: 'NoneType' object is not iterable
> [https://github.com/apache/incubator-airflow/blob/06b62c42b0b55ea55b86b130317594738d2f36a2/airflow/contrib/operators/gcs_to_s3.py#L91]
>
> {code:java}
> >>> set(None)
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> TypeError: 'NoneType' object is not iterable
> {code}
>
> h2. TypeError: set(['a']) is not JSON serializable
> [https://github.com/apache/incubator-airflow/blob/b78c7fb8512f7a40f58b46530e9b3d5562fe84ea/airflow/models.py#L4483]
>
> {code:python}
> >>> json.dumps(set(['a']))
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/usr/local/opt/pyenv/versions/2.7.11/lib/python2.7/json/__init__.py",
> line 244, in dumps
> return _default_encoder.encode(obj)
> File "/usr/local/opt/pyenv/versions/2.7.11/lib/python2.7/json/encoder.py",
> line 207, in encode
> chunks = self.iterencode(o, _one_shot=True)
> File "/usr/local/opt/pyenv/versions/2.7.11/lib/python2.7/json/encoder.py",
> line 270, in iterencode
> return _iterencode(o, 0)
> File "/usr/local/opt/pyenv/versions/2.7.11/lib/python2.7/json/encoder.py",
> line 184, in default
> raise TypeError(repr(o) + " is not JSON serializable")
> TypeError: set(['a']) is not JSON serializable
> {code}
>
> h1. Solution
> * Check that the existing fils is not None.
> * Convert it to the `set` and return it to the `list` after get to the
> difference of files.
> {code:python}
> if existing_files is not None:
> files = list(set(files) - set(existing_files))
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
