Kiyoshi Nomo created AIRFLOW-2939: ------------------------------------- Summary: `set` fails in case of `exisiting_files is None` and in case of `json.dumps` Key: AIRFLOW-2939 URL: https://issues.apache.org/jira/browse/AIRFLOW-2939 Project: Apache Airflow Issue Type: Bug Components: operators Affects Versions: 1.10 Reporter: Kiyoshi Nomo
h1. Problems h2. TypeError: 'NoneType' object is not iterable [https://github.com/apache/incubator-airflow/blob/06b62c42b0b55ea55b86b130317594738d2f36a2/airflow/contrib/operators/gcs_to_s3.py#L91] {code:java} >>> set(None) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'NoneType' object is not iterable {code} h2. TypeError: set(['a']) is not JSON serializable [https://github.com/apache/incubator-airflow/blob/b78c7fb8512f7a40f58b46530e9b3d5562fe84ea/airflow/models.py#L4483] {code:python} >>> json.dumps(set(['a'])) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/opt/pyenv/versions/2.7.11/lib/python2.7/json/__init__.py", line 244, in dumps return _default_encoder.encode(obj) File "/usr/local/opt/pyenv/versions/2.7.11/lib/python2.7/json/encoder.py", line 207, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/local/opt/pyenv/versions/2.7.11/lib/python2.7/json/encoder.py", line 270, in iterencode return _iterencode(o, 0) File "/usr/local/opt/pyenv/versions/2.7.11/lib/python2.7/json/encoder.py", line 184, in default raise TypeError(repr(o) + " is not JSON serializable") TypeError: set(['a']) is not JSON serializable {code} h1. Solution * Check that the existing fils is not None. * Convert it to the `set` and return it to the `list` after get to the difference of files. {code:python} if existing_files is not None: files = list(set(files) - set(existing_files)) {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)