[
https://issues.apache.org/jira/browse/AIRFLOW-5039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117313#comment-17117313
]
ASF GitHub Bot commented on AIRFLOW-5039:
-----------------------------------------
allrod5 commented on pull request #6273:
URL: https://github.com/apache/airflow/pull/6273#issuecomment-633643045
Hi @rifqifatih, I've come to realize that the issue was not with Airflow
itself but that my project was relying on Airflow saving and loading datetimes
objects directly without my code having to serialize and deserialize these
datetimes.
So I'm unsure if this is a behaviour Airflow wants to keep (doesn't seem
that logical to have this datetime parsing enabled by default at least).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> xcom_push execution_date setting is broken when
> core.enable_xcom_pickling=False
> -------------------------------------------------------------------------------
>
> Key: AIRFLOW-5039
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5039
> Project: Apache Airflow
> Issue Type: Bug
> Components: xcom
> Affects Versions: 1.10.2, 1.10.3
> Reporter: Rodrigo Martins de Oliveira
> Assignee: Rodrigo Martins de Oliveira
> Priority: Critical
>
> Function [xcom_push from airflow.models.taskinstance automatically sets the
> execution_date to
> XCOM|https://github.com/apache/airflow/blob/7f66d0a6dc13a790b4d2d31694cb50737983f82e/airflow/models/taskinstance.py#L1336],
> while this is fine when pickling is enabled it just won't work when it isn't
> as [XCOM will attempt to JSON serialize a `datetime`
> object|https://github.com/apache/airflow/blob/7f66d0a6dc13a790b4d2d31694cb50737983f82e/airflow/models/xcom.py#L229],
> which is unsupported.
>
> This issue is critical as having pickle enabled poses security risks and [in
> environments such as Google Cloud Composer pickling is disabled for
> good|[https://cloud.google.com/composer/docs/concepts/airflow-configurations]].
> Attempting to run DAGs with tasks which pushes content to XCOM when
> `core-enable_xcom_pickling=False` in `airflow.cfg` will result in the
> following stacktrace (tested in version 1.10.2-composer):
> {quote}
> TypeError: Object of type 'datetime' is not JSON serializable
> [2019-07-24 19:14:53,193] \{models.py:1796} ERROR - Object of type
> 'datetime' is not JSON serializable
> Traceback (most recent call last):
> File "/usr/local/lib/airflow/airflow/models.py", line 1668, in _run_raw_task
> self.xcom_push(key=XCOM_RETURN_KEY, value=result)
> File "/usr/local/lib/airflow/airflow/models.py", line 2063, in xcom_push
> execution_date=execution_date or self.execution_date)
> File "/usr/local/lib/airflow/airflow/utils/db.py", line 73, in wrapper
> return func(*args, **kwargs)
> File "/usr/local/lib/airflow/airflow/models.py", line 4785, in set
> value = json.dumps(value).encode('UTF-8')
> File "/opt/python3.6/lib/python3.6/json/__init__.py", line 231, in dumps
> return _default_encoder.encode(obj)
> File "/opt/python3.6/lib/python3.6/json/encoder.py", line 199, in encode
> chunks = self.iterencode(o, _one_shot=True)
> File "/opt/python3.6/lib/python3.6/json/encoder.py", line 257, in iterencode
> return _iterencode(o, 0)
> File "/opt/python3.6/lib/python3.6/json/encoder.py", line 180, in default
> o.__class__.__name__)
> TypeError: Object of type 'datetime' is not JSON serializable
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
