kaxil commented on issue #45231: URL: https://github.com/apache/airflow/issues/45231#issuecomment-2659973296
Probably the right thing to do -- I have left some comments on the PR adding more context/background. fyi: We would still have multiple serialization/ de-serialization (after eliminating 1 step in this PR) steps that we should consolidate -- even after this PR. **Storing Data** (for GO-client example): 1. Go serializes the object (json.Marshal) → Sends it as an HTTP request. 2. FastAPI deserializes it (JsonValue) → Converts it into a Python dict. 3. SQLAlchemy stores it (Column(JSON)) → Serializes it back into JSON in the database. https://github.com/apache/airflow/blob/8f63b828ace09b1095229f22b0c6d1f0f85ea81a/airflow/models/xcom.py#L83 **Retrieving Data**: 1. SQLAlchemy loads it as a Python dict from the DB. 2. FastAPI serializes it back into JSON. 3. Go receives it and deserializes it (`json.Unmarshal` or a different library). ---- For Python, for non-JSON objects which should be somewhat common where users store datetime, dataframe etc: **Storing Data** (for GO-client example): 1. Our Serializer module or `XComEncoder` serializes the object (e.g set, Asset, datetime) into Python JSON-serializable object. 2. FastAPI deserializes it (JsonValue) → Converts it into a Python dict. 3. SQLAlchemy stores it (Column(JSON)) → Serializes it back into JSON in the database. --- Initially, I went for "string" as a contract to avoid multiple things doing serialization and instead leave it completely to the client and to retain the current behaviour of `XCom.serialiaze_value` which always returned a string object: https://github.com/apache/airflow/blob/8f63b828ace09b1095229f22b0c6d1f0f85ea81a/airflow/models/xcom.py#L443-L458 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
