kaxil commented on issue #45231:
URL: https://github.com/apache/airflow/issues/45231#issuecomment-2659973296

   Probably the right thing to do -- I have left some comments on the PR adding 
more context/background.
   
   
   fyi: We would still have multiple serialization/ de-serialization (after 
eliminating 1 step in this PR) steps that we should consolidate -- even after 
this PR.
   
   **Storing Data** (for GO-client example):
   
   1. Go serializes the object (json.Marshal) → Sends it as an HTTP request.
   2. FastAPI deserializes it (JsonValue) → Converts it into a Python dict.
   3. SQLAlchemy stores it (Column(JSON)) → Serializes it back into JSON in the 
database.
   
   
https://github.com/apache/airflow/blob/8f63b828ace09b1095229f22b0c6d1f0f85ea81a/airflow/models/xcom.py#L83
   
   
   **Retrieving Data**:
   
   1. SQLAlchemy loads it as a Python dict from the DB.
   2. FastAPI serializes it back into JSON.
   3. Go receives it and deserializes it (`json.Unmarshal` or a different 
library).
   
   
   ----
   
   For Python, for non-JSON objects which should be somewhat common where users 
store datetime, dataframe etc:
   **Storing Data** (for GO-client example):
   
   1. Our Serializer module or `XComEncoder` serializes the object (e.g set, 
Asset, datetime) into Python JSON-serializable object.
   2. FastAPI deserializes it (JsonValue) → Converts it into a Python dict.
   3. SQLAlchemy stores it (Column(JSON)) → Serializes it back into JSON in the 
database.
   
   
   ---
   
   Initially, I went for "string" as a contract to avoid multiple things doing 
serialization and instead leave it completely to the client and to retain the 
current behaviour of `XCom.serialiaze_value` which always returned a string 
object:
   
   
https://github.com/apache/airflow/blob/8f63b828ace09b1095229f22b0c6d1f0f85ea81a/airflow/models/xcom.py#L443-L458
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to