uranusjr opened a new pull request, #77281:
URL: https://github.com/apache/airflow/pull/77281

   Python, Go, Java, and TypeScript all have various differences in handling 
null values. This aligns the behavior between different runtimes as much as 
possible around both explicitly setting an XCom, and implicitly pushing a value 
by returning it from the task function.
   
   ### General guidelines
   
   * Setting an XCom to null explicitly is allowed, as designed in both the 
XCom database table, and the Execution API endpoint.
   * Whether a return value should be pushed is decided by the task function's 
signature.
   
   ### Python
   
   The implementation is already almost the best it can be, deviating only due 
to Python’s natural restrictions (no distinction between “no return value” and 
“return None”).
   
   ### Go
   
   The explicit `PushXCom` already fits the guideline. Two changes to return 
value handling:
   
   1. A task function with return type `(*T, error)` now always pushes its 
result. `return nil, nil` pushes null (previous no value would be pushed). 
Unchanged: No value is pushed when the function’s return type is `error`.
   2. If the task function's return type is `([]T, error)` or `(map[K]V, 
error)`, `return nil, nil` would push an empty array or map (previously no 
value would be pushed). This is due to Go commonly uses nil to represent an 
empty array or map. Pushing `nil` instead would make interfacing with such a 
task more difficult. (This is recommended by Claude, but I’m not entirely sure 
about it, ahving little experience in Go. Advices are welcome.)
   
   ### Java
   
   The `setXCom` function needs to be amended to accept and handle null. Before 
this, runtime would actually crash if null is returned from a task function.
   
   ### TypeScript
   
   Already doing things correctly.
   
   ### Documentation
   
   A line is added to each SDK's documentation to clarify the behavior. (Except 
Python, which already does.)
   
   ### Drive-by improvement
   
   XCom endpoints in the Execution API contain some stale handling carried over 
from SQLAlchemy 1.x no longer needed since we now require 2+.
   
   ### Next work
   
   This covers the _write_ side of the story, and should be enough to unblock 
task mapping design. However, there’s still an issue on the _read_ side. No SDK 
(Python included) currently has a way to distinguish between _no XCom_ and 
_null written to XCom_. Python is maybe the easiest to amend, but other SDKs 
would need further fixes in supervisor comms.
   
   Once we fix the read side interface, we could also add a cute feature to 
Python that mimics typed function returns in other SDKs. Python never 
distinguishes `return` and `return None` _at runtime_, but type checkers (Mypy 
etc.) do based on the function's return type annotation.
   
   ```python
   # These do not write an XCom. Same as current behavior.
   # Mypy accepts both 'return' and 'return None' in these cases.
   def no_annotation():
   def explicit_return_type_none() -> None:
   def return_any() -> Any:
   
   # Bare return fails type checks, only 'return None' satisfies Mypy.
   # So we write a null XCom.
   def return_optional() -> T | None:
   
   # Both bare return and 'return None' would fail type checks.
   # But if the function ends with a None at runtime, a null XCom makes sense?
   def return_optional() -> T:
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to