vishalmore90 opened a new issue, #76973:
URL: https://github.com/apache/airflow/issues/76973
### Under which category would you file this issue?
Task SDK
### Apache Airflow version
`main` (development) / 3.0.0+
### What happened and how to reproduce it?
#### Issue Description
In `task-sdk/src/airflow/sdk/definitions/callback.py`, `Callback.__hash__`
attempts to make dictionary `kwargs` hashable by only flattening the top level
via `tuple(sorted(v.items()))`. If any value within `kwargs` is a `list`,
nested `dict`, `set`, or any non-primitive data structure (for example,
`kwargs={"recipients": ["[email protected]"]}` or `kwargs={"config":
{"timeout": 30}}`), Python raises `TypeError: unhashable type: 'list'` (or
`'dict'`).
Furthermore, in `task-sdk/src/airflow/sdk/definitions/deadline.py`,
`DeadlineAlert.__hash__` delegates directly to `hash(self.callback)`.
Consequently:
1. Calling `hash(callback)` fails.
2. Calling `hash(deadline_alert)` fails.
3. Adding `DeadlineAlert` or `Callback` instances to a `set` (e.g. for Dag
deadline alert deduplication) fails.
4. Using `DeadlineAlert` or `Callback` as a dictionary key fails.
#### Code Pointers
- `task-sdk/src/airflow/sdk/definitions/callback.py` (lines 134–142):
```python
def __hash__(self):
serialized = self.serialize()
hashable_items = []
for k, v in serialized.items():
if isinstance(v, dict):
hashable_items.append((k, tuple(sorted(v.items()))))
else:
hashable_items.append((k, v))
return hash(tuple(sorted(hashable_items)))
```
- `task-sdk/src/airflow/sdk/definitions/deadline.py` (lines 214–221):
```python
def __hash__(self) -> int:
return hash(
(
type(self.reference).__name__,
self.interval,
self.callback,
)
)
```
#### Steps to Reproduce
Run the following Python script using `task-sdk`:
```python
from datetime import timedelta
from airflow.sdk.definitions.callback import SyncCallback
from airflow.sdk.definitions.deadline import DeadlineAlert, DeadlineReference
# 1. Instantiate a callback with a list or nested structure in kwargs
callback = SyncCallback(
"os.path.join",
kwargs={"recipients": ["[email protected]", "[email protected]"]},
)
# 2. Hashing the callback raises TypeError:
hash(callback)
# TypeError: unhashable type: 'list'
# 3. Hashing a DeadlineAlert holding this callback raises TypeError:
alert = DeadlineAlert(
reference=DeadlineReference.DAGRUN_LOGICAL_DATE,
interval=timedelta(hours=1),
callback=callback,
)
hash(alert)
# TypeError: unhashable type: 'list'
# 4. Adding to a set or deduplicating alerts in Dag authoring fails:
alerts = {alert}
# TypeError: unhashable type: 'list'
```
Similarly, with nested dictionaries:
```python
callback = SyncCallback("os.path.join", kwargs={"config": {"retries": 3}})
hash(callback)
# TypeError: unhashable type: 'dict'
```
#### Traceback
```pytb
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "task-sdk/src/airflow/sdk/definitions/callback.py", line 142, in
__hash__
return hash(tuple(sorted(hashable_items)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: unhashable type: 'list'
```
### What you think should happen instead?
`Callback.__hash__` should consistently hash any valid `Callback` instance
with arbitrary JSON-compatible values (such as lists, nested dictionaries,
numbers, strings, booleans, and None) in `kwargs`.
`Callback.serialize()` is JSON-serializable by specification (as it is
persisted to a database JSON column), and `Callback.__eq__` compares
`self.serialize() == other.serialize()`. Hashing a canonical sorted JSON string
handles arbitrary nesting and preserves Python's hash contract (`a == b`
implies `hash(a) == hash(b)` regardless of dictionary key insertion order):
```python
def __hash__(self):
return hash(json.dumps(self.serialize(), sort_keys=True, default=str))
```
### Operating System
All supported operating systems (Linux, macOS, Windows)
### Deployment
Other
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
#### Context & Git History
This defect was previously identified and fixed in commit
`8095abb57188f43305d9561f0f809d5e7900d6e5` (PR #66608):
```python
# ``serialize()`` is JSON-serializable by contract (it is persisted to a
JSON column),
# and ``kwargs`` may legally contain NESTED dicts/lists (e.g.
``kwargs={"config": {...}}``
# or ``kwargs={"tags": [...]}``). The previous implementation only flattened
the top level
# (``tuple(sorted(v.items()))``), so a nested dict/list value remained
unhashable and
# ``hash(callback)`` — and therefore ``hash(DeadlineAlert(...))`` and any
``set``/dict use —
# raised ``TypeError: unhashable type``. Hash a canonical JSON encoding
instead: it handles
# arbitrary nesting, and ``sort_keys=True`` keeps it consistent with
``__eq__`` (which
# compares ``serialize()``) regardless of key insertion order.
return hash(json.dumps(self.serialize(), sort_keys=True, default=str))
```
However, commit `8095abb57188f43305d9561f0f809d5e7900d6e5` was entirely
reverted in commit `609656f7d353b47858bd567efe02065234f62d10` (PR #68909) due
to an unrelated runtime issue in the Execution API callback context fetch path.
The revert unintentionally restored the buggy `tuple(sorted(v.items()))`
hashing logic.
Existing tests in `task-sdk/tests/task_sdk/definitions/test_callback.py`
only covered flat string-to-string mappings like `{"arg1": "value1"}` and did
not test list or nested dictionary kwargs.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
