ChahatKumar opened a new issue, #73712:
URL: https://github.com/apache/airflow/issues/73712
### Apache Airflow version
3.x
### What happened?
When the CeleryExecutor sends a task workload to the task-runner subprocess,
Airflow serializes the message with `msgspec` (msgpack) in
`airflow/sdk/execution_time/comms.py` (`as_bytes` ->
`req_encoder.encode_into(...)`).
If the workload contains a Python `int` outside the range `-2**63 .. 2**64 -
1`, `msgspec` raises:
```
OverflowError: can't serialize ints < -2**63 or > 2**64 - 1
```
The task fails before it runs. This did not occur on Airflow 2, whose
CeleryExecutor dispatched a CLI command (`airflow tasks run ...`) rather than
serializing the workload via msgpack.
Abbreviated traceback:
```
File ".../airflow/providers/celery/executors/celery_executor_utils.py", line
209, in execute_workload
return _execute_workload_pre_3_3(input)
File ".../airflow/providers/celery/executors/celery_executor_utils.py", line
261, in _execute_workload_pre_3_3
supervise(...)
File ".../airflow/sdk/execution_time/supervisor.py", line 2295, in supervise
process = ActivitySubprocess.start(...)
File ".../airflow/sdk/execution_time/supervisor.py", line 1194, in
_on_child_started
self.send_msg(msg, request_id=0)
File ".../airflow/sdk/execution_time/comms.py", line 152, in as_bytes
self.req_encoder.encode_into(self, buffer, 4)
OverflowError: can't serialize ints < -2**63 or > 2**64 - 1
```
### What you think should happen instead?
Large integers (arbitrary-precision Python ints) that are valid task data
should not crash workload serialization. Options:
- Encode out-of-range integers in a lossless way (e.g., as strings/decimal),
or
- Provide a clear, catchable validation error rather than an unhandled
`OverflowError` that silently fails the task.
### How to reproduce
1. Airflow 3.x with CeleryExecutor.
2. A task whose serialized workload includes a Python int > `2**64 - 1` or <
`-2**63` (for example, produced from a very large JSON number that is cast to
int during task processing).
3. Trigger the task; the worker fails with the `OverflowError` above.
Minimal msgspec reproduction of the underlying limitation:
```python
import msgspec
msgspec.msgpack.encode(int(1.7e308)) # OverflowError: can't serialize ints
< -2**63 or > 2**64 - 1
```
### Operating System
Linux (containerized)
### Versions of Apache Airflow Providers
apache-airflow-providers-celery
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Kubernetes, CeleryExecutor.
### Anything else?
Upstream msgspec limitation reference:
https://github.com/msgspec/msgspec/issues/377
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
