jholownia opened a new issue, #50605:
URL: https://github.com/apache/arrow/issues/50605
### Describe the bug, including details regarding any error messages,
version, and platform.
## Summary
With pyarrow==25.0.0 on aarch64 Linux with an SVE-capable CPU (AWS Graviton,
ECS Fargate), reading a double column from a Parquet file silently returns
wrong values — no error, no warning. The same file read on the same machine can
yield either correct or corrupt values across different process invocations,
and setting ARROW_USER_SIMD_LEVEL=NONE makes the corruption disappear entirely.
This points at the ARM64 SVE runtime dispatch newly added in 25.0.0 (GH-49756),
possibly interacting with the reworked CPU detection (GH-49940).
This is a silent data-integrity bug: correct-looking programs read
numerically wrong floating-point data with no indication anything went wrong.
It only surfaced accidentally in data integrity checks.
## Environment
pyarrow: 25.0.0 (manylinux aarch64 wheel)
numpy: 2.4.6, pandas: 3.0.3
OS/arch: Linux aarch64, AWS Graviton (ECS Fargate); /proc/cpuinfo Features
include sve
Python: 3.12
## Minimal reproduction
```Python
import numpy as np, pyarrow as pa, pyarrow.parquet as pq
rng = np.random.default_rng(42)
vals = rng.standard_normal(20000) * 100 + 50
vals[rng.random(20000) < 0.2] = np.nan
pq.write_table(pa.table({"x": vals}), "/tmp/syn.parquet")
ref = float(np.nansum(vals)). # ground truth from the in-memory array
got = float(np.nansum(
pq.read_table("/tmp/syn.parquet").column("x").to_numpy(zero_copy_only=False)
))
print(f"ref={ref:.3f} got={got:.3f} "
f"{'OK' if abs(got - ref) <= 1e-6 * abs(ref) else 'CORRUPT'}")
```
| Condition | Result |
| -------- | ------- |
| default SIMD | got=1393219.089 (CORRUPT) on 5/8 hosts; got=808119.811
(OK) on 3/8 |
| ARROW_USER_SIMD_LEVEL=NONE | got=808119.811 (OK) on 8/8 |
- The corruption is intermittent across process starts / hosts even though
all hosts report sve - but it is deterministic within a process (every read in
a given process agrees) and the wrong value is stable (always 1393219.089 for
this input).
- We first hit this on real data: a byte-identical Parquet file (verified by
md5 == S3 ETag) decoded to different double sums on different Fargate tasks;
the correct value matched an independent ground truth, the corrupt one did not.
ARROW_USER_SIMD_LEVEL=NONE fixed every affected host.
- Downgrading to pyarrow==24.0.0 (no SVE dispatch path) reads correctly on
the same hosts.
## Workarounds
- `ARROW_USER_SIMD_LEVEL=NONE`
- Pin pyarrow<25
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
