deepkilo commented on issue #50482:
URL: https://github.com/apache/arrow/issues/50482#issuecomment-6245996306
Additional Linux evidence from compatibility testing of a
quantitative-research runtime. This was an observed process-exit failure
investigated back to native cleanup, not a finding from source review alone.
Our reproducer uses generated files and a Python filesystem wrapper; it is
related evidence, not a claim that every file-object case has exactly the same
ownership path.
With official PyArrow 26, a malformed fragment raises the expected
`ArrowInvalid`, but the process can still fail during interpreter shutdown. On
Linux x86_64 our reduced case aborted on CPython 3.12.15 (12/12 four-fragment
runs and 12/12 eight-fragment runs), timed out on 3.13.16 (4/4), and also timed
out on unmodified 26 with 3.15.0 (3/3). Upgrading Python alone did not repair
it.
Self-contained reduced case; run in fresh processes with a timeout, since
the symptom is at exit:
```python
import tempfile
from pathlib import Path
import fsspec
import pyarrow as pa
import pyarrow.dataset as ds
import pyarrow.fs as fs
import pyarrow.parquet as pq
root = Path(tempfile.mkdtemp())
pq.write_table(pa.table({'value': [1, 2, 3]}), root / 'valid.parquet')
(root / 'bad.parquet').write_bytes(b'not a parquet file')
filesystem = fs.PyFileSystem(fs.FSSpecHandler(fsspec.filesystem('file')))
paths = [str(root / name) for name in
['valid.parquet', 'bad.parquet', 'valid.parquet', 'valid.parquet']]
dataset = ds.dataset(paths, filesystem=filesystem, format='parquet',
schema=pa.schema([('value', pa.int64())]),
exclude_invalid_files=False)
try:
dataset.to_table(use_threads=True, fragment_readahead=4)
except pa.ArrowInvalid:
pass
else:
raise AssertionError('Expected the malformed-fragment error')
```
Our captured native stack passes through callback-capture destruction,
`MergedGenerator` / `SerialReadahead`, the Parquet reader / `ReadRangeCache`,
and `PyBuffer::~PyBuffer`, then `PyGILState_Ensure` during `Py_FinalizeEx`. On
3.12 the forced unwind crosses a noexcept boundary and terminates; newer Python
parks the late worker while Arrow's pool shutdown joins it.
This also relates to #46945: checking `Py_IsInitialized` or finalization
before acquiring the GIL leaves a check-to-acquire race. A local candidate uses
Python 3.15's PEP 788 view/token admission API for `PyBuffer`, `NumPyBuffer`,
and `OwnedRefNoGIL` cleanup. The complete Python-native wheel was rebuilt, with
the eight computational Arrow/Parquet DSOs retained byte-for-byte. Eighteen
bounded subprocess checks then exited normally, including malformed-input and
valid-input controls; the malformed fragment still raises rather than being
suppressed.
That candidate is qualified only for main-interpreter, GIL CPython 3.15, not
older Python, free-threaded builds or subinterpreters. A failed admission can
also mean allocation failure, so the candidate's skipped cleanup may leak a
reference in that case. I am sharing the mechanism and verification boundary
for discussion, not claiming a universally ready upstream fix or a newly
introduced 26 regression.
Disclosure: OpenAI Codex (GPT-6 family; exact variant/effort unavailable in
this session) assisted with investigation, tool-driven reproduction, candidate
work and this text. The stated runs and native stacks are recorded execution
evidence, not a claim that the operator manually performed or reviewed every
step. No maintainer ping or competing PR is intended.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
