PepijnBrasseur opened a new issue, #51495: URL: https://github.com/apache/arrow/issues/51495
### Describe the bug, including details regarding any error messages, version, and platform. <html> <body> <!--StartFragment--><html><head></head><body><h2>Summary</h2> <p>A <code>pyarrow.dataset</code> built from an explicit list of Parquet files, one of which is not a valid Parquet file, is expected to raise on <code>to_table()</code>. It usually does. <strong>Occasionally it instead returns a table containing only the rows of the file(s) that sort before the invalid fragment, with no exception.</strong> It happens only when the files were written immediately before the scan (a fresh directory per iteration); reusing the same files never reproduced it. Seen on Windows and on Linux, threaded and unthreaded.</p> <h2>Environment</h2> | Windows | Linux -- | -- | -- pyarrow | 23.0.1 | 23.0.1 Python | 3.11.1 | 3.11.15 OS | Windows 11, build 26200 (platform.platform() → Windows-10-10.0.26200-SP0) | Ubuntu 24.04.4 LTS (x86_64; a cloud workspace, the same class on every run) filesystem | NTFS (local disk, the system temp directory) | ext4 on a virtual disk, the system temp directory <h2>What was checked</h2> <ul> <li><strong>The file list is complete when the scan is short.</strong> On a truncated read, the listed paths included the invalid file and the dataset held all 5 fragments (fragment count = number of listed files) — the scan stopped early and returned without error. A fragment-count check before the scan therefore cannot detect it.</li> <li><strong>A 0-byte invalid file in the same position raises too</strong> (in the same test shape) — the truncation is not an empty file being skipped.</li> <li><strong>Threading is not the variable:</strong> it reproduces with <code>use_threads=False</code> on both platforms.</li> <li><strong>Freshness matters:</strong> 0 truncations in 3,600 reads of reused files, against the fresh-file rates above. Why fresh files matter is not understood.</li> <li><strong>The shape of the failing fragment matters:</strong> the 25-byte non-Parquet file truncates; a half-written Parquet file raised every time in 6,000 tries. Not understood.</li> </ul> <p>What remains as candidates: the dataset scanner's handling of a fragment-open error, or something in the filesystem beneath it for files closed moments earlier — the two cannot be told apart from here.</p> <h2>Workaround in use</h2> <p>Opening every listed file's footer independently before the scan (<code>pyarrow.parquet.ParquetFile(path).metadata</code>) turns the silent truncation into a raise (0 truncations in 2,000 tries on the Windows reproduction); comparing the rows read with the rows the footers declare backstops the case where a footer opens but its data does not.</p></body></html><!--EndFragment--> </body> </html> ### Component(s) Python, Parquet -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
