dwsmith1983 opened a new pull request, #7180: URL: https://github.com/apache/datafusion-comet/pull/7180
## Which issue does this PR close? Closes #5657. ## Rationale for this change For a file with an on-disk deletion vector, scan preparation decoded the DV blob before fetching the Parquet footer, though the footer needs nothing from the DV. Each such file paid two round trips in a row. ## What changes are included in this PR? - On-disk DVs: the DV read and the footer read run together (`futures::try_join!`). Descriptor checks still precede any I/O, and memory is still reserved after both reads finish. - Inline DVs are unchanged: a corrupt one still fails before any footer read. - Concurrency stays at 8 files per partition, now with up to two reads per file. - When both reads fail, the first error to complete is returned. - When the DV fails, a valid footer may stay in the metadata cache, keyed by the file's `ObjectMeta`, so it stays correct. Store delaying each request, median of 6 alternating runs: | delay | files | DV | before | after | |---|---|---|---|---| | 20 ms | 8 | sparse | 52 ms | 28 ms | | 20 ms | 32 | dense | 306 ms | 201 ms | | 80 ms | 32 | sparse | 678 ms | 343 ms | | 80 ms | 32 | dense | 839 ms | 454 ms | The pool's peak reservation is identical in every configuration. Max RSS of the test process, one configuration per process: 68.5 to 75.4 MiB (8 sparse files), 78.3 to 85.1 MiB (8 dense), 118.7 to 126.3 MiB (32 dense), and 42.3 to 43.8 MiB with no files. The gap does not grow with file count or density. Against a real store, the in-flight cost per partition is bounded by 8 files x (a footer read of up to the 512 KiB size hint + the framed DV blob). ## How are these changes tested? New tests hold the DV read and check that the footer read is issued, and the reverse; sequential reads in either order fail one of them. Others cover a bad CRC with a slow footer, a missing data file and a corrupt inline DV, each leaving nothing reserved. The Delta DV scan tests pass on Spark 3.5 and 4.0, and the MinIO S3 suite on 3.5. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
