JingsongLi commented on code in PR #10225:
URL: https://github.com/apache/paimon/pull/10225#discussion_r4323132331
##########
paimon-python/pypaimon/read/reader/format_pyarrow_reader.py:
##########
@@ -559,8 +576,7 @@ def __init__(self, file_io: FileIO, file_format: str,
file_path: str,
and self._selected_shared_map_paths)):
import pyarrow.parquet as pq
# ParquetFile(filesystem=...) is unavailable in PyArrow 6.
- self._parquet_source = file_io.filesystem.open_input_file(
- file_path_for_pyarrow)
+ self._parquet_source = _open_pyarrow_input_file(file_io, file_path)
Review Comment:
[P2] Forward the known file size to the direct Parquet reader
`SplitRead` already supplies `DataFileMeta.file_size`, but this call drops
it when row selection, bounded VARIANT or selected MAP reads open their
separate Parquet source. The resulting `CachingInputStream` calls the
delegate's `get_file_size()` before reading cached blocks. With the actual
Jindo handler, the same known-size selected-row read makes 0 backend metadata
requests without the cache wrapper and 1 with it.
I also reproduced this with a fully cached Parquet file: perform a full scan
with `file_size` supplied, remove the source and clear the parsed-footer cache,
then read again with the same `file_size` and `row_indices=[1]`. The full warm
scan succeeds, but the selected-row scan raises `FileNotFoundError` while
querying the source size. Forwarding the known size makes the selected-row scan
succeed as well.
Please pass `file_size` to `_open_pyarrow_input_file` here and add coverage
combining a known size with selected-row warm reads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
