omkar-334 commented on PR #54919:
URL: https://github.com/apache/arrow/pull/54919#issuecomment-6327540411
My understanding before opening the PR -
Function `read_table` supports 2 types of datasets
1. `dataset = ParquetDataset`
- `self._dataset = ds.dataset`
This function returns multiple types, not so straightforward. The
class `Dataset` (_dataset.pyx) has three types of datasets -
```
classes = {
'union': UnionDataset,
'filesystem': FileSystemDataset,
'in-memory': InMemoryDataset,
}
```
- `_filesystem_dataset` -> `FileSystemDatasetFactory`
- `_union_dataset` -> `UnionDataset`
- `_in_memory_dataset` -> `InMemoryDataset`
We don;t need to change code for all the classes, the below unction
makes it easy for us
- `table = self._dataset.to_table` -> `to_table` (_dataset.pyx)
This function in pyx supports MemoryPool -> we just need to edit this.
2. `dataset = ParquetFile`
- `self.reader = ParquetReader()`
- python/pyarrow/_parquet.pyx - ParquetReader supports `CMemoryPool*
pool`
- `self.reader.read_all`
- `return dataset.read`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
