yihua opened a new pull request, #890:
URL: https://github.com/apache/hudi-rs/pull/890
## Description
closes #889
`Table::read` read its file slices with `try_join_all` on the awaiting task,
so all decoding ran on one thread, which for the Python binding is the Python
thread. Each slice is now read on its own runtime task, in input order and
within the existing scan memory budget; the task is aborted if the read is
dropped, and the slice is read in place when there is no multi-thread runtime.
When `hoodie.read.file.slice.read.concurrency` is unset, a `Table` read now
defaults to the available cores; the DataFusion scan and metadata-table reads
keep the default of 4.
Full-table reads through the Python binding got 3 to 6 times faster on local
tables of 1M to 10M rows (for example 2.56 s to 0.42 s for a 10M-row
copy-on-write table).
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
`test_read_decodes_file_slices_off_the_calling_thread` blocks on a read of
four file groups from a non-runtime thread and counts the bytes that thread
allocates: about 140 MB before (four times the batches read), about 12 MB
after. Unit tests cover the spawn helper's abort on drop, panic as error, and
no-runtime paths, and the scan budget test now covers both the core-count and
the configured ceiling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
