Fokko opened a new pull request, #6673: URL: https://github.com/apache/iceberg/pull/6673
PyArrow is still sluggish when it comes into opening files, and we still see many requests being made to S3. This PR removes the Dataset, and uses the lower read_table API. Since the read_table API requires to pass in filters in the DNF form, we need to do some additional conversion. This PR reduces the number of calls from 203 to 165. Requests log: Before: https://gist.github.com/Fokko/96b4d5b65ec85c95d6e875f6ec19bf50 After: https://gist.github.com/Fokko/282f6b803d83a830465d97f64cf10057 Query used: ```python from pyiceberg.catalog import load_catalog catalog = load_catalog('local') tbl = catalog.load_table('nyc.taxis') from pyiceberg.expressions import GreaterThanOrEqual, LessThanOrEqual, And sc = tbl.scan(row_filter=And( GreaterThanOrEqual("tpep_pickup_datetime", "2022-04-01T00:00:00.000000+00:00"), LessThanOrEqual("tpep_pickup_datetime", "2022-04-28T00:00:00.000000+00:00"), )).to_arrow() ``` Logs: Also, the wall clock time is lower: ```python ➜ iceberg git:(fd-optimize-pyarrow) ✗ time python3 /tmp/vo.py python3 /tmp/vo.py 2.38s user 2.75s system 31% cpu 16.067 total python3 /tmp/vo.py 2.55s user 2.57s system 36% cpu 14.097 total python3 /tmp/vo.py 2.60s user 2.57s system 32% cpu 15.954 total ``` ```python ➜ iceberg git:(master) time python3 /tmp/vo.py python3 /tmp/vo.py 2.54s user 2.71s system 28% cpu 18.499 total python3 /tmp/vo.py 2.75s user 2.56s system 24% cpu 21.547 total python3 /tmp/vo.py 2.75s user 2.95s system 17% cpu 32.554 total ``` Keep in mind that these requests are across the great ocean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
