[ https://issues.apache.org/jira/browse/ARROW-12428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17325004#comment-17325004 ]
David Li edited comment on ARROW-12428 at 4/19/21, 12:39 PM: ------------------------------------------------------------- {noformat} Whole file: Pandas/S3FS (no pre-buffer, no readahead): 692.2505334559828 seconds Pandas/S3FS (no pre-buffer, readahead): 99.55904859001748 seconds Pandas/S3FS (pre-buffer, no readahead): 39.282157234149054 seconds Pandas/S3FS (pre-buffer, readahead): 41.564441804075614 seconds PyArrow (no pre-buffer): 242.97687190794386 seconds PyArrow (pre-buffer): 39.5321765630506 seconds === Column selection: Pandas/S3FS (no pre-buffer, no readahead): 153.64498204295523 seconds Pandas/S3FS (no pre-buffer, readahead): 82.44589220592752 seconds Pandas/S3FS (pre-buffer, no readahead): 114.55768134980462 seconds Pandas/S3FS (pre-buffer, readahead): 133.1232347697951 seconds PyArrow (no pre-buffer): 54.11452938010916 seconds PyArrow (pre-buffer): 12.865494727157056 seconds {noformat} {code:python} import time import pandas as pd import pyarrow.fs import pyarrow.parquet as pq columns = ['vendor_id', 'pickup_latitude', 'pickup_longitude', 'extra'] print("Whole file:") start = time.monotonic() df = pd.read_parquet("s3://ursa-labs-taxi-data/2012/01/data.parquet", storage_options={ 'default_block_size': 1, # 0 is ignored 'default_fill_cache': False, }, pre_buffer=False) duration = time.monotonic() - start print("Pandas/S3FS (no pre-buffer, no readahead):", duration, "seconds") start = time.monotonic() df = pd.read_parquet("s3://ursa-labs-taxi-data/2012/01/data.parquet", pre_buffer=False) duration = time.monotonic() - start print("Pandas/S3FS (no pre-buffer, readahead):", duration, "seconds") start = time.monotonic() df = pd.read_parquet("s3://ursa-labs-taxi-data/2012/01/data.parquet", storage_options={ 'default_block_size': 1, # 0 is ignored 'default_fill_cache': False, }, pre_buffer=True) duration = time.monotonic() - start print("Pandas/S3FS (pre-buffer, no readahead):", duration, "seconds") start = time.monotonic() df = pd.read_parquet("s3://ursa-labs-taxi-data/2012/01/data.parquet", pre_buffer=True) duration = time.monotonic() - start print("Pandas/S3FS (pre-buffer, readahead):", duration, "seconds") start = time.monotonic() df = pq.read_pandas("s3://ursa-labs-taxi-data/2012/01/data.parquet", pre_buffer=False).to_pandas() duration = time.monotonic() - start print("PyArrow (no pre-buffer):", duration, "seconds") start = time.monotonic() df = pq.read_pandas("s3://ursa-labs-taxi-data/2012/01/data.parquet", pre_buffer=True).to_pandas() duration = time.monotonic() - start print("PyArrow (pre-buffer):", duration, "seconds") print("===") print("Column selection:") start = time.monotonic() df = pd.read_parquet("s3://ursa-labs-taxi-data/2012/01/data.parquet", storage_options={ 'default_block_size': 1, # 0 is ignored 'default_fill_cache': False, }, columns=columns, pre_buffer=False) duration = time.monotonic() - start print("Pandas/S3FS (no pre-buffer, no readahead):", duration, "seconds") start = time.monotonic() df = pd.read_parquet("s3://ursa-labs-taxi-data/2012/01/data.parquet", columns=columns, pre_buffer=False) duration = time.monotonic() - start print("Pandas/S3FS (no pre-buffer, readahead):", duration, "seconds") start = time.monotonic() df = pd.read_parquet("s3://ursa-labs-taxi-data/2012/01/data.parquet", storage_options={ 'default_block_size': 1, # 0 is ignored 'default_fill_cache': False, }, columns=columns, pre_buffer=True) duration = time.monotonic() - start print("Pandas/S3FS (pre-buffer, no readahead):", duration, "seconds") start = time.monotonic() df = pd.read_parquet("s3://ursa-labs-taxi-data/2012/01/data.parquet", columns=columns, pre_buffer=True) duration = time.monotonic() - start print("Pandas/S3FS (pre-buffer, readahead):", duration, "seconds") start = time.monotonic() df = pq.read_pandas("s3://ursa-labs-taxi-data/2012/01/data.parquet", columns=columns, pre_buffer=False).to_pandas() duration = time.monotonic() - start print("PyArrow (no pre-buffer):", duration, "seconds") start = time.monotonic() df = pq.read_pandas("s3://ursa-labs-taxi-data/2012/01/data.parquet", columns=columns, pre_buffer=True).to_pandas() duration = time.monotonic() - start print("PyArrow (pre-buffer):", duration, "seconds") {code} was (Author: lidavidm): {noformat} Whole file: Pandas/S3FS (no pre-buffer, no readahead): 692.2505334559828 seconds Pandas/S3FS (no pre-buffer, readahead): 99.55904859001748 seconds Pandas/S3FS (pre-buffer, no readahead): 39.282157234149054 seconds Pandas/S3FS (pre-buffer, readahead): 41.564441804075614 seconds PyArrow (no pre-buffer): 242.97687190794386 seconds PyArrow (pre-buffer): 39.5321765630506 seconds === Column selection: Pandas/S3FS (no pre-buffer, no readahead): 153.64498204295523 seconds Pandas/S3FS (no pre-buffer, readahead): 82.44589220592752 seconds Pandas/S3FS (pre-buffer, no readahead): 114.55768134980462 seconds Pandas/S3FS (pre-buffer, readahead): 133.1232347697951 seconds PyArrow (no pre-buffer): 54.11452938010916 seconds PyArrow (pre-buffer): 12.865494727157056 seconds {noformat} {code:python} import time import pandas as pd import pyarrow.fs import pyarrow.parquet as pq path = "/home/lidavidm/Documents/ursa-labs-taxi-data/2009/04/data.parquet" columns = ['vendor_id', 'pickup_latitude', 'pickup_longitude', 'extra'] print("Whole file:") start = time.monotonic() df = pd.read_parquet("s3://ursa-labs-taxi-data/2012/01/data.parquet", storage_options={ 'default_block_size': 1, # 0 is ignored 'default_fill_cache': False, }, pre_buffer=False) duration = time.monotonic() - start print("Pandas/S3FS (no pre-buffer, no readahead):", duration, "seconds") start = time.monotonic() df = pd.read_parquet("s3://ursa-labs-taxi-data/2012/01/data.parquet", pre_buffer=False) duration = time.monotonic() - start print("Pandas/S3FS (no pre-buffer, readahead):", duration, "seconds") start = time.monotonic() df = pd.read_parquet("s3://ursa-labs-taxi-data/2012/01/data.parquet", storage_options={ 'default_block_size': 1, # 0 is ignored 'default_fill_cache': False, }, pre_buffer=True) duration = time.monotonic() - start print("Pandas/S3FS (pre-buffer, no readahead):", duration, "seconds") start = time.monotonic() df = pd.read_parquet("s3://ursa-labs-taxi-data/2012/01/data.parquet", pre_buffer=True) duration = time.monotonic() - start print("Pandas/S3FS (pre-buffer, readahead):", duration, "seconds") start = time.monotonic() df = pq.read_pandas("s3://ursa-labs-taxi-data/2012/01/data.parquet", pre_buffer=False).to_pandas() duration = time.monotonic() - start print("PyArrow (no pre-buffer):", duration, "seconds") start = time.monotonic() df = pq.read_pandas("s3://ursa-labs-taxi-data/2012/01/data.parquet", pre_buffer=True).to_pandas() duration = time.monotonic() - start print("PyArrow (pre-buffer):", duration, "seconds") print("===") print("Column selection:") start = time.monotonic() df = pd.read_parquet("s3://ursa-labs-taxi-data/2012/01/data.parquet", storage_options={ 'default_block_size': 1, # 0 is ignored 'default_fill_cache': False, }, columns=columns, pre_buffer=False) duration = time.monotonic() - start print("Pandas/S3FS (no pre-buffer, no readahead):", duration, "seconds") start = time.monotonic() df = pd.read_parquet("s3://ursa-labs-taxi-data/2012/01/data.parquet", columns=columns, pre_buffer=False) duration = time.monotonic() - start print("Pandas/S3FS (no pre-buffer, readahead):", duration, "seconds") start = time.monotonic() df = pd.read_parquet("s3://ursa-labs-taxi-data/2012/01/data.parquet", storage_options={ 'default_block_size': 1, # 0 is ignored 'default_fill_cache': False, }, columns=columns, pre_buffer=True) duration = time.monotonic() - start print("Pandas/S3FS (pre-buffer, no readahead):", duration, "seconds") start = time.monotonic() df = pd.read_parquet("s3://ursa-labs-taxi-data/2012/01/data.parquet", columns=columns, pre_buffer=True) duration = time.monotonic() - start print("Pandas/S3FS (pre-buffer, readahead):", duration, "seconds") start = time.monotonic() df = pq.read_pandas("s3://ursa-labs-taxi-data/2012/01/data.parquet", columns=columns, pre_buffer=False).to_pandas() duration = time.monotonic() - start print("PyArrow (no pre-buffer):", duration, "seconds") start = time.monotonic() df = pq.read_pandas("s3://ursa-labs-taxi-data/2012/01/data.parquet", columns=columns, pre_buffer=True).to_pandas() duration = time.monotonic() - start print("PyArrow (pre-buffer):", duration, "seconds") {code} > [Python] pyarrow.parquet.read_* should use pre_buffer=True > ---------------------------------------------------------- > > Key: ARROW-12428 > URL: https://issues.apache.org/jira/browse/ARROW-12428 > Project: Apache Arrow > Issue Type: Improvement > Components: Python > Reporter: David Li > Assignee: David Li > Priority: Major > Labels: pull-request-available > Fix For: 5.0.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > If the user is synchronously reading a single file, we should try to read it > as fast as possible. The one sticking point might be whether it's beneficial > to enable this no matter the filesystem or whether we should try to only > enable it on high-latency filesystems. -- This message was sent by Atlassian Jira (v8.3.4#803005)