pitrou commented on issue #38389: URL: https://github.com/apache/arrow/issues/38389#issuecomment-1824684681
Okay, I took a quick look. One thing that takes some time is dictionary decoding for the dictionary-encoded columns. You can keep those columns dictionary-encoded in Arrow by passing `read_dictionary=["l_returnflag", "l_linestatus", "l_shipinstruct", "l_shipmode"]`. This seems to save around 25% CPU time (and also makes the data much more compact in memory). Once that is done, I've got the following breakdown, roughly: - 54% is Snappy decompression - 16% is RLE decoding of dictionary indices - 13% is PLAIN decoding of BYTE_ARRAY values - 10% is PLAIN decoding of FLOAT and DOUBLE values -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
