mapleFU commented on issue #38245: URL: https://github.com/apache/arrow/issues/38245#issuecomment-1759921604
🤔I think it's expected(though this sounds horrible) Your file including 5000 columns with dictionary encoding and SNAPPY compression. The memory might consitituted by: 1. Reading the footer( the peek memory usage might grows higher when deserialize thrift for the footer, maybe you can try to disable statistics, this might make reading it a bit faster) 2. Reading "Physical" Page, which including dictionary page and data page for every column, and because they're compressed, for each column, a decompressed buffer is used. This make the memory usage larger 3. when `read_table` is called, the data in parquet would be "materialized" to arrow table, which make it worse... Some advices: 1. Try to disable statistics if you don't want use it. If you really wants statistics you can try to disable statistics for some columns. This might not helps a lot, but it may reduce some size of this file. 2. Maybe trying to testing PLAIN encoding or disable compression. I'm not advice to just shift to PLAIN or disable compression, but you can have a try. It might works well(and might make it worse) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
