rdbisme commented on issue #38245: URL: https://github.com/apache/arrow/issues/38245#issuecomment-1761371760
> 🤔I think it's expected(though this sounds horrible) > > Your file including 5000 columns with dictionary encoding and SNAPPY compression. The memory might consitituted by: > > 1. Reading the footer( the peek memory usage might grows higher when deserialize thrift for the footer, maybe you can try to disable statistics, this might make reading it a bit faster) > > 2. Reading "Physical" Page, which including dictionary page and data page for every column, and because they're compressed, for each column, a decompressed buffer is used. This make the memory usage larger > > 3. when `read_table` is called, the data in parquet would be "materialized" to arrow table, which make it worse... > > > Some advices: > > 1. Try to disable statistics if you don't want use it. If you really wants statistics you can try to disable statistics for some columns. This might not helps a lot, but it may reduce some size of this file. > > 2. Maybe trying to testing PLAIN encoding or disable compression. I'm not advice to just shift to PLAIN or disable compression, but you can have a try. It might works well(and might make it worse) Thanks for your reply! I must admit that it's a bit underwhelming. ~5x memory increase look enormous to me (considering the parquet is also almost 175M on disk too)... I'll try to play with compression parameters, but the dataframe has been serialized with standard `pd.to_parquet` without additional arguments. Are we really sure this is expected behaviour? :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
