mapleFU commented on issue #36940: URL: https://github.com/apache/arrow/issues/36940#issuecomment-1657224386
Parquet will store the file metadata as a thrift. If you have too many columns or too many row groups, the thrift object would be large. By default, during reading, parquet reader will use 1MB as thrift container size limit, 100MB as thrift object size limit. I guess your file footer is too large. You can solve this by: 1. During writing: Make file having less row-groups, or just make file smaller. 2. During reading: you can enlarge the thrift config -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
