SteveLauC commented on issue #5269: URL: https://github.com/apache/arrow-rs/issues/5269#issuecomment-1873865816
> Parquet encoding is not IO-bound, compression especially so, with many of the encodings very expensive to compute. That's right > we want to treat it as an I/O bound task My last statement is probably not that correct, we are aware that encoding/compression is very CPU-bound, and writing the parquet files is I/O-bound, we want to **separate these 2 things**, leaving only the I/O tasks in our runtime. I guess this is very specific to our usage and we should implement our own `AsyncArrowWriter`. Thanks for your reply:) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
