alamb commented on PR #9836: URL: https://github.com/apache/arrow-rs/pull/9836#issuecomment-4347466670
> Would this kind of bounded per-batch parallelism be acceptable in arrow-ipc, or would it introduce any new hidden costs? I'm really unsure of the impacts of parallelism here for the write process and so I'm really curious I don't think adding parallelism (eg. threasd) to the underlying library calls acceptable as it comes with threading and memory overhead (more buffering) that are not appropriate for all usecases What probably would be useful is if you could find some way (maybe an example) to show people who wanted to make this tradeoff how it could be done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
