alamb commented on issue #11388: URL: https://github.com/apache/datafusion/issues/11388#issuecomment-3469599698
> P.S. @alamb: On a related note, in your podcast with [Developer Voices](https://www.youtube.com/watch?v=8QNNCr8WfDM&t=2105s), you were asked about the RecordBatch size of ~8000 rows and explained that research papers show why 4k-16k is optimal. I'm very interested in this! If you could share a link to that research, I'd appreciate it. I don't plan to put it in the docs (yet), but I'm curious. The paper I had in mind is [Everything You Always Wanted to Know About Compiled and Vectorized Queries But Were Afraid to Ask](https://www.vldb.org/pvldb/vol11/p2209-kersten.pdf) Specifically this chart <img width="363" height="216" alt="Image" src="https://github.com/user-attachments/assets/d03cca92-0500-452b-9370-6e0a32499835"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
