Dandandan commented on issue #16206: URL: https://github.com/apache/datafusion/issues/16206#issuecomment-2923526973
Yeah - one of the options to optimize join is by avoiding the `take` + `concat` (which happens when generating fewer rows per input batch in join, so I think that could use a similar strategy). Ideally, we also would avoid `concat_batches` in the start (and rather maintain a`Vec<RecordBatch>`). This was changed in the early days (by me) as it used to be slow. But now that the hashmap is basically only comparing hash values during probe, I think we can try and avoid the `concat_batches` (and store a u32 / u32 of batch_id + index instead of a u64 index). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
