karlovnv commented on issue #7000:
URL: https://github.com/apache/datafusion/issues/7000#issuecomment-2094202184
Hi! There is great job done here!
I faced with an issues with CoalesceBatches: it seams that there is a
performance killer somewhere in CoalesceBatchesStream.
It's spending too much time in arrow_select::concat::concat (especially in
arrow_data::transform::variable_size::build_extend::_{{closure}}). I think that
it's an issue of expanding MutableBatch (or not vectorized copying)
In Query:
```
select count(*)
from SomeTable as a
join AnotherTable as b
on a.c1 = b.c1
and a.c2 = b.c2
and a.c3 = b.c3
and a.c4 = b.c4
SomeTable consists of 200m rows
Anothertable -- 15m rows
```
**17%** of all execution time is CoalesceBatchesStream:
```
coalesce_batches::CoalesceBatchesStream::poll_next (55,950 samples, 17.03%)
|--arrow_select::concat::concat_batches (36,618 samples, 11.14%)
|--arrow_select::concat::concat (30,947 samples, 9.42%)
```
<img width="1152" alt="image"
src="https://github.com/apache/datafusion/assets/3950601/330792ac-0cbd-4508-a56e-33983c6207bb";>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
