Re: Question about large exec batch in acero

That makes perfect sense, esp. seeing the the zero-copy fashion for slicing the big input. Thanks Weston! *Rossi* Weston Pace 于2023年7月3日周一 22:33写道： > > is this overflow considered a bug? Or is large exec batch something that > should be avoided? > > This is not a bug and it is something that s

Re: Question about large exec batch in acero

> is this overflow considered a bug? Or is large exec batch something that should be avoided? This is not a bug and it is something that should be avoided. Some of the hash-join internals expect small batches. I actually thought the limit was 32Ki and not 64Ki because I think there may be some p

Question about large exec batch in acero

Hi folks, I've encountered a bug when doing swiss join using a big exec batch, say, larger than 65535 rows, on the probe side. It turns out to be that in the algorithm, it is using `uint16_t` to represent the index within the probe exec batch (the materialize_batch_ids_buf