Does boxing a scalar into an array actually build a buffer with the
repeated value, or is it more efficient than that?
Le 29/06/2022 à 17:57, Wes McKinney a écrit :
I'm working on my next PR which addresses the "scalar output modality"
and removes usages of ValueDescr and output shapes in general from the
kernel execution machinery. The ability to provide all scalar input
and scalar output of course is preserved — it's just that the "boxing"
and "unboxing" of the inputs and outputs is happening at a higher
level (i.e. there is only one place that boxes scalars into arrays and
unboxes the output into a scalar) rather than being implemented on a
piecemeal basis in the kernel implementations.
Please keep an eye out for any non-trivial PRs that affect
arrow/compute/kernels since it could cause some rebasing headache. I
am working as fast as I can to complete this next PR.
After this, the next thing is porting the aggregate kernels to use
ExecSpan. Then I think the mass refactoring will cool off and we can
address follow-on improvements like rewriting expression evaluation to
utilize the span data structures to yield performance gains.
On Mon, Jun 13, 2022 at 12:37 PM Wes McKinney <wesmck...@gmail.com> wrote:
I merged the PR a little while ago — thanks for David, Sasha for
helping review. If you have more comments or things you would like to
fix from that PR, please comment there and I can help address them in
follow up PRs.
I'm going to work next on migrating the rest of the kernels to use
ExecSpan (and thus ExecBatchIterator can be removed in favor of
ExecSpanIterator — I'll add benchmarks so we can see the performance
difference), and then my next priority will be removing the ValueDescr
concept and the ValueDescr::SCALAR output path for the
scalar/elementwise kernels which will help us delete a lot of code
I'll attach related Jiras to this umbrella issue:
https://issues.apache.org/jira/browse/ARROW-16755
On Fri, Jun 10, 2022 at 12:56 PM Wes McKinney <wesmck...@gmail.com> wrote:
PR is up: https://github.com/apache/arrow/pull/13364
Look forward to getting this in since there's a bunch of follow on
work that I'd like to get started on ASAP!
On Thu, Jun 9, 2022 at 7:34 AM Wes McKinney <wesmck...@gmail.com> wrote:
I'm making good progress getting my branch PR-ready -- working through
the compute-scalar-test suite and fixing the little things I broke. I
hope I'll have it done by the end of the week.
On Mon, Jun 6, 2022 at 3:21 PM Wes McKinney <wesmck...@gmail.com> wrote:
I created https://issues.apache.org/jira/browse/ARROW-16755 as an
umbrella issue to track improvements to reduce overhead in the
expression and kernel execution machinery. Please help by attaching
related issues and creating new issues for specific individual efforts
here. I'll work as quickly as I can to have my initial patch
ARROW-16756 ready which will unblock the next few projects here
On Mon, Jun 6, 2022 at 10:35 AM Wes McKinney <wesmck...@gmail.com> wrote:
This is definitely only the first stage of cleanup and streamlining —
I anticipate multiple rounds of refactoring (maybe not as invasive and
painful as this one), and this patch I'm not sure will do a lot to
alleviate bottom line expression evaluation overhead but it creates
the environment (i.e. where a whole chain of scalar functions that all
write into preallocated memory can execute without having to touch
shared_ptrs or deal with other objects with excess microperformance
overhead) where such optimization can happen more easily.
On Mon, Jun 6, 2022 at 4:08 AM Antoine Pitrou <anto...@python.org> wrote:
Le 06/06/2022 à 09:34, Sasha Krassovsky a écrit :
Wow that's a lot of progress!
Definitely agree on the scalar outputs point.
One point about the ArraySpan - why does it need to know its data type?
Once a kernel has been resolved by the registry, the kernel will only know
how to execute on the specific type it was resolved for, right?
Because of parametric types for example (e.g. timestamps with a unit and
timezone, or decimals with a precision and scale).
Regards
Antoine.
