mbutrovich opened a new issue, #7697:
URL: https://github.com/apache/datafusion-comet/issues/7697

   ### What is the problem the feature request solves?
   
   Several of Comet's own loops divide every row by a divisor that is fixed for 
the whole batch or the whole shuffle. Each row pays for a hardware divide, or 
for a `__divti3` or `i256` long-division call. apache/arrow-rs#11421 adds 
precomputed divisors to `arrow-buffer`, which turn each division into a high 
multiply and a shift. Its prototype measured `pmod`-style `u32 % 13` at 4.0x 
and Comet's `i256` multiply at 2.7x to 4.2x.
   
   These are the loops that divide by a fixed divisor:
   
   - 
[`pmod`](https://github.com/apache/datafusion-comet/blob/d187e6823f9052e7331e8a810b08a1202debb91c/native/shuffle/src/comet_partitioning.rs#L123-L129)
 divides each row's `i32` hash by the partition count to pick its shuffle 
partition.
   - 
[`div_round_half_up`](https://github.com/apache/datafusion-comet/blob/d187e6823f9052e7331e8a810b08a1202debb91c/native/spark-expr/src/math_funcs/wide_decimal_binary_expr.rs#L121-L146)
 divides each `i256` result of `WideDecimalBinaryExpr` by `10^k` ([add and 
subtract](https://github.com/apache/datafusion-comet/blob/d187e6823f9052e7331e8a810b08a1202debb91c/native/spark-expr/src/math_funcs/wide_decimal_binary_expr.rs#L255),
 
[multiply](https://github.com/apache/datafusion-comet/blob/d187e6823f9052e7331e8a810b08a1202debb91c/native/spark-expr/src/math_funcs/wide_decimal_binary_expr.rs#L293)).
   - The decimal to integer casts divide each `i128` by `10^scale` 
([`cast_decimal_to_int16_down`](https://github.com/apache/datafusion-comet/blob/d187e6823f9052e7331e8a810b08a1202debb91c/native/spark-expr/src/conversion_funcs/numeric.rs#L545),
 
[`cast_decimal_to_int32_up`](https://github.com/apache/datafusion-comet/blob/d187e6823f9052e7331e8a810b08a1202debb91c/native/spark-expr/src/conversion_funcs/numeric.rs#L600)).
   - `ceil` and `floor` on decimals divide each `i128` by `10^scale` when the 
scale isn't a compile-time constant 
([`decimal_round_f`](https://github.com/apache/datafusion-comet/blob/d187e6823f9052e7331e8a810b08a1202debb91c/native/spark-expr/src/math_funcs/ceil_floor.rs#L214-L218)).
   - `round` on decimals divides each `i128` by `10^k` 
([`decimal_round_f`](https://github.com/apache/datafusion-comet/blob/d187e6823f9052e7331e8a810b08a1202debb91c/native/spark-expr/src/math_funcs/round.rs#L218-L233)).
   
   ### Describe the potential solution
   
   Once Comet upgrades to an arrow release that includes them, build a divisor 
once per batch (once per shuffle for `pmod`) and divide each row with it:
   
   - [ ] `pmod` with `DivisorI32`.
   - [ ] `div_round_half_up` with the `i256` divisor, and with `DivisorI128` 
for batches whose values fit in `i128`.
   - [ ] The decimal to integer casts with `DivisorI128`, and with `DivisorI64` 
for batches whose values fit in `i64`.
   - [ ] Decimal `ceil` and `floor` the same way.
   - [ ] Decimal `round` the same way.
   
   apache/arrow-rs#11421 lists what a kernel change must do, and the same 
applies here. Each change should report benchmark bot numbers with 
`-Ctarget-cpu=x86-64-v3` and `-Ctarget-cpu=neoverse-n1`, keep today's behavior 
for nulls and errors in ANSI mode, and include cases with more than half of the 
values null.
   
   Arrow kernels that adopt divisors under apache/arrow-rs#11421, such as 
integer `/` and `%` by a scalar (apache/arrow-rs#12076), need no change in 
Comet.
   
   ### Additional context
   
   The `i32` and `i64` divisors are in apache/arrow-rs#11467, and the `i128` 
and `i256` divisors are in apache/arrow-rs#12232. Comet won't port them ahead 
of the arrow release, so that Comet and arrow-rs share one implementation.
   
   ### Willingness to contribute
   
   I can contribute a fix for this bug independently
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to