stuhood commented on code in PR #22207:
URL: https://github.com/apache/datafusion/pull/22207#discussion_r3249785424
##########
datafusion/physical-plan/src/repartition/mod.rs:
##########
@@ -600,6 +600,11 @@ impl BatchPartitioner {
num_input_partitions,
))
}
+ Partitioning::Expr(_) => {
+ not_impl_err!(
+ "Expression partitioning is not supported by
RepartitionExec"
+ )
+ }
Review Comment:
So, it's worth discussing this in more detail I think.
`Expr` partitioning is much, much more general than `Range` partitioning.
In `Range` partitioning, deciding which partition a row maps to involves
either a binary search or sorted map lookup. But in `Expr` partitioning, it
will always be a linear scan through the expressions, unless the consumer has
reverse-engineered the fact that it is actually Range partitioning under the
hood.
So this operator will be much more expensive than it might be otherwise.
What is the reasoning around using expressions here, and not literally
ranges?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
