kosiew commented on code in PR #26609:
URL: https://github.com/apache/datafusion/pull/26609#discussion_r4582752550


##########
datafusion/physical-expr/src/expressions/case/literal_lookup_table/primitive_lookup_table.rs:
##########
@@ -12,6 +12,26 @@ use half::f16;
 use std::fmt::Debug;
 use std::hash::Hash;
 
+/// The maximum total size in bytes of the literals that are looked up by 
comparing each
+/// value with all of them, rather than with a hash map.
+///
+/// The cost of the comparisons grows with the number of literals and with 
their size,
+/// as fewer wider values fit in a vector register.
+const MAX_LINEAR_LOOKUP_BYTES: usize = 128;
+
+/// The maximum number of literals that are looked up by comparing each value 
with all
+/// of them, for values narrow enough for many of them to fit 
[MAX_LINEAR_LOOKUP_BYTES].
+const MAX_LINEAR_LOOKUP_LITERALS: usize = 32;
+
+/// The maximum number of literals of type `K` that are looked up by comparing 
each
+/// value with all of them, rather than with a hash map.
+fn max_linear_lookup_literals<K>() -> usize {

Review Comment:
   The 128-byte threshold admits eight Decimal128 literals, but the latest 
same-case benchmark regresses from 47.5 to 64.4 µs (~36% slower); earlier local 
runs also showed a reproducible ~45% slowdown with 50% nulls. Please lower the 
wide-key limit or improve the kernel, then benchmark all-valid and nullable 
inputs against hash lookup. A representation-specific four-key cap is a 
reasonable candidate given the measured improvement at four literals, but 
please verify it without unnecessarily reducing the i32/i64 limits.



##########
datafusion/physical-expr/src/expressions/case.rs:
##########
@@ -3289,6 +3289,37 @@ mod tests {
         );
     }
 
+    /// Covers both the linear and the hash lookup of primitive literals, with 
inputs
+    /// spanning several chunks of the linear lookup.
+    #[rstest::rstest]
+    fn test_case_when_literal_lookup_int64_by_number_of_literals(

Review Comment:
   Optional test hardening: could we evaluate a nullable `Int64Array` slice 
starting at a non-byte-aligned offset, with matches and misses spanning the 
256-row boundary? The current test covers chunking but only unsliced arrays; 
reusing the ELSE/no-ELSE helper would verify that values and validity remain 
aligned. No offset bug was found.



##########
datafusion/physical-expr/src/expressions/case/literal_lookup_table/primitive_lookup_table.rs:
##########
@@ -180,7 +269,19 @@ impl_to_hashable_key!(@float | f64 => u64);
 
 #[cfg(test)]
 mod tests {
-    use super::ToHashableKey;
+    use super::{ToHashableKey, max_linear_lookup_literals};
+    use arrow::datatypes::{IntervalMonthDayNano, i256};
+
+    #[test]
+    fn max_linear_lookup_literals_by_type() {
+        assert_eq!(max_linear_lookup_literals::<i8>(), 32);
+        assert_eq!(max_linear_lookup_literals::<u16>(), 32);
+        assert_eq!(max_linear_lookup_literals::<i32>(), 32);
+        assert_eq!(max_linear_lookup_literals::<i64>(), 16);
+        assert_eq!(max_linear_lookup_literals::<i128>(), 8);
+        assert_eq!(max_linear_lookup_literals::<IntervalMonthDayNano>(), 8);
+        assert_eq!(max_linear_lookup_literals::<i256>(), 4);
+    }
     use arrow::array::downcast_primitive;

Review Comment:
   Minor cleanup: `use arrow::array::downcast_primitive;` now appears after the 
new test. Could we move it alongside the other imports at the top of the test 
module?



##########
datafusion/physical-expr/src/expressions/case/literal_lookup_table/primitive_lookup_table.rs:
##########
@@ -180,7 +269,19 @@ impl_to_hashable_key!(@float | f64 => u64);
 
 #[cfg(test)]
 mod tests {
-    use super::ToHashableKey;
+    use super::{ToHashableKey, max_linear_lookup_literals};
+    use arrow::datatypes::{IntervalMonthDayNano, i256};
+
+    #[test]
+    fn max_linear_lookup_literals_by_type() {
+        assert_eq!(max_linear_lookup_literals::<i8>(), 32);
+        assert_eq!(max_linear_lookup_literals::<u16>(), 32);
+        assert_eq!(max_linear_lookup_literals::<i32>(), 32);
+        assert_eq!(max_linear_lookup_literals::<i64>(), 16);
+        assert_eq!(max_linear_lookup_literals::<i128>(), 8);
+        assert_eq!(max_linear_lookup_literals::<IntervalMonthDayNano>(), 8);
+        assert_eq!(max_linear_lookup_literals::<i256>(), 4);

Review Comment:
   Optional coverage: could we add a Decimal256 execution test with four and 
five unique literals, including values with high bits beyond i64, nulls, and 
misses? This would exercise wide-key equality and both sides of the linear/hash 
boundary rather than only asserting the four-key threshold.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to