cyb70289 commented on a change in pull request #10174:
URL: https://github.com/apache/arrow/pull/10174#discussion_r621960840
##########
File path: cpp/src/arrow/compute/kernels/scalar_set_lookup.cc
##########
@@ -75,6 +95,9 @@ struct SetLookupState : public KernelState {
using MemoTable = typename HashTraits<Type>::MemoTableType;
MemoTable lookup_table;
+ // When there are duplicates in value_set, the MemoTable indices must
+ // be mapped back to indices in the value_set.
+ std::vector<int32_t> memo_index_to_value_index;
Review comment:
If there are not many duplicated values (common case?), looks it's
possible to compress this table by storing only indices where duplicated values
are met.
A quick example. I think we can leave it as follow up task if it works.
```
values a a b a b b c
value_index 0 1 2 3 4 5 6
// memo index to value index mapping
memo_index_to_value_index[0] = 0 --> a
memo_index_to_value_index[1] = 2 --> b
memo_index_to_value_index[2] = 6 --> c
// duplicated value indices in sorted order
duplicated_indices = (1, 3, 4, 5)
// lookup 'a'
memo_index = lookup('a') = 0
0 < duplicated_indices[0]
return 0
// lookup 'b'
memo_index = lookup('b') = 1
1 >= duplicated_indices[0]
memo_index++ // = 2
2 < duplicated_indices[1]
return 2
// lookup 'c'
memo_index = lookup('c') = 2
2 >= duplicated_indices[0]
memo_index++ // = 3
3 >= duplicated_indices[1]
memo_index++ // = 4
4 >= duplicated_indices[2]
memo_index++ // = 5
5 >= duplicated_indices[3]
memo_index++ // = 6
return 6
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
