peterxcli commented on issue #23376: URL: https://github.com/apache/datafusion/issues/23376#issuecomment-5035732026
I did a very rough brainstorm with some visualization, so basically, IIUC,
we need new udf that looks like `lowerbound` to replace the `modulo expr` part
of `CASE hash` in the context of hash join.
(right-bottom part)
```
hash join with partition
┌─────────────┐ ┌────────────┐ ┌────────────────┐
┌───────────────┐
│ 0 │ │ │ │ │
│ │
│ │ │ │ │ │
│ │
└─────▲───────┘ └────────────┘ └────────────────┘
└───────────────┘
│
│ ┌───────────────────┐
│ │ │
│ │ hash join │
arrow batch 1 │with validablity bitmask───────────────┘
df["left_key"] % 4 = 0
└───────────┐
│
│
│
│
┌──────┼────────────────────────────────────┐
│ │ key % 4:
hash partotion routting
│ │ range
partition: split points + binary search on the join key
┌───────►│ │
│ │ repartition exec │
│ │ routing │ dynamic
filter:
│ └───────────────────────────────────────▲───┘ hash
part.:CASE key % 4 WHEN 0 THEN evulate stats ... etc
│ │ range
partition: CASE ???? WHEN 0 THEN evulate stats... etc
│ arrow batch 1 │
│ │
┌───────┼──────────┐ ┌───────────────────┐ │
│ │ │ │ │
index: 0 1 2
│ data source 1 │ │ data source 2 ┼───────┘ spilt
point: 2 5 9
│ data file 1 │ │ data file 2 │
└──────────▲───────┘ └──────────▲────────┘
│ │
│ │
hash partition dynamic filter:
│ │
CASE key % 4
│ │
WHEN 0 THEN ...
│ │
WHEN 1 THEN ...
│ │
└──────────────────┬─────────┴─────────┬───
range partition dynamic filter:
│ │
CASE lowerbound(key, split_point)
│ S3 │
WHEN 0 THEN ...
│ │
WHEN 1 THEN ...
│ │
2
└───────────────────┘
3
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
