Rich-T-kid commented on PR #23483: URL: https://github.com/apache/datafusion/pull/23483#issuecomment-4970907345
the test in https://github.com/apache/datafusion/pull/23483/commits/4ca86a1075c6b82447dfc0b677dd256203e644e3 `arrow_cast` builds the dictionary by encoding values in first-seen order. So: ``` ┌──────────────┬────────┬────────┐ │ batch │ key 0 │ key 1 │ ├──────────────┼────────┼────────┤ │ first_batch │ 'west' │ 'east' │ ├──────────────┼────────┼────────┤ │ second_batch │ 'east' │ 'west' │ └──────────────┴────────┴────────┘ ``` Key id 0 means 'west' in the first batch but 'east' in the second. A buggy implementation that groups by raw key id would put those together: so west (group 0) would be seen 6 times while east (group 1) would be seen 2 times. this would be incorrect. A correct implementation looks up the actual string value for each row and groups by that, producing 4 and 4 regardless of what key id was used in each batch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
