sunchao opened a new issue, #7189: URL: https://github.com/apache/datafusion-comet/issues/7189
### What is the problem the feature request solves? A native left-semi sort-merge join returns each matching left row once, regardless of how many right rows match it. When many right rows carry the same key, Comet still sorts and retains every copy. For example, an events table joined to a list of observed account IDs can sort thousands of copies of each ID to answer a membership question. ### Describe the potential solution For an unfiltered left-semi join with an explicit native sort on the right, remove repeated integer key tuples within each batch before sorting. Preserve duplicate left rows and Spark's null semantics. Keep the optimization opt-in, bypass it when the keys are mostly unique or memory is tight, and expose counters for the number of rows entering and leaving the reducer. ### Willingness to contribute I can contribute the implementation and tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
