coderfender opened a new issue, #6484: URL: https://github.com/apache/datafusion-comet/issues/6484
### Describe the problem Comet maps Spark's `ExistenceJoin` (always `BuildRight`) onto DataFusion's mark join, which after the input swap becomes a `RightMark`. The DataFusion 55.1 hash-join lookup enumerates every equal-key candidate pair before the `RightMark` path reduces duplicate probe indices to Boolean markers. With N identical build keys and M matching probe rows this evaluates N*M candidate matches to produce only M output markers, whereas Spark's existence lookup stops after the first match per probe row. Example (from review): 10,000 identical build keys and 1,000 matching probe rows imply ~10 million candidate matches even though only 1,000 markers are needed. ### Impact Correctness is unaffected — the marker is still "at least one match." This is a performance/scalability concern for skewed build keys. The current benchmark uses distinct build keys and does not exercise this path. ### Suggested direction Deduplicate the evaluated build-key tuples for residual-free existence joins, or use a mark-join implementation that stops after the first match. Any dedup helper must account for its own memory and support bounded spilling, and must not discard distinct right payloads while a residual predicate can still use them. ### Context Split out from PR #4587 (native ExistenceJoin). Reviewer thread: https://github.com/apache/datafusion-comet/pull/4587#discussion_r4042586371 . Deferring per discussion (feature is useful now; the mark join can be improved in the background). Related upstream perf work: https://github.com/apache/datafusion/pull/23870 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
