Phucvt123 opened a new pull request, #55854: URL: https://github.com/apache/spark/pull/55854
### What changes were proposed in this pull request? Modified `LSHModel` to conditionally check for the existence of `inputCol` in `approxSimilarityJoin` and `approxNearestNeighbors`. When `inputCol` is missing (user already transformed and dropped it), the methods now fall back to hash-based approximate distance using `outputCol` instead of throwing an error. ### Why are the changes needed? Users who pre-compute hashes and drop the original features column to save memory/storage cannot use `approxSimilarityJoin` or `approxNearestNeighbors`. The documentation implies this should work, but the current implementation always requires `inputCol`. ### Does this PR introduce any user-facing change? Yes. `approxSimilarityJoin` and `approxNearestNeighbors` now accept datasets that contain only the `outputCol` (hashes) without `inputCol`. When `inputCol` is present, behavior is unchanged (exact distance). When `inputCol` is absent, hash-based approximate distance is used with a warning logged. ### How was this patch tested? Added 3 unit tests in `MinHashLSHSuite`: - `approxSimilarityJoin` without `inputCol` - `approxNearestNeighbors` without `inputCol` - `approxSimilarityJoin` self-join without `inputCol` All 17 tests pass (14 existing + 3 new). ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
