IgnatiusPang commented on PR #25744:
URL: https://github.com/apache/datafusion/pull/25744#issuecomment-5835042231
Thank you @alamb for the guidance and feedback! I apologize for opening
several PRs in parallel. I see that you've already put the others into draft
for me, thank you.
Here is a 1-line SQL reproducer demonstrating how users encounter this bug
in `datafusion-cli`:
### SQL Reproducer
```sql
SELECT cosine_distance([1.0, 1.0, 1.0], [1.0, 1.0, 1.0]) AS dist;
```
### Current results (unpatched)
+-----------------------+
| dist |
+-----------------------+
| -2.220446049250313e-16|
+-----------------------+
### Expected results
+------+
| dist |
+------+
| 0.0 |
+------+
### User Query Impact: Silent Row Dropping in Filtering
Because the distance is negative, queries filtering on valid non-negative
distances silently drop identical vectors:
```{sql}
WITH vectors AS (
SELECT 'vec_a' AS id, [1.0, 1.0, 1.0] AS v1, [1.0, 1.0, 1.0] AS v2
)
SELECT id FROM vectors WHERE cosine_distance(v1, v2) >= 0.0;
```
### Output
```
DataFrame has no rows (0 rows returned instead of 1)
```
### Why this happens
In IEEE 754 floating-point math, sqrt(3.0) * sqrt(3.0) evaluates to
2.9999999999999996 (strictly less than 3.0). When comparing identical vectors
like [1.0, 1.0, 1.0]: dot / (norm1 * norm2) = 3.0 / 2.9999999999999996 =
1.0000000000000002 > 1.0 This causes 1.0 - sim to return -2.22e-16, which
produces a negative distance and violates metric space axioms ($d(u, v) \ge
0$). This can cause unexpected behavior in downstream vector search / k-NN
queries and filters like WHERE cosine_distance(...) >= 0.
The fix clamps the similarity to [-1.0, 1.0] and the resulting distance to
[0.0, 2.0]. I also included an update to cosine_distance.slt.
Looking forward to your review on this PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
