Adam Binford created SPARK-54698:
------------------------------------
Summary: Array set-like operations are very slow for complex types
Key: SPARK-54698
URL: https://issues.apache.org/jira/browse/SPARK-54698
Project: Spark
Issue Type: New Feature
Components: SQL
Affects Versions: 4.0.1
Reporter: Adam Binford
Array set-like operations (distinct, union, intersect, etc.) can be very slow
for complex types, especially with larger array sizes. This is because hashed
based operations are not used, but instead a nested comparison checking every
element against every existing element, becoming a O(n^2) operation. This
should be improved to also take advantage of hashing so it can scale to larger
arrays
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
