sarutak commented on PR #55549: URL: https://github.com/apache/spark/pull/55549#issuecomment-4349901187
Hi @shrirangmhalgi, the current implementation does preserve element order, but "the implementation happens to do X" is different from "the API guarantees X." Documenting it as a guarantee constrains future optimizations (sort-based deduplication, parallelization, etc.) and cannot easily be walked back. When `array_union`, `array_intersect`, and `array_except` were added (SPARK-23913/23914/23915), each PR description explicitly stated "The order of elements in the result is not defined." While the PR description for `array_distinct` does not mention the order of the result elements, it should be consistent with `array_union` and the others. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
