dejankrak-db commented on PR #59640: URL: https://github.com/apache/spark/pull/59640#issuecomment-6620602263
Thanks @brkyvz and @holdenk for the reviews. I replied inline and pushed 494fb1ea4e6, which changes only wording. On ordering versus hashing: per the SPIP, FILE is not comparable, as it has neither equality nor an ordering. Operations that need equality (set operations, grouping, hashing, map and partition keys) are rejected because there is no equality, and sorting is rejected because there is no ordering. The scaladoc, comments and `FILE_TYPE_UNSUPPORTED_COLUMN_USAGE` message now say this consistently. I will file follow-up JIRAs for a type-level way to declare equality and ordering support, replacing the per-type checks for MAP, VARIANT and FILE (also covering the shared helper @dtenedor asked for), and for whether FILE should ever define equality, and link them in the description. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
