findepi commented on issue #13704: URL: https://github.com/apache/datafusion/issues/13704#issuecomment-3110803759
> AFAIK the SQL spec is underspecified here Absolutely right > it has been suggested that ordering and comparison could use different ordering That's indeed what databases do. In fact, it's impossible to satisfy two following goals at once: have total ordering AND have any comparison involving NaN return false. > FWIW there is some ongoing discussion around adding this total ordering to parquet as a way to order statistics (the current approach makes statistics unusable for floats). That' a good point. In fact, eg Trino has some conservative treatment of floating point types (like here https://github.com/trinodb/trino/blob/da94e0726fa48bd2e31a91ba26ff7d72aa01085a/core/trino-main/src/main/java/io/trino/sql/planner/DomainTranslator.java#L221-L229). Improving parquet spec (and then Iceberg) around this is a great idea, especially once all writer implementation catch up with this! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
