parthchandra commented on PR #5331: URL: https://github.com/apache/datafusion-comet/pull/5331#issuecomment-5322165649
> > That's not purely hypothetical, because the gate gets evaluated twice against two different points in time. `outputOrdering` is a lazy val forced during planning, this runs later off `serializedPartitionData`, and `reportableOrdering` reads `COMET_ICEBERG_SORT_MERGE_ENABLED` out of the active `SQLConf` each time. Two dynamic conf reads on separate threads at separate stages is the kind of thing that holds until it doesn't. > Fixed. This is evaluated once now. > `serializePartitions` only has the one caller, so could we just pass `outputOrdering` in from the exec and have a single evaluation? Failing that I'd rather the mismatch threw than silently produced a proto that disagrees with what we told Spark. Fixed. This throws if a reported order can't be serialized, instead of the old silent `if (forall(_.isDefined))` drop -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
