mbutrovich commented on PR #20806: URL: https://github.com/apache/datafusion/pull/20806#issuecomment-4025253220
> One thing to mention: to control the flag `datafusion.optimizer.enable_semi_anti_sort_merge_join` or `DATAFUSION_OPTIMIZER_ENABLE_SEMI_ANTI_SORT_MERGE_JOIN` in `bench.sh`, at least document it I'd like to understand the long-term implications of having the config. I added it since we're used to having opt outs in Comet from operators in case it is not compatible with a workload, and it makes it easier to benchmark right now. However, would we then be subject to [API health considerations](https://datafusion.apache.org/contributor-guide/api-health.html) on removing the old SMJ's semi/anti support for X releases/months? If so, I might just propose removing the config and saying "this is the new semi/anti SMJ implementation" as if we had changed code internal to `SortMergeJoinExec` like #18875. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
