mustafasrepo commented on issue #7794: URL: https://github.com/apache/arrow-datafusion/issues/7794#issuecomment-1759571944
Thanks @alamb for great explanation and locating the issue. I have open the [#7811](https://github.com/apache/arrow-datafusion/pull/7811), which I believe solves this issue. As far as I can tell, there are mainly two issues here. - First one is `EnforceSorting` moves up the `SortExec` that satisfy an executer right under the executor. Then during `pushdown` pass that `SortExec` is tried to move down, if it is helpful (In your case, `SortExec` is not pushed down through `RepartitioinExec`). - Second one is when relative position of the `SortExec` changes, some of the executors in between may be invalid. Which is the case, in this issue. The PR solves the second problem, while removing `SortExec` it replaces executors on top of it with variants that do not preserve order. However, after this PR `EnforceSorting` may still move up some `SortExec` to the executor that require it. I tried to replicate the issue in the PR, however I did not use the exact same plan here. Hence this explanation is educated guess. If #7811 doesn't resolve this issue, please let me know. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
