feniljain commented on issue #18595: URL: https://github.com/apache/datafusion/issues/18595#issuecomment-3522179418
>It may be similar issue. I wonder after https://github.com/apache/datafusion/issues/18341 (seehttps://github.com/apache/datafusion/pull/18521#issuecomment-3516585320), will it get better? Seehttps://github.com/apache/datafusion/issues/18341#issuecomment-3505130845 I tested on latest `main`. It is still happening, also I am mentioning the issue due to this statement in it: >I'm not 100% sure this is a bug, but it seems like repartitioning small files is potentially pretty wasteful, this also seems somewhat related to # 4967, which is why I'm filing it. though title of issue does say something else: `Output ordering with a filter over FileSource seems to cause some unexpected repartitioning behavior` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
