milenkovicm commented on issue #17334: URL: https://github.com/apache/datafusion/issues/17334#issuecomment-3477475301
Long story short, in spark operators could ask others to spill when they get under pressure, and I think it makes sense as in the most cases, I saw, there is a spillable operator down the pipeline which could spill. I did not see easy way to implement cooperative memory management in DF. It would add additional set of complexity, threading issues ... KISS solution, ignore unspillable operator allocation error, chances are there is a spillable operator down the road which will get under memory pressure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
