viirya commented on PR #55642: URL: https://github.com/apache/spark/pull/55642#issuecomment-4358061458
> It's one of theoretically proper alternatives, but it's not MUST-HAVE. I chose this simple design for the heavy load executor because all PVCs (of that executor) will be released (affected) if the executor dies with Out-Of-Disk of one PVC. Thanks, I understand the intended trade-off now: if any local-dir PVC can kill the executor when it runs out of space, resizing all local-dir PVCs for that executor is a simpler conservative policy. My remaining concern is that the current implementation does not limit this policy to Spark local-dir PVCs. It resizes every PVC mounted by the executor container, including PVCs that are unrelated to Spark local storage. If the intended contract is “all Spark local-dir PVCs are resized together for a homogeneous local-dir setup,” then I think the code and docs should make that explicit and avoid touching non-local-dir PVCs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
