dongjoon-hyun commented on PR #55642: URL: https://github.com/apache/spark/pull/55642#issuecomment-4357977025
> Multiple local-dir PVCs where only one directory is above the threshold. Only the PVC backing that directory should be resized. It's theoretically proper alternative, but it's not MUST-HAVE. I chose this simple design for the heavy load executor because all PVCs will be released if the executor dies with Out-Of-Disk > Executor pods with additional non-local-dir PVC mounts. Those PVCs should not be resized. `ExecutorPVCResizePlugin` is supposed to be used for homogeneous PVC case and not designed for those heterogeneous scenario. > A PVC whose spec.resources.requests.storage is already larger than status.capacity, meaning a resize is already pending. The plugin should not issue another expansion. As I mentioned, it sounds good but it's not K8s design policy which requesters set the desirable status and controller meets the target asynchronously. > Repeated check intervals with the same high usage report before the filesystem capacity changes. This should not repeatedly double the PVC size. The actual patched storage value, not just that patch() was called. Technically, hourly check is assumed for the long running jobs. The main reason why the interval is MINUTE is only to support the first expansion after creation for small jobs, @viirya . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
