carloea2 commented on issue #4058: URL: https://github.com/apache/texera/issues/4058#issuecomment-3578140838
After we add a backend finish-time check on multipartUpload?type=finish that reads the final size from lakeFS/S3 and rejects anything over singleFileUploadMaxSizeMiB, multipart uploads can no longer produce a dataset file larger than the limit, **the API-side limit can’t be bypassed.** What can still happen is that a client **uploads more** than the limit in raw bytes to lakeFS/S3 **before we reject**, so the remaining risk is bandwidth/storage waste, not correctness. The “watcher” idea is only an **optional** optimization to detect obviously over-limit uploads earlier to save those resources, it doesn’t change the limit semantics. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
