GitHub user carloea2 created a discussion: Enforce 6-Hour Hard Limit on Multipart Uploads (lakeFS LinkAddressTime Constraint)
### Background In our current upload flow we: 1. Call **lakeFS** to **initiate a multipart upload**, which returns a **physical address** + **uploadId**. 2. Upload each part **directly to MinIO** using that address. 3. Finalize by calling **lakeFS** to **complete** (with the MinIO ETags) or **abort** (with the same address + uploadId). ### Problem lakeFS treats the “link address” returned during init as time-bounded. * There’s a constant `LinkAddressTime` set to **6 hours**: [https://github.com/treeverse/lakeFS/blob/master/pkg/catalog/catalog.go#L84](https://github.com/treeverse/lakeFS/blob/master/pkg/catalog/catalog.go#L84) * In the completion path, lakeFS rejects finalize/abort operations if the time since address creation exceeds that window: [https://github.com/treeverse/lakeFS/blob/e77dd296d6e3d84e6083928bcc82fa186eed917c/pkg/catalog/catalog.go#L3197](https://github.com/treeverse/lakeFS/blob/e77dd296d6e3d84e6083928bcc82fa186eed917c/pkg/catalog/catalog.go#L3197) So any multipart upload that takes **longer than 6 hours** will fail on **complete/finish/abort**. ### Options considered **Option A: Build/patch lakeFS ourselves** **Option B: Use MinIO multipart APIs directly, then import into lakeFS** * Pros: Avoids the lakeFS 6-hour “link address” limitation for the multipart upload itself. * Cons: Import commits the file automatically, which breaks assumptions in our frontend: * We currently rely on **diff** to show uploaded files as *uncommitted changes* and then create a version explicitly. * If files are committed immediately via import, we would need broader changes across endpoints and UI flows to preserve expected behavior. **Option C: enforce a hard 6-hour limit for uploads** GitHub link: https://github.com/apache/texera/discussions/4184 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
