nevzheng opened a new issue, #13154: URL: https://github.com/apache/gravitino/issues/13154
## What would you like to be improved? Replace MinIO in Gravitino’s S3 credential-vending integration tests with a maintained backend that preserves our storage and authorization coverage. **We recommend RustFS, primarily to align with PyIceberg and Polaris.** SeaweedFS remains a credible alternative. **Why it matters:** MinIO’s Docker Hub image became unavailable, breaking test startup. [#13112](https://github.com/apache/gravitino/pull/13112) restored access through Quay, but the MinIO community repository is archived. We need an ongoing replacement for this temporary fix. ## How should we improve? Adopt a pinned RustFS release in the shared container fixture and migrate its consumers in a linked PR. The PR must demonstrate compatibility with our actual clients and generated credential policies. **Why this choice:** Ecosystem alignment carries the most weight: PyIceberg and Polaris provide relevant RustFS adoption and implementation references. This is a recommendation based on that priority, not a claim that RustFS is superior on every dimension. Revisit SeaweedFS if compatibility, maintenance, security response, or CI reliability changes the tradeoff. **Keep it maintained:** Add automated image-update detection and scheduled vulnerability reporting for the pinned image, with a documented maintainer response path. Version updates and CVE detection are separate requirements; existing Dependabot configuration does not establish coverage for this image. See Appendix D. **Scope:** S3 test infrastructure for fileset connection testing and Iceberg credential vending. Production storage migration and public API changes are outside this proposal. ## Acceptance criteria - [ ] RustFS passes the compatibility requirements below on a documented, pinned release; material gaps are recorded before adoption. - [ ] The image starts reliably in Linux amd64 CI; readiness, bucket/identity setup, logs, and cleanup work without depending on MinIO tooling. Record arm64 availability for local development. - [ ] The existing fileset connection and Iceberg credential-vending suites pass, preserving owner/modify access, select-only write denial, and active-role narrowing assertions. - [ ] Focused checks confirm the generated inline session policy restricts object paths and bucket-list prefixes, including denied operations outside the allowed scope. A generic client exception alone is insufficient evidence of policy enforcement. - [ ] The PR records the image version/digest, commands, results, and any remaining limitations. Relevant listing, deletion, and multipart behavior must work without weakening coverage. - [ ] Automated update detection tracks the image pin, and scheduled vulnerability reporting covers the image with documented scan limits and a maintainer triage route. Image updates run the affected integration tests before merge. **Decision requested:** Do we agree to adopt RustFS for ecosystem alignment, subject to these compatibility and maintenance criteria? Related to: [#13111](https://github.com/apache/gravitino/issues/13111), [#13112](https://github.com/apache/gravitino/pull/13112). --- ## Supporting evidence ### Appendix A — Apache community context - **Shared outage:** Iceberg Java also [switched MinIO image pulls to Quay](https://github.com/apache/iceberg/pull/18071). The [September 11 discussion](https://www.mail-archive.com/dev%40iceberg.apache.org/msg15098.html) revived replacement discussions across the Iceberg community. - **RustFS adoption:** [PyIceberg #3928](https://github.com/apache/iceberg-python/pull/3928) merged September 11, 2026, replacing the development and integration stacks’ object store. The renewed discussion also reports RustFS use in Polaris. Adoption is useful precedent, but does not establish compatibility with Gravitino’s STS policies. - **Earlier proposals:** Iceberg Java’s [RustFS PR #14928](https://github.com/apache/iceberg/pull/14928) and [SeaweedFS PR #15577](https://github.com/apache/iceberg/pull/15577) both closed without merging. These discussions do not establish an Apache-wide choice. ### Appendix B — What Gravitino actually requires The credential generator and the two test classes below were inspected locally and their Git blob hashes matched upstream Apache Gravitino at commit `20574ccdc7d5ea2c2c77c309c687e61d2fed4c7c`. | Requirement | Evidence and consequence for a replacement | | --- | --- | | Signed STS `AssumeRole` with an inline JSON policy | `S3TokenGenerator` uses AWS SDK v2 and sends role ARN, session name, duration, and policy; it optionally sends an external ID. The fixture may provision a real role instead of retaining MinIO’s dummy ARN. | | Path-scoped object permissions | The policy grants `GetObject`/`GetObjectVersion` for readable paths and `PutObject`/`DeleteObject` only for writable paths. Receiving temporary credentials is not enough: the backend must enforce the restrictions. | | Conditional bucket listing | `ListBucket` uses `StringLike` on `s3:prefix`; the policy also grants `GetBucketLocation`. Fileset configuration includes the bare location prefix for Hadoop directory probing. Preserve that deliberate behavior rather than demanding stricter isolation than the policy expresses. | | Hadoop S3A compatibility | `FilesetS3TokenConnectionIT` calls catalog `testConnection` using temporary credentials, path-style access, and configured S3/STS endpoints. It is a connection test, not a negative authorization suite. | | Spark and Iceberg S3FileIO compatibility | `IcebergRESTCloudTokenAuthorizationBaseIT` checks owner/modify writes and reads, select-only write failure, and active-role narrowing that denies writes while preserving reads. | | Deploy-mode and container integration | The fixture must expose a reachable endpoint, create buckets, and support the existing bundle setup. S3 and STS endpoints are configurable; they need not be the same URL. | Source links: [credential generator](https://github.com/apache/gravitino/blob/20574ccdc7d5ea2c2c77c309c687e61d2fed4c7c/bundles/aws/src/main/java/org/apache/gravitino/s3/credential/S3TokenGenerator.java), [fileset test](https://github.com/apache/gravitino/blob/20574ccdc7d5ea2c2c77c309c687e61d2fed4c7c/catalogs/catalog-fileset/src/test/java/org/apache/gravitino/catalog/fileset/integration/test/FilesetS3TokenConnectionIT.java), [Iceberg authorization tests](https://github.com/apache/gravitino/blob/20574ccdc7d5ea2c2c77c309c687e61d2fed4c7c/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/integration/test/IcebergRESTCloudTokenAuthorizationBaseIT.java). **Coverage boundary:** The existing suites do not directly prove every generated prefix/resource restriction or exercise all multipart edge cases. Focused integration checks are implementation acceptance evidence, not an outstanding desk-research task. ### Appendix C — RustFS vs. SeaweedFS Research checked September 14, 2026. **Source-backed** means supported by released code, tests, or documentation. Neither candidate has been run against Gravitino in this investigation. | Dimension | RustFS | SeaweedFS | Meaning for Gravitino | | --- | --- | --- | --- | | Release status | [1.0.0-rc.6](https://github.com/rustfs/rustfs/releases/tag/1.0.0-rc.6), September 11; marked prerelease. | [4.47](https://github.com/seaweedfs/seaweedfs/releases/tag/4.47), September 14; not marked prerelease. | SeaweedFS offers an established numbered release series; RustFS requires accepting a release candidate. Labels alone do not prove correctness. | | Distribution | Linux x86_64/aarch64 release assets; [container workflow](https://github.com/rustfs/rustfs/blob/1.0.0-rc.6/.github/workflows/docker.yml#L70) targets amd64 and arm64. | Linux amd64/arm64 release assets; [container workflow](https://github.com/seaweedfs/seaweedfs/blob/4.47/.github/workflows/container_release_unified.yml#L138) publishes both architectures. | Both have distribution evidence for CI and local development. Exact image manifests and pulls still need implementation-time verification. | | Fixture setup | Single object-store container; configure credentials/data directory, wait for health, create buckets through S3. PyIceberg provides a [working migration reference](https://github.com/apache/iceberg-python/pull/3928/files). | One process can run master, volume, filer, and S3 gateway. The signed STS path requires credentials, IAM signing configuration, a named role, trust policy, and base permissions. [Configuration](https://github.com/seaweedfs/seaweedfs/blob/4.47/weed/command/server.go#L164). | RustFS appears closer to our current fixture. SeaweedFS can run locally in one process, but needs more identity setup for these tests. This is an implementation-effort inference, not a measured startup comparison. | | Signed STS and inline policy | [Handler](https://github.com/rustfs/rustfs/blob/1.0.0-rc.6/rustfs/src/admin/handlers/sts.rs#L199) issues scoped temporary credentials; [IAM code](https://github.com/rustfs/rustfs/blob/1.0.0-rc.6/crates/iam/src/sys.rs#L1429) intersects the session policy with parent permissions, including restrictions on root-issued sessions. | [Handler](https://github.com/seaweedfs/seaweedfs/blob/4.47/weed/s3api/s3api_sts.go#L342) verifies signed requests and reads inline Policy; [IAM code](https://github.com/seaweedfs/seaweedfs/blob/4.47/weed/iam/integration/iam_manager.go#L1217) requires base and session permission, defaulting to deny. | Both have source evidence for the critical session boundary. Neither is disqualified merely for lacking STS support. | | Read/write and resource restrictions | Released [policy tests](https://github.com/rustfs/rustfs/blob/1.0.0-rc.6/crates/policy/src/policy/policy.rs#L1229) include a Polaris-shaped prefix policy; [session tests](https://github.com/rustfs/rustfs/blob/1.0.0-rc.6/crates/e2e_test/src/list_buckets_iam_filter_test.rs#L416) cover bucket filtering. | Released [IAM tests](https://github.com/seaweedfs/seaweedfs/blob/4.47/weed/iam/integration/iam_integration_test.go#L175) cover read-only denial and session resource boundaries; [prefix tests](https://github.com/seaweedfs/seaweedfs/blob/4.47/weed/s3api/s3_end_to_end_test.go#L222) cover matching, unrelated, and omitted prefixes for V1/V2 listing. | SeaweedFS has particularly direct tests for several relevant boundaries. Both still need Gravitino’s exact generated policies exercised over the real client path. | | Role/trust behavior | Standard signed handler parses role ARN/session name/external ID but derives permissions from caller credentials and inline policy rather than implementing AWS named-role selection. [Source](https://github.com/rustfs/rustfs/blob/1.0.0-rc.6/rustfs/src/admin/handlers/sts.rs#L144). | Evaluates named roles and trust policies. External-ID enforcement is not established: the signed handler does not read it and the [trust context](https://github.com/seaweedfs/seaweedfs/blob/4.47/weed/iam/integration/iam_manager_trust.go#L30) supplies principal fields only. | Neither should be presented as complete AWS IAM/STS emulation. Current MinIO tests do not require external-ID enforcement; SeaweedFS would need a provisioned role in place of the dummy MinIO ARN. | | Listing, deletion, multipart | Released [compatibility matrix](https://github.com/rustfs/rustfs/blob/1.0.0-rc.6/docs/architecture/s3-compatibility-matrix.md) covers common operations, with some multipart edge cases outside its default compatibility gate. | Released [policy engine](https://github.com/seaweedfs/seaweedfs/blob/4.47/weed/iam/policy/policy_engine.go#L621) expands PutObject authorization to multipart operations, including abort/list operations. That must not be described as exact AWS action equivalence. | Verify operations our clients issue and permissions they receive. Neither general S3 claims nor a simple upload demonstrate full compatibility. | | Apache precedent | [Polaris #3679](https://github.com/apache/polaris/pull/3679) added a RustFS test container; [PyIceberg #3928](https://github.com/apache/iceberg-python/pull/3928) migrated its object store using rc.5 and AWS CLI. | [Iceberg #15577](https://github.com/apache/iceberg/pull/15577) proposed SeaweedFS but closed unmerged. | RustFS has stronger verified adoption in the projects reviewed. PyIceberg’s static-credential fixture does not prove Gravitino’s STS requirements. | | License | [Apache-2.0](https://github.com/rustfs/rustfs/blob/1.0.0-rc.6/LICENSE). | [Apache-2.0](https://github.com/seaweedfs/seaweedfs/blob/4.47/LICENSE). | No distinction between the candidates on declared project license. | | Performance and reliability | No comparable Gravitino runtime/resource measurements collected. | No comparable Gravitino runtime/resource measurements collected. | Do not rank candidates using vendor throughput claims. CI startup, stability, and resource use are more relevant here. | #### Assessment **Recommend RustFS because ecosystem alignment is our primary selection criterion.** Its closer fixture model is a secondary benefit. SeaweedFS remains viable, with a non-prerelease release series, named-role handling, and explicit tests for relevant policy boundaries. The recommendation remains conditional on Gravitino acceptance tests. Keep backend-specific setup within the test fixture and use neutral names for shared S3 configuration where practical, so a later switch remains manageable. Reconsider the choice if RustFS cannot preserve authorization coverage, loses maintained distribution, or develops unresolved security or reliability problems. #### Historical issues and evidence limits RustFS’s [listing-consistency report](https://github.com/rustfs/rustfs/issues/1325) was closed after maintainers reported a fix, and its [single-string policy Resource bug](https://github.com/rustfs/rustfs/issues/1337) was reported fixed in alpha.79. SeaweedFS has merged fixes for [STS POST dispatch](https://github.com/seaweedfs/seaweedfs/pull/8320), [multipart authorization](https://github.com/seaweedfs/seaweedfs/pull/8445), and [prefix conditions](https://github.com/seaweedfs/seaweedfs/pull/8971). These are historical compatibility lessons, not evidence that current releases necessarily retain the bugs. Release assets and publication workflows were checked; registry manifest verification was unavailable during research. No image pull, startup benchmark, or Gravitino runtime result is claimed. ### Appendix D — Dependency updates and vulnerability monitoring #### Current coverage As checked September 14, 2026, upstream [apache/gravitino’s Dependabot configuration](https://github.com/apache/gravitino/blob/main/.github/dependabot.yml) schedules monthly GitHub Actions, Gradle, npm, and pip updates. **It has no Docker or Docker Compose entry.** The current MinIO pin lives in a Java fixture constant; Gradle dependency updates do not establish image-update coverage for that reference. Repository-level Dependabot security-alert settings were not verified. Their presence would not by itself prove that the RustFS image or its embedded dependencies are covered. GitHub distinguishes [version-update configuration](https://docs.github.com/en/code-security/concepts/supply-chain-security/about-the-dependabot-yml-file) from [security alerts and updates](https://docs.github.com/en/code-security/concepts/supply-chain-security/dependabot-security-updates). #### Recommended implementation - **Version tracking:** Keep one authoritative image pin in a format the chosen updater can actually discover. Prefer extending upstream’s existing Dependabot setup with a supported Docker manifest if the fixture can consume it directly; otherwise use an explicitly configured image-reference updater. Avoid a duplicate, unused manifest that drifts from the Java fixture. Demonstrate that a newer image is detected, including transitions from release candidates to stable versions. [Dependabot Docker support and prerelease behavior](https://docs.github.com/en/enterprise-cloud%40latest/code-security/reference/supply-chain-security/supported-ecosystems-and-repositories#docker). - **Vulnerability reporting:** Scan the exact pinned image when its digest changes and on a weekly schedule, using a container scanner such as Trivy. Retain reports and route actionable findings to maintainers for triage. Record how urgent fixes and accepted exceptions are handled; do not auto-merge image updates without integration-test results. - **Coverage limits:** Scan OS packages and available application dependency metadata, and include upstream security advisories in the response process. For Rust dependencies, confirm that the image supplies usable metadata or an SBOM. [Trivy documents Cargo.lock and cargo-auditable binary support](https://trivy.dev/docs/latest/coverage/language/rust/); a clean OS-package scan alone does not establish that embedded Rust libraries are free of known vulnerabilities. **Downstream distinction:** The current enterprise checkout intentionally removes Dependabot version-update configuration during upstream sync and receives dependency bumps from upstream. That policy is separate from this Apache proposal. Any enterprise adoption needs its own image-alert/scan path or an explicit way to consume upstream findings; do not assume version sync supplies CVE monitoring. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
