This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/texera.git
The following commit(s) were added to refs/heads/main by this push:
new 322babf620 feat(k8s): make app-tier object storage configurable via
storage.s3 (#5932)
322babf620 is described below
commit 322babf6205cf72277d13a4f2d16c5954a4dfbd9
Author: ali risheh <[email protected]>
AuthorDate: Wed Jul 1 15:46:27 2026 -0700
feat(k8s): make app-tier object storage configurable via storage.s3 (#5932)
### What changes were proposed in this PR?
This makes the **application tier's** object-storage target
configurable, as the first step toward supporting an external S3 store
alongside the in-cluster MinIO. It is **non-breaking**: the default
(on-prem / in-cluster MinIO) install renders an identical set of
resources.
Today `file-service` and `workflow-computing-unit-manager` hardcode the
S3 endpoint and credentials to the in-cluster MinIO Service (`{{
.Release.Name }}-minio`) and its auto-generated Secret. This PR routes
both through a new `storage.s3` values block:
- **`values.yaml`** — new `storage.s3` block: `endpoint`, `region`,
`existingSecret`, `accessKeyId`, `secretAccessKey`. All default to
empty.
- **`templates/base/_helpers.tpl`** (new) — helpers that resolve the S3
endpoint and the credentials Secret name/keys. When
`storage.s3.endpoint` is empty they fall back to the in-cluster MinIO
Service and its `{{ .Release.Name }}-minio` Secret (keys
`root-user`/`root-password`), so the default render is unchanged.
- **`templates/aws/s3-credentials-secret.yaml`** (new) — a `{{
.Release.Name }}-s3-credentials` Secret, rendered **only** when an
external `endpoint` is set and no `existingSecret` is supplied. Renders
nothing on the default install.
- **`file-service` / `workflow-computing-unit-manager` deployments** —
the `STORAGE_S3_*` env now comes from the helpers; `STORAGE_S3_REGION`
is added only in external mode.
How it behaves:
- **Default (no config):** services point at the in-cluster MinIO
exactly as before.
- **External S3:** set `storage.s3.endpoint` + `region` + credentials
(or `existingSecret`) → both services use that store; the chart
materializes the credentials Secret unless you bring your own.
**Out of scope (intentionally deferred to keep this PR small and
atomic):** LakeFS blockstore + Lakekeeper warehouse external-S3 wiring,
the `minio.enabled` switch to drop the in-cluster MinIO entirely, and a
`values-aws.yaml` example overlay. Those are the LakeFS/Lakekeeper half
of "make object storage pluggable" and will land as a follow-up.
### Any related issues, documentation, discussions?
Closes #5931 (app-tier `storage.s3` task).
Part of #5891 — unify AWS (EKS) and on-premise Kubernetes deployment
under `bin/k8s` (parent feature).
Follows #5757 (Helm template reorg) and the design discussion in #5641.
### How was this PR tested?
Verified the default install is unchanged and the external path renders
correctly:
1. **No-op render proof:** `helm template texera bin/k8s` on this branch
vs on `main` renders the **same 102 resources** — identical after
ignoring comments. The only textual artifact is one empty `---` document
from the gated-off `s3-credentials-secret.yaml` (its `# license` header
is emitted by Helm even though the `{{- if }}` body is empty); it
produces no Kubernetes object. `helm lint` passes.
2. **External S3 render:** `helm template ... --set
storage.s3.endpoint=https://s3.us-west-2.amazonaws.com --set
storage.s3.accessKeyId=… --set storage.s3.secretAccessKey=…` renders the
`*-s3-credentials` Secret, repoints both deployments'
`STORAGE_S3_ENDPOINT`/credentials at it (keys
`access-key-id`/`secret-access-key`), and adds `STORAGE_S3_REGION`.
3. **Bring-your-own Secret:** with `--set
storage.s3.existingSecret=my-creds`, the deployments reference
`my-creds` and the chart generates **no** Secret.
4. `helm lint bin/k8s` passes for both the default and the external
value sets.
5. `helm install` locally with Minikube and test creating dataset and
running workflows
No unit tests were added — the change is limited to Helm chart
values/templates, validated by the render diff and `helm lint` above.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
---
bin/k8s/templates/aws/s3-credentials-secret.yaml | 32 +++++++++++++
bin/k8s/templates/base/_helpers.tpl | 56 ++++++++++++++++++++++
.../base/file-service/file-service-deployment.yaml | 14 ++++--
...workflow-computing-unit-manager-deployment.yaml | 14 ++++--
bin/k8s/values.yaml | 12 +++++
5 files changed, 118 insertions(+), 10 deletions(-)
diff --git a/bin/k8s/templates/aws/s3-credentials-secret.yaml
b/bin/k8s/templates/aws/s3-credentials-secret.yaml
new file mode 100644
index 0000000000..8506115167
--- /dev/null
+++ b/bin/k8s/templates/aws/s3-credentials-secret.yaml
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Credentials Secret for an external S3 store. Rendered only when an external
+# endpoint is configured (storage.s3.endpoint) and the deployer has not
supplied
+# their own Secret (storage.s3.existingSecret). On the default in-cluster MinIO
+# install this renders nothing -- the services use MinIO's own Secret instead.
+{{- if and .Values.storage.s3.endpoint (not .Values.storage.s3.existingSecret)
}}
+apiVersion: v1
+kind: Secret
+metadata:
+ name: {{ .Release.Name }}-s3-credentials
+ namespace: {{ .Release.Namespace }}
+type: Opaque
+stringData:
+ access-key-id: {{ .Values.storage.s3.accessKeyId | quote }}
+ secret-access-key: {{ .Values.storage.s3.secretAccessKey | quote }}
+{{- end }}
diff --git a/bin/k8s/templates/base/_helpers.tpl
b/bin/k8s/templates/base/_helpers.tpl
new file mode 100644
index 0000000000..e044b7285a
--- /dev/null
+++ b/bin/k8s/templates/base/_helpers.tpl
@@ -0,0 +1,56 @@
+{{/*
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+*/}}
+
+{{/*
+Object-storage (S3) resolution helpers.
+
+When storage.s3.endpoint is set the services talk to that external
+S3-compatible store (credentials come from storage.s3.existingSecret, or a
+chart-generated "<release>-s3-credentials" Secret). When it is empty the
+services fall back to the in-cluster MinIO Service and its auto-generated
+"<release>-minio" Secret, so the default install is unchanged.
+*/}}
+
+{{/* S3 endpoint URL. */}}
+{{- define "texera.s3.endpoint" -}}
+{{- if .Values.storage.s3.endpoint -}}
+{{- .Values.storage.s3.endpoint -}}
+{{- else -}}
+{{- printf "http://%s-minio:9000"; .Release.Name -}}
+{{- end -}}
+{{- end -}}
+
+{{/* Name of the Secret holding the S3 credentials. */}}
+{{- define "texera.s3.secretName" -}}
+{{- if .Values.storage.s3.endpoint -}}
+{{- .Values.storage.s3.existingSecret | default (printf "%s-s3-credentials"
.Release.Name) -}}
+{{- else -}}
+{{- printf "%s-minio" .Release.Name -}}
+{{- end -}}
+{{- end -}}
+
+{{/* Secret data key for the S3 access key id. */}}
+{{- define "texera.s3.accessKeyIdKey" -}}
+{{- if .Values.storage.s3.endpoint -}}access-key-id{{- else -}}root-user{{-
end -}}
+{{- end -}}
+
+{{/* Secret data key for the S3 secret access key. */}}
+{{- define "texera.s3.secretAccessKeyKey" -}}
+{{- if .Values.storage.s3.endpoint -}}secret-access-key{{- else
-}}root-password{{- end -}}
+{{- end -}}
diff --git a/bin/k8s/templates/base/file-service/file-service-deployment.yaml
b/bin/k8s/templates/base/file-service/file-service-deployment.yaml
index 6c9a4041f9..6a9190bc6c 100644
--- a/bin/k8s/templates/base/file-service/file-service-deployment.yaml
+++ b/bin/k8s/templates/base/file-service/file-service-deployment.yaml
@@ -41,17 +41,21 @@ spec:
env:
# LakeFS & S3 Access
- name: STORAGE_S3_ENDPOINT
- value: http://{{ .Release.Name }}-minio:9000
+ value: {{ include "texera.s3.endpoint" . }}
+ {{- if .Values.storage.s3.endpoint }}
+ - name: STORAGE_S3_REGION
+ value: {{ .Values.storage.s3.region | quote }}
+ {{- end }}
- name: STORAGE_S3_AUTH_USERNAME
valueFrom:
secretKeyRef:
- name: {{ .Release.Name }}-minio
- key: root-user
+ name: {{ include "texera.s3.secretName" . }}
+ key: {{ include "texera.s3.accessKeyIdKey" . }}
- name: STORAGE_S3_AUTH_PASSWORD
valueFrom:
secretKeyRef:
- name: {{ .Release.Name }}-minio
- key: root-password
+ name: {{ include "texera.s3.secretName" . }}
+ key: {{ include "texera.s3.secretAccessKeyKey" . }}
- name: STORAGE_LAKEFS_ENDPOINT
value: http://{{ .Release.Name }}-lakefs:8000/api/v1
- name: STORAGE_LAKEFS_AUTH_USERNAME
diff --git
a/bin/k8s/templates/base/workflow-computing-unit-manager/workflow-computing-unit-manager-deployment.yaml
b/bin/k8s/templates/base/workflow-computing-unit-manager/workflow-computing-unit-manager-deployment.yaml
index 7a0185cd46..a911845041 100644
---
a/bin/k8s/templates/base/workflow-computing-unit-manager/workflow-computing-unit-manager-deployment.yaml
+++
b/bin/k8s/templates/base/workflow-computing-unit-manager/workflow-computing-unit-manager-deployment.yaml
@@ -81,17 +81,21 @@ spec:
value: http://{{ .Values.fileService.name
}}-svc:9092/api/dataset/did/upload
# S3 Access (for R UDF large binary support)
- name: STORAGE_S3_ENDPOINT
- value: http://{{ .Release.Name }}-minio:9000
+ value: {{ include "texera.s3.endpoint" . }}
+ {{- if .Values.storage.s3.endpoint }}
+ - name: STORAGE_S3_REGION
+ value: {{ .Values.storage.s3.region | quote }}
+ {{- end }}
- name: STORAGE_S3_AUTH_USERNAME
valueFrom:
secretKeyRef:
- name: {{ .Release.Name }}-minio
- key: root-user
+ name: {{ include "texera.s3.secretName" . }}
+ key: {{ include "texera.s3.accessKeyIdKey" . }}
- name: STORAGE_S3_AUTH_PASSWORD
valueFrom:
secretKeyRef:
- name: {{ .Release.Name }}-minio
- key: root-password
+ name: {{ include "texera.s3.secretName" . }}
+ key: {{ include "texera.s3.secretAccessKeyKey" . }}
# LakeFS Access (should be removed in production environment)
- name: STORAGE_LAKEFS_ENDPOINT
value: http://{{ .Release.Name }}-lakefs.{{ .Release.Namespace
}}:8000/api/v1
diff --git a/bin/k8s/values.yaml b/bin/k8s/values.yaml
index 2974c27c88..64642b517a 100644
--- a/bin/k8s/values.yaml
+++ b/bin/k8s/values.yaml
@@ -65,6 +65,18 @@ postgresql:
initdb:
scriptsConfigMap: "postgresql-init-script"
+# Object storage (S3) used by the Texera services. Leave storage.s3.endpoint
+# empty to use the in-cluster MinIO (the `minio:` block below). Set it to an
+# S3-compatible endpoint URL -- together with region and credentials -- to
+# point the services at an external store (e.g. AWS S3) instead.
+storage:
+ s3:
+ endpoint: "" # "" => in-cluster MinIO; otherwise an S3 endpoint
URL
+ region: "us-west-2"
+ existingSecret: "" # existing Secret with access-key-id /
secret-access-key; "" => chart creates one
+ accessKeyId: ""
+ secretAccessKey: ""
+
minio:
mode: standalone
image:
