This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-8572-9d768b7ba9c274eb1a2980162af4cda6ef6327ba in repository https://gitbox.apache.org/repos/asf/texera.git
commit f16e7cdc287e6b044473ecf227f6b2251e188dc2 Author: dzueck <[email protected]> AuthorDate: Wed Oct 7 15:03:49 2026 +0000 feat: support for vm runtime configurations for compute units and texera micro services (#8572) <!-- Thanks for sending a pull request (PR)! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: [Contributing to Texera](https://github.com/apache/texera/blob/main/CONTRIBUTING.md) 2. Ensure you have added or run the appropriate tests for your PR 3. If the PR is work in progress, mark it a draft on GitHub. 4. Please write your PR title to summarize what this PR proposes, we are following Conventional Commits style for PR titles as well: - `fix` is for behavior that worked before and no longer does; adding or removing a functionality, or reworking one so that user-facing behavior intentionally changes, is a `feat`; a change that leaves the user-facing behavior unchanged is a `refactor`. - A test-only PR is `test(<module>): ...`; repairing a broken test is `fix(test, <module>): ...`. - A dependency bump is `fix(deps, <module>): ...` when it patches a CVE and `chore(deps, <module>): ...` otherwise; GitHub Actions bumps take `ci` as their module, e.g. `chore(deps, ci): ...`. - A PR targeting a release branch appends the version as the last scope component, e.g. `fix(deps, frontend, v1.2): ...`. See CONTRIBUTING.md for the full convention. 5. Be sure to keep the PR description updated to reflect all changes. --> ### What changes were proposed in this PR? <!-- Please clarify what changes you are proposing. The purpose of this section is to outline the changes. Here are some tips for you: 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, you can clarify why it is a bug. 3. If it is a refactoring, clarify what has been changed. 3. It would be helpful to include a before-and-after comparison using screenshots or GIFs. 4. Please consider writing useful notes for better and faster reviews. --> Adds support for running compute units and micro services inside a vm using [kata containers](https://katacontainers.io/). This is important for security as arbitrary code submitted by users can run inside these compute units. This PR adds the option to use VMs, but does not require it. It is only targeted at k8s deployments and was tested using RKE2. Note: k8s distributions like minikube may need additional steps to support VMs. To enable VMs you must first install kata containers following [these instructions](https://kata-containers.github.io/kata-containers/installation/). Make sure to follow the helm instructions. After this, you can enable Compute Units to use a VM by modifying these two lines in bin/k8s/values.yaml: - name: KUBERNETES_COMPUTING_UNIT_RUNTIME_CLASS_CPU value: "kata-qemu-runtime-rs" - name: KUBERNETES_COMPUTING_UNIT_RUNTIME_CLASS_GPU value: "kata-qemu-nvidia-gpu-runtime-rs" To enable texera specific micro services to run in a VM modify this line: global: texeraMicroServiceRuntimeClass: "kata-qemu-runtime-rs" ### Any related issues, documentation, discussions? <!-- Please use this section to link other resources if not mentioned already. 1. If this PR fixes an issue, please include `Fixes #1234`, `Resolves #1234` or `Closes #1234`. If it is only related, simply mention the issue number. 2. If there is design documentation, please add the link. 3. If there is a discussion in the mailing list, please add the link. --> Closes #8545 Discussed in #8430 ### How was this PR tested? <!-- If tests were added, say they were added here. Or simply mention that if the PR is tested with existing test cases. Make sure to include/update test cases that check the changes thoroughly including negative and positive cases if possible. If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future. If tests were not added, please describe why they were not added and/or why it was difficult to add. --> Primarily tested in v1.2.0 where performance tests were done with various micro benchmark workflows which can be seen in #8430. Additional functionality testing still needs to be done on the latest commits. ### Was this PR authored or co-authored using generative AI tooling? <!-- If generative AI tooling has been used in the process of authoring this PR, please include the phrase: 'Generated-by: ' followed by the name of the tool and its version. If no, write 'No'. Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details. --> Generated-by: Gemini 4.1 Pro --------- Signed-off-by: dzueck <[email protected]> --- .../access-control-service-deployment.yaml | 3 +++ .../base/agent-service/agent-service-deployment.yaml | 3 +++ .../base/config-service/config-service-deployment.yaml | 3 +++ .../base/example-data-loader/example-data-loader-job.yaml | 3 +++ .../templates/base/file-service/file-service-deployment.yaml | 3 +++ bin/k8s/templates/base/lakefs/lakefs-setup-job.yaml | 3 +++ bin/k8s/templates/base/litellm/litellm-deployment.yaml | 3 +++ bin/k8s/templates/base/webserver/webserver-deployment.yaml | 3 +++ .../workflow-compiling-service-deployment.yaml | 3 +++ .../workflow-computing-unit-manager-deployment.yaml | 3 +++ .../workflow-computing-unit-master-prepull-daemonset.yaml | 3 +++ bin/k8s/values.yaml | 12 ++++++++++++ .../org/apache/texera/service/util/KubernetesClient.scala | 11 ++++++++--- 13 files changed, 53 insertions(+), 3 deletions(-) diff --git a/bin/k8s/templates/base/access-control-service/access-control-service-deployment.yaml b/bin/k8s/templates/base/access-control-service/access-control-service-deployment.yaml index 7d63720365..99419d1f54 100644 --- a/bin/k8s/templates/base/access-control-service/access-control-service-deployment.yaml +++ b/bin/k8s/templates/base/access-control-service/access-control-service-deployment.yaml @@ -37,6 +37,9 @@ spec: # for a dedicated identity. serviceAccountName: {{ .Values.accessControlService.serviceAccountName }} {{- end }} + {{- if .Values.global.texeraMicroServiceRuntimeClass }} + runtimeClassName: {{ .Values.global.texeraMicroServiceRuntimeClass | quote }} + {{- end }} containers: - name: {{ .Values.accessControlService.name }} image: {{ .Values.texera.imageRegistry }}/{{ .Values.accessControlService.imageName }}:{{ .Values.texera.imageTag }} diff --git a/bin/k8s/templates/base/agent-service/agent-service-deployment.yaml b/bin/k8s/templates/base/agent-service/agent-service-deployment.yaml index 399b3a9058..4732bae1fa 100644 --- a/bin/k8s/templates/base/agent-service/agent-service-deployment.yaml +++ b/bin/k8s/templates/base/agent-service/agent-service-deployment.yaml @@ -33,6 +33,9 @@ spec: labels: app: {{ .Release.Name }}-{{ .Values.agentService.name }} spec: + {{- if .Values.global.texeraMicroServiceRuntimeClass }} + runtimeClassName: {{ .Values.global.texeraMicroServiceRuntimeClass | quote }} + {{- end }} containers: - name: {{ .Values.agentService.name }} image: {{ .Values.texera.imageRegistry }}/{{ .Values.agentService.imageName }}:{{ .Values.texera.imageTag }} diff --git a/bin/k8s/templates/base/config-service/config-service-deployment.yaml b/bin/k8s/templates/base/config-service/config-service-deployment.yaml index d6775483eb..7b86369f25 100644 --- a/bin/k8s/templates/base/config-service/config-service-deployment.yaml +++ b/bin/k8s/templates/base/config-service/config-service-deployment.yaml @@ -32,6 +32,9 @@ spec: labels: app: {{ .Release.Name }}-{{ .Values.configService.name }} spec: + {{- if .Values.global.texeraMicroServiceRuntimeClass }} + runtimeClassName: {{ .Values.global.texeraMicroServiceRuntimeClass | quote }} + {{- end }} containers: - name: {{ .Values.configService.name }} image: {{ .Values.texera.imageRegistry }}/{{ .Values.configService.imageName }}:{{ .Values.texera.imageTag }} diff --git a/bin/k8s/templates/base/example-data-loader/example-data-loader-job.yaml b/bin/k8s/templates/base/example-data-loader/example-data-loader-job.yaml index 15759be409..f245350cab 100644 --- a/bin/k8s/templates/base/example-data-loader/example-data-loader-job.yaml +++ b/bin/k8s/templates/base/example-data-loader/example-data-loader-job.yaml @@ -27,6 +27,9 @@ spec: metadata: name: {{ .Release.Name }}-example-data-loader spec: + {{- if .Values.global.texeraMicroServiceRuntimeClass }} + runtimeClassName: {{ .Values.global.texeraMicroServiceRuntimeClass | quote }} + {{- end }} restartPolicy: Never containers: - name: example-data-loader diff --git a/bin/k8s/templates/base/file-service/file-service-deployment.yaml b/bin/k8s/templates/base/file-service/file-service-deployment.yaml index 6a9190bc6c..b60eddf11b 100644 --- a/bin/k8s/templates/base/file-service/file-service-deployment.yaml +++ b/bin/k8s/templates/base/file-service/file-service-deployment.yaml @@ -32,6 +32,9 @@ spec: labels: app: {{ .Release.Name }}-{{ .Values.fileService.name }} spec: + {{- if .Values.global.texeraMicroServiceRuntimeClass }} + runtimeClassName: {{ .Values.global.texeraMicroServiceRuntimeClass | quote }} + {{- end }} containers: - name: {{ .Values.fileService.name }} image: {{ .Values.texera.imageRegistry }}/{{ .Values.fileService.imageName }}:{{ .Values.texera.imageTag }} diff --git a/bin/k8s/templates/base/lakefs/lakefs-setup-job.yaml b/bin/k8s/templates/base/lakefs/lakefs-setup-job.yaml index 914a614406..b7fdd51364 100644 --- a/bin/k8s/templates/base/lakefs/lakefs-setup-job.yaml +++ b/bin/k8s/templates/base/lakefs/lakefs-setup-job.yaml @@ -26,6 +26,9 @@ spec: metadata: name: {{ .Release.Name }}-lakefs-setup-job spec: + {{- if .Values.global.texeraMicroServiceRuntimeClass }} + runtimeClassName: {{ .Values.global.texeraMicroServiceRuntimeClass | quote }} + {{- end }} restartPolicy: Never containers: - name: setup-lakefs diff --git a/bin/k8s/templates/base/litellm/litellm-deployment.yaml b/bin/k8s/templates/base/litellm/litellm-deployment.yaml index ba681da7d1..7bc729ef87 100644 --- a/bin/k8s/templates/base/litellm/litellm-deployment.yaml +++ b/bin/k8s/templates/base/litellm/litellm-deployment.yaml @@ -33,6 +33,9 @@ spec: labels: app: {{ .Release.Name }}-{{ .Values.litellm.name }} spec: + {{- if .Values.global.texeraMicroServiceRuntimeClass }} + runtimeClassName: {{ .Values.global.texeraMicroServiceRuntimeClass | quote }} + {{- end }} containers: - name: {{ .Values.litellm.name }} image: {{ .Values.litellm.image }} diff --git a/bin/k8s/templates/base/webserver/webserver-deployment.yaml b/bin/k8s/templates/base/webserver/webserver-deployment.yaml index 983c626994..99a5887095 100644 --- a/bin/k8s/templates/base/webserver/webserver-deployment.yaml +++ b/bin/k8s/templates/base/webserver/webserver-deployment.yaml @@ -32,6 +32,9 @@ spec: labels: app: {{ .Release.Name }}-{{ .Values.webserver.name }} spec: + {{- if .Values.global.texeraMicroServiceRuntimeClass }} + runtimeClassName: {{ .Values.global.texeraMicroServiceRuntimeClass | quote }} + {{- end }} containers: - name: {{ .Values.webserver.name }} image: {{ .Values.texera.imageRegistry }}/{{ .Values.webserver.imageName }}:{{ .Values.texera.imageTag }} diff --git a/bin/k8s/templates/base/workflow-compiling-service/workflow-compiling-service-deployment.yaml b/bin/k8s/templates/base/workflow-compiling-service/workflow-compiling-service-deployment.yaml index 2061e9e53e..a538ce4a86 100644 --- a/bin/k8s/templates/base/workflow-compiling-service/workflow-compiling-service-deployment.yaml +++ b/bin/k8s/templates/base/workflow-compiling-service/workflow-compiling-service-deployment.yaml @@ -32,6 +32,9 @@ spec: labels: app: {{ .Release.Name }}-{{ .Values.workflowCompilingService.name }} spec: + {{- if .Values.global.texeraMicroServiceRuntimeClass }} + runtimeClassName: {{ .Values.global.texeraMicroServiceRuntimeClass | quote }} + {{- end }} containers: - name: {{ .Values.workflowCompilingService.name }} image: {{ .Values.texera.imageRegistry }}/{{ .Values.workflowCompilingService.imageName }}:{{ .Values.texera.imageTag }} diff --git a/bin/k8s/templates/base/workflow-computing-unit-manager/workflow-computing-unit-manager-deployment.yaml b/bin/k8s/templates/base/workflow-computing-unit-manager/workflow-computing-unit-manager-deployment.yaml index f529fe3653..1f6ef3f99b 100644 --- a/bin/k8s/templates/base/workflow-computing-unit-manager/workflow-computing-unit-manager-deployment.yaml +++ b/bin/k8s/templates/base/workflow-computing-unit-manager/workflow-computing-unit-manager-deployment.yaml @@ -32,6 +32,9 @@ spec: labels: app: {{ .Release.Name }}-{{ .Values.workflowComputingUnitManager.name }} spec: + {{- if .Values.global.texeraMicroServiceRuntimeClass }} + runtimeClassName: {{ .Values.global.texeraMicroServiceRuntimeClass | quote }} + {{- end }} serviceAccountName: {{ .Values.workflowComputingUnitManager.serviceAccountName }} initContainers: - name: wait-lakekeeper diff --git a/bin/k8s/templates/base/workflow-computing-unit-pool/workflow-computing-unit-master-prepull-daemonset.yaml b/bin/k8s/templates/base/workflow-computing-unit-pool/workflow-computing-unit-master-prepull-daemonset.yaml index 4edc0104a8..b85a6feedb 100644 --- a/bin/k8s/templates/base/workflow-computing-unit-pool/workflow-computing-unit-master-prepull-daemonset.yaml +++ b/bin/k8s/templates/base/workflow-computing-unit-pool/workflow-computing-unit-master-prepull-daemonset.yaml @@ -31,6 +31,9 @@ spec: labels: app: {{ .Release.Name }}-computing-unit-master-prepuller spec: + {{- if .Values.global.texeraMicroServiceRuntimeClass }} + runtimeClassName: {{ .Values.global.texeraMicroServiceRuntimeClass | quote }} + {{- end }} restartPolicy: Always tolerations: - operator: "Exists" diff --git a/bin/k8s/values.yaml b/bin/k8s/values.yaml index c72120a2e2..2058b2a0f8 100644 --- a/bin/k8s/values.yaml +++ b/bin/k8s/values.yaml @@ -22,6 +22,10 @@ texera: imageTag: 1.4.0-incubating-SNAPSHOT global: + # If using VMs for texera specific microservices, change this to the desired runtime otherwise leave as "". Recommended: "kata-qemu-runtime-rs" + # Note: kata containers must be installed to use kata-qemu-runtime-rs + # Note: This does not change the runtime of dependencies or Compute Units. + texeraMicroServiceRuntimeClass: "" # Required by the Bitnami postgresql sub-chart to allow custom images security: allowInsecureImages: true @@ -533,6 +537,14 @@ texeraEnvVars: - name: AUTH_JWT_SECRET # Development-only default (256-bit HS256 secret). Production environments MUST override this with a different, securely generated secret. value: "a7f3c8e9b14d2e6f5a0b9c3d8e1f4a6b2c5d7e9f0a3b6c8d1e4f7a9b2c5d8e1f" + - name: KUBERNETES_COMPUTING_UNIT_RUNTIME_CLASS_CPU + # If using VMs for CUs change this to the desired runtime otherwise leave as "". Recommended: "kata-qemu-runtime-rs" + # Note: kata containers must be installed to use kata-qemu-runtime-rs + value: "" + - name: KUBERNETES_COMPUTING_UNIT_RUNTIME_CLASS_GPU + # If using VMs for CUs change this to the desired runtime otherwise leave as "". Recommended: "kata-qemu-nvidia-gpu-runtime-rs" + # Note: kata containers must be installed to use kata-qemu-nvidia-gpu-runtime-r + value: "" yWebsocketServer: name: y-websocket-server diff --git a/computing-unit-managing-service/src/main/scala/org/apache/texera/service/util/KubernetesClient.scala b/computing-unit-managing-service/src/main/scala/org/apache/texera/service/util/KubernetesClient.scala index 704513159d..e88c0ea50c 100644 --- a/computing-unit-managing-service/src/main/scala/org/apache/texera/service/util/KubernetesClient.scala +++ b/computing-unit-managing-service/src/main/scala/org/apache/texera/service/util/KubernetesClient.scala @@ -194,9 +194,14 @@ class KubernetesClient( .endMetadata() .withNewSpec() - // Only add runtimeClassName when using NVIDIA GPU - if (gpuLimit != "0" && KubernetesConfig.gpuResourceKey.contains("nvidia")) { - specBuilder.withRuntimeClassName("nvidia") + val runtimeClassOpt: Option[String] = if (gpuLimit != "0") { + sys.env.get("KUBERNETES_COMPUTING_UNIT_RUNTIME_CLASS_GPU").filter(_.nonEmpty) + } else { + sys.env.get("KUBERNETES_COMPUTING_UNIT_RUNTIME_CLASS_CPU").filter(_.nonEmpty) + } + + runtimeClassOpt.foreach { runtimeClass => + specBuilder.withRuntimeClassName(runtimeClass) } val containerBuilder = specBuilder
