This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-8073-a89740810bec0d8487f33e53370e40a9adbbdfff in repository https://gitbox.apache.org/repos/asf/texera.git
commit bda3400261d38e674086f3b44cfcae169ccc55e5 Author: Ryan Zhang <[email protected]> AuthorDate: Wed Sep 9 00:05:53 2026 +0000 feat(notebook-migration, k8s): deploy the notebook migration tool on Kubernetes (#8073) ### What changes were proposed in this PR? The notebook migration tool runs in the single-node Docker stack (#7931) and in local development (#7933), but the Helm chart has no `notebook-migration-service` and no JupyterLab. A Kubernetes deployment shows the tool in the workspace with nothing serving its endpoints. This adds both, following the chart's existing convention that orchestrator services are global and stateful resources are per user. `notebook-migration-service` becomes a global Deployment alongside the other services, with a ServiceAccount whose Role is scoped to the JupyterLab namespace only. Per-user JupyterLab pods live in their own namespace behind a headless Service, addressed as `jupyter-<uid>.<service>.<namespace>.svc.cluster.local`, with a ResourceQuota bounding the pool and an optional prepull DaemonSet. A NetworkPolicy stops one user's pod reaching another's. One `values.yaml` switch, `notebookMigrationService.enabled`, gates the Deployment, the routes, the pool, and the button in the workspace, and it defaults to off. **Enabling the tool now requires a token secret** `notebookMigrationService.jupyterTokenSecret` has no default. Every user's JupyterLab token is `HMAC(secret, uid)` and `/jupyter/<uid>/` is deliberately unauthenticated at the gateway, so a secret shipped in this repo would be public and would let anyone derive any user's token. The Secret template wraps the value in `required`, so enabling the tool without one fails at install with a message naming the key rather than coming up quietly insecure. Generate one with `openssl rand -hex 32`. `values-development.yaml` carries an explicitly dev scoped value so the Minikube flow keeps working. **How browser traffic reaches the right pod** This is the design decision worth reviewing closely. JupyterLab is loaded in an iframe and then issues its own requests for assets, contents and kernel websockets. None of those can carry a Texera token, and Texera has no session cookie, so the caller cannot be authenticated per request. Each user's JupyterLab is therefore served under `/jupyter/<uid>/`, and the gateway's existing ExtAuthz hook resolves that uid to the pod's recorded address and rewrites `Host`. It reuses `dynamic-routes`, its `DynamicResolver` backend, and the `SecurityPolicy` that already forwards `Host`, so no new gateway machinery is introduced. That mechanism routes; it does not authorize. What keeps users apart is the per-user JupyterLab token from #7665, derived from a server held secret and unguessable. Anyone who can reach the gateway can route to any user's pod and will get a 403 from JupyterLab without that user's token. The NetworkPolicy is defence in depth on top, closing the one case where a genuinely hostile neighbour runs: another user's pod. This is a weaker posture than a per-request authorization check, and it is stated plainly in the code rather than left to be inferred. Resolution fails closed. A uid too large for an `Int`, a missing registry row, and a recorded address that will not parse all take one path to 403, never a 500. **This PR is not Helm only, unlike what #8006 anticipated** Four backend changes are inseparable from a working deployment: * JupyterLab serves under a configurable base path, since routing by path requires it, and the recorded internal address carries that path because JupyterLab serves every endpoint under it, `/api` included. * `access-control-service` gains the Jupyter case in `authorize()`. The uid regex is built from `kubernetes.jupyter-base-url` and the prefix is quoted, so a base path is matched as a path and not as a pattern. * JupyterLab is told which origin may embed it. The setting and the pod's `TEXERA_ORIGIN` arrived with #8032; this PR passes it to `--NotebookApp.allow_origin`, without which a proxy that rewrites `Host` makes JupyterLab's own cross origin check fail on cookie authenticated requests and kernel startup is blocked entirely. * The browser facing origin is settable directly, not only derived from a gateway hostname, because a deployment reached by port forward or NodePort has no hostname to derive from. When it is derived, the scheme comes from whether a certificate is configured, `gatewayConfig.issuer` or `tlsSecretName`, because the chart declares both listeners unconditionally and a hostname alone does not say which one serves. `jupyterPool.basePath` is honoured by every layer that reads it: the pod's `base_url`, the recorded internal URL, the public URL template, the gateway route match, and the regex `access-control-service` matches on. All five derive from the one value, so the prefix cannot be obeyed in one place and ignored in another. **Rebuilt pods and pooled connections** A Jupyter pod is named after its uid, so a rebuild reuses the hostname with a new IP. The gateway keeps pooled upstream connections to the old address for its default idle hour, and a request handed one of those hangs until the route timeout, because a departed pod IP is unrouted rather than refused. A BackendTrafficPolicy sets `connectionIdleTimeout` to 30s on the dynamic route so those connections are retired promptly. Measured on the cluster below, a rebuild went from 14 failures in 40 requests scattered over minutes to 4, confined to the moment of the switch. Computing units share this route and are unaffected either way: their pods are keyed on a never reused cuid, so no hostname of theirs ever moves. **Two routes needed explicit timeouts** Envoy's default request timeout is 15 seconds, which is shorter than two things this tool does. `/api/notebook-migration` now carries a 3m timeout. Provisioning waits for a pod to terminate and then to answer, up to 60s each, so the default was truncating work the service would have completed: rebuilds measured 30 seconds and returned 504 to the browser while the pod came up fine underneath, with an immediate retry succeeding. The route serving `/api/chat` and `/api/models` takes its timeout from `gatewayConfig.llmRequestTimeout`, defaulting to 10m to match `python-notebook-migration-timeout-minutes`. LLM completions routinely run past 15 seconds, and the upstream call succeeds and is then discarded, so every conversion failed while burning the API call. This also fixes the agent chat, which shares the route. Known limitations, both worth their own issues: * JupyterLab pods have no persistent volume, so a pod restart empties `work/`. Notebooks survive in the database but are not re-uploaded automatically. The same is true of a single-node container restart, so this is not specific to Kubernetes. * A dead pod self-heals on the next request, but an idle one is never reclaimed. ### Any related issues, documentation, discussions? Closes #8006 Parent issue #4301 Builds on #7665 (PR #8032, merged), which resolves the Jupyter URL and token per user. Roadmap context is the architectural note on #5258. ### How was this PR tested? Unit tests pass across the touched modules: `Config/test` (71), `AccessControlService/test` (68), `NotebookMigrationService/test` (122), with `scalafmtCheckAll` clean. `helm template` renders with the feature on and off; with it disabled the chart emits no Jupyter or notebook-migration objects at all, and with it enabled but no token secret the render fails as intended. Deployed and exercised end to end on a local Minikube cluster running Calico, chosen because Minikube's default CNI creates NetworkPolicy objects without enforcing them, which would make the isolation commit look correct while enforcing nothing. All seven endpoints were driven through the gateway: `get-jupyter-url`, `get-jupyter-iframe-url`, `set-notebook`, `delete-notebook`, and the three mapping endpoints. `set-notebook` was confirmed by reading the file back through Jupyter's own contents API, and `delete-notebook` by that read then returning 404. Requests with no token, a garbage token, and an `INACTIVE` role are refused; `../../etc/x.ipynb`, a non `.ipynb` name, and malformed JSON bodies are all 400. Per-user separation was tested with two real users. Each got its own pod, the quota moved from one to two, and the registry held two rows with uid scoped addresses. Each served token matched an independent HMAC derivation of that uid, and each token returned 200 on its own pod and 403 on the other. Routing to a uid with no pod, and to a uid that does not exist, both return 403. Isolation was tested with an ablation rather than a single observation. `jupyter-1` cannot reach `jupyter-2` on 8888 by pod IP or by DNS, in both directions, while the same pod reaches the `texera-dev` namespace in the same probe, which rules out broken networking. Removing the policy makes the connection succeed and restoring it blocks again. Kernel startup was tested the way a browser does it, with a cookie and an `Origin` header rather than a token query parameter, because a token authenticated request skips JupyterLab's origin check and would pass even when the browser path is broken. `POST /api/sessions` returns 201 and the pod logs `Kernel started`. Cell click sync was confirmed in the browser. The failure paths were provoked rather than assumed. Writing an unparseable address into a registry row returns 403 and logs the `URISyntaxException`, and 200 returns once restored. Deleting a pod without touching its row causes the next request to log that the pod no longer accepts its current token, discard the row and provision a replacement. Setting `jupyterPool.basePath` to `/lab/notebooks` moves the pod's `base_url`, the recorded URL, the gateway route and the regex together, and removing `KUBERNETES_JUPYTER_BASE_URL` from `access-control-service` restores the break, which is what proves the wiring rather than the default doing the work. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) --------- Co-authored-by: Meng Wang <[email protected]> --- .../service/resource/AccessControlResource.scala | 54 ++++++++- .../apache/texera/AccessControlResourceSpec.scala | 130 +++++++++++++++++++++ bin/k8s/templates/base/_helpers.tpl | 6 + .../access-control-service-deployment.yaml | 5 + .../config-service/config-service-deployment.yaml | 5 + .../gateway/gateway-jupyter-traffic-policy.yaml | 40 +++++++ bin/k8s/templates/base/gateway/gateway-routes.yaml | 24 ++++ .../base/jupyter-pool/jupyter-namespace.yaml | 22 ++++ .../base/jupyter-pool/jupyter-network-policy.yaml | 49 ++++++++ .../jupyter-pool/jupyter-prepull-daemonset.yaml | 57 +++++++++ .../base/jupyter-pool/jupyter-resource-quota.yaml | 29 +++++ .../base/jupyter-pool/jupyter-service.yaml | 36 ++++++ .../notebook-migration-service-deployment.yaml | 105 +++++++++++++++++ .../notebook-migration-service-secret.yaml | 30 +++++ ...notebook-migration-service-service-account.yaml | 49 ++++++++ .../notebook-migration-service-service.yaml | 31 +++++ bin/k8s/values-development.yaml | 24 ++++ bin/k8s/values.yaml | 62 ++++++++++ common/config/src/main/resources/kubernetes.conf | 6 + .../texera/common/config/KubernetesConfig.scala | 3 +- .../common/config/KubernetesConfigSpec.scala | 2 + .../src/main/resources/start-texera-jupyter.sh | 14 ++- .../service/util/JupyterKubernetesClient.scala | 12 ++ .../texera/service/util/JupyterProvisioner.scala | 4 +- .../resource/NotebookMigrationResourceSpec.scala | 30 ++++- .../service/util/JupyterKubernetesClientSpec.scala | 17 +++ 26 files changed, 833 insertions(+), 13 deletions(-) diff --git a/access-control-service/src/main/scala/org/apache/texera/service/resource/AccessControlResource.scala b/access-control-service/src/main/scala/org/apache/texera/service/resource/AccessControlResource.scala index 792a0dfd8a..2d209be59f 100644 --- a/access-control-service/src/main/scala/org/apache/texera/service/resource/AccessControlResource.scala +++ b/access-control-service/src/main/scala/org/apache/texera/service/resource/AccessControlResource.scala @@ -27,12 +27,12 @@ import jakarta.ws.rs.{Consumes, DELETE, GET, POST, PUT, Path, Produces} import org.apache.texera.auth.JwtParser.parseToken import org.apache.texera.auth.SessionUser import org.apache.texera.auth.util.{ComputingUnitAccess, HeaderField} -import org.apache.texera.common.config.{GuiConfig, LLMConfig} +import org.apache.texera.common.config.{GuiConfig, KubernetesConfig, LLMConfig} import org.apache.texera.dao.SqlServer import org.apache.texera.dao.jooq.generated.enums.PrivilegeEnum -import org.apache.texera.dao.jooq.generated.tables.daos.WorkflowComputingUnitDao +import org.apache.texera.dao.jooq.generated.tables.daos.{UserJupyterDao, WorkflowComputingUnitDao} -import java.net.URLDecoder +import java.net.{URI, URLDecoder} import java.nio.charset.StandardCharsets import java.util.Optional import scala.jdk.CollectionConverters.{CollectionHasAsScala, MapHasAsScala} @@ -51,6 +51,18 @@ object AccessControlResource extends LazyLogging { private val pvePvesCuidPath: Regex = """^/?(?:auth/)?(?:api/|wsapi/)?pve/pves/([0-9]+)$""".r private val pvePackagesCuidPath: Regex = """^/?(?:auth/)?(?:api/|wsapi/)?pve/([0-9]+)/[^/]+/packages/.+$""".r + // Per-user JupyterLab. The uid is in the path because a browser cannot attach Texera + // credentials to the requests Jupyter's own scripts make, so it is the only place the + // owner can be read from. + private val jupyterPath: Regex = jupyterPathRegex(KubernetesConfig.jupyterBaseUrl) + + // Built from the same setting the gateway route and the pods are rendered from, so the + // prefix cannot be honoured in one place and not another. Quoted: it is a path, not a + // pattern. + private[texera] def jupyterPathRegex(basePath: String): Regex = { + val prefix = basePath.stripPrefix("/").stripSuffix("/") + ("^/?(?:auth/)?" + Regex.quote(prefix) + "/([0-9]+)(?:/.*)?$").r + } /** * Authorize the request based on the path and headers. @@ -68,6 +80,7 @@ object AccessControlResource extends LazyLogging { logger.info(s"Authorizing request for path: $path") path match { + case jupyterPath(uid) => routeToJupyter(uid) case wsapiWorkflowWebsocket() | apiExecutionsStats() | apiExecutionsResultExport() | pveRoute() => checkComputingUnitAccess(uriInfo, headers, bodyOpt) @@ -77,6 +90,41 @@ object AccessControlResource extends LazyLogging { } } + /** + * Resolve which JupyterLab pod a request belongs to. This routes; it does not authorize. + * + * Jupyter is loaded in an iframe and then issues its own requests for assets, contents and + * kernel websockets. None of those can carry a Texera token, and there is no session cookie + * to fall back on, so the caller cannot be authenticated per request. What protects one + * user's notebooks from another is the per-user Jupyter token, which is derived from a + * server-held secret and is unguessable; reaching the right pod without it yields a 403 from + * Jupyter itself. Cross-pod traffic is blocked separately by a NetworkPolicy. + */ + private def routeToJupyter(uid: String): Response = { + // Envoy routes on an authority, so the scheme and base path are stripped off. The parse + // stays inside the guard so a malformed row is denied rather than raised as a 500. + val authority = + try { + val dao = new UserJupyterDao(SqlServer.getInstance().createDSLContext().configuration()) + Option(dao.fetchOneByUid(uid.toInt)) + .map(row => new URI(row.getInternalUrl).getAuthority) + .filter(a => a != null && a.nonEmpty) + } catch { + case e: Exception => + logger.error(s"Failed to resolve the Jupyter registered for user $uid", e) + return Response.status(Response.Status.FORBIDDEN).build() + } + + authority match { + case Some(host) => + logger.info(s"Routing Jupyter for user $uid to recorded host: $host") + Response.ok().header("Host", host).build() + case None => + logger.warn(s"Refusing Jupyter for user $uid: no usable Jupyter address is registered") + Response.status(Response.Status.FORBIDDEN).build() + } + } + private def checkComputingUnitAccess( uriInfo: UriInfo, headers: HttpHeaders, diff --git a/access-control-service/src/test/scala/org/apache/texera/AccessControlResourceSpec.scala b/access-control-service/src/test/scala/org/apache/texera/AccessControlResourceSpec.scala index 10ff44db7f..2892a6026a 100644 --- a/access-control-service/src/test/scala/org/apache/texera/AccessControlResourceSpec.scala +++ b/access-control-service/src/test/scala/org/apache/texera/AccessControlResourceSpec.scala @@ -27,11 +27,13 @@ import org.apache.texera.dao.jooq.generated.enums.{ WorkflowComputingUnitTypeEnum } import org.apache.texera.dao.jooq.generated.tables.daos.{ + UserJupyterDao, ComputingUnitUserAccessDao, UserDao, WorkflowComputingUnitDao } import org.apache.texera.dao.jooq.generated.tables.pojos.{ + UserJupyter, ComputingUnitUserAccess, User, WorkflowComputingUnit @@ -73,6 +75,11 @@ class AccessControlResourceSpec private val testNoAccessRecordedUri: String = "computing-unit-6.compute-unit-svc.default.svc.cluster.local:7777" + // What the provisioner records for a user's Jupyter: scheme, authority and the base path + // the pod serves under. Only the authority may reach Envoy as a Host header. + private val testJupyterInternalUrl: String = + "http://jupyter-1.jupyter-svc.texera-jupyter-pool.svc.cluster.local:8888/jupyter/1"; + private val testUser1: User = { val user = new User() user.setUid(1) @@ -190,6 +197,14 @@ class AccessControlResourceSpec readOnlyAccess.setPrivilege(PrivilegeEnum.READ) computingUnitOfUserDao.insert(readOnlyAccess) + // Per-user Jupyter: user 1 has one registered, user 2 deliberately does not. + val jupyterDao = new UserJupyterDao(getDSLContext.configuration()) + val jupyter = new UserJupyter() + jupyter.setUid(testUser1.getUid) + jupyter.setInternalUrl(testJupyterInternalUrl) + jupyter.setPublicUrl("https://texera.example.com/jupyter/1";) + jupyterDao.insert(jupyter) + token = JwtAuth.jwtToken(JwtAuth.jwtClaims(testUser1)) token2 = JwtAuth.jwtToken(JwtAuth.jwtClaims(testUser2)) } @@ -722,4 +737,119 @@ class AccessControlResourceSpec response.getStatus shouldBe Response.Status.OK.getStatusCode response.getHeaderString("Host") shouldBe testRecordedUri } + + // -- per-user JupyterLab routing -------------------------------------------- + + it should "route a Jupyter request to the pod recorded for the uid in the path" in { + val (uri, headers) = mockRequest("/jupyter/1/notebooks/work/notebook.ipynb", None) + val response = new AccessControlResource().authorizeGet(uri, headers) + + response.getStatus shouldBe Response.Status.OK.getStatusCode + // The scheme and base path are stripped: Envoy routes on an authority alone. + response.getHeaderString("Host") shouldBe + "jupyter-1.jupyter-svc.texera-jupyter-pool.svc.cluster.local:8888" + } + + it should "route Jupyter's own subrequests, which carry no token" in { + // The iframe's asset and API calls cannot present Texera credentials, so routing has to + // work without one. The per-user Jupyter token is what authorizes them. + val (uri, headers) = + mockRequest("/jupyter/1/api/contents", None, authorizationHeader = None) + val response = new AccessControlResource().authorizeGet(uri, headers) + + response.getStatus shouldBe Response.Status.OK.getStatusCode + response.getHeaderString("Host") should startWith("jupyter-1.") + } + + it should "refuse a Jupyter request for a user with none registered" in { + val (uri, headers) = mockRequest("/jupyter/2/tree", None) + new AccessControlResource() + .authorizeGet(uri, headers) + .getStatus shouldBe Response.Status.FORBIDDEN.getStatusCode + } + + it should "refuse a Jupyter request for a uid that does not exist" in { + val (uri, headers) = mockRequest("/jupyter/999999/tree", None) + new AccessControlResource() + .authorizeGet(uri, headers) + .getStatus shouldBe Response.Status.FORBIDDEN.getStatusCode + } + + it should "not treat a Jupyter path without a uid as routable" in { + // Falls through to the catch-all, which denies. + val (uri, headers) = mockRequest("/jupyter/tree", None) + new AccessControlResource() + .authorizeGet(uri, headers) + .getStatus shouldBe Response.Status.FORBIDDEN.getStatusCode + } + + it should "route the gateway-relative form of a Jupyter path" in { + val (uri, headers) = mockRequest("auth/jupyter/1/tree", None) + new AccessControlResource() + .authorizeGet(uri, headers) + .getStatus shouldBe Response.Status.OK.getStatusCode + } + + it should "refuse a Jupyter uid too large to be a user id" in { + // The path regex accepts any run of digits, so the conversion to Int has to be caught + // rather than allowed out as a 500. + val (uri, headers) = mockRequest("/jupyter/9999999999/tree", None) + new AccessControlResource() + .authorizeGet(uri, headers) + .getStatus shouldBe Response.Status.FORBIDDEN.getStatusCode + } + + // Registers an address for testUser2, who otherwise has none, for the rows that should be + // refused rather than routed. + private def withRecordedJupyter(internalUrl: String)(check: => Unit): Unit = { + val jupyterDao = new UserJupyterDao(getDSLContext.configuration()) + val row = new UserJupyter() + row.setUid(testUser2.getUid) + row.setInternalUrl(internalUrl) + row.setPublicUrl(internalUrl) + jupyterDao.insert(row) + try check + finally jupyterDao.deleteById(testUser2.getUid) + } + + it should "refuse a Jupyter request whose recorded address has no authority" in { + // Parses cleanly but yields a null authority, which must not become a Host header of + // "null" for Envoy to route on. + withRecordedJupyter("jupyter-2-with-no-scheme:8888") { + val (uri, headers) = mockRequest(s"/jupyter/${testUser2.getUid}/tree", None) + new AccessControlResource() + .authorizeGet(uri, headers) + .getStatus shouldBe Response.Status.FORBIDDEN.getStatusCode + } + } + + it should "build the uid regex from the configured base path" in { + val custom = AccessControlResource.jupyterPathRegex("/lab/notebooks") + "/lab/notebooks/7/tree" should fullyMatch regex custom + "auth/lab/notebooks/7" should fullyMatch regex custom + "/jupyter/7/tree" should not(fullyMatch regex custom) + } + + it should "tolerate a base path written with surrounding slashes" in { + val slashed = AccessControlResource.jupyterPathRegex("/jupyter/") + "/jupyter/7/tree" should fullyMatch regex slashed + } + + it should "treat the base path as a path, not a pattern" in { + // Unquoted, the dot would match any character and route /axb to the /a.b pool. + val dotted = AccessControlResource.jupyterPathRegex("/a.b") + "/a.b/7" should fullyMatch regex dotted + "/axb/7" should not(fullyMatch regex dotted) + } + + it should "refuse a Jupyter request whose recorded address will not parse" in { + // A stray escape throws from the URI constructor. That parse sits inside the same guard + // as the lookup, so this is denied like any other unusable row instead of raising a 500. + withRecordedJupyter("http://jupyter-2:8888/%zz";) { + val (uri, headers) = mockRequest(s"/jupyter/${testUser2.getUid}/tree", None) + new AccessControlResource() + .authorizeGet(uri, headers) + .getStatus shouldBe Response.Status.FORBIDDEN.getStatusCode + } + } } diff --git a/bin/k8s/templates/base/_helpers.tpl b/bin/k8s/templates/base/_helpers.tpl index bcb8e33a2d..a4e2b9e130 100644 --- a/bin/k8s/templates/base/_helpers.tpl +++ b/bin/k8s/templates/base/_helpers.tpl @@ -77,3 +77,9 @@ values override would let an install quietly hand the privileged mounter to anot {{- printf "system:serviceaccount:%s:%s" .Release.Namespace .Values.accessControlService.serviceAccountName -}} {{- end -}} +{{/* Jupyter base path, as exactly one leading slash and no trailing one. Several places +append the uid to it, so a bare value would render "http://<origin>jupyter/7". */}} +{{- define "texera.jupyter.basePath" -}} +{{- printf "/%s" (trimAll "/" .Values.jupyterPool.basePath) -}} +{{- end -}} + diff --git a/bin/k8s/templates/base/access-control-service/access-control-service-deployment.yaml b/bin/k8s/templates/base/access-control-service/access-control-service-deployment.yaml index 85fbbdd5f7..62bb8f3b21 100644 --- a/bin/k8s/templates/base/access-control-service/access-control-service-deployment.yaml +++ b/bin/k8s/templates/base/access-control-service/access-control-service-deployment.yaml @@ -55,6 +55,11 @@ spec: value: {{ .Values.workflowComputingUnitPool.name }} - name: KUBERNETES_COMPUTE_UNIT_POOL_NAMESPACE value: {{ .Values.workflowComputingUnitPool.namespace }} + {{- if .Values.notebookMigrationService.enabled }} + # The uid regex this service routes Jupyter on is built from the same prefix. + - name: KUBERNETES_JUPYTER_BASE_URL + value: {{ include "texera.jupyter.basePath" . }} + {{- end }} {{- if .Values.litellm.enabled }} # LLM gateway used to serve /api/chat and /api/models to the agent service. - name: LITELLM_BASE_URL diff --git a/bin/k8s/templates/base/config-service/config-service-deployment.yaml b/bin/k8s/templates/base/config-service/config-service-deployment.yaml index f0748785c3..d6775483eb 100644 --- a/bin/k8s/templates/base/config-service/config-service-deployment.yaml +++ b/bin/k8s/templates/base/config-service/config-service-deployment.yaml @@ -47,6 +47,11 @@ spec: secretKeyRef: name: {{ .Release.Name }}-postgresql key: postgres-password + # Shows or hides the notebook migration tool in the workspace. Derived from the + # service's own toggle rather than listed in texeraEnvVars, so enabling the tool + # is one switch instead of two that can disagree. + - name: GUI_WORKFLOW_WORKSPACE_PYTHON_NOTEBOOK_MIGRATION_ENABLED + value: "{{ .Values.notebookMigrationService.enabled }}" {{- range .Values.texeraEnvVars }} - name: {{ .name }} value: "{{ .value }}" diff --git a/bin/k8s/templates/base/gateway/gateway-jupyter-traffic-policy.yaml b/bin/k8s/templates/base/gateway/gateway-jupyter-traffic-policy.yaml new file mode 100644 index 0000000000..4dd4e809ce --- /dev/null +++ b/bin/k8s/templates/base/gateway/gateway-jupyter-traffic-policy.yaml @@ -0,0 +1,40 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Jupyter pods are named after the uid, so a rebuilt pod reuses its predecessor's hostname +# with a new IP. The gateway keeps pooled upstream connections to the old address for the +# default idle hour, and requests handed one of those hang until the route timeout, since a +# departed pod IP is unrouted rather than refused. Retiring idle connections quickly bounds +# that window. Computing units on this route are unaffected either way: their pods are keyed +# on a never-reused cuid, so no hostname of theirs ever moves. +# +# Active requests hold the connection out of idle, so a running kernel's websocket is not cut. +{{- if .Values.notebookMigrationService.enabled }} +apiVersion: gateway.envoyproxy.io/v1alpha1 +kind: BackendTrafficPolicy +metadata: + name: {{ .Release.Name }}-jupyter-traffic-policy + namespace: {{ .Release.Namespace }} +spec: + targetRefs: + - group: gateway.networking.k8s.io + kind: HTTPRoute + name: {{ .Release.Name }}-dynamic-routes + timeout: + http: + connectionIdleTimeout: 30s +{{- end }} diff --git a/bin/k8s/templates/base/gateway/gateway-routes.yaml b/bin/k8s/templates/base/gateway/gateway-routes.yaml index f07a3157a4..eb88a2a09d 100644 --- a/bin/k8s/templates/base/gateway/gateway-routes.yaml +++ b/bin/k8s/templates/base/gateway/gateway-routes.yaml @@ -74,6 +74,19 @@ spec: backendRefs: - name: config-service-svc port: 9094 + {{- if .Values.notebookMigrationService.enabled }} + - matches: + - path: + type: PathPrefix + value: /api/notebook-migration + # Provisioning waits for a pod to terminate and then to answer, 60s each, so Envoy's + # 15s default cuts off requests the service would have completed. + timeouts: + request: "3m" + backendRefs: + - name: {{ .Values.notebookMigrationService.name }}-svc + port: {{ .Values.notebookMigrationService.service.port }} + {{- end }} - matches: - path: type: PathPrefix @@ -81,6 +94,10 @@ spec: - path: type: PathPrefix value: /api/chat + {{- if and .Values.gatewayConfig .Values.gatewayConfig.llmRequestTimeout }} + timeouts: + request: {{ .Values.gatewayConfig.llmRequestTimeout | quote }} + {{- end }} backendRefs: - name: access-control-service-svc port: 9096 @@ -135,6 +152,13 @@ spec: - path: type: PathPrefix value: /api/pve + {{- if .Values.notebookMigrationService.enabled }} + # Per-user JupyterLab. ExtAuthz reads the uid from the path and rewrites Host to + # that user's pod; the per-user Jupyter token is what authorizes the request. + - path: + type: PathPrefix + value: {{ include "texera.jupyter.basePath" . }} + {{- end }} backendRefs: - group: gateway.envoyproxy.io kind: Backend diff --git a/bin/k8s/templates/base/jupyter-pool/jupyter-namespace.yaml b/bin/k8s/templates/base/jupyter-pool/jupyter-namespace.yaml new file mode 100644 index 0000000000..9e228d919e --- /dev/null +++ b/bin/k8s/templates/base/jupyter-pool/jupyter-namespace.yaml @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +{{- if and .Values.notebookMigrationService.enabled .Values.jupyterPool.createNamespaces }} +apiVersion: v1 +kind: Namespace +metadata: + name: {{ .Values.jupyterPool.namespace }} +{{- end }} diff --git a/bin/k8s/templates/base/jupyter-pool/jupyter-network-policy.yaml b/bin/k8s/templates/base/jupyter-pool/jupyter-network-policy.yaml new file mode 100644 index 0000000000..37d9184957 --- /dev/null +++ b/bin/k8s/templates/base/jupyter-pool/jupyter-network-policy.yaml @@ -0,0 +1,49 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +{{- if and .Values.notebookMigrationService.enabled .Values.jupyterPool.networkPolicy.enabled }} +# Stops one user's JupyterLab from reaching another's, and stops computing unit pods +# reaching any of them. Both pools run user code by design and neither has any reason to +# call the other, so both are denied. Allowing every other namespace leaves the real callers +# working: the notebook migration service, and the Envoy proxy wherever the gateway +# installation runs it. +# +# Defence in depth, not the authorisation boundary: the per-user Jupyter token is what stops +# one user reading another's notebooks. Egress is left alone, since notebooks legitimately +# install packages and call out. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: {{ .Values.jupyterPool.name }}-deny-cross-user + namespace: {{ .Values.jupyterPool.namespace }} +spec: + podSelector: + matchLabels: + type: jupyter + policyTypes: + - Ingress + ingress: + - from: + # kubernetes.io/metadata.name is set automatically on every namespace, so the two + # pools can be named here without labelling anything by hand. + - namespaceSelector: + matchExpressions: + - key: kubernetes.io/metadata.name + operator: NotIn + values: + - {{ .Values.jupyterPool.namespace }} + - {{ .Values.workflowComputingUnitPool.namespace }} +{{- end }} diff --git a/bin/k8s/templates/base/jupyter-pool/jupyter-prepull-daemonset.yaml b/bin/k8s/templates/base/jupyter-pool/jupyter-prepull-daemonset.yaml new file mode 100644 index 0000000000..e6fca48a27 --- /dev/null +++ b/bin/k8s/templates/base/jupyter-pool/jupyter-prepull-daemonset.yaml @@ -0,0 +1,57 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +{{- if and .Values.notebookMigrationService.enabled .Values.jupyterPool.prepullImage }} +# Pulls the JupyterLab image onto every node ahead of time. Pods are created on demand and +# the service waits a bounded time for one to answer, so a first-time pull on a cold node can +# outlast that wait and the provisioning attempt is discarded. Mirrors the computing unit +# pool's prepuller. Set jupyterPool.prepullImage to false to trade cold starts for one fewer +# pod per node. +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: {{ .Release.Name }}-jupyter-prepuller + namespace: {{ .Release.Namespace }} + labels: + app: {{ .Release.Name }}-jupyter-prepuller +spec: + selector: + matchLabels: + app: {{ .Release.Name }}-jupyter-prepuller + template: + metadata: + labels: + app: {{ .Release.Name }}-jupyter-prepuller + spec: + restartPolicy: Always + tolerations: + - operator: "Exists" + initContainers: + - name: prepuller + image: {{ .Values.texera.imageRegistry }}/{{ .Values.jupyterPool.imageName }}:{{ .Values.texera.imageTag }} + imagePullPolicy: {{ .Values.texeraImages.pullPolicy }} + command: ["sh", "-c", "true"] + containers: + - name: pause + image: gcr.io/google_containers/pause:3.2 + resources: + limits: + cpu: 1m + memory: 8Mi + requests: + cpu: 1m + memory: 8Mi +{{- end }} diff --git a/bin/k8s/templates/base/jupyter-pool/jupyter-resource-quota.yaml b/bin/k8s/templates/base/jupyter-pool/jupyter-resource-quota.yaml new file mode 100644 index 0000000000..7e186b18eb --- /dev/null +++ b/bin/k8s/templates/base/jupyter-pool/jupyter-resource-quota.yaml @@ -0,0 +1,29 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +{{- if and .Values.notebookMigrationService.enabled .Values.jupyterPool.createNamespaces .Values.jupyterPool.maxRequestedResources }} +# Ceiling for the pool as a whole. Pods are created on demand, one per user, so without this +# a busy deployment has no upper bound on what the tool can consume. +apiVersion: v1 +kind: ResourceQuota +metadata: + name: {{ .Values.jupyterPool.name }}-resource-quota + namespace: {{ .Values.jupyterPool.namespace }} +spec: + hard: + requests.cpu: "{{ .Values.jupyterPool.maxRequestedResources.cpu }}" + requests.memory: {{ .Values.jupyterPool.maxRequestedResources.memory }} +{{- end }} diff --git a/bin/k8s/templates/base/jupyter-pool/jupyter-service.yaml b/bin/k8s/templates/base/jupyter-pool/jupyter-service.yaml new file mode 100644 index 0000000000..2f0fb1f5e9 --- /dev/null +++ b/bin/k8s/templates/base/jupyter-pool/jupyter-service.yaml @@ -0,0 +1,36 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +{{- if .Values.notebookMigrationService.enabled }} +# Headless, so each user's pod is addressable individually rather than load balanced across +# the pool: <pod>.<this service>.<namespace>.svc.cluster.local. The notebook migration +# service creates pods whose hostname is the pod name and whose subdomain is this service's +# name, which is what makes that address resolve. The selector matches the "type" label the +# service stamps on every JupyterLab pod it creates. +apiVersion: v1 +kind: Service +metadata: + name: {{ .Values.jupyterPool.name }}-svc + namespace: {{ .Values.jupyterPool.namespace }} +spec: + clusterIP: None + selector: + type: jupyter + ports: + - protocol: TCP + port: {{ .Values.jupyterPool.service.port }} + targetPort: {{ .Values.jupyterPool.service.targetPort }} +{{- end }} diff --git a/bin/k8s/templates/base/notebook-migration-service/notebook-migration-service-deployment.yaml b/bin/k8s/templates/base/notebook-migration-service/notebook-migration-service-deployment.yaml new file mode 100644 index 0000000000..2d18f7dc6f --- /dev/null +++ b/bin/k8s/templates/base/notebook-migration-service/notebook-migration-service-deployment.yaml @@ -0,0 +1,105 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +{{- if .Values.notebookMigrationService.enabled }} +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ .Release.Name }}-{{ .Values.notebookMigrationService.name }} + namespace: {{ .Release.Namespace }} + labels: + app: {{ .Release.Name }}-{{ .Values.notebookMigrationService.name }} +spec: + replicas: {{ .Values.notebookMigrationService.numOfPods | default 1 }} + selector: + matchLabels: + app: {{ .Release.Name }}-{{ .Values.notebookMigrationService.name }} + template: + metadata: + labels: + app: {{ .Release.Name }}-{{ .Values.notebookMigrationService.name }} + spec: + # Needed to create and delete each user's JupyterLab pod in the pool namespace. + serviceAccountName: {{ .Values.notebookMigrationService.serviceAccountName }} + containers: + - name: {{ .Values.notebookMigrationService.name }} + image: {{ .Values.texera.imageRegistry }}/{{ .Values.notebookMigrationService.imageName }}:{{ .Values.texera.imageTag }} + imagePullPolicy: {{ .Values.texeraImages.pullPolicy }} + ports: + - containerPort: {{ .Values.notebookMigrationService.service.port }} + env: + - name: STORAGE_JDBC_URL + value: jdbc:postgresql://{{ .Release.Name }}-postgresql:5432/texera_db?currentSchema=texera_db,public + - name: STORAGE_JDBC_PASSWORD + valueFrom: + secretKeyRef: + name: {{ .Release.Name }}-postgresql + key: postgres-password + # Resolve each user's JupyterLab rather than one shared server. + - name: KUBERNETES_JUPYTER_ENABLED + value: "true" + - name: KUBERNETES_JUPYTER_NAMESPACE + value: {{ .Values.jupyterPool.namespace }} + - name: KUBERNETES_JUPYTER_SERVICE_NAME + value: {{ .Values.jupyterPool.name }}-svc + - name: KUBERNETES_JUPYTER_IMAGE_NAME + value: {{ .Values.texera.imageRegistry }}/{{ .Values.jupyterPool.imageName }}:{{ .Values.texera.imageTag }} + - name: KUBERNETES_JUPYTER_CPU_LIMIT + value: "{{ .Values.jupyterPool.resources.cpuLimit }}" + - name: KUBERNETES_JUPYTER_MEMORY_LIMIT + value: {{ .Values.jupyterPool.resources.memoryLimit }} + # The pod's own prefix and the browser-facing address are rendered from one + # basePath, so they cannot drift apart. + - name: KUBERNETES_JUPYTER_BASE_URL + value: {{ include "texera.jupyter.basePath" . }} + {{- $origin := .Values.notebookMigrationService.publicOrigin }} + {{- if and (not $origin) .Values.gatewayConfig .Values.gatewayConfig.hostname }} + {{- /* Both listeners always render, so only a configured certificate says + which one actually serves. */}} + {{- if or .Values.gatewayConfig.issuer .Values.gatewayConfig.tlsSecretName }} + {{- $origin = printf "https://%s"; .Values.gatewayConfig.hostname }} + {{- else }} + {{- $origin = printf "http://%s"; .Values.gatewayConfig.hostname }} + {{- end }} + {{- end }} + {{- if $origin }} + - name: KUBERNETES_JUPYTER_PUBLIC_URL_TEMPLATE + value: {{ $origin }}{{ include "texera.jupyter.basePath" . }}/{uid} + - name: KUBERNETES_JUPYTER_TEXERA_ORIGIN + value: {{ $origin }} + {{- end }} + - name: JUPYTER_TOKEN_SECRET + valueFrom: + secretKeyRef: + name: {{ .Release.Name }}-notebook-migration-service-secret + key: jupyter-token-secret + {{- range .Values.texeraEnvVars }} + - name: {{ .name }} + value: "{{ .value }}" + {{- end }} + livenessProbe: + httpGet: + path: /api/healthcheck + port: {{ .Values.notebookMigrationService.service.port }} + initialDelaySeconds: 30 + periodSeconds: 10 + readinessProbe: + httpGet: + path: /api/healthcheck + port: {{ .Values.notebookMigrationService.service.port }} + initialDelaySeconds: 5 + periodSeconds: 5 +{{- end }} diff --git a/bin/k8s/templates/base/notebook-migration-service/notebook-migration-service-secret.yaml b/bin/k8s/templates/base/notebook-migration-service/notebook-migration-service-secret.yaml new file mode 100644 index 0000000000..d9c7f8746a --- /dev/null +++ b/bin/k8s/templates/base/notebook-migration-service/notebook-migration-service-secret.yaml @@ -0,0 +1,30 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +{{- if .Values.notebookMigrationService.enabled }} +# Key the per-user JupyterLab tokens are derived from. Kept in a Secret rather than the +# deployment's env list because it is a credential: anyone holding it can derive any user's +# token. +apiVersion: v1 +kind: Secret +metadata: + name: {{ .Release.Name }}-notebook-migration-service-secret + namespace: {{ .Release.Namespace }} +type: Opaque +{{- $help := "notebookMigrationService.jupyterTokenSecret is required when the notebook migration tool is enabled: it derives every user's JupyterLab token, so generate one per deployment, e.g. openssl rand -hex 32" }} +stringData: + jupyter-token-secret: "{{ required $help .Values.notebookMigrationService.jupyterTokenSecret }}" +{{- end }} diff --git a/bin/k8s/templates/base/notebook-migration-service/notebook-migration-service-service-account.yaml b/bin/k8s/templates/base/notebook-migration-service/notebook-migration-service-service-account.yaml new file mode 100644 index 0000000000..1b5d37ec3a --- /dev/null +++ b/bin/k8s/templates/base/notebook-migration-service/notebook-migration-service-service-account.yaml @@ -0,0 +1,49 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +{{- if .Values.notebookMigrationService.enabled }} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ .Values.notebookMigrationService.serviceAccountName }} + namespace: {{ .Release.Namespace }} +--- +# Scoped to the JupyterLab pool namespace only: the service starts and stops a user's own +# notebook server and needs nothing in the release namespace. +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: {{ .Values.notebookMigrationService.name }} + namespace: {{ .Values.jupyterPool.namespace }} +rules: + - apiGroups: [""] + resources: ["pods"] + verbs: ["get", "list", "watch", "create", "delete"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: {{ .Values.notebookMigrationService.name }}-binding + namespace: {{ .Values.jupyterPool.namespace }} +subjects: + - kind: ServiceAccount + name: {{ .Values.notebookMigrationService.serviceAccountName }} + namespace: {{ .Release.Namespace }} +roleRef: + kind: Role + name: {{ .Values.notebookMigrationService.name }} + apiGroup: rbac.authorization.k8s.io +{{- end }} diff --git a/bin/k8s/templates/base/notebook-migration-service/notebook-migration-service-service.yaml b/bin/k8s/templates/base/notebook-migration-service/notebook-migration-service-service.yaml new file mode 100644 index 0000000000..d13a2d4799 --- /dev/null +++ b/bin/k8s/templates/base/notebook-migration-service/notebook-migration-service-service.yaml @@ -0,0 +1,31 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +{{- if .Values.notebookMigrationService.enabled }} +apiVersion: v1 +kind: Service +metadata: + name: {{ .Values.notebookMigrationService.name }}-svc + namespace: {{ .Release.Namespace }} +spec: + type: {{ .Values.notebookMigrationService.service.type }} + selector: + app: {{ .Release.Name }}-{{ .Values.notebookMigrationService.name }} + ports: + - protocol: TCP + port: {{ .Values.notebookMigrationService.service.port }} + targetPort: {{ .Values.notebookMigrationService.service.port }} +{{- end }} diff --git a/bin/k8s/values-development.yaml b/bin/k8s/values-development.yaml index 01fdc2c957..62a3117650 100644 --- a/bin/k8s/values-development.yaml +++ b/bin/k8s/values-development.yaml @@ -295,6 +295,30 @@ workflowComputingUnitPool: port: 8085 targetPort: 8085 +# Notebook migration tool. The base chart ships it off; development turns it on. +notebookMigrationService: + enabled: true + # Reached through a port-forward, so there is no DNS name to derive an origin from. + publicOrigin: "http://localhost:30080"; + # Development only, and public by virtue of living here. A real deployment must generate + # its own, since this one derives every user's JupyterLab token. + jupyterTokenSecret: "development-only-not-a-secret" + +# Per-user JupyterLab pods, sized for a single-node development cluster. Only the values that +# differ from the base chart are listed, since Helm merges the rest. +jupyterPool: + # The base default of a full CPU and 2Gi per user exhausts a laptop cluster after two of + # them. Limits become requests here, because the pods declare no requests of their own. + resources: + cpuLimit: "0.5" + memoryLimit: 1Gi + # Low enough to actually bind, so a provisioning loop cannot fill the node. + maxRequestedResources: + cpu: 4 + memory: 8Gi + # One node, so prepulling saves nothing and costs a large image pull at install time. + prepullImage: false + texeraEnvVars: - name: USER_SYS_ADMIN_USERNAME value: "texera" diff --git a/bin/k8s/values.yaml b/bin/k8s/values.yaml index 94f30a5b32..46bb349fca 100644 --- a/bin/k8s/values.yaml +++ b/bin/k8s/values.yaml @@ -195,6 +195,61 @@ webserver: type: ClusterIP port: 8080 +notebookMigrationService: + # Turns the whole notebook migration tool on or off: the service, its route, the per-user + # JupyterLab pool, and the button in the workspace. + enabled: false + name: notebook-migration-service + numOfPods: 1 + serviceAccountName: notebook-migration-service-service-account + imageName: texera-notebook-migration-service + service: + type: ClusterIP + port: 9098 + # Origin the browser reaches Texera on, used for the JupyterLab iframe URL and for the + # CSP that lets Texera embed it. Required wherever there is no DNS name, such as a + # port-forward or a NodePort. Falls back to the gateway hostname when left empty. + publicOrigin: "" + # HMAC key each user's JupyterLab token is derived from. Must stay stable across restarts + # or issued tokens stop matching. Empty on purpose: a default here would be public, and + # the token is all that guards /jupyter/<uid>/. Generate with: openssl rand -hex 32 + jupyterTokenSecret: "" + +# Per-user JupyterLab pods, the stateful half of the notebook migration tool. One pod per +# user, addressed through the headless service below. +jupyterPool: + createNamespaces: true + name: texera-jupyter + # Note: like the computing unit pool, this namespace can collide when several Texera + # deployments share a cluster. + namespace: texera-jupyter-pool + imageName: texera-jupyter + # Pull the image onto every node ahead of time, so a first-time pull cannot outlast the + # bounded wait for a new pod to answer. Costs one small pod per node. + prepullImage: true + # Path prefix each user's JupyterLab is served under; the uid is appended, so user 7 is + # served at /jupyter/7/. Drives the pod's base_url, the gateway route and the uid regex + # access-control matches on, so all three move together. + basePath: /jupyter + # Must match kubernetes.jupyter-port-num, which the service reads from its own config + # and no environment variable overrides. + service: + port: 8888 + targetPort: 8888 + # Per-pod limits. + resources: + cpuLimit: "1" + memoryLimit: 2Gi + networkPolicy: + # Denies pod-to-pod traffic inside the pool, so one user's JupyterLab cannot reach + # another's. Requires a cluster with a NetworkPolicy controller; without one the object + # is created but not enforced. + enabled: true + # Ceiling for the pool as a whole. + maxRequestedResources: + cpu: 50 + memory: 50Gi + workflowComputingUnitManager: name: workflow-computing-unit-manager numOfPods: 1 @@ -450,6 +505,13 @@ metrics-server: gatewayConfig: # Routes are available at bin/k8s/templates/gateway-routes.yaml + # Ceiling for LLM requests through /api/chat and /api/models. Completions routinely run + # for tens of seconds, and Envoy's default route timeout is 15s, which severs them while + # the upstream call is still in flight and then succeeds unseen. Keep this at or above + # GUI_WORKFLOW_WORKSPACE_PYTHON_NOTEBOOK_MIGRATION_TIMEOUT_MINUTES, which is what the + # frontend is prepared to wait. + llmRequestTimeout: 10m + # The hostname for the Gateway listener (HTTP/HTTPS). # e.g., "texera.example.com" hostname: "" diff --git a/common/config/src/main/resources/kubernetes.conf b/common/config/src/main/resources/kubernetes.conf index f423784334..da72b930d3 100644 --- a/common/config/src/main/resources/kubernetes.conf +++ b/common/config/src/main/resources/kubernetes.conf @@ -82,6 +82,12 @@ kubernetes { jupyter-texera-origin = "" jupyter-texera-origin = ${?KUBERNETES_JUPYTER_TEXERA_ORIGIN} + # Prefix each user's Jupyter serves under; the uid is appended, so user 7 is served at + # <prefix>/7/. The gateway reads that uid to pick the pod, so this has to match the path + # component of jupyter-public-url-template. + jupyter-base-url = "/jupyter" + jupyter-base-url = ${?KUBERNETES_JUPYTER_BASE_URL} + jupyter-cpu-limit = "1" jupyter-cpu-limit = ${?KUBERNETES_JUPYTER_CPU_LIMIT} diff --git a/common/config/src/main/scala/org/apache/texera/common/config/KubernetesConfig.scala b/common/config/src/main/scala/org/apache/texera/common/config/KubernetesConfig.scala index f52a14fa4e..7cb177c6b6 100644 --- a/common/config/src/main/scala/org/apache/texera/common/config/KubernetesConfig.scala +++ b/common/config/src/main/scala/org/apache/texera/common/config/KubernetesConfig.scala @@ -72,9 +72,10 @@ object KubernetesConfig { val jupyterServiceName: String = conf.getString("kubernetes.jupyter-service-name") val jupyterImageName: String = conf.getString("kubernetes.jupyter-image-name") val jupyterPortNumber: Int = conf.getInt("kubernetes.jupyter-port-num") + val jupyterBaseUrl: String = conf.getString("kubernetes.jupyter-base-url") + val jupyterTexeraOrigin: String = conf.getString("kubernetes.jupyter-texera-origin") val jupyterCpuLimit: String = conf.getString("kubernetes.jupyter-cpu-limit") val jupyterMemoryLimit: String = conf.getString("kubernetes.jupyter-memory-limit") - val jupyterTexeraOrigin: String = conf.getString("kubernetes.jupyter-texera-origin") // Browser-facing address with {uid} substituted; empty means use the in-network one. val jupyterPublicUrlTemplate: String = diff --git a/common/config/src/test/scala/org/apache/texera/common/config/KubernetesConfigSpec.scala b/common/config/src/test/scala/org/apache/texera/common/config/KubernetesConfigSpec.scala index b52568594c..3b3c194b63 100644 --- a/common/config/src/test/scala/org/apache/texera/common/config/KubernetesConfigSpec.scala +++ b/common/config/src/test/scala/org/apache/texera/common/config/KubernetesConfigSpec.scala @@ -86,6 +86,8 @@ class KubernetesConfigSpec extends AnyFlatSpec with Matchers { ) // Empty by default: only a real deployment knows its own origin. ifUnset("KUBERNETES_JUPYTER_TEXERA_ORIGIN")(KubernetesConfig.jupyterTexeraOrigin shouldBe "") + // A prefix, not a full path: the provisioner appends the uid. + ifUnset("KUBERNETES_JUPYTER_BASE_URL")(KubernetesConfig.jupyterBaseUrl shouldBe "/jupyter") ifUnset("KUBERNETES_JUPYTER_CPU_LIMIT")(KubernetesConfig.jupyterCpuLimit shouldBe "1") ifUnset("KUBERNETES_JUPYTER_MEMORY_LIMIT")( KubernetesConfig.jupyterMemoryLimit shouldBe "2Gi" diff --git a/notebook-migration-service/src/main/resources/start-texera-jupyter.sh b/notebook-migration-service/src/main/resources/start-texera-jupyter.sh index 2bfb5a3baf..b38be02604 100644 --- a/notebook-migration-service/src/main/resources/start-texera-jupyter.sh +++ b/notebook-migration-service/src/main/resources/start-texera-jupyter.sh @@ -19,14 +19,22 @@ set -euo pipefail # Texera app origin used by custom.js (postMessage targetOrigin + inbound origin -# check) and by the iframe CSP frame-ancestors. Override TEXERA_ORIGIN for -# deployments under a real hostname; defaults to the local dev origin. +# check), by the iframe CSP frame-ancestors, and by Jupyter's own cross-origin check. +# That last one matters wherever a proxy rewrites the Host header: Jupyter compares +# Origin against Host and rejects cookie-authenticated API calls when they differ, which +# leaves the notebook without a kernel. Override TEXERA_ORIGIN for deployments under a +# real hostname; defaults to the local dev origin. TEXERA_ORIGIN="${TEXERA_ORIGIN:-http://localhost:4200}"; # Weak default token so the server is not fully open to anyone reachable on the # published port. The Texera-side iframe URL must pass this through ?token=<value>. JUPYTER_TOKEN="${JUPYTER_TOKEN:-texera}" +# Path Jupyter serves under. A deployment that puts every user's Jupyter on one hostname +# routes by path, so the server has to know its own prefix. Defaults to "/", which is what +# single-node and local dev use. +JUPYTER_BASE_URL="${JUPYTER_BASE_URL:-/}" + # Substitute the origin placeholder in custom.js before the server starts serving it. sed -i "s|__TEXERA_ORIGIN__|${TEXERA_ORIGIN}|g" /home/jovyan/.jupyter/custom/custom.js @@ -34,5 +42,7 @@ exec start-notebook.sh \ --NotebookApp.token="${JUPYTER_TOKEN}" \ --NotebookApp.password='' \ --NotebookApp.disable_check_xsrf=True \ + --NotebookApp.allow_origin="${TEXERA_ORIGIN}" \ --NotebookApp.tornado_settings="{'headers': {'Content-Security-Policy': 'frame-ancestors ${TEXERA_ORIGIN}'}}" \ + --NotebookApp.base_url="${JUPYTER_BASE_URL}" \ --NotebookApp.default_url=/tree diff --git a/notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterKubernetesClient.scala b/notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterKubernetesClient.scala index 426912516c..1179de65f5 100644 --- a/notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterKubernetesClient.scala +++ b/notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterKubernetesClient.scala @@ -43,6 +43,14 @@ class JupyterKubernetesClient(client: io.fabric8.kubernetes.client.KubernetesCli def generatePodURI(uid: Int): String = s"${generatePodName(uid)}.${KubernetesConfig.jupyterServiceName}.$namespace.svc.cluster.local:${KubernetesConfig.jupyterPortNumber}" + /** + * Path a user's Jupyter serves under. The uid is in the path because the browser cannot + * present Texera credentials on the requests Jupyter's own scripts make, so the gateway + * has to read the owner out of the URL instead. + */ + def basePathFor(uid: Int): String = + s"/${KubernetesConfig.jupyterBaseUrl.stripPrefix("/").stripSuffix("/")}/$uid" + def podExists(uid: Int): Boolean = getPodByName(generatePodName(uid)).isDefined def getPodByName(podName: String): Option[Pod] = @@ -84,6 +92,10 @@ class JupyterKubernetesClient(client: io.fabric8.kubernetes.client.KubernetesCli new EnvVarBuilder() .withName("TEXERA_ORIGIN") .withValue(KubernetesConfig.jupyterTexeraOrigin) + .build(), + new EnvVarBuilder() + .withName("JUPYTER_BASE_URL") + .withValue(s"${basePathFor(uid)}/") .build() ) .withResources(resources) diff --git a/notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterProvisioner.scala b/notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterProvisioner.scala index 2ba3ee0a18..f0b607e5b5 100644 --- a/notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterProvisioner.scala +++ b/notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterProvisioner.scala @@ -104,7 +104,9 @@ class JupyterProvisioner( } private def provision(uid: Int, token: String): Option[JupyterEndpoints] = { - val internalUrl = s"http://${kubernetes.generatePodURI(uid)}" + // Jupyter serves every endpoint under its base path, /api included, so the recorded + // address has to carry it or each later call lands on a 404. + val internalUrl = s"http://${kubernetes.generatePodURI(uid)}${kubernetes.basePathFor(uid)}" val endpoints = JupyterEndpoints(internalUrl, publicUrlFor(uid, internalUrl), token) try { createIfAbsent(uid, token) diff --git a/notebook-migration-service/src/test/scala/org/apache/texera/service/resource/NotebookMigrationResourceSpec.scala b/notebook-migration-service/src/test/scala/org/apache/texera/service/resource/NotebookMigrationResourceSpec.scala index 66cbb4406a..e1c1f7ab8e 100644 --- a/notebook-migration-service/src/test/scala/org/apache/texera/service/resource/NotebookMigrationResourceSpec.scala +++ b/notebook-migration-service/src/test/scala/org/apache/texera/service/resource/NotebookMigrationResourceSpec.scala @@ -722,7 +722,9 @@ class NotebookMigrationResourceSpec kubernetes.created.map(_._1) shouldBe List(writerUid.intValue()) registeredUids() shouldBe List(writerUid) - result.map(_.internalUrl) shouldBe Some(s"http://${kubernetes.generatePodURI(writerUid)}") + result.map(_.internalUrl) shouldBe Some( + s"http://${kubernetes.generatePodURI(writerUid)}${kubernetes.basePathFor(writerUid)}" + ) } it should "give the pod the user's own derived token" in { @@ -752,7 +754,9 @@ class NotebookMigrationResourceSpec kubernetes.deleted shouldBe List(writerUid.intValue()) kubernetes.created.map(_._1) shouldBe List(writerUid.intValue()) - result.map(_.internalUrl) shouldBe Some(s"http://${kubernetes.generatePodURI(writerUid)}") + result.map(_.internalUrl) shouldBe Some( + s"http://${kubernetes.generatePodURI(writerUid)}${kubernetes.basePathFor(writerUid)}" + ) } it should "register nothing and clean up when the pod never becomes ready" in { @@ -765,6 +769,18 @@ class NotebookMigrationResourceSpec registeredUids() shouldBe empty } + it should "record the user's own base path in the internal address" in { + // Jupyter serves /api under its base path too, so an address without the prefix would + // make every later probe and contents call 404. + val kubernetes = new StubKubernetes + val result = provisionerFor(kubernetes, (_, _) => true) + .ensure(writerUid, jupyterEnabled = true, tokenSecret = specSecret) + + result.map(_.internalUrl) shouldBe Some( + s"http://${kubernetes.generatePodURI(writerUid)}${kubernetes.basePathFor(writerUid)}" + ) + } + it should "build the public URL from the configured template" in { val kubernetes = new StubKubernetes val result = @@ -856,7 +872,7 @@ class NotebookMigrationResourceSpec kubernetes.deleted shouldBe List(writerUid.intValue()) kubernetes.created.map(_._1) shouldBe List(writerUid.intValue()) result.map(_.internalUrl) shouldBe Some( - s"http://${kubernetes.generatePodURI(writerUid)}" + s"http://${kubernetes.generatePodURI(writerUid)}${kubernetes.basePathFor(writerUid)}" ) } @@ -871,7 +887,7 @@ class NotebookMigrationResourceSpec kubernetes.created.map(_._1) shouldBe List(writerUid.intValue()) result.map(_.internalUrl) shouldBe Some( - s"http://${kubernetes.generatePodURI(writerUid)}" + s"http://${kubernetes.generatePodURI(writerUid)}${kubernetes.basePathFor(writerUid)}" ) } @@ -908,7 +924,7 @@ class NotebookMigrationResourceSpec .ensure(writerUid, jupyterEnabled = true, tokenSecret = specSecret) result.map(_.internalUrl) shouldBe Some( - s"http://${kubernetes.generatePodURI(writerUid)}" + s"http://${kubernetes.generatePodURI(writerUid)}${kubernetes.basePathFor(writerUid)}" ) kubernetes.deleted shouldBe empty registeredUids() shouldBe List(writerUid) @@ -944,7 +960,9 @@ class NotebookMigrationResourceSpec .ensure(writerUid, jupyterEnabled = true, tokenSecret = specSecret) kubernetes.created.map(_._1) shouldBe List(writerUid.intValue()) - result.map(_.internalUrl) shouldBe Some(s"http://${kubernetes.generatePodURI(writerUid)}") + result.map(_.internalUrl) shouldBe Some( + s"http://${kubernetes.generatePodURI(writerUid)}${kubernetes.basePathFor(writerUid)}" + ) } it should "report unavailable when registration fails for a reason other than a race" in { diff --git a/notebook-migration-service/src/test/scala/org/apache/texera/service/util/JupyterKubernetesClientSpec.scala b/notebook-migration-service/src/test/scala/org/apache/texera/service/util/JupyterKubernetesClientSpec.scala index 624ff3ebc3..e86d8c5668 100644 --- a/notebook-migration-service/src/test/scala/org/apache/texera/service/util/JupyterKubernetesClientSpec.scala +++ b/notebook-migration-service/src/test/scala/org/apache/texera/service/util/JupyterKubernetesClientSpec.scala @@ -167,6 +167,23 @@ class JupyterKubernetesClientSpec extends AnyFlatSpec with Matchers { Some(KubernetesConfig.jupyterTexeraOrigin) } + it should "tell the pod which base path it serves under" in { + // The image passes this to --NotebookApp.base_url. Jupyter wants a trailing slash, and + // without the prefix a path-routed deployment serves every endpoint from the wrong place. + val env = createdPod(7, "tok").getSpec.getContainers.asScala.head.getEnv.asScala + env.find(_.getName == "JUPYTER_BASE_URL").map(_.getValue) shouldBe Some("/jupyter/7/") + } + + "basePathFor" should "put the uid in the path so the gateway can read it" in { + // The browser cannot present Texera credentials on the requests Jupyter's own scripts + // make, so the owner has to be recoverable from the URL alone. + bare.basePathFor(7) shouldBe "/jupyter/7" + } + + it should "give every user a distinct base path" in { + (1 to 50).map(bare.basePathFor).distinct.size shouldBe 50 + } + it should "carry the configured image, pull policy and port" in { val container = createdPod(7, "tok").getSpec.getContainers.asScala.head container.getImage shouldBe KubernetesConfig.jupyterImageName
