This is an automated email from the ASF dual-hosted git repository.
jerryshao pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new eaabffdc68 [#13618] docs(k8s-job-executor): Document the k8s job
executor and split out the local job executor (#13950)
eaabffdc68 is described below
commit eaabffdc6854f96084058a2a719b8a151fd0edb1
Author: Jerry Shao <[email protected]>
AuthorDate: Thu Oct 29 16:16:29 2026 +0800
[#13618] docs(k8s-job-executor): Document the k8s job executor and split
out the local job executor (#13950)
### What changes were proposed in this pull request?
Documents the `k8s` job executor, and gives each built-in job executor a
page of its own with the same layout:
- `docs/k8s-job-executor.md` (new): requirements, how a job template
maps to a SparkApplication, built-in jobs, rejected runs, the job
status, RBAC, the Spark image, the cluster connection, all the
`gravitino.jobExecutor.k8s.*` configurations, queueing with YuniKorn,
environment variables and Secrets, job output and times, multiple
servers, limitations, and a walkthrough that runs a built-in Iceberg job
on a local cluster.
- `docs/local-job-executor.md` (new): the local job executor content
moved out of `docs/manage-jobs-in-gravitino.md`, unchanged in meaning.
- `docs/manage-jobs-in-gravitino.md`: keeps the job API, adds a "Job
Executors" section linking to both pages, and drops a stale note and two
Future Work items that are done.
- `docs/development/custom-job-executor.md`,
`docs/gravitino-server-config.md`, `docs/index.md`, and the optimizer
docs: point to both job executors.
- `K8sJobExecutor`: before treating a 400/403/404/422 response as a
rejection, check whether an earlier attempt created the SparkApplication
of the job, as the client retries requests that timed out or failed with
a server error. Before, the job was removed while its SparkApplication
could run unrecorded, against the contract of #14278. The job now runs
if it exists, and the outcome is unknown if it's being deleted or can't
be read.
### Why are the changes needed?
The `k8s` job executor has no documentation, and the local job
executor's details were mixed into the job API page.
Part of #13618. The `gravitino-spark` Dockerfile and the K3s integration
test of #13618 follow in another PR; until then, the docs show a minimal
Dockerfile to build the image.
### Does this PR introduce _any_ user-facing change?
Documentation, plus one fix: a run whose retried request is rejected
after an earlier attempt created the SparkApplication now runs, rather
than being removed while the SparkApplication runs.
### How was this patch tested?
- The docs audit scripts and the link check
(`.claude/skills/gravitino-docs-refine/scripts/`) report no issues in
the changed pages.
- On a local Kubernetes cluster (OrbStack, Kubernetes 1.35,
spark-kubernetes-operator 0.9.0), with a Gravitino server from the
distribution package of this branch:
- The walkthrough was run from the code blocks of the page: building the
image from the documented Dockerfile, creating the table, running
`builtin-iceberg-rewrite-data-files`, and reading its output.
- The documented Role was checked by removing each verb in turn. Also
verified: the token connection; the startup checks for a missing
`namespace` and for an `http://` `masterUrl` with a token file; the
rejected runs; environment variables from a Secret in the driver and
executor pods; driver eviction; that a cancelled job loses its output;
and YuniKorn 1.10.0 with `maxapplications` and `resources.max`.
- The `spark_conf` behavior of the built-in Iceberg jobs was verified on
the branch stacked on #13703, and the walkthrough was run again on main
with #13982 and #14230.
- `:plugins:k8s-job-executor:build` passes (188 tests), including 4 new
tests in `TestK8sJobExecutor` for a rejected retry. Without the fix, 3
of them fail; the fourth checks that a rejection is kept when the
SparkApplication belongs to another job.
- The docs on #14278's job recovery were checked against the code only,
not run on a cluster.
- Behaviors stated from the code only, not run on a cluster: the
in-cluster connection, the request retries and timeouts, and the mapping
of unknown operator states.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
docs/development/custom-job-executor.md | 7 +-
docs/gravitino-server-config.md | 4 +-
docs/index.md | 2 +
docs/k8s-job-executor.md | 837 +++++++++++++++++++++
docs/local-job-executor.md | 164 ++++
docs/manage-jobs-in-gravitino.md | 200 ++---
.../optimizer-configuration.md | 7 +
docs/table-maintenance-service/optimizer.md | 8 +-
.../apache/gravitino/job/k8s/K8sJobExecutor.java | 55 ++
.../gravitino/job/k8s/TestK8sJobExecutor.java | 118 +++
10 files changed, 1251 insertions(+), 151 deletions(-)
diff --git a/docs/development/custom-job-executor.md
b/docs/development/custom-job-executor.md
index 57844accb7..c121d9acc2 100644
--- a/docs/development/custom-job-executor.md
+++ b/docs/development/custom-job-executor.md
@@ -7,9 +7,10 @@ license: "This software is licensed under the Apache License
version 2."
## Introduction
-The `local` job executor that ships with Gravitino runs jobs as a process on
the Gravitino server
-and is intended for testing. Running jobs anywhere else, such as against a
distributed scheduler,
-means implementing your own job executor.
+Gravitino ships two job executors: the [`local` job
executor](../local-job-executor.md) runs jobs
+as processes on the Gravitino server and is intended for testing, and the
+[`k8s` job executor](../k8s-job-executor.md) runs Spark jobs on Kubernetes.
Running jobs anywhere
+else, such as against another distributed scheduler, means implementing your
own job executor.
## Implement a Custom Job Executor
diff --git a/docs/gravitino-server-config.md b/docs/gravitino-server-config.md
index 54224446dd..3c9d7647de 100644
--- a/docs/gravitino-server-config.md
+++ b/docs/gravitino-server-config.md
@@ -154,7 +154,9 @@ health check at `GET /health/ready` so a server that has
lost its database stops
Jobs run by the default `local` job executor keep their output in
`gravitino.job.stagingDir`. Put
that directory on storage shared by all servers, for example an NFS mount, so
that a request for a
job's output can be served by any server. Otherwise only the server that ran
the job can return
-it, and the others return empty output. See [Manage
Jobs](manage-jobs-in-gravitino.md).
+it, and the others return empty output. See [Local Job
Executor](local-job-executor.md). The
+[Kubernetes job executor](k8s-job-executor.md) keeps no job state on the
servers, so any server can
+return a job's output.
## Server Configuration
diff --git a/docs/index.md b/docs/index.md
index 3514a1e53b..4861407be7 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -169,6 +169,8 @@ Gravitino provides governance features to manage metadata
in a unified way. See:
to manage policies.
* [Manage jobs in Gravitino](./manage-jobs-in-gravitino.md): a complete guide
to using Gravitino
to manage jobs.
+* [Local job executor](./local-job-executor.md) and
+ [Kubernetes job executor](./k8s-job-executor.md): where the jobs run and how
to configure them.
### Gravitino Iceberg REST Catalog Service
diff --git a/docs/k8s-job-executor.md b/docs/k8s-job-executor.md
new file mode 100644
index 0000000000..d8c6c8e332
--- /dev/null
+++ b/docs/k8s-job-executor.md
@@ -0,0 +1,837 @@
+---
+title: "Kubernetes Job Executor"
+slug: "/k8s-job-executor"
+keyword: "job executor, Kubernetes, k8s, Spark operator, SparkApplication,
YuniKorn, Gravitino"
+license: "This software is licensed under the Apache License version 2."
+---
+
+## Introduction
+
+The `k8s` job executor runs Spark jobs on Kubernetes. Each job becomes a
`SparkApplication` resource
+of the [Apache Spark Kubernetes
Operator](https://github.com/apache/spark-kubernetes-operator), which
+starts the driver and executor pods of the job.
+
+All the job state lives in Kubernetes, not in the Gravitino server. Jobs keep
running while Gravitino
+restarts, and every Gravitino server of a deployment can query and cancel any
job.
+
+The `k8s` job executor only runs Spark templates. A run of a shell template is
rejected.
+
+For job templates and the job API, see [Manage
Jobs](./manage-jobs-in-gravitino.md). To run jobs as
+processes on the Gravitino server instead, see [Local Job
Executor](./local-job-executor.md).
+
+## Requirements
+
+- Apache Spark Kubernetes Operator 0.9.x in the Kubernetes cluster that runs
the jobs, watching
+ every namespace the jobs run in. Operator 1.0 and later require Spark 4.0 or
later, while the
+ built-in jobs of Gravitino are built for Spark 3.5.
+- A Spark 3.5 image, see [Spark Image](#spark-image).
+- Access from the Gravitino server to the API server of the cluster, with the
permissions listed in
+ [Kubernetes Permissions](#kubernetes-permissions).
+
+The versions that the job executor supports, and the ones it was tested with:
+
+| Component | Supported
| Tested |
+|----------------------------------|---------------------------------------------------------|--------------------------------------------|
+| Apache Spark Kubernetes Operator | 0.9.x
| 0.9.0, installed with the Helm chart 1.7.0 |
+| Spark | 3.5
| 3.5.9 (`apache/spark:3.5.9-java17`) |
+| Iceberg Spark runtime | 1.11, which the built-in Iceberg jobs are
built against | 1.11.0 |
+| Kubernetes | The versions that operator 0.9.x runs on
| 1.35.6 (OrbStack) |
+| Apache YuniKorn | Optional, see [YuniKorn](#yunikorn)
| 1.10.0 |
+
+Only the versions in the Tested column were run with the job executor.
+
+The job executor is the `gravitino-k8s-job-executor-<version>.jar` file in the
`libs` directory of
+the Gravitino distribution. Nothing else needs to be installed on the
Gravitino server.
+
+## Job Execution
+
+When a job runs, the job executor creates a `SparkApplication` from the job
template and the job
+configuration:
+
+| Job template | SparkApplication
|
+|-----------------------------|----------------------------------------------------------------------------------------|
+| `executable` | `spec.jars`, or `spec.pyFiles` for a file that
ends with `.py` |
+| `className` | `spec.mainClass`
|
+| `arguments` | `spec.driverArgs`
|
+| `configs` | `spec.sparkConf`
|
+| `environments` | `spark.kubernetes.driverEnv.<name>` and
`spark.executorEnv.<name>` in `spec.sparkConf` |
+| `jars`, `files`, `archives` | Appended to `spark.jars`, `spark.files`, and
`spark.archives` in `spec.sparkConf` |
+
+The Spark configurations of a job come from these places, each overriding the
ones before it:
+
+1. The server-level defaults, set with the
`gravitino.jobExecutor.k8s.spark.conf.` prefix.
+2. The `environments` of the job template.
+3. The `configs` of the job template.
+4. For a built-in Iceberg job, its `spark_conf` parameter, see [Built-In
Jobs](#built-in-jobs).
+
+A blank value in the `configs` of a job template or in `spark_conf` overrides
nothing: the
+configuration keeps the value that an earlier place sets, for example a
server-level default. An
+optional placeholder without a value becomes such a blank value. Any other
value replaces the
+earlier one; a list such as `spark.jars` isn't merged with it.
+
+The job executor then adjusts the result:
+
+- The operator decides the master and the deploy mode, so `spark.master` and
+ `spark.submit.deployMode` are removed. A job whose `spark.master` isn't a
Kubernetes master, that
+ is one starting with `k8s://`, such as `local[*]`, is rejected, and so is
one whose
+ `spark.submit.deployMode` is another mode than `cluster`, such as `client`.
+- `spark.kubernetes.container.image` and
`spark.kubernetes.authenticate.driver.serviceAccountName`
+ take the values of `spark.image` and `spark.serviceAccount` of the job
executor if neither the
+ job template nor a server-level default sets them. A job template can
therefore run with its
+ own image and service account.
+- `spark.kubernetes.namespace` is the namespace that the `configs` of the job
template set, or else
+ `namespace` of the job executor. A server-level default of it with the
+ `spark.conf.` prefix is ignored: set `namespace` instead.
+
+The `SparkApplication` is named `<namePrefix>-<job ID>`, for example
+`gravitino-job-688822913391439376`, and carries the labels
`app.kubernetes.io/managed-by=gravitino`
+and `gravitino.apache.org/job-id=<job ID>`. List the jobs of a namespace with:
+
+```shell
+kubectl get sparkapplications -l app.kubernetes.io/managed-by=gravitino
+```
+
+### Job Resources
+
+Gravitino doesn't fetch the resources of a job. It passes their URIs on to
Spark, which fetches them
+in the cluster. The `executable`, `jars`, `files`, and `archives` of a
template must therefore be
+URIs that the driver and executor pods can reach, such as `local://` for a
file in the Spark image,
+`https://`, `s3a://`, or `hdfs://`. So must every entry of `spark.jars`,
`spark.files`, and
+`spark.archives`, whether a job template, `spark_conf`, or a server-level
default sets them. A run
+with a path on the Gravitino server, that is a path without a scheme or with
the `file` scheme, is
+rejected.
+
+The built-in job templates are the exception. Their `executable` is the path
of the built-in jobs
+jar on the Gravitino server, and the job executor replaces it with
`spark.builtinJobsJar`, by default
+`local:///opt/gravitino/jobs/gravitino-jobs.jar`, a file in the [Spark
image](#spark-image).
+
+### Built-In Jobs
+
+The built-in job templates set the resources of the driver and the executors
in their own
+`configs`, from placeholders. Size a built-in job with its job configuration,
for example
+`spark_executor_instances`, `spark_executor_cores`, `spark_executor_memory`,
and
+`spark_driver_memory`, not with server-level defaults: the `configs` of a
template override them.
+Leave `spark_master` empty, its default: any other value than a Kubernetes
master is rejected.
+
+The `spark_conf` parameter of the built-in Iceberg jobs, a flat JSON map of
Spark configurations, is
+set in the `SparkApplication` when the job is submitted, over all the other
Spark configurations of
+the job. So configurations that Spark reads when it starts the driver take
effect, such as
+`spark.jars`, `spark.jars.packages`, and `spark.driver.*`. For example, a job
configuration can add
+the Iceberg Spark runtime to a Spark image without it:
+
+```json
+{
+ "spark_conf":
"{\"spark.jars\":\"https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/1.11.0/iceberg-spark-runtime-3.5_2.12-1.11.0.jar\"}"
+}
+```
+
+- `spark.master` and `spark.submit.deployMode` in `spark_conf` are ignored,
with a warning in the
+ server log.
+- The `spark.kubernetes.*` configurations in `spark_conf`, such as the
namespace, the image, the
+ service account, Secrets, and volumes, are ignored, with a warning in the
server log that leaves
+ out the value. Whoever can run a built-in job sets its `spark_conf`, so only
the job executor
+ configurations and job templates may set them. The driver ignores them as
well.
+- `spark.jars` in `spark_conf` replaces a server-level default of
`spark.jars`, like any other
+ configuration. If that default provides the Iceberg Spark runtime, list the
runtime in
+ `spark_conf` as well.
+- The names of `spark.executorEnv.*` in `spark_conf` are checked like the
`environments` of a job
+ template, see [Environment Variables](#environment-variables).
+- The driver applies the rest of `spark_conf` again when it starts, as it does
with every job
+ executor.
+
+Configurations that every job needs, such as the Iceberg Spark runtime, the
scheduler
+configurations, or volumes, can also be set once as server-level defaults, or
built into the
+[Spark image](#spark-image).
+
+### Rejected Runs
+
+A run is rejected, and its job is removed, when:
+
+- The job template is a shell template.
+- A resource of the job is a path on the Gravitino server, see [Job
Resources](#job-resources).
+- `spark.master` isn't a Kubernetes master, or `spark.submit.deployMode` is
another mode than
+ `cluster`, see [Job Execution](#job-execution).
+- The `spark_conf` parameter of a built-in Iceberg job isn't a flat JSON map.
+- `spark.kubernetes.namespace` isn't a valid namespace name, or an environment
variable name is
+ invalid, see [Environment Variables](#environment-variables).
+- Kubernetes rejects the `SparkApplication`, for example because the namespace
doesn't exist, the
+ job executor lacks a permission, or the operator isn't installed. The error
carries the reason
+ that Kubernetes gives. If Kubernetes rejects a retried request after an
earlier attempt created
+ the `SparkApplication`, the job runs instead.
+- A `SparkApplication` of the same name belongs to another job, for example of
another Gravitino
+ deployment that shares the namespace, see
+ [Multiple Servers and Restarts](#multiple-servers-and-restarts).
+
+### Failed Submissions
+
+When it isn't known whether the `SparkApplication` was created, the run
request fails, but the job
+is kept:
+
+- The API server can't be reached, doesn't answer in time, or answers with
another error than the
+ rejections above, for example a server error or `401 Unauthorized`.
+- The `SparkApplication` of the job exists but is being deleted, or is gone
before it can be read,
+ for example after a retried request.
+- Kubernetes rejects a retried request, and the job executor can't read the
`SparkApplication` to
+ tell whether an earlier attempt created it, for example because of a server
error.
+
+The job executor tries to delete the `SparkApplication` right away, see
+[Cluster Connection](#cluster-connection). Gravitino marks the job as
`CANCELLING`, and about three
+times `gravitino.job.statusPullIntervalInMs` later looks up the
`SparkApplication` by its name and
+namespace. If it's found, it is cancelled. Either way, the job is then marked
as `CANCELLED`. A
+lookup that fails, for example because the API server still can't be reached,
is tried again on
+the next status pull. For the details, see [Run a
Job](./manage-jobs-in-gravitino.md#run-a-job).
+
+The error of the run request names the job. Check the job before running the
job template again, as
+another run request creates another job.
+
+### Job Status
+
+The job status follows the state of the `SparkApplication`:
+
+| Job status | SparkApplication
|
+|-------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `QUEUED` | No state yet, `Submitted`, `DriverRequested`, or
`ScheduledToRestart`. The driver hasn't started, for example because its pod
waits for resources. |
+| `STARTED` | `DriverStarted`, `DriverReady`, `RunningHealthy`,
`RunningWithPartialCapacity`, `InitializedBelowThresholdExecutors`, or
`RunningWithBelowThresholdExecutors` |
+| `SUCCEEDED` | `Succeeded`
|
+| `FAILED` | `Failed`, `SchedulingFailure`, `DriverEvicted`,
`DriverStartTimedOut`, `DriverReadyTimedOut`, or `ExecutorsStartTimedOut`
|
+| `CANCELLED` | Cancelled through Gravitino
|
+
+A job also fails in these cases:
+
+- The operator writes no status for the `SparkApplication` within
`noStatusTimeoutInMs`, for
+ example because the operator isn't running or doesn't watch the namespace.
The job executor
+ deletes the `SparkApplication` before failing the job.
+- The `SparkApplication` is deleted by someone else before Gravitino records
the job as finished,
+ even if the job succeeded.
+
+After its outcome, a `SparkApplication` moves to `ResourceReleased` or
+`TerminatedWithoutReleaseResources`. A job in these states keeps the outcome
that the operator
+recorded before them, or fails if there is none.
+
+A state that the job executor doesn't know, for example of a newer operator,
keeps the job `STARTED`
+if it has started and `QUEUED` otherwise, until the operator records an
outcome. A status that the
+job executor can't read keeps the job `QUEUED` until the operator deletes the
`SparkApplication`.
+Use a supported operator version.
+
+A job is one run of the application: the operator never restarts it.
+
+## Setup
+
+### Kubernetes Permissions
+
+The job executor needs these permissions in every namespace that jobs run in.
Only `list` on
+`sparkapplications` is optional. It needs no permission on Secrets or
ConfigMaps, and no
+cluster-wide permission.
+
+| Resource | Verb | Used to
|
+|------------------------------------------|----------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `sparkapplications` (`spark.apache.org`) | `create` | Submit a job
|
+| `sparkapplications` (`spark.apache.org`) | `get` | Query, submit, and
cancel a job, and read its output
|
+| `sparkapplications` (`spark.apache.org`) | `list` | Query all the jobs of
a namespace with one listing, made 500 at a time. Optional: without it, the job
executor gets the jobs one by one and logs a warning
|
+| `sparkapplications` (`spark.apache.org`) | `update` | Mark a job as
cancelled by Gravitino
|
+| `sparkapplications` (`spark.apache.org`) | `delete` | Cancel a job, and
delete the `SparkApplication` of a job that got no status or whose submission
failed
|
+| `pods` | `list` | Find the driver pod,
for the output and the times of a job, and to check that the driver of a failed
job has terminated. Without it, a failed job stays unfinished until its
`SparkApplication` is deleted after `spark.ttlAfterStopInMs` |
+| `pods` | `delete` | Delete the driver pod
of a failed job that hasn't terminated, for example after the driver start
timeout. Without it, such a job stays unfinished until its `SparkApplication`
is deleted |
+| `pods/log` | `get` | Read the output of a
job
|
+
+For example, in the namespace `default`:
+
+```yaml
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+ name: gravitino-job-executor
+ namespace: default
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+ name: gravitino-job-executor
+ namespace: default
+rules:
+ - apiGroups: ["spark.apache.org"]
+ resources: ["sparkapplications"]
+ verbs: ["create", "get", "list", "update", "delete"]
+ - apiGroups: [""]
+ resources: ["pods"]
+ verbs: ["list", "delete"]
+ - apiGroups: [""]
+ resources: ["pods/log"]
+ verbs: ["get"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+ name: gravitino-job-executor
+ namespace: default
+subjects:
+ - kind: ServiceAccount
+ name: gravitino-job-executor
+ namespace: default
+roleRef:
+ apiGroup: rbac.authorization.k8s.io
+ kind: Role
+ name: gravitino-job-executor
+```
+
+The driver pod of a job runs with another service account,
`spark.serviceAccount`, which Spark uses
+to create the executor pods. The Helm chart of the operator creates the
service account `spark` and
+binds it to the cluster role `spark-workload-clusterrole` in the namespaces it
is installed for, set
+with `workloadResources.namespaces.data` of the chart.
+
+### Job Isolation
+
+The job executor doesn't restrict the Spark configurations of a job. Whoever
can register a job
+template controls the pods of its jobs, within what Kubernetes allows in their
namespace. Through the
+Spark configurations of the template, a job can, for example:
+
+- Run in another namespace that the job executor has permissions in, with
another service account.
+- Mount `hostPath` volumes into its driver and executor pods.
+- Read any Secret of its namespace into an environment variable with
`secretKeyRef`, and print it to
+ its output, which the job API returns.
+
+The same applies to a job template that passes its job configuration on to
these Spark
+configurations, for example through a placeholder. The `spark_conf` parameter
of the built-in
+Iceberg jobs can't set the `spark.kubernetes.*` configurations, see [Built-In
Jobs](#built-in-jobs),
+but anyone who can run one of these jobs can set its other Spark
configurations, such as
+`spark.jars`, which adds code to the job. That code runs in the driver and the
executors, with
+their service accounts and the credentials of the job.
+
+To limit what jobs can do:
+
+- Run the jobs in dedicated namespaces that hold only the Secrets the jobs
need, and grant the Role
+ of the job executor only in those namespaces.
+- Give the driver service account, `spark.serviceAccount`, only the
permissions that Spark needs to
+ manage the executors in the namespace of the job, like the
`spark-workload-clusterrole` of the
+ operator chart, and nothing else.
+- Enforce [Pod Security
Admission](https://kubernetes.io/docs/concepts/security/pod-security-admission/)
+ in those namespaces, for example at the `baseline` level, which rejects pods
with `hostPath`
+ volumes.
+- Grant `REGISTER_JOB_TEMPLATE`, `USE_JOB_TEMPLATE`, and `RUN_JOB` only to
trusted users, see
+ [Access
Control](./security/access-control.md#governance-and-administrative-privileges).
+
+### Spark Image
+
+The image of the driver and the executors, `spark.image`, needs:
+
+- Spark 3.5, whose version is set as `spark.sparkVersion`.
+- The built-in jobs jar of Gravitino at
`/opt/gravitino/jobs/gravitino-jobs.jar`, to run the built-in
+ job templates. It is the `auxlib/gravitino-jobs-<version>.jar` file of the
Gravitino distribution,
+ and must come from the same version as the Gravitino server.
+- The Iceberg Spark runtime for Spark 3.5 and Scala 2.12, to run the built-in
Iceberg jobs. Iceberg
+ 1.11 needs Java 17, so build the image from a Java 17 Spark image.
+
+For example:
+
+```dockerfile
+FROM apache/spark:3.5.9-java17
+USER root
+COPY gravitino-jobs-*.jar /opt/gravitino/jobs/gravitino-jobs.jar
+ADD
--checksum=sha256:94b8e36fc329f0293d44ba9e01b784a56e9501affec1842d898144c51f6e486a
https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/1.11.0/iceberg-spark-runtime-3.5_2.12-1.11.0.jar
/opt/spark/jars/
+RUN chmod 644 /opt/gravitino/jobs/gravitino-jobs.jar
/opt/spark/jars/iceberg-spark-runtime-3.5_2.12-1.11.0.jar
+USER spark
+```
+
+The build fails if the downloaded jar doesn't match `--checksum`, the SHA-256
that Maven Central
+publishes for it. To build without network access, put the jar next to the
Dockerfile and `COPY` it
+instead.
+
+An image without the Iceberg Spark runtime also works if every job gets the
runtime through a
+server-level default, or through the `spark_conf` of a built-in Iceberg job,
see
+[Built-In Jobs](#built-in-jobs). The driver of every job then downloads the
runtime when it starts.
+For example:
+
+```text
+gravitino.jobExecutor.k8s.spark.conf.spark.jars =
https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/1.11.0/iceberg-spark-runtime-3.5_2.12-1.11.0.jar
+```
+
+### Cluster Connection
+
+The cluster that runs the jobs can be the one Gravitino runs in or another
one. The job executor
+connects to it in one of these ways, in this order:
+
+| Way | Configurations | Notes
|
+|-----------------|----------------------------------------|------------------------------------------------------------------------------------------------------------------------|
+| API server URL | `masterUrl`, `caCertFile`, `tokenFile` | The token file is
read again whenever it changes, so the token can be rotated without restarting
Gravitino. |
+| Kubeconfig file | `kubeconfig`, `context` | Uses the current
context of the file if `context` isn't set.
|
+| Default lookup | `context` | The `KUBECONFIG`
environment variable or `~/.kube/config`, then the service account of the pod
that Gravitino runs in. |
+
+The Gravitino server doesn't start if the connection is misconfigured:
+
+- `masterUrl` is set together with `kubeconfig` or `context`, or `caCertFile`
or `tokenFile` is set
+ without `masterUrl`.
+- `masterUrl` doesn't start with `https://` or `http://`.
+- `masterUrl` starts with `http://` and `tokenFile` is set, which would send
the token in clear
+ text, or `caCertFile` is set, which has no effect without TLS.
+- A configured file doesn't exist, or the token file has no token.
+- The kubeconfig is empty or can't be parsed, has no `context` of that name,
or has no current
+ context while `context` isn't set, or the context selects no cluster with a
server.
+- The default lookup finds no kubeconfig, and Gravitino doesn't run in a pod.
+
+These checks don't connect to the API server. An API server that can't be
reached, an invalid or
+expired token, or a missing permission only shows when a job runs, as the
error of the run request,
+or when Gravitino pulls the job status, in the server log.
+
+In every case, the jobs run in `namespace` unless their job template sets its
own, not in the
+namespace of the kubeconfig context or of the pod that Gravitino runs in.
+
+For example, to connect to another cluster with the service account created in
[Kubernetes
+Permissions](#kubernetes-permissions), issue a token for it in that cluster,
and save the CA
+certificate of the cluster:
+
+```shell
+kubectl create token gravitino-job-executor -n default --duration=24h >
/etc/gravitino/k8s/token
+kubectl config view --raw --minify -o
jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d >
/etc/gravitino/k8s/ca.crt
+```
+
+The second command reads the CA certificate from the current context of
`kubectl`, which must be
+the one of that cluster.
+
+A token issued this way expires, and Gravitino doesn't refresh it. Have
whatever provides the file
+replace the token before it expires, for example by updating the Secret that
the file is mounted
+from. Then point the job executor to the API server, its CA certificate, and
the token:
+
+```text
+gravitino.jobExecutor.k8s.masterUrl = https://k8s.example.com:6443
+gravitino.jobExecutor.k8s.caCertFile = /etc/gravitino/k8s/ca.crt
+gravitino.jobExecutor.k8s.tokenFile = /etc/gravitino/k8s/token
+```
+
+`cluster` names the cluster in the job execution ID that Gravitino stores for
each job,
+`<cluster>/sparkapp/<namespace>/<name>`. Set the same name on every Gravitino
server, and don't
+change it while jobs run: the jobs submitted under another name can't be
queried or cancelled, and
+stay unfinished until the name is changed back.
+
+If the API server can't be reached, each request is tried `1 + requestRetries`
times, each waiting
+up to `connectionTimeoutInMs`, or `requestTimeoutInMs` for an API server that
accepts the request
+but doesn't answer. With the defaults, a cancellation fails after about 15
seconds if the API server
+can't be reached, or 30 seconds if it doesn't answer. A submission can take
twice as long, as the
+job executor then also tries to delete the `SparkApplication` in case it was
created. It does the
+same after a server error from the API server. If that deletion fails too,
Gravitino cancels the
+job later, see [Failed Submissions](#failed-submissions), and the server log
names the
+`SparkApplication` to delete by hand. After a status query fails this way, the
status queries of the
+other jobs fail right away for 10 seconds, and Gravitino tries them again on
its next status pull.
+
+### Enable the Job Executor
+
+Set the job executor and its three required configurations in the
`gravitino.conf` file, then
+restart the Gravitino server:
+
+```text
+gravitino.job.executor = k8s
+gravitino.jobExecutor.k8s.namespace = <the namespace of the jobs>
+gravitino.jobExecutor.k8s.spark.image = <the Spark image>
+gravitino.jobExecutor.k8s.spark.sparkVersion = 3.5.9
+```
+
+## Configuration
+
+Set the following configurations in the `gravitino.conf` file. An invalid
value stops the Gravitino
+server from starting.
+
+| Property name | Description
| Default value | Required |
+|--------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------|----------|
+| `gravitino.jobExecutor.k8s.cluster` | The name of
the cluster that runs the jobs, a part of the job execution ID. It can't
contain `/`, and must be the same on every Gravitino server
| `default` | No |
+| `gravitino.jobExecutor.k8s.kubeconfig` | The
kubeconfig file of the cluster
| (none) | No |
+| `gravitino.jobExecutor.k8s.context` | The
kubeconfig context to use
| The current context | No |
+| `gravitino.jobExecutor.k8s.masterUrl` | The URL of
the API server of the cluster, an alternative to the kubeconfig. It starts with
`https://`, or with `http://` without `caCertFile` and `tokenFile`
| (none) | No |
+| `gravitino.jobExecutor.k8s.caCertFile` | The CA
certificate file of the API server, used with `masterUrl`
| (none) | No |
+| `gravitino.jobExecutor.k8s.tokenFile` | The file of
the bearer token to access the API server, used with `masterUrl`. It is read
again whenever it changes
| (none) | No |
+| `gravitino.jobExecutor.k8s.namespace` | The namespace
of the jobs whose job template doesn't set `spark.kubernetes.namespace`
| (none) | Yes |
+| `gravitino.jobExecutor.k8s.namePrefix` | The prefix of
the `SparkApplication` names: at most 24 lowercase letters, digits, and `-`,
starting with a letter and ending with a letter or a digit
| `gravitino-job` | No |
+| `gravitino.jobExecutor.k8s.connectionTimeoutInMs` | How long in
milliseconds to wait to connect to the API server
| `5000` (5 seconds) | No |
+| `gravitino.jobExecutor.k8s.requestTimeoutInMs` | How long in
milliseconds to wait for the response of a request to the API server
| `10000` (10 seconds) | No |
+| `gravitino.jobExecutor.k8s.requestRetries` | How many
times a request to the API server is tried again when the server can't be
reached, doesn't answer in time, or answers with a server error
| `2` | No |
+| `gravitino.jobExecutor.k8s.statusCacheTtlInMs` | How long in
milliseconds the `SparkApplication` resources listed in a namespace are cached
for status queries. Zero or less disables the cache
| `5000` (5 seconds) | No |
+| `gravitino.jobExecutor.k8s.noStatusTimeoutInMs` | How long in
milliseconds a job waits for the operator to write the first status of its
`SparkApplication`, after which the `SparkApplication` is deleted and the job
fails | `600000` (10 minutes) | No |
+| `gravitino.jobExecutor.k8s.spark.image` | The image of
the Spark driver and executors, unless the job template or a server-level
default sets `spark.kubernetes.container.image`
| (none) | Yes |
+| `gravitino.jobExecutor.k8s.spark.sparkVersion` | The Spark
version of the image
| (none) | Yes |
+| `gravitino.jobExecutor.k8s.spark.serviceAccount` | The service
account of the Spark driver, unless the job template or a server-level default
sets `spark.kubernetes.authenticate.driver.serviceAccountName`
| `spark` | No |
+| `gravitino.jobExecutor.k8s.spark.resourceRetainDurationInMs` | How long in
milliseconds the pods of a finished job are retained. The output of a job can
be read during this time
| `86400000` (1 day) | No |
+| `gravitino.jobExecutor.k8s.spark.ttlAfterStopInMs` | How long in
milliseconds the `SparkApplication` of a job is kept after the job stops
| `604800000` (7 days) | No |
+| `gravitino.jobExecutor.k8s.spark.driverStartTimeoutInMs` | How long in
milliseconds the operator waits for the driver pod to start before the job
fails. It includes the time the driver pod waits for resources or in a
scheduler queue | `3600000` (1 hour) | No |
+| `gravitino.jobExecutor.k8s.spark.driverReadyTimeoutInMs` | How long in
milliseconds the operator waits for the started driver to become ready before
the job fails
| `3600000` (1 hour) | No |
+| `gravitino.jobExecutor.k8s.spark.builtinJobsJar` | The URI of
the built-in jobs jar that the Spark pods can reach, see [Job
Resources](#job-resources)
| `local:///opt/gravitino/jobs/gravitino-jobs.jar` | No
|
+| `gravitino.jobExecutor.k8s.spark.conf.<key>` | A default
Spark configuration `<key>` of every job, see [Job Execution](#job-execution)
| (none) | No |
+
+## Queueing
+
+The job executor creates the `SparkApplication` of a job as soon as the job is
submitted. It doesn't
+limit the number of unfinished jobs and has no queue of its own, so a burst of
jobs can exhaust the
+cluster:
+
+- A driver pod that doesn't fit stays pending, and its job stays `QUEUED` until
+ `spark.driverStartTimeoutInMs`, then fails.
+- A job whose driver runs but whose executors can't be scheduled stays
`STARTED` and makes no
+ progress.
+
+To queue jobs, run them with a scheduler that has queues, such as
+[Apache YuniKorn](https://yunikorn.apache.org/).
+
+### YuniKorn
+
+Send every job to a YuniKorn queue with three server-level Spark
configurations. No annotation with
+the application ID is needed: YuniKorn groups the driver and the executors of
a job into one
+application.
+
+```text
+gravitino.jobExecutor.k8s.spark.conf.spark.kubernetes.scheduler.name = yunikorn
+gravitino.jobExecutor.k8s.spark.conf.spark.kubernetes.driver.label.queue =
root.gravitino
+gravitino.jobExecutor.k8s.spark.conf.spark.kubernetes.executor.label.queue =
root.gravitino
+```
+
+A job template can also set these Spark configurations in its `configs`, to
run in a queue of its
+own.
+
+Limit the queue in the YuniKorn configuration, the `queues.yaml` entry of the
ConfigMap
+`yunikorn-configs`:
+
+```yaml
+partitions:
+ - name: default
+ queues:
+ - name: root
+ submitacl: '*'
+ queues:
+ - name: gravitino
+ maxapplications: 10
+ resources:
+ max:
+ vcore: 40
+ memory: 160Gi
+```
+
+- `maxapplications` limits the jobs that run at the same time. The driver pods
of the other jobs
+ stay pending and their jobs are `QUEUED`. The next job starts a few seconds
after the pods of a
+ running job end.
+- `resources.max` limits the resources of the pods in the queue, not the
number of jobs. A driver
+ pod is admitted as soon as it fits, even if its executors don't: such a job
is `STARTED` and
+ waits for the executors of other jobs to end. Combine it with
`maxapplications`, so that the
+ admitted jobs can all get their executors.
+- Cancelling a queued job removes it from the queue.
+- A job that waits in the queue for longer than `spark.driverStartTimeoutInMs`
fails. Set this
+ timeout longer than the longest time a job may wait.
+
+The job executor doesn't depend on a scheduler: queueing works through the
Spark configurations
+above. The behaviors above were observed with YuniKorn 1.10.0 installed
without its admission
+controller (`embedAdmissionController=false`).
+
+## Environment Variables
+
+The `environments` of a job template are set for the driver and the executors
as plain Spark
+configurations: a variable `REGION` becomes
`spark.kubernetes.driverEnv.REGION` and
+`spark.executorEnv.REGION`. A name must consist of letters, digits, `_`, `-`,
and `.`, can't start
+with a digit or `..`, and can't be `.`. Otherwise, the run is rejected. The
same rule applies to the
+names of `spark.executorEnv.*` in the `configs` of a job template and in the
`spark_conf` of a
+built-in Iceberg job. Such a `spark.executorEnv.<name>` overrides the variable
only in the
+executors, as in Spark: `spark_conf` can't set `spark.kubernetes.driverEnv.*`,
so the driver keeps
+the value from the `environments` of the template.
+
+The variables are set in the driver and executor containers, while the local
job executor sets them
+on the `spark-submit` process on the Gravitino server. A template written for
the local job
+executor may set variables to paths on the Gravitino server, such as
`SPARK_HOME`,
+`SPARK_CONF_DIR`, or `HADOOP_CONF_DIR`. In the pods, such a variable overrides
the value of the
+image. A `SPARK_HOME` that isn't the Spark home of the image fails the job, as
the driver can't
+start `$SPARK_HOME/bin/spark-submit`. Any path in `environments` must be a
path in the Spark image.
+
+:::caution
+The values are readable in the `SparkApplication` and in the driver and
executor pods, as well as
+in the job template stored in Gravitino and in the `runtimeJobTemplate` of a
job returned by the
+API. Don't put sensitive values in `environments`.
+:::
+
+### Sensitive Values
+
+Pass a sensitive value from a Kubernetes Secret instead. Create the Secret in
the namespace of the
+job beforehand:
+
+```shell
+kubectl create secret generic warehouse-credentials --from-literal=token=<the
token>
+```
+
+Then reference it in the `configs` of the job template, or in the server-level
defaults with the
+`gravitino.jobExecutor.k8s.spark.conf.` prefix. Each configuration names an
environment variable,
+and its value is the Secret and the key in it:
+
+```json
+{
+ "name": "nightly_rollup",
+ "jobType": "spark",
+ "executable": "s3a://jobs/rollup.jar",
+ "className": "com.example.Rollup",
+ "configs": {
+ "spark.kubernetes.driver.secretKeyRef.WAREHOUSE_TOKEN":
"warehouse-credentials:token",
+ "spark.kubernetes.executor.secretKeyRef.WAREHOUSE_TOKEN":
"warehouse-credentials:token"
+ }
+}
+```
+
+- Kubernetes resolves the Secret when it starts the pods. Neither Gravitino
nor the service account
+ of the driver needs permission on Secrets.
+- The Secret must exist before the job runs. Otherwise, the driver pod can't
start and the job fails
+ after `spark.driverStartTimeoutInMs`.
+- The Secret isn't deleted with the job.
+- A variable set both in `environments` and from a Secret takes the value of
the Secret.
+- The Secret name can be a placeholder, such as `{{secret}}:token`, to choose
the Secret for each
+ run.
+
+Like `environments`, the values of the `spark_conf` parameter of a built-in
Iceberg job, such as
+catalog credentials, are readable in the `sparkConf` and the `driverArgs` of
the `SparkApplication`.
+Don't put sensitive values there. Pass them as environment variables from a
Secret where the job
+reads them from its environment, for example the AWS credentials, which the
AWS SDK reads from
+`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
+
+## Job Output and Times
+
+The output of a job is the log of its driver pod, read from Kubernetes on
every request:
+
+- The standard error is merged into the standard output, so `stderr` is always
empty.
+- The output is available while the driver pod exists. After that, the output
is empty.
+
+The driver pod is deleted, and the output with it:
+
+- `spark.resourceRetainDurationInMs` after the job finishes, 1 day by default,
or earlier when the
+ `SparkApplication` is deleted after `spark.ttlAfterStopInMs`.
+- Right away when the job is cancelled, as cancelling deletes the
`SparkApplication` and its pods.
+- When the driver pod of a failed job hasn't terminated, for example after the
driver start
+ timeout. The job executor deletes it so that it can't run the job later.
+
+The start and finish times of a job are the ones Kubernetes reports for the
driver container, in
+whole seconds. The operator records that a job finished 10 seconds to 2
minutes after the driver
+exits, so the status of a job can lag behind by that time, but its times don't.
+
+The actual times are lost in these cases:
+
+- When the driver pod is gone before Gravitino records the finished job, the
times are the ones the
+ operator recorded, which can be later than the actual ones.
+- When the `SparkApplication` is gone before Gravitino records the finished
job, for example a
+ cancelled job, the finish time is the time Gravitino observes it. The finish
time of a cancelled
+ job is therefore usually up to `gravitino.job.statusPullIntervalInMs` late.
+
+## Multiple Servers and Restarts
+
+The job executor keeps no job state on the Gravitino server:
+
+- When multiple Gravitino servers share the same metadata store, each of them
can query and cancel
+ any job, and read its output. Configure all of them with the same cluster
and the same `cluster`
+ name.
+- Jobs keep running while Gravitino is down, and Gravitino picks up their
status when it is back.
+ The `SparkApplication` of a job is deleted `spark.ttlAfterStopInMs` after
the job stops, 7 days by
+ default. A job that Gravitino hasn't recorded as finished by then fails.
+- If a Gravitino server exits while submitting a job, another server, or the
same server after a
+ restart, looks up the `SparkApplication` of the job about three times
+ `gravitino.job.statusPullIntervalInMs` later. A job that is found is tracked
as usual, and a job
+ that isn't found fails. The lookup derives the name and namespace of the
`SparkApplication` from
+ the job template and the current `namePrefix` and `namespace`
configurations, so a job isn't found
+ if they changed in between.
+- Gravitino deployments that share a namespace need different `namePrefix`
values, so that their
+ `SparkApplication` names don't collide.
+
+## Limitations
+
+- Only Spark templates are supported.
+- The standard error of a job is merged into its standard output.
+- A job whose driver pod is evicted or deleted fails. It isn't run again.
+- A job whose executors can't be scheduled stays `STARTED`: there is no
timeout for the executors.
+- The operator checks the start timeouts about every 2 minutes, so a job can
fail up to 2 minutes
+ after `spark.driverStartTimeoutInMs` or `spark.driverReadyTimeoutInMs`.
+- The finish time of a cancelled job is usually up to
`gravitino.job.statusPullIntervalInMs` late.
+- The output of a job is no longer available
`spark.resourceRetainDurationInMs` after it finishes,
+ and a cancelled job loses its output right away.
+- Unfinished jobs fail if Gravitino is down for longer than
`spark.ttlAfterStopInMs` after they stop.
+- The job executor doesn't limit the number of unfinished jobs, see
[Queueing](#queueing).
+
+## Run a Built-In Job on a Local Cluster
+
+The following steps run the built-in job `builtin-iceberg-rewrite-data-files`
on the Kubernetes
+cluster of [OrbStack](https://orbstack.dev/), which shares the local Docker
images. Another local
+cluster, such as kind, also needs the Spark image loaded into it. The steps
need `kubectl`, `helm`,
+`docker`, and a Gravitino distribution, see [How to build](./how-to-build.md).
+
+Point `kubectl` and `helm` at the cluster first:
+
+```shell
+kubectl config use-context orbstack
+```
+
+### Step 1: Install the Spark Kubernetes Operator
+
+Chart 1.7.0 installs operator 0.9.0. The chart also creates the service
account `spark` for the
+Spark drivers in the namespace it is installed in.
+
+```shell
+helm repo add spark https://apache.github.io/spark-kubernetes-operator
+helm repo update
+helm install spark spark/spark-kubernetes-operator --version 1.7.0
+```
+
+### Step 2: Build the Spark Image
+
+Run the following commands in the directory of the Gravitino distribution. The
Dockerfile is the one
+in [Spark Image](#spark-image).
+
+```shell
+mkdir spark-image
+cp auxlib/gravitino-jobs-*.jar spark-image/
+cat > spark-image/Dockerfile <<'EOF'
+FROM apache/spark:3.5.9-java17
+USER root
+COPY gravitino-jobs-*.jar /opt/gravitino/jobs/gravitino-jobs.jar
+ADD
--checksum=sha256:94b8e36fc329f0293d44ba9e01b784a56e9501affec1842d898144c51f6e486a
https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/1.11.0/iceberg-spark-runtime-3.5_2.12-1.11.0.jar
/opt/spark/jars/
+RUN chmod 644 /opt/gravitino/jobs/gravitino-jobs.jar
/opt/spark/jars/iceberg-spark-runtime-3.5_2.12-1.11.0.jar
+USER spark
+EOF
+docker build -t gravitino-spark:3.5.9 spark-image
+```
+
+### Step 3: Create an Iceberg Table
+
+The built-in Iceberg jobs work on an existing table, in a catalog and a
warehouse that the Spark
+pods can reach. This example keeps a Hadoop catalog on a volume, and creates a
table with three
+data files in it. In production, use the Iceberg catalog and the object store
of the tables instead.
+
+Save the following as `iceberg-table.yaml`:
+
+```yaml
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+ name: iceberg-warehouse
+spec:
+ accessModes: ["ReadWriteOnce"]
+ resources:
+ requests:
+ storage: 1Gi
+---
+apiVersion: v1
+kind: Pod
+metadata:
+ name: iceberg-table
+spec:
+ restartPolicy: Never
+ containers:
+ - name: spark-sql
+ image: gravitino-spark:3.5.9
+ command: ["/opt/spark/bin/spark-sql"]
+ args:
+ - --conf
+ -
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
+ - --conf
+ - spark.sql.catalog.demo=org.apache.iceberg.spark.SparkCatalog
+ - --conf
+ - spark.sql.catalog.demo.type=hadoop
+ - --conf
+ - spark.sql.catalog.demo.warehouse=file:///warehouse
+ - -e
+ - |
+ CREATE TABLE demo.db.events (id BIGINT, name STRING) USING iceberg;
+ INSERT INTO demo.db.events VALUES (1, 'a');
+ INSERT INTO demo.db.events VALUES (2, 'b');
+ INSERT INTO demo.db.events VALUES (3, 'c');
+ volumeMounts:
+ - name: warehouse
+ mountPath: /warehouse
+ volumes:
+ - name: warehouse
+ persistentVolumeClaim:
+ claimName: iceberg-warehouse
+```
+
+Create the table and wait for the pod to finish:
+
+```shell
+kubectl apply -f iceberg-table.yaml
+kubectl wait --for=jsonpath='{.status.phase}'=Succeeded pod/iceberg-table
--timeout=180s
+```
+
+### Step 4: Configure and Start Gravitino
+
+Append the following to `conf/gravitino.conf`. The four `spark.conf.` lines
mount the warehouse
+volume into the driver and the executors of every job. The short status pull
interval shows the job
+progress quickly, and is for this walkthrough only: Gravitino logs a warning
about it.
+
+```text
+gravitino.job.executor = k8s
+gravitino.job.statusPullIntervalInMs = 2000
+gravitino.jobExecutor.k8s.context = orbstack
+gravitino.jobExecutor.k8s.namespace = default
+gravitino.jobExecutor.k8s.spark.image = gravitino-spark:3.5.9
+gravitino.jobExecutor.k8s.spark.sparkVersion = 3.5.9
+gravitino.jobExecutor.k8s.spark.conf.spark.kubernetes.driver.volumes.persistentVolumeClaim.warehouse.options.claimName
= iceberg-warehouse
+gravitino.jobExecutor.k8s.spark.conf.spark.kubernetes.driver.volumes.persistentVolumeClaim.warehouse.mount.path
= /warehouse
+gravitino.jobExecutor.k8s.spark.conf.spark.kubernetes.executor.volumes.persistentVolumeClaim.warehouse.options.claimName
= iceberg-warehouse
+gravitino.jobExecutor.k8s.spark.conf.spark.kubernetes.executor.volumes.persistentVolumeClaim.warehouse.mount.path
= /warehouse
+```
+
+Start the server and create a metalake:
+
+```shell
+./bin/gravitino.sh start
+
+curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" \
+ -d '{"name": "demo", "comment": "", "properties": {}}' \
+ http://localhost:8090/api/metalakes
+```
+
+### Step 5: Run the Job
+
+The Hadoop catalog doesn't use `catalog_uri`, but the job template requires a
value for it. The
+`min-input-files` option makes the job rewrite the three small files of the
table.
+
+```shell
+curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" -d '{
+ "jobTemplateName": "builtin-iceberg-rewrite-data-files",
+ "jobConf": {
+ "catalog_name": "demo",
+ "catalog_type": "hadoop",
+ "catalog_uri": "",
+ "warehouse_location": "file:///warehouse",
+ "table_identifier": "db.events",
+ "where_clause": "",
+ "options": "{\"min-input-files\":\"2\"}"
+ }
+}' http://localhost:8090/api/metalakes/demo/jobs/runs
+```
+
+The response contains the ID of the job, such as `job-688822913391439376`.
+
+### Step 6: Check the Job and Read Its Output
+
+Get the job with its output:
+
+```shell
+curl -H "Accept: application/vnd.gravitino.v1+json" \
+
"http://localhost:8090/api/metalakes/demo/jobs/runs/{job_id}?includeOutput=true"
+```
+
+The `status` of the job turns from `queued` to `started` and then to
`succeeded`, the REST
+values of `QUEUED`, `STARTED`, and `SUCCEEDED`. Its `stdout` is
+the log of the Spark driver, which contains the result of the job:
+
+```text
+Rewrite Data Files Results:
+ Rewritten data files: 3
+ Added data files: 1
+ Rewritten bytes: 1923
+ Failed data files: 0
+ Removed delete files: 0
+Rewrite data files job completed successfully
+```
+
+The job is also visible in Kubernetes:
+
+```shell
+kubectl get sparkapplications -l app.kubernetes.io/managed-by=gravitino
+kubectl get pods -l spark-role=driver
+```
+
+### Step 7: Clean Up
+
+```shell
+./bin/gravitino.sh stop
+kubectl delete sparkapplications -l app.kubernetes.io/managed-by=gravitino
+kubectl delete -f iceberg-table.yaml
+helm uninstall spark
+```
diff --git a/docs/local-job-executor.md b/docs/local-job-executor.md
new file mode 100644
index 0000000000..762c621baf
--- /dev/null
+++ b/docs/local-job-executor.md
@@ -0,0 +1,164 @@
+---
+title: "Local Job Executor"
+slug: "/local-job-executor"
+keyword: "job executor, local job executor, job system, shell job, spark job,
Gravitino"
+license: "This software is licensed under the Apache License version 2."
+---
+
+## Introduction
+
+The `local` job executor runs every job as a process on the Gravitino server.
It is the default job
+executor and is intended for testing and development. To run jobs on a
cluster, use the
+[Kubernetes job executor](./k8s-job-executor.md) or a
+[custom job executor](./development/custom-job-executor.md).
+
+For job templates and the job API, see [Manage
Jobs](./manage-jobs-in-gravitino.md).
+
+## Requirements
+
+- A shell template whose `executable` is a command name, such as `python`,
needs that command on
+ the `PATH` of the Gravitino server process.
+- A Spark template needs either `gravitino.jobExecutor.local.sparkHome` or
`SPARK_HOME` set before
+ the server starts, pointing to a Spark installation with an executable
`bin/spark-submit`.
+ Otherwise, the run request is rejected with an error that names the missing
setting, and the job
+ is removed.
+
+## Job Execution
+
+The local job executor fetches the resources of a job, that is its executable,
scripts, jars, files,
+and archives, into the staging directory of the job, and runs the job there as
a process on the
+Gravitino server host. An `executable` that is a command name, such as
`python`, isn't fetched. The
+resources must be reachable by the Gravitino server, which accepts local paths
and HTTP, HTTPS, FTP,
+and FTPS URLs.
+
+The driver of a built-in Iceberg job applies its `spark_conf` parameter only
after it starts, so
+`spark.jars` there doesn't add a jar to the job, such as the Iceberg Spark
runtime. Install the jar
+into `SPARK_HOME/jars` instead.
+
+The executor always uses `gravitino.job.stagingDir` as its staging directory,
the same one the job
+system stages jobs in. A `gravitino.jobExecutor.local.stagingDir` setting is
ignored.
+
+## Configuration
+
+Set the following configurations in the `gravitino.conf` file:
+
+| Property name | Description
| Default value | Required |
+|--------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------|----------|
+| `gravitino.jobExecutor.local.waitingQueueSize` | The size of the
waiting queue for queued jobs in the local job executor
| `100` | No |
+| `gravitino.jobExecutor.local.maxRunningJobs` | The maximum number
of running jobs in the local job executor
| `max(1, min(available cores / 2, 10))` | No |
+| `gravitino.jobExecutor.local.jobStatusKeepTimeInMs` | The time in
milliseconds to keep the job status in the local job executor
| `3600000` (1 hour) | No |
+| `gravitino.jobExecutor.local.cancelForceKillDelayInMs` | How long in
milliseconds a cancelled job's process may keep running after it is asked to
stop before the executor kills it forcibly. Raise it for jobs that need longer
to shut down cleanly | `30000` (30 seconds) | No |
+| `gravitino.jobExecutor.local.sparkHome` | The home directory
of Spark, Gravitino checks this configuration firstly and then `SPARK_HOME`
env. Either of them should be set to run Spark job
| (none) | No |
+
+## Queueing
+
+The local job executor runs up to `gravitino.jobExecutor.local.maxRunningJobs`
jobs at the same
+time, each in its own process on the Gravitino server host, and queues the
others. Make sure the
+host has enough resources for that many jobs, or lower this value, especially
when running Spark
+jobs.
+
+## Environment Variables
+
+The `environments` of a job template are set as the environment variables of
the job's process,
+which is `spark-submit` for a Spark template.
+
+## Job Output and Times
+
+The local job executor keeps a job's output in the job's staging directory, so
it's available until
+the staging directory is cleaned up (`gravitino.job.stagingDirKeepTimeInMs`
after the job finishes),
+also across server restarts.
+
+:::caution
+When multiple Gravitino servers share the same metadata store,
`gravitino.job.stagingDir` must be on
+storage shared by all servers (for example an NFS mount) for the local job
executor to return a
+job's output from any server. The servers may mount it at different paths.
Otherwise, only the
+server that ran a job can return its output, and the other servers return
empty output rather than
+an error.
+:::
+
+The local job executor finds a job's output through a small index file it
writes to
+`<gravitino.job.stagingDir>/.job-output-index` when the job is submitted:
+
+- Jobs submitted before Gravitino 2.0.0, or during a rolling upgrade by a
server that isn't upgraded
+ yet, have no index file and return empty output.
+- Deleting this directory makes the output of existing jobs unavailable. After
downgrading to an
+ earlier version, it isn't used anymore and can be removed.
+- A missing index returns empty output, but an index that exists and can't be
read, for example
+ while the shared storage is unavailable, fails the request with an error
rather than returning
+ empty output that looks like the job printed nothing.
+
+The local job executor reports when each job actually started and finished,
even for a job that
+starts and finishes between two status pulls. It only keeps a job's state in
memory. When that state
+is lost before Gravitino records the finished job, the actual times are lost
too, and the job's
+`finishedAt` is the time Gravitino marks it as finished instead:
+
+- If Gravitino can't pull the job within
`gravitino.jobExecutor.local.jobStatusKeepTimeInMs` after it
+ finished, the job is marked as `FAILED`, or `CANCELLED` if it was being
cancelled, on the next pull.
+- If the server running the job exits, the job is expired as described in
+ [Multiple Servers and Restarts](#multiple-servers-and-restarts).
+
+## Multiple Servers and Restarts
+
+When multiple Gravitino servers share the same metadata store, each server's
local job executor
+only tracks the jobs it runs itself:
+
+- A job can only be run and tracked by the server that received the run
request. Other servers
+ skip it when pulling job statuses.
+- Cancelling a job on a server that doesn't run it marks the job as
`CANCELLING`, and the server
+ running the job cancels it the next time it pulls job statuses. This can
take up to
+ `gravitino.job.statusPullIntervalInMs`.
+- A job's output can be read from any server only if
`gravitino.job.stagingDir` is shared by all
+ servers, see [Job Output and Times](#job-output-and-times).
+- If a server exits while running jobs, nobody can track these jobs anymore.
When such a job has
+ not been updated for `gravitino.job.stagingDirKeepTimeInMs`, it is marked as
`FAILED`, or as
+ `CANCELLED` if it was being cancelled. Like other finished jobs, it is then
kept for another
+ `gravitino.job.stagingDirKeepTimeInMs` before being cleaned up together with
its staging
+ directory.
+- For a job whose execution id was recorded, cancelling it can clear it
sooner. If it is still
+ `CANCELLING` more than three times `gravitino.job.statusPullIntervalInMs`
after the cancel request,
+ the server running it is considered gone, and another server marks it as
`CANCELLED` when it
+ pulls job statuses. Its job template can then be deleted.
+- A job whose execution id was never recorded is settled within about four
times
+ `gravitino.job.statusPullIntervalInMs` too, as described in
+ [Run a Job](./manage-jobs-in-gravitino.md#run-a-job).
+
+:::caution
+The local job executor can't tell a job left behind by an exited server from a
job that is still
+running without changing its status. A job of the local job executor that is
still queued, started
+or cancelling after `gravitino.job.stagingDirKeepTimeInMs` is marked as
`FAILED` (or `CANCELLED`),
+even if the job is still running, and keeps this status even if it later
finishes. Set this time
+longer than any job can run, or stay queued, without changing its status.
+:::
+
+:::caution
+The local job executor gets a new identity every time the Gravitino server
starts, so a restarted
+server doesn't recognize the jobs it ran before the restart. This also applies
to a single-server
+deployment. The processes of these jobs are usually gone with the previous
server process, but the
+jobs are only marked as `FAILED` once they expire as described above, which
can take up to about
+1.1 times `gravitino.job.stagingDirKeepTimeInMs` (about 7.7 days by default),
as the cleanup runs
+every tenth of that time. Until then, they are still reported as queued,
started or cancelling, and
+they keep their job templates from being deleted.
+
+For jobs whose execution id was recorded, cancelling them settles them as
`CANCELLED` within about
+four times `gravitino.job.statusPullIntervalInMs`, as described above. Lowering
+`gravitino.job.stagingDirKeepTimeInMs` shortens retention, but finished jobs
are then kept for a
+shorter time, and a job that runs longer than that without changing its status
is marked as
+`FAILED`.
+:::
+
+:::caution
+Telling a gone server from a live one by how long a job stays `CANCELLING`
requires all servers to
+use the same `gravitino.job.statusPullIntervalInMs` and to have their clocks
in sync. A job is also
+marked as `CANCELLED` while its server is still running it if that server
doesn't answer the
+cancellation in time, for example because:
+
+- it uses a longer `gravitino.job.statusPullIntervalInMs` than the other
servers, or its clock is
+ off,
+- it fails to pull job statuses for that long, or
+- the job process takes longer than one pull interval to exit, e.g. as
+ `gravitino.jobExecutor.local.cancelForceKillDelayInMs` is longer than the
pull interval.
+
+If the server hasn't asked the job process to stop yet, it never does once the
job is `CANCELLED`,
+so the process keeps running until it finishes, and deleting its job template
also deletes the
+staging directory the process uses.
+:::
diff --git a/docs/manage-jobs-in-gravitino.md b/docs/manage-jobs-in-gravitino.md
index 702e77c61d..a0c0bcace9 100644
--- a/docs/manage-jobs-in-gravitino.md
+++ b/docs/manage-jobs-in-gravitino.md
@@ -14,15 +14,15 @@ This page covers the Gravitino API for job templates and
jobs. For what a templa
relates to a run, and what the job statuses mean, see [Jobs](./jobs.md).
Jobs run through a job executor, set with `gravitino.job.executor`. The
default, `local`, launches
-the job as a process on the Gravitino server and is intended for testing.
Running jobs anywhere else
-means implementing an executor. See
-[Custom job executor](./development/custom-job-executor.md).
+the job as a process on the Gravitino server and is intended for testing. The
`k8s` job executor
+runs Spark jobs on Kubernetes. Running jobs anywhere else means implementing
an executor. See
+[Job Executors](#job-executors).
:::note
1. The job system is still under development, so some features may not be fully
implemented yet.
-2. The aim of the job system is not to replace the existing job executors. So,
it can only
- support running a single job at a time, and it doesn't support job
scheduling for now.
+2. The aim of the job system is not to replace the existing job executors, and
it doesn't
+ support job scheduling for now.
:::
## Job Template Operations
@@ -44,13 +44,15 @@ a job runs. See [Placeholders](#placeholders).
}
```
-`executable` and `scripts` are fetched by the job executor. With the `local`
job executor they must
-be reachable by the Gravitino server, which accepts local paths and HTTP,
HTTPS, FTP, and FTPS URLs.
+`executable` and `scripts` are fetched by the local job executor, so they must
be reachable by the
+Gravitino server, which accepts local paths and HTTP, HTTPS, FTP, and FTPS
URLs.
-Each of them is fetched into the job's working directory under the file name
of its path, so it
-must point to a file, and two different resources of a template must not share
a file name: a run
-is rejected otherwise, instead of one file silently overwriting the other. The
same applies to the
-`executable`, `jars`, `files` and `archives` of a Spark template.
+With the local job executor, each of them is fetched into the job's working
directory under the
+file name of its path, so it must point to a file, and two different resources
of a template must
+not share a file name: a run is rejected otherwise, instead of one file
silently overwriting the
+other. The same applies to the `executable`, `jars`, `files` and `archives` of
a Spark template.
+The Kubernetes job executor doesn't fetch the resources, see
+[Kubernetes Job Executor](./k8s-job-executor.md#job-resources).
`executable` can also be a command name with no path, such as `python` or
`bash`. Such a command is
not fetched, it is looked up where the job runs: the `local` job executor
looks it up on the `PATH`
@@ -94,10 +96,13 @@ curl -X POST -H "Accept: application/vnd.gravitino.v1+json"
\
### Register a Spark Template
-A Spark template submits an application. Running one with the local executor
needs either
-`gravitino.jobExecutor.local.sparkHome` or `SPARK_HOME` set before the server
starts, pointing to a
-Spark installation with an executable `bin/spark-submit`. Otherwise, the run
request is rejected with
-an error that names the missing setting, and no job is created.
+A Spark template submits an application. What it needs depends on the job
executor:
+
+- The local job executor needs a Spark installation on the Gravitino server,
see
+ [Local Job Executor](./local-job-executor.md#requirements).
+- The Kubernetes job executor needs every resource of the template to be a URI
that the cluster can
+ reach, such as `local://` for a file in the Spark image, `https://`, or
`s3a://`, see
+ [Kubernetes Job Executor](./k8s-job-executor.md#job-resources).
```json
{
@@ -284,7 +289,8 @@ Otherwise another server may settle the job, and
cancellation of a late submissi
before its run request fails. Cancellation cannot undo completed work. If the
job was already
settled and this late cancellation fails, check the execution in the runner
before retrying.
`gravitino.job.stagingDirKeepTimeInMs` must also be longer than preparing,
submitting, queueing and
-running a job of the local executor, as described below.
+running a job of the local executor, as described in
+[Local Job Executor](./local-job-executor.md#multiple-servers-and-restarts).
:::
### List Jobs, Get a Job, and Cancel
@@ -338,19 +344,14 @@ A job carries three timestamps:
Gravitino pulls job statuses from the job executor every
`gravitino.job.statusPullIntervalInMs`,
so a job's status can lag behind by up to this interval. The timestamps
usually don't lag: the local
-job executor reports when each job actually started and finished, even for a
job that starts and
-finishes between two pulls. A job executor that doesn't report these times
gets the time Gravitino
-first observes the job running or finished instead. In that case, a job that
finishes between two
-pulls has no `startedAt`.
-
-The local job executor only keeps a job's state in memory. When that state is
lost before Gravitino
-records the finished job, the actual times are lost too, and the job's
`finishedAt` is the time
-Gravitino marks it as finished instead:
+and Kubernetes job executors report when each job actually started and
finished, even for a job that
+starts and finishes between two pulls. A job executor that doesn't report
these times gets the time
+Gravitino first observes the job running or finished instead. In that case, a
job that finishes
+between two pulls has no `startedAt`.
-- If Gravitino can't pull the job within
`gravitino.jobExecutor.local.jobStatusKeepTimeInMs` after it
- finished, the job is marked as `FAILED`, or `CANCELLED` if it was being
cancelled, on the next pull.
-- If the server running the job exits, the job is expired as described in
- [Configurations for Local Job
Executor](#configurations-for-local-job-executor).
+For the cases in which the actual times are lost, see
+[Local Job Executor](./local-job-executor.md#job-output-and-times) and
+[Kubernetes Job Executor](./k8s-job-executor.md#job-output-and-times).
### Get a Job's Output
@@ -388,9 +389,10 @@ stderr = job.stderr()
</TabItem>
</Tabs>
-Output is only kept for as long as the job executor retains it. The local job
executor keeps it in
-the job's staging directory, so it's available until the staging directory is
cleaned up
-(`gravitino.job.stagingDirKeepTimeInMs` after the job finishes), also across
server restarts.
+Output is only kept for as long as the job executor retains it: the local job
executor keeps it in
+the job's staging directory, and the Kubernetes job executor reads it from the
driver pod of the
+job. See [Local Job Executor](./local-job-executor.md#job-output-and-times) and
+[Kubernetes Job Executor](./k8s-job-executor.md#job-output-and-times).
What's returned is always the tail of the output (the most recent content),
capped
by `gravitino.job.outputMaxLines` (line count) and
`gravitino.job.outputMaxBytes` (byte size),
whichever limit is hit first.
@@ -405,132 +407,38 @@ curl -X GET -H "Accept:
application/vnd.gravitino.v1+json" \
"http://localhost:8090/api/metalakes/example/jobs/runs/{job_id}?includeOutput=true&outputMaxLines=50&outputMaxBytes=8192"
```
-:::caution
-When multiple Gravitino servers share the same metadata store,
`gravitino.job.stagingDir` must be on
-storage shared by all servers (for example an NFS mount) for the local job
executor to return a
-job's output from any server. The servers may mount it at different paths.
Otherwise, only the
-server that ran a job can return its output, and the other servers return
empty output rather than
-an error.
-:::
-
-The local job executor finds a job's output through a small index file it
writes to
-`<gravitino.job.stagingDir>/.job-output-index` when the job is submitted:
-
-- Jobs submitted before Gravitino 2.0.0, or during a rolling upgrade by a
server that isn't upgraded
- yet, have no index file and return empty output.
-- Deleting this directory makes the output of existing jobs unavailable. After
downgrading to an
- earlier version, it isn't used anymore and can be removed.
-- A missing index returns empty output, but an index that exists and can't be
read, for example
- while the shared storage is unavailable, fails the request with an error
rather than returning
- empty output that looks like the job printed nothing.
-
### Job System Configuration
Configure the job system through the `gravitino.conf` file. The following are
the
default configurations:
-| Property name | Description
| Default value
| Required |
-|----------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|----------|
-| `gravitino.job.stagingDir` | Directory for managing the staging
files when running jobs. Must be shared by all servers in a multi-server
deployment, see [Get a Job's Output](#get-a-jobs-output) |
`/tmp/gravitino/jobs/staging` | No |
-| `gravitino.job.executor` | The job executor to use for running
jobs
| `local`
| No |
-| `gravitino.job.stagingDirKeepTimeInMs` | The time in milliseconds to keep
the staging directory after the job is completed
| `604800000` (7 days)
| No |
-| `gravitino.job.statusPullIntervalInMs` | The interval in milliseconds to
pull the job status from the job executor
| `300000` (5 minutes)
| No |
-| `gravitino.job.outputMaxLines` | The maximum number of lines
returned when fetching a job's stdout/stderr output
| `1000`
| No |
-| `gravitino.job.outputMaxBytes` | The maximum number of bytes
returned from the tail of a job's stdout/stderr output
| `262144` (256KB)
| No |
-
-#### Configurations for Local Job Executor
-
-The local job executor is used for testing and development purposes, it runs
the job in the local process.
-The following are the default configurations for the local job executor:
-
-| Property name | Description
| Default value | Required |
-|--------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------|----------|
-| `gravitino.jobExecutor.local.waitingQueueSize` | The size of the
waiting queue for queued jobs in the local job executor
| `100` | No |
-| `gravitino.jobExecutor.local.maxRunningJobs` | The maximum number
of running jobs in the local job executor
| `max(1, min(available cores / 2, 10))` | No |
-| `gravitino.jobExecutor.local.jobStatusKeepTimeInMs` | The time in
milliseconds to keep the job status in the local job executor
| `3600000` (1 hour) | No |
-| `gravitino.jobExecutor.local.cancelForceKillDelayInMs` | How long in
milliseconds a cancelled job's process may keep running after it is asked to
stop before the executor kills it forcibly. Raise it for jobs that need longer
to shut down cleanly | `30000` (30 seconds) | No |
-| `gravitino.jobExecutor.local.sparkHome` | The home directory
of Spark, Gravitino checks this configuration firstly and then `SPARK_HOME`
env. Either of them should be set to run Spark job
| `None` | No |
-
-The local job executor always uses `gravitino.job.stagingDir` as its staging
directory, the same one
-the job system stages jobs in. A `gravitino.jobExecutor.local.stagingDir`
setting is ignored.
-
-The local job executor runs up to `gravitino.jobExecutor.local.maxRunningJobs`
jobs at the same
-time, each in its own process on the Gravitino server host, and queues the
others. Make sure the
-host has enough resources for that many jobs, or lower this value, especially
when running Spark
-jobs.
-
-When multiple Gravitino servers share the same metadata store, each server's
local job executor
-only tracks the jobs it runs itself:
-
-- A job can only be run and tracked by the server that received the run
request. Other servers
- skip it when pulling job statuses.
-- Cancelling a job on a server that doesn't run it marks the job as
`CANCELLING`, and the server
- running the job cancels it the next time it pulls job statuses. This can
take up to
- `gravitino.job.statusPullIntervalInMs`.
-- A job's output can be read from any server only if
`gravitino.job.stagingDir` is shared by all
- servers, see [Get a Job's Output](#get-a-jobs-output).
-- If a server exits while running jobs, nobody can track these jobs anymore.
When such a job has
- not been updated for `gravitino.job.stagingDirKeepTimeInMs`, it is marked as
`FAILED`, or as
- `CANCELLED` if it was being cancelled. Like other finished jobs, it is then
kept for another
- `gravitino.job.stagingDirKeepTimeInMs` before being cleaned up together with
its staging
- directory.
-- For a job whose execution id was recorded, cancelling it can clear it
sooner. If it is still
- `CANCELLING` more than three times `gravitino.job.statusPullIntervalInMs`
after the cancel request,
- the server running it is considered gone, and another server marks it as
`CANCELLED` when it
- pulls job statuses. Its job template can then be deleted.
-- A job whose execution id was never recorded is settled within about four
times
- `gravitino.job.statusPullIntervalInMs` too, as described in [Run a
Job](#run-a-job).
+| Property name | Description
| Default value | Required |
+|----------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|----------|
+| `gravitino.job.stagingDir` | Directory for managing the staging
files when running jobs. With the local job executor, it must be shared by all
servers in a multi-server deployment, see [Local Job
Executor](./local-job-executor.md#job-output-and-times) |
`/tmp/gravitino/jobs/staging` | No |
+| `gravitino.job.executor` | The job executor to use for running
jobs, see [Job Executors](#job-executors)
| `local` | No |
+| `gravitino.job.stagingDirKeepTimeInMs` | The time in milliseconds to keep
the staging directory after the job is completed
| `604800000` (7 days) | No |
+| `gravitino.job.statusPullIntervalInMs` | The interval in milliseconds to
pull the job status from the job executor
| `300000` (5 minutes) | No |
+| `gravitino.job.outputMaxLines` | The maximum number of lines
returned when fetching a job's stdout/stderr output
| `1000` | No |
+| `gravitino.job.outputMaxBytes` | The maximum number of bytes
returned from the tail of a job's stdout/stderr output
| `262144` (256KB) | No |
-:::caution
-The local job executor can't tell a job left behind by an exited server from a
job that is still
-running without changing its status. A job of the local job executor that is
still queued, started
-or cancelling after `gravitino.job.stagingDirKeepTimeInMs` is marked as
`FAILED` (or `CANCELLED`),
-even if the job is still running, and keeps this status even if it later
finishes. Set this time
-longer than any job can run, or stay queued, without changing its status.
-:::
+## Job Executors
-:::caution
-The local job executor gets a new identity every time the Gravitino server
starts, so a restarted
-server doesn't recognize the jobs it ran before the restart. This also applies
to a single-server
-deployment. The processes of these jobs are usually gone with the previous
server process, but the
-jobs are only marked as `FAILED` once they expire as described above, which
can take up to about
-1.1 times `gravitino.job.stagingDirKeepTimeInMs` (about 7.7 days by default),
as the cleanup runs
-every tenth of that time. Until then, they are still reported as queued,
started or cancelling, and
-they keep their job templates from being deleted.
-
-For jobs whose execution id was recorded, cancelling them settles them as
`CANCELLED` within about
-four times `gravitino.job.statusPullIntervalInMs`, as described above. Lowering
-`gravitino.job.stagingDirKeepTimeInMs` shortens retention, but finished jobs
are then kept for a
-shorter time, and a job that runs longer than that without changing its status
is marked as
-`FAILED`.
-:::
+A job executor decides where the jobs run. Set it with
`gravitino.job.executor`:
-:::caution
-Telling a gone server from a live one by how long a job stays `CANCELLING`
requires all servers to
-use the same `gravitino.job.statusPullIntervalInMs` and to have their clocks
in sync. A job is also
-marked as `CANCELLED` while its server is still running it if that server
doesn't answer the
-cancellation in time, for example because:
-
-- it uses a longer `gravitino.job.statusPullIntervalInMs` than the other
servers, or its clock is
- off,
-- it fails to pull job statuses for that long, or
-- the job process takes longer than one pull interval to exit, e.g. as
- `gravitino.jobExecutor.local.cancelForceKillDelayInMs` is longer than the
pull interval.
-
-If the server hasn't asked the job process to stop yet, it never does once the
job is `CANCELLED`,
-so the process keeps running until it finishes, and deleting its job template
also deletes the
-staging directory the process uses.
-:::
+| Job executor | Jobs run as
| Job types | Intended for | Configuration
|
+|--------------|------------------------------------------------------------------|--------------|-------------------------|--------------------------------------------------|
+| `local` | Processes on the Gravitino server
| Shell, Spark | Testing and development | [Local Job
Executor](./local-job-executor.md) |
+| `k8s` | `SparkApplication` resources of the Spark operator on
Kubernetes | Spark | Production | [Kubernetes Job
Executor](./k8s-job-executor.md) |
+
+To run jobs anywhere else, implement a
+[custom job executor](./development/custom-job-executor.md).
## Future Work
The job system still needs more work:
-1. Support modification of job templates.
-2. Support running Spark jobs (Java and PySpark) based on the Spark job
template in the local job
- executor.
-3. Support more job templates, like Python, SQL, etc.
-4. Support more job executors, like Apache Airflow, Apache Livy, etc.
-5. Support uploading job template related artifacts to the Gravitino server,
also support
+1. Support more job templates, like Python, SQL, etc.
+2. Support more job executors, like Apache Airflow, Apache Livy, etc.
+3. Support uploading job template related artifacts to the Gravitino server,
also support
downloading the artifacts from more distributed file systems like HDFS, S3,
etc.
-6. Support job scheduling, like running jobs periodically, or based on some
events.
+4. Support job scheduling, like running jobs periodically, or based on some
events.
diff --git a/docs/table-maintenance-service/optimizer-configuration.md
b/docs/table-maintenance-service/optimizer-configuration.md
index 226a21578d..99da41585a 100644
--- a/docs/table-maintenance-service/optimizer-configuration.md
+++ b/docs/table-maintenance-service/optimizer-configuration.md
@@ -25,6 +25,13 @@ gravitino.job.statusPullIntervalInMs=300000
gravitino.jobExecutor.local.sparkHome=/path/to/spark
```
+These settings run the jobs with the [local job
executor](../local-job-executor.md). To run them on
+Kubernetes, set `gravitino.job.executor=k8s` and configure the
+[Kubernetes job executor](../k8s-job-executor.md) instead. The examples on
this page are for the
+local job executor: on Kubernetes, leave `spark_master` empty, as the
Kubernetes job executor
+rejects `local[2]`. See
+[Built-In Jobs](../k8s-job-executor.md#built-in-jobs).
+
`gravitino.job.statusPullIntervalInMs` defaults to five minutes. Job status is
polled rather than pushed, so REST status can lag the real Spark process by a
full interval, which makes a working job look hung. Lower it to `10000` for
local work and restart the server.
### TMS cluster-static client auth
diff --git a/docs/table-maintenance-service/optimizer.md
b/docs/table-maintenance-service/optimizer.md
index eb4d651877..d121226286 100644
--- a/docs/table-maintenance-service/optimizer.md
+++ b/docs/table-maintenance-service/optimizer.md
@@ -84,7 +84,7 @@ Each step ends with a check. If a check fails, stop there,
since every step depe
### Prerequisites
- A running Gravitino server with a metalake. The examples use `test`.
-- Spark available to the job executor, through either `SPARK_HOME` or
`gravitino.jobExecutor.local.sparkHome`.
+- Spark available to the job executor. The [local job
executor](../local-job-executor.md) needs either `SPARK_HOME` or
`gravitino.jobExecutor.local.sparkHome`, and the [Kubernetes job
executor](../k8s-job-executor.md) needs a Spark image.
- An Iceberg Spark runtime on that Spark classpath. Built-in Iceberg templates
configure
`IcebergSparkSessionExtensions` and `SparkCatalog`, but `gravitino-jobs`
does not ship the
Iceberg Spark runtime and the templates leave `jars` empty so your Spark and
Iceberg versions
@@ -97,6 +97,12 @@ Each step ends with a check. If a check fails, stop there,
since every step depe
`org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.11.0`). Without it, the
job fails after
Spark starts with an error naming the missing Iceberg classes. See
[Troubleshooting](./optimizer-troubleshooting.md#job-execution-failures).
+- With the [Kubernetes job executor](../k8s-job-executor.md), the jobs run in
Spark pods: put the
+ Iceberg Spark runtime in the Spark image, in
`gravitino.jobExecutor.k8s.spark.conf.spark.jars`,
+ or in `spark.jars` of `spark_conf`; leave `spark_master` empty, as the
examples' `local[2]` is
+ rejected; and use catalog URIs, warehouse locations, and a `gravitino_uri`
in `updater_options`
+ that the pods can reach, not `localhost`. See
+ [Built-In Jobs](../k8s-job-executor.md#built-in-jobs).
- `gravitino.job.statusPullIntervalInMs` lowered to `10000` and the server
restarted. The default is five minutes, which makes every status check in this
walkthrough feel broken.
If your Iceberg REST backend runs in memory, do not restart it partway
through. Restarting resets both metadata and data files, and you start over.
diff --git
a/plugins/k8s-job-executor/src/main/java/org/apache/gravitino/job/k8s/K8sJobExecutor.java
b/plugins/k8s-job-executor/src/main/java/org/apache/gravitino/job/k8s/K8sJobExecutor.java
index 7e8e5cadcd..7961ae3143 100644
---
a/plugins/k8s-job-executor/src/main/java/org/apache/gravitino/job/k8s/K8sJobExecutor.java
+++
b/plugins/k8s-job-executor/src/main/java/org/apache/gravitino/job/k8s/K8sJobExecutor.java
@@ -185,6 +185,16 @@ public class K8sJobExecutor implements JobExecutor {
throw e;
}
if (REJECTED_REQUEST_CODES.contains(e.getCode())) {
+ // The client retries a request that timed out or failed with a server
error, so the
+ // rejection may answer a retry while an earlier attempt created the
SparkApplication, for
+ // example when that one used up a resource quota.
+ if (createdByEarlierAttempt(jobExecutionId, context.jobId(), e)) {
+ LOG.info(
+ "Submitted job {} as {}, although a retried request was
rejected",
+ context.jobId(),
+ jobExecutionId);
+ return jobExecutionId.toString();
+ }
// The cluster rejects the job as it is, for example its namespace
doesn't exist or isn't
// allowed, so the caller gets the reason rather than an internal
error.
throw new IllegalArgumentException(
@@ -572,6 +582,51 @@ public class K8sJobExecutor implements JobExecutor {
return failed;
}
+ /**
+ * Returns whether an earlier attempt of a rejected request created the
SparkApplication of a job,
+ * which then runs as submitted.
+ *
+ * @param id the job execution id of the SparkApplication
+ * @param jobId the id of the job
+ * @param rejection the rejection of the request
+ * @return whether the SparkApplication of the job exists
+ * @throws IllegalStateException if this can't be told, as the
SparkApplication of the job is
+ * being deleted, or reading it fails other than with a rejection
+ */
+ private boolean createdByEarlierAttempt(
+ K8sJobExecutionId id, long jobId, KubernetesClientException rejection) {
+ Optional<GenericKubernetesResource> existing;
+ try {
+ existing = getSparkApplication(id);
+ } catch (KubernetesClientException e) {
+ rejection.addSuppressed(e);
+ if (REJECTED_REQUEST_CODES.contains(e.getCode())) {
+ // The SparkApplication can't be read at all, so findSubmittedJob()
couldn't find it
+ // either, and keeping the job wouldn't help.
+ return false;
+ }
+ throw new IllegalStateException(
+ String.format(
+ "Kubernetes rejected the SparkApplication %s/%s of job %s, but
whether an earlier"
+ + " attempt of the request created it can't be told",
+ id.namespace(), id.name(), jobId),
+ rejection);
+ }
+ if (!existing.isPresent() || !K8sJobResourceUtils.isOfJob(existing.get(),
jobId)) {
+ return false;
+ }
+ if (existing.get().getMetadata().getDeletionTimestamp() != null) {
+ // Its pods may still run until the deletion finishes, and recovery
finds it until it's gone.
+ throw new IllegalStateException(
+ String.format(
+ "Kubernetes rejected the SparkApplication %s/%s of job %s, which
an earlier attempt"
+ + " of the request created, but it is being deleted",
+ id.namespace(), id.name(), jobId),
+ rejection);
+ }
+ return true;
+ }
+
/**
* Deletes the SparkApplication of a job whose submission failed without
telling whether it was
* created, if it was. Only a SparkApplication of this job is deleted, and
any failure is ignored:
diff --git
a/plugins/k8s-job-executor/src/test/java/org/apache/gravitino/job/k8s/TestK8sJobExecutor.java
b/plugins/k8s-job-executor/src/test/java/org/apache/gravitino/job/k8s/TestK8sJobExecutor.java
index b4fb5a575f..afa90f6be2 100644
---
a/plugins/k8s-job-executor/src/test/java/org/apache/gravitino/job/k8s/TestK8sJobExecutor.java
+++
b/plugins/k8s-job-executor/src/test/java/org/apache/gravitino/job/k8s/TestK8sJobExecutor.java
@@ -369,6 +369,124 @@ public class TestK8sJobExecutor {
KubernetesClientException.class, () -> executor.submitJob(context(1L),
sparkTemplate()));
}
+ @Test
+ public void testRejectedRetryOfCreatedSparkApplication() {
+ // An earlier attempt of the request created the SparkApplication, and the
retry was rejected,
+ // e.g. by a resource quota that the created one used up.
+ client
+ .genericKubernetesResources(SparkApplicationUtils.SPARK_APPLICATION)
+ .inNamespace("default")
+ .resource(
+ SparkApplicationUtils.buildSparkApplication(
+ context(1L),
+ sparkTemplate(),
+ TestSparkApplicationUtils.newConfigs(ImmutableMap.of())))
+ .create();
+ server
+ .expect()
+ .post()
+ .withPath(APPS_PATH)
+ .andReturn(403, new
StatusBuilder().withCode(403).withMessage("exceeded quota").build())
+ .once();
+
+ String jobExecutionId = executor.submitJob(context(1L), sparkTemplate());
+ Assertions.assertEquals(
+ Optional.of(jobExecutionId), executor.findSubmittedJob(context(1L),
sparkTemplate()));
+ Assertions.assertNotNull(sparkApplication("gravitino-job-1"));
+ }
+
+ @Test
+ public void testRejectedRetryOfSparkApplicationOfAnotherJob() {
+ GenericKubernetesResource app =
+ SparkApplicationUtils.buildSparkApplication(
+ context(1L), sparkTemplate(),
TestSparkApplicationUtils.newConfigs(ImmutableMap.of()));
+ app.getMetadata().getLabels().put(K8sJobResourceUtils.LABEL_JOB_ID, "3");
+ client
+ .genericKubernetesResources(SparkApplicationUtils.SPARK_APPLICATION)
+ .inNamespace("default")
+ .resource(app)
+ .create();
+ server
+ .expect()
+ .post()
+ .withPath(APPS_PATH)
+ .andReturn(403, new
StatusBuilder().withCode(403).withMessage("rejected 403").build())
+ .once();
+
+ IllegalArgumentException e =
+ Assertions.assertThrows(
+ IllegalArgumentException.class, () ->
executor.submitJob(context(1L), sparkTemplate()));
+ Assertions.assertTrue(e.getMessage().contains("rejected 403"),
e.getMessage());
+ }
+
+ @Test
+ public void testRejectedRetryOfSparkApplicationBeingDeleted() {
+ GenericKubernetesResource deleting =
+ SparkApplicationUtils.buildSparkApplication(
+ context(1L), sparkTemplate(),
TestSparkApplicationUtils.newConfigs(ImmutableMap.of()));
+
deleting.getMetadata().setFinalizers(ImmutableList.of("foregroundDeletion"));
+ client
+ .genericKubernetesResources(SparkApplicationUtils.SPARK_APPLICATION)
+ .inNamespace("default")
+ .resource(deleting)
+ .create();
+ GenericKubernetesResource created = sparkApplication("gravitino-job-1");
+ created.getMetadata().setDeletionTimestamp(Instant.now().toString());
+ client
+ .genericKubernetesResources(SparkApplicationUtils.SPARK_APPLICATION)
+ .inNamespace("default")
+ .resource(created)
+ .update();
+ server
+ .expect()
+ .post()
+ .withPath(APPS_PATH)
+ .andReturn(403, new StatusBuilder().withCode(403).build())
+ .once();
+
+ // Its pods may still run, so it is not a confirmed rejection, and
recovery still finds it.
+ IllegalStateException e =
+ Assertions.assertThrows(
+ IllegalStateException.class, () -> executor.submitJob(context(1L),
sparkTemplate()));
+ Assertions.assertEquals(403, ((KubernetesClientException)
e.getCause()).getCode());
+ Assertions.assertTrue(executor.findSubmittedJob(context(1L),
sparkTemplate()).isPresent());
+ }
+
+ @Test
+ public void testRejectedWhenLookupFails() {
+ server
+ .expect()
+ .post()
+ .withPath(APPS_PATH)
+ .andReturn(403, new StatusBuilder().withCode(403).build())
+ .always();
+ server
+ .expect()
+ .get()
+ .withPath(APPS_PATH + "/gravitino-job-1")
+ .andReturn(500, new StatusBuilder().withCode(500).build())
+ .once();
+ server
+ .expect()
+ .get()
+ .withPath(APPS_PATH + "/gravitino-job-1")
+ .andReturn(403, new StatusBuilder().withCode(403).build())
+ .once();
+
+ try (KubernetesClient noRetries =
clientWithoutRetries(client.getConfiguration())) {
+
executor.initialize(TestSparkApplicationUtils.newConfigs(ImmutableMap.of()),
noRetries);
+ // Whether an earlier attempt created the SparkApplication can't be told.
+ IllegalStateException unknown =
+ Assertions.assertThrows(
+ IllegalStateException.class, () ->
executor.submitJob(context(1L), sparkTemplate()));
+ Assertions.assertEquals(403, ((KubernetesClientException)
unknown.getCause()).getCode());
+
+ // A SparkApplication that can't be read at all couldn't be found by
recovery either.
+ Assertions.assertThrows(
+ IllegalArgumentException.class, () ->
executor.submitJob(context(1L), sparkTemplate()));
+ }
+ }
+
@Test
public void testFailJobWithoutStatus() {
String jobExecutionId = executor.submitJob(context(1L), sparkTemplate());