This is an automated email from the ASF dual-hosted git repository.
jerryshao pushed a commit to branch branch-1.3
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/branch-1.3 by this push:
new fc1789210e [Cherry-pick to branch-1.3] Docs: update the table
maintenance section (#12401) (#12432)
fc1789210e is described below
commit fc1789210e6f0e18c2585a5a3265844274dc68a3
Author: github-actions[bot]
<41898282+github-actions[bot]@users.noreply.github.com>
AuthorDate: Wed Aug 12 18:07:36 2026 +0800
[Cherry-pick to branch-1.3] Docs: update the table maintenance section
(#12401) (#12432)
**Cherry-pick Information:**
- Original commit: 2731083aa579df578f85f8a82f56c7f5e97b3a8c
- Target branch: `branch-1.3`
- Status: ✅ Clean cherry-pick (no conflicts)
Co-authored-by: Mark Hoerth <[email protected]>
Co-authored-by: Mark Hoerth <[email protected]>
---
.../optimizer-cli-reference.md | 148 +++++++--
.../optimizer-configuration.md | 111 ++++---
.../optimizer-extension-guide.md | 23 +-
.../optimizer-quick-start.md | 245 --------------
.../optimizer-troubleshooting.md | 128 +++-----
docs/table-maintenance-service/optimizer.md | 361 +++++++++++++++------
6 files changed, 501 insertions(+), 515 deletions(-)
diff --git a/docs/table-maintenance-service/optimizer-cli-reference.md
b/docs/table-maintenance-service/optimizer-cli-reference.md
index 855deb34fe..fcfa9130a0 100644
--- a/docs/table-maintenance-service/optimizer-cli-reference.md
+++ b/docs/table-maintenance-service/optimizer-cli-reference.md
@@ -1,7 +1,10 @@
---
-title: "Optimizer CLI Reference"
+title: "CLI Reference"
slug: "/table-maintenance-service/optimizer-cli-reference"
-keyword: "table maintenance, optimizer, cli, commands, metrics, statistics"
+keywords:
+ - table maintenance
+ - cli
+ - job template
license: "This software is licensed under the Apache License version 2."
---
@@ -24,7 +27,7 @@ directory. Use `--conf-path` only when you need a custom
config file.
| `list-job-metrics` | `--identifiers` | None | Query stored job metrics |
| `submit-update-stats-job` | `--identifiers` | `--dry-run`, `--update-mode`,
`--updater-options`, `--spark-conf` | Submit built-in Iceberg update
stats/metrics Spark jobs |
-### Option Field Meanings
+## Option Field Meanings
| Option | Meaning | Used by |
| --- | --- | --- |
@@ -50,7 +53,7 @@ Global option:
`local-stats-calculator` reads JSON Lines (one JSON object per line).
-### Reserved Fields
+## Reserved Fields
- `stats-type`: `table`, `partition`, or `job`
- `identifier`: object identifier
@@ -59,7 +62,7 @@ Global option:
All other fields are treated as metric or statistic values.
-### Supported Examples by Scope
+## Supported Examples by Scope
Use JSON Lines (one JSON object per line). The following examples focus on
table, partition, and
job scopes with multiple metric/statistic fields:
@@ -73,7 +76,7 @@ job scopes with multiple metric/statistic fields:
{"stats-type":"job","identifier":"job-1","timestamp":1735689800,"duration_ms":12500,"rewritten_files":18}
```
-### Identifier Rules
+## Identifier Rules
- Table and partition records: `catalog.schema.table`
- If `gravitino.optimizer.gravitinoDefaultCatalog` is set, `schema.table` is
also accepted
@@ -81,7 +84,7 @@ job scopes with multiple metric/statistic fields:
## CLI Workflow Examples
-### Batch Statistics Update
+## Batch Statistics Update
Calculate and persist table or partition statistics from JSONL input.
@@ -92,7 +95,7 @@ Calculate and persist table or partition statistics from
JSONL input.
--file-path ./table-stats.jsonl
```
-### Batch Metrics Append
+## Batch Metrics Append
Calculate and append table or job metrics from JSONL input.
@@ -103,7 +106,7 @@ Calculate and append table or job metrics from JSONL input.
--file-path ./table-stats.jsonl
```
-### Dry-Run Strategy Submission
+## Dry-Run Strategy Submission
Preview recommendations without actually submitting jobs.
@@ -116,7 +119,7 @@ Preview recommendations without actually submitting jobs.
--limit 10
```
-### Submit Strategy Jobs
+## Submit Strategy Jobs
Submit jobs for identifiers that match the given policy name.
@@ -128,7 +131,7 @@ Submit jobs for identifiers that match the given policy
name.
--limit 10
```
-### Monitor Metrics
+## Monitor Metrics
Evaluate monitor rules around an action time.
@@ -156,7 +159,7 @@ When metrics are produced by `submit-update-stats-job
--update-mode metrics`, me
often `custom-*` (for example `custom-data-file-mse`). Use
`list-table-metrics` first and
configure rules with the exact metric names returned by your environment.
-### Submit Built-In Update Stats Jobs
+## Submit Built-In Update Stats Jobs
Submit built-in Iceberg update stats/metrics Spark jobs directly.
@@ -179,7 +182,7 @@ Notes:
runtime classpath (for example via `spark.jars` in `--spark-conf`).
- `--spark-conf` and `--updater-options` are flat JSON maps.
-### List Table Metrics
+## List Table Metrics
Query stored metrics at table scope.
@@ -198,7 +201,7 @@ For partition scope, provide a partition path JSON array:
--partition-path '[{"dt":"2026-01-01"}]'
```
-### List Job Metrics
+## List Job Metrics
Query stored metrics at job scope.
@@ -229,10 +232,117 @@ MetricsResult{scopeType=TABLE,
identifier=rest_catalog.db.t1, partitionPath=<tab
EvaluationResult{scopeType=TABLE, identifier=rest_catalog.db.t1,
partitionPath=<table-or-job-scope>, evaluation=true,
evaluatorName=gravitino-metrics-evaluator, actionTimeSeconds=1735689600,
rangeSeconds=86400,
beforeMetrics={row_count=[MetricSample{timestampSeconds=1735686000,
value=120}]},
afterMetrics={row_count=[MetricSample{timestampSeconds=1735689600, value=100}]}}
```
+## Built-in Job Templates
+
+Three job templates ship with the service, and they are complementary rather
than alternatives. A full maintenance pass collects statistics, compacts data
files, and then expires the snapshot history that compaction just created.
+
+| Job template | What it does
|
+|---------------------------------------|-------------------------------------------|
+| `builtin-iceberg-update-stats` | Collects file statistics and metrics
|
+| `builtin-iceberg-rewrite-data-files` | Compacts small data files
|
+| `builtin-iceberg-expire-snapshots` | Removes old snapshot metadata
|
+
+Each can be submitted directly over REST, and the first two are also what the
policy-driven workflow submits on your behalf. See [Quick
Start](./optimizer.md#walkthrough) for the policy-driven path.
+
+## Update Statistics
+
+`builtin-iceberg-update-stats` reads a table and writes back the statistics
and metrics that policies evaluate. Compaction policies read
`custom-data-file-mse` and `custom-delete-file-number`, so nothing else will
fire until this job has run at least once.
+
+Its `jobConf` is documented in
[Configuration](./optimizer-configuration.md#job-submission-configuration).
+
+## Rewrite Data Files
+
+`builtin-iceberg-rewrite-data-files` performs the compaction itself, merging
small data files into larger ones. It is what a compaction policy submits when
its thresholds are crossed.
+
+In alpha this works only on Iceberg tables where every partition uses an
identity transform. Tables combining identity with a time or bucket transform
fail during the rewrite, which is covered in
[Troubleshooting](./optimizer-troubleshooting.md#job-execution-failures).
+
+For the policy that drives it, including threshold tuning, see [Iceberg
Compaction Policy](../iceberg-compaction-policy.md).
+
+## Expire Snapshots
+
+`builtin-iceberg-expire-snapshots` removes old Iceberg snapshots and the
metadata files behind them. Without periodic expiration, snapshot JSON files
and manifest lists accumulate indefinitely, which slows table operations and
wastes storage. Compaction makes this worse, since every rewrite creates a
snapshot.
+
+The job calls Iceberg's `expire_snapshots` stored procedure through Spark SQL.
+
+| Property | Value
|
+|-------------|-----------------------------------------------------------------------------|
+| Name | `builtin-iceberg-expire-snapshots`
|
+| Type | Spark
|
+| Version | `v1`
|
+| Main class |
`org.apache.gravitino.maintenance.jobs.iceberg.IcebergExpireSnapshotsJob` |
+
+## Parameters
+
+`catalog_name` and `table_identifier` are required. The rest are optional.
+
+| Key | Description
| Default |
+|------------------|-----------------------------------------------------------------------|-----------------------------|
+| `catalog_name` | Iceberg catalog name as registered in Spark
| Required |
+| `table_identifier` | Fully qualified table name, such as `db.sample`
| Required |
+| `older_than` | Expire snapshots older than this `yyyy-MM-dd HH:mm:ss`
timestamp | Five days ago |
+| `retain_last` | Minimum number of recent snapshots to keep regardless of
age | `1` |
+| `stream_results` | Streams intermediate delete results when present
| Disabled |
+| `spark_conf` | JSON map of Spark configuration
| None |
+
+`older_than` and `retain_last` work together, and `retain_last` wins. Setting
`older_than` to yesterday with `retain_last` at `5` keeps five snapshots even
if all five are older than yesterday.
+
+## Submitting the Job
+
+```bash
+curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "jobTemplateName": "builtin-iceberg-expire-snapshots",
+ "jobConf": {
+ "catalog_name": "rest_catalog",
+ "table_identifier": "db.t1",
+ "older_than": "2024-01-01 00:00:00",
+ "retain_last": "3",
+ "spark_master": "local[2]",
+ "spark_executor_instances": "1",
+ "spark_executor_cores": "1",
+ "spark_executor_memory": "1g",
+ "spark_driver_memory": "1g",
+ "catalog_type": "rest",
+ "catalog_uri": "http://localhost:9001/iceberg";,
+ "warehouse_location": ""
+ }
+ }' \
+ http://localhost:8090/api/metalakes/test/jobs
+```
+
+Omitting `older_than` and passing only `retain_last` is the safer default for
a first run, since it bounds the result by count rather than by a date you have
to reason about.
+
+The job builds this statement, including only the optional parameters you
supplied:
+
+```sql
+CALL `rest_catalog`.system.expire_snapshots(
+ table => 'db.t1',
+ older_than => TIMESTAMP '2024-01-01 00:00:00',
+ retain_last => 3,
+ stream_results => true
+)
+```
+
+## Verifying the Result
+
+```bash
+curl -sS "http://localhost:8090/api/metalakes/test/jobs/{job_id}"; | jq
'.job.state'
+cat
/tmp/gravitino/jobs/staging/test/builtin-iceberg-expire-snapshots/{job_id}/stdout.log
+```
+
+A successful run reports its state as `SUCCEEDED` and logs the counts it
removed:
+
+```text
+Expire Snapshots Results:
+ Deleted data files: 12
+ Deleted manifest files: 8
+ Deleted manifest lists: 3
+```
+
## Related
-- [Table Maintenance Service (Optimizer)](./optimizer.md)
-- [Optimizer Configuration](./optimizer-configuration.md)
-- [Optimizer Extension Guide](./optimizer-extension-guide.md)
-- [Optimizer Quick Start and Verification](./optimizer-quick-start.md)
-- [Optimizer Troubleshooting](./optimizer-troubleshooting.md)
+- [Table Maintenance Service](./optimizer.md) for the concepts and the
walkthrough
+- [Configuration](./optimizer-configuration.md) for the three configuration
layers
+- [Iceberg Compaction Policy](../iceberg-compaction-policy.md) for tuning the
built-in strategy
+- [Manage Jobs](../manage-jobs-in-gravitino.md) for job status and templates
diff --git a/docs/table-maintenance-service/optimizer-configuration.md
b/docs/table-maintenance-service/optimizer-configuration.md
index 28d3663fa5..409a0d70e5 100644
--- a/docs/table-maintenance-service/optimizer-configuration.md
+++ b/docs/table-maintenance-service/optimizer-configuration.md
@@ -1,23 +1,23 @@
---
-title: "Optimizer Configuration"
+title: "Configuration"
slug: "/table-maintenance-service/optimizer-configuration"
-keyword: "table maintenance, optimizer, configuration, job template, spark"
+keywords:
+ - table maintenance
+ - configuration
license: "This software is licensed under the Apache License version 2."
---
-## Configuration Layers
+Three layers of configuration apply, and they are set in different places for
different lifetimes. Server configuration governs how jobs run at all, CLI
configuration governs how the commands reach Gravitino, and `jobConf` governs a
single job submission.
-Use these layers together:
+| Layer | Where it lives | Lifetime |
+|---------------------|----------------------------------|---------------------|
+| Server | `gravitino.conf` | Until server
restart |
+| CLI | `conf/gravitino-optimizer.conf` | Per command
|
+| Job submission | `jobConf` in the request body | One job run
|
-| Layer | Scope | Typical keys |
-| --- | --- | --- |
-| Gravitino server config | Runtime for job manager and executor |
`gravitino.job.executor`, `gravitino.job.statusPullIntervalInMs`,
`gravitino.jobExecutor.local.sparkHome` |
-| Job submission `jobConf` | Per job run | `catalog_name`, `table_identifier`,
`spark_*`, template-specific args |
-| Optimizer CLI config | CLI commands | `gravitino.optimizer.*` in
`conf/gravitino-optimizer.conf` |
+## Server Configuration
-## Server-side Configuration
-
-Set server-level runtime behavior in `gravitino.conf`.
+Set these in `gravitino.conf`. They control the job executor rather than
maintenance itself, so they apply to every job Gravitino runs.
```properties
gravitino.job.executor=local
@@ -25,37 +25,11 @@ gravitino.job.statusPullIntervalInMs=300000
gravitino.jobExecutor.local.sparkHome=/path/to/spark
```
-For local demo environments, you can reduce
`gravitino.job.statusPullIntervalInMs` (for example
-`10000`) to get faster status updates. Restart Gravitino after changing this
value.
-
-## Built-In Update Stats `jobConf`
-
-Use `builtin-iceberg-update-stats` with at least these keys:
-
-```json
-{
- "catalog_name": "rest_catalog",
- "table_identifier": "db.t1",
- "update_mode": "all",
- "updater_options":
"{\"gravitino_uri\":\"http://localhost:8090\",\"metalake\":\"test\",\"statistics_updater\":\"gravitino-statistics-updater\",\"metrics_updater\":\"gravitino-metrics-updater\"}";,
- "spark_conf":
"{\"spark.master\":\"local[2]\",\"spark.hadoop.fs.defaultFS\":\"file:///\"}",
- "spark_master": "local[2]",
- "spark_executor_instances": "1",
- "spark_executor_cores": "1",
- "spark_executor_memory": "1g",
- "spark_driver_memory": "1g",
- "catalog_type": "rest",
- "catalog_uri": "http://localhost:9001/iceberg";,
- "warehouse_location": ""
-}
-```
-
-`warehouse_location` can be empty for local filesystem testing. Set it to your
warehouse URI
-for HDFS or cloud object storage environments.
+`gravitino.job.statusPullIntervalInMs` defaults to five minutes. Job status is
polled rather than pushed, so REST status can lag the real Spark process by a
full interval, which makes a working job look hung. Lower it to `10000` for
local work and restart the server.
-## Strategy Submission Configuration
+## CLI Configuration
-`submit-strategy-jobs` needs optimizer CLI config. This is a minimal working
example:
+The CLI needs to know where Gravitino is and which components to use. This is
a minimal working file for `submit-strategy-jobs`.
```properties
gravitino.optimizer.gravitinoUri = http://localhost:8090
@@ -74,34 +48,59 @@
gravitino.optimizer.jobSubmitterConfig.spark_executor_memory = 1g
gravitino.optimizer.jobSubmitterConfig.spark_driver_memory = 1g
gravitino.optimizer.jobSubmitterConfig.catalog_type = rest
gravitino.optimizer.jobSubmitterConfig.catalog_uri =
http://localhost:9001/iceberg
-# Leave empty for local filesystem; set to your warehouse URI for cloud/HDFS
storage.
+# Leave empty for a local filesystem; set to your warehouse URI for cloud or
HDFS storage.
gravitino.optimizer.jobSubmitterConfig.warehouse_location =
gravitino.optimizer.jobSubmitterConfig.spark_conf =
{"spark.master":"local[2]","spark.hadoop.fs.defaultFS":"file:///"}
```
-`--strategy-name` must be the policy name, for example
`iceberg_compaction_default`.
+Everything under `gravitino.optimizer.jobSubmitterConfig.` becomes the
`jobConf` of jobs this CLI submits, so the two layers carry the same keys under
different names.
+
+## Job Submission Configuration
+
+A direct job submission carries its own `jobConf`. This is
`builtin-iceberg-update-stats` with the keys it needs.
+
+```json
+{
+ "catalog_name": "rest_catalog",
+ "table_identifier": "db.t1",
+ "update_mode": "all",
+ "updater_options":
"{\"gravitino_uri\":\"http://localhost:8090\",\"metalake\":\"test\",\"statistics_updater\":\"gravitino-statistics-updater\",\"metrics_updater\":\"gravitino-metrics-updater\"}";,
+ "spark_conf":
"{\"spark.master\":\"local[2]\",\"spark.hadoop.fs.defaultFS\":\"file:///\"}",
+ "spark_master": "local[2]",
+ "spark_executor_instances": "1",
+ "spark_executor_cores": "1",
+ "spark_executor_memory": "1g",
+ "spark_driver_memory": "1g",
+ "catalog_type": "rest",
+ "catalog_uri": "http://localhost:9001/iceberg";,
+ "warehouse_location": ""
+}
+```
+
+`updater_options` and `spark_conf` are JSON strings inside a JSON object, so
their quotes are escaped. That nesting is the most common source of malformed
submissions.
+
+`warehouse_location` may be empty for local filesystem testing. Set it to the
warehouse URI for HDFS or cloud object storage.
-## Local Filesystem
+## Running Against a Local Filesystem
-If your environment is local and not HDFS-based, set:
+On a machine with no HDFS, Spark still defaults to `hdfs://localhost:9000` and
fails. Set the default filesystem explicitly, in `spark_conf` for job
submissions and in the CLI `spark_conf` value:
```properties
spark.hadoop.fs.defaultFS=file:///
```
-Without this, Spark jobs may try `hdfs://localhost:9000` and fail.
+## Checking Your Configuration
-## Validation Checklist
+Four things are worth confirming before assuming a configuration problem is a
code problem.
-- Job templates exist: `builtin-iceberg-update-stats`,
`builtin-iceberg-rewrite-data-files`.
-- Policies are attached to target tables.
-- `submit-strategy-jobs` prints `SUBMIT` lines.
-- Rewrite logs show `Rewritten data files: <N>` where `N > 0` for non-empty
tables.
+- `builtin-iceberg-update-stats` and `builtin-iceberg-rewrite-data-files`
appear in the job template list.
+- The policy is attached to the target table, not merely created.
+- `submit-strategy-jobs` prints `SUBMIT` lines rather than nothing.
+- The rewrite log shows `Rewritten data files: N` with `N` greater than zero
for a non-empty table.
## Related
-- [Table Maintenance Service (Optimizer)](./optimizer.md)
-- [Optimizer Extension Guide](./optimizer-extension-guide.md)
-- [Optimizer Quick Start and Verification](./optimizer-quick-start.md)
-- [Optimizer CLI Reference](./optimizer-cli-reference.md)
-- [Optimizer Troubleshooting](./optimizer-troubleshooting.md)
+- [Table Maintenance Service](./optimizer.md) for the concepts and the
walkthrough
+- [CLI Reference](./optimizer-cli-reference.md) for every command and the
built-in job templates
+- [Troubleshooting](./optimizer-troubleshooting.md) when a command or job fails
+- [Extension Guide](./optimizer-extension-guide.md) for custom strategies and
providers
diff --git a/docs/table-maintenance-service/optimizer-extension-guide.md
b/docs/table-maintenance-service/optimizer-extension-guide.md
index fbf7aa2fa2..9d33021cb1 100644
--- a/docs/table-maintenance-service/optimizer-extension-guide.md
+++ b/docs/table-maintenance-service/optimizer-extension-guide.md
@@ -1,17 +1,18 @@
---
-title: "Optimizer Extension Guide"
+title: "Extension Guide"
slug: "/table-maintenance-service/extension-guide"
-keyword: "table maintenance, optimizer, extension, provider, ServiceLoader"
+keywords:
+ - table maintenance
license: "This software is licensed under the Apache License version 2."
---
## Introduction
-Use this guide when built-in optimizer components do not match your
environment and you need custom implementations.
+Use this guide when the built-in components do not match your environment and
you need custom implementations.
## Extension Model
-Optimizer supports three loading patterns:
+The service supports three loading patterns:
1. `Provider` SPI (`name()` + `initialize()`): loaded by `ServiceLoader` and
selected by config value.
2. Class-name mapping for strategy handlers and job adapters.
@@ -48,7 +49,7 @@ public class MyStatisticsProvider implements
StatisticsProvider {
@Override
public void initialize(OptimizerEnv optimizerEnv) {
- // Initialize clients/resources from optimizer config.
+ // Initialize clients and resources from the configuration file.
}
@Override
@@ -110,8 +111,8 @@ Notes:
## Package and Deploy
- Build a JAR containing your classes and `META-INF/services` files.
-- Put the JAR on optimizer runtime classpath, for example
`${GRAVITINO_HOME}/optimizer/libs/`.
-- Restart optimizer process before testing.
+- Put the JAR on the runtime classpath, for example
`${GRAVITINO_HOME}/optimizer/libs/`.
+- Restart the process before testing.
If you also extend Gravitino server job execution, see [Manage jobs in
Gravitino](../manage-jobs-in-gravitino.md).
@@ -124,7 +125,7 @@ If you also extend Gravitino server job execution, see
[Manage jobs in Gravitino
## Related
-- [Table Maintenance Service (Optimizer)](./optimizer.md)
-- [Optimizer Configuration](./optimizer-configuration.md)
-- [Optimizer CLI Reference](./optimizer-cli-reference.md)
-- [Optimizer Troubleshooting](./optimizer-troubleshooting.md)
+- [Table Maintenance Service](./optimizer.md)
+- [Configuration](./optimizer-configuration.md)
+- [CLI Reference](./optimizer-configuration.md)
+- [Troubleshooting](./optimizer-troubleshooting.md)
diff --git a/docs/table-maintenance-service/optimizer-quick-start.md
b/docs/table-maintenance-service/optimizer-quick-start.md
deleted file mode 100644
index cb06483db8..0000000000
--- a/docs/table-maintenance-service/optimizer-quick-start.md
+++ /dev/null
@@ -1,245 +0,0 @@
----
-title: "Optimizer Quick Start"
-slug: "/table-maintenance-service/quick-start"
-keyword: "table maintenance, optimizer, quick start, compaction, update stats"
-license: "This software is licensed under the Apache License version 2."
----
-
-## Before Running Quick Start
-
-- Prepare a running Gravitino server.
-- Ensure target metalake exists (examples use `test`).
-- Configure `SPARK_HOME` or `gravitino.jobExecutor.local.sparkHome` for Spark
templates.
-- For faster status feedback during verification, set
`gravitino.job.statusPullIntervalInMs`
- to a smaller value (for example `10000`) and restart Gravitino.
-- If your Iceberg REST backend is in-memory, avoid restarting it during this
quick start because
- restart resets metadata and data files.
-
-For full config details, see [Optimizer
Configuration](./optimizer-configuration.md).
-
-## Success Criteria
-
-- Update-stats job finishes and table statistics/metrics include
`custom-data-file-mse` and
- `custom-delete-file-number`.
-- `submit-strategy-jobs` prints `SUBMIT` with a rewrite job ID.
-- Rewrite job log shows `Rewritten data files: <N>` where `N > 0` for
non-empty tables.
-
-## Quick Start a: Built-in Table Maintenance Workflow
-
-This workflow uses:
-
-- Built-in policy type: `system_iceberg_compaction`
-- Built-in update stats job template: `builtin-iceberg-update-stats`
-- Built-in rewrite data files job template:
`builtin-iceberg-rewrite-data-files`
-
-### Step 1: Preflight Checks
-
-```bash
-# Check metalake
-curl -sS "http://localhost:8090/api/metalakes/test"; | jq
-
-# Check built-in templates
-curl -sS
"http://localhost:8090/api/metalakes/test/jobs/templates?details=true"; | jq
'.jobTemplates[].name'
-```
-
-Expected names include:
-
-- `builtin-iceberg-update-stats`
-- `builtin-iceberg-rewrite-data-files`
-
-If missing, verify `gravitino-jobs` JAR in `auxlib`, then restart Gravitino.
-
-### Step 2: Prepare Demo Metadata Objects
-
-Create a REST Iceberg catalog, schema, and table:
-
-```bash
-# Create catalog (ignore "already exists" errors)
-curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
- -H "Content-Type: application/json" \
- -d '{
- "name": "rest_catalog",
- "type": "RELATIONAL",
- "comment": "Iceberg REST catalog",
- "provider": "lakehouse-iceberg",
- "properties": {
- "catalog-backend": "rest",
- "uri": "http://localhost:9001/iceberg";
- }
- }' \
- http://localhost:8090/api/metalakes/test/catalogs
-
-# Create schema
-curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
- -H "Content-Type: application/json" \
- -d '{
- "name": "db",
- "comment": "optimizer demo schema",
- "properties": {}
- }' \
- http://localhost:8090/api/metalakes/test/catalogs/rest_catalog/schemas
-
-# Create table
-curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
- -H "Content-Type: application/json" \
- -d '{
- "name": "t1",
- "comment": "optimizer demo table",
- "columns": [
- {"name": "id", "type": "integer", "nullable": true},
- {"name": "name", "type": "string", "nullable": true}
- ],
- "properties": {}
- }' \
-
http://localhost:8090/api/metalakes/test/catalogs/rest_catalog/schemas/db/tables
-```
-
-### Step 3: Seed Demo Data (Recommended)
-
-Use Spark SQL to create enough small files so compaction has visible effect:
-
-```bash
-${SPARK_HOME}/bin/spark-sql \
- --conf spark.hadoop.fs.defaultFS=file:/// \
- --conf spark.sql.catalog.rest_catalog=org.apache.iceberg.spark.SparkCatalog \
- --conf spark.sql.catalog.rest_catalog.type=rest \
- --conf spark.sql.catalog.rest_catalog.uri=http://localhost:9001/iceberg \
- -e "CREATE NAMESPACE IF NOT EXISTS rest_catalog.db; \
- SET spark.sql.files.maxRecordsPerFile=1000; \
- INSERT INTO rest_catalog.db.t1 \
- SELECT id, concat('name_', CAST(id AS STRING)) FROM range(0, 100000);"
-```
-
-### Step 4: Create and Attach Built-In Compaction Policy
-
-```bash
-# Create policy
-curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
- -H "Content-Type: application/json" \
- -d '{
- "name": "iceberg_compaction_default",
- "comment": "Built-in iceberg compaction policy",
- "policyType": "system_iceberg_compaction",
- "enabled": true,
- "content": {}
- }' \
- http://localhost:8090/api/metalakes/test/policies
-
-# Attach policy to table
-curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
- -H "Content-Type: application/json" \
- -d '{
- "policiesToAdd": ["iceberg_compaction_default"]
- }' \
-
http://localhost:8090/api/metalakes/test/objects/table/rest_catalog.db.t1/policies
-```
-
-Verify association:
-
-```bash
-curl -sS
"http://localhost:8090/api/metalakes/test/objects/table/rest_catalog.db.t1/policies?details=true";
| jq
-```
-
-### Step 5: Submit Built-In Update Stats Job
-
-```bash
-update_stats_job_id=$(curl -sS -X POST -H "Accept:
application/vnd.gravitino.v1+json" \
- -H "Content-Type: application/json" \
- -d '{
- "jobTemplateName": "builtin-iceberg-update-stats",
- "jobConf": {
- "catalog_name": "rest_catalog",
- "table_identifier": "db.t1",
- "update_mode": "all",
- "updater_options":
"{\"gravitino_uri\":\"http://localhost:8090\",\"metalake\":\"test\",\"statistics_updater\":\"gravitino-statistics-updater\",\"metrics_updater\":\"gravitino-metrics-updater\"}";,
- "spark_conf":
"{\"spark.master\":\"local[2]\",\"spark.hadoop.fs.defaultFS\":\"file:///\"}",
- "spark_master": "local[2]",
- "spark_executor_instances": "1",
- "spark_executor_cores": "1",
- "spark_executor_memory": "1g",
- "spark_driver_memory": "1g",
- "catalog_type": "rest",
- "catalog_uri": "http://localhost:9001/iceberg";,
- "warehouse_location": ""
- }
- }' \
- http://localhost:8090/api/metalakes/test/jobs/runs | jq -r '.job.jobId')
-
-echo "update-stats job id: ${update_stats_job_id}"
-```
-
-### Step 6: Trigger Rewrite Submission with `submit-strategy-jobs`
-
-```bash
-# Required optimizer CLI config for strategy submission.
-# Note: --strategy-name is policy name, not strategy.type.
-cat > /tmp/gravitino-optimizer-submit.conf <<'EOF_CONF'
-gravitino.optimizer.gravitinoUri = http://localhost:8090
-gravitino.optimizer.gravitinoMetalake = test
-gravitino.optimizer.gravitinoDefaultCatalog = rest_catalog
-gravitino.optimizer.recommender.statisticsProvider =
gravitino-statistics-provider
-gravitino.optimizer.recommender.strategyProvider = gravitino-strategy-provider
-gravitino.optimizer.recommender.tableMetaProvider =
gravitino-table-metadata-provider
-gravitino.optimizer.recommender.jobSubmitter = gravitino-job-submitter
-gravitino.optimizer.strategyHandler.iceberg-data-compaction.className =
org.apache.gravitino.maintenance.optimizer.recommender.handler.compaction.CompactionStrategyHandler
-gravitino.optimizer.jobSubmitterConfig.catalog_name = rest_catalog
-gravitino.optimizer.jobSubmitterConfig.spark_master = local[2]
-gravitino.optimizer.jobSubmitterConfig.spark_executor_instances = 1
-gravitino.optimizer.jobSubmitterConfig.spark_executor_cores = 1
-gravitino.optimizer.jobSubmitterConfig.spark_executor_memory = 1g
-gravitino.optimizer.jobSubmitterConfig.spark_driver_memory = 1g
-gravitino.optimizer.jobSubmitterConfig.catalog_type = rest
-gravitino.optimizer.jobSubmitterConfig.catalog_uri =
http://localhost:9001/iceberg
-# Leave empty for local filesystem; set to your warehouse URI for cloud/HDFS
storage.
-gravitino.optimizer.jobSubmitterConfig.warehouse_location =
-gravitino.optimizer.jobSubmitterConfig.spark_conf =
{"spark.master":"local[2]","spark.hadoop.fs.defaultFS":"file:///"}
-EOF_CONF
-
-# Optional: preview recommendations without submitting jobs.
-./bin/gravitino-optimizer.sh \
- --type submit-strategy-jobs \
- --identifiers rest_catalog.db.t1 \
- --strategy-name iceberg_compaction_default \
- --dry-run \
- --limit 10 \
- --conf-path /tmp/gravitino-optimizer-submit.conf
-
-# Submit rewrite job through strategy evaluation.
-submit_output=$(./bin/gravitino-optimizer.sh \
- --type submit-strategy-jobs \
- --identifiers rest_catalog.db.t1 \
- --strategy-name iceberg_compaction_default \
- --limit 10 \
- --conf-path /tmp/gravitino-optimizer-submit.conf)
-echo "${submit_output}"
-
-strategy_job_id=$(echo "${submit_output}" | sed -n
's/.*jobId=\([^[:space:]]*\).*/\1/p')
-[[ -z "${strategy_job_id}" ]] && echo 'ERROR: failed to extract strategy job
ID' && exit 1
-echo "strategy rewrite job id: ${strategy_job_id}"
-```
-
-### Step 7: Track Status and Verify Results
-
-```bash
-# Check job status by id
-curl -sS
"http://localhost:8090/api/metalakes/test/jobs/runs/${update_stats_job_id}"; | jq
-curl -sS
"http://localhost:8090/api/metalakes/test/jobs/runs/${strategy_job_id}"; | jq
-
-# Verify table statistics after update-stats
-curl -sS
"http://localhost:8090/api/metalakes/test/objects/table/rest_catalog.db.t1/statistics";
| jq
-
-# Staging path is controlled by `gravitino.job.stagingDir` (default:
`/tmp/gravitino/jobs/staging`).
-# Verify rewrite actually rewrote files (N should be > 0 for non-empty table).
-log_dir="/tmp/gravitino/jobs/staging/test/builtin-iceberg-rewrite-data-files/${strategy_job_id}"
-grep -E "Rewritten data files|Added data files|completed successfully" \
- "${log_dir}/output.log"
-```
-
-By default, Gravitino pulls job status every `300000` ms
(`gravitino.job.statusPullIntervalInMs`).
-REST status may lag real Spark process state by up to about 5 minutes.
-
-## Related
-
-- [Optimizer Configuration](./optimizer-configuration.md)
-- [Optimizer CLI Reference](./optimizer-cli-reference.md)
-- [Optimizer Troubleshooting](./optimizer-troubleshooting.md)
diff --git a/docs/table-maintenance-service/optimizer-troubleshooting.md
b/docs/table-maintenance-service/optimizer-troubleshooting.md
index 13332336b6..3a2111d69d 100644
--- a/docs/table-maintenance-service/optimizer-troubleshooting.md
+++ b/docs/table-maintenance-service/optimizer-troubleshooting.md
@@ -1,124 +1,96 @@
---
-title: "Optimizer Troubleshooting"
+title: "Troubleshooting"
slug: "/table-maintenance-service/troubleshooting"
-keyword: "table maintenance, optimizer, troubleshooting, spark, strategy"
+keywords:
+ - table maintenance
+ - troubleshooting
+ - spark
license: "This software is licensed under the Apache License version 2."
---
-## `Invalid --type`
+## Overview
-Use kebab-case values such as `update-statistics`, not `update_statistics`.
+Failures fall into three groups, matching where they occur in the workflow.
Command and argument errors surface immediately. Evaluation problems produce no
output rather than an error, which is what makes them confusing. Execution
failures happen inside Spark, so the real message is in the staging log rather
than the API response.
-## `--statistics-payload and --file-path cannot be used together`
+Staging logs live under
`/tmp/gravitino/jobs/staging/{metalake}/{job_template_name}/{job_id}/`,
controlled by `gravitino.job.stagingDir`. Read `error.log` for failures and
`output.log` for results.
-For `local-stats-calculator`, use exactly one of them.
+## Command and Argument Errors
-## `requires one of --statistics-payload or --file-path`
+These come back from the CLI immediately and name the problem.
-When `--calculator-name local-stats-calculator` is used, one input source is
required.
+**`Invalid --type`** — command names are kebab-case. Use `update-statistics`,
not `update_statistics`.
-## `--partition-path must be a JSON array`
+**`--statistics-payload and --file-path cannot be used together`** —
`local-stats-calculator` takes exactly one input source.
-Use a JSON array format, for example:
+**`requires one of --statistics-payload or --file-path`** — the same rule from
the other side. With `--calculator-name local-stats-calculator`, one of the two
is mandatory.
+
+**`--partition-path must be a JSON array`** — even for a single partition,
pass an array:
```text
[{"dt":"2026-01-01"}]
```
-## Job Status Appears Stale (`queued` or `started` for a Long Time)
+**`Specified optimizer config file does not exist`** — check the `--conf-path`
value and the file's permissions.
-Check `gravitino.job.statusPullIntervalInMs` and local staging logs under:
+**`No StrategyHandler class configured for strategy type ...`** — the strategy
handler mapping is missing from the CLI configuration:
-`/tmp/gravitino/jobs/staging/<metalake>/<job-template-name>/<job-id>/error.log`.
+```properties
+gravitino.optimizer.strategyHandler.iceberg-data-compaction.className =
org.apache.gravitino.maintenance.optimizer.recommender.handler.compaction.CompactionStrategyHandler
+```
-For local verification, reduce `gravitino.job.statusPullIntervalInMs` (for
example `10000`) and
-restart Gravitino so REST status can refresh faster.
+The packaged default configuration already contains this, so seeing it usually
means a hand-written config file.
-## `No identifiers matched strategy name ...`
+## Evaluation Produces Nothing
-`--strategy-name` must be the policy name (for example
`iceberg_compaction_default`), not the policy type
(`system_iceberg_compaction`) and not the strategy type
(`iceberg-data-compaction`).
+These are the hard ones, because success and "the policy decided not to act"
look identical.
-## Dry-Run Returns No `DRY-RUN` or `SUBMIT` Lines
+**`No identifiers matched strategy name ...`** — `--strategy-name` takes the
policy name, for example `iceberg_compaction_default`. It does not take the
policy type `system_iceberg_compaction` or the strategy type
`iceberg-data-compaction`, despite being called strategy name.
-This usually means trigger conditions are not met. For compaction, verify
-`custom-data-file-mse` and `custom-delete-file-number` in table
statistics/metrics are large
-enough to satisfy policy rules.
+**A dry run prints no `DRY-RUN` or `SUBMIT` lines** — the trigger conditions
were not met. For compaction, check that `custom-data-file-mse` and
`custom-delete-file-number` in the table's statistics are large enough to
satisfy the policy rules. A table with too few small files is the usual cause,
and the fix is more data rather than more configuration.
-## `monitor-metrics` Returns `evaluation=false` Unexpectedly
+**`monitor-metrics` returns `evaluation=false` unexpectedly** — check the rule
names and the sample window together:
-Check both rule names and metric samples:
+1. Query the current metrics with `list-table-metrics`, adding
`--partition-path` for partition scope.
+2. Use the exact metric names your environment returns in
`gravitino.optimizer.monitor.gravitinoMetricsEvaluator.rules`. Names that look
close enough are not.
+3. Make sure `--action-time` falls inside a range where both a before and an
after sample exist.
-1. Query current metrics first with `list-table-metrics` (and
`--partition-path` for partition scope).
-2. Use the exact metric names returned by your environment in
- `gravitino.optimizer.monitor.gravitinoMetricsEvaluator.rules`.
-3. Ensure `--action-time` is inside the range where both before and after
samples exist.
+## Job Execution Failures
-## `No StrategyHandler class configured for strategy type ...`
+**Status stays `queued` or `started` for a long time** — REST status is
polled, not pushed, and `gravitino.job.statusPullIntervalInMs` defaults to five
minutes. Lower it to `10000` and restart the server for local work. If the
status is genuinely stuck rather than lagging, read `error.log` in the staging
directory.
-Add strategy handler mapping to optimizer config, for example:
+**Spark fails with `hdfs://localhost:9000` or other filesystem errors** —
Spark is defaulting to HDFS on a machine that has none:
```properties
-gravitino.optimizer.strategyHandler.iceberg-data-compaction.className =
org.apache.gravitino.maintenance.optimizer.recommender.handler.compaction.CompactionStrategyHandler
+spark.hadoop.fs.defaultFS=file:///
```
-If you already use the packaged default optimizer config, this mapping may
already exist.
-
-## Spark Job Fails with `hdfs://localhost:9000` or Filesystem Errors
-
-Set local filesystem explicitly in Spark config:
+**`submit-update-stats-job` fails with JDBC metrics errors** — when
`--updater-options` includes `gravitino.optimizer.jdbcMetrics.*`, the JDBC
driver has to be on the Spark runtime classpath. `ClassNotFoundException` and
`No suitable driver` both mean the same thing:
-```properties
-spark.hadoop.fs.defaultFS=file:///
+```json
+{
+ "spark.jars": "/path/to/postgresql-42.7.4.jar"
+}
```
-## Rewrite Fails on Multi-level Partition (`identity + day(...)`)
-
-In release `1.2.0`, rewrite may fail for partition filters combining identity
and day transform
-(for example `PARTITIONED BY (p, days(ts))`) with error:
+**Rewrite fails on a multi-level partition** — in release `1.2.0`, rewriting a
table partitioned by an identity transform combined with a time transform, such
as `PARTITIONED BY (p, days(ts))`, fails with:
```text
Cannot translate Spark expression ... day(cast(ts as date)) ... to data source
filter
```
-How to verify:
-
-1. Check job run status by rewrite job id under
- `/api/metalakes/<metalake>/jobs/runs/<job-id>`.
-2. Check staging log:
-
`/tmp/gravitino/jobs/staging/<metalake>/builtin-iceberg-rewrite-data-files/<job-id>/error.log`.
-
-Workaround:
-
-- Use identity-only partition compaction path for release `1.2.0`.
-- Keep this failure case as a reproducible regression test for later fix
validation.
-
-Observed compatibility matrix in release `1.2.0` (rewrite path):
-
-- PASS: `p`, `p, c2` (identity-only partition transforms)
-- FAIL: `p, years(ts)`, `p, months(ts)`, `p, days(ts)`, `p, hours(ts)`,
- `p, truncate(1, c2)`, `p, bucket(8, id)`
-
-## `submit-update-stats-job` Fails with JDBC Metrics Errors
-
-When `--updater-options` includes `gravitino.optimizer.jdbcMetrics.*`, ensure
the JDBC driver is
-available to Spark runtime classpath. Typical failures include
`ClassNotFoundException` for driver
-class or `No suitable driver`.
-
-Example in `--spark-conf`:
-
-```json
-{
- "spark.jars": "/path/to/postgresql-42.7.4.jar"
-}
-```
+Confirm it by checking the job run at
`/api/metalakes/{metalake}/jobs/runs/{job_id}` and reading `error.log` under
`builtin-iceberg-rewrite-data-files`. The only workaround is to compact
identity-partitioned tables and leave the rest alone.
-## `Specified optimizer config file does not exist`
+Observed in `1.2.0`:
-Check your `--conf-path` and file permissions.
+| Partitioning
| Rewrite |
+|-------------------------------------------------------------------------------|---------|
+| `p`, `p, c2`
| Works |
+| `p, years(ts)`, `p, months(ts)`, `p, days(ts)`, `p, hours(ts)`
| Fails |
+| `p, truncate(1, c2)`, `p, bucket(8, id)`
| Fails |
## Related
-- [Table Maintenance Service (Optimizer)](./optimizer.md)
-- [Optimizer Configuration](./optimizer-configuration.md)
-- [Optimizer Quick Start and Verification](./optimizer-quick-start.md)
-- [Optimizer CLI Reference](./optimizer-cli-reference.md)
+- [Table Maintenance Service](./optimizer.md)
+- [Configuration](./optimizer-configuration.md)
+- [Quick Start](./optimizer.md#walkthrough)
+- [CLI Reference](./optimizer-configuration.md)
diff --git a/docs/table-maintenance-service/optimizer.md
b/docs/table-maintenance-service/optimizer.md
index 08566001c4..22c3362528 100644
--- a/docs/table-maintenance-service/optimizer.md
+++ b/docs/table-maintenance-service/optimizer.md
@@ -1,134 +1,283 @@
---
-title: "Table Maintenance Service (Optimizer)"
+title: "Table Maintenance Service"
slug: "/table-maintenance-service/optimizer"
-keyword: "table maintenance, optimizer, statistics, metrics, monitor"
+keywords:
+ - table maintenance
+ - compaction
+ - statistics
+ - quick start
license: "This software is licensed under the Apache License version 2."
---
## Overview
-The Table Maintenance Service (Optimizer) automates table maintenance by
connecting:
+The table maintenance service keeps tables healthy without anyone watching
them. You attach a policy to a catalog, schema, or table; the service collects
statistics, evaluates them against that policy, and submits a job when the
policy says work is needed.
-- Statistics and metrics collection
-- Rule evaluation and strategy recommendation
-- Job template based execution
+The framework is generic. Metrics collection, policy evaluation, and job
submission are not tied to any particular table format, and each is a Java
ServiceLoader extension point. What ships built in is deliberately narrower,
and in alpha that means Iceberg data file compaction on identity-partitioned
tables.
-The CLI commands and configuration keys use the `optimizer` name.
+The CLI binary, its configuration file, and its configuration keys carry the
older name `optimizer`, so you will see `gravitino-optimizer.sh`,
`gravitino-optimizer.conf`, and `gravitino.optimizer.*` throughout. Those are
literal strings rather than a second product.
-## Alpha Status and Limitations
+## Alpha Scope
-The Table Maintenance Service is in **alpha** stage.
+Confirm your environment matches this list before starting an evaluation
against the built-ins. Anything outside it needs a custom extension, which is
covered in the [Extension Guide](./optimizer-extension-guide.md).
-Limitations:
+- Compaction is the only built-in strategy. There is no built-in snapshot
expiration, orphan file cleanup, or sort and cluster maintenance.
+- Compaction applies to Iceberg tables only, and only where every partition
uses an identity transform.
+- The service is driven through the CLI workflow rather than running on a
schedule of its own.
-- It is operated through the optimizer CLI workflow.
-- The built-in maintenance strategy focuses on Iceberg table compaction.
-- Compaction support is limited to Iceberg tables with identity partition
transforms.
-## Extensibility and Roadmap
+## How It Works
-Although the built-in capability is intentionally narrow in alpha, the
framework is designed for
-extension:
+Maintenance runs as four steps. Each is a separate command, so you can stop
after any of them, and the dry run on step two shows what would be submitted
before anything runs.
-- Integrate external systems by implementing custom providers and adapters.
-- Add new strategies and handlers beyond built-in compaction.
-- Plug in custom metrics, evaluators, and job submitters for different
environments.
-
-See [Optimizer Extension Guide](./optimizer-extension-guide.md) for extension
points.
-
-Future versions will continue improving the out-of-the-box experience and
evolve toward a more
-ready-to-use maintenance service.
-
-## Architecture Overview
-
-The optimizer workflow is based on six parts:
-
-1. Metadata objects: catalog/schema/table in a metalake.
-2. Statistics and metrics: table/partition signals used for decision making.
-3. Policies: strategy intent, for example `system_iceberg_compaction`.
-4. Job templates: executable contracts, for example built-in Spark templates.
-5. Job executor: local or custom backend that runs submitted jobs.
-6. Status and logs: REST job state plus local staging logs.
-
-
-
-The following diagram shows the end-to-end interactions between CLI, Gravitino
server, Spark jobs,
-JDBC metrics repository, and the Recommender/Updater/Monitor modules.
-
-Typical data flow:
-
-1. Collect statistics and metrics for target tables.
-2. Evaluate rules and produce candidate actions.
-3. Submit jobs using a concrete template and `jobConf`.
-4. Track status and verify results on table metadata and logs.
+| Step | What you run | What it
produces |
+|----------|---------------------------------------------------|---------------------------------------------------------|
+| Collect | `update-statistics`, `append-metrics` | Statistics on
the table, metrics in the JDBC repository |
+| Evaluate | `submit-strategy-jobs --dry-run` | Candidate
actions, with nothing submitted |
+| Submit | `submit-strategy-jobs`, `submit-update-stats-job` | A Spark job,
tracked by job status and staging logs |
+| Observe | `monitor-metrics`, `list-table-metrics` | Before and
after metrics, and rewritten data files |
## Execution Modes
-| Mode | Main entry | Best for | Output |
-| --- | --- | --- | --- |
-| Built-in maintenance workflow | Gravitino REST + built-in templates |
Server-side operational runs | Submitted Spark jobs and updated metadata |
-| Optimizer CLI local calculator | `gravitino-optimizer.sh` | Local
file-driven testing and batch scripts | Statistics/metrics updates and optional
submissions |
-
-Use built-in maintenance workflow when you want policy-driven server execution.
-Use CLI local calculator when you want to feed JSONL input directly.
-
-## Start Here
-
-- Configuration first: read [Optimizer
Configuration](./optimizer-configuration.md).
-- Need custom integrations: read [Optimizer Extension
Guide](./optimizer-extension-guide.md).
-- First-time enablement: run [Optimizer Quick Start and
Verification](./optimizer-quick-start.md).
-- CLI-only usage: read [Optimizer CLI Reference](./optimizer-cli-reference.md).
-- Runtime failures or mismatched results: check [Optimizer
Troubleshooting](./optimizer-troubleshooting.md).
-
-## Lifecycle
-
-### Step 1: Collect
-
-Generate or ingest table and partition statistics/metrics.
-
-### Step 2: Evaluate
-
-Apply policies and rules to decide whether maintenance should run.
-
-### Step 3: Submit
-
-Pick a job template and submit job with concrete `jobConf`.
-
-### Step 4: Observe
-
-Check REST job status and validate resulting statistics, metrics, or rewritten
data files.
-
-## Configuration Model
+There are two ways in, and they differ in where the numbers come from rather
than in what they do.
-| Layer | Scope | Typical keys |
-| --- | --- | --- |
-| Gravitino server config | Runtime for job manager and executor |
`gravitino.job.executor`, `gravitino.job.statusPullIntervalInMs`,
`gravitino.jobExecutor.local.sparkHome` |
-| Job submission `jobConf` | Per job run | `catalog_name`, `table_identifier`,
`spark_*`, template-specific args |
-| Optimizer CLI config | CLI commands | `gravitino.optimizer.*` in
`conf/gravitino-optimizer.conf` |
+The built-in workflow drives everything through the Gravitino server and its
job templates, using the policy attached to a table to decide what runs. Use it
for server-side operational runs.
-## Terminology Mapping
+The local calculator reads a JSONL file you supply and updates statistics and
metrics directly from it. Use it for testing and batch scripts, where you
already have the numbers and want to feed them in without the server computing
them.
-| Term | Example value | Used in |
-| --- | --- | --- |
-| Policy name | `iceberg_compaction_default` | Policy identity and CLI
`--strategy-name` |
-| Policy type | `system_iceberg_compaction` | REST policy creation field
`policyType` |
-| Strategy type | `iceberg-data-compaction` | Policy content field
`strategy.type` and strategy handler config key |
+## Naming
-For strategy submission, `--strategy-name` must use policy name, not policy
type or strategy type.
+Three identifiers look interchangeable and are not.
-## Prerequisites and Verification
+| Term | Example | Where it appears
|
+|---------------|------------------------------|------------------------------------------------------------|
+| Policy name | `iceberg_compaction_default` | The policy's own name, and
the CLI `--strategy-name` |
+| Policy type | `system_iceberg_compaction` | The `policyType` field when
creating a policy over REST |
+| Strategy type | `iceberg-data-compaction` | The `strategy.type` field,
and the strategy handler config |
+
+`--strategy-name` takes the **policy name**, despite what it is called.
Passing either of the other two reports no matching identifiers rather than
naming the mistake.
+
+## Walkthrough
+
+This takes one Iceberg table through the whole workflow: create it, fill it
with small files, attach a compaction policy, collect statistics, and let the
service decide to compact it. It runs against a local Spark and takes about
fifteen minutes.
+
+Each step ends with a check. If a check fails, stop there, since every step
depends on the one before it.
+
+### Prerequisites
+
+- A running Gravitino server with a metalake. The examples use `test`.
+- Spark available to the job executor, through either `SPARK_HOME` or
`gravitino.jobExecutor.local.sparkHome`.
+- `gravitino.job.statusPullIntervalInMs` lowered to `10000` and the server
restarted. The default is five minutes, which makes every status check in this
walkthrough feel broken.
-Quick start prerequisites and success checks are documented in
-[Optimizer Quick Start and Verification](./optimizer-quick-start.md).
+If your Iceberg REST backend runs in memory, do not restart it partway
through. Restarting resets both metadata and data files, and you start over.
+
+### Step 1: Confirm the Job Templates Exist
+
+```bash
+curl -sS "http://localhost:8090/api/metalakes/test"; | jq
+curl -sS
"http://localhost:8090/api/metalakes/test/jobs/templates?details=true"; \
+ | jq '.jobTemplates[].name'
+```
+
+The template list must include `builtin-iceberg-update-stats` and
`builtin-iceberg-rewrite-data-files`. If it does not, the `gravitino-jobs` JAR
is missing from `auxlib`. Add it and restart the server before going on.
+
+### Step 2: Create the Demo Catalog, Schema, and Table
+
+```bash
+# Catalog. An "already exists" error here is fine.
+curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "name": "rest_catalog",
+ "type": "RELATIONAL",
+ "comment": "Iceberg REST catalog",
+ "provider": "lakehouse-iceberg",
+ "properties": {
+ "catalog-backend": "rest",
+ "uri": "http://localhost:9001/iceberg";
+ }
+ }' \
+ http://localhost:8090/api/metalakes/test/catalogs
+
+# Schema
+curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" \
+ -d '{"name": "db", "comment": "maintenance demo schema", "properties": {}}' \
+ http://localhost:8090/api/metalakes/test/catalogs/rest_catalog/schemas
+
+# Table
+curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "name": "t1",
+ "comment": "maintenance demo table",
+ "columns": [
+ {"name": "id", "type": "integer", "nullable": true},
+ {"name": "name", "type": "string", "nullable": true}
+ ],
+ "properties": {}
+ }' \
+
http://localhost:8090/api/metalakes/test/catalogs/rest_catalog/schemas/db/tables
+```
+
+### Step 3: Create Something Worth Compacting
+
+An empty table gives the policy nothing to react to, so write 100,000 rows
capped at 1,000 rows per file. That produces the many small files compaction
exists to merge.
+
+```bash
+${SPARK_HOME}/bin/spark-sql \
+ --conf spark.hadoop.fs.defaultFS=file:/// \
+ --conf spark.sql.catalog.rest_catalog=org.apache.iceberg.spark.SparkCatalog \
+ --conf spark.sql.catalog.rest_catalog.type=rest \
+ --conf spark.sql.catalog.rest_catalog.uri=http://localhost:9001/iceberg \
+ -e "CREATE NAMESPACE IF NOT EXISTS rest_catalog.db; \
+ SET spark.sql.files.maxRecordsPerFile=1000; \
+ INSERT INTO rest_catalog.db.t1 \
+ SELECT id, concat('name_', CAST(id AS STRING)) FROM range(0, 100000);"
+```
+
+Without `spark.hadoop.fs.defaultFS=file:///`, Spark reaches for
`hdfs://localhost:9000` and fails.
+
+### Step 4: Attach a Compaction Policy
+
+Creating the policy is not enough. It has to be attached to the table, and the
attachment is what the service reads.
+
+```bash
+curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "name": "iceberg_compaction_default",
+ "comment": "Built-in Iceberg compaction policy",
+ "policyType": "system_iceberg_compaction",
+ "enabled": true,
+ "content": {}
+ }' \
+ http://localhost:8090/api/metalakes/test/policies
+
+curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" \
+ -d '{"policiesToAdd": ["iceberg_compaction_default"]}' \
+
http://localhost:8090/api/metalakes/test/objects/table/rest_catalog.db.t1/policies
+```
+
+Confirm the attachment before moving on:
+
+```bash
+curl -sS
"http://localhost:8090/api/metalakes/test/objects/table/rest_catalog.db.t1/policies?details=true";
| jq
+```
+
+### Step 5: Collect Statistics
+
+```bash
+update_stats_job_id=$(curl -sS -X POST -H "Accept:
application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "jobTemplateName": "builtin-iceberg-update-stats",
+ "jobConf": {
+ "catalog_name": "rest_catalog",
+ "table_identifier": "db.t1",
+ "update_mode": "all",
+ "updater_options":
"{\"gravitino_uri\":\"http://localhost:8090\",\"metalake\":\"test\",\"statistics_updater\":\"gravitino-statistics-updater\",\"metrics_updater\":\"gravitino-metrics-updater\"}";,
+ "spark_conf":
"{\"spark.master\":\"local[2]\",\"spark.hadoop.fs.defaultFS\":\"file:///\"}",
+ "spark_master": "local[2]",
+ "spark_executor_instances": "1",
+ "spark_executor_cores": "1",
+ "spark_executor_memory": "1g",
+ "spark_driver_memory": "1g",
+ "catalog_type": "rest",
+ "catalog_uri": "http://localhost:9001/iceberg";,
+ "warehouse_location": ""
+ }
+ }' \
+ http://localhost:8090/api/metalakes/test/jobs/runs | jq -r '.job.jobId')
+
+echo "update-stats job id: ${update_stats_job_id}"
+```
+
+Wait for it to finish, then confirm the statistics landed:
+
+```bash
+curl -sS
"http://localhost:8090/api/metalakes/test/objects/table/rest_catalog.db.t1/statistics";
| jq
+```
+
+The response must include `custom-data-file-mse` and
`custom-delete-file-number`. Those two are what the compaction policy
evaluates, so if they are absent the next step has nothing to decide on.
+
+### Step 6: Evaluate and Submit
+
+Write the CLI configuration first. `--strategy-name` takes the **policy
name**, not the policy type or the strategy type.
+
+```bash
+cat > /tmp/gravitino-optimizer-submit.conf <<'EOF_CONF'
+gravitino.optimizer.gravitinoUri = http://localhost:8090
+gravitino.optimizer.gravitinoMetalake = test
+gravitino.optimizer.gravitinoDefaultCatalog = rest_catalog
+gravitino.optimizer.recommender.statisticsProvider =
gravitino-statistics-provider
+gravitino.optimizer.recommender.strategyProvider = gravitino-strategy-provider
+gravitino.optimizer.recommender.tableMetaProvider =
gravitino-table-metadata-provider
+gravitino.optimizer.recommender.jobSubmitter = gravitino-job-submitter
+gravitino.optimizer.strategyHandler.iceberg-data-compaction.className =
org.apache.gravitino.maintenance.optimizer.recommender.handler.compaction.CompactionStrategyHandler
+gravitino.optimizer.jobSubmitterConfig.catalog_name = rest_catalog
+gravitino.optimizer.jobSubmitterConfig.spark_master = local[2]
+gravitino.optimizer.jobSubmitterConfig.spark_executor_instances = 1
+gravitino.optimizer.jobSubmitterConfig.spark_executor_cores = 1
+gravitino.optimizer.jobSubmitterConfig.spark_executor_memory = 1g
+gravitino.optimizer.jobSubmitterConfig.spark_driver_memory = 1g
+gravitino.optimizer.jobSubmitterConfig.catalog_type = rest
+gravitino.optimizer.jobSubmitterConfig.catalog_uri =
http://localhost:9001/iceberg
+# Leave empty for a local filesystem; set to your warehouse URI for cloud or
HDFS storage.
+gravitino.optimizer.jobSubmitterConfig.warehouse_location =
+gravitino.optimizer.jobSubmitterConfig.spark_conf =
{"spark.master":"local[2]","spark.hadoop.fs.defaultFS":"file:///"}
+EOF_CONF
+```
+
+Preview first. A dry run evaluates the policy and prints what it would do
without submitting anything.
+
+```bash
+./bin/gravitino-optimizer.sh \
+ --type submit-strategy-jobs \
+ --identifiers rest_catalog.db.t1 \
+ --strategy-name iceberg_compaction_default \
+ --dry-run \
+ --limit 10 \
+ --conf-path /tmp/gravitino-optimizer-submit.conf
+```
+
+`DRY-RUN` lines mean the policy fired. No output at all means it did not,
which usually means the statistics from step 5 are below the policy thresholds
rather than that anything is broken.
+
+Then submit for real:
+
+```bash
+submit_output=$(./bin/gravitino-optimizer.sh \
+ --type submit-strategy-jobs \
+ --identifiers rest_catalog.db.t1 \
+ --strategy-name iceberg_compaction_default \
+ --limit 10 \
+ --conf-path /tmp/gravitino-optimizer-submit.conf)
+echo "${submit_output}"
+
+strategy_job_id=$(echo "${submit_output}" | sed -n
's/.*jobId=\([^[:space:]]*\).*/\1/p')
+[[ -z "${strategy_job_id}" ]] && echo 'ERROR: failed to extract strategy job
ID' && exit 1
+echo "strategy rewrite job id: ${strategy_job_id}"
+```
+
+### Step 7: Verify the Rewrite
+
+```bash
+curl -sS
"http://localhost:8090/api/metalakes/test/jobs/runs/${update_stats_job_id}"; | jq
+curl -sS
"http://localhost:8090/api/metalakes/test/jobs/runs/${strategy_job_id}"; | jq
+
+log_dir="/tmp/gravitino/jobs/staging/test/builtin-iceberg-rewrite-data-files/${strategy_job_id}"
+grep -E "Rewritten data files|Added data files|completed successfully"
"${log_dir}/output.log"
+```
+
+`Rewritten data files: N` with `N` greater than zero means the workflow worked
end to end. The staging path comes from `gravitino.job.stagingDir`, which
defaults to `/tmp/gravitino/jobs/staging`.
+
+REST job status is polled rather than pushed, so it lags the real Spark
process by up to one poll interval. That is why the prerequisites lower it to
ten seconds.
## Related
-- [Optimizer Configuration](./optimizer-configuration.md)
-- [Optimizer Extension Guide](./optimizer-extension-guide.md)
-- [Optimizer Quick Start and Verification](./optimizer-quick-start.md)
-- [Optimizer CLI Reference](./optimizer-cli-reference.md)
-- [Optimizer Troubleshooting](./optimizer-troubleshooting.md)
-- [Manage policies in Gravitino](../manage-policies-in-gravitino.md)
-- [Iceberg compaction policy](../iceberg-compaction-policy.md)
-- [Manage jobs in Gravitino](../manage-jobs-in-gravitino.md)
-- [Manage statistics in Gravitino](../manage-statistics-in-gravitino.md)
+- [Configuration](./optimizer-configuration.md) for the three configuration
layers
+- [CLI Reference](./optimizer-cli-reference.md) for every command and the
built-in job templates
+- [Troubleshooting](./optimizer-troubleshooting.md) when something above does
not behave
+- [Extension Guide](./optimizer-extension-guide.md) for custom strategies and
providers
+- [Iceberg Compaction Policy](../iceberg-compaction-policy.md) for tuning the
built-in strategy
