This is an automated email from the ASF dual-hosted git repository.
ayushtkn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 980064b76be HIVE-29855: Optimize K8s Operator LLAP scaling (#6723)
980064b76be is described below
commit 980064b76be20049edd4048ce9c2b411e1fba534
Author: Tanishq Chugh <[email protected]>
AuthorDate: Mon Aug 31 09:49:11 2026 +0530
HIVE-29855: Optimize K8s Operator LLAP scaling (#6723)
---
packaging/src/kubernetes/README.md | 31 +++++----
.../crds/hiveclusters.hive.apache.org-v1.yml | 52 +++++++++++----
.../helm/hive-operator/templates/hivecluster.yaml | 1 +
.../src/kubernetes/helm/hive-operator/values.yaml | 3 +-
.../operator/autoscaling/ComponentAutoscaler.java | 9 ++-
.../autoscaling/HiveClusterAutoscaler.java | 15 +++++
.../operator/autoscaling/LlapScalingStrategy.java | 74 +++++++++++++++-------
.../operator/dependent/HiveDependentResource.java | 7 ++
.../operator/model/spec/AutoscalingSpec.java | 7 +-
.../operator/model/spec/HiveServer2Spec.java | 2 +-
.../kubernetes/operator/model/spec/LlapSpec.java | 4 +-
.../operator/model/spec/MetastoreSpec.java | 2 +-
.../kubernetes/operator/model/spec/TezAmSpec.java | 2 +-
.../operator/reconciler/HiveClusterReconciler.java | 11 ++--
.../hive/kubernetes/operator/util/ConfigUtils.java | 2 +
.../operator/util/HiveConfigBuilder.java | 6 ++
16 files changed, 167 insertions(+), 61 deletions(-)
diff --git a/packaging/src/kubernetes/README.md
b/packaging/src/kubernetes/README.md
index 2202885d459..1135413f612 100644
--- a/packaging/src/kubernetes/README.md
+++ b/packaging/src/kubernetes/README.md
@@ -580,7 +580,8 @@ cluster:
autoscaling:
enabled: true
minReplicas: 2
- scaleUpThreshold: 2
+ scaleUpThreshold: 80
+ scaleDownThreshold: 20
- name: analytics
enabled: true
@@ -744,7 +745,7 @@ When `autoscaling.enabled: true` is set for a component,
the operator:
- HS2: deregisters from ZK, drains open sessions, kills JVM
- HMS: kills JVM (stateless HTTP — no drain needed)
- LLAP: waits until all executors become idle, kills JVM
- - TezAM: no drain (DAGAppMaster does not expose JMX metrics)
+ - TezAM: deregisters from ZK, drains running DAG, kills JVM
5. terminationGracePeriodSeconds = gracePeriodSeconds (safety cap)
6. Pod terminates immediately once drain completes (does NOT wait
the full grace period — it's only the upper safety bound)
@@ -762,14 +763,14 @@ The autoscaling system uses three independent timing
controls:
| Timer | Config Field | Default | Purpose |
|-------|-------------|---------|---------|
| **Metrics scrape interval** | `metricsScrapeIntervalSeconds` | `10` | How
often the operator scrapes JMX Exporter `/metrics` on each pod. This is the
**biggest bottleneck** for autoscaling reaction time. |
-| **Scale-up stabilization** | `scaleUpStabilizationSeconds` | `60` | Window:
picks the highest recommendation within this period before scaling up. Prevents
flapping when metrics oscillate. Set to `0` for LLAP and TezAM (reactive
dependents). |
+| **Scale-up stabilization** | `scaleUpStabilizationSeconds` | `60` | Window:
picks the highest recommendation within this period before scaling up. Prevents
flapping when metrics oscillate. Set to `0` for LLAP and TezAM (reactive
dependents). LLAP picks the lowest recommendation to prevent over-scaling due
to momentary Reducer Burst. |
| **Scale-down stabilization** | `scaleDownStabilizationSeconds` | `300-900` |
Window: picks the most conservative (highest) recommendation within this period
before scaling down. Also acts as the cooldown between consecutive scale-downs
— no separate cooldown needed. |
**How they interact:**
- Load spike detected → operator scrapes metrics within
`metricsScrapeIntervalSeconds` → waits `scaleUpStabilizationSeconds` then
scales up
- Load drops → operator waits `scaleDownStabilizationSeconds` (stabilization
window must confirm low demand consistently) then scales down
-**Tuning reaction time:** With defaults (`metricsScrapeIntervalSeconds: 10`,
`scaleUpStabilizationSeconds: 0` for LLAP/TezAM), scale-up latency is ~10-20s
(one scrape cycle). For HS2 with `scaleUpStabilizationSeconds: 60`, expect ~70s.
+**Tuning reaction time:** With defaults (`metricsScrapeIntervalSeconds: 10`,
`scaleUpStabilizationSeconds: 60`), expect ~70s for stabilized scale-up.
### Per-Component Scaling Logic
@@ -777,7 +778,7 @@ The autoscaling system uses three independent timing
controls:
|-----------|-----------------|------------|------------|
| **HiveServer2** | `max(ceil(sessions / threshold), cpu_desired)` | Sessions
drop to 0 AND CPU below threshold → scale to minReplicas | `hs2_open_sessions`,
`jvm_process_cpu_load` |
| **Metastore** | `max(ceil(api_rate / threshold), cpu_desired)` | Rate drops
to 0 AND CPU below threshold → scale to minReplicas | `api_*_total`,
`jvm_process_cpu_load` |
-| **LLAP** | `ceil(avg(queued + configured - available) / scaleUpThreshold)` |
All executors idle + no HS2 sessions | `hadoop_llapdaemon_executor*` |
+| **LLAP** | `ceil(totalClusterLoad / capacityPerDaemon)` when pending load is
above threshold | No TezAM pending tasks AND avg daemon load is below threshold
| `hadoop_llapdaemon_executor*`, `tez_am_pending_tasks` |
| **Tez AM** | `max(sum(hs2_open_sessions), count(HS2_pods) *
sessions_per_queue)` | All HS2 sessions closed | `hs2_open_sessions` (from HS2
pods) |
**TezAM Scaling Model:** TezAM uses demand-driven scaling with two formulas
(max wins):
@@ -1092,8 +1093,8 @@ status:
```
**Applicability:** CPU-based scaling only applies to HS2 and HMS. LLAP and
TezAM
-do not use CPU as a scaling signal (LLAP scales on busy executor slots which
already
-correlates with CPU; TezAM is demand-based from HS2 session count).
+do not use CPU as a scaling signal (LLAP scales on combined TezAM pending tasks
+and daemon busy slots; TezAM is demand-based from HS2 session count).
---
@@ -1187,7 +1188,8 @@ cluster:
autoscaling:
enabled: true
# minReplicas: 0 # default — scale to zero when no sessions
target this cluster
- # scaleUpThreshold: 1 # default — total busy slots (queued+running)
triggering scale-up
+ # scaleUpThreshold: 80 # default — pending load percent of total
cluster load triggers scale-up
+ # scaleDownThreshold: 20 # default — avg daemon utilization percent
triggering scale-down
# scaleUpStabilizationSeconds: 60 # default — scale-up window
# scaleDownStabilizationSeconds: 900 # default — scale-down window (long
— scaling down destroys cache)
# gracePeriodSeconds: 600 # default — 10 min drain for in-flight
fragments
@@ -1224,8 +1226,8 @@ When autoscaling is enabled, the operator automatically:
|-----------|---------|---------|
| **HiveServer2** | `hs2_open_sessions`, `jvm_process_cpu_load` | Session
count for primary scaling + CPU for secondary scaling signal |
| **Metastore** | `api_*_total`, `jvm_process_cpu_load` | API call counters
(operator computes request rate from deltas) + CPU for secondary scaling signal
|
-| **LLAP** | `hadoop_llapdaemon_executornumqueuedrequests`,
`hadoop_llapdaemon_executornumexecutorsconfigured`,
`hadoop_llapdaemon_executornumexecutorsavailable` | Total busy slots = queued +
configured - available |
-| **Tez AM** | N/A (scales on HS2 metrics) | TezAM scaling is demand-driven
from `hs2_open_sessions` — no TezAM-specific metrics needed |
+| **LLAP** | `hadoop_llapdaemon_executornumqueuedrequests`,
`hadoop_llapdaemon_executornumexecutorsconfigured`,
`hadoop_llapdaemon_executornumexecutorsavailable`,
`hadoop_llapdaemon_executormaxfreeslotsconfigured` | Daemon busy slots and
capacity; combined with TezAM pending tasks for scaling |
+| **Tez AM** | `tez_am_pending_tasks`, `tez_am_dag_running` | Pending task
count drives LLAP scale-up; DAG status ranks idle AMs for cost-based scale-down
|
### Enabling Autoscaling — Example
@@ -1248,7 +1250,7 @@ cluster:
scaleUpThreshold: 75 # API requests/sec threshold
```
-> **Note:** LLAP scales on total busy slots (queued + running executors).
+> **Note:** LLAP scales based on combined TezAM pending tasks and daemon busy
slots.
> TezAM scales on demand — the number of active HS2 pods multiplied by
> `hive.server2.tez.sessions.per.default.queue` (default 1).
@@ -1260,6 +1262,7 @@ cluster:
| `cluster.<component>.autoscaling.enabled` | `false` | Enable operator-driven
autoscaling |
| `cluster.<component>.autoscaling.minReplicas` | `1` (HS2/HMS), `0`
(LLAP/TezAM) | Minimum replica count. Set to 0 for scale-to-zero (LLAP, TezAM
only; HS2 minimum is 1) |
| `cluster.<component>.autoscaling.scaleUpThreshold` | varies | Metric
threshold triggering scale-up |
+| `cluster.<component>.autoscaling.scaleDownThreshold` | `20` (LLAP only) |
Average daemon utilization percent below which LLAP scale-down triggers |
| `cluster.<component>.autoscaling.scaleUpStabilizationSeconds` | `60` |
Stabilization window for scale-up (picks highest recommendation in window) |
| `cluster.<component>.autoscaling.scaleDownStabilizationSeconds` | `300-900`
| Stabilization window for scale-down (picks most conservative recommendation
in window). Also acts as cooldown between consecutive scale-downs. |
| `cluster.<component>.autoscaling.gracePeriodSeconds` | `3600` | Safety cap:
max drain time before forced termination. Pod exits immediately once drain
completes. |
@@ -1406,7 +1409,8 @@ and a paired TezAM Deployment (when `tezAm.enabled:
true`).
| `cluster.llapClusters[].extraVolumeMounts` | `[]` | Additional volume mounts
for LLAP containers |
| `cluster.llapClusters[].autoscaling.enabled` | `false` | Enable per-cluster
autoscaling |
| `cluster.llapClusters[].autoscaling.minReplicas` | `0` | Min replicas (0 =
scale to zero) |
-| `cluster.llapClusters[].autoscaling.scaleUpThreshold` | `1` | Busy-slot
threshold for scale-up |
+| `cluster.llapClusters[].autoscaling.scaleUpThreshold` | `80` | Pending load
percent of total cluster load for scale-up |
+| `cluster.llapClusters[].autoscaling.scaleDownThreshold` | `20` | Avg daemon
utilization percent for scale-down |
HS2 routes sessions to LLAP clusters server-side based on user/group identity:
@@ -1448,7 +1452,8 @@ controls shared settings (enabled flag, scratch PVC).
Per-LLAP TezAM settings
|-------|---------|-------------|
| `cluster.<component>.autoscaling.enabled` | `false` | Enable operator-driven
autoscaling for this component |
| `cluster.<component>.autoscaling.minReplicas` | `0` | Floor replica count. 0
enables scale-to-zero (LLAP, TezAM only; HS2 minimum is 1) |
-| `cluster.<component>.autoscaling.scaleUpThreshold` | `100` (HS2/HMS), `10`
(LLAP) | Metric threshold per pod triggering scale-up (sessions for HS2,
connections for HMS, busy slots for LLAP). TezAM scales 1:1 with demand (no
threshold). |
+| `cluster.<component>.autoscaling.scaleUpThreshold` | `100` (HS2/HMS), `80`
(LLAP) | Metric threshold per pod triggering scale-up (sessions for HS2, API
rate for HMS, pending load percent for LLAP). TezAM scales 1:1 with demand (no
threshold). |
+| `cluster.<component>.autoscaling.scaleDownThreshold` | `20` (LLAP) | Average
daemon utilization percent below which LLAP scale-down triggers |
| `cluster.<component>.autoscaling.scaleUpStabilizationSeconds` | `60` |
Stabilization window for scale-up decisions (prevents flapping) |
| `cluster.<component>.autoscaling.scaleDownStabilizationSeconds` | `300-900`
| Stabilization window for scale-down decisions (also acts as cooldown between
consecutive scale-downs) |
| `cluster.<component>.autoscaling.gracePeriodSeconds` | `3600` | Safety cap
(seconds) — pod terminates immediately once drain completes, this is only the
upper bound |
diff --git
a/packaging/src/kubernetes/helm/hive-operator/crds/hiveclusters.hive.apache.org-v1.yml
b/packaging/src/kubernetes/helm/hive-operator/crds/hiveclusters.hive.apache.org-v1.yml
index 9947f44b75b..9bdb3863c52 100644
---
a/packaging/src/kubernetes/helm/hive-operator/crds/hiveclusters.hive.apache.org-v1.yml
+++
b/packaging/src/kubernetes/helm/hive-operator/crds/hiveclusters.hive.apache.org-v1.yml
@@ -129,6 +129,12 @@ spec:
replicas before scale-down occurs. Also acts as the
cooldown
between consecutive scale-downs.
type: integer
+ scaleDownThreshold:
+ default: 20
+ description: "Threshold that triggers scale-down (LLAP
only):\
+ \ average daemon utilization percentage (0-100)
below which\
+ \ scale-down triggers."
+ type: integer
scaleUpStabilizationSeconds:
default: 60
description: Stabilization window in seconds for
scale-up
@@ -138,8 +144,8 @@ spec:
scaleUpThreshold:
default: 80
description: "Threshold that triggers scale-up
(component-specific:\
- \ sessions per pod for HS2, request rate for HMS,
busy slots\
- \ per daemon for LLAP). Not used by TezAM
(demand-based:\
+ \ sessions per pod for HS2, request rate for HMS,
load percentage\
+ \ pending in TezAM for LLAP). Not used by TezAM
(demand-based:\
\ 1 TezAM per session)."
type: integer
type: object
@@ -312,6 +318,12 @@ spec:
fewer replicas before scale-down occurs. Also acts
as
the cooldown between consecutive scale-downs.
type: integer
+ scaleDownThreshold:
+ default: 20
+ description: "Threshold that triggers scale-down
(LLAP only):\
+ \ average daemon utilization percentage (0-100)
below\
+ \ which scale-down triggers."
+ type: integer
scaleUpStabilizationSeconds:
default: 60
description: Stabilization window in seconds for
scale-up
@@ -321,9 +333,9 @@ spec:
scaleUpThreshold:
default: 80
description: "Threshold that triggers scale-up
(component-specific:\
- \ sessions per pod for HS2, request rate for HMS,
busy\
- \ slots per daemon for LLAP). Not used by TezAM
(demand-based:\
- \ 1 TezAM per session)."
+ \ sessions per pod for HS2, request rate for HMS,
load\
+ \ percentage pending in TezAM for LLAP). Not used
by TezAM\
+ \ (demand-based: 1 TezAM per session)."
type: integer
type: object
configOverrides:
@@ -467,6 +479,12 @@ spec:
fewer replicas before scale-down occurs. Also
acts
as the cooldown between consecutive
scale-downs.
type: integer
+ scaleDownThreshold:
+ default: 20
+ description: "Threshold that triggers scale-down
(LLAP\
+ \ only): average daemon utilization percentage
(0-100)\
+ \ below which scale-down triggers."
+ type: integer
scaleUpStabilizationSeconds:
default: 60
description: Stabilization window in seconds for
scale-up
@@ -477,8 +495,8 @@ spec:
default: 80
description: "Threshold that triggers scale-up
(component-specific:\
\ sessions per pod for HS2, request rate for
HMS,\
- \ busy slots per daemon for LLAP). Not used by
TezAM\
- \ (demand-based: 1 TezAM per session)."
+ \ load percentage pending in TezAM for LLAP).
Not\
+ \ used by TezAM (demand-based: 1 TezAM per
session)."
type: integer
type: object
replicas:
@@ -545,6 +563,12 @@ spec:
replicas before scale-down occurs. Also acts as the
cooldown
between consecutive scale-downs.
type: integer
+ scaleDownThreshold:
+ default: 20
+ description: "Threshold that triggers scale-down (LLAP
only):\
+ \ average daemon utilization percentage (0-100)
below which\
+ \ scale-down triggers."
+ type: integer
scaleUpStabilizationSeconds:
default: 60
description: Stabilization window in seconds for
scale-up
@@ -554,8 +578,8 @@ spec:
scaleUpThreshold:
default: 80
description: "Threshold that triggers scale-up
(component-specific:\
- \ sessions per pod for HS2, request rate for HMS,
busy slots\
- \ per daemon for LLAP). Not used by TezAM
(demand-based:\
+ \ sessions per pod for HS2, request rate for HMS,
load percentage\
+ \ pending in TezAM for LLAP). Not used by TezAM
(demand-based:\
\ 1 TezAM per session)."
type: integer
type: object
@@ -753,6 +777,12 @@ spec:
replicas before scale-down occurs. Also acts as the
cooldown
between consecutive scale-downs.
type: integer
+ scaleDownThreshold:
+ default: 20
+ description: "Threshold that triggers scale-down (LLAP
only):\
+ \ average daemon utilization percentage (0-100)
below which\
+ \ scale-down triggers."
+ type: integer
scaleUpStabilizationSeconds:
default: 60
description: Stabilization window in seconds for
scale-up
@@ -762,8 +792,8 @@ spec:
scaleUpThreshold:
default: 80
description: "Threshold that triggers scale-up
(component-specific:\
- \ sessions per pod for HS2, request rate for HMS,
busy slots\
- \ per daemon for LLAP). Not used by TezAM
(demand-based:\
+ \ sessions per pod for HS2, request rate for HMS,
load percentage\
+ \ pending in TezAM for LLAP). Not used by TezAM
(demand-based:\
\ 1 TezAM per session)."
type: integer
type: object
diff --git
a/packaging/src/kubernetes/helm/hive-operator/templates/hivecluster.yaml
b/packaging/src/kubernetes/helm/hive-operator/templates/hivecluster.yaml
index 278ae40e8df..a2b465ba0b3 100644
--- a/packaging/src/kubernetes/helm/hive-operator/templates/hivecluster.yaml
+++ b/packaging/src/kubernetes/helm/hive-operator/templates/hivecluster.yaml
@@ -157,6 +157,7 @@ spec:
enabled: true
minReplicas: {{ .autoscaling.minReplicas }}
scaleUpThreshold: {{ .autoscaling.scaleUpThreshold }}
+ scaleDownThreshold: {{ .autoscaling.scaleDownThreshold }}
scaleUpStabilizationSeconds: {{ .autoscaling.scaleUpStabilizationSeconds
}}
scaleDownStabilizationSeconds: {{
.autoscaling.scaleDownStabilizationSeconds }}
gracePeriodSeconds: {{ .autoscaling.gracePeriodSeconds }}
diff --git a/packaging/src/kubernetes/helm/hive-operator/values.yaml
b/packaging/src/kubernetes/helm/hive-operator/values.yaml
index c16f7240ea2..dda082695f8 100644
--- a/packaging/src/kubernetes/helm/hive-operator/values.yaml
+++ b/packaging/src/kubernetes/helm/hive-operator/values.yaml
@@ -200,7 +200,8 @@ cluster:
autoscaling:
enabled: false
minReplicas: 0
- scaleUpThreshold: 10
+ scaleUpThreshold: 80
+ scaleDownThreshold: 20
scaleUpStabilizationSeconds: 60
scaleDownStabilizationSeconds: 900
gracePeriodSeconds: 600
diff --git
a/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/autoscaling/ComponentAutoscaler.java
b/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/autoscaling/ComponentAutoscaler.java
index fcf4da2aefb..a55973bfe34 100644
---
a/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/autoscaling/ComponentAutoscaler.java
+++
b/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/autoscaling/ComponentAutoscaler.java
@@ -94,8 +94,13 @@ public EvaluationResult evaluate(List<PodMetrics> metrics,
AutoscalingSpec spec,
int target;
if (clamped > currentReplicas) {
- // Scale up: use stabilized max (highest recommendation in window —
don't under-scale)
- target = scaleUpWindow.stabilizedMax();
+ if (component.startsWith(ConfigUtils.COMPONENT_LLAP + "-")) {
+ // HS2 sessions activation gate scales up the LLAP pods to atleast 1
+ // in presence of sessions. Avoid stabilizedMin in this start-up case.
+ target = currentReplicas == 0 ? clamped :
scaleUpWindow.stabilizedMin();
+ } else {
+ target = scaleUpWindow.stabilizedMax();
+ }
} else if (clamped < currentReplicas) {
// Scale down: use stabilized max (highest/most conservative
recommendation in window —
// prevents premature scale-down, matches HPA selectPolicy: Max
behavior).
diff --git
a/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/autoscaling/HiveClusterAutoscaler.java
b/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/autoscaling/HiveClusterAutoscaler.java
index 3c69a957ef9..f6016043fc6 100644
---
a/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/autoscaling/HiveClusterAutoscaler.java
+++
b/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/autoscaling/HiveClusterAutoscaler.java
@@ -321,6 +321,21 @@ public List<PodMetrics> getHs2MetricsFromCache(HiveCluster
cluster) {
return metricsCache.getOrEmpty(key, maxStale);
}
+ /**
+ * Returns cached TezAM metrics for the given LLAP cluster (used by
LlapScalingStrategy).
+ */
+ public List<PodMetrics> getTezAmMetricsFromCache(HiveCluster cluster, String
llapName) {
+ String namespace = cluster.getMetadata().getNamespace();
+ String clusterName = cluster.getMetadata().getName();
+ int maxStale = cluster.getSpec().llapClusters().stream()
+ .filter(l -> llapName.equals(l.name()))
+ .findFirst()
+ .map(l -> l.tezAm().autoscaling().metricsScrapeIntervalSeconds() * 3)
+ .orElse(30);
+ return metricsCache.getOrEmpty(
+ cacheKey(namespace, clusterName,
ConfigUtils.tezAmComponentKey(llapName)), maxStale);
+ }
+
private void evaluateComponent(HiveCluster cluster, KubernetesClient client,
String namespace, String clusterName, String component,
AutoscalingSpec autoscaling, int maxReplicas,
diff --git
a/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/autoscaling/LlapScalingStrategy.java
b/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/autoscaling/LlapScalingStrategy.java
index 368d38474d6..546800403ac 100644
---
a/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/autoscaling/LlapScalingStrategy.java
+++
b/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/autoscaling/LlapScalingStrategy.java
@@ -28,11 +28,12 @@
/**
* Scaling strategy for LLAP daemons.
- * Formula: avg(QueuedRequests + Configured - Available) across all pods.
- * This represents average "busy slots + queued" per daemon.
- * desired = ceil(avg_busy / scaleUpThreshold)
+ * For Scale-Up: Pending Load across all TezAM pods should be above the
threshold.
+ * For Scale-Down: Running Load across all LLAP pods should be below the
threshold.
+ * desired = ceil(totalClusterLoad / capacityPerDaemon)
* <p>
- * Activation gate: only scale if HS2 has open sessions (prevents zombie
scaling).
+ * Activation gate: only scale if HS2 has open sessions, TezAM metrics are
present &
+ * cluster has load (prevents zombie scaling).
*/
public class LlapScalingStrategy implements ScalingStrategy {
@@ -41,7 +42,9 @@ public class LlapScalingStrategy implements ScalingStrategy {
static final String METRIC_QUEUED =
"hadoop_llapdaemon_executornumqueuedrequests";
static final String METRIC_CONFIGURED =
"hadoop_llapdaemon_executornumexecutorsconfigured";
static final String METRIC_AVAILABLE =
"hadoop_llapdaemon_executornumexecutorsavailable";
+ static final String METRIC_MAX_FREE_SLOTS_CONFIGURED =
"hadoop_llapdaemon_executormaxfreeslotsconfigured";
static final String METRIC_LLAP_TARGET_PREFIX = "hs2_llap_target_sessions_";
+ static final String METRIC_TEZ_PENDING_TASKS = "tez_am_pending_tasks";
private final HiveClusterAutoscaler orchestrator;
private final HiveCluster cluster;
@@ -67,7 +70,7 @@ public int computeDesiredReplicas(List<PodMetrics> podMetrics,
// HS2 scrape returned no data but LLAP is running — hold current state
LOG.debug("[llap] HS2 scrape returned no pods; preserving LLAP (has {}
running pods)", podMetrics.size());
lastMetric = 0;
- return Math.max(1, autoscaling.minReplicas());
+ return podMetrics.size();
}
if (sessionsDetected == null || !sessionsDetected) {
LOG.debug("[llap] HS2 has no open sessions, scaling to minReplicas");
@@ -83,36 +86,59 @@ public int computeDesiredReplicas(List<PodMetrics>
podMetrics,
return minReplica;
}
- // Compute average busy slots across all LLAP pods
- double totalBusy = 0;
- int podCount = 0;
+ List<PodMetrics> tezAmMetrics =
orchestrator.getTezAmMetricsFromCache(cluster, llapName);
+ double totalPending = 0;
+ for (PodMetrics pm : tezAmMetrics) {
+ totalPending += pm.metrics().getOrDefault(METRIC_TEZ_PENDING_TASKS, 0.0);
+ }
+
+ double totalLLAPCapacity = 0;
+ double totalLLAPLoad = 0;
for (PodMetrics pm : podMetrics) {
double queued = pm.metrics().getOrDefault(METRIC_QUEUED, 0.0);
double configured = pm.metrics().getOrDefault(METRIC_CONFIGURED, 0.0);
double available = pm.metrics().getOrDefault(METRIC_AVAILABLE, 0.0);
- double busy = queued + configured - available;
- totalBusy += busy;
- podCount++;
+ double capacity =
pm.metrics().getOrDefault(METRIC_MAX_FREE_SLOTS_CONFIGURED, 0.0);
+ if (capacity <= 0) {
+ capacity = configured;
+ }
+ totalLLAPCapacity += capacity;
+ totalLLAPLoad += queued + configured - available;
}
- double avgBusy = totalBusy / podCount;
- lastMetric = (int) Math.round(avgBusy);
-
- if (avgBusy <= 0) {
- // HS2 has sessions (passed activation gate above) but executors are
idle between queries.
- // Keep at least 1 daemon to avoid flapping: scaling to 0 here would
cause immediate
- // scale-back-up on the next evaluation when the empty-pod path triggers.
+ // HS2 has sessions (passed activation gate above) but either
+ // 1. there is no tezAM running, so LLAP running any work is zombie if any.
+ // 2. there are tezAMs running, but no tasks running or pending.
+ if (tezAmMetrics.isEmpty() || (totalPending + totalLLAPLoad) == 0) {
return Math.max(1, autoscaling.minReplicas());
}
- if (LOG.isDebugEnabled()) {
- LOG.debug("[llap] avgBusy={}, threshold={}", String.format("%.2f",
avgBusy),
- autoscaling.scaleUpThreshold());
+ double capacityPerDaemon = Math.max(1.0, totalLLAPCapacity /
podMetrics.size());
+ double avgLLAPLoadPercent = totalLLAPCapacity > 0 ? (totalLLAPLoad /
totalLLAPCapacity) * 100.00 : 0.0;
+
+ double totalClusterLoad = totalPending + totalLLAPLoad;
+ double pendingLoadPercent = (totalPending / totalClusterLoad) * 100.0;
+
+ int scaleUpThreshold = autoscaling.scaleUpThreshold();
+ int scaleDownThreshold = autoscaling.scaleDownThreshold();
+
+ lastMetric = (int) totalClusterLoad;
+
+ // Scale-up: pending load share of total load exceeds threshold
+ // Scale-down: no pending work AND daemon load below threshold
+ if (pendingLoadPercent >= scaleUpThreshold || (totalPending == 0 &&
avgLLAPLoadPercent <= scaleDownThreshold)) {
+ int desired = (int) Math.ceil(totalClusterLoad / capacityPerDaemon);
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("[llap-{}] totalClusterLoad={}, capacityPerDaemon={},
pendingLoadPercent={}, avgLLAPLoadPercent={}",
+ llapName, totalClusterLoad, capacityPerDaemon,
String.format("%.2f", pendingLoadPercent),
+ String.format("%.2f", avgLLAPLoadPercent));
+ }
+ return desired;
}
- int threshold = Math.max(1, autoscaling.scaleUpThreshold());
- int desired = (int) Math.ceil(avgBusy / threshold);
- return Math.max(desired, autoscaling.minReplicas());
+ // Work is in flight (pending share below threshold) or daemons are
moderately loaded.
+ // Return current pod count so the stabilization window keeps the replica
count stable.
+ return podMetrics.size();
}
@Override
diff --git
a/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/dependent/HiveDependentResource.java
b/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/dependent/HiveDependentResource.java
index ebe89d06143..4cb348bfd47 100644
---
a/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/dependent/HiveDependentResource.java
+++
b/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/dependent/HiveDependentResource.java
@@ -854,14 +854,21 @@ private static String buildJmxExporterConfig(String
component) {
sb.append("- pattern:
'").append(llapBean).append("ExecutorNumExecutors'\n");
sb.append(" name: hadoop_llapdaemon_executornumexecutors\n");
sb.append(" type: GAUGE\n");
+ sb.append("- pattern:
'").append(llapBean).append("ExecutorMaxFreeSlotsConfigured'\n");
+ sb.append(" name: hadoop_llapdaemon_executormaxfreeslotsconfigured\n");
+ sb.append(" type: GAUGE\n");
break;
case ConfigUtils.COMPONENT_TEZAM:
// LlapMetricsSystem registers beans under LlapTaskScheduler service
// SchedulerDagStatus tracks if the AM is running a dag or is idle so
exported as GAUGE.
+ // SchedulerPendingTaskCount tracks currently waiting tasks in the AM to
be scheduled.
String schedulerBean = "Hadoop<service=LlapTaskScheduler,
name=LlapTaskSchedulerMetrics.+><>";
sb.append("- pattern:
'").append(schedulerBean).append("SchedulerDagStatus'\n");
sb.append(" name: tez_am_dag_running\n");
sb.append(" type: GAUGE\n");
+ sb.append("- pattern:
'").append(schedulerBean).append("SchedulerPendingTaskCount'\n");
+ sb.append(" name: tez_am_pending_tasks\n");
+ sb.append(" type: GAUGE\n");
break;
default:
sb.append("- pattern: '.*'\n");
diff --git
a/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/model/spec/AutoscalingSpec.java
b/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/model/spec/AutoscalingSpec.java
index 1cdb06bdad4..a1bc6ea3a10 100644
---
a/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/model/spec/AutoscalingSpec.java
+++
b/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/model/spec/AutoscalingSpec.java
@@ -33,10 +33,14 @@ public record AutoscalingSpec(
@Default("0")
Integer minReplicas,
@JsonPropertyDescription("Threshold that triggers scale-up
(component-specific: "
- + "sessions per pod for HS2, request rate for HMS, busy slots per
daemon for LLAP). "
+ + "sessions per pod for HS2, request rate for HMS, load percentage
pending in TezAM for LLAP). "
+ "Not used by TezAM (demand-based: 1 TezAM per session).")
@Default("80")
Integer scaleUpThreshold,
+ @JsonPropertyDescription("Threshold that triggers scale-down (LLAP only): "
+ + "average daemon utilization percentage (0-100) below which
scale-down triggers.")
+ @Default("20")
+ Integer scaleDownThreshold,
@JsonPropertyDescription("Stabilization window in seconds for scale-up
decisions. "
+ "Picks the highest recommendation within this window to prevent
flapping.")
@Default("60")
@@ -73,6 +77,7 @@ public record AutoscalingSpec(
enabled = enabled != null ? enabled : false;
minReplicas = minReplicas != null ? minReplicas : 0;
scaleUpThreshold = scaleUpThreshold != null ? scaleUpThreshold : 80;
+ scaleDownThreshold = scaleDownThreshold != null ? scaleDownThreshold : 20;
scaleUpStabilizationSeconds = scaleUpStabilizationSeconds != null ?
scaleUpStabilizationSeconds : 60;
scaleDownStabilizationSeconds = scaleDownStabilizationSeconds != null ?
scaleDownStabilizationSeconds : 600;
gracePeriodSeconds = gracePeriodSeconds != null ? gracePeriodSeconds :
3600;
diff --git
a/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/model/spec/HiveServer2Spec.java
b/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/model/spec/HiveServer2Spec.java
index 62855537bcd..b4962d7e6f3 100644
---
a/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/model/spec/HiveServer2Spec.java
+++
b/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/model/spec/HiveServer2Spec.java
@@ -63,6 +63,6 @@ public record HiveServer2Spec(
extraVolumeMounts = extraVolumeMounts != null ? extraVolumeMounts :
List.of();
externalJars = externalJars != null ? externalJars : List.of();
autoscaling = autoscaling != null ? autoscaling : new AutoscalingSpec(
- false, 1, 80, 60, 600, 300, 10, 90, 30, null);
+ false, 1, 80, 0, 60, 600, 300, 10, 90, 30, null);
}
}
diff --git
a/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/model/spec/LlapSpec.java
b/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/model/spec/LlapSpec.java
index 61279829c90..eb42d8c8c0e 100644
---
a/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/model/spec/LlapSpec.java
+++
b/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/model/spec/LlapSpec.java
@@ -81,7 +81,7 @@ public record LlapTezAmSpec(
public LlapTezAmSpec {
replicas = replicas != null ? replicas : 1;
autoscaling = autoscaling != null ? autoscaling : new AutoscalingSpec(
- false, 0, 0, 60, 600, 120, 10, 0, 0, null);
+ false, 0, 0, 0, 60, 600, 120, 10, 0, 0, null);
}
}
@@ -102,7 +102,7 @@ public record LlapTezAmSpec(
extraVolumes = extraVolumes != null ? extraVolumes : List.of();
extraVolumeMounts = extraVolumeMounts != null ? extraVolumeMounts :
List.of();
autoscaling = autoscaling != null ? autoscaling : new AutoscalingSpec(
- false, 0, 1, 60, 900, 600, 10, 0, 0, null);
+ false, 0, 1, 20, 60, 900, 600, 10, 0, 0, null);
tezAm = tezAm != null ? tezAm : new LlapTezAmSpec(null, null);
}
diff --git
a/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/model/spec/MetastoreSpec.java
b/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/model/spec/MetastoreSpec.java
index 956bac65bcb..6548d999918 100644
---
a/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/model/spec/MetastoreSpec.java
+++
b/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/model/spec/MetastoreSpec.java
@@ -70,7 +70,7 @@ public record MetastoreSpec(
extraVolumes = extraVolumes != null ? extraVolumes : List.of();
extraVolumeMounts = extraVolumeMounts != null ? extraVolumeMounts :
List.of();
autoscaling = autoscaling != null ? autoscaling : new AutoscalingSpec(
- false, 1, 75, 60, 300, 60, 10, 90, 30, null);
+ false, 1, 75, 0, 60, 300, 60, 10, 90, 30, null);
}
public boolean isEnabled() {
diff --git
a/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/model/spec/TezAmSpec.java
b/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/model/spec/TezAmSpec.java
index 98d1c976dd9..8165b9b829a 100644
---
a/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/model/spec/TezAmSpec.java
+++
b/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/model/spec/TezAmSpec.java
@@ -64,7 +64,7 @@ public record TezAmSpec(
extraVolumes = extraVolumes != null ? extraVolumes : List.of();
extraVolumeMounts = extraVolumeMounts != null ? extraVolumeMounts :
List.of();
autoscaling = autoscaling != null ? autoscaling : new AutoscalingSpec(
- false, 0, 0, 60, 600, 120, 10, 0, 0, null);
+ false, 0, 0, 0, 60, 600, 120, 10, 0, 0, null);
}
public boolean isEnabled() {
diff --git
a/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/reconciler/HiveClusterReconciler.java
b/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/reconciler/HiveClusterReconciler.java
index ebc5c4f1ae8..a7da524a3db 100644
---
a/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/reconciler/HiveClusterReconciler.java
+++
b/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/reconciler/HiveClusterReconciler.java
@@ -704,10 +704,13 @@ private int resolveTezAmReplicaCount(HiveCluster resource,
if (llapManaged != null && llapManaged == 0) {
return 0;
}
- if (llapSpec.autoscaling().isEnabled() && llapManaged == null) {
- // First reconcile, LLAP starts at minReplicas (likely 0) — TezAM matches
- return llapSpec.autoscaling().minReplicas() > 0
- ? tezAmSpec.replicas() : 0;
+ if (llapSpec.autoscaling().isEnabled() && llapManaged == null
+ && llapSpec.autoscaling().minReplicas() == 0) {
+ // First reconcile before autoscaler runs: LLAP starts at 0, so TezAM
stays down too.
+ return 0;
+ }
+ if (tezAmSpec.autoscaling().isEnabled()) {
+ return tezAmSpec.autoscaling().minReplicas();
}
return tezAmSpec.replicas();
}
diff --git
a/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/util/ConfigUtils.java
b/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/util/ConfigUtils.java
index 6253681455b..76dac7a640b 100644
---
a/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/util/ConfigUtils.java
+++
b/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/util/ConfigUtils.java
@@ -141,6 +141,8 @@ public static String tezAmComponentKey(String llapName) {
public static final String HIVE_LLAP_DAEMON_UMBILICAL_PORT_KEY =
"hive.llap.daemon.umbilical.port";
public static final String HIVE_LLAP_DAEMON_UMBILICAL_PORT_DEFAULT = "0";
+ public static final String HIVE_LLAP_TASK_SCHEDULER_LOCALITY_DELAY_KEY =
"hive.llap.task.scheduler.locality.delay";
+
public static final String METASTORE_SERVER_TRANSPORT_MODE_KEY =
"metastore.server.thrift.transport.mode";
public static final String METASTORE_SERVER_TRANSPORT_MODE_DEFAULT = "http";
diff --git
a/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/util/HiveConfigBuilder.java
b/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/util/HiveConfigBuilder.java
index d71011f9289..9c78fa15342 100644
---
a/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/util/HiveConfigBuilder.java
+++
b/packaging/src/kubernetes/src/java/org/apache/hive/kubernetes/operator/util/HiveConfigBuilder.java
@@ -204,6 +204,12 @@ public static Map<String, String>
getTezSite(HiveClusterSpec spec, LlapSpec llap
if (spec.tezAm().configOverrides() != null) {
tezProps.putAll(spec.tezAm().configOverrides());
}
+
+ // Disable Infinite locality Delay when LLAP Auto-scaling is enabled, as
they are mutually exclusive.
+ if (llap != null && llap.isEnabled() && llap.autoscaling().isEnabled() &&
+ ConfigUtils.getTimeMs(tezProps,
ConfigUtils.HIVE_LLAP_TASK_SCHEDULER_LOCALITY_DELAY_KEY, 0) == -1) {
+ tezProps.put(ConfigUtils.HIVE_LLAP_TASK_SCHEDULER_LOCALITY_DELAY_KEY,
"0ms");
+ }
return tezProps;
}
