This is an automated email from the ASF dual-hosted git repository.
baibaichen pushed a commit to branch feature/ansi-baseline-workflow
in repository https://gitbox.apache.org/repos/asf/gluten.git
The following commit(s) were added to refs/heads/feature/ansi-baseline-workflow
by this push:
new 800de149ae [GLUTEN] Extract ANSI analysis prompt into shared SKILL
800de149ae is described below
commit 800de149aed6519fd62c7e6bc11bce02919bd76f
Author: Chang chen <[email protected]>
AuthorDate: Tue Apr 21 12:06:32 2026 +0800
[GLUTEN] Extract ANSI analysis prompt into shared SKILL
Refactor the AI prompt embedded in .github/scripts/analyze-ansi.py into
a shared markdown file so both the CI Python pipeline and local agents
(GitHub Copilot CLI / Claude Code) consume a single source of truth,
preventing prompt drift.
Layout follows the SKILL convention (one directory per skill):
.github/skills/ansi-analysis/
SKILL.md - orchestration (Step 0 mandates reading shared.md)
shared.md - the analysis prompt, also loaded by analyze-ansi.py
analyze-ansi.py changes:
- Delete inline ANALYSIS_PROMPT_TEMPLATE constant
- Add _load_prompt_template() with fail-fast (no silent fallback)
- Use str.replace(PROMPT_PLACEHOLDER, json_output) instead of
.format(json_data=...), since shared.md contains many literal '{...}'
blocks that would crash str.format
shared.md additions over the original prompt:
- Extended source location reference (Spark plan / Substrait / Native
bridge / C++ Velox layers)
- Self-investigation guide pointing AI agents at Velox file:line
embedded in failCause and at isAnsiSupported() in SparkCastExpr.cpp
(root cause of most NO_EXCEPTION/Cast failures)
Co-authored-by: Copilot <[email protected]>
---
.github/scripts/analyze-ansi.py | 70 ++++++++-----------------
.github/skills/ansi-analysis/SKILL.md | 95 ++++++++++++++++++++++++++++++++++
.github/skills/ansi-analysis/shared.md | 76 +++++++++++++++++++++++++++
3 files changed, 193 insertions(+), 48 deletions(-)
diff --git a/.github/scripts/analyze-ansi.py b/.github/scripts/analyze-ansi.py
index c03bb0f62d..b870b72afe 100644
--- a/.github/scripts/analyze-ansi.py
+++ b/.github/scripts/analyze-ansi.py
@@ -26,64 +26,38 @@ import argparse
import glob
import json
import os
+import pathlib
import re
import subprocess
import sys
import xml.etree.ElementTree as ET
from collections import defaultdict
+# Shared analysis prompt — single source of truth consumed by both this script
+# and the local agent SKILL (`.github/skills/ansi-analysis.md`).
+SHARED_PROMPT_PATH = (
+ pathlib.Path(__file__).resolve().parent.parent
+ / "skills" / "ansi-analysis" / "shared.md"
+)
+PROMPT_PLACEHOLDER = "{json_data}"
+
+
+def _load_prompt_template():
+ """Load the shared prompt template. Fail-fast if missing — drift between
+ this script and the SKILL is exactly the bug this layout prevents."""
+ if not SHARED_PROMPT_PATH.is_file():
+ sys.exit(
+ f"FATAL: shared prompt not found at {SHARED_PROMPT_PATH}. "
+ f"Repository layout is broken — refusing to fall back to a stale "
+ f"in-script copy."
+ )
+ return SHARED_PROMPT_PATH.read_text(encoding="utf-8")
+
NO_EXCEPTION_RE = re.compile(
r"Expected .+ to be thrown, but no exception was thrown")
WRONG_EXCEPTION_RE = re.compile(r"Expected (\S+) but got (\S+):")
MSG_MISMATCH_RE = re.compile(r"Expected error message containing")
-ANALYSIS_PROMPT_TEMPLATE = """\
-You are an ANSI mode test analysis expert for the Gluten project. Gluten is a
native engine \
-acceleration plugin for Apache Spark that offloads expression evaluation to
Velox (C++). \
-ANSI mode requires throwing exceptions on overflow, invalid type casts, etc.
-
-Below is the structured output from JSON expression tests (not XML suite
tests):
-
-```json
-{json_data}
-```
-
-Analyze only the JSON expression test data. Output key findings directly — no
overview section.
-
-Generate analysis in Markdown:
-
-## Key Findings
-- Failure hotspot table (Suite / Failures / Root Cause)
-- failCause type statistics table (Type / Count / % / Interpretation):
- - WRONG_EXCEPTION: Velox threw an exception but Spark's scheduling layer
wrapped it as \
-SparkException, losing the original exception type
- - NO_EXCEPTION: Velox did not throw the expected exception in ANSI mode
- - OTHER: Result mismatch or other errors
-- Root cause deep analysis for WRONG_EXCEPTION (exception wrapping chain path,
key code locations)
-- Breakdown of NO_EXCEPTION by root cause (arithmetic/cast/datetime etc.)
-
-## Fix Recommendations (P0 / P1 / P2 only)
-Each recommendation includes:
-- Symptom: test failure pattern
-- Root Cause: specific code path and logic issue
-- Fix Point: file path + change direction
-- Representative Tests: affected test names
-- Estimated Impact: number of tests that would turn green after fix
-
-Key source locations (for reference):
-- Exception lookup:
gluten-ut/common/src/test/scala/org/apache/spark/sql/GlutenTestsTrait.scala
(findCause method)
-- Exception wrapping:
gluten-arrow/src/main/java/org/apache/gluten/vectorized/ColumnarBatchOutIterator.java
(translateException)
-- Expression conversion: gluten-core/.../ExpressionConverter.scala
-- Type mapping: gluten-substrait/.../ConverterUtils.scala (getTypeNode)
-- Velox Cast:
ep/build-velox/build/velox_ep/velox/functions/sparksql/SparkCastExpr.cpp
-- Velox Arithmetic:
ep/build-velox/build/velox_ep/velox/functions/sparksql/Arithmetic.cpp
-
-Constraints:
-- Use Markdown tables, no ASCII box drawing characters
-- Maximum 3 fix recommendations
-- If source code is accessible, read key files to verify root cause analysis
-"""
-
def classify_fail_cause(message):
if not message:
@@ -508,7 +482,7 @@ def call_ai_analysis(json_output, model=None):
models_to_try.append(model)
models_to_try.extend(["gpt-4.1", "gpt-4o"])
- prompt = ANALYSIS_PROMPT_TEMPLATE.format(json_data=json_output)
+ prompt = _load_prompt_template().replace(PROMPT_PLACEHOLDER, json_output)
headers = {
"Authorization": f"Bearer {token}",
"Content-Type": "application/json",
diff --git a/.github/skills/ansi-analysis/SKILL.md
b/.github/skills/ansi-analysis/SKILL.md
new file mode 100644
index 0000000000..1f9d9a0a95
--- /dev/null
+++ b/.github/skills/ansi-analysis/SKILL.md
@@ -0,0 +1,95 @@
+---
+name: ansi-analysis
+description: Analyze Gluten ANSI-mode test results (run
dev/verify-ansi-expressions.sh, parse JSON tracker output, produce root-cause
analysis and fix recommendations). Trigger on user requests like "analyze ANSI
tests", "run ANSI matrix", "why is this ANSI test failing".
+---
+
+# ANSI Test Analysis Skill
+
+## Step 0 — MUST READ FIRST: shared analysis prompt
+
+Before doing anything else, read the shared prompt that defines the analysis
output format and reference source locations:
+
+```
+.github/skills/ansi-analysis/shared.md
+```
+
+This file is the single source of truth — the same content is consumed by the
CI Python pipeline (`.github/scripts/analyze-ansi.py --ai-analysis`). Your
output structure, reference source locations, and self-investigation steps MUST
follow it. If the file is missing, STOP and tell the user the repo is in a
broken state.
+
+## Step 1 — Decide entry point
+
+Ask the user (or infer from request):
+- Run new tests? → Step 2
+- Re-analyze existing JSON in `target/ansi-offload/`? → Step 3
+- Diagnose a single test failure? → Step 4
+
+## Step 2 — Run the verification script
+
+```bash
+./dev/verify-ansi-expressions.sh <category> <spark41|spark40|all> [--clean]
+```
+
+Categories: `cast | arithmetic | collection | datetime | math | decimal |
string | aggregate | errors | all`
+
+Logs: `/tmp/ansi-matrix/latest/` (bash logs).
+JSON: `target/ansi-offload/*.json` (written by
`GlutenExpressionOffloadTracker.scala`, this is the structured input for
analysis).
+
+Notes from prior runs:
+- Use `all` mode in single JVM (~28 min) when full coverage is needed
+- After rebase / branch switch, run `./dev/builddep-veloxbe-inc.sh` first to
refresh `libvelox.so` / `libgluten.so`
+
+## Step 3 — Analyze JSON results
+
+Two options:
+
+### 3a. Local AI orchestration (this skill, recommended for interactive review)
+
+1. Read `.github/skills/ansi-analysis/shared.md` (Step 0)
+2. List `target/ansi-offload/*.json`
+3. Read each JSON; extract: suite name, total/passed/failed/ignored counts,
per-test `failCause`
+4. Apply the analysis template from shared.md verbatim (sections, tables,
constraints)
+5. For each failure: extract Velox file:line from `failCause`, read those C++
files, verify root cause
+6. **Always** grep `isAnsiSupported` in
`ep/build-velox/build/velox_ep/velox/functions/sparksql/specialforms/SparkCastExpr.cpp`
when the failure involves Cast — most NO_EXCEPTION/Cast failures stem from the
small whitelist there
+7. Output the markdown report
+
+### 3b. Python script (CI / batch)
+
+```bash
+python3 .github/scripts/analyze-ansi.py \
+ --json-dir target/ansi-offload/ \
+ --ai-analysis \
+ --output ansi-report.md
+```
+
+The script loads the same shared prompt and calls the GitHub Models API.
+
+## Step 4 — Single-failure diagnosis
+
+When the user pastes one failing test:
+1. Locate its JSON entry under `target/ansi-offload/`
+2. Apply the self-investigation steps from shared.md (extract Velox file:line,
check `isAnsiSupported`, cross-check `withAnsiEvalMode` in the shim)
+3. Output: Symptom / Root Cause / Fix Point / Representative Tests / Estimated
Impact
+
+## Step 5 — Optional PR comment
+
+If the user wants the report posted to a PR:
+
+```bash
+gh pr comment <pr-number> --body-file ansi-report.md
+```
+
+(or use the GitHub MCP server tool when available)
+
+## Environment requirements
+
+For Step 2 (running tests):
+- `SPARK_ANSI_SQL_MODE=true`
+- `SPARK_TESTING=true`
+- `SPARK_SCALA_VERSION=2.13`
+- JVM: `-Dspark.gluten.sql.ansiFallback.enabled=false`
+- Maven profile: include `-Pdelta`
+
+## What NOT to do
+
+- Do NOT invent reference paths or line numbers — always grep / verify
+- Do NOT skip Step 0 — drift between shared.md and your output is the failure
mode this skill is designed to prevent
+- Do NOT bypass the shared prompt by writing your own analysis structure
diff --git a/.github/skills/ansi-analysis/shared.md
b/.github/skills/ansi-analysis/shared.md
new file mode 100644
index 0000000000..1f2d0496cf
--- /dev/null
+++ b/.github/skills/ansi-analysis/shared.md
@@ -0,0 +1,76 @@
+You are an ANSI mode test analysis expert for the Gluten project. Gluten is a
native engine acceleration plugin for Apache Spark that offloads expression
evaluation to Velox (C++). ANSI mode requires throwing exceptions on overflow,
invalid type casts, etc.
+
+Below is the structured output from JSON expression tests (not XML suite
tests):
+
+```json
+{json_data}
+```
+
+Analyze only the JSON expression test data. Output key findings directly — no
overview section.
+
+Generate analysis in Markdown:
+
+## Key Findings
+- Failure hotspot table (Suite / Failures / Root Cause)
+- failCause type statistics table (Type / Count / % / Interpretation):
+ - WRONG_EXCEPTION: Velox threw an exception but Spark's scheduling layer
wrapped it as SparkException, losing the original exception type
+ - NO_EXCEPTION: Velox did not throw the expected exception in ANSI mode
+ - OTHER: Result mismatch or other errors
+- Root cause deep analysis for WRONG_EXCEPTION (exception wrapping chain path,
key code locations)
+- Breakdown of NO_EXCEPTION by root cause (arithmetic/cast/datetime etc.)
+
+## Fix Recommendations (P0 / P1 / P2 only)
+Each recommendation includes:
+- Symptom: test failure pattern
+- Root Cause: specific code path and logic issue
+- Fix Point: file path + change direction
+- Representative Tests: affected test names
+- Estimated Impact: number of tests that would turn green after fix
+
+Key source locations (for reference):
+
+Spark plan layer (Scala):
+- ANSI Cast/arithmetic detection:
shims/sparkXX/src/main/scala/org/apache/gluten/sql/shims/sparkXX/SparkXXShims.scala
(withAnsiEvalMode). Variants per Spark version: shims/spark34/,
shims/spark35/, shims/spark40/, shims/spark41/
+- ANSI fallback rule:
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/FallbackRules.scala
(enableAnsiMode && enableAnsiFallback check)
+- ANSI config:
gluten-substrait/src/main/scala/org/apache/gluten/config/GlutenConfig.scala
(enableAnsiFallback, GLUTEN_ANSI_FALLBACK_ENABLED =
spark.gluten.sql.ansiFallback.enabled, default true)
+
+Substrait conversion layer (Scala/Java):
+- Expression conversion: gluten-core/.../ExpressionConverter.scala
+- Type mapping: gluten-substrait/.../ConverterUtils.scala (getTypeNode)
+
+Native bridge (Java):
+- Exception lookup:
gluten-ut/common/src/test/scala/org/apache/spark/sql/GlutenTestsTrait.scala
(findCause method)
+- Exception wrapping:
gluten-arrow/src/main/java/org/apache/gluten/vectorized/ColumnarBatchOutIterator.java
(translateException)
+
+C++ Velox layer:
+- ANSI config plumbing: cpp/velox/compute/WholeStageResultIterator.cc
(kSparkAnsiEnabled)
+- ANSI gate function (CRITICAL):
ep/build-velox/build/velox_ep/velox/functions/sparksql/specialforms/SparkCastExpr.cpp
(isAnsiSupported)
+ - Currently only String→{Boolean, Date, Integral} are ANSI-supported
+ - All other casts silently fall back to try_cast when ANSI is on → root
cause of most NO_EXCEPTION failures involving Cast
+ - Always grep `isAnsiSupported` to see the current whitelist (do not trust
hard-coded line numbers)
+- ANSI gate header: ep/build-velox/.../specialforms/SparkCastExpr.h
+- Velox Cast construction: same SparkCastExpr.cpp (constructSpecialForm) —
uses `!config.sparkAnsiEnabled() || !isAnsiSupported(...)` to decide isTryCast
+- Velox Arithmetic: ep/build-velox/.../sparksql/Arithmetic.cpp (uses
kSparkAnsiEnabled)
+- Velox QueryConfig: ep/build-velox/.../core/QueryConfig.h (kSparkAnsiEnabled)
+- Velox tests for reference behavior:
+ - ep/build-velox/.../sparksql/tests/SparkCastExprTest.cpp
+ - ep/build-velox/.../sparksql/tests/ArithmeticTest.cpp
+
+Self-investigation (when stack info is available in failCause):
+
+The failCause field in JSON often contains rich diagnostic info:
+- Velox error code (e.g., INVALID_ARGUMENT, ARITHMETIC_ERROR)
+- Velox file + line (e.g., "File: .../EvalCtx.cpp, Line: 183")
+- Top-level expression context (e.g., "Top-level Expression: checked_add(...)")
+- Java stack trace from ColumnarBatchOutIterator.translateException
+
+You SHOULD:
+1. Extract Velox file path + line number from failCause strings
+2. Read those Velox source files to verify your root cause analysis
+3. Always check `isAnsiSupported()` in SparkCastExpr.cpp when the failure
involves Cast — this function gates which casts honor ANSI semantics. Currently
only String→{Boolean, Date, Integral} are supported; all other ANSI casts
silently fall back to try_cast (most common root cause of NO_EXCEPTION failures
involving Cast). Use grep to locate the current implementation.
+4. Cross-reference with `withAnsiEvalMode` in the appropriate
shims/sparkXX/.../SparkXXShims.scala to confirm the Spark plan sent the
expression with the ANSI tag.
+
+Constraints:
+- Use Markdown tables, no ASCII box drawing characters
+- Maximum 3 fix recommendations
+- If source code is accessible, read key files to verify root cause analysis
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
