felipepessoto commented on code in PR #12388: URL: https://github.com/apache/gluten/pull/12388#discussion_r3617361437
########## .github/workflows/util/delta-spark-ut/run-delta-tests.sh: ########## @@ -0,0 +1,240 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# +# Runs the Delta `spark` module tests for one shard under the Gluten bundle: +# arms a hang watchdog (thread-dumps + kills a wedged fork), invokes sbt +# spark/test with the tuned JVM/heap flags, prints cgroup memory forensics, and +# then gates the results against the baseline (compare-test-results.py). Extracted +# from delta_spark_ut.yml so the workflow step stays readable. +# +# Driven by environment (set by the workflow step / job): +# SHARD_ID - this shard's id (matrix.shard) +# SPARK_VERSION - Delta -DsparkVersion value +# UPDATE_BASELINE - 'true' -> gate seed mode; else enforce +# FAIL_ON_FIXED - passed through to the gate +# DELTA_SCALA_VERSION, NUM_SHARDS, TEST_PARALLELISM_COUNT, DELTA_TESTING +# - test env (see the workflow step's `env:` block) +# GITHUB_WORKSPACE - repo root (holds the Delta clone + util scripts) +# +# JAVA_TOOL_OPTIONS is set by sourcing java-test-args.sh (below), not the caller. + +set -euo pipefail +export JAVA_HOME=/usr/lib/jvm/java-17-openjdk +export PATH=$JAVA_HOME/bin:$PATH +# Gluten/JDK17 test JVM flags (--add-opens + Netty property), shared with local +# dev runs. Sets JAVA_TOOL_OPTIONS so it reaches the sbt launcher + forked JVMs. +# shellcheck source=./java-test-args.sh +source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/java-test-args.sh" +cd "$GITHUB_WORKSPACE/delta" +chmod +x build/sbt +# Only run the unified `spark` sbt project, NOT `sparkGroup/test` -- +# `sparkGroup` aggregates many other projects (sparkV2, contribs, +# sharing, connect*, ...) that are out of scope for this pipeline. +# +# JVM heap layout -- two memory consumers on the ~16G runner: +# * sbt launcher JVM: -J-Xmx4G for the test compile, then forced to +# return idle memory during the (long) test phase via G1 periodic GC +# (G1PeriodicGCInterval=10s; G1PeriodicGCSystemLoadThreshold=0 so the +# busy fork doesn't suppress it; -XX:-G1PeriodicGCInvokesConcurrent +# forces each periodic GC to a full STW collection) that uncommits to a +# tight free ratio (Min/MaxHeapFreeRatio 5/15, JEP 346) above a low +# -Xms512m floor. +# Without this the idle launcher holds ~5.3G for the whole run; with +# it, it drops back to ~1-2G. These flags touch no Gluten/Spark runtime +# config, so they cannot affect the measured pass/fail signal. +# * Forked test JVM: -Xmx2G via the `set ... Test / javaOptions` command +# below. Delta caps its fork at -Xmx1024m in build.sbt; `++=` appends +# so our -Xmx2G comes last and wins. Gluten offloads data to Velox +# off-heap (capped at 2g via spark.memory.offHeap.size in the patched +# DeltaSQLCommandTest), so the fork's heap need is modest. A larger +# fork heap pushed the cgroup peak past the ~16G OOM threshold and the +# kernel OOM-killed the fork mid-shard (no hs_err), wedging sbt -- 2G +# keeps headroom. Keep heap-dump-on-OOM so a real >2G heap OOM is +# analyzable. +# `-u target/test-reports` enables ScalaTest's JUnit XML reporter so +# every suite writes per-test results. Delta itself only configures +# the console reporter (-oDF), so without this we'd have no machine- +# readable results to gate on. The path is relative to the forked +# test JVM's working dir (Test / baseDirectory = spark/), i.e. +# delta/spark/target/test-reports/TEST-*.xml. +# +# We deliberately do NOT let an sbt non-zero exit (which fires on the +# MANY expected Delta-on-Gluten failures) fail this step directly. +# Instead the known-failures gate below decides pass/fail: the build +# is green when the only failures are ones already recorded in the +# baseline, and red on a genuine regression. +set +e +# --- hang watchdog --------------------------------------------------- +# Shard 2 (and occasionally others) hangs indefinitely after a suite's +# last test with no further output. ScalaTest's failAfter only wraps +# individual test BODIES, so a wedge in suite teardown/afterAll -- or in +# a non-interruptible native Velox/JNI call that ignores +# Thread.interrupt() -- has no timeout and stalls until the 350-min job +# limit with zero diagnostics. This watchdog dumps the forked test JVM's +# threads (to the job log, and to a file for the artifact) once the test +# output has been silent for too long, so the deadlock is diagnosable. +SBT_LOG="/tmp/sbt-spark-test-shard-${SHARD_ID}.log" +: > "$SBT_LOG" +rm -f /tmp/sbt-done +( + # CRITICAL: the step shell runs with `bash -eo pipefail`, which the + # subshell inherits. Without `set +e` here, ANY non-zero command -- + # e.g. fork detection finding no match, or `kill`/`jps` returning + # non-zero -- silently kills this watchdog. That errexit kill (plus a + # /proc detection miss) once made the watchdog capture ZERO dumps. A + # diagnostic must never abort on a failed probe. + set +e +o pipefail + JSTACK="${JAVA_HOME}/bin/jstack" + JPS="${JAVA_HOME}/bin/jps" + silent_limit=900 # 15 min with no new test output => treat as hung + dumps=0 + fork_pids() { + # The sbt test fork's main class is sbt.ForkMain. Prefer jps (reads + # the main class from hsperfdata, robust to sbt's @argfile launch); + # fall back to scanning /proc cmdline + @argfile. + "$JPS" -l 2>/dev/null | awk '/sbt\.ForkMain/ {print $1}' + local p cl arg + for p in /proc/[0-9]*; do + [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] || continue + cl="$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null)" + case "$cl" in *sbt.ForkMain*) echo "${p##*/}"; continue ;; esac + arg="$(printf '%s' "$cl" | tr ' ' '\n' | sed -n 's/^@//p' | head -1)" + [ -n "$arg" ] && [ -f "$arg" ] && grep -qa 'sbt\.ForkMain' "$arg" 2>/dev/null \ + && echo "${p##*/}" + done + } + all_java_pids() { + "$JPS" -q 2>/dev/null + local p + for p in /proc/[0-9]*; do + [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] && echo "${p##*/}" + done + } + echo "HANG WATCHDOG armed: dumps the test JVM after ${silent_limit}s of output silence" + hb=0 + while [ ! -f /tmp/sbt-done ]; do + sleep 60 + [ -f "$SBT_LOG" ] || continue + now=$(date +%s) + mtime=$(stat -c %Y "$SBT_LOG" 2>/dev/null || echo "$now") + silent=$(( now - mtime )) + # Per-minute memory profile: heap tuning proved the ~16G OOM peak is + # NATIVE-driven, so log which JVM (sbt launcher vs fork) actually grows + # toward it -- the last lines before a hang reveal the real hog to cut. + # Read /proc directly (no `ps` dependency in the minimal container). + memnow=$(awk '{printf "%.2fG",$1/1073741824}' /sys/fs/cgroup/memory.current 2>/dev/null) + jvmrss="" + for mp in $(all_java_pids 2>/dev/null | sort -un); do + r=$(awk '/^VmRSS:/{print $2}' "/proc/$mp/status" 2>/dev/null) + [ -n "$r" ] && jvmrss="$jvmrss $(( r / 1024 ))M(p$mp)" + done + echo "MEM cgroup=${memnow} JVMs=[${jvmrss# }]" + hb=$(( hb + 1 )) + # Heartbeat every ~5 min so we can SEE the watchdog is alive (and how + # long the test has been silent) without waiting for a hang. + [ $(( hb % 5 )) -eq 0 ] && echo "HANG WATCHDOG: alive; last test output ${silent}s ago" + if [ "$silent" -ge "$silent_limit" ] && [ "$dumps" -lt 3 ]; then + dumps=$(( dumps + 1 )) + pids="$(fork_pids | sort -un)" + # Safety net: if the fork JVM cannot be pinpointed, dump EVERY JVM. + [ -n "$pids" ] || pids="$(all_java_pids | sort -un)" + echo "::group::HANG WATCHDOG: test output silent ${silent}s -- thread dump #${dumps} (pids:$(printf ' %s' $pids))" + [ -n "$pids" ] || echo "HANG WATCHDOG: no java process found to dump" Review Comment: Agreed. I split the dump set from the kill set: we still dump *every* JVM for diagnostics when no `sbt.ForkMain` fork can be pinpointed, but we now only KILL matched fork pids. During a pre-fork stall (dependency resolution or a cold-cache compile) the watchdog dumps and leaves sbt running instead of killing the launcher, so we don't waste the slot on a confusing "compile/launch failure". Only a real wedged fork is killed (and only then is the fail-the-shard marker set). I also made the per-episode dump/kill budget reset when output resumes, so a transient pre-fork stall we dump-but-don't-kill can't starve the budget for a real fork hang later. (73e5c90) ########## .github/workflows/util/delta-spark-ut/compare-test-results.py: ########## @@ -0,0 +1,747 @@ +#!/usr/bin/env python3 +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Gate / seed / aggregate the Delta-on-Gluten unit test results. + +Running delta-io/delta's ScalaTest suite against the Gluten Velox bundle +produces many *expected* failures (Gluten does not yet support every Delta +code path). To keep the red/green signal meaningful while we fix those +failures incrementally, we maintain a committed baseline of known failing +tests (``known-failures.txt``) and compare each CI run against it. + +This script has three modes: + +``enforce`` (default, per shard) + Parse the JUnit XML produced by ``sbt spark/test`` (ScalaTest ``-u`` + reporter) and compare against the baseline: + + * regression -- a test that FAILED but is NOT in the baseline. These + fail the build: a previously-passing test just started failing. + * expected -- a test that failed and IS in the baseline. Ignored. + * fixed -- a baseline test that now PASSES. By default these also + fail the build (``--fail-on-fixed true``) so the baseline stays honest + and contributors remove entries as they fix them. + + If the baseline file exists but is empty (not yet bootstrapped) the mode + automatically degrades to ``seed`` so the first run is never spuriously red. + A *missing* ``--known-failures`` file is treated as a configuration error + (the gate fails) so a mis-referenced path can't silently pass. + +``seed`` (bootstrap / ``update_baseline``) + Never fails. Just writes the current shard's failing tests so the baseline + can be (re)generated from a real run. + +``aggregate`` (final job) + Merge every shard's ``--failures-out`` / ``--ran-out`` file into a single, + sorted, ready-to-commit ``known-failures.txt`` and report stale baseline + entries (tests no longer present in any shard). Pass ``--expected-shards N`` + to fail when fewer than ``N`` shards contributed gate lists (a shard that + died before writing them), so an incomplete baseline is never produced. + +Flaky quarantine (``--flaky-tests``) + Some Delta-on-Gluten failures are non-deterministic (e.g. a native bug that + only triggers on certain runtime plans), so they are neither a stable pass + nor a stable failure and cannot live in the baseline: baselining them turns + the gate red on every run where they pass, and leaving them out turns it red + on every run where they fail. ``flaky-tests.txt`` quarantines them -- a + quarantined test never counts as a regression (when it fails) nor as + now-passing (when it passes), and is excluded from the regenerated baseline. + Its SUITE is an fnmatch glob (so one line covers a root-cause family across + generated suite variants); its TEST name is matched exactly. + +Flaky quarantine by error signature (``--flaky-error-patterns``) + When a failure is caused by a known nondeterministic bug that surfaces on a + *different test each run* (e.g. the native Delta DV bitmap row-index error), + matching by test name is whack-a-mole. ``flaky-error-patterns.txt`` instead + quarantines by root cause: each line is a regex matched against a failed + test's <failure>/<error> text, and any failure that matches is treated as + flaky regardless of which test it landed on. + +Baseline file format (``known-failures.txt``):: + + # comment lines start with '#' + <fully.qualified.SuiteName>#<test display name> + +The suite is always a JVM class name (dot-separated, never starts with '#'), +so a line whose first non-space character is '#' is unambiguously a comment, +and the FIRST '#' after the suite separates suite from the (possibly +'#'-containing) test name. + +Only the Python standard library is used so the script runs in the bare +centos image used by the Delta UT pipeline with no ``pip install``. +""" + +import argparse +import fnmatch +import glob +import os +import re +import sys +import xml.etree.ElementTree as ET + +# Synthetic "test name" recorded when a whole suite aborts (e.g. beforeAll +# throws) so that the JUnit XML reports a suite-level error with no per-test +# <testcase>. Without this, a suite that used to pass but now aborts entirely +# would record zero failing testcases and the regression would be missed. +SUITE_ABORTED = "<suite aborted>" + + +class NoReportsError(RuntimeError): + """Raised when no JUnit <testsuite> elements are found under reports_dir.""" + + +class CorruptReportError(NoReportsError): + """Raised when an expected JUnit report file (TEST-*.xml) fails to parse. + + Subclasses NoReportsError so the enforce/seed handler treats a truncated + report as a hard data error (exit 2) instead of silently dropping the + suite's results and letting the gate pass on partial data. + """ + + +SEP = "#" + + +def eprint(*args, **kwargs): + print(*args, file=sys.stderr, **kwargs) + + +# --------------------------------------------------------------------------- # +# Baseline (known-failures.txt) parsing / formatting +# --------------------------------------------------------------------------- # +def format_entry(suite, test): + return "{}{}{}".format(suite, SEP, test) + + +def parse_entry(line): + """Parse a 'suite#test' line into (suite, test) or return None for blanks/comments.""" + stripped = line.strip() + if not stripped or stripped.startswith("#"): + return None + idx = stripped.find(SEP) + if idx < 0: + # No separator: treat the whole line as a suite-level entry. + return (stripped, SUITE_ABORTED) + return (stripped[:idx], stripped[idx + len(SEP) :]) + + +def load_entries(path): + """Load a set of (suite, test) tuples from a baseline/shard-list file.""" + entries = set() + if not path or not os.path.exists(path): + return entries + with open(path, "r", encoding="utf-8") as fh: + for line in fh: + parsed = parse_entry(line) + if parsed is not None: + entries.add(parsed) + return entries + + +def make_is_flaky(flaky_entries): + """Build a predicate that matches a (suite, test) tuple against flaky entries. + + A flaky entry quarantines a test whose failure is known to be non-deterministic + (see flaky-tests.txt). The entry's SUITE is treated as an fnmatch glob so a + single line can cover a root-cause family across generated suite variants + (e.g. ``*DVs*Suite`` matches every deletion-vector merge suite, ``*`` matches + any suite); the TEST name is matched exactly (test names are freeform and may + contain glob metacharacters, so they are never globbed). + """ + exact = set() + globbed = [] + for suite, test in flaky_entries: + if any(ch in suite for ch in "*?["): + globbed.append((suite, test)) + else: + exact.add((suite, test)) + + def is_flaky(entry): + if entry in exact: + return True + suite, test = entry + for glob_suite, glob_test in globbed: + if test == glob_test and fnmatch.fnmatchcase(suite, glob_suite): + return True + return False + + return is_flaky + + +def load_patterns(path): + """Load flaky-error regex patterns from a file (one per line). + + Blank lines and ``#`` comments are ignored. Each remaining line is compiled + as a case-sensitive regex. These match the FAILURE TEXT of a failed test + (its JUnit <failure>/<error> message + stack), so a test that fails with a + known-nondeterministic native error (e.g. the Delta DV bitmap row-index bug) + can be quarantined by root cause instead of by exact test name. + """ + patterns = [] + if not path or not os.path.exists(path): + return patterns + with open(path, encoding="utf-8") as fh: + for line in fh: + line = line.rstrip("\n") + if not line.strip() or line.lstrip().startswith("#"): + continue + patterns.append(re.compile(line)) + return patterns + + +def make_signature_matcher(patterns): + """Return a predicate matching a failure text against any flaky-error pattern.""" + + def matches(text): + if not text: + return False + return any(p.search(text) for p in patterns) + + return matches + + +def write_entries(path, entries, header=None): + """Write a sorted set of (suite, test) tuples to a file.""" + os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True) + with open(path, "w", encoding="utf-8") as fh: + if header: + for hl in header.splitlines(): + fh.write(hl.rstrip() + "\n") + for suite, test in sorted(entries): + # Defensive: collapse any stray newlines so each entry stays on one line. + safe_test = test.replace("\r", " ").replace("\n", " ") + fh.write(format_entry(suite, safe_test) + "\n") + + +# --------------------------------------------------------------------------- # +# JUnit XML parsing +# --------------------------------------------------------------------------- # +def _iter_testsuites(root): + """Yield every <testsuite> element regardless of whether the file root is + <testsuites> (wrapper) or a single <testsuite>.""" + tag = root.tag.split("}")[-1] # strip any namespace + if tag == "testsuites": + for child in root: + if child.tag.split("}")[-1] == "testsuite": + yield child + elif tag == "testsuite": + yield root + + +def _child_local_tags(elem): + return {c.tag.split("}")[-1] for c in elem} + + +def _failure_text(tc): + """Concatenate the message attribute + body text of a testcase's + <failure>/<error> children, for error-signature matching.""" + parts = [] + for c in tc: + if c.tag.split("}")[-1] in ("failure", "error"): + msg = c.get("message") + if msg: + parts.append(msg) + if c.text: + parts.append(c.text) + return "\n".join(parts) + + +def parse_reports(reports_dir): + """Walk reports_dir for JUnit XML and classify every test. + + Returns (passed, failed, skipped, fail_texts). The first three are sets of + (suite, test) tuples; fail_texts maps each failed (suite, test) to its + combined <failure>/<error> message + stack text (used for error-signature + quarantine). A test is 'failed' if its <testcase> has a <failure> or <error> + child, 'skipped' if it has a <skipped> child, otherwise 'passed'. Suite-level + aborts (a <testsuite> reporting errors/failures with no failing <testcase>) + are recorded as a synthetic (suite, SUITE_ABORTED) failure. + """ + passed, failed, skipped = set(), set(), set() + fail_texts = {} + + xml_files = [] + # ScalaTest's -u reporter and Maven surefire both write `TEST-<suite>.xml` + # under a `target/.../*-reports/` dir. Restrict the secondary glob to + # `target/` so we never parse Delta's own XML *test resources* (which live + # under src/test/resources and are not reports). The <testsuite>-root guard + # below is a final safety net. + for pattern in ("**/TEST-*.xml", "**/target/**/*.xml"): + xml_files.extend(glob.glob(os.path.join(reports_dir, pattern), recursive=True)) + xml_files = sorted(set(xml_files)) + + parsed_any = False + for xml_file in xml_files: + try: + tree = ET.parse(xml_file) + except ET.ParseError as exc: + # A TEST-*.xml that fails to parse is almost always a report truncated + # when a forked test JVM was killed mid-write (e.g. OOM). Silently + # skipping it drops that suite's results and could let the gate go + # green on partial data, so fail hard for report files. Other XML that + # merely matched the broad `target/**` glob is still skipped. + if os.path.basename(xml_file).startswith("TEST-"): + raise CorruptReportError( + "corrupt or truncated JUnit report {}: {}. Refusing to " + "evaluate the gate on partial data.".format(xml_file, exc) + ) + eprint("WARNING: could not parse {}: {}".format(xml_file, exc)) + continue + root = tree.getroot() + root_tag = root.tag.split("}")[-1] + if root_tag not in ("testsuites", "testsuite"): + continue # not a JUnit report + + for ts in _iter_testsuites(root): + parsed_any = True + suite_name = ts.get("name") or "" + suite_has_failing_tc = False + for tc in ts: + if tc.tag.split("}")[-1] != "testcase": + continue + suite = tc.get("classname") or suite_name + name = tc.get("name") or "" + key = (suite, name) Review Comment: Nice one — subtle but real. Added a `normalize_key(suite, test)` helper that runs the XML-derived key through the same round-trip the baseline uses (`write_entries` collapses CR/LF in the test name to spaces; `parse_entry` strips the whole `suite#test` line), applied at both key sites (the testcase key and the suite-abort key). It's a no-op for normal names, but now a name with a trailing newline/whitespace normalizes to match a line pasted from the gate's `REGRESSION` output, so it's actually suppressible. Verified with an enforce-mode run using a report whose testcase `name` carries a trailing newline. (73e5c90) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
