Copilot commented on code in PR #13057: URL: https://github.com/apache/gluten/pull/13057#discussion_r4049186101
########## .github/workflows/util/delta-spark-ut/apply-delta-test-patches.sh: ########## @@ -0,0 +1,183 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# +# Applies temporary Gluten compatibility patches to a delta-io/delta checkout. +# +# Usage: +# apply-delta-test-patches.sh <delta_dir> <delta_ref> +# +# Remove each patch group when DELTA_REF contains the corresponding upstream +# fix or Gluten no longer needs the workaround. +# + +set -euo pipefail + +if [ "$#" -ne 2 ]; then + echo "Usage: $0 <delta_dir> <delta_ref>" >&2 + exit 1 +fi + +DELTA_DIR="$1" +DELTA_REF="$2" + +# Delta's tests collect file-source scans by matching the concrete +# `FileSourceScanExec` case class; Gluten offloads the scan to +# DeltaScanTransformer, a `FileSourceScanLike` sibling, so those matches miss +# (`scala.MatchError: List()`, empty partition filters, broken column-pruning / +# scan-metric checks across many suites). delta-io/delta#7104 and #7105 widen the +# matches to the shared `FileSourceScanLike` interface that both the vanilla and +# Gluten scans implement (behavior-preserving for vanilla). Both are merged +# upstream but land after the pinned DELTA_REF (v4.2.0), so apply them here; once +# DELTA_REF includes a fix, cherry_pick_delta_fix detects it and skips (see below). +# +# Depth-2 fetch brings each fix commit and its parent, which cherry-pick needs to +# diff against (a depth-1 fetch grafts the parent away); `-n` stages the change +# without requiring a committer identity. +cherry_pick_delta_fix() { + local sha="$1" pr="$2" + git -C "$DELTA_DIR" fetch --quiet --depth 2 origin "$sha" + echo "Cherry-picking delta-io/delta${pr}" + if git -C "$DELTA_DIR" cherry-pick -n "$sha"; then + return 0 + fi + # The cherry-pick did not apply. The usual cause is that the pinned DELTA_REF + # already contains this fix (e.g. after a version bump), which makes the patch + # empty/conflicting and would -- under `set -e` -- abort the whole setup. We + # can't use ancestry to tell "already contained" from a genuine conflict here + # (the clone is shallow, so `merge-base --is-ancestor` can't see past the graft), + # so recover the exact paths this fix touches -- leaving other setup such as the + # DeltaSQLCommandTest patch intact -- and continue. This is self-correcting: if + # the fix is genuinely still needed, the FileSourceScanLike failures it prevents + # resurface as gate regressions rather than being hidden by a hard abort here. + echo "Cherry-pick of delta-io/delta${pr} did not apply cleanly" \ + "(most likely already contained in ${DELTA_REF}); skipping it." + local f + while IFS= read -r f; do + [ -n "$f" ] || continue + git -C "$DELTA_DIR" reset -q -- "$f" 2>/dev/null || true + git -C "$DELTA_DIR" checkout -q -- "$f" 2>/dev/null || true + done < <(git -C "$DELTA_DIR" diff-tree --no-commit-id --name-only -r "$sha") + # Clear any leftover sequencer state (harmless if none exists). + git -C "$DELTA_DIR" cherry-pick --quit 2>/dev/null || true + return 0 +} + +echo "::group::Cherry-picking upstream Delta FileSourceScanLike test fixes" +cherry_pick_delta_fix 46bd45d57eadd7e528002a0ae7bd36ce5a456eca "#7104 (ScanReportHelper.collectScans)" +cherry_pick_delta_fix 959e00e15f41f56afc1c9bb95d160c55c6dc7068 "#7105 (9 more test suites)" Review Comment: This SHA is not the merged #7105 commit (`959e00e15f41f56afc1c9bb95d160c55c6dc7068`); the final digit is `9` here instead of `8`. The `git fetch` in `cherry_pick_delta_fix` therefore fails under `set -e`, so setup aborts before applying the remaining patches or running Delta tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
