This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-8723-564070a56b0a184b1cb1dd4e61a943915a5e79d7 in repository https://gitbox.apache.org/repos/asf/texera.git
commit bcd0ec4f742264503114aa4388d82df8d35e9647 Author: Meng Wang <[email protected]> AuthorDate: Sun Sep 27 23:45:05 2026 +0000 fix(deploy): load example datasets against the account's real email (#8723) ### What changes were proposed in this PR? `bin/single-node/examples/load-examples.sh` derived the owner email that file-service matches dataset lookups against by concatenating `@example.com` onto the username. On a single-node deployment that never matches: the admin seeded from `USER_SYS_ADMIN_USERNAME` has the username itself as its email (`AuthResource.createAdminUser`), so the loader logged in as that admin, skipped its register branch, and every example file upload failed with `400 Dataset not found`. Both example datasets were created but left with no files and no version, so the two example workflows pointed at CSV paths that do not exist. The owner email now comes from the access token's `email` claim, which is whatever the server actually stored — correct for the seeded admin and for an account the script had to register itself. The register call keeps an `@`-shaped address, which the server's format check requires. Second, the loader now exits non-zero once it has logged an error. Each failure is still non-fatal so one bad file does not abandon the rest, but they used to leave no trace: the container printed `=== Example data loading complete ===` and exited 0. That is why the broken lookup went unnoticed for two months. ### Any related issues, documentation, discussions? Closes #8721. Introduced by #6402, which had to give the register call a regex-valid address and changed the shared variable rather than splitting its two roles. `release/v1.3` carries it too. ### How was this PR tested? New `bin/single-node/tests/test_load_examples_sh.sh` (9 assertions, picked up automatically by build.yml's shell-test discovery) drives the loader against a stub `curl` — no docker, no network, no sleeping. It pins the owner email to the token's claim, asserts it is never `<username>@example.com`, and asserts that a failed upload or version create exits non-zero. Both fixes were reverted individually to confirm the relevant assertions go red and the suite exits 1. End to end on a fresh single-node deployment (`down --volumes` then `up --with-examples`): the loader now reports `Owner email resolved as 'texera'` and uploads both files, `dataset_version` goes from 0 rows to 2 (both `v1`), and presigned download of the two paths the example workflows scan returns HTTP 200. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (claude-opus-5) --- bin/single-node/examples/load-examples.sh | 56 ++++++- bin/single-node/tests/test_load_examples_sh.sh | 199 +++++++++++++++++++++++++ 2 files changed, 251 insertions(+), 4 deletions(-) diff --git a/bin/single-node/examples/load-examples.sh b/bin/single-node/examples/load-examples.sh index e6452d1152..93ad591ece 100755 --- a/bin/single-node/examples/load-examples.sh +++ b/bin/single-node/examples/load-examples.sh @@ -27,7 +27,15 @@ TEXERA_DASHBOARD_SERVICE_URL=${TEXERA_DASHBOARD_SERVICE_URL:-"http://dashboard-s TEXERA_FILE_SERVICE_URL=${TEXERA_FILE_SERVICE_URL:-"http://file-service:9092/api"} USERNAME=${TEXERA_EXAMPLE_USERNAME:-"texera"} PASSWORD=${TEXERA_EXAMPLE_PASSWORD:-"texera"} -OWNER_EMAIL="[email protected]" +# Email this script registers the account with, on the path where it has to +# create one. The server requires an `@`-shaped address here. +REGISTER_EMAIL="[email protected]" +# The account's real email, resolved from the access token in authenticate(). +# It is not derivable from $USERNAME: a freshly registered account carries +# $REGISTER_EMAIL, while an admin seeded from USER_SYS_ADMIN_USERNAME carries +# the username itself (AuthResource.createAdminUser). file-service looks +# datasets up by owner email, so guessing it fails every upload (#8721). +OWNER_EMAIL="" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" DATASET_DIR="$SCRIPT_DIR/datasets" @@ -36,6 +44,12 @@ WORKFLOW_DIR="$SCRIPT_DIR/workflows" MAX_RETRIES=60 RETRY_INTERVAL=5 +# Counts print_error calls so main() can exit non-zero. Each failure below is +# deliberately non-fatal so one bad file does not abandon the rest, but they +# used to leave no trace at all: the loader reported success even when nothing +# was uploaded (#8721). +FAILURES=0 + # Color codes for output GREEN='\033[0;32m' RED='\033[0;31m' @@ -47,6 +61,7 @@ print_status() { } print_error() { + FAILURES=$((FAILURES + 1)) echo -e "${RED}[ERROR]${NC} $1" } @@ -54,6 +69,33 @@ print_warn() { echo -e "${YELLOW}[WARN]${NC} $1" } +# Read one claim out of a server-issued JWT's payload. Only the payload is +# decoded: the token came back from our own authenticated call, so there is +# nothing to verify here -- this just avoids guessing a value the server has +# already told us. jq does the base64 work because the `base64` binary's decode +# flag differs between GNU and BSD, and this script's tests run on both. +jwt_claim() { + local payload="$1" claim="$2" + payload=$(printf '%s' "$payload" | cut -d. -f2) + while [ $(( ${#payload} % 4 )) -ne 0 ]; do + payload="${payload}=" + done + printf '%s' "$payload" \ + | jq -Rr --arg c "$claim" '(gsub("-";"+") | gsub("_";"/") | @base64d | fromjson)[$c] // empty' +} + +# Resolve the authenticated account's email. file-service identifies a dataset +# by (ownerEmail, datasetName), so this has to be the address the server +# actually stored for this account. +resolve_owner_email() { + OWNER_EMAIL=$(jwt_claim "$TOKEN" email 2>/dev/null || true) + if [ -z "$OWNER_EMAIL" ]; then + print_error "Could not read the 'email' claim from the access token" + return 1 + fi + print_status "Owner email resolved as '$OWNER_EMAIL'" +} + # Wait for a service to become healthy wait_for_service() { local service_name="$1" @@ -86,17 +128,19 @@ authenticate() { if echo "$LOGIN_RESPONSE" | grep -q '"accessToken"'; then TOKEN=$(echo "$LOGIN_RESPONSE" | grep -o '"accessToken":"[^"]*' | cut -d'"' -f4) print_status "Login successful" + resolve_owner_email || return 1 return 0 fi print_status "User doesn't exist, attempting to register..." REGISTER_RESPONSE=$(curl -s -X POST "$TEXERA_DASHBOARD_SERVICE_URL/auth/register" \ -H "Content-Type: application/json" \ - -d "{\"username\": \"$USERNAME\", \"email\": \"$OWNER_EMAIL\", \"password\": \"$PASSWORD\"}") + -d "{\"username\": \"$USERNAME\", \"email\": \"$REGISTER_EMAIL\", \"password\": \"$PASSWORD\"}") if echo "$REGISTER_RESPONSE" | grep -q '"accessToken"'; then TOKEN=$(echo "$REGISTER_RESPONSE" | grep -o '"accessToken":"[^"]*' | cut -d'"' -f4) print_status "Registration successful" + resolve_owner_email || return 1 return 0 fi @@ -261,8 +305,7 @@ load_workflows() { wid=$(echo "$response" | grep -o '"wid":[0-9]*' | cut -d':' -f2) print_status "Workflow '$workflow_name' created with ID $wid" else - print_error "Failed to create workflow '$workflow_name'" - print_error "Response: $response" + print_error "Failed to create workflow '$workflow_name': $response" fi done @@ -281,6 +324,11 @@ main() { load_datasets load_workflows + if [ "$FAILURES" -ne 0 ]; then + echo -e "${RED}[ERROR]${NC} === Example data loading finished with $FAILURES error(s) ===" + return 1 + fi + print_status "=== Example data loading complete ===" } diff --git a/bin/single-node/tests/test_load_examples_sh.sh b/bin/single-node/tests/test_load_examples_sh.sh new file mode 100755 index 0000000000..a79b6f2aa0 --- /dev/null +++ b/bin/single-node/tests/test_load_examples_sh.sh @@ -0,0 +1,199 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Tests for bin/single-node/examples/load-examples.sh. Run from the repo root: +# bash bin/single-node/tests/test_load_examples_sh.sh +# Exits 0 if every check passes, 1 otherwise. +# +# The loader only talks to Texera over HTTP, so a stub `curl` placed ahead of +# the real one on PATH is enough to drive every branch deterministically -- no +# docker, no network, no sleeping. The two things guarded here both regressed +# silently once (#8721): +# +# * the ownerEmail sent to file-service must be the email the server issued +# in the token, not one concatenated from the username; and +# * a failed upload must make the loader exit non-zero instead of printing +# "complete". + +set -u + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)" +LOADER="$REPO_ROOT/bin/single-node/examples/load-examples.sh" + +PASS=0 +FAIL=0 + +_pass() { printf " \e[32m✓\e[0m %s\n" "$1"; PASS=$((PASS+1)); } +_fail() { + printf " \e[31m✗\e[0m %s\n" "$1" + [[ $# -ge 2 ]] && printf " %s\n" "$2" + FAIL=$((FAIL+1)) +} + +# The loader itself needs jq, so a runner without it cannot exercise anything. +if ! command -v jq >/dev/null 2>&1; then + _pass "skip: jq not installed — load-examples.sh cannot be exercised" + printf "\n%d passed, %d failed\n" "$PASS" "$FAIL" + exit 0 +fi + +# base64url, the JWT flavour: standard base64 with a different alphabet and no +# padding. Built with jq so no `base64` binary is needed (GNU takes -d, BSD -D). +_b64url() { jq -Rr '@base64' | tr '+/' '-_' | tr -d '='; } + +# A syntactically real JWT whose payload is $1. Never verified by anything — +# the loader only reads claims out of a token its own login call returned. +_mk_jwt() { + printf '%s.%s.%s' \ + "$(printf '%s' '{"alg":"HS256","typ":"JWT"}' | _b64url)" \ + "$(printf '%s' "$1" | _b64url)" \ + "c2lnbmF0dXJl" +} + +# One isolated run of the loader: a scratch tree with one dataset and one +# workflow, a stub curl on PATH, and the loader copied in so that its +# SCRIPT_DIR-derived datasets/ and workflows/ point at the scratch tree. +# Behaviour knobs are read from the environment by the stub (STUB_*). +# Called directly (never in a command substitution, which would lose the +# variables): sets $RUN_RC, $RUN_OUT and $RUN_LOG for the caller. The login +# token is $1; any further KEY=VAL arguments become stub knobs. +_run_loader() { + local token="$1"; shift + local tmp + tmp=$(mktemp -d) + mkdir -p "$tmp/bin" "$tmp/examples/datasets/iris-species" "$tmp/examples/workflows" + printf 'sepal,species\n1.0,setosa\n' > "$tmp/examples/datasets/iris-species/Iris.csv" + printf 'an example\n' > "$tmp/examples/datasets/iris-species/description.txt" + printf '{"operators":[],"links":[]}\n' > "$tmp/examples/workflows/Example Workflow.json" + cp "$LOADER" "$tmp/examples/load-examples.sh" + + RUN_LOG="$tmp/requests.log" + : > "$RUN_LOG" + + cat > "$tmp/bin/curl" <<STUB +#!/usr/bin/env bash +# Stub curl: log the requested URL, then answer with the shape the loader +# greps for. Only the cases the loader actually calls are handled. +url="" +for a in "\$@"; do case "\$a" in http*) url="\$a"; break;; esac; done +printf '%s\n' "\$url" >> "$RUN_LOG" + +case "\$url" in + *healthcheck*) printf '200' ;; + */auth/login) + if [ -n "\${STUB_NO_LOGIN:-}" ]; then printf '{"error":"no such user"}' + else printf '{"accessToken":"%s"}' "\${STUB_LOGIN_TOKEN}"; fi ;; + */auth/register) printf '{"accessToken":"%s"}' "\${STUB_REGISTER_TOKEN:-\${STUB_LOGIN_TOKEN}}" ;; + */dataset/list|*/workflow/list) printf '[]' ;; + */dataset/create) printf '{"did":1}' ;; + *multipart-upload/part*) printf '200' ;; + *type=init*) + if [ -n "\${STUB_FAIL_INIT:-}" ]; then printf '{"code":400,"message":"Dataset not found"}' + else printf '{"missingParts":[1],"completedPartsCount":0}'; fi ;; + *type=finish*) printf '{"message":"upload finished"}' ;; + *type=abort*) printf '{"message":"aborted"}' ;; + */version/create) + if [ -n "\${STUB_FAIL_VERSION:-}" ]; then printf '{"code":500}' + else printf '{"datasetVersion":{"dvid":1}}'; fi ;; + */workflow/create) printf '{"wid":1}' ;; + *) printf '{}' ;; +esac +STUB + chmod +x "$tmp/bin/curl" + + RUN_RC=0 + RUN_OUT=$(PATH="$tmp/bin:$PATH" STUB_LOGIN_TOKEN="$token" \ + env "$@" bash "$tmp/examples/load-examples.sh" 2>&1) || RUN_RC=$? +} + +ADMIN_TOKEN=$(_mk_jwt '{"sub":"texera","userId":1,"email":"texera","role":"ADMIN"}') + +# 1) The owner email used for dataset lookups comes from the token, so an +# account whose email is not "<username>@example.com" still gets its files. +# This is the #8721 regression itself: every upload 400'd because the +# loader guessed [email protected] for an admin stored as "texera". +_run_loader "$ADMIN_TOKEN" +if [[ "$RUN_RC" == 0 ]]; then + _pass "all-success run exits 0" +else + _fail "all-success run should exit 0" "rc=$RUN_RC out=$(echo "$RUN_OUT" | tail -3)" +fi + +if grep -q 'ownerEmail=texera&' "$RUN_LOG"; then + _pass "ownerEmail is taken from the token's email claim" +else + _fail "ownerEmail should be the token's email claim" \ + "$(grep -o 'ownerEmail=[^&]*' "$RUN_LOG" | sort -u | tr '\n' ' ')" +fi + +if grep -q '[email protected]' "$RUN_LOG"; then + _fail "ownerEmail must not be concatenated from the username" \ + "$(grep -o 'ownerEmail=[^&]*' "$RUN_LOG" | sort -u | tr '\n' ' ')" +else + _pass "ownerEmail is never guessed as <username>@example.com" +fi + +# 2) A failed upload has to surface in the exit status. The loader keeps going +# so one bad file does not abandon the rest, but it must not claim success. +_run_loader "$ADMIN_TOKEN" STUB_FAIL_INIT=1 +if [[ "$RUN_RC" != 0 ]]; then + _pass "failed upload makes the loader exit non-zero" +else + _fail "failed upload should exit non-zero" "rc=$RUN_RC" +fi +if [[ "$RUN_OUT" == *"error(s)"* && "$RUN_OUT" != *"loading complete"* ]]; then + _pass "failed run reports the error count, not 'complete'" +else + _fail "failed run should not report completion" "$(echo "$RUN_OUT" | tail -2)" +fi + +# 3) Same for a failed version create — the other formerly-silent path. +_run_loader "$ADMIN_TOKEN" STUB_FAIL_VERSION=1 +if [[ "$RUN_RC" != 0 ]]; then + _pass "failed version create makes the loader exit non-zero" +else + _fail "failed version create should exit non-zero" "rc=$RUN_RC" +fi + +# 4) The register fallback still sends an @-shaped address (the server rejects +# anything else), and the owner email then comes from the register token. +REG_TOKEN=$(_mk_jwt '{"sub":"texera","userId":2,"email":"[email protected]","role":"REGULAR"}') +_run_loader "$ADMIN_TOKEN" STUB_NO_LOGIN=1 STUB_REGISTER_TOKEN="$REG_TOKEN" +if [[ "$RUN_RC" == 0 ]] && grep -q '/auth/register' "$RUN_LOG"; then + _pass "register fallback runs when login fails" +else + _fail "register fallback should run and succeed" "rc=$RUN_RC out=$(echo "$RUN_OUT" | tail -3)" +fi +if grep -q 'ownerEmail=texera%40example.com\|[email protected]' "$RUN_LOG"; then + _pass "a registered account's own email is used for lookups" +else + _fail "registered account should use its register email" \ + "$(grep -o 'ownerEmail=[^&]*' "$RUN_LOG" | sort -u | tr '\n' ' ')" +fi + +# 5) A token with no email claim must abort with a clear message rather than +# sending an empty ownerEmail and getting an opaque 400 per file. +_run_loader "$(_mk_jwt '{"sub":"texera","userId":1,"role":"ADMIN"}')" +if [[ "$RUN_RC" != 0 ]] && [[ "$RUN_OUT" == *"email"* ]]; then + _pass "token without an email claim aborts with a clear message" +else + _fail "missing email claim should abort clearly" "rc=$RUN_RC out=$(echo "$RUN_OUT" | tail -2)" +fi + +printf "\n%d passed, %d failed\n" "$PASS" "$FAIL" +(( FAIL == 0 ))
