wombatu-kun commented on code in PR #20621: URL: https://github.com/apache/hudi/pull/20621#discussion_r4573883714
########## hudi-lakehouse/docker/compose.yaml: ########## @@ -0,0 +1,167 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +name: hudi-lakehouse + +# Six default services and one optional Ollama service. Startup order: +# minio -> minio-init -> hive-metastore -> spark-connect -> seed -> gateway. +# minio-init and seed exit after completing their work; the other services stay running. +services: + # S3-compatible storage for Hudi data, log files and timelines. + # S3 is internal at http://minio:9000; the console defaults to localhost:9001. + minio: + image: ${MINIO_IMAGE:-cgr.dev/chainguard/minio:latest@sha256:fb893e36fc369d41e92f678b17ec27378fa5bf0576aeef7ac15460c13629ab5a} + command: [server, /data, --console-address, ':9001'] + environment: + MINIO_ROOT_USER: admin + MINIO_ROOT_PASSWORD: password + volumes: + - minio-data:/data + ports: + - '127.0.0.1:${MINIO_CONSOLE_PORT:-9001}:9001' + + # One-shot job: wait for MinIO, then create the warehouse bucket if absent. + minio-init: + image: ${MINIO_CLIENT_IMAGE:-cgr.dev/chainguard/minio-client:latest-dev@sha256:fe649f71872c299489deb3cb9e057e5d2f76a45c85b7599f7c21616ac80b5e63} + depends_on: + minio: + condition: service_started + entrypoint: [/bin/sh, -ec] + command: + - | + for attempt in $$(seq 1 60); do + if mc alias set local http://minio:9000 admin password; then + mc mb --ignore-existing local/warehouse + exit 0 + fi + sleep 2 + done + echo 'MinIO did not become ready' >&2 + exit 1 + + # Stores database/table schemas, locations and partition metadata in persistent + # Derby storage. Serves the internal Hive Metastore Thrift endpoint on port 9083. + hive-metastore: + image: apache/hive:3.1.3 + # The existing local-dev HMS image is amd64-only (emulated on ARM). + platform: linux/amd64 + depends_on: + minio-init: + condition: service_completed_successfully + user: '0:0' + entrypoint: [/bin/bash, /lakehouse/start-metastore.sh] + environment: + SERVICE_NAME: metastore + volumes: + - metastore-data:/opt/hive/data + # Shared schema/server settings, including HMS's own S3A access to MinIO. + - ./conf/hive-site.xml:/opt/hive/conf/hive-site.xml:ro + - ./scripts/start-metastore.sh:/lakehouse/start-metastore.sh:ro + healthcheck: + test: [CMD, bash, -c, 'exec 3<>/dev/tcp/127.0.0.1/9083'] + interval: 10s + timeout: 5s + retries: 30 + start_period: 30s + + # Executes SQL with an HMS-backed HoodieCatalog and reads/writes Hudi data in + # MinIO. Runs local[2]; external Connect clients use localhost:15002 by default. + spark-connect: + image: ${SPARK_CONNECT_IMAGE:-hudi-lakehouse-spark-connect:4.1.3-hudi1.2.0} + depends_on: + hive-metastore: + condition: service_healthy + environment: + AWS_ACCESS_KEY_ID: admin + AWS_SECRET_ACCESS_KEY: password + AWS_REGION: us-east-1 + volumes: + - ./conf/spark-defaults.conf:/opt/spark/conf/spark-defaults.conf:ro + - ./scripts/check-connect.py:/lakehouse/check-connect.py:ro + ports: + - '127.0.0.1:${SPARK_CONNECT_PORT:-15002}:15002' + healthcheck: + test: [CMD, python3, /lakehouse/check-connect.py] + interval: 30s + timeout: 30s + retries: 20 + start_period: 30s + stop_grace_period: 30s + + # One-shot Python Connect client: create/populate trips and drivers, then verify + # the sample data. Reuses the Spark image without starting another Spark server. + seed: + image: ${SPARK_CONNECT_IMAGE:-hudi-lakehouse-spark-connect:4.1.3-hudi1.2.0} + depends_on: + spark-connect: + condition: service_healthy + entrypoint: [python3, /lakehouse/seed.py] Review Comment: `seed.py` runs as PID 1 here, where the kernel ignores a default-action SIGALRM, so `signal.alarm(600)` never fires and a stalled seed is never stopped. Adding `init: true` to the `seed` service lets the alarm terminate it (exit 142). ########## hudi-lakehouse/docker/lakehouse.sh: ########## @@ -0,0 +1,160 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -euo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ACTION= +BUILD=0 +LOCAL_LLM=0 +CHAT_CONFIG_ERROR= +usage() { + cat <<'HELP' +Usage: lakehouse.sh <up|down|status|logs|test> [--build] [--local-llm] + up Start services, seed demo data, verify MCP and report chat readiness. + --build Build Spark Connect and Gateway images in Docker from this checkout. + --local-llm Start containerized Ollama and pull GATEWAY_LLM_MODEL (default qwen3:8b). + test Verify sample data and gateway MCP without reseeding. + down Remove containers/network; preserve all named data volumes. + status|logs Inspect this Compose installation. + -h, --help Show this help (also shown when no arguments are supplied). +Set COMPOSE_PROJECT_NAME to isolate an installation; configure ports/provider in .env. +An unavailable chat LLM does not fail startup; MCP clients use their own model. +HELP +} +if [[ $# -eq 0 ]]; then + usage + exit 0 +fi +while [[ $# -gt 0 ]]; do + case "$1" in + up|down|status|logs|test) + if [[ -n "$ACTION" ]]; then + echo 'Specify exactly one command.' >&2; usage >&2; exit 2 + fi + ACTION="$1" ;; + --build) BUILD=1 ;; + --local-llm) LOCAL_LLM=1 ;; + -h|--help) usage; exit 0 ;; + *) echo "Unknown argument: $1" >&2; usage >&2; exit 2 ;; + esac + shift +done +if [[ -z "$ACTION" ]]; then + echo 'Specify a command, for example: lakehouse.sh up' >&2 + usage >&2 + exit 2 +fi +command -v docker >/dev/null || { echo 'Install Docker with Compose 2.27.2 or newer first.' >&2; exit 1; } +COMPOSE_VERSION="$(docker compose version --short)" || { + echo 'Docker Compose 2.27.2 or newer is required.' >&2; exit 1; +} +# config --environment was introduced in 2.27.2. Allow Docker Desktop suffixes. +if [[ ! "$COMPOSE_VERSION" =~ ^v?([0-9]+)\.([0-9]+)\.([0-9]+)([-+].*)?$ ]] || + (( 10#${BASH_REMATCH[1]} < 2 || + (10#${BASH_REMATCH[1]} == 2 && (10#${BASH_REMATCH[2]} < 27 || + (10#${BASH_REMATCH[2]} == 27 && 10#${BASH_REMATCH[3]} < 2))) )); then + echo "Docker Compose 2.27.2 or newer is required; found '$COMPOSE_VERSION'. Upgrade Docker Compose and retry." >&2 + exit 1 +fi +MEMORY="$(docker info --format '{{.MemTotal}}')" || { + echo 'Docker is not running or is inaccessible. Start Docker and retry.' >&2; exit 1; +} +COMPOSE=(docker compose --project-directory "$HERE" -f "$HERE/compose.yaml") +if [[ -f "$HERE/.env" ]]; then + COMPOSE+=(--env-file "$HERE/.env") +fi +if [[ "$BUILD" == 1 ]]; then + COMPOSE+=(-f "$HERE/compose.build.yaml") +fi +if [[ "$LOCAL_LLM" == 1 ]]; then + export GATEWAY_LLM_PROVIDER=ollama GATEWAY_OLLAMA_BASE_URL=http://ollama:11434 + COMPOSE+=(--profile local-llm) +fi +case "$ACTION" in + down) "${COMPOSE[@]}" --profile local-llm down; exit ;; + status) "${COMPOSE[@]}" --profile local-llm ps -a; exit ;; + logs) "${COMPOSE[@]}" --profile local-llm logs --tail=100; exit ;; +esac +if [[ "$ACTION" == up ]]; then + if [[ ! "$MEMORY" =~ ^[0-9]+$ ]] || (( MEMORY < 7500000000 )); then + echo 'Allocate at least 8 GB of memory to Docker (more for containerized Ollama).' >&2 + exit 1 + fi + # Auto-select API credentials from the environment only when the user has not + # supplied a .env configuration or an explicit provider. Never persist API keys. + if [[ ! -f "$HERE/.env" && -z "${GATEWAY_LLM_PROVIDER:-}" ]]; then Review Comment: With `OPENAI_API_KEY` or `ANTHROPIC_API_KEY` exported for other tools and no `.env`, `up` silently moves chat off the documented host-Ollama default, and once `GATEWAY_LLM_MODEL` names an Ollama model nothing flags it while every chat request fails. Is the key-based auto-selection needed, or could the installer at least print the provider and model it resolved and say it was auto-selected? ########## hudi-lakehouse/docker/scripts/seed.py: ########## @@ -0,0 +1,96 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Seed the demo tables through Spark Connect; retries never replace existing rows.""" +import argparse +import signal + +from pyspark.sql import SparkSession + + +def seed(spark): + spark.sql("""CREATE TABLE IF NOT EXISTS default.drivers ( + driver_id STRING, driver_name STRING, home_city STRING, + vehicle_type STRING, onboarded_date DATE + ) USING hudi TBLPROPERTIES (type = 'cow', primaryKey = 'driver_id')""").collect() + spark.sql("""CREATE TABLE IF NOT EXISTS default.trips ( + trip_id STRING, rider_id STRING, driver_id STRING, city STRING, + trip_status STRING, fare_amount DECIMAL(10,2), tip_amount DECIMAL(10,2), + distance_km DOUBLE, pickup_ts TIMESTAMP, dropoff_ts TIMESTAMP, trip_date DATE + ) USING hudi PARTITIONED BY (trip_date) + TBLPROPERTIES (type = 'cow', primaryKey = 'trip_id', preCombineField = 'pickup_ts')""").collect() + spark.sql("""INSERT INTO default.drivers + SELECT source.* FROM ( + SELECT * FROM VALUES + ('driver_1', 'Ananya', 'chennai', 'sedan', DATE '2024-01-01'), + ('driver_2', 'Alex', 'san_francisco', 'sedan', DATE '2024-01-01'), + ('driver_3', 'Ana', 'sao_paulo', 'sedan', DATE '2024-01-01') + AS drivers(driver_id, driver_name, home_city, vehicle_type, onboarded_date) + ) source LEFT ANTI JOIN default.drivers target + ON target.driver_id = source.driver_id""").collect() + # Stable keys make reruns safe, including a retry after only one table committed. + # First-run dates are yesterday; subsequent installs preserve the original data. + spark.sql("""INSERT INTO default.trips + SELECT source.* FROM ( + SELECT concat('trip_', CAST(id AS STRING)) AS trip_id, + concat('rider_', CAST(id AS STRING)) AS rider_id, + concat('driver_', CAST(CAST(id / 3 AS INT) + 1 AS STRING)) AS driver_id, + element_at(array('chennai', 'san_francisco', 'sao_paulo'), + CAST(id / 3 AS INT) + 1) AS city, + CASE WHEN id % 3 = 2 THEN 'cancelled' ELSE 'completed' END AS trip_status, + CAST(10 * (CAST(id / 3 AS INT) + 1) + (id % 3) * 10 AS DECIMAL(10,2)) AS fare_amount, + CAST(CAST(id / 3 AS INT) + 1 + id % 3 AS DECIMAL(10,2)) AS tip_amount, + CAST(5 + id AS DOUBLE) AS distance_km, + CAST(date_sub(current_date(), 1) AS TIMESTAMP) AS pickup_ts, + CAST(date_sub(current_date(), 1) AS TIMESTAMP) + INTERVAL 15 MINUTES AS dropoff_ts, + date_sub(current_date(), 1) AS trip_date + FROM range(9) + ) source LEFT ANTI JOIN default.trips target + ON target.trip_id = source.trip_id""").collect() + + +def verify(spark): + assert spark.conf.get("spark.sql.catalog.spark_catalog") == ( + "org.apache.spark.sql.hudi.catalog.HoodieCatalog" + ) + tables = {row.tableName for row in spark.sql("SHOW TABLES IN spark_catalog.default").collect()} + assert {"trips", "drivers"} <= tables, tables + assert spark.sql("SELECT count(*) FROM default.trips").collect()[0][0] == 9 Review Comment: Every `up` reruns the seed and the Gateway waits for it, so editing a demo row through the advertised Spark Connect endpoint (an `UPDATE` on `trips`, say) makes every later `up` fail here, and after a `down` the Gateway and MCP no longer start. Is that intended, or should a mismatch on tables that already existed only warn and leave the strict check to `lakehouse.sh test`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
