bobbai00 commented on code in PR #4274:
URL: https://github.com/apache/texera/pull/4274#discussion_r3070429874
##########
bin/single-node/.env:
##########
@@ -58,6 +58,18 @@
STORAGE_ICEBERG_CATALOG_POSTGRES_URI_WITHOUT_SCHEME=texera-postgres:5432/texera_
STORAGE_ICEBERG_CATALOG_POSTGRES_USERNAME=texera
STORAGE_ICEBERG_CATALOG_POSTGRES_PASSWORD=password
+LAKEKEEPER__PG_DATABASE_URL_READ=postgres://texera:password@texera-postgres:5432/texera_lakekeeper
Review Comment:
please group the env vars together. e.g. for Lakekeeper put their relevent
env var together and put comments
##########
bin/single-node/docker-compose.yml:
##########
@@ -75,6 +75,142 @@ services:
timeout: 5s
retries: 10
+ # Lakekeeper migration init container
+ # This runs once to migrate the database before the lakekeeper server starts
+ lakekeeper-migrate:
+ image: vakamo/lakekeeper:v0.11.0
+ container_name: texera-lakekeeper-migrate
+ depends_on:
+ postgres:
+ condition: service_healthy
+ env_file:
+ - .env
+ restart: "no"
+ entrypoint: ["/home/nonroot/lakekeeper"]
+ command: ["migrate"]
+
+ # Lakekeeper is the Iceberg REST catalog service
+ lakekeeper:
+ image: vakamo/lakekeeper:v0.11.0
+ container_name: texera-lakekeeper
+ restart: always
+ depends_on:
+ postgres:
+ condition: service_healthy
+ minio:
+ condition: service_started
+ lakekeeper-migrate:
+ condition: service_completed_successfully
+ env_file:
+ - .env
+ entrypoint: ["/home/nonroot/lakekeeper"]
+ command: ["serve"]
+ ports:
+ - "8181:8181"
+ healthcheck:
+ test: ["CMD", "/home/nonroot/lakekeeper", "healthcheck"]
+ interval: 10s
+ timeout: 5s
+ retries: 10
+ start_period: 10s
+
+ lakekeeper-init:
+ image: alpine:3.19
+ container_name: texera-lakekeeper-init
+ depends_on:
+ lakekeeper:
+ condition: service_healthy
+ minio:
+ condition: service_started
+ env_file:
+ - .env
+ restart: "no"
+ entrypoint: [ "/bin/sh", "-c" ]
+ command:
+ - |
+ set -e
+
+ echo "Installing dependencies..."
+ apk add --no-cache curl ca-certificates
+
+ echo "Installing MinIO Client..."
+ wget -q https://dl.min.io/client/mc/release/linux-amd64/mc -O
/usr/local/bin/mc
+ chmod +x /usr/local/bin/mc
+
+ check_status() {
+ if [ "$$1" -ge 200 ] && [ "$$1" -lt 300 ]; then
+ echo "Created $$2 successfully (HTTP $$1)."
+ elif [ "$$1" -eq 409 ]; then
+ echo "$$2 already exists (HTTP 409). Treating as success."
+ else
+ echo "Failed to create $$2. HTTP Code: $$1"
+ echo "ERROR RESPONSE:"
+ if [ -f /tmp/response.txt ]; then cat /tmp/response.txt; fi
+ echo ""
+ exit 1
+ fi
+ }
+
+ echo "Step 1: Initializing MinIO bucket
'$$STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET'..."
+ mc alias set minio "$$STORAGE_S3_ENDPOINT"
"$$STORAGE_S3_AUTH_USERNAME" "$$STORAGE_S3_AUTH_PASSWORD" || true
+ if mc ls minio/$$STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET > /dev/null
2>&1; then
+ echo "MinIO bucket '$$STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET'
already exists."
+ else
+ mc mb minio/$$STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET || {
+ echo "Failed to create MinIO bucket
'$$STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET'"
+ exit 1
+ }
+ echo "MinIO bucket '$$STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET'
created successfully."
Review Comment:
for initializing the minio bucket, can we put it as the job for minio, not
for Lakekeeper? There should be a way for minio to create a bucket when it gets
started for the 1st time
##########
bin/single-node/.env:
##########
@@ -58,6 +58,18 @@
STORAGE_ICEBERG_CATALOG_POSTGRES_URI_WITHOUT_SCHEME=texera-postgres:5432/texera_
STORAGE_ICEBERG_CATALOG_POSTGRES_USERNAME=texera
STORAGE_ICEBERG_CATALOG_POSTGRES_PASSWORD=password
+LAKEKEEPER__PG_DATABASE_URL_READ=postgres://texera:password@texera-postgres:5432/texera_lakekeeper
+LAKEKEEPER__PG_DATABASE_URL_WRITE=postgres://texera:password@texera-postgres:5432/texera_lakekeeper
+LAKEKEEPER__PG_ENCRYPTION_KEY=texera_key
+LAKEKEEPER_BASE_URI=http://texera-lakekeeper:8181
+STORAGE_ICEBERG_CATALOG_REST_WAREHOUSE_NAME=texera
+STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET=texera-iceberg
+STORAGE_S3_AUTH_USERNAME=texera_minio
+STORAGE_S3_AUTH_PASSWORD=password
Review Comment:
can you remove the MINIO's related env vars and unify to use this one you
are introducing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
