zuozhiw commented on code in PR #5377: URL: https://github.com/apache/texera/pull/5377#discussion_r4346789865
########## bin/single-node/docker-compose.override.yml: ########## @@ -0,0 +1,29 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Local-dev override, auto-loaded by `docker compose` when run from this +# directory. It publishes the OTel collector's OTLP receiver ports on the +# host loopback so a backend running outside docker (sbt / IntelliJ) can +# emit telemetry to the collector. The released bundle does not ship this +# file, so the base compose stays network-clean in a deploy. +name: texera-single-node + +services: + otel-collector: + ports: Review Comment: These port mappings are not loaded by `bin/local-dev.sh`, so a JVM running on the developer's host cannot reach the collector at `127.0.0.1:4317`. For example, the collector container can be running and telemetry can be enabled in the host JVM, yet its exports fail because that port is not published to the host. `bin/local-dev/main.sh` explicitly selects `bin/single-node/docker-compose.yml` and `bin/local-dev/docker-compose.override.yml`. The mappings here are in `bin/single-node/docker-compose.override.yml`, which that command does not include. Please move/add the mappings to the local-dev override the launcher actually uses, or explicitly include this file there. A configuration test should render that exact file combination and check the published collector ports. ########## bin/k8s/templates/base/observability/loki.yaml: ########## @@ -0,0 +1,168 @@ +{{/* +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +*/}} + +{{- if and .Values.observability.enabled .Values.observability.logs.enabled }} +# Loki: log store, k8s counterpart of the docker-compose loki service. +# OTLP ingest (/otlp) + LogQL query API on one port. Loki is AGPLv3, so +# it is an optional operator-run component, not part of any Texera release +# artifact (see bin/observability/README.md). Only this ASF-licensed +# template lives in the tree. +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ .Release.Name }}-loki-config + namespace: {{ .Release.Namespace }} + labels: + app: {{ .Release.Name }}-loki +data: + config.yaml: | + # No multi-tenancy: one implicit tenant for the in-cluster stack. + auth_enabled: false + server: + http_listen_port: {{ .Values.observability.logs.port }} + grpc_listen_port: 9096 + common: + instance_addr: 127.0.0.1 + # Every component's data path defaults under this prefix, which is + # the PVC mounted at /loki below. + path_prefix: /loki + storage: + filesystem: + chunks_directory: /loki/chunks + rules_directory: /loki/rules + replication_factor: 1 + ring: + kvstore: + store: inmemory + schema_config: + configs: + - from: 2024-01-01 + store: tsdb + object_store: filesystem + schema: v13 + index: + prefix: index_ + period: 24h + limits_config: + # Required for OTLP ingest (OTLP attributes -> structured metadata). + allow_structured_metadata: true + reject_old_samples: false + compactor: Review Comment: The configured 30-day log retention is not applied. `observability.logs.retentionPeriod` defaults to `30d`, but this template never uses it, and the Compose Loki config also lacks retention settings. Loki disables retention cleanup by default, so logs keep accumulating. For example, an operator expecting old logs to disappear after a month would instead keep consuming disk until the volume fills and log ingestion fails. Please use the configured value for `limits_config.retention_period` and enable deletion of expired logs with `compactor.retention_enabled`, including the required `compactor.delete_request_store`. Apply the equivalent settings to Compose. See [Loki's retention configuration](https://grafana.com/docs/loki/latest/operations/storage/retention/). A regression check of the generated settings is sufficient here; it does not need to wait for logs to expire. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
