Saketkr21 commented on code in PR #73259: URL: https://github.com/apache/airflow/pull/73259#discussion_r4101552656
########## providers/dbt/core/README.rst: ########## @@ -0,0 +1,305 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +``apache-airflow-providers-dbt-core`` +===================================== + +Run `dbt Core <https://docs.getdbt.com/docs/core/installation-overview>`__ jobs +on Kubernetes from Apache Airflow. The provider ships a single operator, +:class:`~airflow.providers.dbt.core.operators.dbt.DbtKubernetesRunOperator`, +which mirrors a dbt Cloud job run inside one Kubernetes pod — without dbt Cloud. + +.. contents:: :local: + + +What the operator does +---------------------- + +In a single pod, in order: + +1. **Clone** the dbt project from git (optional; the project may be baked into + the image instead). A git-repo cache on S3/GCS lets runs recover if git is + temporarily unreachable. +2. **Install packages** — runs ``dbt deps`` (optional). +3. **Run your dbt commands** (``dbt build``, ``dbt test``, …) in the order + given. The first command to exit non-zero stops the run and fails the task. +4. **Upload** the ``target/`` directory to S3 or GCS on **both** success and + failure, so ``run_results.json`` and the compiled SQL are always available + for inspection. Review Comment: Good point — worth discussing why this operator was built and what gaps it fills. The DbtKubernetesRunOperator addresses real production needs that Cosmos + manual KPO don't provide: - Git caching (Optional) — S3/GCS cache per DAG+repo+branch; falls back if git is unreachable. - Artifact upload (Optional) — target/ uploaded to S3/GCS on success/failure (so run_results.json + compiled SQL are always available) - Connection security — credentials resolved on worker, injected as env vars; pod never touches Airflow connections directly - One image for all adapters — Supports generic python+uv+git public image; Or Specific Privately hosted Image. - Works on every dbt project setup — each project carries its own uv.lock or requirement.txt. Valid for every kind of dbt project. - Steps beyond dbt — steps accepts any shell command (uv sync, pip install, etc.). Project may required for initial setup cmd The operator gives teams a shared, tested implementation instead of each building their own custom operator. I'm happy to discuss further or adjust the design. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
