ashb commented on code in PR #73259: URL: https://github.com/apache/airflow/pull/73259#discussion_r4101778503
########## providers/dbt/core/README.rst: ########## @@ -0,0 +1,305 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +``apache-airflow-providers-dbt-core`` +===================================== + +Run `dbt Core <https://docs.getdbt.com/docs/core/installation-overview>`__ jobs +on Kubernetes from Apache Airflow. The provider ships a single operator, +:class:`~airflow.providers.dbt.core.operators.dbt.DbtKubernetesRunOperator`, +which mirrors a dbt Cloud job run inside one Kubernetes pod — without dbt Cloud. + +.. contents:: :local: + + +What the operator does +---------------------- + +In a single pod, in order: + +1. **Clone** the dbt project from git (optional; the project may be baked into + the image instead). A git-repo cache on S3/GCS lets runs recover if git is + temporarily unreachable. +2. **Install packages** — runs ``dbt deps`` (optional). +3. **Run your dbt commands** (``dbt build``, ``dbt test``, …) in the order + given. The first command to exit non-zero stops the run and fails the task. +4. **Upload** the ``target/`` directory to S3 or GCS on **both** success and + failure, so ``run_results.json`` and the compiled SQL are always available + for inspection. Review Comment: I'm not saying is not useful, just that these decisions all add up to this not really being a dbt operator, but a "how you want to use dbt and git and kube and connections" For example, > Connection security — credentials resolved on worker, injected as env vars; pod never touches Airflow connections directly I'd say that this, specifically is a huge security risk. Connections in env means they are visible casually to anything that can list pods. The point though is that i don't think this operator is useful enough/will have enough usage by people to make it worth including in the core registry. But you can publish this on PyPI (just not with an `apache-airflow-` prefix. `airflow-` is fine though) and then if it proves more popular we can re-examine -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
