dolfinus opened a new issue, #51321: URL: https://github.com/apache/airflow/issues/51321
### Description OpenLineage integration can provide information which DAGs and Task were executed, that datasets were read or written, and so on. One of facets OpenLineage have is [SourceCodeLocation](https://openlineage.io/docs/spec/facets/job-facets/source-code-location/) which allows users to see the repo DAG source code was originated from, as well as branch name, commit, tag and so on. But there is no way to add annotations to DAG which allow to collect this information and send it as some facet. Probably something like this could be implemented: ```python from airflow.sdk import DAG, SourceCodeLocation dag = DAG( dag_id="my_dag", source_code_location=SourceCodeLocation( repo_url="http://github.com/user/repo";, branch="main", path="/some/dag.py", version="6f1cab2f" ), ) ``` ### Use case/motivation My use case is: * There is a repo with ETL scripts and DAG .py files. Multiple users work with code within this repo, and create PRs with scripts/DAGs updates. * Each PR (e.g. `bugfix/123` or `feature/123` branch) is deployed to Airflow. To prevent overriding working files of different users, `/opt/airflow/dags` folder have a structure `./repo_name/branch_name/dag.py`. * For Airflow on host branch can be cloned using `git clone` or `rsync`. * For Airflow in Kubernetes this is done via custom sidecar which clones every branch within a repo to this directory structure. * After PR is merged, directory containing DAGs for this branch is removed from Airflow. I'm not using `gitsync` sidecar because it clones only one specific branch (e.g. `main`), and cannot be configured without redeploying Airflow workers & dag processor. Same for `GitDagBundle` in Airflow 3.x - bundles are set up using airflow.cfg, not via API, and changing this config require restarting DagProcessor. Also OpenLineage integration for Airflow is set up for the entire Airflow instance, not for each DAG specifically. But there can be DAGs from multiple repos, branches or commits within the same branch, and this cannot be passed to OpenLineage via config https://github.com/OpenLineage/OpenLineage/issues/3745. It is possible to parse all this information from `.git` folder content https://github.com/OpenLineage/OpenLineage/issues/3746, but the precense of this folder actually depends on the way Airflow DAG are reployed. Also, having `.git` folder may be a security issue in some usecases (e.g. public SAAS instance with private Git repos). That's why here I mention here only some kind of DAG object _annotation_/option. ### Related issues _No response_ ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
