dilnazanlid opened a new pull request, #72474:
URL: https://github.com/apache/airflow/pull/72474
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
* **Project Board:** [AIP-85 Dag Importer (Project
#672)](https://github.com/orgs/apache/projects/672)
* **Related PRs:**
* Part 1: #72369 ([AIP-85] Move DAG importer abstractions and registry
into Task SDK)
* Part 2: #72442 ([AIP-85] Add configuration support for custom DAG
importers)
This pull request is one of the incremental parts of the **[AIP-85: DAG
Importer](https://cwiki.apache.org/confluence/x/_Q7OEg)** implementation and
can be submitted separately.
### Description
Under **AIP-85: DAG Importer**, Dag definitions are no longer restricted to
local `.py` files with flat filesystem paths. Pluggable and non-Python Dag
importers (e.g. YAML/JSON declarative importers, composite archive importers
like `ZipImporter`, or external bundle sources) introduce new requirements for
diagnostics, source location tracking, and presentation:
1. **Generalized Source References in Import Errors**: Importers may import
Dags from composite archives (such as `archive.zip:dag.py`) or arbitrary
locations rather than strictly local relative file paths.
2. **Extensible Dag Warnings & Context**: Custom importers may detect
format-specific non-fatal issues and emit custom warning types accompanied by
structured context (such as schema issue locations or suggestions).
3. **Language-Aware Dag Sources**: Dag source code may be written in formats
other than Python, requiring language identification for proper syntax
highlighting in the Airflow UI Code view.
This PR adjusts the metadata models, database schema, FastAPI endpoints, and
UI components to support these capabilities.
---
### Summary of Changes
#### 1. Import Error Source References (`ParseImportError` & DB Migration)
- **Database Migration (`ca8499dc1004`)**: Added migration
`0134_3_4_0_rename_filename_to_source_reference_in_import_error.py` renaming
the `filename` column to `source_reference` (`sa.String(length=1024)`) in the
`import_error` table. Updated `_REVISION_HEADS_MAP` and migration reference
docs.
- **Model Compatibility**: Added `filename = synonym("source_reference")` in
`ParseImportError` for backward compatibility with existing code.
- **Path Resolution**: Updated `ParseImportError.full_file_path()` to
support composite and absolute references (`archive.zip:member.py` or absolute
paths) in addition to bundle-relative paths.
- **Collection & Deletion**:
- Updated `airflow.api.common.delete_dag` to delete import error entries
matching either relative or absolute filelocs via
`ParseImportError.source_reference`.
- Updated `_update_import_errors` in `airflow.dag_processing.collection`
and old error cleanup in `DagFileProcessorManager` to query, update, and delete
using `source_reference`.
- **FastAPI Endpoints**:
- Maintained API compatibility by setting `source_reference: str =
Field(alias="filename")` in `ImportErrorResponse`.
- Updated `/importErrors` filtering and search parameter factories
(`QueryParseImportErrorFilename*`) and order-by mapping to target
`ParseImportError.source_reference`.
#### 2. Open-Ended Dag Warnings & Contextual Metadata (`DagWarning`)
- **Arbitrary Warning Types**: Updated `DagWarning.__init__` to gracefully
fall back to storing raw string types if a warning type does not match the
built-in `DagWarningType` enum, enabling custom importers to emit
domain-specific warnings.
- **Context Payload**: Added an optional `context: dict | None = None` field
to `DAGWarningResponse` to allow rich diagnostic metadata to be attached to
warnings.
- **FastAPI Filtering**: Updated `list_dag_warnings` to accept
`warning_type: str | None` instead of strict `DagWarningType | None`. Querying
for non-matching or unknown warning types now returns HTTP 200 with an empty
list rather than HTTP 422 Unprocessable Entity.
#### 3. Language Metadata in Dag Sources & UI Syntax Highlighting
- **API Response**: Added `language: str | None = None` to
`DAGSourceResponse` in `airflow.api_fastapi.core_api.datamodels.dag_sources`.
- **UI Code View**: Updated
`airflow-core/src/airflow/ui/src/pages/Dag/Code/Code.tsx` (and generated
TypeScript schemas) to pass `code?.language ?? "python"` to both Monaco
`Editor` and `CodeDiffViewer`, dynamically supporting syntax highlighting for
non-Python Dags while defaulting to Python.
#### 4. Tests
- Updated unit tests in `test_dag_sources.py` to verify `language` in DAG
source responses.
- Updated `test_dag_warning.py` to verify the `context` field and
non-matching warning type queries.
- Updated `test_import_error.py` and `test_collection.py` to instantiate and
assert against `source_reference`.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes
Generated-by: Google Gemini following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
