bnspopi opened a new pull request, #77313:
URL: https://github.com/apache/airflow/pull/77313

   `DagBundlesManager.parse_config` keys bundles by name and never compares 
where they read their files from. If the same Git repository, branch and subdir 
is configured twice, once with an SSH URL and once with an HTTPS URL, Airflow 
accepts both as independent bundles. The Dag processor then clones and parses 
everything twice, every import error is reported twice, and each Dag keeps 
moving from one bundle to the other. That is what #77241 reports.
   
   This adds the detection described in the issue, as a warning only:
   
   - `BaseDagBundle.source_key(**kwargs)` is a new classmethod that says where 
a bundle with those kwargs reads from. It returns `None` by default, which 
means "not known", so existing and third-party bundles are unaffected.
   - `LocalDagBundle` returns its resolved directory.
   - `GitDagBundle` returns the repository as `host/path` together with the 
tracking ref and the subdir. The user, port and `.git` suffix are dropped and 
the host is lower-cased, so `[email protected]:org/repo.git`, 
`ssh://[email protected]:22/org/repo.git` and `https://[email protected]/org/repo` 
give the same key. When `repo_url` is not in the kwargs (it comes from the 
connection) or is a local path, the key is `None` and the bundle is not 
compared.
   - `DagBundlesManager.parse_config` logs a warning naming the bundles that 
share a key.
   - `dag-bundles.rst` and the duplicate dag id FAQ entry now say that two 
bundles must not read the same source.
   
   Left out on purpose:
   
   - The `[dag_processor] reject_duplicate_bundle_sources` flag that would turn 
the warning into an error. It is a new config option and a separate decision.
   - Nested subdirs of one repository (`subdir=""` and `subdir="dags"`). The 
issue notes this can be intentional.
   - The warning is logged every time a `DagBundlesManager` parses the config, 
so it appears in the log of each component that loads the bundle configuration.
   
   @basantjamwal proposed this same plan in the issue. I am happy to close this 
in favour of their PR if they already have one in progress.
   
   Tests, run on Linux with SQLite:
   
   - `test_dag_bundle_manager.py`, `test_local.py`, `test_base.py`: 107 passed 
(5 new cases).
   - `providers/git/tests/unit/git/bundles/test_git.py`: 142 passed (11 new 
cases).
   - Without the source changes, 14 of the new cases fail.
   
   related: #77241
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes (Claude Code)
   
   Generated-by: Claude Code following [the 
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to