seanmuth opened a new issue, #71125:
URL: https://github.com/apache/airflow/issues/71125
### Description
\`find_dag_file_paths\` (airflow-core/src/airflow/utils/file.py) decides
whether
to attempt DAG discovery on a file using:
if path.is_file() and (path.suffix == ".py" or zipfile.is_zipfile(path)):
if might_contain_dag(file_path, safe_mode):
file_paths.append(file_path)
\`zipfile.is_zipfile()\` is a content sniff (checks for the PK zip magic
bytes/central directory), not an extension check. There is no accompanying
\`path.suffix == ".zip"\` condition. Since the ZIP format underlies many
common file types beyond \`.zip\` itself — \`.jar\`, \`.pptx\`, \`.docx\`,
\`.xlsx\`,
\`.apk\`, \`.epub\`, \`.odt\`, \`.whl\`, etc. — any of these dropped into a
DAGs
folder (e.g. a build artifact, a supporting doc, a packaged dependency)
will pass this check and get opened and scanned via \`might_contain_dag\`,
purely because it happens to share the underlying zip container format
with Airflow's own zipped-DAG-bundle feature.
This was reported previously in #45718 with a \`.pptx\` file, but that issue
was closed as invalid because the specific symptom described there (garbled
metric names) turned out to be an unrelated stat-sanitization bug, not this
zip-detection design question. The zip-detection behavior itself was never
actually addressed.
### Impact
At minimum this is wasted work (opening and scanning irrelevant files on
every DAG processor cycle). Depending on \`might_contain_dag\`'s heuristic
and
the archive's contents, it can also produce confusing log noise, and in the
originally reported case, appears able to feed corrupted/unexpected data
further into DAG processing.
### What you think should happen instead
Gate the zip-bundle branch on the file extension in addition to (or instead
of) the content sniff, e.g.:
if path.is_file() and (path.suffix == ".py" or (path.suffix == ".zip"
and zipfile.is_zipfile(path))):
This preserves the documented \`.zip\` DAG-bundle behavior while no longer
opening arbitrary non-\`.zip\` files that merely share the same underlying
container format. \`.airflowignore\` is a viable per-deployment workaround
today (matching by extension), but it means every deployment that happens
to keep e.g. \`.jar\` files anywhere under its DAGs folder has to know to add
this rule proactively rather than it being a non-issue by default.
### How to reproduce
1. Place any non-\`.zip\` PK-zip-format file (a \`.jar\`, \`.pptx\`,
\`.docx\`, etc.)
anywhere under the DAGs folder.
2. Wait for the DAG processor to walk the directory.
3. Observe that the file is opened and passed through \`might_contain_dag\`
(visible via DAG processor debug logs), the same as a \`.zip\` DAG bundle
would be, despite not being one.
### Are you willing to submit a PR?
- [X] Yes
### Code of Conduct
- [X] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
