Re: [I] `dag_discovery_safe_mode not scanning all files when set to False [airflow]
dayonet commented on issue #66104: URL: https://github.com/apache/airflow/issues/66104#issuecomment-4586573583 Did the DAG stay in the UI after the next DAG serialization? In my experience it shows up in the UI for a few minutes, and then disappears. It returns upon running `airflow dags reserialize`, and disappears again on the next automatic reserialization, just like @saidiraja experienced. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] `dag_discovery_safe_mode not scanning all files when set to False [airflow]
kir-rin commented on issue #66104:
URL: https://github.com/apache/airflow/issues/66104#issuecomment-4534569623
I'm the author of #67325. **I'm leaving this comment to explain why I closed
the PR.**
Since config changes require a restart, I don't think the default argument
evaluation at module load time is a bug.
I tried to reproduce the issue to identify the actual root cause, but
couldn't — the DAG files appeared in the UI correctly in my tests.
I tested both `airflow standalone` and separate `scheduler` + `api-server` +
`processor` setups.
Example files I used (same wrapper pattern from the issue description):
- `dags/wrapper/flow.py` — the wrapper decorator
- `dags/test.py` — uses the wrapper, also no keywords
```python
from datetime import datetime
from wrapper.flow import flow
@flow(schedule=none, start_date=datetime(2024, 1, 1))
def test():
print("hello")
test()
```
The DAG was discovered and visible in the UI after setting
`DAG_DISCOVERY_SAFE_MODE=False` and restarting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] `dag_discovery_safe_mode not scanning all files when set to False [airflow]
dayonet commented on issue #66104: URL: https://github.com/apache/airflow/issues/66104#issuecomment-4528923808 From what I understand from the PR, the conclusion is that the bug's behavior is as expected due to Airflow's evaluation of default arguments at module load time. Although I understand that this is the default behavior, I'd appreciate it if a workaround or more robust solution could be provided, since it is causing a concrete bug that's a big issue in our organization. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] `dag_discovery_safe_mode not scanning all files when set to False [airflow]
dayonet commented on issue #66104: URL: https://github.com/apache/airflow/issues/66104#issuecomment-4495857738 Hi, Is this supposed to be solved in future versions? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] `dag_discovery_safe_mode not scanning all files when set to False [airflow]
dayonet commented on issue #66104: URL: https://github.com/apache/airflow/issues/66104#issuecomment-4361574148 Thank you for the quick and helpful reply. Since this problem is currently a showstopper for me, I'd like to know if fixing it is on your roadmap (at least in terms of a workaround). I'm not sure I'm in a position to submit a PR, but I'd like to understand my options. Happy to help test or provide more details if that’s useful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] `dag_discovery_safe_mode not scanning all files when set to False [airflow]
saidiraja commented on issue #66104: URL: https://github.com/apache/airflow/issues/66104#issuecomment-4354851156 Thanks for sharing this , I was able to reproduce the same behavior on my side. With `dag_discovery_safe_mode=False`, the DAG is: - visible via `airflow dags list` - triggerable via CLI However, I also observed that it does not appear in the UI unless `airflow dags reserialize` is run manually. From what I understand, this may be because DAG discovery (used by CLI) and DAG serialization (used by the UI) are handled separately. The UI relies on the scheduler to serialize DAGs into the metadata DB, so if that step is not triggered automatically, the DAG won’t appear there. So it seems the issue might not only be about file scanning, but also how DAGs are picked up for serialization when safe mode is disabled. Curious to hear if others have seen similar behavior or have more insights on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] `dag_discovery_safe_mode not scanning all files when set to False [airflow]
dayonet commented on issue #66104: URL: https://github.com/apache/airflow/issues/66104#issuecomment-4353781320 Thank you for the quick reply. I ran the commands and I found that they really do appear when running `airflow dags list` and can be triggered via `airflow dags trigger `, which is great! Still, they don't appear in the Airflow UI without running `airflow dags reserialize` every few minutes. We give our Airflow to users in our company so it's important they can see them in the UI and trigger from there as well. Is there a way to make them appear in the UI permanently? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] `dag_discovery_safe_mode not scanning all files when set to False [airflow]
saidiraja commented on issue #66104: URL: https://github.com/apache/airflow/issues/66104#issuecomment-4347057227 Reproduction attempt (Airflow 3.2.0) Environment - Airflow: 3.2.0 - Python: 3.12.3 - OS: Ubuntu (WSL2) - Deployment: local virtualenv What I tested I created a DAG file (hidden_flow.py) that does not explicitly contain typical keywords like "airflow" or "dag". Then I disabled safe mode: export AIRFLOW__CORE__DAG_DISCOVERY_SAFE_MODE=False and verified: airflow config get-value core dag_discovery_safe_mode # False Observations - The DAG was detected when loading via DagBag: HAS SAFE_MODE_TEST_66104: True - However, it was not listed initially using: airflow dags list - After starting Airflow using: airflow standalone the DAG appeared in the list, could be triggered, and runs were created successfully. Result I was not able to fully reproduce the exact issue described. DAG parsing works correctly when safe mode is disabled, but DAG visibility seems to depend on the scheduler/services being running. Evidence Screenshots attached showing: - DAG not listed initially - DagBag detecting the DAG - DAG appearing after starting Airflow - Successful DAG run https://github.com/user-attachments/assets/b57a51cc-4dba-44d7-8b42-b9e9ee10cdf3"; /> https://github.com/user-attachments/assets/386bccfc-900b-42b3-a837-5441a162931e"; /> https://github.com/user-attachments/assets/0f6dfe4a-ea8c-45d7-b6f4-af1de1c1dc21"; /> https://github.com/user-attachments/assets/4d4847b7-f3e8-495e-860f-bf0287a7fd05"; /> https://github.com/user-attachments/assets/c042fd57-7c37-47c1-9898-a7416d2d414b"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] `dag_discovery_safe_mode not scanning all files when set to False [airflow]
boring-cyborg[bot] commented on issue #66104: URL: https://github.com/apache/airflow/issues/66104#issuecomment-4345123289 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
