Re: [I] Airflow incorrectly `AirflowInactiveAssetInInletOrOutletException` when using `dag.test()` [airflow]
uranusjr commented on issue #51644: URL: https://github.com/apache/airflow/issues/51644#issuecomment-2973585147 I think for the context of this specific issue, we should process the specific dag you’re trying to test against when `test` is run to do the necessary setup for things to work as expected. In the future, we should add a flag to control whether this setup phase modifies the real metadatabase, or creates an ephemeral mock that the dag test interacts with instead of the real database, so it does not introduce and/or be afected by side effects. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Airflow incorrectly `AirflowInactiveAssetInInletOrOutletException` when using `dag.test()` [airflow]
tatiana commented on issue #51644: URL: https://github.com/apache/airflow/issues/51644#issuecomment-2969614191 I believe the `airflow dags test` or the `dag.test()` method is one of the most valuable tools Airlfow offers to end-users. It empowers any DAG author or Airflow provider developer who wants to efficiently validate the behaviour of a DAG without the overhead of using the Airflow UI. It also enables smooth development and troubleshooting without relying on multiple processes or containers - you can easily use the Python standard library's `breakpoint` to troubleshoot. Finally, they enable users to conduct effective integration tests that can be run in the CI, providing confidence that, from a DAG logic perspective, things will behave as expected when orchestrated by Airlfow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Airflow incorrectly `AirflowInactiveAssetInInletOrOutletException` when using `dag.test()` [airflow]
tatiana commented on issue #51644: URL: https://github.com/apache/airflow/issues/51644#issuecomment-2969588877 @uranusjr thank you very much for investigating this! > Or do we just ignore all the assets when using test (and not emit an event at all—which might make sense instead since you probably wouldn’t expect a downstream to be triggered when testing a dag) From my perspective, users should have the option to decide whether they want to enable side effects or not when running `airflow dags test`. If we could have a flag, it would be awesome. I am personally interested in both testing asset events using dags test, both from an integration test to be run in the CI, but also being able to view the results in the AF UI later. At the moment, there isn't a good way to verify if assets were emitted as expected in operators that generate asset events dynamically. We also cannot validate listeners' behaviour when using `dag.test` - something that is quite important for people building Airflow integrations and providers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Airflow incorrectly `AirflowInactiveAssetInInletOrOutletException` when using `dag.test()` [airflow]
uranusjr commented on issue #51644: URL: https://github.com/apache/airflow/issues/51644#issuecomment-2969138202 I played around a bit. Assuming a fresh database… 1. `airflow dags test example_inactive_asset` immediately → fail 2. Trigger from UI with `airflow standalone` → success 3. `airflow dags test example_inactive_asset` after running `standalone` → success I think this is caused by the dag processor being responsible for collecting assets, and it does not kick in when you do `airflow dags test`. After you let the dag processor collect the assets, everything would work as expected. I’m not quite sure how to handle this. Should we do a round of dag collection when a dag is run from CLI (on the to-be-run dag)? Or do we just ignore all the assets when using `test` (and not emit an event at all—which might make sense instead since you probably wouldn’t expect a downstream to be triggered when testing a dag)? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
[I] Airflow incorrectly `AirflowInactiveAssetInInletOrOutletException` when using `dag.test()` [airflow]
tatiana opened a new issue, #51644:
URL: https://github.com/apache/airflow/issues/51644
### Apache Airflow version
3.0.2
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
The following DAG used to work in Airflow 2:
```
from datetime import datetime
from airflow import DAG
from airflow.datasets import Dataset, DatasetAlias
from airflow.operators.empty import EmptyOperator
dataset = Dataset("something")
with DAG("example_inactive_asset", start_date=datetime(2023, 4, 20)) as dag:
do_nothing = EmptyOperator(
task_id="start_task",
outlets=[dataset]
)
```
However, in Airflow 3.0.2 it raises an exception if I try to run it using
the `airflow dags test example_inactive_asset` command:
```
│ /Users/tati/Library/Application
│
│
Support/hatch/env/virtual/astronomer-cosmos/ekzAVDNG/tests.py3.11-3.0-1.9/lib/python3.11/site-pa
│
│ ckages/airflow/sdk/execution_time/task_runner.py:792 in
_validate_task_inlets_and_outlets│
│
│
│789 │ if TYPE_CHECKING:
│
│790 │ │ assert isinstance(inactive_assets_resp,
InactiveAssetsResult) │
│791 │ if inactive_assets := inactive_assets_resp.inactive_assets:
│
│ ❱ 792 │ │ raise AirflowInactiveAssetInInletOrOutletException(
│
│793 │ │ │ inactive_asset_keys=[
│
│794 │ │ │ │ AssetUniqueKey.from_profile(asset_profile) for
asset_profile in inactive │
│795 │ │ │ ]
│
│
│
│ ╭─── locals
───╮ │
│ │ inactive_assets = [AssetProfile(name='something', uri='something',
type='Asset')] │ │
│ │ inactive_assets_resp = InactiveAssetsResult(
│ │
│ ││ inactive_assets=[
│ │
│ ││ │ AssetProfile(
│ │
│ ││ │ │ name='something',
│ │
│ ││ │ │ uri='something',
│ │
│ ││ │ │ type='Asset'
│ │
│ ││ │ )
│ │
│ ││ ],
│ │
│ ││ type='InactiveAssetsResult'
│ │
│ │)
│ │
│ │ log =
│ │
│ │ ti =│ │
│ ││ start_date=datetime.datetime(2025, 6, 12, 9,
34, 38, 74867, │ │
│ │tzinfo=datetime.timezone.utc)
│ │
│ │>
│ │
│
╰──╯
│
╰──╯
AirflowInactiveAssetInInletOrOutletException: Task has the following
inactive assets in its inlets
or outlets: Asset(name='something', uri='something')
2025-06-12 10:34:38 [debug] Sending request[task]
msg=TaskState(state=,
end_date=datetime.datetime(2025, 6, 12, 9, 34, 38, 630590,
tzinfo=datetime.timezone.utc), type='TaskState', rendered_map_index=None)
2025-06-12 10:34:38 [debug] Received message from task runner [task]
msg=TaskState(state=,
end_date=datetime.datetime(2025, 6, 12, 9, 34, 38, 630590,
tzinfo=datetime.timezone.utc), type='TaskState', rendered_map_index=None)
2025-06-12 10:34:38 [debug] Running finalizers [task]
ti=RuntimeTaskInstance(id=UUID('0197637d-cb2d-7d46-b079-6585b7ee853e'),
task_id='start_task', dag_id='example_inactive_asset',
run_id='manual__2025-06-12T09:34:37.857251+00:00', try_number=1, map_index=-1,
hostname='alchueyr', context_carrier=None, task=, max_t
Re: [I] Airflow incorrectly `AirflowInactiveAssetInInletOrOutletException` when using `dag.test()` [airflow]
boring-cyborg[bot] commented on issue #51644: URL: https://github.com/apache/airflow/issues/51644#issuecomment-2965924551 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
