Re: Guidance/best practice for mocking task instance output for unit testing DAGs
Here is a related SO question https://stackoverflow.com/questions/58875195/airflow-test-mode-xcom-pull-push-not-working The unverified answer is as follows > As far as I know, "test" runs without saving anything to the metadata > database which is why when you run the puller task, you get "None" as a > result and when you actually run the DAG code, it works. You can query the > metadata database directly after testing the first task to verify this. and > Context seems to be missing here, along with xcom_push=True, we need to use > provide_context=True That's what we are going off. lewismc On 2021/07/29 16:39:02, Lewis John McGibbney wrote: > Hi users@, > Say I want to unit test Task A which accepts one or more parameters. These > parameters are the outputs of some other task which I want to mock. Can > someone provide a code example of how I would do that? > We are using pytest as opposed to unittest but an example of either would be > greatly appreciated. > I studied several articles [0] [1] [2] which were informative but I think > lack the content which explains what I want to do above. > I did find the following code example [3] which I feel is on the correct > track but I am not confident. > Thanks for any consideration. > lewismc > > [0] > https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#testing-a-dag > [1] > https://blog.usejournal.com/testing-in-airflow-part-1-dag-validation-tests-dag-definition-tests-and-unit-tests-2aa94970570c > [2] > https://medium.com/@chandukavar/testing-in-airflow-part-2-integration-tests-and-end-to-end-pipeline-tests-af0555cd1a82 > [3] > https://github.com/apache/airflow/blob/main/tests/ti_deps/deps/test_dagrun_exists_dep.py >
Re: Guidance/best practice for mocking task instance output for unit testing DAGs
I will add that the unit tests we are writing relate to DAGs which are written using the Taskflow API. We have come to understand that XCom cannot be mocked in unit tests... and I think that this is the source of our problem. Therefore we have experimented with manually defining XCom pull and push constructs as follows pulled_value = ti.xcom_pull(key='someKey', task_ids='push_to_xcoms') Is this the correct way to write task instance unit tests which require input parameters from XCom? Thanks lewismc On 2021/07/29 16:39:02, Lewis John McGibbney wrote: > Hi users@, > Say I want to unit test Task A which accepts one or more parameters. These > parameters are the outputs of some other task which I want to mock. Can > someone provide a code example of how I would do that? > We are using pytest as opposed to unittest but an example of either would be > greatly appreciated. > I studied several articles [0] [1] [2] which were informative but I think > lack the content which explains what I want to do above. > I did find the following code example [3] which I feel is on the correct > track but I am not confident. > Thanks for any consideration. > lewismc > > [0] > https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#testing-a-dag > [1] > https://blog.usejournal.com/testing-in-airflow-part-1-dag-validation-tests-dag-definition-tests-and-unit-tests-2aa94970570c > [2] > https://medium.com/@chandukavar/testing-in-airflow-part-2-integration-tests-and-end-to-end-pipeline-tests-af0555cd1a82 > [3] > https://github.com/apache/airflow/blob/main/tests/ti_deps/deps/test_dagrun_exists_dep.py >
Re: Guidance/best practice for mocking task instance output for unit testing DAGs
Hi users@, Is anyone able to assist here? Does my example make sense? Thank you lewismc On 2021/07/29 16:39:02, Lewis John McGibbney wrote: > Hi users@, > Say I want to unit test Task A which accepts one or more parameters. These > parameters are the outputs of some other task which I want to mock. Can > someone provide a code example of how I would do that? > We are using pytest as opposed to unittest but an example of either would be > greatly appreciated. > I studied several articles [0] [1] [2] which were informative but I think > lack the content which explains what I want to do above. > I did find the following code example [3] which I feel is on the correct > track but I am not confident. > Thanks for any consideration. > lewismc > > [0] > https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#testing-a-dag > [1] > https://blog.usejournal.com/testing-in-airflow-part-1-dag-validation-tests-dag-definition-tests-and-unit-tests-2aa94970570c > [2] > https://medium.com/@chandukavar/testing-in-airflow-part-2-integration-tests-and-end-to-end-pipeline-tests-af0555cd1a82 > [3] > https://github.com/apache/airflow/blob/main/tests/ti_deps/deps/test_dagrun_exists_dep.py >
Guidance/best practice for mocking task instance output for unit testing DAGs
Hi users@, Say I want to unit test Task A which accepts one or more parameters. These parameters are the outputs of some other task which I want to mock. Can someone provide a code example of how I would do that? We are using pytest as opposed to unittest but an example of either would be greatly appreciated. I studied several articles [0] [1] [2] which were informative but I think lack the content which explains what I want to do above. I did find the following code example [3] which I feel is on the correct track but I am not confident. Thanks for any consideration. lewismc [0] https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#testing-a-dag [1] https://blog.usejournal.com/testing-in-airflow-part-1-dag-validation-tests-dag-definition-tests-and-unit-tests-2aa94970570c [2] https://medium.com/@chandukavar/testing-in-airflow-part-2-integration-tests-and-end-to-end-pipeline-tests-af0555cd1a82 [3] https://github.com/apache/airflow/blob/main/tests/ti_deps/deps/test_dagrun_exists_dep.py
