This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new ea3aa67891b Fix system test for dataform operators (#44729)
ea3aa67891b is described below
commit ea3aa67891b68c69d35af455ad3106c1253244fe
Author: Maksim <[email protected]>
AuthorDate: Fri Dec 6 06:41:07 2024 -0800
Fix system test for dataform operators (#44729)
---
.../operators/cloud/dataform.rst | 122 ++++++++++++---------
.../google/cloud/dataform/example_dataform.py | 58 +++++-----
2 files changed, 98 insertions(+), 82 deletions(-)
diff --git a/docs/apache-airflow-providers-google/operators/cloud/dataform.rst
b/docs/apache-airflow-providers-google/operators/cloud/dataform.rst
index 208035af53c..d192ad2dd9e 100644
--- a/docs/apache-airflow-providers-google/operators/cloud/dataform.rst
+++ b/docs/apache-airflow-providers-google/operators/cloud/dataform.rst
@@ -25,20 +25,21 @@ Dataform lets you manage data transformation in the
Extraction, Loading, and Tra
for data integration. After raw data is extracted from source systems and
loaded into BigQuery, Dataform
helps you to transform it into a well-defined, tested, and documented suite of
data tables.
-For more information about the task visit `Dataform production documentation
<Product documentation <https://cloud.google.com/dataform/docs>`__
+For more information about the task visit `Dataform documentation
<https://cloud.google.com/dataform/docs>`__
Configuration
-------------
Before you can use the Dataform operators you need to initialize repository
and workspace, for more information
-about this visit `Dataform Production documentation <Product documentation
<https://cloud.google.com/dataform/docs>`__
+about this visit `Dataform documentation
<https://cloud.google.com/dataform/docs>`__
Create Repository
-----------------
-Create repository for tracking your code in Dataform service. Example of usage
can be seen below:
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformCreateRepositoryOperator`
+To create a repository for tracking your code in Dataform service use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformCreateRepositoryOperator`.
+Example of usage can be seen below:
.. exampleinclude::
/../../providers/tests/system/google/cloud/dataform/example_dataform.py
:language: python
@@ -46,11 +47,13 @@ Create repository for tracking your code in Dataform
service. Example of usage c
:start-after: [START howto_operator_create_repository]
:end-before: [END howto_operator_create_repository]
+
Create Workspace
----------------
-Create workspace for storing your code in Dataform service. Example of usage
can be seen below:
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformCreateWorkspaceOperator`
+To create a workspace for storing your code in Dataform service use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformCreateWorkspaceOperator`.
+Example of usage can be seen below:
.. exampleinclude::
/../../providers/tests/system/google/cloud/dataform/example_dataform.py
:language: python
@@ -61,9 +64,10 @@ Create workspace for storing your code in Dataform service.
Example of usage can
Create Compilation Result
-------------------------
-A simple configuration to create Compilation Result can look as followed:
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformCreateCompilationResultOperator`
+To create a Compilation Result use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformCreateCompilationResultOperator`.
+A simple configuration can look as followed:
.. exampleinclude::
/../../providers/tests/system/google/cloud/dataform/example_dataform.py
:language: python
@@ -71,12 +75,12 @@ A simple configuration to create Compilation Result can
look as followed:
:start-after: [START howto_operator_create_compilation_result]
:end-before: [END howto_operator_create_compilation_result]
+
Get Compilation Result
----------------------
-To get a Compilation Result you can use:
-
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformGetCompilationResultOperator`
+To get a Compilation Result you can use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformGetCompilationResultOperator`.
.. exampleinclude::
/../../providers/tests/system/google/cloud/dataform/example_dataform.py
:language: python
@@ -84,22 +88,12 @@ To get a Compilation Result you can use:
:start-after: [START howto_operator_get_compilation_result]
:end-before: [END howto_operator_get_compilation_result]
+
Create Workflow Invocation
--------------------------
-To create a Workflow Invocation you can use:
-
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformCreateWorkflowInvocationOperator`
-
-We have possibility to run this operation in the sync mode and async, for
async operation we also have
-a sensor:
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformWorkflowInvocationStateSensor`
-
-We also have a sensor to check the status of a particular action for a
workflow invocation triggered
-asynchronously.
-
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformWorkflowInvocationActionStateSensor`
-
+To create a Workflow Invocation you can use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformCreateWorkflowInvocationOperator`.
.. exampleinclude::
/../../providers/tests/system/google/cloud/dataform/example_dataform.py
:language: python
@@ -107,24 +101,34 @@ asynchronously.
:start-after: [START howto_operator_create_workflow_invocation]
:end-before: [END howto_operator_create_workflow_invocation]
+
+We have possibility to run this operation in the sync mode and async, for
async operation we also have
+a sensor
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformWorkflowInvocationStateSensor`.
+
.. exampleinclude::
/../../providers/tests/system/google/cloud/dataform/example_dataform.py
:language: python
:dedent: 4
:start-after: [START howto_operator_create_workflow_invocation_async]
:end-before: [END howto_operator_create_workflow_invocation_async]
+
+We also have a sensor
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformWorkflowInvocationActionStateSensor`
+to check the status of a particular action for a workflow invocation triggered
asynchronously.
+
.. exampleinclude::
/../../providers/tests/system/google/cloud/dataform/example_dataform.py
:language: python
:dedent: 4
:start-after: [START
howto_operator_create_workflow_invocation_action_async]
:end-before: [END howto_operator_create_workflow_invocation_action_async]
+
Get Workflow Invocation
-----------------------
-To get a Workflow Invocation you can use:
-
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformGetWorkflowInvocationOperator`
+To get a Workflow Invocation you can use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformGetWorkflowInvocationOperator`.
.. exampleinclude::
/../../providers/tests/system/google/cloud/dataform/example_dataform.py
:language: python
@@ -132,12 +136,12 @@ To get a Workflow Invocation you can use:
:start-after: [START howto_operator_get_workflow_invocation]
:end-before: [END howto_operator_get_workflow_invocation]
+
Query Workflow Invocation Action
--------------------------------
-To query Workflow Invocation Actions you can use:
-
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformQueryWorkflowInvocationActionsOperator`
+To query Workflow Invocation Actions you can use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformQueryWorkflowInvocationActionsOperator`.
.. exampleinclude::
/../../providers/tests/system/google/cloud/dataform/example_dataform.py
:language: python
@@ -145,12 +149,12 @@ To query Workflow Invocation Actions you can use:
:start-after: [START howto_operator_query_workflow_invocation_actions]
:end-before: [END howto_operator_query_workflow_invocation_actions]
+
Cancel Workflow Invocation
--------------------------
-To cancel a Workflow Invocation you can use:
-
-:class:`~airflow.providers.google.cloud.sensors.dataform.DataformCancelWorkflowInvocationOperator`
+To cancel a Workflow Invocation you can use
+:class:`~airflow.providers.google.cloud.sensors.dataform.DataformCancelWorkflowInvocationOperator`.
.. exampleinclude::
/../../providers/tests/system/google/cloud/dataform/example_dataform.py
:language: python
@@ -158,35 +162,41 @@ To cancel a Workflow Invocation you can use:
:start-after: [START howto_operator_cancel_workflow_invocation]
:end-before: [END howto_operator_cancel_workflow_invocation]
+
Delete Repository
-----------------
-Deletes repository. Example of usage can be seen below:
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformDeleteRepositoryOperator`
+To delete a repository use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformDeleteRepositoryOperator`.
+Example of usage can be seen below:
.. exampleinclude::
/../../providers/tests/system/google/cloud/dataform/example_dataform.py
:language: python
:dedent: 0
- :start-after: [START howto_operator_delete_workspace]
- :end-before: [END howto_operator_delete_workspace]
+ :start-after: [START howto_operator_delete_repository]
+ :end-before: [END howto_operator_delete_repository]
+
Delete Workspace
----------------
-Deletes workspace. Example of usage can be seen below:
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformDeleteRepositoryOperator`
+To delete a workspace use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformDeleteWorkspaceOperator`.
+Example of usage can be seen below:
.. exampleinclude::
/../../providers/tests/system/google/cloud/dataform/example_dataform.py
:language: python
:dedent: 0
- :start-after: [START howto_operator_delete_repository]
- :end-before: [END howto_operator_delete_repository]
+ :start-after: [START howto_operator_delete_workspace]
+ :end-before: [END howto_operator_delete_workspace]
+
Remove file
-----------
-Removes file. Example of usage can be seen below:
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformRemoveFileOperator`
+To remove a file use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformRemoveFileOperator`.
+Example of usage can be seen below:
.. exampleinclude::
/../../providers/tests/system/google/cloud/dataform/example_dataform.py
:language: python
@@ -194,11 +204,13 @@ Removes file. Example of usage can be seen below:
:start-after: [START howto_operator_remove_file]
:end-before: [END howto_operator_remove_file]
+
Remove directory
----------------
-Removes directory. Example of usage can be seen below:
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformRemoveDirectoryOperator`
+To remove a directory use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformRemoveDirectoryOperator`.
+Example of usage can be seen below:
.. exampleinclude::
/../../providers/tests/system/google/cloud/dataform/example_dataform.py
:language: python
@@ -206,11 +218,12 @@ Removes directory. Example of usage can be seen below:
:start-after: [START howto_operator_remove_directory]
:end-before: [END howto_operator_remove_directory]
+
Initialize workspace
--------------------
-Creates default projects structure for provided workspace. Before it can be
done workspace and repository should be created. Example of usage can be seen
below:
-:class:`~airflow.providers.google.cloud.utils.dataform.make_initialization_workspace_flow`
+Creates default projects structure for provided workspace. Before it can be
done workspace and repository should be created.
+Example of usage can be seen below:
.. exampleinclude::
/../../providers/tests/system/google/cloud/dataform/example_dataform.py
:language: python
@@ -218,11 +231,12 @@ Creates default projects structure for provided
workspace. Before it can be done
:start-after: [START howto_initialize_workspace]
:end-before: [END howto_initialize_workspace]
+
Write file to workspace
-----------------------
-Writes file with given content to specified workspace.
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformWriteFileOperator`
+To write a file with given content to specified workspace use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformWriteFileOperator`.
.. exampleinclude::
/../../providers/tests/system/google/cloud/dataform/example_dataform.py
:language: python
@@ -230,11 +244,12 @@ Writes file with given content to specified workspace.
:start-after: [START howto_operator_write_file]
:end-before: [END howto_operator_write_file]
+
Make directory in workspace
---------------------------
-Make directory with given path in specified workspace.
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformMakeDirectoryOperator`
+To make a directory with given path in specified workspace use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformMakeDirectoryOperator`.
.. exampleinclude::
/../../providers/tests/system/google/cloud/dataform/example_dataform.py
:language: python
@@ -242,11 +257,12 @@ Make directory with given path in specified workspace.
:start-after: [START howto_operator_make_directory]
:end-before: [END howto_operator_make_directory]
+
Install NPM packages
--------------------
-Installs npm packages for specified workspace
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformInstallNpmPackagesOperator`
+To install npm packages for specified workspace use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformInstallNpmPackagesOperator`.
.. exampleinclude::
/../../providers/tests/system/google/cloud/dataform/example_dataform.py
:language: python
diff --git a/providers/tests/system/google/cloud/dataform/example_dataform.py
b/providers/tests/system/google/cloud/dataform/example_dataform.py
index bf2a2a3eb8e..5d3268b6ea9 100644
--- a/providers/tests/system/google/cloud/dataform/example_dataform.py
+++ b/providers/tests/system/google/cloud/dataform/example_dataform.py
@@ -171,33 +171,22 @@ with DAG(
region=REGION,
repository_id=REPOSITORY_ID,
workflow_invocation_id=(
- "{{
task_instance.xcom_pull('create-workflow-invocation')['name'].split('/')[-1] }}"
+ "{{
task_instance.xcom_pull('create-workflow-invocation-async')['name'].split('/')[-1]
}}"
),
expected_statuses={WorkflowInvocation.State.SUCCEEDED},
)
# [END howto_operator_create_workflow_invocation_async]
# [START howto_operator_create_workflow_invocation_action_async]
- create_workflow_invocation_async_action =
DataformCreateWorkflowInvocationOperator(
- task_id="create-workflow-invocation-async-action",
- project_id=PROJECT_ID,
- region=REGION,
- repository_id=REPOSITORY_ID,
- asynchronous=True,
- workflow_invocation={
- "compilation_result": "{{
task_instance.xcom_pull('create-compilation-result')['name'] }}"
- },
- )
-
is_workflow_invocation_action_done =
DataformWorkflowInvocationActionStateSensor(
task_id="is-workflow-invocation-action-done",
project_id=PROJECT_ID,
region=REGION,
repository_id=REPOSITORY_ID,
workflow_invocation_id=(
- "{{
task_instance.xcom_pull('create-workflow-invocation')['name'].split('/')[-1] }}"
+ "{{
task_instance.xcom_pull('create-workflow-invocation-async')['name'].split('/')[-1]
}}"
),
- target_name="YOUR_TARGET_HERE",
+ target_name="first_view",
expected_statuses={WorkflowInvocationAction.State.SUCCEEDED},
failure_statuses={
WorkflowInvocationAction.State.SKIPPED,
@@ -340,26 +329,37 @@ with DAG(
)
(
last_initialization_step
- >> install_npm_packages
- >> create_compilation_result
- >> get_compilation_result
- >> create_workflow_invocation
- >> get_workflow_invocation
- >> query_workflow_invocation_actions
- >> create_workflow_invocation_async
- >> is_workflow_invocation_done
- >> create_workflow_invocation_async_action
- >> is_workflow_invocation_action_done
- >> create_workflow_invocation_for_cancel
- >> cancel_workflow_invocation
>> make_test_directory
>> write_test_file
- # TEST TEARDOWN
>> remove_test_file
>> remove_test_directory
>> delete_dataset
- >> delete_workspace
- >> delete_repository
+ )
+ (
+ last_initialization_step
+ >> install_npm_packages
+ >> create_compilation_result
+ >> [
+ get_compilation_result,
+ create_workflow_invocation,
+ create_workflow_invocation_async,
+ create_workflow_invocation_for_cancel,
+ ]
+ )
+ (
+ create_workflow_invocation
+ >> [get_workflow_invocation, query_workflow_invocation_actions]
+ >> delete_dataset
+ )
+ (
+ create_workflow_invocation_async
+ >> [is_workflow_invocation_done, is_workflow_invocation_action_done]
+ >> delete_dataset
+ )
+ create_workflow_invocation_for_cancel >> cancel_workflow_invocation >>
delete_dataset
+ (
+ # TEST TEARDOWN
+ delete_dataset >> delete_workspace >> delete_repository
)
# ### Everything below this line is not part of example ###
