marshall7m commented on issue #10454:
URL: https://github.com/apache/airflow/issues/10454#issuecomment-1321250230
Hey all, just wondering if there's potential for a Terraform/Terragrunt
operator that's simply for returning Terraform output values to be used for
downstream tasks. Since this operator will be scoped to only the `terraform
output` command, the operator would only need a minimal amount of dependencies
and probably wouldn't require setting up a new environment. The operator would
only need to know the: terraform/terragrunt version, infrastructure repository
URL, directory to run the output command, and the associated read credentials
needed to access the remote tfstate file. I think this would be ideal for users
who have CI/CD pipelines already in place for running `terraform apply` and
they just need a way to sync their Terraform resources to their DAG tasks.
The operator would look something like this:
``` python
terra_output = TerraOutputOperator(
binary='terraform',
repo_url="https://company/infra-live";,
sub_path='infra/region/project',
tf_version='1.3.5',
tg_version="32.0.0", # only used if `binary` arg is set to `terragrunt`
gcp_iam_role=None,
aws_iam_role="arn:aws:iam::role"
)
some_task = SomeCloudOperator(
task_id="some_task",
instance_arn=terra_output["instance_arn"]
)
```
within the `execute()` of the `TerraOutputOperator`, the [tftest]() PyPi
package can be used to parse the Terraform output into a Python dictionary that
can be accessed by downstream tasks. The operator execute() function could look
like something along the lines of:
``` python
from tftest import TerraformTest, TerragruntTest
def execute(self, context: Context):
terra = TerraformHook()
# install repo containing IaC files to local tmp directory
tfdir = terra.get_source(self.repo_url)
# install terraform binary if not already installed and sets the version
in PATH
# could use terraform version manager like tfenv
terra.set_terraform_version(self.tf_version)
if self.binary == "terraform":
terra = TerraformTest(tfdir)
elif self.binary == "terragrunt":
# install terragrunt binary if not already installed and sets the
version in PATH
# could use terragrunt version manager like tgswitch
terra.set_terragrunt_version(self.tg_version)
terra = TerragruntTest(tfdir)
# parses terraform/terragrunt output results into python dictionary
return terra.output()
```
If this is within the scope of Airflow's built-in operators/hooks, I'd be
happy to make a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
