This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 9b3b403ae9c Cloudbuild+manualsetup+playground (#24144)
9b3b403ae9c is described below
commit 9b3b403ae9c071dcef0e801c892eab2203db88df
Author: ruslan-ikhsan <114978215+ruslan-ikh...@users.noreply.github.com>
AuthorDate: Mon Dec 5 23:46:30 2022 +0500
Cloudbuild+manualsetup+playground (#24144)
* Commit related to apache #24007, #24006
* Commit related to apache #24007, #24006
* Update README.md
* Updates related to apache #24007, #24006
* Updates to comments in apache #24144
* Update README.md
* Update README.md
* Update README.md
* Optimizing cloud build yaml files
* Updates related to apache #24007, #24006
* Updates related to apache #24006, #24007
* Update README.md
* Update cloudbuild_pg_to_gke.yaml
* Added curly brackets for variables
* Update cloudbuild_pg_to_gke.yaml
* Test updates for apache #24007
* Create terraform.tfvars
* Latest updates related to PR apache #24144
* Delete terraform.tfvars
* Update README.md
* Update README.md
* Update README.md
* Updates related to #24144
* Update .gitignore
* Updates related to PR #24144
* Update README.md
* Updates for apache #24144
* Update README.md
* Update README.md
* Update variables.tf
* Update state.tfbackend
* Revert "Update state.tfbackend"
This reverts commit 0aa6f1ffd81854e950b35ae5b36337e0eb693372.
* Update README.md
* Update README.md
* Update README.md
* Update variables and logging type #24144
* Updates related to tfvars file and Readme
* Updates related to apache beam PR #24144
* Updates related to apache beam PR #24144
* Update iam.tf
* Update cloudbuild_pg_infra.yaml
* Updating cloud build configs
* Update provider version to 4.0
* Update README.md
* Updates with bash scripts for cloud builder
* Update cloudbuild_pg_infra.yaml
* Update cloudbuild_pg_infra.yaml
* Updates of cloud build logic
* Update env_init.sh
* Update cloudbuild_pg_infra.yaml
* Update cloudbuild_pg_infra.yaml
* Updates to cb bash scripts
* Update cloudbuild_pg_infra.yaml
* Update cloudbuild_pg_infra.yaml
* Shell scripts updates
* Update cloudbuild_pg_infra.yaml
* Update cloudbuild_pg_infra.yaml
* Shell script updates cloud build
* Updates
* Updates related to PR #24144
* Update terraform.tf
* Update README.md
* Updates related to PR #24144
* Update cloudbuild_pg_to_gke.yaml
* Revert "Update cloudbuild_pg_to_gke.yaml"
This reverts commit 6545170605995fcf75cb628dee1c640ecd90f864.
* Update triggers.tf
* Updates with new variables for region/location #24144
* Update README.md
* Updated for SDK_TAG
* Update README.md
* Update cloudbuild_pg_to_gke.yaml
* Update cloudbuild_pg_to_gke.yaml
* Update README.md
* Update build.gradle.kts
* Updates
* Update build.gradle.kts
* Update cloudbuild_pg_to_gke.yaml
* Update Dockerfile
* Update cloudbuild_pg_to_gke.yaml
* Update build.gradle.kts
* test faster without back
* Update env_init.sh
* Updates to cloud build
* Update cloudbuild_pg_to_gke.yaml
* Update build.gradle.kts
* Updates cloud build
* Update cloudbuild_pg_to_gke.yaml
* Update env_init.sh
* Updates
* Update build.gradle.kts
* Updates
* trying to set run helm verbose
* remove unneeded task temporarily
* remove extraargs
* verbose
* remove extra args
* remove commented code
* Update build.gradle.kts
* Update build.gradle.kts
* Update build.gradle.kts
* Update build.gradle.kts
* Update build.gradle.kts
* Update cloudbuild_pg_to_gke.yaml
* Update cloudbuild_pg_to_gke.yaml
* Update build.gradle.kts
* Update build.gradle.kts
* Update build.gradle.kts
* Updates related to PR #24144
* Updates related to PR #24144
* Update README.md
* Update build.gradle.kts
* Update README.md
* Update README.md
* Update README.md
* Update variables.tf
* Update main.tf
* Revert "Update main.tf"
This reverts commit 119d31ba5fe8a78e44ebfa474601a8d6f0bee8ef.
* Revert "Update variables.tf"
This reverts commit afddc1669f64031a39009b7d5808683634e32415.
* Majority of updates following the comments in #24144
* Rest of updates following the comments in #24144
* Updates to the comments in PR #24144
* Update variables.tf
* Update variables.tf
* Update README.md
* Updated TF and CB variables to Zone
* Update cloudbuild_pg_to_gke.yaml
* Added integration between two Readme files
* Removed var.region from triggers tf script and changed readme file link
to relative #24144
* Update build.gradle.kts to resolve merge conflict
* Update build.gradke.kts file to resolve merge conflict
Co-authored-by: oborysevych <oleg.borisev...@akvelon.com>
---
.../cloudbuild/cloudbuild_pg_infra.yaml | 45 ++++++++
.../cloudbuild/cloudbuild_pg_to_gke.yaml | 48 +++++++++
playground/infrastructure/cloudbuild/env_init.sh | 44 ++++++++
playground/terraform/README.md | 1 +
playground/terraform/build.gradle.kts | 16 ++-
.../cloudbuild-manual-setup/01.setup/iam.tf | 44 ++++++++
.../cloudbuild-manual-setup/01.setup/provider.tf | 20 ++++
.../cloudbuild-manual-setup/01.setup/services.tf | 33 ++++++
.../cloudbuild-manual-setup/01.setup/terraform.tf | 28 +++++
.../cloudbuild-manual-setup/01.setup/variables.tf | 27 +++++
.../02.builders/provider.tf | 20 ++++
.../02.builders/terraform.tf | 28 +++++
.../02.builders/triggers.tf | 81 ++++++++++++++
.../02.builders/variables.tf | 108 +++++++++++++++++++
.../cloudbuild-manual-setup/README.md | 116 +++++++++++++++++++++
15 files changed, 650 insertions(+), 9 deletions(-)
diff --git a/playground/infrastructure/cloudbuild/cloudbuild_pg_infra.yaml
b/playground/infrastructure/cloudbuild/cloudbuild_pg_infra.yaml
new file mode 100644
index 00000000000..262adc469a7
--- /dev/null
+++ b/playground/infrastructure/cloudbuild/cloudbuild_pg_infra.yaml
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+steps:
+# This stage uses Docker container from image built in step above to run
gradle task for Playground Infrastructure deployment
+ - name: 'gcr.io/google.com/cloudsdktool/google-cloud-cli'
+ entrypoint: "/bin/bash"
+ args:
+ - "-c"
+ - |
+ chmod +x playground/infrastructure/cloudbuild/env_init.sh \
+ && ./playground/infrastructure/cloudbuild/env_init.sh
+ mkdir playground/terraform/environment/$_ENVIRONMENT_NAME
+ printf '%s\n' \
+ 'project_id = "$PROJECT_ID"' \
+ 'network_name = "$_NETWORK_NAME"' \
+ 'gke_name = "$_GKE_NAME"' \
+ 'region = "$_PLAYGROUND_REGION"' \
+ 'location = "$_PLAYGROUND_ZONE"' \
+ 'state_bucket = "$_STATE_BUCKET"' \
+ > playground/terraform/environment/$_ENVIRONMENT_NAME/terraform.tfvars
+ printf \
+ 'bucket = "$_STATE_BUCKET"'\
+ > playground/terraform/environment/$_ENVIRONMENT_NAME/state.tfbackend
+ ./gradlew playground:terraform:InitInfrastructure
-Pproject_environment="$_ENVIRONMENT_NAME" -Pdns-name="$_DNS_NAME"
+
+# This option enables writing logs to Cloud Logging
+options:
+ logging: CLOUD_LOGGING_ONLY
+
+timeout: 3600s
\ No newline at end of file
diff --git a/playground/infrastructure/cloudbuild/cloudbuild_pg_to_gke.yaml
b/playground/infrastructure/cloudbuild/cloudbuild_pg_to_gke.yaml
new file mode 100644
index 00000000000..3d9c8d40566
--- /dev/null
+++ b/playground/infrastructure/cloudbuild/cloudbuild_pg_to_gke.yaml
@@ -0,0 +1,48 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+steps:
+ # This stage uses pre-built Docker container to run gradle task for
Playground pre-config and deployment to GKE
+ - name: 'gcr.io/google.com/cloudsdktool/google-cloud-cli'
+ entrypoint: "/bin/bash"
+ args:
+ - "-c"
+ - |
+ chmod +x playground/infrastructure/cloudbuild/env_init.sh \
+ && ./playground/infrastructure/cloudbuild/env_init.sh
+ mkdir playground/terraform/environment/$_ENVIRONMENT_NAME
+ printf '%s\n' \
+ 'project_id = "$PROJECT_ID"' \
+ 'network_name = "$_NETWORK_NAME"' \
+ 'gke_name = "$_GKE_NAME"' \
+ 'region = "$_PLAYGROUND_REGION"' \
+ 'location = "$_PLAYGROUND_ZONE"' \
+ 'state_bucket = "$_STATE_BUCKET"' \
+ > playground/terraform/environment/$_ENVIRONMENT_NAME/terraform.tfvars
+ printf \
+ 'bucket = "$_STATE_BUCKET"'\
+ > playground/terraform/environment/$_ENVIRONMENT_NAME/state.tfbackend
+ gcloud container clusters get-credentials --region '$_PLAYGROUND_ZONE'
'$_GKE_NAME' --project '$PROJECT_ID'
+ ./gradlew playground:terraform:prepareConfig -Pdns-name="$_DNS_NAME"
+ ./gradlew playground:terraform:gkebackend
-Pdocker-repository-root="$_DOCKER_REPOSITORY_ROOT" \
+ -Pproject_environment="$_ENVIRONMENT_NAME" -Pdocker-tag="$_TAG"
-Psdk-tag="$_SDK_TAG" -Pdns-name="$_DNS_NAME"
+
+# This option enables writing logs to Cloud Logging
+options:
+ logging: CLOUD_LOGGING_ONLY
+
+timeout: 7600s
\ No newline at end of file
diff --git a/playground/infrastructure/cloudbuild/env_init.sh
b/playground/infrastructure/cloudbuild/env_init.sh
new file mode 100644
index 00000000000..c612472433b
--- /dev/null
+++ b/playground/infrastructure/cloudbuild/env_init.sh
@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apt-get update > /dev/null
+
+# Install dependencies
+apt-get install -y build-essential unzip apt-transport-https ca-certificates
curl software-properties-common gnupg2 wget > /dev/null
+
+# Install Docker
+curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
+add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian
$(lsb_release -cs) stable" > /dev/null
+apt update > /dev/null && apt install -y docker-ce > /dev/null
+
+#Install Helm
+curl -fsSLo get_helm.sh
https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 > /dev/null
+chmod +x get_helm.sh && ./get_helm.sh > /dev/null
+
+# Install Terraform
+wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor | tee
/usr/share/keyrings/hashicorp-archive-keyring.gpg
+echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg]
https://apt.releases.hashicorp.com $(lsb_release -cs) main" \
+| tee /etc/apt/sources.list.d/hashicorp.list
+apt update -y > /dev/null && apt install -y terraform > /dev/null
+
+# Install kubectl
+curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s
https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl"
\
+&& chmod +x ./kubectl \
+&& mv ./kubectl /usr/local/bin/kubectl
+
+# Install jdk
+apt-get install openjdk-11-jdk -y > /dev/null
\ No newline at end of file
diff --git a/playground/terraform/README.md b/playground/terraform/README.md
index 5c431d6a0c0..2e1449723d6 100644
--- a/playground/terraform/README.md
+++ b/playground/terraform/README.md
@@ -18,6 +18,7 @@
-->
# Playground deployment on GCP
This guide shows you how to deploy full Playground environment on Google Cloud
Platform (GCP) environment.
+Alternatively, you can automate Playground environment deployment with Cloud
Build as described in
[readme](infrastructure/cloudbuild-manual-setup/README.md).
## Prerequisites:
diff --git a/playground/terraform/build.gradle.kts
b/playground/terraform/build.gradle.kts
index e3be9773540..cb3241ebf7b 100644
--- a/playground/terraform/build.gradle.kts
+++ b/playground/terraform/build.gradle.kts
@@ -464,6 +464,7 @@ dns_name: ${dns_name}
""")
}
}
+
helm {
val playground by charts.creating {
chartName.set("playground")
@@ -478,23 +479,20 @@ helm {
tasks.register("gkebackend") {
group = "deploy"
val init = tasks.getByName("terraformInit")
- val apply = tasks.getByName("terraformApplyInf")
- val indexcreate = tasks.getByName("indexcreate")
val takeConfig = tasks.getByName("takeConfig")
+ val back = tasks.getByName("pushBack")
val front = tasks.getByName("pushFront")
- val push = tasks.getByName("pushBack")
+ val indexcreate = tasks.getByName("indexcreate")
val helm = tasks.getByName("helmInstallPlayground")
dependsOn(init)
- dependsOn(apply)
dependsOn(takeConfig)
- dependsOn(push)
+ dependsOn(back)
dependsOn(front)
dependsOn(indexcreate)
dependsOn(helm)
- apply.mustRunAfter(init)
- takeConfig.mustRunAfter(apply)
- push.mustRunAfter(takeConfig)
- front.mustRunAfter(push)
+ takeConfig.mustRunAfter(init)
+ back.mustRunAfter(takeConfig)
+ front.mustRunAfter(back)
indexcreate.mustRunAfter(front)
helm.mustRunAfter(indexcreate)
}
diff --git
a/playground/terraform/infrastructure/cloudbuild-manual-setup/01.setup/iam.tf
b/playground/terraform/infrastructure/cloudbuild-manual-setup/01.setup/iam.tf
new file mode 100644
index 00000000000..3117134ffcc
--- /dev/null
+++
b/playground/terraform/infrastructure/cloudbuild-manual-setup/01.setup/iam.tf
@@ -0,0 +1,44 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+resource "google_service_account" "cloudbuild_service_account_id" {
+ account_id = var.cloudbuild_service_account_id
+ display_name = var.cloudbuild_service_account_id
+ description = "The service account cloud build will use to deploy
Playground"
+}
+
+// Provision IAM roles to the IaC service account required to build and
provision resources
+resource "google_project_iam_member" "cloud_build_roles" {
+ for_each = toset([
+ "roles/appengine.appAdmin",
+ "roles/appengine.appCreator",
+ "roles/artifactregistry.admin",
+ "roles/redis.admin",
+ "roles/compute.admin",
+ "roles/iam.serviceAccountCreator",
+ "roles/container.admin",
+ "roles/servicemanagement.quotaAdmin",
+ "roles/iam.securityAdmin",
+ "roles/iam.serviceAccountUser",
+ "roles/datastore.indexAdmin",
+ "roles/storage.admin",
+ "roles/logging.logWriter"
+ ])
+ role = each.key
+ member =
"serviceAccount:${google_service_account.cloudbuild_service_account_id.email}"
+ project = var.project_id
+}
\ No newline at end of file
diff --git
a/playground/terraform/infrastructure/cloudbuild-manual-setup/01.setup/provider.tf
b/playground/terraform/infrastructure/cloudbuild-manual-setup/01.setup/provider.tf
new file mode 100644
index 00000000000..c23ddd6f9bf
--- /dev/null
+++
b/playground/terraform/infrastructure/cloudbuild-manual-setup/01.setup/provider.tf
@@ -0,0 +1,20 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+provider "google" {
+ project = var.project_id
+}
\ No newline at end of file
diff --git
a/playground/terraform/infrastructure/cloudbuild-manual-setup/01.setup/services.tf
b/playground/terraform/infrastructure/cloudbuild-manual-setup/01.setup/services.tf
new file mode 100644
index 00000000000..d44822e37ba
--- /dev/null
+++
b/playground/terraform/infrastructure/cloudbuild-manual-setup/01.setup/services.tf
@@ -0,0 +1,33 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+// Provision the required Google Cloud services
+resource "google_project_service" "required_services" {
+ project = var.project_id
+ for_each = toset([
+ "cloudresourcemanager",
+ "cloudbuild",
+ "appengine",
+ "artifactregistry",
+ "compute",
+ "redis",
+ "iam",
+ ])
+
+ service = "${each.key}.googleapis.com"
+ disable_on_destroy = false
+}
\ No newline at end of file
diff --git
a/playground/terraform/infrastructure/cloudbuild-manual-setup/01.setup/terraform.tf
b/playground/terraform/infrastructure/cloudbuild-manual-setup/01.setup/terraform.tf
new file mode 100644
index 00000000000..e7baa96ccd7
--- /dev/null
+++
b/playground/terraform/infrastructure/cloudbuild-manual-setup/01.setup/terraform.tf
@@ -0,0 +1,28 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+terraform {
+ backend "gcs" {
+ prefix = "01.setup"
+ }
+ required_providers {
+ google = {
+ source = "hashicorp/google"
+ version = "~> 4.0.0"
+ }
+ }
+}
\ No newline at end of file
diff --git
a/playground/terraform/infrastructure/cloudbuild-manual-setup/01.setup/variables.tf
b/playground/terraform/infrastructure/cloudbuild-manual-setup/01.setup/variables.tf
new file mode 100644
index 00000000000..3ab4de68af3
--- /dev/null
+++
b/playground/terraform/infrastructure/cloudbuild-manual-setup/01.setup/variables.tf
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+variable "project_id" {
+ type = string
+ description = "The ID of the Google Cloud project within which resources are
provisioned"
+}
+
+variable "cloudbuild_service_account_id" {
+ type = string
+ description = "The ID of the cloud build service account responsible for
provisioning Google Cloud resources"
+ default = "playground-cloudbuild-sa"
+}
diff --git
a/playground/terraform/infrastructure/cloudbuild-manual-setup/02.builders/provider.tf
b/playground/terraform/infrastructure/cloudbuild-manual-setup/02.builders/provider.tf
new file mode 100644
index 00000000000..c23ddd6f9bf
--- /dev/null
+++
b/playground/terraform/infrastructure/cloudbuild-manual-setup/02.builders/provider.tf
@@ -0,0 +1,20 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+provider "google" {
+ project = var.project_id
+}
\ No newline at end of file
diff --git
a/playground/terraform/infrastructure/cloudbuild-manual-setup/02.builders/terraform.tf
b/playground/terraform/infrastructure/cloudbuild-manual-setup/02.builders/terraform.tf
new file mode 100644
index 00000000000..e0da73116ef
--- /dev/null
+++
b/playground/terraform/infrastructure/cloudbuild-manual-setup/02.builders/terraform.tf
@@ -0,0 +1,28 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+terraform {
+ backend "gcs" {
+ prefix = "02.builders"
+ }
+ required_providers {
+ google = {
+ source = "hashicorp/google"
+ version = "~> 4.40.0"
+ }
+ }
+}
\ No newline at end of file
diff --git
a/playground/terraform/infrastructure/cloudbuild-manual-setup/02.builders/triggers.tf
b/playground/terraform/infrastructure/cloudbuild-manual-setup/02.builders/triggers.tf
new file mode 100644
index 00000000000..5fb2957eea7
--- /dev/null
+++
b/playground/terraform/infrastructure/cloudbuild-manual-setup/02.builders/triggers.tf
@@ -0,0 +1,81 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+data "google_service_account" "cloudbuild_sa" {
+ account_id = var.cloudbuild_service_account_id
+}
+
+resource "google_cloudbuild_trigger" "playground_infrastructure" {
+ name = var.infra_trigger_name
+ project = var.project_id
+
+ description = "Builds the base image and then runs cloud build config file
to deploy Playground infrastructure"
+
+ github {
+ owner = var.github_repository_owner
+ name = var.github_repository_name
+ push {
+ branch = var.github_repository_branch
+ }
+ }
+
+ substitutions = {
+ _PLAYGROUND_REGION : var.playground_region
+ _PLAYGROUND_ZONE : var.playground_zone
+ _ENVIRONMENT_NAME : var.playground_environment_name
+ _DNS_NAME : var.playground_dns_name
+ _NETWORK_NAME : var.playground_network_name
+ _GKE_NAME : var.playground_gke_name
+ _STATE_BUCKET : var.state_bucket
+ }
+
+ filename = "playground/infrastructure/cloudbuild/cloudbuild_pg_infra.yaml"
+
+ service_account = data.google_service_account.cloudbuild_sa.id
+}
+
+resource "google_cloudbuild_trigger" "playground_to_gke" {
+ name = var.gke_trigger_name
+ project = var.project_id
+
+ description = "Builds the base image and then runs cloud build config file
to deploy Playground to GKE"
+
+ github {
+ owner = var.github_repository_owner
+ name = var.github_repository_name
+ push {
+ branch = var.github_repository_branch
+ }
+ }
+
+ substitutions = {
+ _PLAYGROUND_REGION : var.playground_region
+ _PLAYGROUND_ZONE : var.playground_zone
+ _ENVIRONMENT_NAME : var.playground_environment_name
+ _DNS_NAME : var.playground_dns_name
+ _NETWORK_NAME : var.playground_network_name
+ _GKE_NAME : var.playground_gke_name
+ _STATE_BUCKET : var.state_bucket
+ _TAG : var.image_tag
+ _DOCKER_REPOSITORY_ROOT : var.docker_repository_root
+ _SDK_TAG : var.sdk_tag
+ }
+
+ filename = "playground/infrastructure/cloudbuild/cloudbuild_pg_to_gke.yaml"
+
+ service_account = data.google_service_account.cloudbuild_sa.id
+}
\ No newline at end of file
diff --git
a/playground/terraform/infrastructure/cloudbuild-manual-setup/02.builders/variables.tf
b/playground/terraform/infrastructure/cloudbuild-manual-setup/02.builders/variables.tf
new file mode 100644
index 00000000000..4316d2406f2
--- /dev/null
+++
b/playground/terraform/infrastructure/cloudbuild-manual-setup/02.builders/variables.tf
@@ -0,0 +1,108 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+variable "project_id" {
+ type = string
+ description = "The ID of the Google Cloud project within which resources are
provisioned"
+}
+
+variable "region" {
+ type = string
+ description = "The Google Cloud Platform (GCP) region (For example:
us-central1) where Cloud Build triggers will be created at"
+}
+
+variable "infra_trigger_name" {
+ type = string
+ description = "The name of the trigger that will deploy Playground
infrastructure"
+ default = "playground-infrastructure-trigger"
+}
+
+variable "gke_trigger_name" {
+ type = string
+ description = "The name of the trigger that will deploy Playground to GKE"
+ default = "playground-to-gke-trigger"
+}
+
+variable "cloudbuild_service_account_id" {
+ type = string
+ description = "The ID of the cloud build service account responsible for
provisioning Google Cloud resources"
+ default = "playground-cloudbuild-sa"
+}
+
+variable "github_repository_name" {
+ type = string
+ description = "The name of the GitHub repository. For example the repository
name for https://github.com/example/foo is 'foo'."
+}
+
+variable "github_repository_owner" {
+ type = string
+ description = "The owner of the GitHub repository. For example the owner for
https://github.com/example/foo is 'example'."
+}
+
+variable "github_repository_branch" {
+ type = string
+ description = "The GitHub repository branch regex to match cloud build
trigger"
+}
+
+variable "playground_environment_name" {
+ description = <<EOF
+Environment name where to deploy Playground. Located in
playground/terraform/environment/{environment_name}. E.g. test, dev, prod.
+More details:
https://github.com/akvelon/beam/blob/cloudbuild%2Bmanualsetup%2Bplayground/playground/terraform/README.md#prepare-deployment-configuration";
+ EOF
+}
+
+variable "playground_dns_name" {
+ description = <<EOF
+The DNS record name for Playground website.
+More details:
https://github.com/apache/beam/blob/master/playground/terraform/README.md#deploy-playground-infrastructure";
+ EOF
+}
+
+variable "playground_network_name" {
+ description = "The Google Cloud Platform (GCP) VPC Network Name for
Playground deployment"
+}
+
+variable "playground_gke_name" {
+ description = "The Playground GKE Cluster name in Google Cloud Platform
(GCP)"
+}
+
+variable "state_bucket" {
+ description = "The Google Cloud Platform (GCP) GCS bucket name for Beam
Playground temp files and Terraform state"
+}
+
+variable "image_tag" {
+ description = "The docker images tag for Playground containers"
+}
+
+variable "docker_repository_root" {
+ description = "The name of Google Cloud Platform (GCP) Artifact Registry
Repository where Playground images will be saved to"
+}
+
+variable "playground_region" {
+ description = "The Google Cloud Platform (GCP) region (For example:
us-central1) where playground infrastructure will be deployed to"
+}
+
+variable "playground_zone" {
+ description = "The Google Cloud Platform (GCP) zone (For example:
us-central1-b) where playground infrastructure will be deployed to"
+}
+
+variable "sdk_tag" {
+ description = <<EOF
+Apache Beam Golang and Python images SDK tag. (For example: 2.43.0)
+See more: https://hub.docker.com/r/apache/beam_python3.7_sdk/tags and
https://hub.docker.com/r/apache/beam_go_sdk";
+ EOF
+}
\ No newline at end of file
diff --git
a/playground/terraform/infrastructure/cloudbuild-manual-setup/README.md
b/playground/terraform/infrastructure/cloudbuild-manual-setup/README.md
new file mode 100644
index 00000000000..235dfe71275
--- /dev/null
+++ b/playground/terraform/infrastructure/cloudbuild-manual-setup/README.md
@@ -0,0 +1,116 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Beam Playground Cloud Build Setup
+
+This directory organizes Infrastructure-as-Code to provision dependent
resources and set up Cloud Build for Beam Playground.
+Cloud Build triggers created by terraform scripts from this directory automate
steps described in
[readme](https://github.com/apache/beam/blob/master/playground/terraform/README.md).
+
+## Requirements:
+
+- [GCP
project](https://cloud.google.com/resource-manager/docs/creating-managing-projects)
+- [GCP User
account](https://cloud.google.com/appengine/docs/standard/access-control?tab=python)
_(Note: You will find the instruction "How to create User account" for your
new project)_<br>
+ Ensure that the account has at least the following [IAM
roles](https://cloud.google.com/iam/docs/understanding-roles):
+ - Service Account Admin
+ - Storage Admin
+ - Service Usage Admin
+ - Cloud Build Editor
+ - Security Admin
+ - Service Account User
+- [gcloud CLI](https://cloud.google.com/sdk/docs/install-sdk)
+- An existing GCP Bucket to save Terraform state - `state-bucket`
+- DNS name for your Playground deployment instance
+- [Terraform](https://www.terraform.io/)
+- [Apache Beam GitHub](https://github.com/apache/beam) repository cloned
locally
+
+## 1. Set up the Google Cloud Build for your GCP project
+
+The `playground/terraform/infrastructure/cloudbuild-manual-setup/01.setup`
provisions dependencies required to set up Cloud Build for Playground:
+- Required API services
+- Cloud Build service account
+- IAM roles for Cloud Build service account
+
+#### To execute the module:
+
+**Note:** Please see [Cloud Build
locations](https://cloud.google.com/build/docs/locations) for the list of all
supported locations.
+
+1. Run commands:
+
+
+```console
+# Set environment variable for state bucket
+export STATE_BUCKET="state-bucket"
+
+# Create a new authentication configuration for GCP Project with the created
user account
+gcloud init
+
+# Command imports new user account credentials into Application Default
Credentials
+gcloud auth application-default login
+
+# Navigate to 01.setup directory
+cd playground/terraform/infrastructure/cloudbuild-manual-setup/01.setup/
+
+# Run terraform commands
+terraform init -backend-config="bucket=$STATE_BUCKET"
+terraform apply -var="project_id=$(gcloud config get-value project)"
+```
+
+## 2. Connect Apache Beam GitHub repository and GCP Cloud Build
+
+**Note:** Ensure correct `region` is set in [Cloud Build
Triggers](https://console.cloud.google.com/cloud-build/triggers) page before
proceeding further.
+
+Follow [Connect to a GitHub
repository](https://cloud.google.com/build/docs/automating-builds/github/connect-repo-github)
to connect Apache Beam GitHub repository and GCP Cloud Build.
+
+## 3. Set up the Google Cloud Build triggers
+
+The `playground/terraform/infrastructure/cloudbuild-manual-setup/02.builders`
provisions:
+- Cloud Build triggers to build and deploy Beam Playground
+
+#### To execute the module
+
+```
+# Navigate to
playground/terraform/infrastructure/cloudbuild-manual-setup/02.builders
directory
+cd ../02.builders
+
+# Run terraform commands
+terraform init -backend-config="bucket=$STATE_BUCKET"
+terraform apply -var="project_id=$(gcloud config get-value project)"
-var="state_bucket=$STATE_BUCKET"
+```
+
+## 4. Run Cloud Build `Playground-infrastructure-trigger` to deploy Playground
infrastructure
+
+1. Navigate to [GCP Console Cloud Build
Triggers](https://console.cloud.google.com/cloud-build/triggers) page. Choose
the region (In our example: us-central1).
+2. Open Trigger: `Playground-infrastructure-trigger`.
+3. Scroll down to `Source` - `Repository` to ensure that Apache Beam GitHub
repository is connected.
+ - Click on drop-down menu and press `CONNECT NEW REPOSITORY` in case it was
not automatically connected.
+4. Click `Save` and Run the trigger `Playground-infrastructure-trigger`.
+
+5. Once Playground infrastructure has been deployed, please navigate to
+ [Playground deployment
README](https://github.com/apache/beam/tree/master/playground/terraform#deploy-playground-infrastructure)
and execute step #2:
+ `Add following DNS A records for the discovered static IP address`
expanding use of variable `DNS_NAME`.
+
+## 5. Run Cloud Build `Playground-to-gke-trigger` to deploy Playground to GKE
+
+1. Navigate to [GCP Console Cloud Build
Triggers](https://console.cloud.google.com/cloud-build/triggers) page. Choose
the region (In our example: us-central1).
+2. Open Trigger: `Playground-to-gke-trigger`.
+3. Scroll down to `Source` - `Repository` to ensure that Apache Beam GitHub
repository is connected.
+ - Click on drop-down menu and press `CONNECT NEW REPOSITORY` in case it
was not automatically connected.
+4. Click `Save` and Run the trigger `Playground-to-gke-trigger`.
+
+## 6. Validate Playground deployment
+
+Once Playground has been deployed to GKE, please navigate to
[Validation](https://github.com/apache/beam/tree/master/playground/terraform#validate-deployed-playground)
to perform Playground deployment steps.
\ No newline at end of file
