Pablo Alejandro Costesich's Blog

The Build Pipeline

Posted by @pcostesi on 2021-07-14Time to read: 25 minutes

This blog is a static site built on Gatsby, as previously stated. As such, it requires a place to host the source code and another to host the content (which will be the subject of a follow-up post).

It's trivial to build the site locally. As with any Gatsby project, the only step is to run npm run build and then the output will be readily available in the /public/ folder.

UPDATE 2021-07-14: I have split this blog post into a series. This post just contains the basic outline of what we're going to automate later with Terraform. I will cover how to minimize your expenses by using a third party certificate provider.

The Need for Automation

Repetitive tasks are error-prone and take time from pressing issues and important goals. As such, the build and deployment pipeline reacts to the push events on the main branch and triggers the npm command, storing the result in a GCS bucket.

This process is carried out by Google Cloud Build using the following cloudbuild.yaml file in the root of the project:

resource "google_cloudbuild_trigger" "site_trigger" {
provider = google-beta
name = "${var.project}-trigger"
project = var.project
github {
push {
branch = "master"
}
owner = "pcostesi"
name = var.static_site
}
filename = "cloudbuild.yaml"
}
view raw build.tf hosted with ❤ by GitHub
steps:
- name: node:current
entrypoint: npm
args: ["install"]
- name: node:current
entrypoint: npm
args: ["run", "build"]
- name: gcr.io/cloud-builders/gsutil
args: ["-m", "rsync", "-r", "-c", "-d", "./public", "gs://pcostesi.dev"]
view raw cloudbuild.yaml hosted with ❤ by GitHub
# Global IP, needed for the load balancer (maybe this should go to `routing.tf`)
resource "google_compute_global_address" "static_sites" {
provider = google
name = "static-sites"
}
# Managed DNS zone
data "google_dns_managed_zone" "main_site" {
provider = google
name = var.project
}
# Bind IP to the DNS
resource "google_dns_record_set" "static_site" {
provider = google
name = "${var.static_site}."
type = "A"
ttl = 300
managed_zone = data.google_dns_managed_zone.main_site.name
rrdatas = [google_compute_global_address.static_sites.address]
}
# Site verification
resource "google_dns_record_set" "static_site_txt" {
provider = google
name = "${var.static_site}."
type = "TXT"
ttl = 300
managed_zone = data.google_dns_managed_zone.main_site.name
rrdatas = ["google-site-verification=${var.site_verification}"]
}
view raw dns.tf hosted with ❤ by GitHub
terraform {
required_providers {
google = {
source = "hashicorp/google"
}
google-beta = {
source = "hashicorp/google-beta"
}
}
backend "gcs"{
bucket = "a-bucket-for-storing-terraform-state"
prefix = "blog"
credentials = "/path/to/your/google/credentials.json"
}
}
provider "google" {
credentials = file(var.credentials)
project = var.project
region = var.region
zone = var.zone
}
provider "google-beta" {
credentials = file(var.credentials)
project = var.project
region = var.region
zone = var.zone
}
view raw main.tf hosted with ❤ by GitHub
resource "google_monitoring_notification_channel" "basic" {
display_name = "Main Notification Channel"
type = "email"
project = var.project
provider = google-beta
labels = {
email_address = var.email_address
}
}
resource "google_monitoring_uptime_check_config" "static_site_https" {
provider = google-beta
display_name = "${var.project}-https-uptime-check"
timeout = "60s"
project = var.project
http_check {
path = "/"
port = "443"
use_ssl = true
validate_ssl = true
}
monitored_resource {
type = "uptime_url"
labels = {
project_id = var.project
host = var.static_site
}
}
}
resource "google_monitoring_alert_policy" "static_site_alert_policy" {
display_name = "Static Site Alert Policy"
combiner = "OR"
conditions {
display_name = "SSL certificate expiring soon"
condition_threshold {
filter = "metric.type=\"monitoring.googleapis.com/uptime_check/time_until_ssl_cert_expires\" AND resource.type=\"uptime_url\""
duration = "600s"
comparison = "COMPARISON_LT"
threshold_value = 15
trigger {
count = 1
}
aggregations {
alignment_period = "1200s"
per_series_aligner = "ALIGN_NEXT_OLDER"
cross_series_reducer = "REDUCE_MEAN"
group_by_fields = ["resource.label.*"]
}
}
}
user_labels = {
"uptime" = "ssl_cert_expiration"
"version" = "1"
}
notification_channels = [ google_monitoring_notification_channel.basic.name ]
}
view raw monitoring.tf hosted with ❤ by GitHub
# Add the bucket as a CDN backend
resource "google_compute_backend_bucket" "static_site" {
provider = google
name = "${var.project}-backend"
description = "Contains files needed by the website"
bucket_name = google_storage_bucket.static_site.name
enable_cdn = true
depends_on = [google_dns_record_set.static_site_txt]
}
# HTTPS certificate
resource "google_compute_managed_ssl_certificate" "static_site" {
provider = google-beta
name = "${var.project}-cert"
managed {
domains = [google_dns_record_set.static_site.name]
}
}
# GCP URL MAP (frontend config)
resource "google_compute_url_map" "static_site" {
provider = google
name = "${var.project}-url-map"
default_service = google_compute_backend_bucket.static_site.self_link
}
# GCP proxy (load balancer)
resource "google_compute_target_https_proxy" "static_site" {
provider = google
name = "${var.project}-target-proxy"
url_map = google_compute_url_map.static_site.self_link
ssl_certificates = [google_compute_managed_ssl_certificate.static_site.self_link]
}
# GCP forwarding rule (IP <-> Load Balancer mapping)
resource "google_compute_global_forwarding_rule" "default" {
provider = google
name = "${var.project}-forwarding-rule"
load_balancing_scheme = "EXTERNAL"
ip_address = google_compute_global_address.static_sites.address
ip_protocol = "TCP"
port_range = "443"
target = google_compute_target_https_proxy.static_site.self_link
}
view raw routing.tf hosted with ❤ by GitHub
resource "google_storage_bucket" "static_site" {
name = var.static_site
force_destroy = true
uniform_bucket_level_access = false
website {
main_page_suffix = "index.html"
not_found_page = "404.html"
}
cors {
origin = ["https://${var.static_site}"]
method = ["GET", "HEAD", "PUT", "POST", "DELETE"]
response_header = ["*"]
max_age_seconds = 3600
}
}
resource "google_storage_default_object_acl" "static_site_read" {
bucket = google_storage_bucket.static_site.name
role_entity = ["READER:allUsers"]
}
view raw storage.tf hosted with ❤ by GitHub
variable "project" {
default = "pcostesi-dev"
}
variable "credentials" {
default = "../credentials.json"
}
variable "region" {
default = "us-central1"
}
variable "zone" {
default = "us-central1-c"
}
variable "name" {
default = "main"
}
variable "static_site" {
default = "pcostesi.dev"
}
variable "email_address" {}
variable "site_verification" {}
view raw variables.tf hosted with ❤ by GitHub

Please, keep in mind this is the bare minimum to get this going. Caching node_modules and .cache would speed up the process, and the rsync step needs to set up compression, too.

Infrastructure as Code

If we ever want to deploy another instance of this site (or any other), we would have to repeat the process, potentially making a mistake. I'm going to cover this topic in a follow-up post, but for now let's just pretend we're provisioning everything manually (the pipeline, bucket, and everything else).

A Breakdown of the Pipeline

Before We Continue

Make sure to create a service account with limited access to the resources we are going to create, and to enable the Google Cloud Build app in GitHub, as well as registering the relevant service account in the Google Search Console and to follow the procedures needed for verifying the site.

Storage

We need to create a bucket with a public read policy so the load balancer can fetch the files. We also have to set up the 404 and index pages here.

CI/CD Pipeline

The pipeline is everything Google CloudBuild needs to build and push the site to the bucket. The cloudbuild.yaml file contains all the steps to do so, and it goesin the blog repo:

resource "google_cloudbuild_trigger" "site_trigger" {
provider = google-beta
name = "${var.project}-trigger"
project = var.project
github {
push {
branch = "master"
}
owner = "pcostesi"
name = var.static_site
}
filename = "cloudbuild.yaml"
}
view raw build.tf hosted with ❤ by GitHub
steps:
- name: node:current
entrypoint: npm
args: ["install"]
- name: node:current
entrypoint: npm
args: ["run", "build"]
- name: gcr.io/cloud-builders/gsutil
args: ["-m", "rsync", "-r", "-c", "-d", "./public", "gs://pcostesi.dev"]
view raw cloudbuild.yaml hosted with ❤ by GitHub
# Global IP, needed for the load balancer (maybe this should go to `routing.tf`)
resource "google_compute_global_address" "static_sites" {
provider = google
name = "static-sites"
}
# Managed DNS zone
data "google_dns_managed_zone" "main_site" {
provider = google
name = var.project
}
# Bind IP to the DNS
resource "google_dns_record_set" "static_site" {
provider = google
name = "${var.static_site}."
type = "A"
ttl = 300
managed_zone = data.google_dns_managed_zone.main_site.name
rrdatas = [google_compute_global_address.static_sites.address]
}
# Site verification
resource "google_dns_record_set" "static_site_txt" {
provider = google
name = "${var.static_site}."
type = "TXT"
ttl = 300
managed_zone = data.google_dns_managed_zone.main_site.name
rrdatas = ["google-site-verification=${var.site_verification}"]
}
view raw dns.tf hosted with ❤ by GitHub
terraform {
required_providers {
google = {
source = "hashicorp/google"
}
google-beta = {
source = "hashicorp/google-beta"
}
}
backend "gcs"{
bucket = "a-bucket-for-storing-terraform-state"
prefix = "blog"
credentials = "/path/to/your/google/credentials.json"
}
}
provider "google" {
credentials = file(var.credentials)
project = var.project
region = var.region
zone = var.zone
}
provider "google-beta" {
credentials = file(var.credentials)
project = var.project
region = var.region
zone = var.zone
}
view raw main.tf hosted with ❤ by GitHub
resource "google_monitoring_notification_channel" "basic" {
display_name = "Main Notification Channel"
type = "email"
project = var.project
provider = google-beta
labels = {
email_address = var.email_address
}
}
resource "google_monitoring_uptime_check_config" "static_site_https" {
provider = google-beta
display_name = "${var.project}-https-uptime-check"
timeout = "60s"
project = var.project
http_check {
path = "/"
port = "443"
use_ssl = true
validate_ssl = true
}
monitored_resource {
type = "uptime_url"
labels = {
project_id = var.project
host = var.static_site
}
}
}
resource "google_monitoring_alert_policy" "static_site_alert_policy" {
display_name = "Static Site Alert Policy"
combiner = "OR"
conditions {
display_name = "SSL certificate expiring soon"
condition_threshold {
filter = "metric.type=\"monitoring.googleapis.com/uptime_check/time_until_ssl_cert_expires\" AND resource.type=\"uptime_url\""
duration = "600s"
comparison = "COMPARISON_LT"
threshold_value = 15
trigger {
count = 1
}
aggregations {
alignment_period = "1200s"
per_series_aligner = "ALIGN_NEXT_OLDER"
cross_series_reducer = "REDUCE_MEAN"
group_by_fields = ["resource.label.*"]
}
}
}
user_labels = {
"uptime" = "ssl_cert_expiration"
"version" = "1"
}
notification_channels = [ google_monitoring_notification_channel.basic.name ]
}
view raw monitoring.tf hosted with ❤ by GitHub
# Add the bucket as a CDN backend
resource "google_compute_backend_bucket" "static_site" {
provider = google
name = "${var.project}-backend"
description = "Contains files needed by the website"
bucket_name = google_storage_bucket.static_site.name
enable_cdn = true
depends_on = [google_dns_record_set.static_site_txt]
}
# HTTPS certificate
resource "google_compute_managed_ssl_certificate" "static_site" {
provider = google-beta
name = "${var.project}-cert"
managed {
domains = [google_dns_record_set.static_site.name]
}
}
# GCP URL MAP (frontend config)
resource "google_compute_url_map" "static_site" {
provider = google
name = "${var.project}-url-map"
default_service = google_compute_backend_bucket.static_site.self_link
}
# GCP proxy (load balancer)
resource "google_compute_target_https_proxy" "static_site" {
provider = google
name = "${var.project}-target-proxy"
url_map = google_compute_url_map.static_site.self_link
ssl_certificates = [google_compute_managed_ssl_certificate.static_site.self_link]
}
# GCP forwarding rule (IP <-> Load Balancer mapping)
resource "google_compute_global_forwarding_rule" "default" {
provider = google
name = "${var.project}-forwarding-rule"
load_balancing_scheme = "EXTERNAL"
ip_address = google_compute_global_address.static_sites.address
ip_protocol = "TCP"
port_range = "443"
target = google_compute_target_https_proxy.static_site.self_link
}
view raw routing.tf hosted with ❤ by GitHub
resource "google_storage_bucket" "static_site" {
name = var.static_site
force_destroy = true
uniform_bucket_level_access = false
website {
main_page_suffix = "index.html"
not_found_page = "404.html"
}
cors {
origin = ["https://${var.static_site}"]
method = ["GET", "HEAD", "PUT", "POST", "DELETE"]
response_header = ["*"]
max_age_seconds = 3600
}
}
resource "google_storage_default_object_acl" "static_site_read" {
bucket = google_storage_bucket.static_site.name
role_entity = ["READER:allUsers"]
}
view raw storage.tf hosted with ❤ by GitHub
variable "project" {
default = "pcostesi-dev"
}
variable "credentials" {
default = "../credentials.json"
}
variable "region" {
default = "us-central1"
}
variable "zone" {
default = "us-central1-c"
}
variable "name" {
default = "main"
}
variable "static_site" {
default = "pcostesi.dev"
}
variable "email_address" {}
variable "site_verification" {}
view raw variables.tf hosted with ❤ by GitHub

Serving the site

Once we have all these resources, we need to set up the domain name, IP, verify our ownership of the bucket, and point the A record.

Note: I set up the site verification id before creating the bucket. This is a domain-wide verification id, and the service account creating the bucket is an owner in the search console.

Routing is quite complex:

  • We need to create a load balancer...
  • ...and a backend, which points to the bucket.
  • Then, we need to set-up forwarding rules (frontend) so a domain or url pattern gets routed to the backend we just selected. This is crucial to set-up http and https.
  • Finally, we have to add a certificate.