Blue Green Deployment or How You Can Test on Production and Not Kill Your Project
April 4th, 2017
Managing releases is one of the major causes of headaches for DevOps and the rest of the project team alike. Especially for large enterprise products, continuous delivery is required and here's one of the ways how to do it right.
This is one of the most powerful techniques we know for managing releases. The idea is to have two identical versions of your production environment, which we’ll call blue and green. ("Continuous Delivery" by Jez Humble and David Farley)
The main idea of the approach is to have two easily switchable project environments so that you can quickly switch to another one for updates. There are many ways to do this when it comes to architecture, but the one we're going to focus for this article is Blue-Green Deployment. In our case, the environments use the same database, while the switch is done for the web and domain levels.
Benefits
- You can test your production without pausing the production
- This way offers a rapid way to rollback
Drawbacks
- The project becomes more complex (which is solved by automating the workflow)
- The database might be incompatible (which is solved by database refactoring techniques)
How do we this for one of our projects?
We have a client who has a product that is definitely a high-load one and requests have to be managed fast and efficiently. An additional fact about the system - there are many updates and features coming quite often, therefore a coherent and working continuous delivery system is a must.
Implementation of Google Cloud Platform
Here’s how our web tier looks like:
For the Blue-Green deployment, we need to have two copies of each service in green and blue so that the system is defined.
Before making the switch from Blue to Green, a new version of the tracking server for requests handling is added to the Green environment.
The switch is made and the servers exchange roles - green handles production requests and blue is available for rollback.
Here’s how the Nginx requests tracking graph looks like at the moment of switching:
Here’s the CPU usage graph:
Infrastructure as a code
Image Building
For image building and automation, we use packer and ansible.
Packer is easy to use and automates the creation of any type of machine image. It embraces modern configuration management by encouraging you to use automated scripts to install and configure the software within your Packer-made images. Packer brings machine images into the modern age, unlocking untapped potential and opening new opportunities. © Packer
Ansible - the simple, yet powerful IT automation engine that thousands of companies are using to drive complexity out of their environments and accelerate DevOps initiatives. © Ansible
Here’s how the code looks like:
{
"variables": {
"ssh_user": "{{env `USER`}}"
},
"builders": [
{
"type": "googlecompute",
"account_file": "../account.json",
"project_id": "<Project ID>",
"source_image_family": "debian-8",
"zone": "us-central1-b",
"machine_type": "n1-highcpu-4",
"image_name": "operations-{{isotime \"2006-01-02\"}}-v{{user `version`}}",
"image_family": "operations",
"ssh_username": "{{user `ssh_user`}}"
}
],
"provisioners": [
{
"type": "ansible",
"playbook_file": "operations.yml"
}
]
}
GCP Infrastructure
For managing the Google Cloud Platform (GCP) infrastructure, we use Terraform.
Terraform enables you to safely and predictably create, change, and improve production infrastructure. It is an open source tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned. © Terraform
resource "google_compute_instance" "mail-1" {
name = "mail-1"
machine_type = "g1-small"
zone = "us-central1-b"
disk {
image = "https://www.googleapis.com/compute/v1/projects/<project ID>/global/images/family/mail"
size = 20
}
network_interface {
network = "default"
access_config {
# Ephemeral
}
}
service_account {
scopes = [
"https://www.googleapis.com/auth/monitoring.write",
"https://www.googleapis.com/auth/logging.write",
"https://www.googleapis.com/auth/servicecontrol",
"https://www.googleapis.com/auth/service.management.readonly",
"https://www.googleapis.com/auth/devstorage.read_only",
]
}
}
Terraform blue/green environment switch:
resource "google_compute_url_map" "www-admin-urlmap" {
name = "www-admin-urlmap"
default_service = "${ var.admin-prod == "green" ? google_compute_backend_service.www-admin-green-backend.self_link : google_compute_backend_service.www-admin-blue-backend.self_link }"
}
The data was presented at #DevInnerConf by Gleb, our DevOps specialist, and Alexander, Solutions Architect.
Are you looking for enterprise-level solutions?
