terraform & ansible for a loadbalanced site at digitalocean

Apr 6, 2025

VPS ansible devops digitalocean doctl environmnetal variables infrastructure as code (IaC) secret-tool terraform

Introduction

This post will document my notes for creating a loadbalancer in front of nginx webservers at digitalocean. As in the previous post, the goal is to make this process quick and reproducible using terraform and ansible. The material presented here is very similar to the previous post on setting up a single nginx server . However, there are some differences that I will cover in more detail, including:

  • terraform is used to setup the firewall for the machines, and
  • terraform and digitalocean create the certificate, instead of certbot.

As in the previous post, code for doing all of this is posted at github . However, remember that launching these services is not free! In this case, a basic launch includes a single-node loadbalancer ($12/month) and two nginx/ubuntu webservers ($6/month each) totalling $24/month at this time. Of course, these can be tested for an hour or two without much cost. Remember to destroy the setup when done-- thankfully terraform makes this easy.

website source

I won't go over creating the html (web site) source for this example. However, I will use the site built by the 11ty demo blog . I have previously covered how to build this site in multiple posts. For example, see this post: docker for development and build of 11ty project if you want to look over one example I have created. However you choose to create the website, it should be available locally for upload via rsync. In the github repository I have created for this example, the website code is located in the _site directory. If your website is located elsewhere, be sure to look for changes to variables defined below.

terraform for infrastructure

terraform is going to be our tool for creating infrastructure, including:

  • 2 ubuntu servers (called droplets at digitalocean),
  • a single-node loadbalancer,
  • a firewall to limit access to the ubuntu/nginx servers,
  • domain records for the domain (I'll use "example.com" as usual) and "www" subdomain, and
  • a certificate for the loadbalancer to handle https traffic.

All of this infrastructure is defined in the main.tf file and variables are assigned in the vars.auto.tfvars file. The main.tf is heavily commented to help make sense of what is being defined. Briefly,

  • The provider section at the top of the file shows that these resources will created at digitalocean and define the "token" to be accessed later. The digitalocean token should already be created at the digitalocean website, or using a tool like doctl.
  • The server(s) section defines the servers to be created: www-server-01 and www-server-02. A third server, www-server-03, is commented out. I will discuss how I would add this server if/when needed.
    • Most of the properties are set by variables, which are discussed below.
    • Notice that the do_test ssh key is added to each server. This key should already be uploaded to digitalocean and named there. This ssh key is treated as data and accessed from digitalocean in a section below.
  • The loadbalancer section adds a name, region (has to be same as the servers/droplets), a list of droplets (these are the ubuntu servers defined above), a forwarding rule that uses SSL termination (https traffic is decrypted at loadbalancer, using the certificate, and sent to servers), a health check that queries backend servers for health and eliminates "broken" servers from rotation.
  • The domain, records section declares the default domain and adds an A record for handling the "www" subdomain. This assumes that you have pointed your domain to the name servers at digitalocean.
  • The certificate section tells digitalocean to create a "Let's Encrypt" certificate for the domain and "www" subdomain. You will receive an email explaining that the certificate wil automatically be renewed by digitalocean.
  • The firewall for webservers section defines a firewall for the ubuntu servers/droplets. The droplets/servers are added in a list and rules are added to inbound and outbound traffic.
    • In this example, incoming traffic for ssh (tcp on port 22) and ping are allowed from any source. However, web traffic (tcp on port 80) is only allowed from the loadbalancer- nice!
    • Outgoing traffic from the servers is allowed to any destination.
  • The output section defines "outputs" that will print the IPs of the servers and loadbalancer when they have been created. This is useful for adding to ansible invetory, ssh to the servers after creation, etc.
  • The data section documents the ssh key that has been uploaded to digitalocean. The key can be retrieved from digitalocean by name, in this case "do_test", and uploaded to the servers created. This is done above, in the servers section.
  • Finally, the variables section defines variables that are used throught this file. The actual values of the variables are set in the vars.auto.tfvars file, shown below.
  • Some of the commented-out code shows how a 3rd server would be added to the situation, but I'll discuss that in more detail below.

main.tf

 
##
## provider
##
terraform {
  required_providers {
    digitalocean = {
      source  = "digitalocean/digitalocean"
      version = "~> 2.0"
    }
  }
}

provider "digitalocean" {
  token = var.do_token
}

##
## server(s)
##

## www-server-01
resource "digitalocean_droplet" "www-server-01" {
  image  = var.server_image
  name   = "www-server-01"
  region = var.region
  size   = var.server_size
  ipv6   = true
  ssh_keys = [
    data.digitalocean_ssh_key.do_test.id
  ]

  lifecycle {
    create_before_destroy = true
  }
}

## www-server-02
resource "digitalocean_droplet" "www-server-02" {
  image  = var.server_image
  name   = "www-server-02"
  region = var.region
  size   = var.server_size
  ipv6   = true
  ssh_keys = [
    data.digitalocean_ssh_key.do_test.id
  ]

  lifecycle {
    create_before_destroy = true
  }
}

# ## www-server-03
# resource "digitalocean_droplet" "www-server-03" {
#   image  = var.server_image
#   name   = "www-server-03"
#   region = var.region
#   size   = var.server_size
#   ipv6   = true
#   ssh_keys = [
#     data.digitalocean_ssh_key.do_test.id
#   ]
# 
#   lifecycle {
#     create_before_destroy = true
#   }
# }

##
## loadbalancer
##
resource "digitalocean_loadbalancer" "www" {
  name                   = "lb-www"
  region                 = var.region
  size_unit              = var.lb_size_unit
  redirect_http_to_https = true

  # 2 droplets
  droplet_ids = [
    digitalocean_droplet.www-server-01.id,
    digitalocean_droplet.www-server-02.id
  ]

  #  # 3 droplets
  #  droplet_ids = [
  #    digitalocean_droplet.www-server-01.id,
  #    digitalocean_droplet.www-server-02.id,
  #    digitalocean_droplet.www-server-03.id
  #  ]

  forwarding_rule {
    entry_port     = 443
    entry_protocol = "https"

    target_port     = 80
    target_protocol = "http"

    certificate_name = digitalocean_certificate.cert.name
  }

  healthcheck {
    port     = 80
    protocol = "http"
    path     = "/"
  }

  lifecycle {
    create_before_destroy = true
  }

}

##
## domain, records
##
# add domain
resource "digitalocean_domain" "default" {
  name = var.domain
}

# add an A record for www.domain
resource "digitalocean_record" "www" {
  domain = digitalocean_domain.default.id
  type   = "A"
  name   = "www"
  value  = digitalocean_loadbalancer.www.ip
}

##
## certficate
##
resource "digitalocean_certificate" "cert" {
  name    = "le-cert"
  type    = "lets_encrypt"
  domains = ["${var.domain}", "www.${var.domain}"]

  lifecycle {
    create_before_destroy = true
  }
}

##
## firewall for webservers
##
resource "digitalocean_firewall" "www" {
  name = "www-server-firewall"

  # 2 droplets
  droplet_ids = [
    digitalocean_droplet.www-server-01.id,
    digitalocean_droplet.www-server-02.id
  ]

  # 3 droplets
  #  droplet_ids = [
  #    digitalocean_droplet.www-server-01.id,
  #    digitalocean_droplet.www-server-02.id,
  #    digitalocean_droplet.www-server-03.id
  #  ]

  ## ssh from all sources
  inbound_rule {
    protocol         = "tcp"
    port_range       = "22"
    source_addresses = ["0.0.0.0/0", "::/0"]
  }

  ## http inbound from loadbalancer
  inbound_rule {
    protocol                  = "tcp"
    port_range                = "80"
    source_load_balancer_uids = [digitalocean_loadbalancer.www.id]
  }

  ## ping
  inbound_rule {
    protocol         = "icmp"
    source_addresses = ["0.0.0.0/0", "::/0"]
  }

  ##
  ## open outbound traffic
  ##
  outbound_rule {
    protocol              = "tcp"
    port_range            = "1-65535"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }

  outbound_rule {
    protocol              = "udp"
    port_range            = "1-65535"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }

  outbound_rule {
    protocol              = "icmp"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }
}


##
## output
##
# output server-01 ip
output "www-server-01" {
  value = digitalocean_droplet.www-server-01.ipv4_address
}
# output server-02 ip
output "www-server-02" {
  value = digitalocean_droplet.www-server-02.ipv4_address
}
# # output server-03 ip
# output "www-server-03" {
#   value = digitalocean_droplet.www-server-03.ipv4_address
# }

# output loadbalancer's ip
output "lb-www" {
  value = digitalocean_loadbalancer.www.ip
}

##
## data
##
# ssh key uploaded to digitalocean
data "digitalocean_ssh_key" "do_test" {
  name = "do_test"
}

##
## variables
##
variable "do_token" {
  type        = string
  description = "Personal access token setup at digitalocean."
}

variable "domain" {
  type        = string
  description = "The domain name for the server"
  default     = "example.com"
}

variable "region" {
  type        = string
  description = "Digitalocean region"
  default     = "nyc3"
}

variable "server_image" {
  type        = string
  description = "Image used as foundation for webservers"
  default     = "ubuntu-22-04-x64"
}

variable "server_size" {
  type        = string
  description = "Server size for webservers"
  default     = "s-1vcpu-1gb"
}

variable "server_count" {
  type        = number
  description = "Number of webservers to create"
  default     = 2
}

variable "lb_size_unit" {
  type        = number
  description = "Number of nodes in load balancer"
  default     = 1
}

The variables are assigned values in the vars.auto.tfvars file. The ".auto.tfvars" ending to the file name lets terraform know to load these values without extra options.

vars.auto.tfvars

 
domain       = "example.com"
region       = "nyc1"
server_image = "ubuntu-24-04-x64"
server_size  = "s-1vcpu-1gb"

lb_size_unit = 1

Briefly, the assignments to the variables above are:

  • The domain is set to "example.com",
  • the region for the droplets and load balancer is set to "nyc1",
  • the droplet/server uses an Ubuntu 24.04 LTS image,
  • a smaller droplet, 1 virtual CPU and 1 GB of RAM, is selected, and
  • a small load balancer, with 1 node, is chosen.

Of course, you can change these settings as you want. However, note that the small droplets and load balancer are the least expensive options at the moment. More nodes/droplets and larger droplets will increase the cost. So, research before you choose your droplets/balancer sizes!

running the terraform code

init: The starting point is initializing the directory. This downloads the files needed for the provider, digitalocean in the case, to be used. This is done with: 

$ terraform init

fmt: With the files in place (be sure to set domain, region, etc) it is good to format the files using: 

$ terraform fmt .

If the formating of any file(s) has changed, they will be listed as output and you should be sure to save the file with changes. When all files are formatted correctly, the output should be blank/empty as shown above.

validate: Next, validate the files using 

$ terraform validate .
Success! The configuration is valid.

If there are errors, these will be listed in the output. A successful validation will look like the example above.

plan: Next up, we get ready to launch "plan" and "apply" things. These commands require a digitalocean token. I use secret-tool to store and retrieve mine, as detailed in this post . Of course, you can do this however you like. Using my setup, the plan looks like 

$ terraform plan -var "do_token=$(secret-tool lookup token digitalocean)"

or, in general, the command looks like 

$ terraform plan -var "do_token=SUB-YR-TOKEN"

Either way, the output should be a long list of resources that will be created including 2 servers, a load balancer, domain records, a certificate, etc. It is worth your time to look over this output and make sure that it seems sensible. This is particularly true when using a new terrafrom setup.

apply: If the "plan" output looks okay, it's time to "apply" the terraform code and have digitalocean create all of the created resources: 

$ terraform apply -var "do_token=$(secret-tool lookup token digitalocean)"

or 

$ terraform apply -var "do_token=SUB-YR-TOKEN"

Terraform will ask you to type "yes" to put the plan into action, avoiding an "accidental" creation of resources :) It will likely take a two or three minutes for all of the resources to be created. Once complete, terraform will "output" the IPs for the two servers and the load balancer. These IPs can be added to the ansible inventory, as detailed below.

output: If you need this "output" repeated later, a simple: 

$ terraform output

will repeat the output information.

destroy: Finally, an important command is destroy, which destroys all of the resources created by terraform. Remember, it takes money to have all of this running, so be sure to destroy when the resources are not needed for testing or other purposes. The command is simple: 

$ terraform apply -destroy -var "do_token=$(secret-tool lookup token digitalocean)"

or 

$ terraform apply -destroy -var "do_token=SUB-YR-TOKEN"

NOTE: the deletion of the domain doesn't seem to work using terraform. I always get an error and have to delete the domain at the digitalocean website, or using doctl like so: 

$ doctl compute domain delete example.com

ansible to install and update servers and website

There are two main files for using ansible (of course, they can be split into more files, if that helps organize):

  • inventory.ini - holds information about the "inventory", including IPs, information for ssh to the servers, etc.
  • playbook.yml - holds tasks for installing software, creating users, updating servers, etc.

ansibleuser

In an effort to move away from using root, a non-root sudo account called ansibleuser will be created. This user will need an ssh key, created using 

$ mkdir ssh
$ ssh-keygen -t rsa -b 4096 -f ssh/ansible_user

The above commands will create the ansible_user public and private ssh keys in the local ssh directory. If you are using git, you probably want to add ssh/ to your .gitignore file. This has already been done in the example github repository.

inventory

In this example the inventory is pretty simple. It contains IPs for the two servers as well as infomation used by ansible to ssh to the server. This looks like:

inventory.ini

 
[www_servers]
xxx.xxx.xxx.xxx
yyy.yyy.yyy.yyy

[new_servers]


[all:vars]
ansible_user=ansibleuser
ansible_ssh_private_key_file=ssh/ansible_user
ansible_ssh_extra_args="-o IdentitiesOnly=yes"
ansible_python_interpreter=/usr/bin/python3

In this file, IPs xxx.xxx.xxx.xxx and yyy.yyy.yyy.yyy are the two servers created by terraform (see above). Remember, terraform should "output" the IPs of these servers when digitalocean has finished creating them. These servers are in the [www_servers] group. A [new_servers] groups has no IPs, but it can be used to later add a new server. Finally, the [all:vars] section givers the user, ssh key, etc that will be used by ansible. The creation of this user and key will be the first task below.

playbook.yml

The ansible playbook, called playbook.yml has all of the tasks needed to setup the ubuntu servers with nginx and upload the website, as well as other needed setup and maintainance tasks.

The playbook.yml file has lots of comments, but let's go through the sections and tasks briefly:

  • At the top of the file we say that "all" hosts should have these tasks applied and sudo should be used, indicated by the become: true line. The non-root username is given, as is the domain and location of the html source on the local machine. Be sure to change these as needed :)
  • SERVER SETUP: this section creates the ansibleuser, sets up passwordless sudo, uploads the ansible_user public ssh key, and disables password authentication for the root account.
  • SERVER UPDATE: this section updates and upgrades packages on the ubuntu servers and reboots the droplets, if needed.
  • NGINX SETUP: this section installs nginx, sets up the directories, uploads the nginx conf, enables the site etc.
  • WEB RSYNC: this section uses rsync to upload and sync the local _site directory with the appropriate nginx directory on the servers.
  • HANDLERS: this section has a single handler that restarts nginx when "notified".

playbook.yml

 
---
- hosts: all
  become: true
  vars:
    username: ansibleuser
    domain: example.com
    local_html: _site

  tasks:
  #
  # SERVER SETUP
  #
  # reference 
  # https://www.digitalocean.com/community/tutorials/how-to-use-ansible-to-automate-initial-server-setup-on-ubuntu-22-04
  #
    - name: Setup passwordless sudo
      lineinfile:
        path: /etc/sudoers
        state: present
        regexp: '^%sudo'
        line: '%sudo ALL=(ALL) NOPASSWD: ALL'
        validate: '/usr/sbin/visudo -cf %s'
      tags: [ never, setup ]

    - name: Create a new regular user with sudo privileges
      user:
        name: "{{ username }}"
        state: present
        groups: sudo
        append: true
        create_home: true
      tags: [ never, setup ]

    - name: Set authorized key for remote user
      ansible.posix.authorized_key:
        user: "{{ username }}"
        state: present
        key: "{{ lookup('file', 'ssh/ansible_user.pub') }}"
      tags: [ never, setup ]

    - name: Disable password authentication for root
      lineinfile:
        path: /etc/ssh/sshd_config
        state: present
        regexp: '^#?PermitRootLogin'
        line: 'PermitRootLogin prohibit-password'
      tags: [ never, setup ]

  #
  # SERVER UPDATE
  #
  # reference 
  # https://www.jeffgeerling.com/blog/2022/ansible-playbook-upgrade-ubuntudebian-servers-and-reboot-if-needed
  # - some commands out of date
  #
    - name: Run apt update and apt upgrade
      ansible.builtin.apt:
        upgrade: yes
        update_cache: yes
        cache_valid_time: 86400 # one day, in seconds
      tags: [ never, update ]

    - name: Check if reboot is required
      ansible.builtin.stat:
        path: /var/run/reboot-required
        get_checksum: false
      register: reboot_required_file
      tags: [ never, update ]

    - name: Reboot the server (if needed)
      ansible.builtin.reboot:
      when: reboot_required_file.stat.exists == true
      tags: [ never, update ]

    - name: Autoremove deps that are no longer needed
      ansible.builtin.apt:
        autoremove: true
      tags: [ never, update ]

  #
  #
  # NGINX SETUP
  #
  #
    - name: Update apt and install required system packages
      apt:
        pkg:
          - nginx
        state: latest
        update_cache: true
      tags: [ never, nginx_setup ]

    - name: Ensure Nginx is runnning
      service:
        name: nginx
        state: started
        enabled: yes
      tags: [ never, nginx_setup ]

    - name: Create remote html directory
      ansible.builtin.file:
        path: /var/www/{{ domain }}/html
        state: directory
        mode: '0755'
      tags: [ never, nginx_setup ]

    - name: Change ownership of html directory to ansibleuser
      ansible.builtin.file:
        path: /var/www/{{ domain }}/html
        state: directory
        recurse: yes
        owner: ansibleuser
        group: ansibleuser
      tags: [ never, nginx_setup ]

    - name: Apply Nginx template
      template:
        src: ansible_files/nginx.conf.j2
        dest: /etc/nginx/sites-available/default
      notify: Restart Nginx
      tags: [ never, nginx_setup ]

    - name: Enable new site
      file:
        src: /etc/nginx/sites-available/default
        dest: /etc/nginx/sites-enabled/default
        state: link
      notify: Restart Nginx
      tags: [ never, nginx_setup ]

  #
  #
  # WEB RSYNC
  #
  #
    - name: rsync local html with server html directory
      ansible.builtin.shell:
        cmd: "rsync -av -e 'ssh -o \"IdentitiesOnly=yes\" -i ssh/ansible_user' {{ local_html }}/ ansibleuser@{{ hostvars[inventory_hostname]['ansible_default_ipv4']['address'] }}:/var/www/{{ domain }}/html/"
      delegate_to: localhost
      register: rsync_output
      tags: [ never, web_rsync ]
      vars:
        ansible_become: false

    - name: Print rsync output
      debug:
        var: rsync_output.stdout_lines
      tags: [ never, web_rsync ]
      vars:
        ansible_become: false

  #
  #
  # HANDLERS
  #
  #
  handlers:
    - name: Restart Nginx
      service:
        name: nginx
        state: restarted

running the ansible code

ssh to server: first up, it's good to ssh to the servers using the root account (be sure to use your ssh-key in the command below) and make sure that server fingerprints for the servers have been accepted: 

$ ssh -o "IdentitiesOnly=yes" -i ~/.ssh/do_test root@xxx.xxx.xxx.xxx
$ ssh -o "IdentitiesOnly=yes" -i ~/.ssh/do_test root@yyy.yyy.yyy.yyy

server-setup: first up we use the root account to setup the server (again, be sure to change to point to your ssh-key): 

$ ansible-playbook -i inventory.ini playbook.yml --tags "setup" \
  -e "ansible_user=root ansible_ssh_private_key_file=~/.ssh/do_test"

the --tags "setup" flag makes sure only the "setup' sections of the playbook are run. Once this section finishes, ansibleuser should be ready to use.

server-update: next we update the server and reboot if needed. A new digitalocean droplet often has many updates that need to be done, and the first update often takes a few minutes becaude of this, so this is a good place to start: 

$ ansible-playbook -i inventory.ini playbook.yml --tags "update"

Notice that this command is run as ansibleuser because of the variables set in the inventory file.

nginx-setup: next we install nginx and do all of the configuration with: 

$ ansible-playbook -i inventory.ini playbook.yml --tags "nginx_setup"

web-rsync: a final step is to upload the code for the website. This is done using: 

$ ansible-playbook -i inventory.ini playbook.yml --tags "web_rsync"

server maintainance

After all on the setup above, things should be good to go. A couple of commands are useful for reuse. First, update the server on whatever schedule you like using: 

$ ansible-playbook -i inventory.ini playbook.yml --tags "update"

Next, went there are update to the website code, sync those using: 

$ ansible-playbook -i inventory.ini playbook.yml --tags "web_rsync"

adding a server

If you're getting lots of traffic to your site, congrats! As an example of increasing capacity, a third server ( or more ) can be added to the infrastructure by following these steps:

  • Uncomment the code for "www-server-03" and its "output" section in the main.tf file and use terraform to spin this up with the usual: 
    $ terraform fmt .
    $ terraform validate .
    $ terraform plan -var "do_token=SUB-YR-TOKEN" 
    $ terraform apply -var "do_token=SUB-YR-TOKEN"
  • Make sure that ssh to the new server is working using ( assuming IP is zzz.zzz.zzz.zzz ): 
    $ ssh -o "IdentitiesOnly=yes" -i ~/.ssh/do_test root@zzz.zzz.zzz.zzz
  • Use ansible to setup and provision the new server by adding it's IP to the new [new_servers]

    inventory.ini

     
    [www_servers]
    xxx.xxx.xxx.xxx
    yyy.yyy.yyy.yyy

    [new_servers]
    zzz.zzz.zzz.zzz

    [all:vars]
    ansible_user=ansibleuser
    ansible_ssh_private_key_file=ssh/ansible_user
    ansible_ssh_extra_args="-o IdentitiesOnly=yes"
    ansible_python_interpreter=/usr/bin/python3
    section and running 
    $ ansible-playbook -i inventory.ini playbook.yml --tags "setup" --limit "new_servers"\
      -e "ansible_user=root ansible_ssh_private_key_file=~/.ssh/do_test"
    $ ansible-playbook -i inventory.ini playbook.yml --tags "update"
    $ ansible-playbook -i inventory.ini playbook.yml --tags "nginx_setup" --limit "new_servers"
    $ ansible-playbook -i inventory.ini playbook.yml --tags "web_rsync"
  • In the main.tf file, add the new server to the list of "droplet_ids" in both the loadbalancer and the firewall sections. Examples for this are commented out near the appropriate parts of the file. When this is done, run the usual terraform commands to change the resources: 
    $ terraform fmt .
    $ terraform validate .
    $ terraform plan -var "do_token=SUB-YR-TOKEN" 
    $ terraform apply -var "do_token=SUB-YR-TOKEN"
  • Once this is complete, move the new servers IP from the [new_servers] to the [www_servers] section of the inventory file so the update and syncs can be done for all three servers easily.

    inventory.ini

     
    [www_servers]
    xxx.xxx.xxx.xxx
    yyy.yyy.yyy.yyy
    zzz.zzz.zzz.zzz

    [new_servers]

    [all:vars]
    ansible_user=ansibleuser
    ansible_ssh_private_key_file=ssh/ansible_user
    ansible_ssh_extra_args="-o IdentitiesOnly=yes"
    ansible_python_interpreter=/usr/bin/python3

that's all folks

That's all my notes for now. Remember to look at the github repository for the complete files. I have also added a Makefile that can be used to run the commands. I find it very helpful to have all of the main commands organized this way.

prev post »