Bryan Kendall

Terraform
09 Apr 2016

I had a bit of free time this afternoon and a task from work to explore some various… “container running services”. More or less, this requirement is that our applications (contained in Docker containers) is run in a cluster of servers in order to be able to horizontally scale the infrastructure.

One of these tools that I discovered is Nomad by HashiCorp. While thinking about setting up a full environment, I cringed at the thought of manually setting up a VPC, creating Instances, attaching IPs, and etc., that you have to do in AWS. One of the tools we currently use is Ansible to do some configuration, but it doesn’t really set up our infrastructure.

It happens that HashiCorp has another project Terraform that does infrastructure management and works with AWS. I figured I would give it a shot and see how much work it was to set up. Turns out, it’s not too bad.

The Goal

Let me lay out the infrastructure for which I am aiming:

  1. A VPC (/16)
  2. with two Subnets (/24)
  3. 3 Nomad masters (each with Elastic IPs)
  4. 3 Nomad slaves
  5. Appropriate security groups for all servers

Here I am going to describe what I made in Terraform to make this happen.

Basics

To get everything started, we need to get some variables set up:

variable "access_key" {
  default = "ACCESS_KEY"
}

variable "secret_key" {
  default = "SECRET_KEY"
}

variable "region" {
  default = "us-west-1"
}

variable "az" {
  default = "us-west-1a"
}

variable "master_servers" {
  default = 3
}

variable "slave_servers" {
  default = 3
}

variable "instance_size" {
  default = {
    master = "t2.small"
    worker = "m3.medium"
  }
}

variable "ubuntu_amis" {
  default = {
    us-west-1 = "ami-06116566"
  }
}

provider "aws" {
  access_key = "${var.access_key}"
  secret_key = "${var.secret_key}"
  region = "${var.region}"
}

This provides some sane defaults for me. The are pretty self-explanatory; the AMI ID is one that is publicly available on AWS (default Ubuntu 14.04). These variables are used throughout the rest of the components.

VPC

To give me an easy template for the VPC, I spun up a quick test VPC to get more details. There’s a surprising number of things that go into a private/public subnet, so I needed to keep track of a lot of things.

  1. The VPC
  2. 2 Subnets
  3. 2 Routing Tables (one created by the VPC initially)
  4. an Internet Gateway
  5. a NAT

We end up creating quite a few resources to make this setup possible:

# The VPC
resource "aws_vpc" "nomad_vpc" {
  cidr_block = "10.0.0.0/16"
  tags {
    Name = "nomad-vpc"
  }
}

# Public Subnet
resource "aws_subnet" "nomad_public" {
  vpc_id = "${aws_vpc.nomad_vpc.id}"
  cidr_block = "10.0.0.0/24"
  tags {
    Name = "nomad-public-subnet"
  }
}

# Private Subnet
resource "aws_subnet" "nomad_private" {
  vpc_id = "${aws_vpc.nomad_vpc.id}"
  cidr_block = "10.0.1.0/24"
  tags {
    Name = "nomad-private-subnet"
  }
}

# Route Table for the Public Subnet
resource "aws_route_table" "public" {
  vpc_id = "${aws_vpc.nomad_vpc.id}"
  tags {
    Name = "nomad public subnet routing table"
  }
}

# Associate the Public Subnet with the new Route Table
resource "aws_route_table_association" "public" {
  subnet_id = "${aws_subnet.nomad_public.id}"
  route_table_id = "${aws_route_table.public.id}"
}

# Associate the Private subnet with the Main Route Table
# (which was created by the VPC)
resource "aws_route_table_association" "private" {
  subnet_id = "${aws_subnet.nomad_private.id}"
  route_table_id = "${aws_vpc.nomad_vpc.main_route_table_id}"
}

# the Internet Gateway for the Public Subnet
resource "aws_internet_gateway" "nomad_gateway" {
  vpc_id = "${aws_vpc.nomad_vpc.id}"
  tags {
    Name = "nomad-vpc-border"
  }
}

# Elastic IP for the nat
resource "aws_eip" "nat" {
  vpc = true
}

# Specify the NAT for the private subnet (lives in the public subnet)
resource "aws_nat_gateway" "nat" {
  allocation_id = "${aws_eip.nat.id}"
  subnet_id = "${aws_subnet.nomad_public.id}"
}

# The Public Route Table needs the Internet Gateway
resource "aws_route" "public" {
  route_table_id = "${aws_route_table.public.id}"
  destination_cidr_block = "0.0.0.0/0"
  gateway_id = "${aws_internet_gateway.nomad_gateway.id}"
}

# The Private Route Table needs the NAT
resource "aws_route" "private" {
  route_table_id = "${aws_vpc.nomad_vpc.main_route_table_id}"
  destination_cidr_block = "0.0.0.0/0"
  nat_gateway_id = "${aws_nat_gateway.nat.id}"
}

Instances

Now with the VPC set up, we can start populating it with our Instances! For the instances residing in the Public Subnet, we are going to assign each of them an Elastic IP.

This does assume a Private Key already exists named “nomad-key” (and that you have it available).

# Nomad Master
resource "aws_instance" "nomad_master" {
  ami = "${lookup(var.ubuntu_amis, var.region)}"
  instance_type = "${var.instance_size.master}"
  key_name = "nomad-key"
  count = "${var.master_servers}"
  availability_zone = "${var.az}"
  vpc_security_group_ids = [
    "${aws_security_group.nomad.id}",
    "${aws_security_group.nomad_master.id}"
  ]
  subnet_id = "${aws_subnet.nomad_public.id}"
  root_block_device {
    volume_size = 50
  }
}

# Nomad Slave
resource "aws_instance" "nomad_slave" {
  ami = "${lookup(var.ubuntu_amis, var.region)}"
  instance_type = "${var.instance_size.worker}"
  key_name = "nomad-key"
  count = "${var.slave_servers}"
  availability_zone = "${var.az}"
  vpc_security_group_ids = [
    "${aws_security_group.nomad.id}",
    "${aws_security_group.nomad_slave.id}"
  ]
  subnet_id = "${aws_subnet.nomad_private.id}"
  root_block_device {
    volume_size = 500
  }
}

resource "aws_eip" "ip" {
  count = "${var.master_servers}"
  instance = "${element(aws_instance.nomad_master.*.id, count.index)}"
  vpc = true
}

Security Groups

Finally, we need everything to be able to talk with each other. There was one small caveat about Terraform’s Security Groups: they automatically remove the default egress rule that allows all outbound connections. We have to add it manually.

# Nomad Cluster group
resource "aws_security_group" "nomad" {
  name = "nomad"
  description = "nomad general security group"
  vpc_id = "${aws_vpc.nomad_vpc.id}"
  tags {
    Name = "nomad cluster"
  }
}

# Allow All Egress on Nomad Cluster
resource "aws_security_group_rule" "allow_all_egress" {
  security_group_id = "${aws_security_group.nomad.id}"
  type = "egress"
  from_port = 0
  to_port = 0
  protocol = "-1"
  cidr_blocks = ["0.0.0.0/0"]
}

# Expose the Nomad HTTP port to the Nomad Cluster
resource "aws_security_group_rule" "nomad_http" {
  security_group_id = "${aws_security_group.nomad.id}"
  type = "ingress"
  from_port = 4646
  to_port = 4646
  protocol = "tcp"
  source_security_group_id = "${aws_security_group.nomad.id}"
}

# Nomad Master group
resource "aws_security_group" "nomad_master" {
  name = "nomad-master"
  description = "nomad master sg"
  vpc_id = "${aws_vpc.nomad_vpc.id}"
  tags {
    Name = "nomad master"
  }
}

# Allow SSH to Nomad Masters from anywhere (sometimes bad idea in production)
resource "aws_security_group_rule" "nomad_ssh" {
  security_group_id = "${aws_security_group.nomad_master.id}"
  type = "ingress"
  protocol = "tcp"
  from_port = 22
  to_port = 22
  cidr_blocks = ["0.0.0.0/0"]
}

# Allow Nomad RPC calls to Masters from Cluster
resource "aws_security_group_rule" "nomad_rpc" {
  security_group_id = "${aws_security_group.nomad_master.id}"
  type = "ingress"
  from_port = 4647
  to_port = 4647
  protocol = "tcp"
  source_security_group_id = "${aws_security_group.nomad.id}"
}

# Allow Nomad SERF between Masters
resource "aws_security_group_rule" "nomad_serf" {
  security_group_id = "${aws_security_group.nomad_master.id}"
  type = "ingress"
  from_port = 4648
  to_port = 4648
  protocol = "tcp"
  source_security_group_id = "${aws_security_group.nomad_master.id}"
}

# Nomad Slave group
resource "aws_security_group" "nomad_slave" {
  name = "nomad-slave"
  description = "nomad slave sg"
  vpc_id = "${aws_vpc.nomad_vpc.id}"
  tags {
    Name = "nomad slave"
  }
}

# Allow SSH access to Slaves from Masters (to help set them up)
resource "aws_security_group_rule" "nomad_ssh_slave" {
  security_group_id = "${aws_security_group.nomad_slave.id}"
  type = "ingress"
  protocol = "tcp"
  from_port = 22
  to_port = 22
  source_security_group_id = "${aws_security_group.nomad_master.id}"
}

Output

The quickest way to get information out of AWS is to setup Output blocks for Terraform.

# Elastic IPs assigned to Nomad Masters
output "master_ips" {
  value = "${join(",", aws_eip.ip.*.public_ip)}"
}

# Private IPs assigned to Nomad Master
output "master_private_ips" {
  value = "${join(",", aws_instance.nomad_master.*.private_ip)}"
}

# Private IPs assigned to Nomad Slaves
output "slave_ips" {
  value = "${join(",", aws_instance.nomad_slave.*.private_ip)}"
}

Apply

There you go! Combine all of these values into a single file and run terraform plan and terraform apply. All the AWS resources are created and you should be able to SSH into a master and set it up!


BrandYourself